Lewati ke isi

Runbook Cutover Big-Bang

Version: 1.0
Last Updated: 2026-05-26
Status: DRAFT — Requires team review

📋 Overview

Dokumen ini berisi prosedur lengkap cutover big-bang migrasi dari MariaDB legacy ke PostgreSQL + ClickHouse.

Timeline: Idealnya dilakukan saat tutup buku (akhir bulan/tahun) untuk minimalkan disruption.

⚠️ Prerequisites

Gate Cutover (WAJIB SEMUA ✅)

# Checkpoint Status Owner
1 Rekonsiliasi saldo, stok, jurnal 100% per tenant prioritas [ ] DBA
2 UAT operasional PG + API + Flutter pada tenant pilot [ ] QA
3 Runbook cutover + rollback teruji [ ] Lead
4 Legacy MariaDB read-only / arsip [ ] DevOps
5 OpenAPI + versi klien terkunci untuk rilis cutover [ ] Backend Lead
6 Semua tenant sudah di-initial load ke ClickHouse [ ] DBA
7 CDC consumer running dengan lag < 1000 [ ] DevOps
8 Backup PostgreSQL + ClickHouse pre-cutover [ ] DevOps

📅 Timeline Cutover

H-7 (1 Minggu Sebelum)

  • [ ] Final reconciliation semua tenant
  • [ ] UAT sign-off dari business user
  • [ ] Communication ke stakeholder (downtime window)
  • [ ] Backup full database legacy

H-1 (1 Hari Sebelum)

  • [ ] Pre-cutover backup: 
    # PostgreSQL
pg_dump -h localhost -U erp_owner erp > erp_pre_cutover.sql

# ClickHouse
clickhouse-client --query "SELECT * FROM erp_clickhouse.transaksi" > ch_transaksi.csv
  • [ ] Verify backup size dan integrity
  • [ ] Notify user: "Sistem akan downtime besok jam XX:XX"

H (Cutover Day)

Phase 1: Freeze Legacy (T=0)

# 1. Set legacy MariaDB to READ-ONLY
mysql -u root -p -e "SET GLOBAL read_only = ON;"

# 2. Stop semua aplikasi yang connect ke legacy
pm2 stop legacy-backend

# 3. Verify tidak ada active connection
mysql -u root -p -e "SHOW PROCESSLIST;" | grep -v Sleep

Duration: 15 menit

Phase 2: Final Delta Sync (T+15min)

# 1. Run ETL delta terakhir
cd /home/sic/Projects/migrasi/database/etl
python3 etl_final.py --tenant=all --delta-only

# 2. Verify counts match
python3 ../scripts/rekonsiliasi-kpi.py --tenant=all

Acceptance Criteria: - PG vs CH count match ✅ - Jurnal tidak imbang = 0 ✅

Duration: 30-60 menit (tergantung volume)

Phase 3: CDC Consumer Start (T+1hr)

# 1. Verify publication complete
psql "$ERP_MIGRASI_URL" -c "SELECT * FROM pg_publication_tables WHERE pubname='debezium_erp_pub'"

# 2. Start CDC consumer
pm2 restart clickhouse-etl --update-env
pm2 restart cdc-monitor

# 3. Monitor lag
watch -n 5 'sudo docker exec cdc-kafka-1 kafka-consumer-groups --bootstrap-server localhost:9092 --group clickhouse-etl-group --describe'

Acceptance Criteria: - Lag Kafka < 1000 ✅ - Consumer running tanpa error ✅

Duration: 30 menit

Phase 4: Switch Backend (T+1.5hr)

# 1. Update backend environment
cd /home/sic/Projects/migrasi/backend
# Update .env: DATABASE_URL → PostgreSQL

# 2. Restart backend
pm2 restart erp-backend

# 3. Health check
curl http://localhost:3000/api/health | jq .

Duration: 15 menit

Phase 5: Flutter App Switch (T+2hr)

# 1. Force update Flutter app (via Firebase App Distribution)
# 2. Monitor crash rate di Firebase Crashlytics
# 3. Verify API calls successful

Duration: 30 menit

Phase 6: Validation (T+2.5hr)

# 1. Run validation script
python3 /home/sic/Projects/migrasi/database/scripts/rekonsiliasi-kpi.py --tenant=all

# 2. Manual spot check
# - Login ke Flutter app
# - Create transaksi test
# - Verify muncul di dashboard < 10 detik

# 3. Business user sign-off

Duration: 30 menit

Phase 7: Go-Live (T+3hr)

# 1. Notify stakeholder: "Sistem LIVE"
# 2. Monitor dashboard:
#    - CDC lag
#    - API response time
#    - Error rate
# 3. Standby support team 24 jam

Duration: Ongoing

🔄 Rollback Procedure

TRIGGER ROLLBACK JIKA: - Critical bug yang block operasional > 30 menit - Data corruption terdeteksi - CDC lag > 10000 selama > 1 jam

Step 1: Stop New System

# Stop backend
pm2 stop erp-backend

# Stop CDC
pm2 stop clickhouse-etl
pm2 stop cdc-monitor

Step 2: Restore Legacy

# Set legacy back to READ-WRITE
mysql -u root -p -e "SET GLOBAL read_only = OFF;"

# Restart legacy backend
pm2 start legacy-backend

Step 3: Notify

"Rollback dilakukan. Sistem kembali ke legacy. 
Tim teknis akan investigasi issue."

Step 4: Post-Mortem

  • Root cause analysis dalam 24 jam
  • Fix issue
  • Re-schedule cutover

📊 Monitoring Dashboard

Pre-Cutover

Metric Target Current
PG vs CH count match 100% [ ]
Jurnal tidak imbang 0 [ ]
CDC lag < 1000 [ ]
API response time < 500ms [ ]

During Cutover

Metric Target Current
Downtime duration < 4 jam [ ]
Data loss 0 rows [ ]
Rollback triggered No [ ]

Post-Cutover (Week 1)

Metric Target Current
CDC lag < 1000 [ ]
API error rate < 0.1% [ ]
User complaints < 5 [ ]

📞 Contact List

Role Name Contact
Cutover Manager [Name] [Phone]
DBA Lead [Name] [Phone]
Backend Lead [Name] [Phone]
DevOps Lead [Name] [Phone]
Business Owner [Name] [Phone]

✅ Post-Cutover Checklist

Week 1

  • [ ] Daily reconciliation PG vs CH
  • [ ] Monitor CDC lag < 1000
  • [ ] Review error logs daily
  • [ ] User feedback collection

Month 1

  • [ ] Performance benchmark vs legacy
  • [ ] Cost analysis (infra cost)
  • [ ] Decommission legacy servers (after 30 days)

Appendix: Runbook Cutover Big-Bang