Skip to content

Prod Cutover Runbook — France → new Mumbai (New Prod)

✅ CUTOVER COMPLETE — 2026-07-04 ~09:05–09:30 IST (Attempt 2)

Prod is now New Prod 147.93.168.216. Sequence executed: DNS (4 records) → active-call gate (waited for 0) → NUC wg0 flip (handshake 3 s) → staging wg0 (was pointing at the DELETED old Mumbai 82.180.146.80 — caught & fixed) → France frozen (asterisk stopped to force client re-resolution — converged 31→62 contacts in ~2 min; wg0 down; backups paused; watchdogs off; OpenVPN kept UP as ATA access path) → NP watchdogs on (scripts had to be copied from France — missing from image). Live-verified: answered queue calls (10 completed within 25 min), CDR→ODBC writing, recordings landing, editor/devpbx/gateway 200 via Cloudflare, fail2ban banned a live attacker, GeoIP active. Post-flip fixes: cdr-csv dir, watchdog scripts. Key learning: a mid-flip "all SIP users unregistered" panic is the expected convergence window — stopping the OLD box's asterisk is what accelerates it; do NOT revert DNS for it. Outstanding: V7 router re-handshake (repointed, handshook once 09:10, needs reboot) and ATA migrations — audit proved V7 + 4 ATAs had pointed at the deleted Mumbai IP since 2026-07-03 (only ACR was on France): they were already down pre-cutover; 4 ATAs need hospital-LAN access to repoint (unreachable remotely).

Adopted 2026-07-04. The one-shot flip of live prod from the France box back to a faithful Mumbai restore. Work top-to-bottom; each phase has a Gate and a Rollback.

The three boxes — never confuse them

Name IP Role at cutover
New Prod 147.93.168.216 (vmi3415819) the faithful Mumbai restore we flip TO
Old Prod (France) 89.116.31.109 (vmi3132044) currently live; becomes frozen rollback
Staging VPS 94.136.188.221 unchanged

Where New Prod stands (verified 2026-07-04)

  • Telephony proven by a real call: softphone registered (e343xhubaae, 41 ms) + *43 echo call ran through the dialplan. features show = ## / *2 / *5 (what France lost).
  • Services up: asterisk, mariadb, postgresql, redis, nginx, wg0 (NUC), wg1 (customer), openvpn astradial-ata (tun0), exim, pm2 apps (astrapbx/editor/workflow-engine).
  • Fixes applied (real gaps found in testing): libgsm1→codec_gsm (prompt transcode gsm→ulaw/alaw), libsrtp2-1→res_srtp (mobile/WebRTC), libspeex1/libspandsp2/libiksemel3, Tata gateway pinned allow=alaw (Tata rejects ulaw), fail2ban asterisk-pjsip jail, GeoIP-India SIP firewall (india_sip ipset + eth0 DROP rules, tcp+udp).
  • pipecat IS a soft cutover-blocker (learned in Attempt 1): the editor dashboard depends on the gateway (7860) — when pipecat is down, the dashboard 502s and shows no orgs. Must be UP before cutover.
  • Temp preview DNS (delete after cutover): sip.astradial.com (DNS-only→New Prod), my.astradial.com (proxied→New Prod; editor vhost has a temp server_name add).

⚠️ Attempt 1 — 2026-07-04 — ROLLED BACK (audited post-mortem)

Phase 0 completed; DNS flipped (devsip/devpbx/editor/gateway → New Prod); rolled back within minutes — France was never actually down (the NUC wg0 was never flipped, so Tata hospital inbound stayed on France throughout — 2 live calls the whole time). Rollback (clean): revert the 4 DNS records → 89.116.31.109; endpoints re-converge in ~1–2 min.

Definitive root cause (full audit, evidence-verified)

One missing MariaDB user took down the whole cutover

New Prod MariaDB was built with ONLY the pbx_api user (pipecat was "deferred")
  → gateway .env user=pipecat → ERROR 1045 Access denied for 'pipecat'@'localhost'
  → gateway init_db() fails → 8 gunicorn workers crash-loop (NRestarts=33) → :7860 dead
  → nginx maps editor.astradial.com/api/gateway/* → 127.0.0.1:7860  (hard dependency)
  → dashboard renders the Cloudflare 502 body + empty org list
  → read as "prod is down" → abort/rollback
Evidence: France mysql.user has pbx_api, pbx_api_user, pipecat; New Prod has only pbx_api. Live auth test with the gateway's own .env creds → ERROR 1045.

Two masking layers hid the DB bug (each fix peeled one, import crashed before init_db()):

  1. Broken venv: py3.13 (llvmlite won't build) → py3.12 rebuild → correct is py3.11.15 via uv (France's exact interpreter; uv sync with the identical uv.lock works clean).
  2. NLTK punkt_tab corrupt-zip race: pipecat imports nltk punkt_tab at module load; with -w 8 gunicorn workers all missing it, they concurrently download and trample the zip → BadZipFile at import. (France's copy is intact from Apr-02; New Prod's became valid only after many crash-loop iterations.)

Ruled OUT by audit (all byte-verified against France): DB data (22/219/63 identical), auth stack (JWT_SECRET, Firebase misssellerai, SA-key identical), and — correcting this doc's earlier claim — "3-weeks-stale code" was WRONG: editor source (app/lib/components) and workflow-engine/src are byte-identical to France; astrapbx/src differs only by the intentional IP patches. The only real drift is 2 files: pipecat-flow/gateway/{models.py,router_admin.py} (France got newer gateway code than the Mumbai image). The Jun-09/Jun-29-vs-Jul-02 mtimes were misleading.

Process failures (why it reached the flip):

  • pipecat was misclassified "AI bots only — not a cutover blocker". nginx proves the editor dashboard hard-depends on it. The one gate that would have caught this — login to the dashboard on the preview hostname and see the 22 orgs render — was never run.
  • Execution deviated from this runbook's order (UFW → NUC → DNS together) to "DNS-first, watch registrations": that created the softphone split-brain (endpoints drifting to New Prod while inbound stayed on France) which amplified the "prod is down" perception.

Prerequisites for Attempt 2 (Phase 0, before any flip)

  • Create the missing MariaDB users on New Prod — DONE 2026-07-04: pipecat@localhost + pbx_api_user@localhost replicated from France via SHOW CREATE USER/SHOW GRANTS (password hashes, piped host→host, no plaintext). Gateway DB auth verified OK.
  • Sync the 2 drifted gateway files — DONE 2026-07-04: gateway/{models.py,router_admin.py} byte-identical to France; pipecat-flow active, NRestarts=0, 8 workers, /health 200.
  • Parity audits (config that lives OUTSIDE dumps/images) — DONE 2026-07-04: mysql.user ✔ (fixed), PostgreSQL roles identical, per-app .env key sets: editor/workflow/ pipecat identical, astrapbx = NP strict superset (extras: APNS push + SMTP — France lacks mobile push), nginx vhosts match (+harmless default), asterisk was disabled at boot → enabled, crons: added move-recordings.sh hourly + GeoIP monthly + image-backup daily (France-only: vaultwarden backup — stays on France; runner unit — CI/CD deferred). CI/CD runner DONE 2026-07-05: mumbai-prod-astradial-platform registered on 147.93.168.216 (/opt/runners/astradial-platform, root systemd, v2.335.1, label production) — one runner serves all 4 prod deploy workflows. Pre-flight audit (Fable 5) confirmed prod == main (zero drift, no pending migrations) before enabling deploys.
  • THE gate that was missed: editor E2E on my.astradial.com → New Prod — PASSED 2026-07-04: all 22 orgs render, Services & Schedulers "All up", no 502.
  • Pre-flip functional audit — PASSED 2026-07-04 (Hari-mandated, full France-vs-NP runtime comparison): endpoint sets identical 230=230 after regenerating all 22 orgs' configs from the freshly-synced DB via POST /organizations/:id/regenerate (x-internal-key) — this caught + fixed: post-image org org_mr3ay0u4 (dialplan+trunk was MISSING on NP), 2 post-image devices (e1001wx5c9q London Ortho, e1007ceaz8y Salem Gopi), 2 stale deleted users removed; MOH 23=23 classes after syncing /var/lib/asterisk/{moh,sounds/greetings} + /var/spool/asterisk/voicemail from France; queues/transports/wg-peers/OVPN-CCDs/ODBC/redis/ rclone identical; Tata allow=alaw, *5, transport externals all survived regen. Features prove NP > France: NP has ##/*2/*5; live France still lacks *2+*5 (degraded). LESSON for the runbook: generated asterisk configs are a DB-derived artifact — after any DB restore/sync, regenerate all org configs; and client audio (moh/greetings/voicemail) is file-based state that must be synced separately. Known accepted deltas: stale org_mqzglv1q conf files on NP (deleted org — cleanup post-cutover), vaultwarden stays France-only.
  • Tighter flip sequence — flip NUC wg0 + the 4 DNS + V7's wg1 together in the same minute (runbook order, no observation gap). Tiered abort criteria: web-layer glitch → fix forward on New Prod (telephony unaffected); only a telephony failure triggers full rollback.
  • ATA hospitals — the 5 OpenVPN ATAs stay on France (access bridge); their inbound breaks if the cloud moves but they don't → migrate their ATAs onto New Prod's OpenVPN (access shifts to New Prod's tunnel) OR hold those 5 DIDs on France during the interim.

Symptom-indexed entry: Troubleshooting → Error 84.

Phase 0 — Pre-cutover (do in advance, no downtime)

  • Lower DNS TTL on devsip, devpbx, gateway, editor.astradial.com to 60 s (so the flip — and any rollback — propagates in a minute).
  • Arm New Prod's OWN backups (right now only France runs them; New Prod must before it is prod): deploy astradial-backup.sh + astradial-backup-check.sh crons (:05 db / 02:30 full / :35 check), astradial-image-backup.sh daily, and the monthly restore drill. Verify one cycle lands in gcs:astradial-prod-backups. Gotcha: France's rclone gcs: remote needed bucket_policy_only = true; New Prod's already has it (set during restore).
  • Arm monitoring on New Prod: Netdata, host-timer probe, nuc-linkqual. Leave nuc-watchdog/nuc-linkqual DISABLED until the NUC actually points here (else false "NUC down" alerts).
  • Finish pipecat (or accept bots-down at cutover): uv sync rebuild of /opt/pipecat-flow/.venv, then revert the systemd drop-in to uv run (or point ExecStart at the rebuilt .venv/bin/gunicorn).
  • Confirm Contabo panel hardening (why prod was lost): 2FA, no delete rights for team members, servers renamed PROD-/SECTEST-/STAGING-.

Gate: New Prod backup cycle verified in GCS; monitoring armed; pipecat resolved or accepted.

Phase 1 — Fresh final data sync (start of the quiet window)

The 18:39 restore DB is stale by cutover — capture the delta from live France:

  • On France, run astradial-backup.sh db (hourly cron already does this) → note the newest gcs:astradial-prod-backups/db/hourly/pbx_api_db-*.sql.gz + pgdb/hourly/....
  • On New Prod, reload from that freshest dump: zcat <newest pbx_api_db>.sql.gz | mysql pbx_api_db and the PG equivalent into workflow_db (role workflow already exists).
  • Regenerate per-org Asterisk configs if the DB changed materially (configDeploymentService / AMI reload), or trust the restored /etc/asterisk if no config-affecting changes since restore.

Gate: mysql -N -e "SELECT COUNT(*) FROM pbx_api_db.organizations" on New Prod matches France; CDR/tickets current.

Rollback: none needed yet — nothing user-facing has changed.

Phase 2 — The flip (one shot)

Order matters: open the door on New Prod, then send traffic.

  • Open UFW SIP/RTP to the world on New Prod (remove the temp per-IP allow rules; GeoIP + fail2ban now protect it): ufw allow 5080/udp, 5060/udp, 8089/tcp, 10000:20000/udp, plus WireGuard 51820/51821 (already open), OpenVPN 1194.
  • Repoint the NUC's WireGuard Endpoint from 89.116.31.109 → 147.93.168.216 (the cloud pubkey is identical — both restored from Mumbai's key — so only the Endpoint IP changes). wg-quick down wg0 && up wg0 on the NUC. This moves Tata hospital inbound.
  • Repoint the reseller (Skylink FusionPBX) WireGuard Endpoint → 147.93.168.216.
  • Repoint DNS → 147.93.168.216: devsip, devpbx, gateway, editor.astradial.com.
  • Enable nuc-watchdog / nuc-linkqual on New Prod now that the NUC points here.
  • Remove the temp preview hack: delete my.astradial.com from the editor vhost server_name; retire sip.astradial.com / my.astradial.com records (or keep as spares).

Gate: proceed to verification immediately.

Phase 3 — Verify (the full path)

  • Hospital inbound: place a real PSTN call to a hospital DID → rings via Tata→NUC→(wg0)→New Prod → correct org context. pjsip show contacts shows customer endpoints registering to New Prod.
  • Features: *2 attended transfer, *5 agentnum, queue ring, IVR greeting audio plays (codec_gsm fix).
  • Outbound via Tata (alaw) connects with two-way audio.
  • Reseller (Skylink) inbound+outbound over wg1.
  • Recordings land in GCS; CDR rows write (func_odbc→MariaDB).
  • Web: editor.astradial.com loads + logs in (Firebase auth domain now matches), org list populated; devpbx API healthy.
  • Mobile app registers (WSS 8089 + res_srtp) and receives a VoIP-push call.
  • Protection live: a non-India SIP probe is dropped (GeoIP); repeated auth fails get fail2ban-banned.

Rollback (fast, ≤ a few min): DNS devsip/devpbx/gateway/editor back to 89.116.31.109; NUC + reseller WireGuard Endpoint back to 89.116.31.109; wg-quick down/up. Low TTL makes this quick. France was untouched and stays authoritative.

Phase 4 — Post-cutover

  • Freeze France as rollback ≥ 1 week — do NOT change or decommission it. Its hourly backups can be paused to avoid dueling writes to the shared GCS bucket (or point New Prod's backups to the canonical path and stop France's).
  • After the soak, France returns to security-test-only (per CLAUDE.md — never delete).
  • Un-defer the off-account backup copy (a 2nd GCS project or B2/S3) — the accidental deletion is exactly the risk it hedges.
  • Finish pipecat (bots) if deferred; re-enable its editor /api/gateway routes.
  • Update CLAUDE.md / docs: prod IP is now 147.93.168.216 (the 82.180.146.80 references are historical). DONE 2026-07-05 — swept 50 operational/reference docs 82.180.146.80 → 147.93.168.216; kept the IP in the 6 historical records (this doc, prod-india-migration, lessons-learned, recovery-and-scale-roadmap, astrapbx-commissioning-log, troubleshooting).