Skip to content

Prod migration: France → Mumbai (India) — context & runbook

Status: ✅ CUTOVER COMPLETE — Mumbai 82.180.146.80 is now PROD. France 89.116.31.109 kept running as rollback only.

⚠️ Prod identity changed. Prod is now Mumbai 82.180.146.80, not France 89.116.31.109. CLAUDE.md and other docs that still say "prod = 89.116.31.109" need a follow-up sweep. SSH: root@82.180.146.80.

This is a live handoff doc. The migration + cutover are done; what follows is the runbook (for reference), the post-cutover fix log, and the remaining tail items.

Cutover outcome (verified)

  • Hospitals: Tata → NUC → Mumbai, India-local 17 ms, 0% loss (was ~200 ms + 5–8% loss on France). MOH, greetings, prompts, recordings all working.
  • V7 hotel: tunnel up (30 ms); IP phones registering over wg1.
  • DBs (pbx_api_db + workflow_db), apps (astrapbx/editor/workflow-engine/pipecat gateway), dashboard, recordings playback, GCS+Firebase backups, ufw+fail2ban+geoip, WireGuard wg0+wg1 — all working & boot-persistent.

Post-cutover fixes & lessons (things missed in the first copy, caught & fixed)

  1. Audio assets — /var/lib/asterisk/sounds (greetings/TTS/digits) + /var/lib/asterisk/moh weren't in the initial copy → MOH/greetings/prompts silent. Copy them; Asterisk loads MOH classes only after the files exist (restart or moh reload post-copy).
  2. PostgreSQL — workflow-engine uses PG (workflow_db), not just MariaDB. Easy to miss; install PG17 + migrate role(with hash)+data.
  3. pipecat venv — uv venv symlinks to a uv-managed Python 3.11 interpreter under ~/.local/share/uv/python; copying /opt/pipecat-flow alone isn't enough (interpreter + uv cache needed). The :7860 "gateway" is also the dashboard's admin API, not just the voicebot — don't leave it down.
  4. rclone remotes — copying France's rclone.conf (firebase remote) overwrote the gcs remote → GCS backups silently failed (script suppresses rclone errors). Keep both remotes; firebase GCS-type remote needs service_account_file=/opt/astrapbx/firebase-sa-key.json.
  5. GeoIP SIP firewall ORDER (caused a V7 outage) — a blanket --dports 5060,5080 -j DROP inserted at the TOP of INPUT drops tunnel SIP (wg1) before the per-interface ACCEPT. Correct rule is eth0-scoped only: -i eth0 ... -m set ! --match-set india_sip src -j DROP — physically cannot touch wg0/wg1. Persist via a small boot service after ipset-restore.
  6. V7 GDMS peer — duplicate public key — adding a 2nd peer (Astradial-Mumbai) with the SAME pubkey as Astradial-Cloud does NOT work; WireGuard keys peers by pubkey, so the GWN router ignores the duplicate. Fix: edit the existing peer's Endpoint Address France→Mumbai (keep key/PSK/AllowedIP). Router needs a reload (toggle peer or reboot) to switch a live tunnel's endpoint.
  7. Watchdog conflict — France's nuc-watchdog/nuc-linkqual keep alerting "NUC DOWN" after cutover (NUC moved to Mumbai); both ntfy the same topic as Mumbai's → flapping Down/Up. Stop France's monitors after cutover. nuc-linkqual now has NOALERT_LABELS="V7" (logs V7 but no pages — hotel BSNL link flaps).
  8. Reaching V7 LAN phones to reconfigure — Mumbai routes to 192.168.0.0/24 over wg1. SSH-forward through Mumbai works for HTTP (port 80) but HTTPS/443 stalls (nested-tunnel MTU; wg1 MTU 1420). Use ssh -L <port>:192.168.0.x:80 + http://localhost:<port>. Phones should point SIP server at the tunnel IP 10.20.0.1 (constant across migrations — never needs reconfig), not a public IP.

Host-level gaps that broke calls org-by-org (caught later — diff against the still-alive France box)

These three live outside the astradial-platform deploy (which only rsyncs api/ + regenerates org configs), so they were silently absent until calls failed. Symptom pattern: every org behaved differently — the tell that it's environment, not app config. France was the reference for all three.

  1. unixODBC never set up → no call logs (all orgs). /etc/odbc.ini + /etc/odbcinst.ini were EMPTY and the MariaDB ODBC driver was not installed → res_odbc had 0 connections → Adaptive ODBC CDR couldn't write asterisk_cdr AND func_odbc (SPAM_BLOCKED / STICKY_AGENT / USER_OUTBOUND_DID) failed. Fix: apt install odbc-mariadb; write /etc/odbcinst.ini ([MariaDB Unicode] Driver=libmaodbc.so) + /etc/odbc.ini ([asterisk-connector] Server=localhost/Database=pbx_api_db/Socket=/run/mysqld/mysqld.sock) matching staging; module reload res_odbc.so. (Call logs = asterisk_cdr via ODBC — there is no call_logs table.)
  2. UFW missing the RTP range → no audio for public-carrier orgs. UFW allowed 5060,5080 but not 10000:20000/udp. Tunnel media (V7 over wg1) worked via the custom wg1 accept rule; public-eth0 media (PSTN carriers, e.g. Om Chambers) was dropped → connect-then-no-audio-then-drop. Fix: ufw allow 10000:20000/udp.
  3. codec_opus not loaded → no audio for Opus orgs (V7). codec_opus.so was "Not Running" because its XML doc /var/lib/asterisk/documentation/thirdparty/codec_opus_config-en_US.xml was missing → ERROR: failed to register sorcery object type 'opus'. V7 negotiates allow=opus,... first → no codec → no audio + drop. Fix: copy that XML from France, then restart Asterisk — XML docs load ONLY at startup; xmldoc reload does nothing. Use core restart when convenient (waits for 0 active calls — never drops a live call).

  4. ffmpeg missing → call recordings unplayable. Recording itself worked (MixMonitor wrote per-leg WAVs, hourly move-recordings.sh uploaded them to Firebase — no data loss), but ffmpeg/ffprobe were not installed, so the multi-leg stitcher (stitch-recordings.js) produced stitched=0 and the playback endpoint GET /calls/:id/recording failed on all 3 tiers (all need ffmpeg). Queue calls (most calls) are multi-leg → looked like "no recordings." Single-leg recordings still played. Fix: apt install ffmpeg. Existing recordings then play via on-demand rebuild (fetch legs from Firebase + ffmpeg concat); new calls stitch via the hourly cron.

  5. WireGuard keypair COPIED to Mumbai → France impersonates prod and hijacks staging's tunnel. The migration copied prod's WG private key onto Mumbai, so France (89.116.31.109) and Mumbai (82.180.146.80) present the same pubkey mBYgadDa8w/... and both claim 10.10.10.1 on the wg0 mesh (peers: NUC .2, staging .3). Staging's /etc/wireguard/wg0.conf still pointed its "prod" peer Endpoint at the old France IP, and France kept keepaliving staging every 25s → roamed staging's endpoint back to France. Result: Mumbai→staging 10.10.10.3 = 100% packet loss / SIP qualify Unavail, while staging → 10.10.10.1 (0% loss) was silently reaching France, not Mumbai. Classic asymmetric WG — outbound-from-staging worked, inbound-to-staging failed. Effect: PSTN calls to the staging-bridge DIDs (prod [tata-did-route] Dials …@cloud-endpoint-stage to let a real DID exercise staging, e.g. 918065978001/08065978001) died with number-not-in-service, and staging queue announcements never played (the call never arrived). Debugged 2026-06-05 (symptom: "number incorrect often" + "queue announcement toggle not working"). Fix: re-point staging's prod-peer endpoint to Mumbai — live wg set wg0 peer mBYgadDa8w/… endpoint 82.180.146.80:51820 (no interface bounce) + sed the conf 89.116.31.109→82.180.146.80 (backup first) — then stop + disable wg-quick@wg0 on France (systemctl stop wg-quick@wg0 && systemctl disable wg-quick@wg0; box stays up for vuln-testing, config kept, reversible). The re-pin alone flapped back until France's WG was down. ⚠️ Do NOT bounce prod's wg0 — it carries the live NUC↔prod PSTN tunnel; scope the fix to staging / use targeted wg set. Same root lesson as item 6: never reuse a WG keypair on two live boxes on the same mesh — give each box its own key. Durable follow-up: rotate Mumbai's WG key so it no longer collides with France.

Migration checklist must cover these (and ideally move to provisioning/IaC, since a future VPS rebuild loses them again): unixODBC pkg+config, UFW RTP range, codec_opus doc+load, ffmpeg, and a DISTINCT WireGuard keypair for the new box (never copy prod's WG key — it makes the old box impersonate prod, item 13). All persist on disk now.

Remaining tail (non-blocking)

  • V7 straggler phones still hard-coded to 89.116.31.109 → repoint to 10.20.0.1.
  • GitHub Actions runners (CI/CD) — register on Mumbai (PAT held).
  • ~~codec_opus (V7 Opus FEC)~~ — was broken on Mumbai (missing doc XML + not loaded); fixed — see item 11 above. Hospitals use alaw, were unaffected.
  • Contabo snapshot of Mumbai. ~~decommission France after a stable day~~ — SUPERSEDED: France 89.116.31.109 is RETAINED as a vulnerability/security-test box — do NOT decommission (per CLAUDE.md, the authoritative source).
  • Doc sweep: update all 89.116.31.109 prod references → 82.180.146.80. Partially done (2026-06-22); the main remaining offender is operations/prod-monorepo-cutover.md (historical runbook — flagged inline, not rewritten).

1. Why we're migrating (root cause)

Customers (first reported: Thangavelu Kolathur hospital) had calls with ~2s delayed voice + very choppy audio + MOH dropping out, starting suddenly.

Root cause = packet loss + high jitter on the India↔Europe internet transit, because the call-bridging Asterisk is hosted in France (Contabo, Lauterbourg — 89.116.31.109, hostname vmi3132044) while the NUC SIP gateway and all customers are in India.

Two compounding factors: - Geographic latency (constant): ~160ms floor France↔India. The cloud box's tz is set to Asia/Kolkata but it is physically in France (confirmed ipinfo=FR + traceroute through Telia/Arelion 62.115.x). - Double-hairpin: Tata-PSTN orgs (e.g. Kolathur) have queue agents that are PSTN mobiles dialed back out the same Tata trunk. So a connected call's media crosses India→France→India twice (caller leg + agent leg, both bridged in France).

Evidence (what was proven vs ruled out)

  • Loss is on the India↔Europe leg, not the NUC's last-mile. From the NUC @50pps: →India-local (1.1.1.1/8.8.8.8) 0–0.25% loss, →France 5.6–8% loss (intermittent; samples within one hour ranged 0.6%→5.6%→7%→8% = congestion windows). Per-hop from France: every European/Tata-intl hop 0%, loss only at the far (India) end.
  • NUC's own internet is healthy — wired USB-eth behind a router on a Jio line; 38ms & ~0% to India destinations. Not the problem.
  • Ruled out — codec change: the May Opus codec-profile work (per-org profiles) only applied Opus to V7's 29 endpoints; every other org's user legs are still ulaw,alaw,g722. The live choppy Kolathur call was alaw, no Opus, no transcoding. Timing was coincidence.
  • Ruled out — CPU/RAM: Asterisk steady ~10.5% CPU during calls; kernel NIC drops = 0. Adding cores/RAM does NOT fix network loss. (High France load avg was netdata ~34% + iowait from debug,verbose(5) Asterisk logging writing ~7GB/rotation — a separate cleanup item, not the call cause.)
  • "Why now?" Nothing on our side changed (config, codec, CPU, and traffic actually dropped per netdata 7-day history). An unchanged system breaking = an external carrier/transit change on the India↔Europe path ~2 days before reporting. No public incident confirmed for the exact date; the India↔Europe subsea routes (e.g. Red Sea SMW4/IMEWE family) are a known fragility.

Conclusion: the fix is to bridge calls in India, eliminating the long-haul leg and the double-hairpin. Verified that more RAM/CPU and codec reverts do nothing for this.

2. The decision & the new box

Migrate prod (whole stack) to a Contabo VPS in the Mumbai/India region, built fresh and cut over with France left running as instant rollback. (NOT Contabo's paid region-migration — €40 + 48h + opaque; self-redeploy gives control + rollback. Real data is tiny, ~330MB.)

New box (ordered & validated): - IP: 82.180.146.80 — ssh root@82.180.146.80 (key auth works) - Contabo Cloud VPS 40, region Asia (India) / Mumbai (AS141995 Contabo Asia) - 12 vCPU / 47GB RAM / 250GB NVMe, Debian 13 (trixie) — matches prod OS - ~$44.70/mo ($30 VPS + $14.70 India surcharge), 1-month term (switch to annual once proven) - No Contabo Auto Backup (we do off-site GCS backups instead)

Network validation (the make-or-break test) — PASSED: - NUC → 82.180.146.80 UDP (RTP-representative): 0% loss, 0.3–0.8ms jitter, 16ms RTT (at 200kbps and 1mbps). - The 5% ICMP "loss" seen with ping is Contabo's ICMP policing — irrelevant to media (UDP is clean). Box firewall is open (iptables ACCEPT all). - vs France path: 185ms, intermittent 0.6–8% loss, 36–123ms jitter.

→ The Mumbai box delivers a pristine call path. Migration will fix the issue.

3. Current prod (France 89.116.31.109) inventory

  • OS/stack: Debian 13, MariaDB 11.8.6, Node 20.20.1, Redis 8.0.2, custom-built Asterisk 20.18.2 (built WITH libxslt so codec_opus loads — required for Opus profiles).
  • Apps run via pm2 (pm2-root.service): astrapbx, editor, workflow-engine, pipecat-flow (AI voicebot). Also workflow-engine.service (systemd) and GitHub Actions self-hosted runners per app (actions.runner.astradial-*) — this is how code deploys (same path used to build the India staging box 94.136.188.221).
  • nginx serves editor.astradial.com and gateway.astradial.com; TLS cert /etc/ssl/certs/astrapbx.crt; proxies to localhost :7860 (editor/admin), :3001, :3002, :8000.
  • Data to migrate (~330MB total): MariaDB pbx_api_db 56MB; recordings /var/spool/asterisk/monitor 264MB / 177 files (offloaded hourly by move-recordings.sh); /etc/asterisk 6.8MB; /etc/wireguard 44KB. (Box shows 66GB used — almost all is discardable debug logs.)
  • Crontab: 0 * * * * /opt/astrapbx/scripts/move-recordings.sh; monthly geoip update; daily vaultwarden backup.
  • WireGuard:
  • wg0 (ListenPort 51820, Address 10.10.10.1/24): peers = NUC Gateway 10.10.10.2 and India-staging 10.10.10.3 (Endpoint 94.136.188.221:51820).
  • wg1 (ListenPort 51821, Address 10.20.0.1/16): peer = V7 Tirupathur 10.20.0.2 + 192.168.0.0/24 (UCM6301). Auto-generated from customer_tunnels table.
  • ⚠️ CRITICAL cutover item — /etc/asterisk/pjsip_transport.conf hardcodes the public IP:
  • external_media_address=89.116.31.109
  • external_signaling_address=89.116.31.109
  • These must become 82.180.146.80 on the new box or SIP/RTP breaks.

4. Migration runbook

France stays live serving real calls throughout. Build + verify Mumbai in parallel, then cut over.

  1. Provision — DONE (82.180.146.80, Debian 13, validated).
  2. Base setup on Mumbai box: fix locale (en_US.UTF-8 warning), install deps, set timezone, harden SSH/firewall (open only 22, 80, 443, 5060/5080/8089 SIP, 51820/51821 WG UDP).
  3. Deploy app stack via the existing GitHub Actions runners (register runners on the new box, same as staging) — astrapbx, editor, workflow-engine, pipecat-flow under pm2. Install MariaDB 11.8, Redis 8, nginx.
  4. Build Asterisk 20.18.2 from source WITH libxslt (so codec_opus loads). Don't copy the binary — rebuild on matching OS.
  5. Restore data: mysqldump pbx_api_db from France → import on Mumbai; rsync recordings; copy /etc/asterisk (then fix pjsip externals) and /etc/wireguard (keep keys; private keys stay, only remote Endpoints change — see step 7).
  6. Edit pjsip_transport.conf → external_media_address / external_signaling_address = 82.180.146.80. Run config generator / pjsip reload.
  7. Re-point WireGuard remotes to the new IP:
  8. On the NUC: update the cloud peer Endpoint to 82.180.146.80:51820.
  9. On V7's UCM/router: update its peer Endpoint to 82.180.146.80:51821.
  10. Cloud box keeps the same WG private keys → peer public keys remain valid; only the Endpoint IP changes on the remote side.
  11. Set up GCS backups (off-site, the real protection): cron mysqldump pbx_api_db (daily or hourly) + recordings + /etc/asterisk + /etc/wireguard → GCS bucket, with retention + a tested restore. (Reuse the vaultwarden backup pattern.)
  12. Install nuc-linkqual monitor on the new box too (loss/jitter history + ntfy alerts).
  13. Test on Mumbai while France still serves: DB integrity, dashboard login (editor./gateway.), a test inbound + outbound call, recording, queue.
  14. Cutover (quiet window): final incremental pbx_api_db dump → import; re-point Tata trunk + WireGuard (NUC + V7) + DNS A-records (editor.astradial.com, gateway.astradial.com, SIP) to 82.180.146.80. Lower DNS TTLs beforehand.
  15. Rollback: if anything's wrong, point trunk/WG/DNS back to France (kept running). Decommission France after 1–2 stable days.

Build progress (Mumbai 82.180.146.80) — DONE

Hardened (fail2ban/ufw/key-only/auto-updates). Stack matched: MariaDB 11.8.6, PostgreSQL 17 (workflow-engine dep — was missed initially, now migrated), Redis 8.0.2, nginx, Node 20, pm2. Asterisk 20.18.2 = France's exact binary+modules (fresh build had codec_opus ABI issues; replicating France's /usr/sbin/asterisk + /usr/lib/asterisk + bundled libs was the fix). 152 contexts, 127 endpoints, Kolathur DID routes. Apps online: astrapbx(:8000, AMI connected), editor(:3001), workflow-engine(:3002, PG). DBs migrated: pbx_api_db (43 tbl) + workflow_db (5 tbl). IP swapped. WG configured+down. GCS backups live. nuc-linkqual installed. logger.conf debug-level dropped.

Two deferred follow-ups (NON-blocking; hospitals unaffected)

  1. codec_opus (V7 Opus FEC) — ✅ RESOLVED 2026-06-03 (root cause was a missing doc XML, not the binary; see "Host-level gaps" item 11 above and the Remaining-tail note). The historical note below is kept for context: ~~won't load on Mumbai (failed to register sorcery object type 'opus' — France's binary built it for France's exact build). Hospitals use alaw → unaffected; V7 falls back to g722/ulaw.~~
  2. pipecat-flow voicebot (/opt/pipecat-flow) — uv sync fails building llvmlite 0.43.0 (numba dep) on Python 3.13/new setuptools. France has an older already-built copy. Services stopped+disabled. Fix later: tar France's fully-resolved .venv (tar --hard-dereference) to the same path + set the unit to use the venv directly (not uv run/sync), OR pin setuptools<80, OR install the prebuilt llvmlite wheel. Not on the call path.

Open / gotchas to finish before cutover

  • Re-run the full "every reference to 89.116.31.109" scan across /etc /opt /usr/local /root and app .env files — the inventory scan timed out mid-run. pjsip externals are confirmed; check for app env vars, DNS configs, customer endpoints registering by IP vs DNS name.
  • Final DB sync at cutover (France keeps writing until the switch).
  • Customer SIP endpoints (phones/ONT/UCM): if they register to a DNS name → just update DNS; if hardcoded IP → each must be reconfigured.
  • Don't forget Redis is mostly ephemeral (can start fresh); the GitHub runners need re-registering on the new host.

5. Capacity plan (10 hospitals + 4 gated communities, ~2 months)

  • Concurrent calls are NOT the bottleneck — Asterisk bridges hundreds of G.711 calls per core (was 10.5% CPU during live calls). Realistic busy-hour peak ~100–120 concurrent; design for ~200.
  • Real sizing drivers: RAM for the multi-service stack + DB growth, and Opus-transcoding margin (gated-community/ONT legs). Bandwidth (~35Mbps @200 calls) is a non-issue vs the 800Mbit/s port.
  • VPS 40 NVMe (12c/48GB) chosen — comfortable headroom for hospital-grade + growth. NVMe chosen over SSD (IOPS for concurrent recording/DB writes; recordings offload hourly so capacity isn't needed).
  • Plan soon (hospital-grade): off-box nightly pbx_api_db backup to GCS (step 8), and eventually a warm standby / HA — a single VPS is a SPOF for all 10 hospitals.

6. Monitoring already deployed

  • nuc-linkqual.service on France cloud (/usr/local/bin/nuc-linkqual.sh; local copy AstradialDevelopment/NUC/nuc-linkqual.sh): probes NUC + V7 tunnels at 50pps every 60s, logs CSV to /var/log/linkqual/, ntfy-alerts on >2% loss / >60ms jitter (reuses topic in /etc/nuc-watchdog.env). Fills the prior "no latency history" gap. Complements the reachability-only nuc-watchdog. Install on the Mumbai box too.

Key facts / IPs

  • France prod (migrate FROM): 89.116.31.109 — Contabo Lauterbourg, FR
  • Mumbai prod (migrate TO): 82.180.146.80 — Contabo Mumbai, IN, validated
  • India staging: 94.136.188.221 (wg peer 10.10.10.3)
  • NUC SIP gateway: wg 10.10.10.2 (behind Jio, reach via nuc.astradial.com — note that hostname resolves to Cloudflare, not the NUC's real line)
  • V7 Tirupathur UCM: wg 10.20.0.2 / LAN 192.168.0.0/24
  • Org slug for Thangavelu Kolathur: org_mpdzkauf (DB org 4e753c3a-7801-4a49-b3a0-fa48bf25cf4a)