Recovery & Scale Roadmap (plan of record)¶

Adopted 2026-07-03, after the prod VPS termination. This is the active execution plan — work top-to-bottom, phase by phase. Check items off as they complete; each phase has a Gate (don't start the next phase until it passes) and a Rollback note.

Goals (the six requirements)¶

Sequencing decision (do not revisit mid-incident): restore the known monolith first; prove Kamailio on staging; only then front prod. No big-bang — deploying an unproven proxy during disaster recovery couples two failure domains.

Growth model (governs all phases): the only standing spend is the small Kamailio front box. Every media node is bought when a client signs, funded by their subscription, onboarded via the scripted node runbook (≤ half a day), cancelled on churn (loss ≈ 1 month). Nodes are cattle: all per-org state regenerates from the central DB; recordings/CDR are central. Never provision capacity ahead of revenue.

Phase 1 — Rebuild prod on a new VDS (NOW)¶

1a. De-risk the account (same day as purchase)¶

Root cause was human error, not provider action: a team member deleted the Mumbai VPS from the Contabo panel instead of the France one. The fix is access control, not abuse-desk diplomacy:

• Contabo panel: enable 2FA on the account; remove or restrict team-member access — if Contabo offers sub-accounts/roles, nobody but the owner gets delete/cancel rights on servers.
• Rename servers in the panel with unambiguous labels (PROD-mumbai-..., SECTEST-france-..., STAGING-...) so the wrong-target mistake can't repeat.
• Write the two-person rule into team practice: nobody deletes/cancels any server without a second confirmation from Hari (mirrors the internal-docs prod-change rule).
• Confirm terms in writing: monthly cancellability, notice period, setup fee, resize policy.
• Off-account backup copy (1f) still stands — it hedges account-level mistakes exactly like this one, as well as provider action.

1b. Buy the VDS¶

• Region MUST be India (Mumbai) — non-negotiable, verify the datacenter dropdown BEFORE paying. An EU box reintroduces the exact France↔India packet loss that forced the original Mumbai migration and is degrading service right now.
• Dedicated-CPU VDS line (NOT shared "Cloud VPS"), NVMe — the "Physical Cores" wording confirms it's the dedicated line; that's the structural Error 73 fix.
• Decision 2026-07-03: Contabo VDS S — 3 physical cores (AMD EPYC 7282 @ 2.8 GHz), 24 GB RAM, 180 GB NVMe, 250 Mbit/s port. Approved as node-1 (today's base load).

1c. Clean rebuild (not an image restore)¶

Clean install + regen-from-DB beats cloning a terminated box's image; the restic image stays available as an emergency fallback only.

• OS = Debian 13 (trixie) — matches France/NUC/staging (do NOT jump versions during recovery). Baseline + hardening: UFW, fail2ban (incl. asterisk jails — runbook), all DBs bound 127.0.0.1 (ransomware-incident lesson), SSH keys only, unattended-upgrades.
• Generate the en_US.UTF-8 locale — every fresh Contabo Debian ships only C.UTF-8, but SSH clients forward LC_ALL=en_US.UTF-8, causing setlocale warnings that can break apt/perl/dpkg. Standard first step on every new node (→ folds into the P5 node-onboarding runbook):

  DEBIAN_FRONTEND=noninteractive apt-get install -y locales
  sed -i 's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
  locale-gen && update-locale LANG=en_US.UTF-8
  # reconnect to clear the warning in your current shell
  

• Install stack at the pinned prod versions (per prod-india-migration): MariaDB 11.8.x (collation- sensitive — the ODBC queries assume 11.8 utf8mb4_uca1400 behavior), PostgreSQL 17, Redis 8.0.x, Node 20.x, nginx; pm2 apps from git main (/opt/astrapbx, /opt/pipecat-flow-editor, /opt/workflow-engine, /opt/pipecat-flow); WSS certs.
• Asterisk 20.18.2 — do NOT apt install. Known landmine (docs flag it twice): replicate France's exact /usr/sbin/asterisk + /usr/lib/asterisk + bundled libs, OR rebuild 20.18.2 from source WITH libxslt and verify codec_opus loads (asterisk -rx "module show like opus") before cutover. A naive fresh build hit codec_opus ABI issues. Also note OpenSSL 3 on trixie may need tls-version-min 1.1 for old-firmware ATAs (ata guide).
• Restore data from latest GCS dumps (pbx_api_db, workflow_db, Redis) per the backup runbook.
• Regenerate all per-org Asterisk configs from DB (configDeploymentService) rather than restoring config tars; restore only static/per-host files from the config tar, fixing IP-bound values for the NEW IP: external_media_address, external_signaling_address, WireGuard server config, UFW rules, .env files.
• Re-establish WireGuard: NUC peer (10.10.10.x), reseller tunnel(s) (wg-reseller), staging forwarding.

1d. Re-arm protection BEFORE cutover¶

• Backup suite crons: astradial-backup.sh (hourly db / daily full), astradial-image-backup.sh (daily restic image), astradial-backup-check.sh (hourly landing alert), monthly astradial-restore-drill.sh.
• Monitoring: Netdata (→ Netdata Cloud), host-timer probe + nuc-linkqual (monitoring), ntfy alerts, Upptime target updated to the new IP.
• Gate: one full backup cycle verified landed in GCS before any DNS cutover.

1e. Cutover (quiet window)¶

• Lower TTL on devsip / devpbx / gateway / editor records, then point to new VDS.
• Repoint NUC WireGuard.
• Verify full paths: Tata→NUC→cloud hospital inbound · outbound · queues · V7 trunk · Skylink reseller fork · mobile VoIP push · recordings landing · staging DID forward.
• Explicitly test the functions not fully verified on France: bots/Pipecat, mobile push, recording sync, WhatsApp alerts (extend this list during execution).
• France returns to security-test-only role (do NOT decommission).

Rollback: DNS back to France (low TTL) + NUC WG back. Keep France untouched until the new box has run clean for ≥1 week.

1f. Post-incident¶

• Incident writeup: docs/operations/incident-prod-vps-accidental-deletion.md (title by what broke; date as metadata; include the panel-access-control fixes from 1a as the prevention section).
• Un-defer the off-account backup copy (deferred 2026-06-28 — an account-level mistake deleting prod is exactly the risk it hedges, alongside provider action): second GCS project or different provider (B2/S3), pull-based or scoped-key push. Execution lands in Phase 8; decision is made now.

Phase 2 — Inbound resilience (parallel track; carrier lead time)¶

Execute the documented Tata DID failover:

• Send Tata the 5 failover questions (already listed in the HA doc).
• Build the cloud-side failover endpoint: pjsip identify for Tata's failover SBC IP, DID→org routes, UFW/GeoIP allow.
• Mobile hunt-group per hospital as the guaranteed tertiary.
• NUC UPS + dual-WAN work continues as already planned.

Gate: simulated NUC-down test → calls arrive via the cloud-SIP secondary (or mobile tertiary).

Phase 3 — Kamailio + RTPengine proof on STAGING¶

Per the design doc (its own rule: staging first). Small VPS (~€10/mo) in front of staging Asterisk (94.136.188.221).

• Registration through Kamailio (usrloc); endpoints unchanged except server name.
• Dispatcher routing + OPTIONS health-check ejection (kill staging Asterisk → ejected in seconds).
• Drain with live test calls: active call survives, new calls hold off / reroute.
• RTPengine media anchoring: packet capture proves all RTP flows via the front box (single-IP promise — without RTPengine, media leaks backend IPs).
• Per-tenant routing for 1–2 staging orgs (org/DID → node lookup from MariaDB).
• CDR + recording coherence for calls through the proxy.

Gate: all green before ANY prod exposure.

Phase 4 — Multi-node config-gen (main code lift, astradial-platform)¶

• Node registry table + organizations.backend_node.
• configDeploymentService.deployOrganizationConfiguration() deploys to a specific node: per-node file push + per-node AMI, targeted reloads (kills the global core reload — also shrinks mobile-push dialplan-regen blast radius).
• Kamailio reads org/DID→node from the same MariaDB.
• Every node on WireGuard to central MariaDB (func_odbc), AMI, GCS recordings.

Gate: deploying an org to node-X touches only node-X (verified on staging).

Phase 5 — Front prod with Kamailio¶

• Quiet window; devsip (low TTL) → prod Kamailio+RTPengine; the Phase-1 VDS becomes node-1, sole dispatcher backend.
• Direct-to-Asterisk DNS fallback retained until Phase 8 (Kamailio now fronts hospital calls; the fallback is load-bearing — test it).
• Script the Phase-1 provisioning steps into the node-onboarding runbook (the pay-as-you-grow recipe: base install → WG → stack → regen configs → probes/backups → join dispatcher).
• When node-2 arrives, exercise the R4 org-move playbook on a friendly tenant first: drain → regen on target → flip routing row → force re-register.

Rollback: DNS flip back to direct Asterisk. Follow internal-docs CLAUDE.md prod rules (backup → written rollback → explicit confirm) at every step.

Phase 6 — Call-center client onboarding (client-funded)¶

• Dedicated-CPU node(s) sized for 150+ concurrent — 2 mid nodes > 1 huge; G.711 end-to-end, no transcode on backends (offload to RTPengine if unavoidable).
• Org + trunk concurrent-call caps configured.
• Recording pipeline at volume: raise move-recordings.sh cadence; client pricing covers their storage (the ~$2/mo GCS cap is for base ops, not client media).
• SIPp load test to target concurrency before go-live.
• Churn procedure documented: drain → remove from dispatcher → cancel VDS (loss ≈ 1 month).

Gate: load test passes; kill -9 a backend → dispatcher ejects it, new calls reroute (R1).

Phase 7 — Per-node observability (R5)¶

• Node-onboarding checklist includes: Netdata + host-timer probe + backup suite + backup-landing check + Upptime entry + dispatcher membership.
• Synthetic end-to-end test calls that page — the biggest current gap.
• Per-trunk SIP health monitoring.
• Kamailio dispatcher-state changes → ntfy.

Gate: induced failure (stop Asterisk on a node) pages within minutes via BOTH dispatcher alert and synthetic call.

Phase 8 — State plane + control-plane HA¶

• MariaDB replica (it is now a shared dependency of ALL nodes; today backup-only, RPO ≤1h); Redis AOF + replica.
• Kamailio A/B + DMQ + SIP SRV (funded when the big client signs); then retire the direct-to-Asterisk DNS fallback.
• Off-account backup copy implemented and restore-tested (from 1f).
• Timed rebuild-node-from-DB drill alongside the monthly restore drill.

Gate: Kamailio failover drill keeps registrations; off-account restore drill green; replica lag monitored.

Risks being carried¶

• Panel/human error — prod was lost to an accidental deletion; until 1a's access controls (2FA, no delete rights for team members, unambiguous server labels) are in place, it can happen again. Hedge = France warm fallback + off-account backups.
• 4-core headroom — node-1 is sized for today's load; Pipecat bot concurrency is the first pressure point (mitigation: bots to their own VPS). New client load never lands on node-1 — clients get their own nodes.
• Kamailio is a new SPoF for hospital calls between P5 and P8 → the DNS fallback is the mitigation; keep it tested.
• RTPengine bandwidth/CPU at 150+ concurrent (~13 Mbps each way at G.711) — verify front-box plan limits before Phase 6.
• Recordings bucket not ransomware-hardened (known follow-up from the backup runbook) — matters more at call-center volume.

Appendix A — Restore backups onto a new VDS (tested recipe, 2026-07-03/04)¶

The concrete, exercised procedure behind Phase 1c/1e, captured after node-1 (147.93.168.216) was rebuilt. It supersedes the "clean rebuild + latest GCS dump" wording in 1c: a clean rebuild silently lost France's static features.conf features (blindxfer ##, atxfer *2, astradial_agentnum *5), so a restic full-image restore is the faithful path; GCS DB dumps were also stale (see A1).

A0. Backup inventory + credentials¶

rclone/restic gotchas (both hit this session): buckets use Uniform Bucket-Level Access → the rclone gcs: remote needs bucket_policy_only = true (else rclone copy / restic lock 400s with a "legacy ACL" error); restic against the image needs --no-lock.

A1. DB source — check freshness FIRST (do not blindly restore GCS)¶

France ran no DB backup cron until re-armed, so the newest GCS DB dump can be stale (07:00 at the incident, ~31h old by rebuild time). If the source box is still serving, take a fresh direct dump and load that — a GCS restore would lose hours of live data:

mysqldump --single-transaction --routines --triggers --events pbx_api_db | gzip > /root/pbx_api_db-$(date +%F-%H%M).sql.gz
runuser -u postgres -- pg_dump workflow_db | gzip > /root/workflow_db-$(date +%F-%H%M).sql.gz

A2. Restic image selective restore (engine + prod-defining trees)¶

export RESTIC_REPOSITORY=rclone:gcs:astradial-prod-image/restic
export RESTIC_PASSWORD_FILE=/root/.restic-prod.pass          # from the Secrets file
export RCLONE_CONFIG=/root/.config/rclone/rclone.conf        # gcs remote, bucket_policy_only=true
restic snapshots --no-lock                                   # newest = Mumbai vmi3341755 (e.g. 61e483c5)
restic restore <snap> --no-lock --target / \
  --include /etc/asterisk --include /etc/openvpn --include /etc/odbc.ini --include /etc/odbcinst.ini \
  --include /usr/sbin/asterisk --include /usr/lib/asterisk \
  --include /var/lib/asterisk --include /var/spool/asterisk \
  --include /opt/astrapbx --include /opt/pipecat-flow-editor --include /opt/workflow-engine --include /opt/pipecat-flow

A3. Rewrite the OLD prod IP → new node IP — rewrite BOTH old IPs¶

The image bakes the old box's IP, and the app code additionally hardcodes the France IP. Rewrite both 82.180.146.80 and 89.116.31.109 → <new-IP>:

grep -rlE "82\.180\.146\.80|89\.116\.31\.109" /etc/asterisk /opt/astrapbx/.env \
  /opt/astrapbx/src/services/network /opt/astrapbx/src/routes/customer-tunnels.js \
  /opt/astrapbx/src/services/zombieChannelWatchdog.js \
  | xargs sed -i "s/82\.180\.146\.80/<new-IP>/g; s/89\.116\.31\.109/<new-IP>/g"

A4. Asterisk runtime lib deps (image restores modules, NOT their OS libs)¶

apt-get install -y libgsm1 libsrtp2-1 libspeex1 libspandsp2 libiksemel3 libxslt1.1 ffmpeg odbc-mariadb

A5. Load data + verify BEFORE cutover¶

gunzip -c pbx_api_db-*.sql.gz | mysql pbx_api_db
gunzip -c workflow_db-*.sql.gz | runuser -u postgres -- psql workflow_db
asterisk -rx "features show"     | grep -E "blindxfer|atxfer|agentnum"  # ## / *2 / *5 present
asterisk -rx "pjsip show endpoints" | grep -c Endpoint:                 # ~229
# then a real *43 echo test from a softphone = telephony proven end-to-end

A6. Cutover repoints — do not forget the OpenVPN hospital gateways¶

Phase 1e names NUC + reseller WireGuard, but the 5 FXS hospital gateways ride the astradial-ata OpenVPN tunnel, not WireGuard — each needs its OpenVPN Server Address changed to <new-IP> on the device web UI (field/LAN action per Grandstream/HT814):

The server binds 0.0.0.0:1194 (no server-side IP edit) — just open 1194/udp in UFW. Local mgmt scripts + client .ovpn remote lines in ~/AstradialDevelopment/OpenVPN/* also carry the IP; update them.

A7. Known post-restore bug to fix¶

audit_log.id is bigint but the code inserts UUID() → Cannot cast 'uuid' as 'bigint', so compliance-audit rows for some actions silently fail (caught, non-blocking). Fix: ALTER TABLE audit_log MODIFY id VARCHAR(36); (account for existing auto-increment rows).

Related¶

• High availability — tier roadmap this executes
• Kamailio SIP proxy design
• Database backup & restore runbook
• Monitoring & alerts · Concurrent-call cap