Skip to content

Cell Architecture — Per-Client Placement, Automated Failover, Cost-Elastic Scaling

Design adopted 2026-07-04, immediately after the France→Mumbai cutover. This is the concrete architecture behind roadmap Phases 3–6 and 8 (Recovery & Scale Roadmap). It exists because the July-3 prod deletion proved we need org-mobility and unattended failover — and because the cutover itself proved we already own the core machinery.

1. Goal

Run Astradial as a fleet of small, disposable cells (one VDS = one cell) so that:

  1. A high-value client gets a dedicated cell ("big room") within ~1 hour of asking — and when they churn, the cell is deleted the same day. Cost tracks revenue.
  2. A cell failure heals itself: the orgs on the dead cell are regenerated onto a standby cell and traffic repointed, unattended, in minutes (target RTO ≤ 10 min; RPO ≤ 1 h from the backup program).
  3. Zero downtime for existing prod while any of this is built — the live telephony VM is not touched until each pattern is proven on a pilot cell.

Non-goals: Kubernetes for the telephony layer (Asterisk/RTP is stateful UDP with a 10k-port range — the workload containers handle worst; every serious PBX operator runs cells-on-VMs). Multi-master databases. Live-call preservation across a cell failure (calls in flight on a dying cell drop; the next call succeeds).

2. Why not Kubernetes (decision record)

K8s promise How the cell model delivers it cheaper
Scheduling / placement org_placement table + config regeneration (already exists)
Self-healing health probe + auto-regenerate script (watchdog pattern already running)
Rolling upgrades Kamailio dispatcher drains a cell, upgrade, undrain (Kamailio design)
Horizontal scale order another VDS, run the bootstrap playbook (~30 min)
Cost no 3-node control plane, no ingress LB, no CNI complexity — a VDS-S-class node is cheaper than the k8s tax

K8s/k3s remains a fine fit later for stateless workloads only (pipecat bot workers), as a separate track. Nothing in this design blocks that.

3. Architecture overview

flowchart TB
    subgraph edge [Routing plane]
        NUC["NUC gateway (on-prem)\nTata NNI inbound\nDID -> cell map"]
        KAM["Kamailio edge (Phase 5)\nclient REGISTER/INVITE -> cell\ndispatcher per org"]
    end
    subgraph cells [Telephony cells]
        C1["Cell shared-1 = today's prod\n147.93.168.216\n~20 small orgs"]
        C2["Cell bigclient-1 (on demand)\nnew VDS, 1 org"]
        CF["Cell standby\n(France box after soak /\ncheap Mumbai VDS)"]
    end
    subgraph control [Control plane]
        API["astrapbx API\norg regenerate -> any cell"]
        DB[("MariaDB pbx_api_db\n+ PG workflow_db\n(central, on shared-1;\nreplica -> standby, Phase 8)")]
    end
    subgraph store [State plane]
        GCS[("GCS: db dumps, restic images")]
        FB[("Firebase: recordings")]
    end
    NUC -- "wg per cell (10.10.10.x)" --> C1
    NUC -- wg --> C2
    NUC -.-> CF
    KAM --> C1
    KAM --> C2
    API --> C1
    API --> C2
    C1 --- DB
    C2 -- "wg to DB" --> DB
    C1 --> FB
    C2 --> FB
    C1 --> GCS

A cell runs: Asterisk + the per-org generated configs + fail2ban + GeoIP + UFW + watchdog + backup crons. A cell does NOT run: the API, editor, workflow-engine, pipecat, or the DB — those stay on the control-plane node (today: the same box as cell shared-1; separable later).

4. The five primitives (what we own vs what we build)

# Primitive Status
P1 Org regeneration — POST /api/v1/organizations/:id/regenerate (x-internal-key) rebuilds an org's complete dialplan/pjsip/queues/MOH from the DB ✅ exists, battle-proven in the 2026-07-04 cutover audit (22/22 orgs)
P2 Placement map — which cell hosts which org / receives which DID 🔨 build: org_placement table + generator awareness (roadmap Phase 4)
P3 Inbound steering — NUC dialplan routes each DID to its org's cell tunnel IP 🔨 build: generate the NUC DID→cell include from the placement map
P4 Client steering — registrations/calls reach the right cell Phase A: per-org DNS SRV; Phase B: Kamailio edge (full spec)
P5 Cell bootstrap — bring a blank VDS to cell-parity in ~30 min 🔨 build: playbook from §6 (every line learned from Errors 84/85)

5. Data & state plane (the small details that bite)

Prime principle (Hari, 2026-07-04): cells are disposable pointers

All durable state lives on the static state plane (central DB + Firebase/GCS object storage). A cell holds only derived artifacts: configs regenerated from the DB and a local audio cache synced at placement time. Therefore: moving an org is a re-derivation, not a data migration; losing a cell loses nothing durable; and the protection budget concentrates on ONE node (dumps + images + Phase-8 replica) instead of being smeared across the fleet. The single deliberate exception: Asterisk must play prompt/MOH audio from local disk (network-mounted audio = dead air when storage hiccups), hence cache-not-mount.

Residual cell-state elimination (work items): 1. Recordings: mover cadence hourly → every 5 min (cell loss costs ≤5 min of recordings). 2. Voicemail: switch app_voicemail to ODBC storage in the central DB (staging first). 3. MOH/greetings: canonical store on the control plane; cells hold auto-synced caches.

  • DB access from cells: cells reach central MariaDB over WireGuard (10.10.10.1:3306 today). In-Mumbai WG round-trip ≈ 1–3 ms — fine for func_odbc/CDR. Each cell needs the pbx_api, pipecat MariaDB users created for its tunnel IP — a dump does NOT carry mysql.user (Error 84).
  • Per-org file state travels WITH the org at move time (none of it is in the DB): /var/lib/asterisk/moh/<org>*, /var/lib/asterisk/sounds/greetings/*, /var/spool/asterisk/voicemail/<org>/. The move procedure (§7) rsyncs exactly these.
  • Recordings: each cell runs move-recordings.sh hourly → the SAME Firebase bucket; the playback API fetches by filename regardless of which cell recorded it. Each cell therefore needs ffmpeg, the gcloud ADC json, and the rclone firebase:+gcs: remotes (Error 85).
  • Media/RTP: direct client↔cell (no central media hop). Each cell's pjsip_transport.conf carries its OWN public IP in external_media_address / external_signaling_address — the single highest-risk line in any cell bring-up.
  • Registrations converge, they don't switch: clients re-register on their own timers. Lesson from the cutover: to force convergence, stop answering on the old cell (clients fail → re-resolve → land on the new cell in ~2 min). Never "wait and see" with both cells answering.
  • WireGuard keys are per-cell, never copied. Reusing a keypair across two live cells lets one roam/steal the other's tunnel (we relied on same-keypair for the cutover on purpose; in a multi-cell fleet it's a footgun). Every new cell generates fresh keys; peers are added to the NUC explicitly.

6. Cell bootstrap playbook (blank Contabo VDS → cell in ~30 min)

Run top-to-bottom; every item exists because its absence broke something real.

  1. Base: Debian 13, apt install baseline from the dpkg manifest of a healthy cell — includes asterisk (20.x), mariadb-client, ffmpeg, libgsm1, libsrtp2-1, libspeex1, libspandsp2, rclone, restic, fail2ban, ipset, wireguard, ufw, netdata. Gate: dpkg -l diff vs reference cell is clean (the sweep that caught ffmpeg).
  2. Keys & creds: fresh WG keypair; copy gcloud ADC json + rclone remotes (gcs: with bucket_policy_only = true, firebase:) + restic pass file + ntfy env. Never in git.
  3. Network: UFW profile (22, 80/443 if edge-fronted, 5060/5080/udp, 8089/tcp, 10000:20000/udp, 51820/udp); GeoIP india_sip ipset + DROP rules; fail2ban asterisk-pjsip jail; add cell as peer on the NUC wg0 (next free 10.10.10.x).
  4. Asterisk statics: /etc/asterisk base from the reference cell (features.conf ##/*2/*5, [astradial-star5], Tata gateway allow=alaw, ODBC config) — then set THIS cell's public IP in pjsip_transport.conf externals, and CLOUD_PUBLIC_IP if the cell hosts app code.
  5. DB users: create pbx_api/pipecat grants for the cell's WG IP on central MariaDB (via SHOW CREATE USER hash replication).
  6. Crons/services: astradial-backup-check, move-recordings.sh hourly, GeoIP monthly, astradial-image-backup.sh daily (stagger the hour per cell — GCS cost cap!), watchdog scripts (/usr/local/bin/nuc-*.sh — NOT in the image, copy them), systemctl enable asterisk.
  7. Verification gate (from the cutover audit): features show 3 rows; pjsip show transports; ODBC connected; test org regenerated onto the cell + dialplan show org_test__incoming; echo call over the tunnel; recording of that call plays via the API.

Deliverable: scripts/bootstrap-cell.sh in astradial-platform encoding the above (roadmap Phase 4 work item).

7. Moving an org between cells (the core operation, step by step)

Zero-to-low downtime: seconds of registration convergence for that org only; all other orgs untouched. Everything is reversible at every step.

  1. Pre-stage on target cell (no traffic impact): POST /organizations/:id/regenerate?cell=<target> → writes the org's pjsip/ext/MOH confs on the target; rsync the org's moh/greetings/voicemail files; core reload on target.
  2. Verify on target: dialplan show org_<id>__incoming, MOH classes present, a staged test endpoint registers against the target and completes an echo call.
  3. Flip inbound: regenerate the NUC DID→cell include (this org's DIDs → target tunnel IP); dialplan reload on the NUC. Inbound now lands on the target (its endpoint registrations may still be converging — do steps 3–4 back-to-back).
  4. Flip clients: Phase A: update the org's SRV/hostname → target IP (TTL 60 s). Phase B (Kamailio): update the org's dispatcher set — instant, no DNS wait. Force convergence: remove the org's endpoints on the source cell (regenerate source WITHOUT the org, or pjsip qualify+unload) so client refreshes fail fast and re-resolve. Lesson: never leave both cells answering for the same org.
  5. Verify: live call to one of the org's DIDs → target cell (core show channels); agent registration count on target ≥ source's pre-move count; recording plays.
  6. Clean source: delete the org's conf files + audio from the source cell (avoid the stale org_mqzglv1q-style leftovers the audit found).

Rollback at any step: the source still has everything until step 6 — re-flip NUC + SRV back.

8. High-value client onboarding ("big room" flow)

  1. Order VDS sized to the client (calls ≈ 30–40 per dedicated core with recording; the VDS-S class ≈ 3 cores / 24 GB — plenty for a single hospital/hotel).
  2. Bootstrap cell (§6, ~30 min). 3. Move their org (§7, minutes).
  3. Tag the cell dedicated:<org> in the placement map; its monthly cost is now attributable to that client's invoice. 5. On churn: move the org back to a shared cell (§7 reversed), destroy the VDS. Cost returns to zero the same day.

9. Automated failover (the July-3 insurance)

Detector (extends the existing nuc-watchdog pattern; runs on the control plane + NUC): - Per-cell probes every 15 s: WG handshake age, OPTIONS ping to :5060 over the tunnel, asterisk -rx "core show uptime" via SSH. Trip = all three failing for 90 s (one probe can lie; three can't — and 90 s avoids flapping on a reboot).

Actor (failover-cell.sh, dry-runnable, also invocable manually): 1. Page ntfy + WhatsApp immediately (CELL <x> DOWN — auto-failover starting). 2. Freeze the dead cell's WG peer on the NUC (prevents zombie return mid-failover). 3. For each org on the dead cell (placement map): regenerate onto the standby cell (P1); rsync org audio from the hourly GCS full backup if the source is unreachable (RPO ≤ 1 h; voicemail/greetings only — recordings are already in Firebase). 4. Regenerate NUC DID→cell include → standby; flip the orgs' SRV / dispatcher sets. 5. Verify loop: one synthetic call per moved org (echo DID) — page results. 6. Mark placement map; the dead cell, if it comes back, boots with its WG peer disabled and its orgs' configs marked stale (no split-brain).

Standby sizing: one warm standby per region, bootstrapped and idle (France after its soak = free EU standby; a second Mumbai VDS-S when budget allows = proper standby; Phase 8 adds DB replica to it, removing the last single point).

Drill (quarterly, and before first real use): kill asterisk on the pilot cell, watch the detector trip, measure RTO end-to-end, restore, write the numbers into this doc.

10. Admin-panel Fleet Control (UI) — REQUIREMENT

Hari, 2026-07-04: "Complete structure and control should be there in admin panel, with UI." Nothing in this design is CLI-only at the end state; every operation in §6–§9 is a button with a progress view. The scripts are the engine; the admin panel is the only steering wheel operators use.

New top-level Fleet section in the editor's admin area (platform-admin RBAC only; every action writes audit_log; long operations stream progress over the existing socket channel):

10.1 Cells page

Table: cell name · role (shared / dedicated:<org> / standby) · region · public IP · WG IP · org count · health badge (live from the §9 probes: WG handshake, OPTIONS, uptime) · active calls · CPU/RAM · monthly cost tag. Actions: ➕ Add Cell (wizard: paste new-VDS IP + root key → runs bootstrap-cell.sh remotely, renders the §6 checklist as live check-marks, ends with the verification gate) · Drain · Decommission (blocked unless org count = 0) · Re-run verification gate.

10.2 Placement page

Org ⇄ cell matrix (the org_placement table, live). Select org → Move to cell… wizard: runs §7 step-by-step with per-step status (pre-stage ✓ / verify ✓ / inbound flipped ✓ / clients converging n/m endpoints / cleanup), a convergence meter (registered endpoints on target vs expected), and a Rollback button active until step 6.

10.3 Failover page

Standby readiness (bootstrap gate age, last drill RTO) · per-cell detector state (three probes, trip timer) · event timeline of every automated action · Manual failover NOW (typed- confirmation, same actor script as auto) · Drill mode (dry-run against the pilot cell, records measured RTO into this doc's §9).

10.4 Cell detail page

Live: registered endpoints, core show channels, per-org list with per-org "Move" shortcut, recent asterisk log tail, backup/cron status, the cell's §6 checklist state (drift = red).

10.5 API surface backing the UI (control plane, admin-authenticated)

GET    /api/v1/admin/cells                    list + health + load
POST   /api/v1/admin/cells                    add (bootstrap) — body: {ip, ssh_key_ref, role}
POST   /api/v1/admin/cells/:id/drain|verify|decommission
GET    /api/v1/admin/placement                org ⇄ cell matrix
POST   /api/v1/admin/orgs/:id/move            body: {target_cell} → §7 job, progress via socket
POST   /api/v1/admin/failover/:cell           manual trigger (typed confirmation token)
POST   /api/v1/admin/failover/:cell/drill     dry-run
GET    /api/v1/admin/fleet/events             failover/move/bootstrap audit timeline
Server-side these call the same scripts as the automation (§6 bootstrap, §7 move, §9 actor) — one code path, whether triggered by a human button or the detector.

11. Cost model (indicative)

Fleet state Monthly infra (VDS-S-class ≈ €10–18/node)
Today: 1 prod + frozen France + staging ~3 nodes
+ 1 dedicated big-client cell +1 node, billed to that client
+ warm Mumbai standby (real failover) +1 node — the price of never repeating July-3
Client leaves −1 node same day

No k8s control-plane nodes, no load-balancer fees; Kamailio runs on the control-plane node (or a €5 VPS) when Phase 5 lands.

12. Rollout plan (zero prod downtime, gated; each step ships its UI)

Step Engine work UI shipped with it (§10) Gate Roadmap
A org_placement table + ?cell= on regenerate + NUC include generator Placement page (read-only matrix) unit: moving a TEST org between staging boxes Phase 4
B bootstrap-cell.sh + pilot cell (cheapest VDS) Cells page + Add-Cell wizard §6 verification gate passes from the wizard Phase 4
C §7 move procedure as a job Move-org wizard + convergence meter + rollback live call each way; convergence < 3 min Phase 4
D Failover detector + actor Failover page + drill mode measured RTO ≤ 10 min unattended, from the UI Phase 8
E Kamailio edge on staging → prod front dispatcher set editor per org Kamailio rollout plan Phases 3+5
F First real dedicated-client cell cost tags on Cells page client-funded Phase 6

Prod is only ever fronted (E) or sourced from (C/D on test orgs) — never rebuilt or moved.

13. Risks & mitigations

  • Central DB is a SPoF until Phase 8 → hourly dumps already land in GCS (RPO 1 h); Phase 8 puts a MariaDB replica on the standby cell (RPO → seconds).
  • Kamailio edge becomes a SPoF in Phase B → the Kamailio doc's own HA section (keepalived pair / DNS failover) applies; until then per-org SRV keeps cells directly reachable.
  • Placement map drift vs reality → the cutover taught us: configs are DB-derived artifacts. A nightly auditor compares each cell's on-disk org list vs the placement map and pages on drift (the org_mqzglv1q class of bug, automated away).
  • Contabo panel remains the ultimate blast radius → 2FA + no team delete rights + cells named after their role (cell-shared-1, cell-standby) so no human ever again deletes "the wrong France".