Skip to content

High Availability & Reliability (target architecture + roadmap)

Goal: carrier-grade reliability for hospital telephony — on-premise (NUC, bare metal) as the primary media anchor, cloud as backup, with no single point of failure. This page is the evergreen plan; it states what IS today, the target, and the phased path between them.

“100% uptime” — the honest framing

No operator sells 100.000%. Class-5 telecom switches target 99.999% (≈5 min/yr). You get there not with one box but by removing every single point of failure (N+1 at each layer) + fast automatic failover + monitoring that catches degradation before customers do. Cost rises ~10× per extra nine, so this is tiered — climb a tier only when client count / SLA commitments justify it.

Target Downtime/yr Tier
99.9% 8.8 h Tier 1 (quick wins)
99.99% 52 min Tier 2 (HA pair + Kamailio)
99.999% 5.3 min Tier 3 (full carrier-grade)

Current single points of failure (today)

Layer SPoF Tier that fixes it
PSTN carrier Tata NNI only — one carrier outage kills all numbers 3
SIP control none (no proxy in front of Asterisk) 2
Media / Asterisk single cloud node; was on shared vCPU (Error 73) 1→3
NUC gateway single box, single Jio line, UPS dependent 1→3
WireGuard single tunnel NUC↔cloud 2
MariaDB single instance (backed up hourly, but no replica) 2→3
Redis / PostgreSQL single instance, no replica (both backed up hourly since 2026-06-28 — see backup runbook) 1→2
DNS / edge Cloudflare (external) 1
Monitoring steal=0 blind spot (now covered by the host-timer probe) done

Target architecture — on-prem primary + cloud backup

                ┌──────────────── PSTN / CARRIERS ────────────────┐
   Tier 3 →     │  Tata NNI (→NUC)          Carrier-2 SIP (→cloud) │
                └──────┬──────────────────────────────┬───────────┘
                       ▼                               ▼
  ON-PREM (primary, bare metal)              CLOUD (backup, Mumbai DEDICATED-CPU VDS)
  ┌───────────────────────────────┐  DMQ /  ┌───────────────────────────────┐
  │  KAMAILIO-A  (floating VIP)    │◄──SRV──►│  KAMAILIO-B (SRV secondary)   │  CONTROL PLANE
  │  registrar · dispatcher · SBC  │ replic. │  registrar · dispatcher · SBC │  (active/active)
  └───────┬───────────────────────┘         └──────────────┬────────────────┘
          │   dispatcher pool, OPTIONS health-checked       │
   ┌──────▼─────┐  ┌──────────┐   add nodes to scale  ┌─────▼──────┐
   │ Asterisk   │  │ Asterisk │  ───────────────────► │  Asterisk  │            MEDIA PLANE
   │  NUC-1     │  │  NUC-2   │                        │  cloud-1   │            (active/active)
   └──────┬─────┘  └────┬─────┘                        └─────┬──────┘
          └─── func_odbc · AMI · CDR(ODBC) ───┬──────────────┘
                                              ▼
                    ┌────────────────────────────────────────────┐
                    │ STATE: MariaDB Galera (NUC + cloud + arbiter)│            DATA PLANE
                    │ Redis (AOF + replica)   PostgreSQL (stream)  │            (replicated)
                    └───────────────────┬──────────────────────────┘
                                        ▼
            Recordings → GCS (lifecycle tiers)   +   off-site DB backups (tested restore)

Bare-metal on-prem gives the best audio and lowest cost-per-call (it is the structural cure for Error 73; the NUC measures timer p99 ≈0.37 ms vs a contended cloud VM's 29 ms) and keeps media in-India. The cloud is a right-sized dedicated-CPU backup. Kamailio on top turns two boxes into a self-healing, shardable, zero-downtime platform.

Layer-by-layer plan

  • Power (NUC): line-interactive UPS + AVR is the mandatory minimum (see Error 75); generator/extended battery at Tier 3.
  • Internet (NUC): dual ISP (Jio + a second wired/5G) with automatic failover routing; one L3 path per subnet (Error 74 discipline).
  • PSTN / carrier: the near-term inbound SPoF fix is Tata DID failover (NUC → cloud-SIP → mobile) — see Inbound carrier resilience below; longer-term, add a 2nd carrier per the multi-carrier design with Kamailio routing inbound from whichever carrier.
  • SIP control: Kamailio A/B — floating VIP on-prem, SRV secondary in cloud, usrloc replicated via DMQ.
  • Media: Asterisk node pool behind Kamailio dispatcher (OPTIONS health-checked). Node failure → new calls reroute in seconds. (In-progress calls on a dying node still drop — full media redundancy is the last, rarely-worth-it nine.)
  • State: MariaDB → Galera (NUC + cloud + arbiter) for synchronous multi-master; Redis AOF + replica; PostgreSQL streaming replication. Backups already run — see the backup runbook.
  • Storage: recordings already durable on GCS; add lifecycle tiering (hot→nearline→coldline→delete) as volume grows.
  • DNS/edge: low-TTL DNS + SIP NAPTR/SRV priority records so endpoints and carriers have an ordered proxy list.
  • Observability: active synthetic probes that page — the host-timer probe (Error 73), per-trunk SIP health, end-to-end synthetic test calls, DB replication lag, backup-landing check.

Inbound carrier resilience (Tata DID failover)

Problem: each hospital has one DID on the single Tata NNI, and the NNI terminates at the one NUC. Anything that drops the NUC — power cut (Error 75), ISP outage, hardware, or the WireGuard tunnel — takes all hospital inbound offline (Tata sends the INVITE to a dead gateway). This is the highest-impact availability gap today (a ~9-min outage occurred 2026-06-28 from a NUC power cut).

Fix in layers (cheap → robust):

  1. Keep the NUC alive (the common causes): UPS+AVR (power) + dual internet (2nd ISP / 4G-5G auto-failover). Removes the two usual triggers (power, Jio flap) with no carrier involvement.
  2. Tata failover when the NUC is unreachable (the carrier ask): configure alternate routing on the DIDs/NNI so Tata reroutes on INVITE-timeout / primary-unreachable. Ideally both, in priority order:
  3. (best) → cloud public SIP (devsip.astradial.com / cloud public IP): calls bypass the NUC and hit the cloud Asterisk directly → IVR/queue/AI-bot keep working even if the NUC is fully dead, and the cloud (datacenter) is far more reliable than the on-prem NUC. Needs: Tata to support an internet-SIP secondary route and the cloud configured to accept Tata's failover (pjsip identify for Tata's failover SBC IP, DID→org routes, UFW/GeoIP allow). The platform's multi-carrier "SIP-direct to cloud" topology already anticipates this.
  4. (simplest, guaranteed) → mobile hunt-group per hospital: Tata forwards the DID to reception mobiles when unreachable — the call reaches a human (no platform features, but it connects). Every carrier supports this; zero infra. Use as the always-available tertiary net.
  5. Target chain: NUC → cloud-SIP → mobile.
  6. 2nd carrier (if Tata itself fails): a second provider (Jio/Vodafone/ITSP) SIP-direct to the cloud with DIDs duplicated/ported — full carrier redundancy (Tier 3; commercial + porting). See multi-carrier trunks.

Questions to put to Tata: (1) do the DIDs/NNI support failover/alternate routing on primary-unreachable? (2) trigger = INVITE timeout (how many seconds?) or registration/OPTIONS? (3) can the secondary be a SIP route to our cloud public IP over the internet (not just the private NNI)? (4) else, per-DID failover to a PSTN/mobile number? (5) failover + failback timing and cost.

Status / notes: not yet implemented. The Tata NNI is IP-identified (no registration), so failover triggers on INVITE timeout, not a lost registration. Build items: the cloud-side SIP-direct failover endpoint + the NUC dual-WAN; the UPS is the prerequisite (Error 75).

Scaling as the client base grows

  • Vertical first: NUC RAM 16→64 GB (i5-13420H takes 2×32 GB SODIMM), bigger NVMe; resize the cloud VDS tier.
  • Horizontal (Kamailio enables it): add Asterisk media nodes to the dispatcher pool and shard orgs by DID/linkedid. Today all orgs co-locate on one Asterisk with a global core reload and no org-sharding mechanism — Kamailio's routing layer is that mechanism. One node already bridges hundreds of G.711 calls, so multi-node is about redundancy + tenant isolation, not raw capacity.
  • Storage/CDR: asterisk_cdr grows ~3.2 GB/yr (trivial; partition by month, archive cold); recordings 0.5–2.2 TB/yr on tiered GCS.

Phased roadmap (cost vs the nine it buys)

Tier Add Buys
1 — Stop the bleeding dedicated-CPU VDS-Mumbai (kills Error 73); UPS on NUC; PostgreSQL backup + backup-landing alert; low-TTL DNS + SIP SRV ~99.9%, no data-loss risk
2 — HA pair + Kamailio Kamailio A/B + dispatcher (NUC primary + cloud backup); MariaDB replication; Redis AOF+replica; dual internet; customer dual-SIP/outbound-proxy ~99.99%, zero-downtime upgrades
3 — Carrier-grade 2nd PSTN carrier + SBC failover; Galera 3-node; 2nd NUC (local HA) + active cloud node; generator/2nd ISP; full NOC + DR drills 99.99%→99.999%

Cost levers: bare-metal on-prem is the cheapest high-quality capacity; the cloud backup can be a right-sized warm standby scaled up only on failover; the whole stack (Kamailio, Asterisk, MariaDB/Galera, RTPengine, Prometheus, Debian) is open-source (₹0 licensing); GCS lifecycle tiering keeps the fastest-growing cost on the cheapest storage.