Incident: Apr 14 — Monorepo Migration & Prod Outbound Breakage¶

Timeline¶

What Went Wrong¶

1. Monorepo editor was 40 commits behind¶

The local copy of astradial-editor repo was at an old commit. The monorepo was built from the local working directory, not from git pull origin main. Missing: CRM, DID marketplace, admin pages, mobile responsive, RBAC.

2. firebase-admin not in package.json¶

The firebase-admin npm package was installed manually on prod/staging but never added to package.json. When monorepo CI/CD ran npm ci --omit=dev, it wasn't installed.

3. rsync --delete removed server-side files¶

The CI/CD rsync -a --delete removed firebase-sa-key.json and .env files that only existed on the server.

4. Prod Asterisk restart broke outbound¶

Restarting Asterisk on prod caused tata_gateway trunk to return busy/congested on all outbound Dial attempts. The PJSIP endpoint with static contact lost its ability to create outbound channels.

5. NUC tata-endpoint CHANUNAVAIL after restart¶

On Asterisk 22, a PJSIP AOR with qualify_frequency=0 and max_contacts=0 results in CHANUNAVAIL for outbound Dial after restart. The contact exists but is NonQual, and Asterisk 22 requires Available status.

6. NUC from-cloud wrong number format¶

The original from-cloud context used Dial(PJSIP/0${EXTEN}@tata-endpoint) and Set(CALLERID(num)=+91${CALLERID(num):1}). This produced:

• CallerID: +91918065978001 (double country code — original CallerID already had +91)
• Dial: 09944421125 — worked before qualify fix but broke after

Set(CALLERID(all)=+918065978001)   ; Use CALLERID(all) not CALLERID(num)
Dial(PJSIP/0${EXTEN}@tata-endpoint,60)  ; 0-prefix = local dialing format

What Was Fixed¶

Staging outbound call path¶

Staging Asterisk (94.136.188.221)
  → tata_gateway endpoint → sip:10.10.10.2:5060
  → WireGuard wg0 (10.10.10.3)
  → Prod VPS (10.10.10.1) — IP forwarding only, no Asterisk involvement
  → WireGuard → NUC (10.10.10.2)
  → NUC Asterisk → from-cloud context
  → Dial(PJSIP/0${EXTEN}@tata-endpoint)
  → Tata SBC (10.79.215.102) → PSTN

WireGuard changes¶

Prod (89.116.31.109):

• Enabled net.ipv4.ip_forward=1 in /etc/sysctl.conf
• Added iptables FORWARD rules for wg0: 10.10.10.3 ↔ 10.10.10.2

Staging (94.136.188.221):

• Added 10.10.10.2/32 to AllowedIPs for prod peer in /etc/wireguard/wg0.conf

NUC (nuc.astradial.com):

• Added 10.10.10.3/32 to AllowedIPs for prod peer in /etc/wireguard/wg0.conf

NUC Asterisk changes¶

pjsip.conf:

• cloud-identify: Added match=10.10.10.3 (accept staging as cloud endpoint)
• tata-aor: Added qualify_frequency=30 and max_contacts=1

extensions.conf — [from-cloud]:

[from-cloud]
exten => _X.,1,NoOp(Cloud Outbound via Tata: ${EXTEN} CID: ${CALLERID(all)})
 same => n,Set(CALLERID(all)=+918065978001)
 same => n,Dial(PJSIP/0${EXTEN}@tata-endpoint,60)
 same => n,NoOp(Tata dial status: ${DIALSTATUS})
 same => n,Hangup()

Staging Asterisk changes¶

pjsip_tata_gateway.conf:

• Changed contact=sip:10.10.10.2:5060 (direct to NUC, was 10.10.10.1 pointing to prod)
• Changed match=10.10.10.2 (identify NUC directly)

Recovery Checklist¶

If staging outbound breaks again:

1. Check NUC Tata contact status:

   ssh user@nuc.astradial.com
   sudo asterisk -x "pjsip show contacts" | grep tata
   # Must show "Avail" — if "NonQual" or "Unavail", restart NUC Asterisk
   

2. Check WireGuard connectivity:

   # From staging:
   ping -c 2 10.10.10.2   # Must reach NUC
   # From NUC:
   ping -c 2 10.10.10.3   # Must reach staging
   

3. Check prod IP forwarding:

   ssh root@89.116.31.109
   sysctl net.ipv4.ip_forward          # Must be 1
   iptables -L FORWARD | grep wg0      # Must have ACCEPT rules
   

4. Test NUC outbound directly:

   sudo asterisk -x "channel originate Local/919944421125@testonly-outbound application Wait 30"
   # If this works, NUC → Tata is fine; issue is upstream
   

Backups¶

All configs backed up at /root/asterisk-backup-20260415/ on:

• Prod (89.116.31.109)
• Staging (94.136.188.221)
• NUC (nuc.astradial.com)