Skip to content

Incident — Jul 09 2026: Prod SSH lockout → reboot → Asterisk headless (calls down)

Severity: P0 (all prod calls down ~23:27–00:03 IST). Host: prod PBX 147.93.168.216 (vmi3415819), Debian 13, Contabo. Resolved: 2026-07-10 ~00:20 IST. No data loss.

Summary

A three-stage cascade, all from latent misconfig — no single fault:

  1. SSH lockout (trigger). fail2ban's asterisk-pjsip jail whole-IP-banned the operator's home IP (120.60.127.178) after a SIP client on that network sent ≥4 failed REGISTERs (maxretry=4 / findtime=10m). The ban is an iptables REJECT on all ports, so it also killed SSH. UFW's only port-22 allow rule was IPv6-only, so there was no IPv4 fallback. Operator could reach the box only via Contabo VNC.
  2. Manual reboot (amplifier). To recover SSH, the operator rebooted at 23:27. Clean reboot — prior-boot log shows no OOM/panic.
  3. Calls down (main outage). asterisk.service had no RuntimeDirectory=asterisk and there was no tmpfiles.d entry. On Debian 13 /var/run is tmpfs (wiped on reboot), so nothing recreated /var/run/asterisk as asterisk:asterisk. Asterisk starts as the unprivileged asterisk user and could not create its control socket / PID file → came up headless (no CLI, degraded). Boot log 23:27:39: Unable to bind socket to /var/run/asterisk/asterisk.ctl: No such file or directory.

Secondary breakage surfaced during recovery:

  1. Live Calls dashboard 503. astrapbx holds a persistent AMI/ARI connection and does not auto-reconnect. Restarting Asterisk (to fix #3) dropped its connection → /api/v1/calls/live returned 503 until astrapbx was restarted.

Root causes

# Root cause Fix applied
A asterisk.service missing RuntimeDirectory=asterisk → tmpfs /var/run wipe on reboot leaves Asterisk unable to create its control socket Added RuntimeDirectory=asterisk + RuntimeDirectoryMode=0750 to /etc/systemd/system/asterisk.service. Survives reboots. Backup: asterisk.service.bak-2026-07-10.
B fail2ban ignoreip was scoped to [sshd] only — the asterisk-pjsip jail had no whitelist and its whole-IP ban blocks SSH too Moved ignoreip to [DEFAULT] in jail.local (applies to ALL jails); added operator IP 120.60.127.178, ATA VPN 10.21.0.0/24, hospital 103.164.181.2.
C UFW port-22 allow rule was IPv6-only Re-enabled UFW with ufw allow 22/tcp (IPv4).
D astrapbx does not reconnect AMI/ARI after an Asterisk restart Operational: always pm2 restart astrapbx after restarting Asterisk. (Code follow-up: add AMI/ARI auto-reconnect.)

Recovery steps (what was run)

# A — restore calls
install -d -o asterisk -g asterisk -m 0750 /var/run/asterisk
# add RuntimeDirectory=asterisk + RuntimeDirectoryMode=0750 to [Service]
systemctl daemon-reload && systemctl restart asterisk
asterisk -rx "pjsip show registrations"   # Tata trunk Avail (IP-auth, no reg object)

# B — fail2ban whitelist (jail.local [DEFAULT] ignoreip)
fail2ban-client set asterisk-pjsip unbanip 10.21.0.97
fail2ban-client set asterisk-pjsip unbanip 103.164.181.2
systemctl reload fail2ban

# C — firewall
ufw allow 22/tcp && ufw --force enable

# D — live-calls feed
pm2 restart astrapbx --update-env       # reconnects AMI + ARI

Verification (post-fix, all green)

  • Voice: Tata trunk tata_gateway Avail RTT ~15ms; endpoints loaded; transports bound (udp 5060/5080, tcp 5080, wss 8089). Live inbound test call answered.
  • fail2ban: 0 currently banned; operator + ATA subnet + hospital in ignoreip.
  • UFW active with IPv4 rules for 22, 5060, 5080/udp, 8089/tcp, 10000:20000/udp.
  • Live Calls dashboard: astrapbx logs AMI manager connected + Connected to ARI; /api/v1/calls/live → 401 (healthy, was 503).

Prevention / follow-ups

  • RuntimeDirectory=asterisk in the prod unit — reboot-safe. Add to the asterisk.service baseline so staging + any rebuild carries it.
  • fail2ban ignoreip in [DEFAULT] with ATA/operator subnets — see Fail2Ban Runbook.
  • astrapbx AMI/ARI auto-reconnect — until shipped, restarting Asterisk requires a follow-up pm2 restart astrapbx, else Live Calls goes 503.
  • Reboot drill in a maintenance window to confirm A holds end-to-end.
  • Track down the rogue SIP client on the operator network that triggered the original asterisk-pjsip bans (stale softphone hitting devsip with bad creds).
  • Unrelated display bug found same session: Live Calls To column showed n for Local-channel forwards (Local/fwd_<ext>@ctx/n — the /n option flag was mis-parsed as the destination). Fix in liveCallsService.js To-derivation.