Lessons Learned — Mistakes Ledger¶

A running knowledge base of mistakes we've made and the rule that prevents a repeat. Add a row when a mistake costs real time. Index by symptom/mistake, never by date (dates go inside the entry). For product/code bugs with diagnostic commands, use troubleshooting.md instead — this page is for operational / process mistakes and their prevention rules.

New entry: append to the table, then add a ## L-N section below (symptom → root cause → fix → rule). Keep the rule one sentence, actionable, and phrased so a future agent avoids it.

L-1 — "Registered" on the device ≠ reachable from the server¶

Date: 2026-07-04 · Where: ACR Hospital GXW4216V2 (ext 31–39 / e601–e610)

Symptom. The gateway's Port Status showed all FXS ports Registered and Network Status showed VPN Up; the platform dashboard showed the SIP users orange ("registered but unreachable"). Calls in didn't ring.

Root cause. SIP REGISTER is device-initiated, so it punches out through the hospital NAT and always succeeds — the device looks registered. The real health signal is the server's qualify (OPTIONS server→device), which was timing out. pjsip show contacts showed …@<public-ip> Unavail -nan (never a single reply).

Fix / rule. Diagnose from the server, not the device UI. Truth = asterisk -rx "pjsip show contacts" showing Avail + a real RTT. A device showing "Registered" while the server shows Unavail/-nan = return path broken, inbound will fail. Ping is not VPN health (many ATAs drop ICMP). See vpn-fxs-gateway-register-over-tunnel.

Resolved 2026-07-17 — and the deeper cause was worse than "return path broken". The 07-04 symptom was treated as a NAT/qualify problem, but the reason the qualify had to cross NAT at all went unnoticed for two more weeks: ACR's tunnel was Up the whole time and SIP simply wasn't using it. All 10 FXS ports sat on Profile 1 → devsip.astradial.com:5080 (public) with NAT Traversal = Keep-Alive, while the healthy 10.21.0.65 tunnel carried only management traffic and Profile 2 (the one configured with NAT Traversal = VPN) was used by no port.

Evidence that settled it — the contact source IP, ACR vs a known-good sibling:

ACR    : sip:e601u9t4ktb@120.56.231.104:12005;x-ast-orig-host=192.168.1.35:5060   ← public + hospital NAT
Paavai : sip:e201fedxg6e@10.21.0.33:6060;x-ast-orig-host=192.168.11.4:6060        ← tunnel (correct)

The hospital's BSNL WAN IP rotates (≥5 distinct IPs observed; one mid-session float inside 25 h; 19 tunnel re-handshakes in 8 days). Every rotation dead-ends all 10 contacts until the next REGISTER — the mechanism behind "some calls are not reached". The platform's own monitor had been logging it all along: 45 sip_down episodes over 17 consecutive days (2026-06-30 → 07-16, ~2.6/day) — but only on ext 33, the sole port with down_alert_enabled=1, so the true rate was higher.

Fix: Profile 1 → SIP Server 10.21.0.1:5080, NAT Traversal = VPN. Contacts moved to 10.21.0.65 immediately; measured 0% loss, 0.019 ms jitter on the first call after (vs RTT swinging 30→50 ms on the public path). Full provisioning table: Provision a Grandstream ATA.

New rules this cost us two weeks to learn:

1. "VPN Up" ≠ "SIP over the VPN". A tunnel can be perfectly healthy and carry nothing but management traffic. The only proof is the contact source IP being 10.21.0.x. Add this to every ATA handover check.
2. NAT Traversal = Keep-Alive on a tunnelled device is a red flag — it's the public-path pattern (from the HT813 FXO runbook) and its presence means someone provisioned the device as if it had no tunnel.
3. Monitor every port, not one. ACR had 1 of 10 ports alerting. The signal existed from day one; nobody could see its true size.
4. OpenVPN is not the latency cost people fear it is. Measured on this same link, same instant, both paths live: tunnel 35.6 ms vs public 37.0 ms — the tunnel was faster. Encryption cost ~0.3–0.9 ms; the hospital router's NAT lookup cost more. Never reject the tunnel path on latency grounds without measuring.

L-2 — Host cutover left the ATA overlay behind¶

Date: 2026-07-04 · Where: France→Mumbai prod cutover; ACR (and 4 other ATAs)

Symptom. After prod moved to Mumbai 147.93.168.216, ACR's extensions went Unavail. The device's OpenVPN tunnel was still terminating on the frozen France box (89.116.31.109, asterisk stopped), while SIP tried the public path where the hospital NAT dropped return OPTIONS. Neither path gave a qualifiable registration on the new prod.

Root cause. The cloud (Asterisk + OpenVPN concentrator) moved hosts, but the ATA devices and helper scripts still pointed at the old server. Device-initiated REGISTER masked it as "working". Tracked as an outstanding item in prod-cutover-france-to-mumbai.md but not closed per device.

Fix / rule. On the device: Maintenance → VPN → OpenVPN Server Address → 147.93.168.216 (same CA/cert — verify the CA SHA1 matches on both boxes before repointing; ours did, so no re-provision). Then verify acr-gxw01 appears in the new box's /var/log/openvpn/astradial-ata-status.log and pjsip show contacts flips to Avail. Rule: a prod IP move is incomplete until every ATA tunnel is repointed and re-verified server-side — schedule the on-site repoints as part of the cutover, not after complaints.

L-3 — HTTP 000 was a wrong-port false negative¶

Date: 2026-07-04 · Where: Thangavelu Hospitals thangavelu-hospitals-ata01 (HT814)

Symptom. Details …command reported ⛔ HTTP 000 — not reachable for the web UI, while VPN was CONNECTED and ping was OK — looked like the device was half-dead.

Root cause. The helper (and the provision-client.sh generator) probed HTTP on :80, but this HT814's admin UI is HTTPS on :443 (port 80 closed) and returns 302 to the login page. Two bugs stacked: wrong port/scheme, and a reachable-check that only accepted 200/401.

Fix / rule. For HTTPS ATAs set DEVICE_PORT=443, URL=https://…, curl -k, and accept 302 as reachable. provision-client.sh now emits this by default; older generated folders need the manual edit. Rule: HTTP 000 + healthy VPN/ping = you probed the wrong port/scheme — this is not Dinstar-only, some HT8xx firmware is HTTPS-only too.

L-4 — Dead server IP lived on in scripts and docs¶

Date: 2026-07-04 · Where: OpenVPN/provision-client.sh, per-customer .commands, internal-docs

Symptom. After the cutover, provision-client.sh still defaulted to France 89.116.31.109, and the docs still called the terminated Contabo box 82.180.146.80 the "current PROD" — so a runbook pointed at a server that no longer exists.

Root cause. IP was duplicated across many files with no single source of truth; a migration updated the servers but not the tooling/docs.

Fix / rule. Corrected provision-client.sh defaults → 147.93.168.216, reference/ip-addresses.md, and the core ATA guides. Rule: on a host cutover, treat the IP as living in one authoritative place (reference/ip-addresses.md) and grep the whole repo + the OpenVPN/ tooling for the old IP before calling the migration done.

Durable fix (2026-07-05). Root cause was that a raw IP was hardcoded in every ATA device and every local script, so each prod move meant an on-site visit per hospital. We introduced openvpn.astradial.com (Cloudflare A, DNS-only, → 147.93.168.216) as the tunnel endpoint, repointed every ATA's OpenVPN Server Address to the hostname, and set provision-client.sh + all client .ovpn/SETTINGS to emit it. Now a prod IP move = one DNS record change, zero device visits. (The .command SSH-management targets keep the IP — they're edited from the Mac, not on-site.) ⚠️ Still open: internal-docs/CLAUDE.md and ~46 other docs still carry 82.180.146.80 as prod — needs a scoped repo-wide sweep (82.180.146.80 → 147.93.168.216), skipping intentional historical references (migration/incident writeups).