Cloud Operations Runbook¶

SSH Access¶

ssh root@147.93.168.216

Check NUC / Tata Gateway Status¶

Endpoint Status¶

asterisk -rx "pjsip show endpoint tata_gateway"

AOR and Contact¶

asterisk -rx "pjsip show aor tata_gateway"

WireGuard Tunnel¶

wg show wg0
ping -c 3 10.10.10.2   # Ping NUC end of tunnel

DID Routing¶

View Current Routes¶

asterisk -rx "dialplan show tata-did-route"

Current DID Routing Table¶

For adding new DIDs, see DID Management.

Reload Commands¶

# Reload PJSIP (endpoints, transports, AORs)
asterisk -rx "module reload res_pjsip.so"

# Reload dialplan only
asterisk -rx "dialplan reload"

# Full restart (drops all active calls)
asterisk -rx "core restart now"

Regenerating org configs¶

Default behaviour (as of 2026-05-23, PR #306)¶

Every Deploy API to staging / Deploy API to production run now ends with a Regenerate all org configs step. After the standard rsync + npm ci + migrations + pm2 reload + func_odbc reload, the workflow runs node /opt/astrapbx/scripts/regen-all-org-configs.js on the deployed VPS. This rewrites every active org's pjsip_*.conf, ext_*.conf, queues_*.conf from the freshly-deployed generator code and fires a single AMI reload at the end.

So in the normal case you don't need to do anything — merging the api PR is sufficient. Check the deploy workflow log; the new step is visible right at the bottom and prints Done — deployed=N failed=0 total=N.

The rule for the assistant to never run this manually after a normal deploy is captured in CLAUDE.md → Rule 7 in the astradial-platform repo.

Why this exists (the stale-org failure mode)¶

Until 2026-05-23, autoDeploy only regenerated an org's config when that org had a config-mutation in the editor (PUT to user/DID/queue/IVR). When a generator change shipped — e.g. PR #296 (per-user outbound DID), PR #297 (engaged-state failover), PR #304 (queue empty-state routing) — orgs that nobody had touched in the editor since the last bulk regen kept running the old generated dialplans on the VPS even though the API code was already updated.

The Fintax incident on 2026-05-23 surfaced 7 stale orgs at once (vseven_hotels, grandestancia, autostyles, thangavelu_kolathur, pvs_global_network_, thangam_communication, zauto_ai). Recovery required a manual SSH + node scripts/regen-all-org-configs.js. Auto-regen prevents the next recurrence.

Manual fallbacks (when the CI step fails)¶

If the deploy workflow's Regenerate all org configs step goes red — or you need to force a regen between deploys for a hotfix — two options remain:

(A) Dispatch the standalone workflow (audited):

1. GitHub → Actions → Regen org configs → Run workflow → pick staging or production.
2. Pre-flight grep, snapshot every queues_*.conf to /root/queues-bak-<ts>/, run the regen script, audit non-zero penalty members before and after, print the rollback recipe.
3. Single AMI reload at the end (dialplan + res_pjsip + app_queue + devstate seed) — no core restart, no dropped active calls.

Source: .github/workflows/regen-org-configs.yml in astradial-platform. See Queue Routing Architecture → Regen org configs workflow for the design rationale.

(B) Raw SSH (last resort, no audit log):

ssh root@147.93.168.216 'cd /opt/astrapbx && node scripts/regen-all-org-configs.js'

Same script, no pre-flight grep or snapshot. Use only when (A) is unavailable. Counts as a deployed runbook script per astradial-platform/CLAUDE.md Rule 3.

How to detect stale orgs¶

The auto-regen should keep all orgs fresh. If you suspect an org is stale (a deployed feature isn't showing for them):

# Compare deployed-API mtime to per-org dialplan mtimes — anything older
# than the deployed API code rsync time is stale.
ssh root@147.93.168.216 'api_t=$(stat -c %Y /opt/astrapbx/src/services/asterisk/dialplanGenerator.js); for f in /etc/asterisk/ext_*.conf; do ft=$(stat -c %Y "$f"); if [ "$ft" -lt "$api_t" ]; then echo "STALE: $f ($(date -u -d @$ft))"; fi; done'

If anything prints STALE, run option (A) above to refresh.

GitHub Actions staging-sync trap¶

When a PR merges to main, the sync-staging workflow auto-fast-forwards the staging branch — but it pushes using the default GITHUB_TOKEN, which does not trigger downstream workflows (GitHub's loop-prevention rule). So Deploy API to staging and Deploy Editor to staging do not run, and the staging VPS keeps the old code even though the branch shows the new commits.

To recover after a direct-to-main merge that skipped staging:

gh workflow run "Deploy API to staging" --ref staging
gh workflow run "Deploy Editor to staging" --ref staging

Both have workflow_dispatch: enabled. Re-run on the staging self-hosted runner. After they succeed, pm2 list | grep astrapbx on the staging VPS should show a recent uptime; sentinel-grep the deployed source to confirm new code landed.

The long-term fix is to swap sync-staging.yml's GITHUB_TOKEN for a PAT or GitHub App token so downstream workflows DO fire on the auto-sync push.

API Endpoints¶

Base URL: https://devpbx.astradial.com

Swagger documentation: https://devpbx.astradial.com/docs

Organization Reference¶

To look up UUIDs:

curl -s https://devpbx.astradial.com/api/v1/organizations | jq '.[] | {name, id, prefix}'

Or via database:

USE pbx_api_db;
SELECT id, name, context_prefix, type FROM organizations;

Restart Services¶

# Asterisk PBX
systemctl restart asterisk

# AstraPBX API (Node.js)
systemctl restart astrapbx

# Nginx reverse proxy
systemctl restart nginx

# MariaDB
systemctl restart mariadb

# WireGuard tunnel
systemctl restart wg-quick@wg0

# Netdata monitoring
systemctl restart netdata

Log Locations¶

Tailing Logs¶

# Asterisk messages (most useful for SIP debugging)
tail -f /var/log/asterisk/messages

# Asterisk full log (verbose, all output)
tail -f /var/log/asterisk/full

# AstraPBX API
journalctl -u astrapbx -f

# Nginx errors
tail -f /var/log/nginx/error.log