Backup & Restore Runbook¶

Scope: off-site backup + restore for prod (147.93.168.216) and the NUC gateway. As-built 2026-06-28. Scripts are synced in this repo under scripts/.

What's backed up¶

Crons live in root's crontab on each host. The hourly DB dump uses --single-transaction (MVCC snapshot, no locks → safe on the live DB). All image/restic uploads are bandwidth-capped + niced so they never compete with live call RTP.

Ransomware posture (immutability)¶

• astradial-prod-backups (the critical DB/CDR/config backups): the prod key astradial-backup-writer is Object Creator + Viewer only — it cannot delete or purge (verified). Object Versioning ON. Expiry is done by GCS lifecycle (Google-side), not the key. → a rooted prod cannot erase these backups.
• Image buckets (astradial-prod-image, astradial-nuc-backups): restic needs delete (locks/prune), so their keys have Object Admin scoped to that one bucket only. They hold rebuildable OS images (no customer data) → limited blast radius; the DB backups stay immutable.
• Recordings (misssellerai.firebasestorage.app): single Google-replicated copy; the Firebase key still has delete → not ransomware-hardened (follow-up: tighten / add versioning).
• A 2nd off-account copy was deferred (2026-06-28); the controls above are the protection.

Monitoring¶

• Landing alert — astradial-backup-check.sh hourly at :15 → ntfy if newest db/hourly object is stale (>90 min) or <1 MB. Reads status marker /var/lib/astradial-backup/last.env. (This caught a 6-day silent PostgreSQL backup failure on day one — a cron PATH bug, now fixed.)
• Restore drill — astradial-restore-drill.sh monthly (1st, 03:30) → restores the latest dump into scratch pbx_api_db_restore, asserts row counts vs live, ntfy on mismatch, drops scratch. Proves RTO without touching live.

Retention policy (recordings + CDR)¶

• Uniform: 1095 d recording (3 yr) / 1825 d CDR (5 yr) for ALL orgs (set 2026-06-28; new-org column default also 1095/1825). Chosen over per-industry presets for simplicity + fail-safe (never under-retain a hospital). SQL: scripts/fix-org-retention.sql.
• The "industry presets" in compliance.md were never implemented in code — the API does INSERT IGNORE INTO org_compliance (org_id) and relies on the column default.
• Purge (enforcement) is not built and not needed until ~April 2029 — with 3-yr retention the oldest recording (Apr 2026) isn't purge-eligible until then (CDR ~2031). Until then, lifecycle tiering → Archive is the cost control. Build the per-org purge (honoring org_compliance.retention_*, right-to-erasure, audit_log) before 2029.

Cost / budget¶

Hard cap $2/mo GCS. Backups (DB+PG+Redis+configs+images) ≈ $0.85/mo; recordings auto-tier to Archive (~$0.0012/GB). Total ≈ $1.5/mo at current volume. At many-hospital scale, 3-yr no-purge recordings could approach $2/mo in ~2–3 yr → revisit retention/cost then (the alert warns first).

Budget alert (Zazmic India billing acct 018B3E-366D99-31DE78 → Budgets & alerts → "Astradial"): ₹200/mo, triggers 50/90/100% actual, emails billing admins. Scope = projects Astradial(misssellerai) + AstraDial(astradial), Services = Cloud Storage only — so GCS spend on either Astradial project pages, without false alerts from Gemini API spend on misssellerai. (It was mis-scoped to project monsoon and silently missed the Jul-2026 blowout — corrected 2026-07-14. If you add a new project that stores backups/recordings, add it to this budget's scope.)

⚠️ RCA — the ₹4,482 Jul-2026 blowout (list-egress, NOT storage). After the Jul-4 box cutover, move-recordings.sh ran */5 and each rclone copy/move traversed the flat astra_pbx/recordings/ prefix (~100k objects) in a us-central1 bucket → ~90 MB of list-JSON per sweep billed from Mumbai as inter-continent Download APAC egress (₹4,141 = 91% of the bill) + Class A ListObjects (₹291, +513%). Actual stored bytes were only ~₹99. Fix: --no-traverse on all 3 rclone calls → per-source-file HEAD instead of listing the huge destination (few new WAVs per run; move deletes sources on success → identical upload, no data-loss). Cut ~95% of the cost. Watch for this whenever a flat, large-object-count prefix is synced repeatedly. Long-term: date-partition recordings (recordings/YYYY/MM/) and/or move the bucket to asia-south1.

Verify backups are healthy¶

ssh root@147.93.168.216
rclone lsl gcs:astradial-prod-backups/db/hourly/ | sort -k2 | tail -3   # fresh (<70 min) + ~4 MB
cat /var/lib/astradial-backup/last.env                                   # LAST_STATUS=ok
gsutil versioning get gs://astradial-prod-backups                        # Enabled
restic -r rclone:gcs:astradial-prod-image/restic snapshots              # prod image (needs RESTIC_PASSWORD_FILE)

Restore pbx_api_db (DR)¶

⚠️ Restoring over live is destructive — restore to a scratch DB first, verify, then cut over. Confirm per CLAUDE.md rule 1.

rclone lsl gcs:astradial-prod-backups/db/hourly/ | sort -k2 | tail        # pick newest
rclone copy gcs:astradial-prod-backups/db/hourly/<file> /tmp/
mysql -e "CREATE DATABASE pbx_api_db_restore"
gunzip -c /tmp/<file> | mysql pbx_api_db_restore
mysql pbx_api_db_restore -e "SELECT COUNT(*) FROM organizations; SELECT COUNT(*) FROM did_numbers"
# only after approval, real restore: mysql pbx_api_db < <(gunzip -c <file>)   # back up live first

Restore a whole box (restic image)¶

# On a fresh VM: install restic+rclone, restore the rclone key + repo password.
# The image-repo password is inside the configs backup:
#   astradial-prod-backups/configs/<tier>/configs-*.tar.gz  → /etc/astradial-prod-image-restic.pass
restic -r rclone:gcs:astradial-prod-image/restic restore latest --target /
# Then load the latest DB dump on top, and re-point WireGuard + DNS.
# NUC: same with rclone:nucgcs:astradial-nuc-backups/restic (password in the NUC config tar).

Restic repo passwords (where they live + can they be stolen?)¶

The restic image repos are encrypted — lose the password and the image is unrecoverable. Each password is stored in two places, plus a recommended third: - On the host (chmod 600, root-only): /etc/astradial-prod-image-restic.pass (prod), /etc/astradial-nuc-restic.pass (NUC). - In GCS, inside that host's config tar — prod → astradial-prod-backups/configs/… (the immutable bucket); NUC → astradial-nuc-backups/config/…. The config tar is a plain rclone copy (readable with the GCS key, not restic-encrypted), so DR isn't circular: fetch the tar → extract the password → restore the image. - Recommended third copy: Vaultwarden. cat each file and paste it in. With host + GCS + Vaultwarden, losing any two still leaves the images recoverable.

Can an attacker read it on the box? Only with root. 600 blocks every non-root user, but root reads everything — as it does for every secret the box uses (rclone GCS key, WireGuard keys, .env, Firebase key). An unattended cron backup inherently needs the password readable by root; you can't hide it from root on the machine that uses it. Therefore: - The real defense is preventing root compromise — SSH key-only, fail2ban, ufw + GeoIP, auto-updates (all in place). Vaultwarden guards against losing the password, not theft. - Critical data survives a full root compromise regardless: the DB/CDR/config backups use a no-delete key + versioning in astradial-prod-backups, so a rooted box can't wipe them (and they're plain mysqldump, no restic password). The password only matters for the image backups (rebuild convenience) — a root attacker could decrypt/delete those, but you'd still rebuild from the immutable DB backups. - restic encryption still protects the image at rest in GCS against a GCS-only leak (bucket read without the host ≠ readable contents).

Scripts (synced in this repo)¶

astradial-backup.sh · astradial-backup-check.sh · astradial-restore-drill.sh · astradial-image-backup.sh · nuc-backup.sh · fix-org-retention.sql — under scripts/. The deployed copies live in /usr/local/sbin/ on each host (VPS is source of truth; keep these in sync).