Monitoring¶

Overview¶

Astradial uses three monitoring layers:

• Netdata -- Real-time system metrics, alerts, and dashboards
• Upptime -- Public status page with uptime tracking via GitHub Actions
• NUC offline alerting -- Two independent watchers that page via ntfy within ~75s when the on-premise NUC SIP gateway goes offline (see below)

Netdata¶

Installations¶

Both instances are connected to Netdata Cloud under the hari-space-company space.

Monitored Metrics¶

• CPU usage and temperature
• RAM usage
• Disk usage and I/O
• Network throughput

Alerts¶

Alerts are configured in Netdata Cloud with email and mobile push notifications:

Upptime¶

Upptime provides a public-facing status page powered by GitHub Actions. It checks service availability every 5 minutes and publishes results to a static site.

Monitored Services¶

Upptime checks 5 services every 5 minutes.

GitHub Actions Workflows¶

Deployment¶

The status page is built as a static site and deployed to GitHub Pages using the peaceiris/actions-gh-pages action, which pushes to the gh-pages branch.

# Deployment step in workflow
- uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: ./build

NUC Offline Alerting¶

The on-premise NUC is the Tata PSTN gateway and sits on an unstable office WiFi link. When it drops off the network, all Tata inbound/outbound stops. This alerting fires a phone push within ~75s so the outage is acted on immediately.

Why a phone call can't be the alert¶

A PSTN call to Hari's mobile routes over the Tata trunk, which runs on the NUC. When the NUC is down, that call path is down too. The alert channel must be out-of-band: a data push (ntfy) that does not touch the NUC. The watcher must also run somewhere other than the NUC — a dead device can't alert.

Architecture — two independent watchers¶

                 ntfy.sh topic  (in /etc/nuc-watchdog.env & HC integration)
                       ▲                         ▲
   Layer 1: active     │ push                    │ push
   ping over WG  ──────┘                         │
   [cloud 147.93.168.216]                         │
   nuc-watchdog.service                          │
        │ ping 10.10.10.2 q15s                   │
        ▼                              Layer 2: dead-man's-switch
   [ NUC 10.10.10.2 ] ──heartbeat q60s──► [ Healthchecks.io ]
   nuc-heartbeat.timer

The two layers detect "NUC down" over independent paths, so a failure of either the cloud VPS or the Healthchecks side still leaves one working detector. Netdata Cloud's "node unreachable" alert is a third, bonus detector.

Components¶

Healthchecks.io check config¶

• Project owner: hari@astradial.com
• Check name: NUC SIP gateway
• Schedule: Simple, Period 1 minute, Grace 1 minute
• Integration: ntfy → topic from /etc/nuc-heartbeat.env, server https://ntfy.sh

Test it¶

Layer 1 — simulate a real outage from the cloud (safe: drops only ICMP, so SIP/RTP keep flowing). Auto-removes the rule on exit:

ssh root@147.93.168.216 '
  trap "iptables -D OUTPUT -p icmp -d 10.10.10.2 -j DROP 2>/dev/null" EXIT
  iptables -I OUTPUT -p icmp -d 10.10.10.2 -j DROP
  sleep 90   # watch /var/lib/nuc-watchdog/state.env flip to DOWN, expect 🔴 push
'
# rule removed on exit → expect 🟢 recovery push ~60s later

Layer 2 — exercise the Healthchecks → ntfy path without disrupting anything:

HC=$(ssh root@nuc.astradial.com 'grep -o "https://hc-ping.com/[^ ]*" /etc/nuc-heartbeat.env')
curl -fsS "$HC/fail"   # expect a DOWN push from Healthchecks
sleep 10
curl -fsS "$HC"        # back to green

Tune¶

• Faster detection (~45s, slightly more flap-prone): on the cloud, set DOWN_THRESHOLD=3 in /etc/nuc-watchdog.env, then systemctl restart nuc-watchdog. Other knobs: INTERVAL, UP_THRESHOLD, REMINDER_SECS.
• Check status: systemctl status nuc-watchdog (cloud); systemctl list-timers nuc-heartbeat.timer (NUC).

Scope¶

Ping detects network/device offline. The Layer-2 heartbeat additionally gates on Asterisk being active and the WG handshake being fresh, so it also catches a "NUC pingable but gateway dead" condition. Neither layer inspects call quality or per-trunk SIP registration — that's nuc-linkqual (below) / Netdata territory.

Link Quality & Host vCPU Monitoring (nuc-linkqual)¶

nuc-watchdog (above) catches NUC offline. nuc-linkqual catches what degrades calls while everything still pings "up": tunnel packet loss/jitter, and — since 2026-06-22 — prod host vCPU-scheduling stalls (the Error 73 noisy-neighbour failure mode that is invisible to steal=0 dashboards).

• Runs on cloud 147.93.168.216, nuc-linkqual.service (Restart=always). Script /usr/local/bin/nuc-linkqual.sh (local copy AstradialDevelopment/NUC/nuc-linkqual.sh).
• Every ~60 s it logs a sample to /var/log/linkqual/linkqual-YYYYMMDD.csv (30-day retention) and ntfy-alerts on BAD/RECOVERED transitions with hysteresis (same topic + pattern as nuc-watchdog).

Host-timer probe (Error 73 early warning)¶

Each round, on prod itself: (1) a load-free 2000-sample 5 ms-sleep wakeup test → p99/max lateness + count of stalls >10 ms (healthy KVM p99 <1 ms; the Jun-2026 incident was p99 29–33 ms / 5–7% stalls), and (2) worst-of-3 asterisk -rx "timing test" → ticks/sec (target 50; broken dipped to 18–46). Tunables (env): TIMER_P99_WARN=2, TIMER_TICKS_WARN=48, TIMER_BAD_THRESHOLD=3, TIMER_SAMPLES=2000; set TIMER_PROBE=0 to disable.

# on the cloud box: host timing should read p99<2ms, ticks 50, stalls 0
ssh root@147.93.168.216
LQ=$(ls -t /var/log/linkqual/*.csv | head -1)
grep ',TIMER,' "$LQ" | tail -5
grep ',NUC,'   "$LQ" | tail -5      # tunnel loss/jitter
systemctl status nuc-linkqual