Skip to content

Hindi Daily Word — Voice-Driven Vocabulary Tutoring

Status: Shipped on staging (2026-04-29). First-tenant org Hindi-Tutoring running 1 active learner end-to-end. Org: Hindi-Tutoring (9dede1af-c88d-454d-b2a5-d837788a71c8) Feature Name: Hindi Daily Word

What

A daily voice tutoring service that:

  1. At 00:00 IST, picks the next-untaught Hindi word for each active learner using SM-2 spaced repetition.
  2. Schedules an outbound call to the learner at their preferred time (default 09:00 IST).
  3. The Hindi-tutor bot greets the learner, teaches one word + transliteration + meaning + example, listens for the learner to repeat it, praises with "Shabaash!" + a brief farewell, and hangs up.
  4. Sends a WhatsApp recap message with today's word — either "you learned X" or "you missed X" depending on call outcome — including for calls that never connected (no answer, busy).

Designed as a minimal voice-tutoring template that can be extended to other subjects (multiplication tables, vocabulary in any language, dietary reminders, medication adherence, etc.) by swapping the bot module's _lesson shape and the WhatsApp template.

Why

  • Real-life daily-habit reinforcement. A scheduled outbound call + WhatsApp summary is a stickier learning loop than an app the learner has to open.
  • Validates Astradial's full bot stack end-to-end. Hindi Daily Word exercises every layer of the platform — workflow scheduler, AstraPBX originate-to-ai, NUC SIP forwarding, pipecat-flow gateway, Gemini Live, WhatsApp via MSG91 — in a self-contained vertical.
  • Tenant template. The org-as-tenant pattern (Hindi-Tutoring is one org out of many) means future verticals reuse the same scheduling + voice + WhatsApp infrastructure with org-scoped data.

Architecture

┌────────────────────────────────────────────────────────────────────┐
│  Cloud VPS (94.136.188.221, staging)                               │
│                                                                    │
│  ┌─────────────────┐   ┌──────────────────┐   ┌────────────────┐   │
│  │  workflow-engine │   │  astradial-editor │   │  pipecat-flow  │   │
│  │  :3002 / Bull    │   │  :3001 / Next.js  │   │  :7860 / FastAPI│   │
│  └────────┬─────────┘   └─────────┬─────────┘   └────────┬───────┘   │
│           │                        │                      │           │
│           │  fires workflow        │  bot webhooks        │  loads    │
│           │  trigger               │  (today-lesson,      │  bots/    │
│           ▼                        │   mark-progress,     │  hindi_   │
│  ┌─────────────────┐               │   call-log,          │  teacher/ │
│  │   AstraPBX      │◀──────────────┘   whatsapp)          │           │
│  │   :8000 (Node)  │                                       │           │
│  └────────┬─────────┘                                       │           │
│           │  originate-to-ai                                │           │
│           ▼                                                 │           │
│  ┌─────────────────┐                  WireGuard            │           │
│  │   Asterisk       │ ─────────────────────────────────────▶│           │
│  │   PJSIP/AMI/ARI  │  10.10.10.3 → NUC at 10.10.10.1      │           │
│  └─────────────────┘                                        │           │
│                                                             │           │
│              shared MySQL (pbx_api_db) + Firebase Firestore │           │
│              ─────────────────────────────────────────────  │           │
│              organizations, sip_trunks, did_numbers,        │           │
│              pipecat_bots, pipecat_org_config (MySQL)       │           │
│              hindi_learners, hindi_words, hindi_word_progress│           │
│              hindi_lesson_queue, hindi_call_logs (Firestore) │           │
└─────────────────────────────────────────────────────────────────────────┘

External services:
  ┌──────────────────┐    ┌────────────────────────┐    ┌────────────┐
  │ NUC (on-prem)    │───▶│ Tata Communications NNI │───▶│ Learner's  │
  │ Asterisk +       │    │ SBC 10.79.215.102:5060  │    │ phone      │
  │ WireGuard tunnel │    │                          │    │            │
  └──────────────────┘    └────────────────────────┘    └────────────┘

  ┌──────────────────┐
  │ MSG91 WhatsApp   │   POST control.msg91.com/api/v5/whatsapp/...
  │ Business API     │   per-org authkey + integrated number
  └──────────────────┘

Daily call lifecycle

00:00 IST                    workflow-engine                 Firestore
─────────                    ──────────────────              ──────────

Bull cron fires    ───────▶  runDailyQueueForOrgs(...)
                              │
                              ├─ For each active learner:
                              │   ├─ Pick word via buildLessonForUser
                              │   │   (lowest unseen frequency_rank,
                              │   │    plus due review words)
                              │   ├─ Insert hindi_lesson_queue row     ──▶ pending
                              │   └─ POST /trigger/<workflow_id>
                              │        with {name, phone, learner_id,
                              │             lesson_id, scheduled_date,
                              │             scheduled_time}
                              │
                              ▼
                            workflow-engine schedules
                            place_call at scheduled_call_time
                            (per learner's preferred_call_time)


At scheduled time            workflow-engine                 AstraPBX
─────────────────            ──────────────────              ─────────

scheduled job fires ───────▶ place_call executor
                              │
                              ├─ POST /api/v1/calls/originate-to-ai
                              │   {to: 9042321214, caller_id: 918065978021,
                              │    wss_url: ws://localhost:7860/...,
                              │    variables: {WORKFLOW_BOT_ID, learner_id,
                              │                org_id, lesson_id, ...}}
                              │
                              ▼
                          AstraPBX → AMI Originate
                              │
                              └─ PJSIP/<phone>@<trunk>
                                   ▼ (Asterisk)
                                   PJSIP INVITE to NUC (10.10.10.1)
                                   → NUC dialplan adds 0-prefix → Tata
                                   → Tata routes to PSTN
                                   → Learner's phone rings


Phone answers                AstraPBX ariClient              pipecat-flow
─────────────                ────────────────────             ────────────

Stasis Start ──────────────▶ handleAiAgentCall(channel)
                              │
                              ├─ Create mixing bridge
                              ├─ Open WebSocket to wss_url
                              │   ws://localhost:7860/ws/<org>/<bot>?...
                              │
                              ├─ Send Twilio-format handshake ────▶ router_ws
                              │                                       │
                              │                                       ├─ load_bot_module("hindi_teacher")
                              │                                       ├─ bot.set_call_metadata(...)
                              │                                       │   (loads today's lesson sync via
                              │                                       │    today-lesson webhook)
                              │                                       ├─ build LLM with full Settings
                              │                                       │   (system_instruction, VAD HIGH/HIGH,
                              │                                       │    temperature 0.7, voice "Puck")
                              │                                       └─ flow_manager.initialize +
                              │                                          queue LLMRunFrame (eager greet)
                              │
                              ├─ Bridge external media channel ◀────▶ Gemini Live audio loop
                              │   (UnicastRTP) for audio relay
                              │
                              └─ Bot teaches the word, listens for repeat


Bot ends                     pipecat-flow / bot module        Editor + Firestore
────────                     ─────────────────────────         ──────────────────

end_lesson(success | user_requested) tool fires
                              │
                              ├─ POST /api/hindi/bot/mark-progress  ──▶ hindi_word_progress
                              │   (mastery 5 if success, 2 if user_requested;
                              │    next_review_at via SM-2 schedule)
                              │
                              ├─ POST /api/hindi/bot/call-log        ──▶ hindi_call_logs
                              │   (status flips to "completed" or "early_hangup")
                              │
                              ├─ POST /api/hindi/bot/whatsapp         ──▶ MSG91
                              │   (success template / missed template)
                              │
                              └─ asyncio.create_task(_force_end):
                                  ├─ sleep 5.5s (success) or 2.5s (user_requested)
                                  │   so bot's last line plays out
                                  ├─ queue EndFrame
                                  ├─ flow_manager.transport._client.disconnect()
                                  │   ↳ underlying WebSocket force-closed
                                  │
                                  ▼ (back to AstraPBX)
                          ws.on('close') ────▶ bridge.destroy()
                                                hangupChannel(channel.id)
                                                hangupChannel(extChannelId)
                                                ↓
                                                SIP BYE → NUC → Tata → caller's phone drops


Calls that never connected   workflow-engine                 Editor + MSG91
──────────────────────────   ──────────────────────────       ───────────────

every 5 min IST   ────────▶ missed-sweeper Bull cron
                              │
                              ├─ Find queue rows:
                              │     status=pending AND
                              │     scheduled_call_time < (now - 5 min)
                              │
                              ├─ For each: POST /api/hindi/bot/whatsapp
                              │             outcome=missed                 ──▶ MSG91
                              │
                              └─ Flip queue row status to "missed"
                                  (idempotent — won't re-attempt next tick)

Components

Editor (editor/)

File Purpose
app/dashboard/[orgId]/learn/page.tsx Learners list + dropdown actions: Edit, Call now, Deactivate/Reactivate
app/dashboard/[orgId]/learn/{new,[id]/edit,lessons}/page.tsx Learner CRUD + today's lesson queue view
components/layout/app-sidebar.tsx "Learn" sidebar section gated to the Hindi-Tutoring org id
app/api/hindi/learners/route.ts + [id]/route.ts POST/PATCH learner via firebase-admin (Firestore client-SDK writes blocked by security rules)
app/api/hindi/bot/today-lesson/route.ts Returns today's primary + review words for a learner. Read-only.
app/api/hindi/bot/mark-progress/route.ts Updates hindi_word_progress with SM-2 next_review_at schedule
app/api/hindi/bot/call-log/route.ts Inserts hindi_call_logs row, transitions queue row to terminal status
app/api/hindi/bot/whatsapp/route.ts Sends recap via MSG91 with success/missed template
lib/hindi/{repos,types,admin}.ts Client-SDK reads + admin-SDK writes
lib/workflow/client.ts hindi.runDailyQueue and hindi.triggerForLearner workflow-engine clients

All bot webhooks use X-Internal-Key matching INTERNAL_API_KEY env. The learners CRUD routes are gated by Cloudflare Access on stageeditor.astradial.com.

Workflow-engine (workflow-engine/src/hindi/)

File Purpose
admin.js firebase-admin client targeting the misssellerai project. Reads FIREBASE_ADMIN_KEY_PATH env.
lessonBuilder.js Pure-logic SM-2 helpers. buildLessonForUser(learner, allWords, progress, today) returns the lesson row to insert.
runDailyQueue.js runDailyQueueForOrg: builds lessons + fires workflow trigger. triggerForLearner: one-off "Call now" path that always builds fresh + supersedes any pending row. Also normalises Indian phone numbers to 10-digit local format before sending to the NUC dialplan.
missedSweeper.js sweepMissedForOrg: finds pending rows past scheduled_call_time + 5 min grace, sends missed WhatsApp via the editor route, flips status to missed.
index.js wireHindi(app, queue) + registerHindiCron(queue). Two Bull repeatables: daily-builder (00:00 IST) and missed-sweeper (every 5 min IST). HTTP routes: POST /hindi/run-daily-queue, POST /hindi/trigger-for-learner, POST /hindi/run-missed-sweeper.

Workflow template (workflow-engine/templates/hindi_daily_lesson.json)

Webhook-triggered workflow with one place_call node. bot_id and wss_url are patched into the workflow row after instantiation (see setup guide). place_call uses timing: "scheduled" so the workflow engine delays the dial to {trigger.scheduled_time}.

Pipecat-flow gateway (pipecat-flow/)

File Purpose
bots/hindi_teacher/__init__.py Single-node freeform Gemini Live tutor. Mirrors ~/pipecat-quickstart/bots/hindi_teacher_bot.py (the local reference): same prompt, same end_lesson(reason) schema, same voice "Puck". Loads today's lesson synchronously in set_call_metadata (before LLM is built). Force-closes the WebSocket from the handler to bypass pipecat's 30-second EndFrame deferral.
gateway/pipeline.py Two opt-in hooks for module bots: (1) calls bot_module.set_call_metadata(call_metadata) before LLM construction so the bot can load per-call data, (2) reads bot_module.get_system_instruction() and passes it as Gemini Live's first-class system_instruction along with VAD HIGH/HIGH 200/400 ms + temperature 0.7. Also queues LLMRunFrame() on on_client_connected for opted-in bots so the bot starts speaking immediately rather than waiting for VAD. Existing JSON-flow bots / hotel_concierge are unaffected.
gateway/router_ws.py Already-existing WS handler. We injected the path-level org_id into extra_metadata so it's available in call_metadata (purely additive).

AstraPBX (api/src/services/asterisk/ariClient.js)

handleAiAgentCall.ws.on('close') now does three parallel teardowns when the bot's WebSocket closes: bridge.destroy(), hangupChannel(channel.id), hangupChannel(extChannelId). Each is independently .catch'd. Without this, the SIP channel stayed up 22+ seconds after the bot ended and the caller heard silence until they hung up themselves.

Data model

Firestore (per-org under astrapbx_stage/{org_id}/ on staging, astrapbx/{org_id}/ on prod)

Collection Schema
hindi_learners { name, phone_number, preferred_call_time (HH:MM), timezone, active, current_streak, longest_streak, last_successful_call_at, consent_recorded, created_at }
hindi_words { hindi_word, transliteration, english_meaning, frequency_rank, difficulty_tier, category, example_sentence_hindi, example_transliteration, example_english, pronunciation_audio_url, part_of_speech }. Word ids are stable strings like rank_0001, rank_0002.
hindi_word_progress { user_id, word_id, mastery_score (0–5 SM-2), times_taught, first_taught_at, last_reviewed_at, next_review_at }
hindi_lesson_queue { user_id, scheduled_date, scheduled_call_time, primary_word_id, review_word_ids[], status, attempt_count, created_at, missed_at?, whatsapp_missed_sent? }. Status: pending → dialing → in_progress → completed / early_hangup / missed / failed.
hindi_call_logs { user_id, lesson_id, started_at, ended_at, duration_seconds, outcome, transcript_ref, pipecat_session_id, words_actually_covered[], user_performance_json }
hindi_user_memory (optional, future) { user_id, rolling_summary, last_call_summary, total_calls_completed, updated_at }

MySQL (pbx_api_db)

The Hindi feature reuses the existing platform tables. Per-org rows for the Hindi-Tutoring tenant:

  • organizations.id = 9dede1af-c88d-454d-b2a5-d837788a71c8
  • organizations.settings.msg91_authkey (JSON) — MSG91 account authkey
  • organizations.settings.ticket_whatsapp.sender_number — registered MSG91 integrated number
  • pipecat_org_config — Google Gemini API key for this org's bots
  • pipecat_bots — one row for the Hindi Teacher bot, module_path="hindi_teacher", voice "Puck"
  • did_numbers — at least one DID assigned with is_default=1 (currently +918065978021)
  • sip_trunks — one row per org, host pointing at NUC's WireGuard IP

workflow-engine postgres

  • workflows — one row per org instantiated from the hindi_daily_lesson template
  • scheduled_jobs — Bull-backed scheduled place_call jobs for each lesson queue row

SM-2 spaced repetition

buildLessonForUser (in workflow-engine/src/hindi/lessonBuilder.js) is pure logic, no I/O:

  1. Primary word: lowest frequency_rank among words the learner hasn't been taught yet (hindi_word_progress doesn't contain the word_id).
  2. Review words: up to 3, drawn from hindi_word_progress rows whose next_review_at <= today, ordered by weakest mastery first.
  3. Schedule: scheduled_call_time = learner.preferred_call_time in learner.timezone, that day's date.

mark-progress schedules next_review_at based on SM-2 buckets:

mastery_score next_review_at
0 (didn't try) +1 day
1 +1 day
2 (hesitant / needed help) +3 days
3 +7 days
4 (clear / remembered cleanly) +14 days
5 (perfect first try / instant recall) +30 days

The bot maps end_lesson(reason) → mastery: success→5, user_requested→2.

Known limitations

  1. Initial speech latency — caller hears ~2–3 s of silence after pickup before the bot greets. Most of that is Gemini Live session setup + first-TTS-token roundtrip to Google. Pre-warming the LLM session before the call answers would shave most of it but requires a pipecat-flow gateway architecture change.
  2. Pipeline completed normally masks failed executions — the workflow_executions table records failed for the immediate-execution path on timing: "scheduled" nodes (the immediate runner sees the node deferred and propagates failure). Cosmetic only; the deferred run executes fine.
  3. Sidebar "Learn" entry is hardcoded to the Hindi-Tutoring org id in app-sidebar.tsx. If a second tutoring-style org onboards, generalise to a per-org feature flag.
  4. Missed template was PENDING Meta approval at ship time — the missed path API call returns ok but Meta gates final delivery until approval. Approved once status flips in the MSG91 dashboard; no code change needed.
  5. Phone normalisation is India-specific — runDailyQueue.fireWorkflow strips +91 / 91 / leading 0. Generalise if non-Indian learners onboard.
  6. MSG91 authkey is per-org but template name is global — env vars MSG91_HINDI_SUCCESS_TEMPLATE / MSG91_HINDI_MISSED_TEMPLATE apply to all orgs. Move to per-org settings if templates diverge across tenants.