Real-time stampede prevention: CCTV person detection (YOLOv8) + voice reports + live NDMA government alerts, fused into one state machine that drafts, voices (Sarvam AI), and broadcasts multilingual PA announcements — and phones every responder in their own language via Twilio.
Sarvam Epoch Buildathon 2026. On 29 January 2025, at least 30 pilgrims died in a crowd crush at the Sangam during Mauni Amavasya at the Kumbh Mela. People on the ground knew the crowd was turning. What failed was the announcement: by the time a constable radioed a control room, someone decided, and someone read it out (in Hindi and English, at a gathering that speaks a dozen languages), the crush had already happened.
Sanket fuses three live sensors into one governed mela state and computes every announcement from state changes. Nobody types an announcement, ever.
The metric we move: incident-to-announcement time, from minutes to about four seconds, in five languages.
SENSE (n→1) STATE ANNOUNCE (1→n)
───────────── ───── ──────────────
🎙 Voice reports ──┐
(Hinglish, Saaras) │ zone state machine computed PA text
📷 CCTV person ├──► NORMAL→CROWDED→CRITICAL ──► voiced in 5 langs
count (YOLOv8) │ escalation memory (Bulbul v3)
🏛 LIVE NDMA SACHET┘ provenance per change + pilgrims ask back
govt alert feed in THEIR language
- Voice is a sensor, not a command line. A marshal at a gate does not open a dashboard, he just talks: "Gate do pe bahut bheed hai, log gir rahe hain." Saaras v3
translatemode understands the code-mixed speech and detects the speaker's language. - CCTV without a cloud vision API. Four mela cameras: Gate 1 (Sangam Entry), Gate 2 (Akshayavat Entry), and two ambient ghat cams. Local YOLOv8 counts people per second on real footage, each camera's thresholds calibrated to its own field of view. When Gate 1 crosses critical, Sanket closes it and computes a redirect to Gate 2, through the same pipeline as voice.
- Real government data, live. We poll NDMA SACHET's public CAP feed, the actual national disaster-alert system, with 40+ monsoon alerts live today. When a red alert arrives in Gujarati, we voice the government's original text in Gujarati, plus translations. Pilgrims hear their government verbatim.
- Memory that judges can see. A second report on a flagged gate does NOT re-announce. The board shows "already under management, escalating". Announcement spam is exactly what erodes crowd trust in a real emergency. Stand-downs propagate, and state survives a server kill (SQLite).
- Two-way. "Bhaiya, khaali gate kaunsa hai?" is answered from live state, spoken back in the asker's language.
- Call notifications (1→n over the phone). The same computed announcement can be phoned to every marshal, each hearing it in their own language, voiced by Sarvam (Bulbul). Gate marshals get Hindi, zone wardens get Bengali. Real calls go out via Twilio when configured; with no telephony account it runs in a fully-logged simulation so the flow is demoable anywhere. Manage recipients and place calls at
/callers.
| Path | Scenario |
|---|---|
/mela |
Kumbh Mela gate redirection. Two guarded entry gates plus two ambient ghat cams, live YOLOv8 counts, gate closure and redirect announcements. |
/ |
Railway station. The same engine on CSMT Mumbai platform footage, showing the pipeline is not tuned to one venue. |
computed PA announcement (already voiced in EN/HI/BN/MR/KN by Bulbul)
│
├─ recipient speaks Hindi → Play 1_hi-IN.wav on the call
├─ recipient speaks Bengali → Play 1_bn-IN.wav on the call
└─ recipient speaks Tamil → Mayura translate + Bulbul voice on the fly → Play
- Per-recipient language. Each contact has a phone number and a language. We play that language's clip; if the announcement was not voiced in it, we translate the English master (Mayura) and voice it (Bulbul) on demand. Nobody is called in a language they do not speak.
- Provider is Twilio. An outbound call is placed with inline TwiML that
<Play>s the.wavtwice, because a phone alert is heard once, in a noisy place. Twilio must be able to fetch the audio, so setPUBLIC_BASE_URLto your public tunnel. - Simulation mode. With no Twilio credentials, every call is recorded in the log exactly as it would be dialled (number, language, audio URL, TwiML). Perfect for the demo without a phone bill.
In .env:
TWILIO_ACCOUNT_SID=ACxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxx
TWILIO_FROM_NUMBER=+1XXXXXXXXXX # a Twilio voice number
PUBLIC_BASE_URL=https://abc.trycloudflare.com # so Twilio can GET the .wav filesLeave them blank for simulation. Endpoints:
| Method | Path | Purpose |
|---|---|---|
| GET | /callers |
Call-notifications console (add recipients, call announcements out, watch statuses) |
| GET/POST/DELETE | /contacts |
Manage recipients (name, phone, language) |
| POST | /announcements/{id}/call |
Phone every recipient with that announcement, each in their language |
| GET | /calls |
Recent call log + current mode |
Everything lives in the nto1/ folder. Run all commands from inside it.
cd nto1
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt # or: uv venv + uv pip install
cp .env.example .env && echo "SARVAM_API_KEY=sk_..." >> .env # add your key
.venv/bin/uvicorn app:app --port 8000 # open http://localhost:8000/mela
cloudflared tunnel --url http://127.0.0.1:8000 # public link, also set as PUBLIC_BASE_URL for real callsrequirements.txt covers the app and call notifications. CCTV analysis (cctv.py, gates_detect.py) additionally needs ultralytics (commented in requirements.txt).
python gates_detect.py re-annotates the mela gate clips, python cctv.py regenerates station CCTV analysis, and python test_demo.py runs the full demo sequence 3× with assertions (36 checks).
| API | Model | Role |
|---|---|---|
| Speech-to-Text | saaras:v3, mode=translate | code-mixed reports and queries → English, plus language detect |
| Text-to-Speech | bulbul:v3 | every announcement in en/hi/bn/mr/kn, original-script govt alerts, per-recipient phone calls |
| Chat Completions | sarvam-30b (JSON mode) | intent and field extraction, state-grounded query answering |
| Translate | mayura:v1 | announcement text → regional languages, including on demand for a caller's language |
| Text LID | text-lid | detect the real language of govt alerts (their lang field lies) |
Selected Sarvam parameter: Voice Experience.
- (30s) 29 Jan 2025, Mauni Amavasya at the Kumbh: at least 30 pilgrims died in a crush at the Sangam. Two weeks later, 18 more died at New Delhi station waiting for a train to that same Kumbh. In both cases the announcement was the thing that failed.
- (30s) Baseline today: a constable radios a control room, someone decides, someone types or reads it out in Hindi and English only. Three to four people, several minutes per incident.
- (2min live, on
/mela)- Gate 1 person count crosses critical → gate closes → five-language redirect to Gate 2 plays, about 4s from detection to PA.
- Hinglish crush report into the mic → board flips CRITICAL → announcement computed from the state change, not typed.
- Same report again → "already under management, escalating", no repeat announcement.
- Real NDMA red alert (Gujarat SDMA) → voiced in Gujarati original plus translations.
- Judge speaks their own report or question into the mic → answered from live state in their language.
- Phone tree on
/callers: every marshal called in their own language. - Stand-down clears the gate. Reset. Repeatable.
- Announcement generation takes about 8 to 15s for five languages (parallel TTS); voice-report ack is about 4s. Latency is shown per-hop in the UI ticker.
- TTS covers 11 languages. Alerts in unsupported languages (Mizo, for example) degrade to translated Hindi and English with the original text shown.
- Mela and station footage is pre-recorded (10 downloaded Kumbh clips, four public CSMT Mumbai recordings). The analysis is genuinely local YOLOv8, replayed through the live sensor endpoint.