A shopkeeper speaks their day. It comes back as a ledger.
"आज तीन किलो चावल बेचा डेढ़ सौ रुपए में और दो हज़ार का सामान लिया"
│
├─ ASR ────────────► transcript
├─ Claude ─────────► [{sale, चावल, 3kg, ₹150}, {purchase, ₹2000}]
├─ SQLite ─────────► one ledger row per line
└─ UI / WhatsApp ──► "लिख लिया: बिक्री — 3 किलो चावल — 150 रुपए"
Over months that ledger becomes a financial history — which is the actual product, because the reason 60M+ Indian MSMEs can't borrow isn't that they don't make money, it's that they can't prove it.
| Workspace | What it is | Runs on |
|---|---|---|
apps/api |
Pipeline + HTTP API + WhatsApp webhook. Owns the ledger. | :3000 |
apps/web |
Browser front end — record, watch it parse, see the day. | :5173 |
They share nothing but HTTP. The web app is a client like any other; the WhatsApp webhook
is a second client of the same pipeline (ingest.ts),
so a fix to correction handling or the clarification flow lands on both at once instead of
drifting apart.
npm install
npm run dev # both apps, colour-tagged outputThen open http://localhost:5173. Vite proxies /api to the API, so the browser stays
on one origin and there's no CORS to think about.
Run one at a time if you prefer: npm run dev:api / npm run dev:web.
Press the mic, say your day in Hindi, release. You'll see the transcript land first (~3s), then the parsed entries (~5s) — two steps rather than one long spinner, because when the transcript is visibly wrong you want to see why before the entries appear.
- Type instead — the text box under the mic runs the identical pipeline. Fastest way to test extraction without talking.
गलतon any row voids it. Voids soft-delete and can be restored; the audit trail a lender would eventually read stays intact.- Ask for a total — say or type "आज का हिसाब". Keyword-matched before any model call, so it's free and instant.
- Answer a question — when the model can't resolve something it asks (
चीनी पचास रुपए की बेची या पचास किलो?). Your next message is read as the answer, with the original utterance still in context. - The
दुकानfield in the header is the shopkeeper id. Change it to keep separate ledgers; it persists in localStorage.
Mic access needs localhost or HTTPS. Opening the Vite URL from your phone over LAN will
load the page but the browser will refuse the microphone — use a tunnel (ngrok http 5173)
if you want to test on a real phone.
ASR_PROVIDER in .env. The pipeline behind it is identical, so switching later touches
nothing else.
| Provider | Cost | Setup | Hindi accuracy |
|---|---|---|---|
mock |
free | none | n/a — returns a fixture, ignores your audio |
whisper |
free, local | brew install whisper-cpp + npm run setup:whisper |
good on clear speech, weaker on accent/noise |
bhashini |
free for non-commercial | ULCA registration | best — purpose-built for Indic languages |
Bhashini registration is a queue you can't engineer around, so don't wait on it:
brew install whisper-cpp
npm run setup:whisper # ~1.6 GB, one time
# .env: ASR_PROVIDER=whisperNo microphone handy? npm run sample:audio synthesises a Hindi clip with the macOS
say voice, and npm run transcribe -- fixtures/sample-hi.aiff runs the whole path in
the terminal. Synthetic speech is clean and evenly paced though — it proves the plumbing,
not the accuracy.
If Bhashini stays blocked, Sarvam AI is the usual commercial
Indic fallback — same AsrProvider interface, one new file in apps/api/src/asr/.
Measured on the 13-case suite (npm run eval), effort=low:
| Model | Passed | Per voice note | 100 shops × 3/day | Avg latency | Prompt cached? |
|---|---|---|---|---|---|
claude-opus-5 |
13/13 | ₹0.61 | ₹5,460/mo (₹55/shop) | 5.2s | yes |
claude-sonnet-5 |
13/13 | ₹0.36 | ₹3,219/mo (₹32/shop) | 3.9s | yes |
claude-haiku-4-5 |
12/13 | ₹0.35 | ₹3,108/mo | 3.0s | no |
Sonnet 5 is the pick — identical accuracy at ~40% less cost and lower latency.
Haiku is a trap, for two separate reasons. Its tokens are 3× cheaper but it lands at
the same effective price, because the system prompt (~3.2K tokens) falls below Haiku
4.5's 4,096-token minimum cacheable prefix — nothing caches, so every call pays full
freight for the whole few-shot block. Sonnet's minimum is 1,024. And it failed the case
you least want failed: given दूध बेचा चालीस रुपए लीटर बारह लीटर it got ₹480 right then
wrote 480000 paise — ₹4,800, a 100× ledger error, at confidence: "high".
npm run eval warns when cache_read comes back zero. That's how the Haiku problem
surfaced, and it's a silent 10× on your largest cost line.
ASR is free running Whisper locally, so the model call is your only marginal cost.
apps/api/src/
extract/prompt.ts ← the system prompt. Most of the real work.
extract/schema.ts ← Zod schema the model is constrained to
extract/extractor.ts ← the Claude call
pipeline/ingest.ts ← transcript → ledger. Shared by web + WhatsApp.
pipeline/process.ts ← the WhatsApp entry point
pipeline/summary.ts ← Hindi confirmation and daily summary text
api/routes.ts ← HTTP surface for the browser
asr/ ← AsrProvider interface + whisper / bhashini / mock
whatsapp/ ← Cloud API client, webhook types, signature verification
db/schema.sql ← the ledger. Portable SQL; SQLite-isms flagged in comments.
scripts/ ← eval, replay, transcribe, smoke, setup
apps/web/src/
App.tsx ← state machine: record → transcribe → extract → render
hooks/useRecorder.ts ← MediaRecorder + level metering
components/ ← Recorder, EntryCard, DayPanel
styles.css ← design tokens, light/dark, all of the visual design
Money is integer paise, everywhere. amount_paise, never amount. Floats and money
don't mix, and a bookkeeping app that's off by a rupee is worse than no app.
Structured output, not JSON-in-a-prompt. The schema is enforced by the API
(output_config.format), so there's no parse-retry loop and no "output ONLY valid JSON"
begging in the prompt.
The system prompt is byte-stable and cached. No timestamps, no shopkeeper names, no conditional sections — everything per-request lives in the user turn, after the cache breakpoint. The few-shot block is a cache read from the second call onward.
Corrections void, they don't mutate. "नहीं, वो चार सौ का था" sets voided = 1 on the
old rows and writes new ones.
The model asks instead of guessing. A missing amount produces a spoken-style question
and no invented number. Unresolved lines park in the clarifications table and the next
message is read as the answer — the whole human-in-the-loop story for a user who can't
type a correction.
Every API response returns the full day. Routes hand back the refreshed snapshot, so the client replaces state wholesale instead of stitching together deltas.
Spoken Hindi is not written Hindi, and this is where accuracy lives:
| Problem | Handled in |
|---|---|
डेढ़ = 1.5, साढ़े तीन = 3.5, पौने दो = 1.75 |
prompt.ts → "Numbers as people actually say them" |
150 रुपए में (total) vs 150 रुपए किलो (rate) |
prompt.ts → "Rate versus total" |
किलो/दर्जन/बोरी/पाव → closed unit enum |
prompt.ts → "Units", schema.ts → UnitSchema |
उधार — a sale that isn't cash |
on_credit; excluded from net cash, tracked as outstanding |
| Old debt settled ≠ new sale | payment_received kind |
ASR mangling (चवाल, किलू) |
prompt.ts → "Speech recognition noise" |
| Several transactions in one breath | one row per line of business |
Every one has a case in npm run eval. When the model gets a real voice note wrong, add
the transcript there with the correct answer before touching the prompt — otherwise a fix
for today's bug silently breaks last week's.
Optional — the web app is a complete client on its own.
- Meta app → add WhatsApp → note the Phone number ID and access token.
- Fill
.env:WHATSAPP_PHONE_NUMBER_ID,WHATSAPP_ACCESS_TOKEN,WHATSAPP_APP_SECRET, and invent aWHATSAPP_VERIFY_TOKEN. npx ngrok http 3000, set the callback tohttps://<ngrok>/webhook, subscribe to messages.
Two things that bite people, both already handled: signature verification needs the raw
bytes (re-serializing the parsed JSON breaks the HMAC), and media download is two hops
where the second fetch still needs the bearer token. The webhook also acknowledges with a
200 before doing any work, because transcription plus a model call can exceed Meta's ~20s
timeout — redelivery is safe since the WhatsApp message id is the messages primary key.
Verified by running it:
npm run typecheckclean across both workspaces;npm run buildproduces a bundle.npm run smoke— 10/10 (storage, aggregation, udhaar excluded from cash, formatting).npm run eval— 13/13 on Opus 5 and Sonnet 5, 12/13 on Haiku 4.5.- API routes exercised over HTTP:
/api/config,/api/text(real extraction),/api/transcribe(real audio through local Whisper),/api/ledger. - Web dev server serves and proxies
/apicorrectly.
Not verified: the browser UI in an actual browser. It builds and serves, but I have no way to click it here — the recorder, animations, and layout are unexercised. Expect to find something.
Whisper mangles Hindi number words. On the sample clip it heard डेढ़ सौ (150) as
देर सो. The extractor recovered 150 from context, but a garbled number is the one error
class that silently corrupts a ledger. Worth a targeted eval case once you have real
recordings.
Bhashini adapter is written against the documented two-call shape but untested.
- Open the UI and use it. Fix whatever's broken.
- Set
ANTHROPIC_MODEL=claude-sonnet-5. - Record 20–30 real voice notes from one actual shopkeeper. Synthetic clips prove the plumbing, not the accuracy.
- Every wrong ledger entry becomes an eval case before you touch the prompt.
- Then: CSV/PDF export of
transactions— that export, with 3–6 months behind it, is what a lender actually reads.