A WhatsApp-native assistant for a mortgage/loan advisor. It reads the advisor's real WhatsApp conversations (personal number, via Green API) and automatically extracts commitments — who owes whom what, and by when — so the advisor never has to re-read a thread to remember what's still open.
Separately, it can run as a 360dialog-based business bot that replies to customers directly with an OpenAI agent (text + voice notes). The two capabilities share one FastAPI app and one database but don't otherwise interact.
Status: active prototype, ~3 days old (first commit 2026-07-05). The commitment-extraction pipeline is validated against a 93-example eval set; it has not been run against real production traffic yet, and there are two known reliability gaps before it should be (see Known Limitations).
- Why this exists
- How a message becomes a commitment
- Project layout
- Commitment extraction & evals
- Interactive failure inspector
- Data model
- Setup
- Configuration
- Run locally
- Seed the database
- Tests
- Deploy (Render)
- Endpoints
- Known limitations / roadmap
Advisors run their business out of dozens of parallel WhatsApp threads — clients, banks, lawyers. Nothing is ever explicitly "closed," so the mental load of tracking who's waiting on what falls entirely on the advisor. Tami One turns that implicit tracking into an explicit, queryable list, extracted automatically from ordinary conversation:
{
"committed_party": "me",
"required_action": "Send the material",
"deadline": "in one hour",
"status": "waiting",
"context": "Message: 'אני אשלח לך את החומר עוד שעה' ..."
}Given a real, mixed-Hebrew/English WhatsApp thread, the pipeline correctly separates a fulfilled promise (status=done) from a still-waiting one (status=waiting), attributes each to the right party, and keeps a plain-language context trail for why. That's the core proof-of-concept the project has validated so far.
- Ingest — Green API POSTs to
/webhook/green-api.normalize_green_api_message_event()turns the raw payload into aMessageEvent(chat id, sender, direction, text, timestamp). - Upsert + buffer —
upsert_contact_and_chat():- Creates a
Contactrow if this(tenant_id, chat_id)hasn't been seen before (insert-only; existing contacts are never updated). - Skips chats with more distinct senders than
MAX_GROUP_PARTICIPANTS(this is a 1:1/small-group tool, not a broadcast-list reader). - Appends the event to an in-memory
MessageBuffer, keyed by(tenant_id, chat_id).
- Creates a
- Drain — a background
asynciotask inside the running app process wakes up every 2 minutes, atomically drains the whole buffer, and hands each chat's batch of messages to the commitment extractor. - Extract —
extract_commitments()sends the batch, plus that chat's currently-open commitments, through a DSPyCommitmentAgent(ExtractCommitmentssignature +dspy.Predict), configured at startup withdspy.JSONAdapter()so it behaves like a structured-output call while staying measurable and optimizable. - Persist —
CommitmentItemrows are upserted into Postgres/SQLite, tagged with thesource_message_idsthat produced them.
The 360dialog path (/webhook/360dialog) is unrelated to this flow — it's a direct reply bot for business-number customers, using agents/core.py for in-memory per-thread conversation and services/transcription.py for voice notes.
app/
├── main.py # FastAPI app; startup wires init_db + load_cache + drain loop
├── config.py # Settings, loaded from env vars at import time
├── routers/
│ ├── green_api.py # Green API payload → normalized MessageEvent
│ ├── personal_webhook.py # POST /webhook/green-api, /admin/seed, /debug/settings
│ └── business_webhook.py # POST /webhook/360dialog — agent-driven customer replies
├── db/
│ ├── engine.py # SQLAlchemy engine (Postgres or SQLite)
│ ├── models.py # SQLModel tables
│ ├── cache.py # In-memory account/contact cache + MessageBuffer
│ ├── upsert.py # Insert-only Contact creation + buffering
│ └── seed.py # Demo data seeding
├── commitments/
│ ├── models.py # Canonical Commitment / CommitmentList schema
│ ├── commitments_agent.py # DSPy Signature + Module for commitment extraction
│ ├── extractor.py # Adapter: messages + existing commitments → DSPy agent
│ └── processor.py # Drains buffer, calls extractor, upserts CommitmentItem rows
├── agents/
│ ├── core.py # OpenAI agent used by the 360dialog business bot
│ └── memory.py # Per-thread in-memory conversation history
└── services/
├── whatsapp.py # 360dialog API client
└── transcription.py # Voice-note transcription via OpenAI
eval/
├── dataset.py # Example construction + devset loading
├── metrics.py # commitment_metric, act_vs_ignore_metric, token-F1, word-overlap
├── localize.py # Failure → root cause / subcause / repair-type / priority scoring
└── llm_judge.py # Optional LLM-as-judge for semantic field matching
scripts/
├── seed.py # CLI seed entry point
├── eval_runner.py # Local commitment-extraction eval runner (train/dev/test/challenge)
├── eval_inspector.py # Interactive CLI for examining eval failures (load or run fresh, then REPL)
└── compare_runs.py # Compare two eval runs side by side
tests/
├── evals/
│ ├── data/*.yaml # Hand-authored example definitions, 6 categories + challenge split
│ ├── generate_devset.py # YAML → train/dev/test JSON splits
│ ├── trainset.json / devset.json / testset.json / challenge_act_ignore.json
│ └── localize_cases/ # Fixtures for testing the localizer itself
└── test_*.py # upsert, webhook, commitment extraction/eval, localizer
The extractor is a DSPy Signature + Module, not a one-off prompt string:
Commitment/CommitmentList(app/commitments/models.py) are the canonical domain schema.ExtractCommitments(dspy.Signature)inapp/commitments/commitments_agent.pyencodes the extraction rules directly in the docstring — most importantly a set of act-vs-ignore rules ("we should do X" is an opinion, not a commitment; a request needs explicit acceptance; "started"/"almost done" are progress reports, not completions).eval/metrics.pyscores predictions against a hand-labeled devset: exact match on structured fields, token-F1 onrequired_action(sosettle the invoicevspay the invoiceisn't an automatic fail), and word-overlap oncontext. Three semantic fields (required_action,deadline,context) can optionally use an LLM-as-judge fallback (eval/llm_judge.py) when the deterministic check fails — enabled with--llm-judgeon the eval runner.eval/dataset.pybuilds train/dev/test splits from YAML fixtures intests/evals/data/, each tagged by category (act_vs_ignore,args_party,args_deadline,args_required_action,lifecycle_update_vs_new,lifecycle_completion), difficulty (easy/medium/hard), and a specific contrastive scenario name.
Current split sizes: train 40 · dev 22 · test 31 (93 total), generated with:
python tests/evals/generate_devset.pyModel: gpt-5.4-mini, run 20260707_023551.
python scripts/eval_runner.py --all # deterministic metrics (default)
python scripts/eval_runner.py --all --llm-judge # with LLM-as-judge fallbackRuns are saved to runs/<run_id>/ by default (use --no-save to disable). Runs with --llm-judge get a -judge suffix in the run ID.
| Split | N | TP | FP | FN | TN | Precision | Recall | F1 | Act/Ignore Acc. | Full-Commitment | Over-Extraction |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Train | 40 | 28 | 3 | 4 | 5 | 0.90 | 0.88 | 0.89 | 0.82 | 50.0% (16/32) | 38% |
| Dev | 22 | 18 | 1 | 1 | 2 | 0.95 | 0.95 | 0.95 | 0.91 | 52.6% (10/19) | 33% |
| Test | 31 | 25 | 0 | 2 | 4 | 1.00 | 0.93 | 0.96 | 0.94 | 48.1% (13/27) | 0% |
Field accuracy, measured only on true-positive commitment detections:
| Field | Train | Dev | Test |
|---|---|---|---|
committed_party |
100% | 94% | 96% |
required_action |
71% | 78% | 76% |
deadline |
86% | 78% | 80% |
context |
82% | 89% | 84% |
status |
100% | 100% | 96% |
Reading these numbers: the binary act-vs-ignore decision (should this message produce a commitment at all?) is solid across all three splits. The full-commitment metric is intentionally much stricter — it requires the right party, action, deadline, context, and status simultaneously — and sits around 50% everywhere, which is the real signal on where to invest next.
With --llm-judge enabled, the dev split's full-commitment metric rises from 52.6% to 68.4% — the gap is almost entirely semantic equivalence (reserve vs book, by Friday vs Friday) that the deterministic metric penalizes but a judge correctly accepts. This confirms the localizer's top-2 repair recommendation: fix required_action and context matching in the metric/postprocessing layer first.
eval/localize.py takes the failures.jsonl from an eval run and buckets every failure into a root cause, subcause, repair type, and a priority = impact × confidence / cost score, so effort goes to the highest-leverage fix first rather than the loudest one:
python scripts/eval_runner.py --all --save # writes runs/<run_id>/{train,dev,test,summary}.md + failures.jsonl
python -m eval.localize runs/<run_id>/failures.jsonl # rich table
python -m eval.localize runs/<run_id>/failures.jsonl --json # machine-readableAcross all 45 failures in the latest run:
| Root Cause | Count | Top Subcause | Repair Type | Priority |
|---|---|---|---|---|
required_action_normalization |
15 | verb/object too specific vs. expected | metric/postprocess | 6.8 |
context_metric_noise |
7 | paraphrase, same meaning | metric | 6.3 |
deadline_normalization |
7 | missing/extra by prefix |
metric/postprocess | 3.1 |
under_extraction_policy |
6 | external party / group obligation | signature rule | 1.7 |
update_vs_new_matching |
4 | new commitment mismatched to existing one | postprocess | 1.5 |
over_extraction_policy |
2 | refusal read as acceptance | signature rule | 0.6 |
lifecycle_policy |
2 | "started"/"almost done" read as done | signature rule | 0.6 |
party_resolution |
2 | third-party obligation implied by role | signature rule | 0.6 |
The top two repairs are both about normalization, not the model's judgment — required_action and context mismatches are usually semantically correct but worded differently than the reference (send over the docs vs send the documents), which the current exact/near-exact matching penalizes. That's cheap to fix in the metric or a postprocessing step, before touching the prompt at all.
One pattern worth calling out even though it scores lower on priority: several under_extraction_policy and party_resolution misses across all three splits are the same shape — a commitment implied by someone's role or an external party rather than stated directly ("I'm waiting for the bank to process the loan", "we need to send the documents", "the contractor needs to finish by next week"). These recur often enough that they're a real, cross-split gap even though today's failure count per category is small.
The localizer is validated with 16 deterministic tests (tests/test_localize.py) against fixed fake failures in tests/evals/localize_cases/, so its scoring logic isn't just eyeballed.
scripts/compare_runs.py diffs two eval runs side by side — showing failures fixed, new failures, and persistent failures per scenario:
python scripts/compare_runs.py runs/<run_a> runs/<run_b>Output is a rich console table (summary, per-category, per-scenario diff) saved as markdown to runs/compares/<run_a>_vs_<run_b>.md.
scripts/eval_inspector.py is an interactive CLI agent that either loads an existing run or runs a fresh eval, then drops into a REPL for examining each failure without leaving the terminal:
# Load an existing run (interactive picker lists recent runs)
python scripts/eval_inspector.py
# Load a specific run directly
python scripts/eval_inspector.py --run 20260707_183434-judge
# Run a fresh eval on a split, then enter inspector
python scripts/eval_inspector.py --split dev
python scripts/eval_inspector.py --split dev --limit 5 --llm-judgeWhen loading an existing run, the inspector joins failures.jsonl with predictions.jsonl (on split + category + scenario) to recover the original inputs (chat_id, current_datetime, existing_commitments_json) needed for re-runs.
| Command | Description |
|---|---|
list / ls |
Numbered table of all failures (or filtered subset) |
<N> |
Show full detail for failure #N: input messages, expected vs actual side-by-side diff, root-cause localization |
next / prev |
Navigate to next/previous failure from detail view |
filter <key>=<value> |
Filter by error_type, category, difficulty, split, or field |
filter clear |
Clear all filters |
filters |
Show active filters |
localize |
Run failure localization on the current (filtered) failure set — shows root-cause summary table + top suggested repairs |
rerun <N> |
Re-run failure #N through a fresh CommitmentAgent — shows pass/fail, what was fixed, what's still wrong, and any new problems |
summary |
Aggregate counts by error type, category, and difficulty |
help / h |
Show available commands |
quit / q |
Exit |
The detail view shows three sections: (1) the input messages and chat metadata, (2) a field-by-field expected vs actual diff table with ✓/✗/~ indicators (mismatches shown in red), and (3) root cause, subcause, repair type, confidence, and suggested repair text from the localizer.
Example inspector output (click to expand)
inspector>: 1
╭─────────────────────── Failure #6 — FIELD MISMATCH | args_deadline/deadline_by_end_of_week (medium) ────────────────────────╮
│ Messages: │
│ Gaddy: I will send the documents by end of week │
│ │
│ chat_id: 972501111111@c.us │
│ chat_name: Rachel │
│ current_datetime: 2025-01-06T10:00:00Z │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Expected vs Actual
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ # ┃ Field ┃ Expected ┃ Actual ┃ Match ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ 0 │ id │ None │ None │ ✓ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ committed_party │ Gaddy │ Gaddy │ ✓ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ required_action │ Send the documents │ send the documents │ ~ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ deadline │ 2025-01-10 │ None │ ✗ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ context │ I will send the documents by │ Gaddy: I will send the │ ~ │
│ │ │ end of week │ documents by end of week │ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ status │ waiting │ waiting │ ✓ │
├─────┼────────────────────┼────────────────────────────────┼────────────────────────────────┼───────┤
│ │ notification │ none │ none │ ✓ │
└─────┴────────────────────┴────────────────────────────────┴────────────────────────────────┴───────┘
╭─────────────────────────────────────────────────────── Localization ────────────────────────────────────────────────────────╮
│ Root Cause: deadline_normalization │
│ Subcause: format_mismatch │
│ Repair Type: metric_or_postprocess │
│ Confidence: 0.90 │
│ Suggested Repair: Canonicalize deadline phrases │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The rerun command is useful after making code changes — it re-runs a single failing example through a fresh agent instance and shows whether the fix worked, without re-running the entire eval set.
| Table | Status | Notes |
|---|---|---|
Tenant |
in use | multi-tenant root |
WhatsAppAccount |
in use | one row per Green API instance / 360dialog number; stores provider + provider_instance_id, not API credentials |
Contact |
in use | one row per (tenant_id, chat_id); kind field (client/bank/lawyer/internal/family/unknown) exists but nothing populates it yet |
CommitmentItem |
in use | the actual product surface — party, action, deadline, status, context, source message ids |
Chat, ChatMessage |
defined, not wired up | raw per-message history is not persisted anywhere; messages only exist transiently in the in-memory buffer until drained, then are discarded once extraction runs |
WaitingStatus, WaitingParty, Urgency, WaitingItemStatus enums |
defined, no table | scaffolding for a more general waiting/urgency model beyond commitments |
In-memory only (lost on process restart):
accounts_by_instance,contacts_by_tenant_chat_id— rebuilt from the DB on startup viaload_cache(), so these are safe.MessageBuffer._messages_by_chat— not rebuilt from anywhere. Any message ingested but not yet drained when the process restarts (deploy, crash, autoscale) is gone permanently, and there's no raw-message table to recover it from either. This is the single biggest reliability gap right now.
Also unwired: app/services/green_api_client.py defines a GreenApiClient (get chats / group data / chat history) but nothing in the app instantiates it, and WhatsAppAccount has no field to store the per-instance token it would need anyway — this is scaffolding for a future feature (e.g. backfilling chat history), not a bug in the current flow.
There's no requirements.txt — dependencies are pinned in pyproject.toml / uv.lock.
# recommended (uv.lock is committed, so this is reproducible)
uv sync
# or with plain pip
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp example.env .envEdit .env (example.env currently only lists a subset of these — worth syncing it up):
| Var | Required | Default | Purpose |
|---|---|---|---|
D360_API_KEY |
yes — app raises on startup without it | — | needed even if you're only using the Green API path |
D360_API_BASE_URL |
no | https://waba-v2.360dialog.io |
use the sandbox URL for testing |
OPENAI_API_KEY |
yes, for either pipeline to function | — | used by both the commitment extractor and the 360dialog agent |
OPENAI_MODEL |
no | gpt-5.4-mini |
model for commitment extraction and agent replies |
OPENAI_TRANSCRIBE_MODEL |
no | gpt-4o-transcribe |
voice-note transcription |
WEBHOOK_AUTH_MODE |
no | none |
none / bearer / basic — protects /webhook/360dialog only, not /webhook/green-api |
WEBHOOK_BEARER_TOKEN |
if bearer mode |
— | |
WEBHOOK_BASIC_USER / WEBHOOK_BASIC_PASS |
if basic mode |
— | |
MAX_GROUP_PARTICIPANTS |
no | 3 |
group chats with more distinct senders than this are skipped entirely |
ALLOWED_CHAT_IDS |
no | empty (no filter) | parsed into settings and exposed via /debug/settings, but not currently enforced in the Green API webhook handler |
DATABASE_PATH |
no | tami.db |
SQLite file, used when DATABASE_URL is unset |
DATABASE_URL |
no | — | Postgres URL; takes priority over DATABASE_PATH when set |
LOG_LEVEL |
no | INFO |
|
LANGSMITH_TRACING_V2 / LANGSMITH_API_KEY / LANGSMITH_PROJECT |
no | tracing off | optional observability |
CONVERSATION_MAX_MESSAGES |
no | 20 |
per-thread history cap for the 360dialog agent |
GREEN_API_BASE_URL / GREEN_API_TOKEN currently appear in example.env but aren't read anywhere in the code — Green API credentials live in the WhatsAppAccount table lookup path, not a global env var. Safe to ignore or remove.
uvicorn app.main:app --reload --port 8000
curl http://localhost:8000/health# Local, against Postgres
DATABASE_URL="postgresql://user:password@host:5432/dbname" .venv/bin/python scripts/seed.py
# On Render, after first deploy
curl -X POST https://your-app.onrender.com/admin/seedrun_seed(overwrite=True) drops the entire public schema before recreating tables. The /admin/seed route currently calls it with overwrite=False, but the route has no auth of its own — don't leave it reachable on a public URL longer than you need to.
.venv/bin/python -m pytest tests/ -vCovers Green API payload normalization, contact upsert + buffering, webhook handling, commitment extraction helpers, commitment-eval utilities, the failure localizer (16 deterministic tests), and the LLM judge (11 tests with mocked DSPy responses).
Run the commitment eval, optionally with LLM-as-judge, and regenerate the deterministic train/dev/test splits if the YAML fixtures change:
python scripts/eval_runner.py --all # deterministic (default)
python scripts/eval_runner.py --all --llm-judge # with LLM judge
python tests/evals/generate_devset.py- Connect the repo.
- Set env vars (
D360_API_KEY,OPENAI_API_KEY,DATABASE_URL, etc. — see Configuration). - Build:
pip install -e . - Start:
uvicorn app.main:app --host 0.0.0.0 --port $PORT - Seed once:
curl -X POST https://your-app.onrender.com/admin/seed
Because the drain loop and message buffer live inside the app process, any redeploy or restart between ingestion and the 2-minute drain cycle loses buffered-but-undrained messages, and Render's free tier spinning an idle instance down has the same effect. This is the top item in the roadmap below.
| Endpoint | Provider | Purpose |
|---|---|---|
POST /webhook/green-api |
Green API | Ingest personal WhatsApp messages, buffer for commitment extraction |
POST /webhook/360dialog |
360dialog | Business WhatsApp — AI agent replies, text + voice |
POST /admin/seed |
— | Seed demo data (unauthenticated) |
GET /health |
— | Health check |
GET /debug/settings |
— | Dev debug info (no secrets) |
Roughly in priority order:
- Durable ingestion. Persist
ChatMessagerows on arrival (the table already exists) instead of relying solely on the in-memory buffer. This removes the "lose messages on restart" risk and gives a permanent audit trail independent of what the LLM does with them. - Green API webhook auth is disabled (
verify_green_api_authorizationis defined but commented out inpersonal_webhook.py). Re-enable before this is exposed on a public URL for real. required_actionandcontextwording drive most eval failures (15 + 7 of 45, see Failure localization) — mostly verb/object synonyms and paraphrasing that the current metric treats as wrong even when semantically correct. Highest-leverage fix available, and it's in the metric/postprocessing layer, not the prompt.deadlineis free text ("in one hour","by Friday"), not a resolved timestamp — 7 failures trace to missing/extraby-style phrasing. Needs normalizing before it can drive urgency sorting or scheduled digests.- Commitment dedup/update matching is still partly model-driven (matching by
idinside one extraction call); 4 failures are the model creating a near-duplicate instead of updating the existing row. Worth a deterministic fallback matcher. - Under-extraction on implied/external-party commitments — "waiting on the bank," "the contractor needs to," "we need to" — recurs across all three splits. Loosening the act-vs-ignore rule for implied-party cases is the natural next prompt change.
Contact.kind(client/bank/lawyer/…) is modeled but not populated. No classifier sets it yet, so nothing downstream can filter or route by relationship type.- No digest job yet.
CommitmentItemdata is good enough to query; a scheduled per-tenant job rendering waiting commitments into a daily WhatsApp message is the natural next milestone. /admin/seedhas no auth of its own — fine for a private dev deploy, worth gating before wider use.