Consumer-Finance Complaint Classification, RAG-Grounded Draft Replies & Confidence Routing — with a Human Always in the Loop
LangChain · LangGraph · Chroma · FastAPI · Pydantic v2 · Claude / OpenAI
Ingests a raw consumer-finance complaint (real CFPB data), returns structured JSON — category, urgency, team, confidence, and a draft reply grounded in similar past cases. Every number below is measured, not estimated.
All numbers below come from evals/eval_report.json — a single run of the full
pipeline over 300 real CFPB complaints (golden set) with gpt-4o, plus a
free local retrieval eval. Nothing here is projected or estimated.
| Metric | Value | How it's measured |
|---|---|---|
| Category accuracy | 0.803 | 300 real complaints vs CFPB product labels |
| Category macro-F1 | 0.817 | Equal-weight mean over the 6 ground-truth classes |
| Retrieval recall@1 / @3 / @5 | 0.76 / 0.91 / 0.95 | Top-k similar past complaints share the query's category (v1.1 chunked index) |
| Retrieval MRR | 0.84 | Mean reciprocal rank of first relevant hit |
| Groundedness | 50/50 = 1.00 | LLM-judge over sampled draft replies: zero fabricated claims |
| Latency | 4.75 s avg · 8.66 s p95 | End-to-end pipeline (2 sequential LLM calls + local retrieval) |
| Cost per ticket | $0.0079 | 752K input + 48K output tokens measured across the run |
| Full eval run cost | $2.36 | 300 tickets + 50 judge calls |
| Auto-routed | 99.0% | Confidence ≥ 0.85 (see limitations — confidence is poorly calibrated) |
Per-class F1: bank_account 0.96 · mortgage 0.93 · loans 0.84 ·
debt_collection 0.73 · credit_card 0.73 · credit_reporting 0.72.
The three ~0.73s share one confusion: complaints about cards/debts that
mention a credit report get pulled into credit_reporting
(precision 0.57, recall 0.98). See Limitations.
Quality gates in CI: category macro-F1 ≥ 0.80 and groundedness ≥ 0.90
and retrieval recall@1 ≥ 0.60 — a bad prompt or model change fails the
build (evals/run_evals.py, evals/eval_retrieval.py, .github/workflows/ci.yml).
Support agents manually read, tag, and route incoming complaints — minutes of pure triage overhead per ticket before any actual help happens. Routing is agent-dependent, urgency is guessed, and first responses wait in a queue.
This system ingests a complaint and returns structured JSON in ~5 seconds:
{
"category": "debt_collection",
"urgency": "high",
"suggested_team": "collections",
"confidence": 0.94,
"draft_reply": "Thank you for reaching out... Our collections team will review...",
"retrieved_ticket_ids": ["9390210", "12497926", "21732177"],
"routing_action": "auto_route",
"is_draft": true
}It is trained and evaluated on the CFPB Consumer Complaint Database — millions of labeled, free-text finance complaints — so the accuracy numbers describe real consumer writing, not synthetic templates.
⚠️ Hard rule: we draft, we never auto-send. Every reply is flaggedis_draft: trueand posted for agent review. Routing is a suggestion gated by a confidence threshold — not blind automation. No auto-send path exists anywhere in the codebase.
flowchart LR
subgraph Sources
ZD[Zendesk Webhook]
CL[API Client / curl]
end
subgraph Service["FastAPI Service (triage/api.py)"]
EP["POST /triage · POST /triage/batch<br/>GET /health · GET /metrics"]
end
subgraph Pipeline["LangGraph Pipeline (triage/pipeline.py)"]
IG[Input Guard]
CLS[LLM Classifier]
RET[Retrieve Similar Cases]
RG[Reply Generator]
OG[Output Guard]
AS[Assemble TriageOut]
FB[Safe Fallback]
end
subgraph RAG["Retrieval Layer (local)"]
CH[(Chroma<br/>1,200 CFPB cases)]
EMB[MiniLM embeddings<br/>all-MiniLM-L6-v2]
end
subgraph LLM["Provider-Agnostic LLM Layer"]
CLAUDE[Claude API<br/>primary]
OAI[OpenAI<br/>swappable]
end
ZD --> EP
CL --> EP
EP --> IG
IG --> CLS
CLS --> RET
RET --> RG
RG --> OG
OG --> AS
IG -. "guard fails" .-> FB
OG -. "guard fails" .-> FB
RET --- CH
CH --- EMB
CLS --> CLAUDE
CLS -.-> OAI
RG --> CLAUDE
AS --> EP
FB --> EP
The pattern is classify → retrieve → reply → guard → route. Retrieval is an enhancement, never a dependency: if the vector store is missing or fails, the reply is generated without context and the pipeline continues. Every hard failure lands in the safe fallback — the system never crashes the caller and never emits unvalidated output.
| Principle | Meaning |
|---|---|
| Schema is law | Everything the LLM returns is validated against Pydantic models. Invalid → retry once → fallback. |
| Human in the loop | Low confidence and anything customer-facing is drafted, never sent. |
| Fail safe, never silent | Every error returns a valid TriageOut with routing_action: "human_review" and a logged error. |
| Real data, real numbers | The golden set is 300 real CFPB complaints; every README metric is measured by the eval harness. |
| Quality gates as code | CI fails on F1, groundedness, or recall regression — not on vibes. |
stateDiagram-v2
[*] --> InputGuard : Complaint text
InputGuard : Input Guard
InputGuard : • reject empty / oversized
InputGuard : • block prompt injection
InputGuard : • redact PII before LLM
Classify : LLM Classify
Classify : • with_structured_output()
Classify : • CFPB-domain few-shots
Classify : • category / urgency / team / confidence
Retrieve : Retrieve Similar Cases
Retrieve : • Chroma + local MiniLM
Retrieve : • top-3, filtered by predicted category
Retrieve : • failure → empty context, never fallback
Reply : Reply Generator
Reply : • grounded in complaint + retrieved cases
Reply : • cites case IDs like [#9390210]
Reply : • never fabricates specifics
OutputGuard : Output Guard
OutputGuard : • validate taxonomy values
OutputGuard : • block risky promises / PII / profanity
OutputGuard : • enforce confidence threshold
Assemble : Assemble TriageOut
Fallback : Safe Fallback
InputGuard --> Classify : pass
InputGuard --> Fallback : fail
Classify --> Retrieve : valid schema
Classify --> Fallback : LLM error / invalid
Retrieve --> Reply : context (or empty)
Reply --> OutputGuard
OutputGuard --> Assemble : pass
OutputGuard --> Fallback : fail
Assemble --> [*] : TriageOut
Fallback --> [*] : TriageOut (human_review)
- One orchestrator, two callers. The API and the eval runner call the same graph, so eval numbers describe exactly what production does.
- Fallback is a first-class node, not scattered exception handlers.
- Retrieval fails soft — a missing/broken vector store degrades quality,
never availability (contract pinned by
tests/test_retrieval.py).
Every choice below states what it buys, what it costs, and why it won.
Pros: real consumer writing (typos, rants, half-stories); labels come with
the data; results transfer to production-like distributions.
Cons: labels are consumer-selected products — inherently noisy (a card
dispute filed under "credit reporting" is wrong-ish ground truth, and it
shows in the confusion pattern); narratives are pre-redacted (XXXX), which
under-exercises our PII guards; only ~half of complaints have published
narratives (consent bias).
Verdict: noisy real labels beat clean fake ones. The measured 0.80 accuracy
is an honest number; the old synthetic set's numbers were not.
The original SaaS taxonomy (billing/technical/shipping…) had no honest mapping
onto finance complaints. Rather than force a lossy map, the taxonomy was
rebuilt around CFPB product groups (see table below). general was kept as a
fallback-only class — never sampled as ground truth, so predictions into it
always cost recall.
Verdict: a taxonomy that matches the data keeps every eval number
interpretable. The alternative (mapping "Mortgage" → "billing") would make F1
meaningless.
Pros: two dependencies instead of four; retrieval is a metadata-filtered top-k query, no chain composition needed; local embeddings are free ($0/query), private (complaints never leave the box for embedding), and fast. Measured recall@3 of 0.91 says 384-dim MiniLM is good enough for this corpus. Cons: MiniLM trails large API embedding models on nuanced similarity; the index must be rebuilt from the corpus on deploy (one command, baked into the Docker image). Verdict: measured recall settles the debate — no API embedding bill needed.
Pros: ~100 lines using the exact with_structured_output house pattern;
claim-level evidence (fabricated_claims listed verbatim); zero new
dependencies — deepeval, presidio, and guardrails-ai were removed from
requirements since nothing imported them.
Cons: no framework-standard metrics to compare against; judge quality is
our own responsibility.
Verdict: a legible bespoke judge beats an unused dependency. Measured
result: 50/50 replies grounded, zero fabricated claims.
Pros: conditional routing (fail → fallback) is declarative; every node — guards and retrieval included — appears in traces; adding the retrieve node was a 3-line graph change. Cons: heavy dependency for a mostly linear flow; framework churn. Verdict: the RAG addition partially vindicated the choice — the graph absorbed a new node with no orchestration rewrite. Still heavier than strictly necessary.
Pros: the #1 reliability lever — the LLM cannot return free-form mush; schema violations are detectable, retryable, countable. Cons: edge-case complaints get shoehorned into the nearest enum rather than flagged; schema tokens cost input budget every call. Verdict: non-negotiable. Free-text parsing is where LLM products go to die.
Pros: classification fails on accuracy, replies fail on tone and hallucination — separation keeps prompts and evals attributable. Cons: two sequential LLM calls per ticket → measured 4.75s average latency and roughly 2× the cost of a combined call. Verdict: correctness of attribution won. The measured $0.0079/ticket makes the 2× cost concrete — and acceptable at this volume.
The original pin (claude-sonnet-4-20250514) is retired — a hard lesson in
pinning dated model snapshots. Default is now claude-sonnet-4-6 ($3/$15 per
MTok); Sonnet 5 was rejected because it rejects the temperature=0.0 the
classifier relies on. The provider layer let this eval run execute entirely on
gpt-4o ($2.50/$10 per MTok) with a config switch — which is the point of
having one.
Verdict: provider-agnostic wasn't hypothetical insurance; it was used in
anger the first week it mattered.
v1.0 embedded whole narratives, but all-MiniLM-L6-v2 truncates at 256
tokens (~1,000 chars) — the tail of long complaints (up to 2,000 chars) never
influenced similarity. v1.1 chunks narratives into ~200-token windows with
25% overlap (1,200 docs → 2,030 chunks) and dedupes hits back to parent
complaints, so the reply prompt sees the best-matching passage instead of
the narrative head.
| Retrieval config | recall@1 | recall@3 | recall@5 | MRR | Latency |
|---|---|---|---|---|---|
| v1.0 whole-narrative | 0.763 | 0.907 | 0.957 | 0.838 | ~15 ms |
| v1.1 chunked (shipped) | 0.763 | 0.907 | 0.950 | 0.838 | 23 ms |
| v1.1 chunked + cross-encoder rerank (rejected) | 0.740 | 0.900 | 0.943 | 0.825 | 83 ms |
Readout: chunking holds the proxy metrics at parity while eliminating the
truncation defect and improving reply context — shipped. The cross-encoder
(ms-marco-MiniLM-L-6-v2) regressed recall@1 by 2.3 points — most likely a
proxy-metric mismatch: it promotes semantically relevant chunks from
neighboring categories, which our category-match relevance can't credit — but
we gate ships on the measured number, so it ships off by default
(rerank_enabled flag; implementation + tests retained, revisit with
human-labeled relevance).
Verdict: the eval harness did its job — it killed a plausible-sounding
improvement before it reached production.
Pros: explainable in one sentence; one number in config.py.
Cons: measured routing shows the problem — 99% of complaints cleared the
0.85 bar, because LLM self-reported confidence clusters high regardless of
correctness (accuracy was 0.80, so ~1 in 5 auto-routed complaints was
mis-categorized). The threshold as-is barely filters.
Verdict: kept for transparency, flagged as mis-calibrated by our own eval.
Next step is calibrating the threshold per-category against the measured
confusion matrix — the eval harness now produces exactly the data needed.
Pros: zero chance of a hallucination reaching a customer unsupervised — and in finance, a fabricated promise is a regulatory problem, not just a bad look. Cons: caps the automation ceiling; an agent still touches every reply. Verdict: deliberate product decision. Judged groundedness of 1.00 is encouraging, not a license to auto-send.
- Ground-truth labels are noisy. CFPB categories are chosen by consumers.
The dominant confusion — card/debt complaints mentioning credit reports
pulled into
credit_reporting(P=0.57, R=0.98) — is partly label ambiguity, not just model error. - Confidence ≠ probability. 99% of complaints exceed the 0.85 threshold while accuracy is 0.80. The confidence gate needs calibration before the auto-route split means anything operationally.
- No urgency ground truth. CFPB has none, so urgency is predicted but ungated — reported as a distribution only. Don't quote urgency accuracy; there isn't one.
- Retrieval relevance is a proxy. recall@k counts a category match as relevant. True usefulness-for-reply relevance would need human labels.
- Judge is an LLM judging an LLM. Same-family self-preference bias is real; the claim-level-evidence prompt mitigates but doesn't eliminate it.
- PII guards are under-tested by this dataset. CFPB narratives arrive
pre-redacted (
XXXX), so the regex redaction paths are exercised by unit tests, not by the eval set. - Latency is honest but slow. 4.75s average / 8.66s p95 — two sequential LLM calls dominate. Batching, caching, or a distilled classifier are the levers at scale.
- English-only. Prompts, guards, and eval data assume English.
- No authentication on the API. Assumes deployment behind a trusted gateway.
- Regex guardrails are bypassable. Defense-in-depth, not a security boundary; false positives route to a human (the cheap failure).
support-ticket-triage/
│
├── triage/
│ ├── config.py # CFPB taxonomy enums, thresholds, RAG settings
│ ├── schema.py # Pydantic models — the contract
│ ├── data_prep.py # CFPB narrative cleaning + dedupe helpers
│ ├── classifier.py # Structured classifier (finance few-shots)
│ ├── retrieval.py # Chroma + MiniLM similar-case retrieval
│ ├── reply_gen.py # Draft-reply chain, grounded in retrieved cases
│ ├── guardrails.py # Input/output safety guards (regex)
│ ├── pipeline.py # LangGraph: guard→classify→retrieve→reply→guard
│ ├── api.py # FastAPI: /triage /triage/batch /health /metrics
│ └── monitoring.py # Prometheus metrics + structured logs
│
├── evals/
│ ├── eval_set.jsonl # 300 real CFPB complaints (50 × 6 categories)
│ ├── metrics.py # Accuracy, macro-F1, confusion, cost-from-tokens
│ ├── judge.py # LLM-as-judge groundedness verdicts
│ ├── eval_retrieval.py # recall@k + MRR (free, runs in every CI)
│ ├── run_evals.py # Full harness + regression gates
│ └── eval_report.json # The measured numbers quoted in this README
│
├── data/kb/
│ └── cfpb_corpus.jsonl # 1,200 complaints powering the RAG layer
│
├── scripts/
│ ├── build_eval_set.py # CFPB API → golden set + corpus (disjoint IDs)
│ └── ingest_kb.py # Corpus → Chroma index (rebuildable artifact)
│
├── tests/ # 52 tests (guards, pipeline, retrieval, API)
├── integrations/zendesk.py # Webhook in → tags + internal-note draft out
├── Dockerfile # CPU torch + pre-baked MiniLM + baked index
├── .github/workflows/ci.yml # lint + tests + eval gates
└── CLAUDE.md # Build specification
| Category | CFPB product group | Routed team |
|---|---|---|
credit_reporting |
Credit reporting or other personal consumer reports | credit_disputes |
debt_collection |
Debt collection | collections |
mortgage |
Mortgage | mortgage_team |
credit_card |
Credit card (+ legacy "Credit card or prepaid card") | card_services |
bank_account |
Checking or savings account | deposit_accounts |
loans |
Student / vehicle / payday & personal loans | lending |
general |
(fallback only — never ground truth) | general_support |
Urgency (critical / high / medium / low) is predicted for routing context but
has no CFPB ground truth — it is reported, never gated.
Built by scripts/build_eval_set.py: downloads recent (2024+) complaints with
published narratives from the CFPB public Search API, cleans them (collapses
XXXX redaction runs, 200–2,000 char length filter, near-duplicate removal),
and stratified-samples 50 per category. The remaining ~1,200 go to the RAG
corpus — disjoint by complaint ID, so retrieval evals can never fetch the
query's own ticket.
| Eval | Cost | Gate |
|---|---|---|
| Classification accuracy / macro-F1 / confusion | ~$2.40 per full run (measured) | macro-F1 ≥ 0.80 |
| Groundedness (LLM judge, 50-reply sample) | <$0.50 | grounded rate ≥ 0.90 |
| Retrieval recall@k + MRR | free (local embeddings) | recall@1 ≥ 0.60 |
| Token/cost accounting | — | reported per run |
python -m evals.eval_retrieval # free — no API key needed
python -m evals.run_evals --max-tickets 60 # capped smoke (~$0.70)
python -m evals.run_evals # full 300 + judge (~$3)CI runs the retrieval eval on every push, and the LLM evals (capped at 60
tickets + 20 judge samples) whenever the ANTHROPIC_API_KEY secret is
configured — skipped gracefully on fork PRs. A regression on any gate fails
the build.
| Guard | Catches |
|---|---|
| Empty/Blank | Whitespace-only complaints |
| Oversized | > 10,000 chars |
| Prompt Injection | "Ignore previous instructions…", DAN mode, system overrides |
| PII Redaction | Emails, phones, SSNs, credit cards — redacted before the model sees them |
| Guard | Catches |
|---|---|
| Taxonomy Validation | Invalid category/urgency/team values |
| Risky Promises | "I've processed your refund", "Account updated" |
| PII in Reply | Email/phone/card echoed back |
| Profanity / Empty | Inappropriate or blank drafts |
On any failure the caller still gets a valid TriageOut with
routing_action: "human_review" and confidence: 0.0.
The RAG prompt adds one more rule: retrieved cases may be cited ([#id])
for grounding, but their specifics (amounts, companies, outcomes) must never
be copied into the reply — and the groundedness judge measures compliance
(50/50 grounded in the last run).
{
"text": "A debt collector keeps calling me at work about a medical debt I already paid off...",
"ticket_id": "TICKET-42"
}{
"triage_id": "7a39924d-e46c-40d9-9b11-596398883c66",
"ticket_id": "TICKET-42",
"category": "debt_collection",
"urgency": "high",
"suggested_team": "collections",
"confidence": 0.94,
"draft_reply": "Thank you for reaching out... Our collections team will review your case...",
"retrieved_ticket_ids": ["9390210", "12497926", "21732177"],
"is_draft": true,
"routing_action": "auto_route",
"model_used": "gpt-4o",
"processing_time_ms": 4581.2,
"error": null
}POST /triage/batch— up to 50 complaints, one failure never poisons the batchGET /health— liveness + active modelGET /metrics— Prometheus payload (volume, errors, routing split, latency, confidence histograms)- Swagger UI at
http://localhost:8000/docs
- Python 3.12+
- An Anthropic or OpenAI API key (the provider layer takes either)
git clone https://github.com/vigoroth/support-ticket-triage.git
cd support-ticket-triage
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # then edit .env — add your key. NEVER put keys in .env.example (it's tracked)python scripts/build_eval_set.py # golden set + corpus from CFPB API (already committed)
python scripts/ingest_kb.py # corpus → Chroma index (gitignored artifact)pytest tests/ -m "not slow" # 51 fast tests, no API key needed
python -m triage.api # http://localhost:8000/docs
python -m evals.eval_retrieval # free retrieval eval
python -m evals.run_evals # full eval (~$3 with gpt-4o)docker build -t triage . # CPU torch, pre-baked MiniLM + index (~2.1GB)
docker run --rm -p 8000:8000 --env-file .env triage- Confidence calibration — per-category thresholds fitted to the measured confusion matrix (the 99% auto-route rate is the smoking gun)
-
credit_reportingdisambiguation — prompt or post-hoc rule for the card/debt-vs-report confusion cluster - Human-labeled retrieval relevance — replace the category-match proxy; then re-evaluate the (currently rejected) cross-encoder reranker against labels it can actually win on
- Hybrid BM25 + dense retrieval — next measurable retrieval experiment (one variable at a time)
- Latency — parallelize classify+retrieve, or distill the classifier (4.75s → target <2s)
- Cross-provider A/B — same harness, Claude vs GPT, decided by F1-per-dollar
- API authentication — key-based auth outside trusted gateways
- NER-based PII redaction — beyond regex patterns
MIT — see LICENSE.
Built with LangChain, LangGraph, Chroma, and FastAPI
Production LLM engineering: real data, measured numbers, quality gates as code, and a human always in the loop.