Self-correcting multi-agent document extraction on LangGraph: classify, extract, validate, and loop validator feedback back into the extractor until the output is right or a human needs to see it.
- Single-shot LLM extraction fails silently: malformed dates, currency symbols in amounts, missing fields, all shipped downstream. Here every attempt passes a deterministic validator before anything leaves the pipeline.
- Retry loops without feedback just repeat the same mistake. The validator's specific error messages ("field 'due_date': '01/01/2026' is not a valid ISO date") are injected into the next extraction prompt, so the second attempt knows exactly what to fix.
- Unbounded agent loops burn tokens forever. Retries are capped, and documents that cannot be fixed escalate to a terminal human-review state with the full trace and unresolved errors attached.
flowchart LR
IN[Document] --> C[classify agent]
C -- known type --> E[extract agent<br/>pluggable backend]
C -- unknown --> U[unsupported]
E --> V[validate agent<br/>deterministic rules]
V -- valid --> DONE[extracted]
V -- "errors, retries left" --> E
V -- "errors, retries exhausted" --> H[escalate<br/>human review queue]
The validate to extract edge is the core of the design: a real cycle with bounded retries and feedback flowing through typed shared state, which is exactly the case where a graph beats a function pipeline (ADR 0001).
Actual output of python examples/demo.py with the offline rule backend:
=== malformed invoice -> escalated
classify: invoice
extract[rules] attempt 1: 2/4 fields
validate: 2 error(s)
extract[rules] attempt 2: 2/4 fields
validate: 2 error(s)
extract[rules] attempt 3: 2/4 fields
validate: 2 error(s)
escalate: 2 unresolved error(s) after 2 retry(ies)
unresolved: missing required field 'currency'
unresolved: missing required field 'due_date'
Every agent step is recorded in the state's trace, so a failed document arrives at the human queue with its full history.
| Technology | Why it was chosen for this project |
|---|---|
| LangGraph | The retry cycle and conditional routing are first-class edges rather than control flow hidden inside a function; typed state is the inter-agent contract (ADR 0001) |
| Deterministic validator | Format rules have ground truth; an LLM critic grading an LLM extractor converges on confident garbage (ADR 0002) |
| Pluggable backend protocol | RuleBackend runs the whole graph offline for tests, CI, and demos; LLMBackend wraps any caller-supplied completion function and never touches API keys |
| pytest + scripted backends | The correction loop is tested deterministically with a backend scripted to fail once then recover, asserting the exact feedback it received |
pip install .
PYTHONPATH=src python examples/demo.pyWith a real LLM backend (any OpenAI-compatible client):
from openai import OpenAI
from agentic_extract.backends import LLMBackend
from agentic_extract.graph import run
client = OpenAI()
backend = LLMBackend(lambda prompt: client.chat.completions.create(
model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}]
).choices[0].message.content)
result = run(open("invoice.txt").read(), backend)
print(result["status"], result.get("extraction"))Adding a document type is one entry in SCHEMAS plus its validator rules; the graph itself does not change.
Full compiled graph with the rule backend on a shared x86_64 container, Python 3.12, 2,000 documents per case. Reproduce with python benchmark/run_benchmark.py.
| Path | Docs per second | ms per doc |
|---|---|---|
| Clean document (3 nodes, no retry) | ~590 | 1.69 |
| Escalating document (full retry loop, 7 node visits) | ~300 | 3.30 |
With a real LLM backend, model latency dominates by orders of magnitude; these numbers isolate what the graph itself costs.
- ADR 0001: why a state graph instead of three function calls, and what the dependency costs
- ADR 0002: why the quality gate is code, not a critic model
There is no checkpointing or human-in-the-loop interrupt in v0.1. LangGraph supports both, and escalation is where an interrupt would naturally live, but wiring persistence without a real review UI to resume from would be architecture theater. The escalate node carries everything (trace, errors, last attempt) that a resume flow will need.
Document classification is keyword rules, not a model. For two document types with distinctive markers, rules are free, instant, and testable. The classify node is one function swap away from an LLM classifier when the type count grows.
- Backend returns malformed JSON: the error propagates immediately rather than being coerced into an empty extraction that would waste a retry.
- Backend hallucinates fields outside the schema: flagged as validation errors by name, fed back for correction.
- Persistently failing documents: bounded by MAX_RETRIES, then escalated with full trace; the loop cannot run away.
- Unknown document types: routed to a terminal unsupported state before any extraction cost is incurred.
src/agentic_extract/ state, backends, validator, graph
tests/ 18 tests: routing, correction loop, validator, backends
examples/ offline demo with per-agent traces
benchmark/ orchestration overhead benchmark
docs/adr/ architecture decision records
- LangGraph checkpointing with an interrupt before escalate for human-in-the-loop resume
- LLM-based classify node behind the same routing contract for open-ended document types
- Structured trace export (JSON) for observability pipelines
MIT