Agentic Fact Verifier

A multi-agent system, orchestrated with LangGraph, that researches a factual claim (an entity + attribute, e.g. "Product X — weight in kg") across multiple independent web sources, cross-checks the results, and returns a confidence-scored, source-attributed answer — instead of trusting a single LLM's guess.

Reusable tool logic (search, extraction, trust scoring) is also exposed via a standalone MCP (Model Context Protocol) server, decoupling tool implementation from agent orchestration so the same tools can be used by this graph, a chat agent, or any other MCP-compatible client.

Why this architecture?

Decision	Rationale
ReAct-style search agent with tool-calling	Rather than a single search call, the agent decides iteratively how many searches to run and how to rephrase queries, gathering multiple independent sources before concluding.
Structured-output enforcement (Pydantic) at extraction time	Every source is parsed into a strict schema (value, unit, entity_mentioned, llm_confidence) instead of free text — this makes downstream verification logic deterministic and testable.
Fuzzy entity matching to discard mismatches	Search results often surface a similarly-named but wrong entity (a different product, company, or version). Token-based fuzzy matching (rapidfuzz) filters these out before they can pollute the trust score.
Statistical trust layer (cross-source deviation)	Confidence isn't just the LLM's self-reported score — it's computed independently from how tightly the surviving sources agree with each other (coefficient of variation). A single source, or wildly disagreeing sources, yields low trust even if the LLM sounded confident.
Conditional retry loop in the graph	If trust is too low, the graph loops back with a refined, more specific query (e.g. appending "official specification") instead of giving up after one pass.
MCP server for tool logic	Search, extraction, and trust-scoring tools are also exposed over MCP, so the same reusable logic isn't locked inside the LangGraph process — any MCP client (Claude Desktop, another agent) can call them directly.

Architecture

flowchart TD
    Start([Verification request: entity + attribute]) --> Plan[plan_query]
    Plan --> SearchExtract["search_and_extract
ReAct search agent + structured extraction"]
    SearchExtract --> Verify["verify
fuzzy match + trust score"]
    Verify -->|trust too low & retries left| Refine[refine_query]
    Refine --> SearchExtract
    Verify -->|confident, or out of retries| Finalize[finalize]
    Finalize --> End([VerifiedResult: value, trust_score, sources]))

    subgraph MCP["MCP server (standalone)"]
        T1[search_web]
        T2[compute_trust]
    end

Loading

Project structure

app/
  agents/
    search_agent.py       ReAct-style tool-calling search agent
    extraction_agent.py    LLM structured extraction into Pydantic schema
  search/
    web_search.py           Pluggable search backend (Tavily by default)
  verification/
    fuzzy_match.py          Entity-mismatch filtering
    trust_score.py          Cross-source statistical confidence scoring
  graph/
    workflow.py              LangGraph StateGraph wiring the pipeline above
  api/routes.py              FastAPI endpoint (POST /api/verify)
  schemas.py                  Pydantic request/response/intermediate models
  config.py                   Centralized settings
  main.py                      FastAPI app
mcp_server/server.py          Standalone MCP server exposing the same tools
scripts/run_cli.py             CLI entry point for local runs
tests/                          Unit tests for fuzzy matching, trust scoring, retry routing

Getting started

1. Configure environment

cp .env.example .env
# fill in OPENAI_API_KEY and TAVILY_API_KEY

2. Run the API

pip install -r requirements.txt
uvicorn app.main:app --reload

curl -X POST http://localhost:8000/api/verify \
  -H "Content-Type: application/json" \
  -d '{"entity": "Raspberry Pi 5", "attribute": "weight in grams"}'

3. Or run from the CLI

python -m scripts.run_cli "Raspberry Pi 5" "weight in grams"

4. Run the MCP server standalone

python -m mcp_server.server

Running tests

pytest

Tests cover the pure logic — fuzzy entity matching, trust-score computation, and retry-vs-finalize routing — without requiring API keys or network access.

Tech stack

Python · LangGraph · LangChain (tool-calling) · OpenAI · Tavily Search · MCP (Model Context Protocol) · rapidfuzz · FastAPI · Pydantic

License

MIT — see LICENSE.