Agentic financial claim verification.
FinVet verifies one financial claim at a time. Given "Microsoft's fiscal 2025 revenue was $282 billion," it returns a verdict of supported, refuted, or not enough information, a confidence score, and the number it compared the claim against. Each response also names the agent that answered, lists every tool call it made in order, and keeps an audit trail of the run.
Not financial advice. A research and demonstration system; outputs may be wrong and must be independently verified. It applies model-risk principles but certifies nothing and is not a compliance product. Not affiliated with any data provider named here.
The language model is responsible for retrieval only. It selects the tools, extracts the figure from the filing or the quote, and proposes a verdict, but the returned verdict is computed by comparing that figure to the claim under a fixed tolerance. Where the two disagree, the computed verdict is retained and both are reported. Where no source yields a figure, the claim is declined as not enough information.
FinVet is evaluated at three levels: the claim parser, the full pipeline from claim to verdict, and its RAG component in isolation. The RAG component is a hybrid search over filing text that the SEC agent invokes as a tool for claims about a filing's narrative sections, such as risk factors or legal proceedings, where no reported figure applies.
1) Claim parsing. The claim parser performs structured extraction, mapping each claim to a JSON object. A parse is correct when every field matches the gold parse. Scored over four runs on the held-out claim set described below: 309 of its 325 claims reach the parser, and the other 16 are refused by the input guardrails before parsing.
| DeepSeek-V4.1-Flash | Qwen3.8 | |
|---|---|---|
| Exact match | 97.5% | 89.3% |
2) Claim verification. The pipeline was evaluated on 325 held-out claims about 72 US-listed companies, each labelled with an expected verdict. Two models were run on identical code, four runs each, so that the reliability of a verdict could be measured alongside its accuracy. The claims were generated by script or with AI and reviewed by the author; their composition and labelling are described in the dataset card.
| Layer | DeepSeek-V4.1-Flash | Qwen3.8 |
|---|---|---|
| 1. Outcome — verdict accuracy | 97.5% | 96.9% |
| 2. Tool trajectory — tools called | 100% | 98.8% |
| 3. Grounding — numbers traced | 100% | 100% |
| 4. Calibration — decisive ECE | 0.049 | 0.050 |
| 5. Asymmetric risk, dangerous | 1 | 0 |
| 5. Asymmetric risk, declined | 2.2% | 3.1% |
| 6. Reliability — pass^4 | 95.4% | 92.0% |
| 7. Routing — expected path | 95.4% | 98.5% |
Rows 1 to 5 and 7 report each model's first run; the pass^4 row uses all four runs. Accuracy is computed over all 325 claims; excluding the 28 claims about live stock prices, whose answers depend on the day of the run, it is 97.3% and 97.0%. Across all eight runs, one claim was answered wrongly, in two of the four DeepSeek runs: a net loss was read as a gain. Every other miss was a decline or a hand-off to a human reviewer.
3) RAG component. The search was evaluated on 50 held-out queries and the model's explanations on 50 held-out claims.
| Test | Sample | Result |
|---|---|---|
| Finds the right passage in its top five | 30 queries | 24 of 30 (80%) |
| Ranks it near the top (mean reciprocal rank) | 30 queries | 0.68 |
| Returns nothing for off-topic or wrong-filing queries | 20 queries | 20 of 20 |
| Explanation stays inside the passages (RAGAS faithfulness) | 50 claims | 0.69 mean, 0.13 lowest |
Four of the six queries whose passage was absent from the top five returned a passage on the same subject from another section of the same filing. No explanation determines a verdict: a claim without a figure is declined.
The dataset, method and every run's results, with the files the verification and RAG numbers
are computed from, are in docs/eval/.
One agent runs per claim, chosen by claim type.
| Agent | Handles | Sources |
|---|---|---|
| SEC | GAAP financials, filing text | SEC EDGAR XBRL via MCP, hybrid RAG over filings |
| Market | Prices, valuation, market cap | Finnhub |
| News | Events, fines, settlements | Tavily search, and one-way delegation to SEC (cannot recurse) |
A news claim settled by delegation: the SEC agent found the same €500 million in Apple's filing.
Controls. Regex and PII guards screen every claim, with Llama Guard optional. Low confidence, unsafe output, or a conflict between press and filing pauses the graph at a checkpoint until a reviewer decides. Every tool call and verdict is persisted.
Full design, node by node: docs/ARCHITECTURE.md.
git clone https://github.com/danielberhane/finvet.git && cd finvet
cp .env.example .env # add DEEPSEEK_API_KEY, TAVILY_API_KEY, POSTGRES_PASSWORD
docker compose --profile sec up --build # --profile sec adds the SEC EDGAR MCP container
docker exec finvet-ollama ollama pull nomic-embed-text # embeddings, one-time
# UI → http://localhost:8501 API → http://localhost:8000Claims. One English sentence stating one fact about one US-listed company. Compound claims are not decomposed; ranges, relative changes, and figures derived from other figures, such as a fourth quarter computed from annual and nine-month totals, are declined rather than estimated.
Evidence. FinVet gives a verdict on a number only when it can fetch that number from a trusted source: a figure the company reported in its SEC filings, a current market quote, or a fine or settlement stated in a filing. Claims about other measures, such as margins, EBITDA, analyst price targets or economic indicators, are declined with a stated reason. Market data is the current delayed quote only; claims about past prices are declined.
Evaluation. Only two models were evaluated. The RAGAS faithfulness score is a second model's judgement and has not been verified by hand.
Deployment. A single-process research system with no authentication, rate limiting or tenant isolation (SECURITY.md); a pending review cannot be completed after a restart. If the SEC MCP server is unavailable, SEC claims decline; if Ollama is unavailable, the Llama Guard check is skipped and flagged, and filing-text search falls back to keyword matching.
Bug reports and focused fixes are welcome; see CONTRIBUTING.md.
To cite this software, use CITATION.cff.
Prior work. FinVet v1 (IEEE BigData 2025, with Duoduo Liao; code) decided verdicts by confidence-weighted vote over two retrieval pipelines. This release replaces that with one agent per claim and a deterministic comparator.
Apache-2.0. FinVet runs third-party components without distributing them, notably the AGPL-3.0 SEC EDGAR MCP server, which runs as its own container; see THIRD_PARTY_NOTICES.md. Users must honor each provider's terms, including the SEC Fair Access policy.