knowledge-agent-evals

This project is a small, reproducible evaluation harness for comparing how different knowledge-access strategies answer questions over an external document collection. It focuses on failure modes that are easy to miss in demos: outdated sources, ambiguous questions, missing information, weak citations, and multi-hop retrieval.

Why this exists

External knowledge systems often look good on happy-path examples and then fail on version drift, incomplete documentation, or unsupported questions. This repository keeps the system intentionally small so the behavior is inspectable: the corpus is synthetic, the retrieval strategies are deterministic, and the metrics are simple enough to audit.

What this demonstrates

• Local ingestion and chunking for markdown knowledge bases.
• Three retrieval strategies: BM25-style keyword search, TF-IDF cosine search, and deterministic multi-step agentic search.
• Extractive answers with chunk citations and abstention.
• Evaluation metrics for retrieval, answer overlap, citations, and abstention.
• Generated JSONL and markdown reports suitable for error analysis.

Quickstart

python -m pip install -e ".[dev]"
python -m pytest
python -m knowledge_agent_evals evaluate --strategy all --k 5 --output-dir results

The default project runs locally and offline. It does not require API keys or network services.

Example commands

python -m knowledge_agent_evals evaluate --strategy agentic --k 5 --output-dir results
python -m knowledge_agent_evals ask "What changed in the latest vehicle software release?" --strategy agentic --k 5
python -m knowledge_agent_evals inspect-question q001 --strategy agentic --k 5

How the evaluation works

The corpus contains 10 synthetic markdown documents for a fictional mobility operations company. The eval set contains 32 hard questions across single-hop lookup, multi-hop reasoning, ambiguous phrasing, conflicting current/deprecated sources, temporal/version-sensitive facts, citation support, and missing information.

For each question, the runner retrieves the top k chunks, passes them to a deterministic extractive answerer, scores the result, and writes:

• results/summary.json
• results/summary.md
• results/per_question.jsonl

Strategies compared

Strategy	Description
keyword_bm25	Local BM25-style keyword retriever implemented in Python.
tfidf	Scikit-learn TF-IDF retriever with cosine similarity.
agentic	Deterministic query decomposition, subquery retrieval, reciprocal-rank fusion, and lightweight reranking for status/version hints.

The agentic strategy does not use an LLM. It demonstrates the architecture of multi-step search while staying reproducible.

Metrics

Retrieval metrics:

• recall_at_k
• mrr
• citation_doc_recall

Answer metrics:

• token_f1
• answer_contains_expected
• citation_precision
• citation_recall
• abstention_accuracy
• overall_score

overall_score is a documented weighted average of retrieval recall, reciprocal rank, answer overlap, citation quality, and abstention accuracy. It is meant for comparing runs on this benchmark, not as a general quality claim.

Example result table

Generated with:

python -m knowledge_agent_evals evaluate --strategy all --k 5 --output-dir results

Strategy	Overall	Recall@k	MRR	Token F1	Citation Precision	Citation Recall	Abstention Accuracy
keyword_bm25	0.855	0.984	0.922	0.513	0.969	0.906	1.000
tfidf	0.857	0.984	0.943	0.512	0.969	0.906	1.000
agentic	0.871	1.000	0.898	0.531	0.984	0.938	1.000

Failure modes this benchmark tries to expose

• Retrieval finds the right topic but the wrong version.
• Deprecated policy text outranks current guidance.
• Multi-hop questions need two documents, but only one is retrieved or cited.
• Missing information tempts the answerer to over-answer from adjacent evidence.
• Extractive answers cite a plausible chunk while omitting the required detail.

Limitations

The answerer is deliberately simple. It extracts sentences and uses heuristic abstention, so it can miss synthesis questions or cite too narrowly. Token F1 and substring checks are useful debugging signals, but they do not prove faithfulness. The corpus is small and synthetic, so absolute scores should not be generalized.

Roadmap

• Add metadata-aware reranking for dates, document status, and release versions.
• Add a citation verifier that checks whether each answer sentence is supported by its cited chunk.
• Add adversarial near-duplicate documents and harder missing-information cases.
• Separate retrieval quality, answer faithfulness, and citation faithfulness in the report.

Data policy

All data in this repository is synthetic. The corpus is fictional, hand-written, and public-style. The project does not scrape the internet and does not include confidential, proprietary, employer-specific, or customer-specific data.