Agentic Search Benchmark

Agentic Search Benchmark is an offline benchmark for comparing search agents on external-knowledge tasks that require planning, source selection, table lookup, citation support, abstention, and budget-aware tool use. It focuses on failure modes that simple RAG demos often hide: stale sources, conflicting documents, false premises, missing information, weak citations, and tool-use inefficiency.

Why This Exists

Search agents are often evaluated by looking at final answers. That hides the part that matters most for external-knowledge reliability: which sources were selected, which tools were used, whether current sources beat deprecated sources, and whether the system knew when not to answer.

This repository asks a narrower question:

When does agentic search actually improve external-knowledge reliability, and when does it fail?

The benchmark is deliberately offline, deterministic by default, and small enough to inspect in 10 to 15 minutes. It is not a chatbot, a toy RAG demo, a scraping project, or a leaderboard with broad claims.

What This Benchmark Tests

• Single-shot retrieval versus multi-step retrieval.
• Query decomposition, iterative search, and graph-aware search.
• Structured-data lookup from CSV tables.
• Current versus deprecated source selection.
• Citation-grounded extractive answering.
• Abstention under missing evidence or false premises.
• Budget-aware tool use.
• Failure modes across task categories.

What "Agentic" Means Here

An agentic search system can choose tools, perform intermediate search steps, inspect evidence, compare source versions, verify citations, abstain, and emit a structured trace. The benchmark does not assume an LLM is required. The included agents are deterministic baselines designed to make planning and failure modes inspectable.

flowchart TD
  Q[User question or benchmark task] --> P[Agent planner]
  P --> T{Tool selection}
  T --> R[Text retrieval]
  T --> G[Graph expansion]
  T --> L[Table lookup]
  T --> O[Open / inspect source]
  R --> E[Evidence set]
  G --> E
  L --> E
  O --> E
  E --> A[Extractive answerer]
  A --> C[Citation verifier]
  C --> S[Abstention / source freshness policy]
  S --> F[Final answer]
  F --> M[Benchmark metrics]
  P --> Tr[Structured trace]
  T --> Tr
  C --> Tr
  Tr --> M
  M --> Rep[Reports]

Loading

Quickstart

python -m pip install -e ".[dev]"
python -m pytest
python -m asbench list-corpora
python -m asbench run --suite smoke --agents all --budget 8 --output-dir results/smoke

No API keys, scraping, Docker, vector database, model download, LangChain, or LlamaIndex setup is required.

Example Commands

python -m asbench list-agents
python -m asbench stats --corpus all
python -m asbench ask "What authentication method should a new Helio integration use?" --corpus helio_api_docs --agent verify_then_answer --budget 10
python -m asbench inspect-task core:q012 --agent graph_navigator --show-trace
python -m asbench run --suite core --agents all --budget 12 --output-dir results
python -m asbench sweep --suite budgeted --agents all --budgets 2,4,8,12,16 --output-dir results
python -m asbench ablate --agent graph_navigator --suite core --output-dir results
python -m asbench report --results-dir results

Example Output

graph_navigator: overall=0.862 citation_f1=0.970 budget=7.00
static_topk: overall=0.860 citation_f1=0.930 budget=2.00
iterative: overall=0.853 citation_f1=0.950 budget=6.30
decompose: overall=0.838 citation_f1=0.923 budget=6.60
verify_then_answer: overall=0.835 citation_f1=0.970 budget=7.00

The weighted score is intentionally secondary. Read it with citation F1, abstention accuracy, forbidden-source rate, current-source preference, and budget used.

Architecture

The package is named asbench and exposes the CLI through python -m asbench.

• corpus: markdown, CSV, ontology loading, metadata validation, chunking, and indexing.
• retrieval: manual BM25, scikit-learn TF-IDF, hybrid reciprocal-rank fusion, table lookup, graph expansion, and version-aware reranking.
• agents: deterministic search strategies with trace-emitting tool calls.
• answering: extractive answer construction, citation verification, claim parsing, and abstention.
• benchmark: task loading, metrics, failure taxonomy, runner, sweeps, ablations, and reporting.
• traces: step-level trace schema, recorder, and markdown renderer.

Corpora

All corpora are synthetic and fictional.

• aster_mobility_ops: autonomy operations policies, release notes, incident response, map updates, tables, and ontology.
• helio_api_docs: fictional API documentation with current/deprecated authentication, endpoints, rate limits, error codes, SDK tables, and ontology.
• northstar_enterprise_policy: fictional enterprise policy with retention, access control, incident escalation, vendor review, privacy exceptions, audit logging, tables, and ontology.
• orion_research_archive: fictional research notes on retrieval, evaluation, agentic search, citation quality, negative results, experiment tables, and ontology.

Task Taxonomy

The task suites cover single-hop lookup, multi-hop composition, temporal current-versus-deprecated selection, conflicting sources, missing information, false premises, table lookup, document-plus-table evidence, entity disambiguation, cross-document references, policy exceptions, incident triage, release risk, research ablation, citation traps, budget sensitivity, and paraphrase robustness.

Suites:

• smoke.jsonl: 10 tiny tasks for fast sanity checks.
• core.jsonl: 72 balanced tasks.
• adversarial.jsonl: 30 failure-seeking tasks.
• budgeted.jsonl: 24 budget-sensitive tasks.
• paraphrase_groups.jsonl: 12 paraphrase groups.

Agents Compared

• static_topk: one hybrid search call and extractive answer. Strong on simple lookup, cheap, brittle on multi-hop tasks.
• decompose: deterministic query decomposition plus reciprocal-rank merge. Better on some multi-hop tasks, spends more budget.
• iterative: searches, optionally queries tables, inspects chunks, compares versions, verifies citations. More evidence-aware, less cheap.
• graph_navigator: uses ontology aliases and graph expansion to follow related sources and version links. Helpful for entity-linked and temporal tasks, can add noise.
• verify_then_answer: runs graph navigation, verifies claims, and may abstain. Reduces unsupported claims, but can over-abstain.

Metrics

The benchmark separates evidence retrieval, answer quality, citation support, abstention/source freshness, and agentic efficiency.

Default weighted score:

• 30 percent evidence retrieval.
• 25 percent answer correctness.
• 20 percent citation support.
• 15 percent abstention and source freshness.
• 10 percent budget efficiency.

Component metrics matter more than the overall score.

Trace Format

Every agent emits an AgentTrace with ordered tool calls. Allowed actions are:

search_text, search_hybrid, query_table, open_doc, inspect_chunk, expand_entity, follow_reference, compare_versions, verify_citations, abstain, and final_answer.

Each step includes action input, output references, an inspectable short reason, budget cost, elapsed milliseconds, and warnings.

Results

Benchmark runs write:

• summary.json
• summary.md
• leaderboard.csv
• per_task_results.jsonl
• traces.jsonl
• failure_analysis.md
• budget_sweep.csv
• ablation_report.md
• report.html

The HTML report is static, local, and generated with Jinja2. It uses no external assets.

Interpreting Failures

Failures are classified into categories such as retrieval_miss, stale_source_selected, forbidden_source_used, table_row_missed, graph_expansion_noise, query_decomposition_bad, over_abstention, under_abstention, unsupported_claim, citation_mismatch, budget_exhausted, and answer_extraction_error.

Healthy results usually show tradeoffs: a cheap baseline may win easy lookup, graph navigation may improve current-source handling, and verification may reduce unsupported claims while refusing more often.

Related Repositories

• knowledge-agent-evals: a simpler local evaluation harness for retrieval, citations, abstention, and external-knowledge failure modes.
• autonomous-driving-knowledge-agent: a domain-specific knowledge agent with graph-aware retrieval and safety-sensitive answering over synthetic autonomous-driving operations data.
• agentic-search-benchmark: this repository generalizes the evaluation question to heterogeneous corpora, tools, planning styles, citation requirements, source freshness, structured data, and budget constraints.

Limitations

• Synthetic tasks are not production workloads.
• Included agents are deterministic baselines, not full LLM agents.
• Answer scoring uses heuristic token overlap.
• Citation verification is lexical.
• Corpora are intentionally small.
• There is no learned reranker, distributed retrieval, or external search by default.

Roadmap

• Add larger synthetic corpora with richer contradictory evidence.
• Add optional LLM-backed agents behind explicit configuration.
• Add more paraphrase and perturbation suites.
• Add stricter trace validators.
• Add richer report visualizations without external assets.

Data Policy

All data is synthetic. The repository contains no proprietary data, no scraped content, no real customer information, no real operational guidance, and no confidential documents. Gold labels are included only for benchmark evaluation and should never be exposed to comparable agents.