Autonomous Driving Knowledge Agent

This project is a small, offline knowledge agent for a synthetic autonomous-driving operations corpus. It focuses on failure modes that matter when language models use external knowledge: stale policies, conflicting sources, missing information, weak citations, and safety-sensitive questions.

The system is not a real autonomous-driving system, not operational guidance, and not connected to vehicles, maps, telemetry, or the internet. All data is synthetic.

Why This Exists

Most retrieval demos show a happy path: search a few documents, cite the top chunk, and produce a fluent answer. This repository instead treats external knowledge as an engineering problem. The corpus includes current and deprecated policy, effective dates, entity relationships, safety boundaries, and hard eval questions that reward abstention when the answer is not supported.

What This Demonstrates

• Offline, reproducible execution with no API keys.
• Typed Python models for documents, chunks, query plans, evidence, claims, answers, and eval results.
• Markdown ingestion with required YAML front matter validation.
• Stable heading-aware chunks with source metadata.
• BM25, TF-IDF, hybrid, and graph-aware retrieval.
• Ontology-backed document/entity expansion using NetworkX.
• Temporal and source-status reranking for current versus deprecated policy.
• Safety-aware extractive answering with claim-level evidence ledgers.
• Citation verification and conservative abstention.
• Evaluation reports for retrieval, citation quality, source handling, and safety caveats.

Quickstart

python -m pip install -e ".[dev]"
python -m ad_knowledge_agent stats
python -m ad_knowledge_agent ask "What are the current ODD limits for operating near school zones in heavy rain?" --strategy graph --k 6

Example Commands

python -m ad_knowledge_agent inspect "What changed in release 2025.08?" --strategy graph --k 6
python -m ad_knowledge_agent incident-brief --scenario "Vehicle reported repeated map mismatch near a temporary construction zone during light rain after release 2025.08." --output examples/sample_incident_brief.md
python -m ad_knowledge_agent release-risk --release-doc vehicle_software_release_2025_08 --output examples/sample_release_risk_review.md
python -m ad_knowledge_agent evaluate --strategy all --k 6 --output-dir results

Example Output

The ask command prints a concise answer, confidence, citations, safety flags, missing information, and warnings. Safety-sensitive answers include the synthetic-corpus caveat and cite chunk IDs such as odd_policy_current::chunk_002.

Architecture

flowchart LR
  A["User question"] --> B["Query planner"]
  B --> C["Hybrid retriever"]
  C --> D["Ontology graph expansion"]
  D --> E["Temporal/safety reranker"]
  E --> F["Extractive answerer"]
  F --> G["Citation verifier"]
  G --> H["Safety policy"]
  H --> I["Answer or abstention"]
  I --> J["Evaluation/reporting"]

Loading

Information Architecture

Every corpus document has YAML front matter with doc_id, status, version, effective_date, risk_level, tags, entities, owners, and related documents. Deprecated documents stay in the index so evals can test stale-source failures, but current policy is preferred for present-policy questions.

The ontology in data/ontology.yml defines domain entities such as Operational Design Domain, School Zone, Heavy Rain, Remote Assistance, Map Mismatch, Safety Case, and Release Gate. Graph-aware retrieval uses these relationships to add relevant policy and procedure documents that text similarity alone may miss.

Retrieval Strategies

• bm25: manual lexical baseline.
• tfidf: scikit-learn TF-IDF and cosine similarity baseline.
• hybrid: reciprocal-rank fusion of BM25 and TF-IDF.
• graph: hybrid seeds plus ontology expansion, source-status reranking, and safety-risk boosts.

Safety-Aware Answering

The default answerer is deterministic and extractive. It does not use an LLM. It selects evidence sentences from retrieved chunks, creates claim-level citations, verifies lexical support, and abstains when support is weak. For safety-sensitive questions, it states that the answer is limited to the synthetic knowledge base and is not real autonomous-vehicle operational guidance.

Evaluation Design

evals/hard_questions.jsonl contains questions across single-hop, multi-hop, temporal, conflicting-source, missing-information, safety-sensitive, incident-triage, release-risk, glossary, and hallucination-bait categories. The runner writes:

• results/summary.json
• results/summary.md
• results/per_question.jsonl
• results/failure_analysis.md
• results/report.html

Results

Run the evaluation locally:

python -m ad_knowledge_agent evaluate --strategy all --k 6 --output-dir results

The current checked-in report compares strategies on required-document recall, forbidden-source rate, current-source preference, abstention accuracy, citation precision, unsupported-claim rate, and overall score. In the latest local run, graph-aware retrieval had the strongest overall score and required-document recall:

strategy	required doc recall	abstention accuracy	citation precision	overall
bm25	0.97	1.00	0.74	0.84
tfidf	0.95	1.00	0.77	0.84
hybrid	0.97	1.00	0.73	0.84
graph	0.99	1.00	0.74	0.86

The graph-aware strategy helps most on temporal, multi-hop, and incident-triage categories. Simple lexical strategies remain competitive on direct lookup and glossary questions.

Failure Modes

Known limitations are intentionally visible:

• Citation verification is lexical, not semantic entailment.
• The answerer extracts evidence sentences instead of synthesizing deep explanations.
• Ontology expansion can add neighboring documents that dilute simple top-k retrieval.
• Temporal reasoning is based on metadata and query heuristics, not a full time model.
• The corpus is small and synthetic by design.

Limitations

This is not a production RAG system, not a vehicle safety system, and not real operational guidance. It does not validate autonomous-driving behavior. It demonstrates information architecture, retrieval, evidence handling, and evaluation mechanics for an offline synthetic domain.

Roadmap

• Add better date normalization and temporal conflict tests.
• Add a stricter claim verifier behind the same interface.
• Add richer scenario-task scoring.
• Add optional LLM answerer integration while preserving offline deterministic defaults.
• Expand adversarial evals for aliases, policy near-duplicates, and citation traps.

Data Policy

All corpus content is synthetic and hand-authored for this repository. The project uses no real employer data, no OEM data, no customer data, no partner data, and no scraped data.