Skip to content

WORLDCLASS_PLAN.md

Latest commit

 

History

History
675 lines (502 loc) · 25.9 KB

WORLDCLASS_PLAN.md

File metadata and controls

675 lines (502 loc) · 25.9 KB
Raw

Athena: World-Class Roadmap

SUPERSEDED — This document has been replaced by STRATEGY.md (2026-04-16). The project has narrowed from a full-stack RAG system to a focused runtime verification layer (athena-verify). This file is kept for historical reference only. All active work follows the new strategy. The code preserved on the legacy/full-stack branch corresponds to the plan described below.

Executive Summary

Athena 2.0 (Verifiable RAG with sentence-level hallucination detection) is currently very impressive — benchmarked, reproducible, shipped. To become world-class and unlock top-0.1% hiring, we need to:

  1. Prove verification works universally (not just legal)
  2. Go deep on verification science (hierarchical, uncertainty quantification, adversarial testing)
  3. Publish research (ArXiv paper + citations)
  4. Show real-world deployment (production case study with metrics)
  5. Integrate into the ecosystem (LangChain, LlamaIndex, HF, Vercel)

Timeline: 5–8 weeks of focused work. Hiring signal multiplication: 3–5×.

Phase 1: Multi-Domain Benchmark Suite

Goal: Show that verification works across all domains, not just legal.

Why this first: It's the fastest path to credibility. LegalBench-RAG alone is strong but domain-specific. Running the same evaluation on medical + research + finance + technical docs transforms the narrative from "Athena is good at legal RAG" to "Verification is the universal solution to hallucination."

Deliverables

A. Medical Q&A Benchmark

  • Source: PubMed-QA (open-domain QA over PubMed abstracts) or MedQA (USMLE exam questions)
  • Size: 500–1,000 questions
  • Documents: Medical literature, clinical guidelines, drug references
  • File: backend/eval/medqa/loader.py, backend/eval/medqa/runner.py
  • Metrics: Precision@1, Recall@64, mean trust score, hallucination rate
  • Expected result: 5–10× improvement vs. baseline RAG

B. Research Paper / ArXiv Benchmark

  • Source: SciQA (questions about academic papers) or hand-curated arXiv abstracts
  • Size: 500–1,000 questions
  • Documents: PDF abstracts, section summaries, method descriptions
  • File: backend/eval/sciqa/loader.py, backend/eval/sciqa/runner.py
  • Metrics: Same as medical
  • Expected result: Verification shows particularly strong gains on multi-paragraph citations

C. Financial Q&A Benchmark

  • Source: Earnings call transcripts, 10-K filings, analyst reports
  • Size: 300–500 questions (smaller corpus but high stakes)
  • Curated manually or via SEC EDGAR dataset
  • File: backend/eval/financial/loader.py, backend/eval/financial/runner.py
  • Metrics: Same + cost-per-query (financial queries are expensive)
  • Expected result: 6–8× improvement; show cost/accuracy tradeoff

D. Technical Docs Benchmark

  • Source: API documentation, software manuals, GitHub READMEs
  • Size: 500–1,000 questions
  • Examples: React docs, Django docs, PostgreSQL manual
  • File: backend/eval/techdocs/loader.py, backend/eval/techdocs/runner.py
  • Metrics: Same as others
  • Expected result: Verification excels on precise technical claims (method signatures, version numbers)

Public Leaderboard

File: docs/BENCHMARKS.md (NEW)

# Athena Verification Leaderboard

| Domain | Baseline RAG | Athena Verified | Improvement | Mean Trust |
|--------|---|---|---|---|
| Legal (LegalBench-RAG) | 2.65% | 18.2% | 6.9× | 0.82 |
| Medical (PubMed-QA) | 8.2% | 54.3% | 6.6× | 0.80 |
| Research (SciQA) | 5.1% | 38.7% | 7.6× | 0.84 |
| Financial (10-K) | 6.8% | 42.1% | 6.2× | 0.79 |
| Technical (API Docs) | 12.4% | 71.8% | 5.8× | 0.85 |

**Hallucination Rate Reduction:**
| Domain | Baseline | Athena | Reduction |
|--------|----------|--------|-----------|
| Legal | 23% | 5% | 78% |
| Medical | 28% | 6% | 79% |
| Research | 31% | 7% | 77% |
| Financial | 26% | 6% | 77% |
| Technical | 18% | 4% | 78% |

Average hallucination reduction: **78%** across all domains.

Implementation Roadmap

  1. Week 1: Build medical benchmark (loader + runner + evaluation harness)
  2. Week 2: Build research + financial + technical benchmarks (parallel)
  3. Week 3: Run full evaluation suite, create leaderboard table, commit results
  4. Week 4: Write blog post: "Verifiable RAG Works Everywhere: 78% Hallucination Reduction Across 5 Domains"

Success Metrics

  • ✅ 5–10× improvement on all benchmarks
  • ✅ Consistent 77–79% hallucination reduction across domains
  • ✅ Public leaderboard in docs
  • ✅ Reproducible evaluation code + CI/CD for each domain
  • ✅ Blog post with results published on dev.to, Medium, HN

Phase 2: Advanced Verification Techniques

Goal: Go beyond single-sentence NLI. Show scientific depth in the verification approach.

Why this matters: Hiring committees want to see you think deeply, not just string together existing models. Advanced verification shows you understand information theory, uncertainty quantification, and adversarial robustness.

A. Hierarchical Verification

Concept: Don't verify sentences in isolation. Decompose answers into claims → sub-claims → evidence chains, and verify each level.

Example:

Claim: "The indemnification cap is $1M."
  ├─ Sub-claim 1: "There is an indemnification clause."
  ├─ Sub-claim 2: "The cap is mentioned explicitly."
  └─ Sub-claim 3: "The cap amount is $1M."

Verify each step independently, then propagate confidence upward.
Result: 0.95 (clause exists) × 0.92 (cap mentioned) × 0.88 (amount correct) = 0.77 overall confidence.

Implementation:

  • File: backend/app/verification/hierarchical_verifier.py (NEW)
  • Use Claude to decompose answers into claim trees (JSON schema)
  • Verify each leaf claim independently
  • Propagate uncertainty upward using Bayesian inference
  • API response now includes claim_tree with per-node confidence

Expected impact: Identifies exactly where trust breaks down. Shows why a sentence is weak, not just that it is.

B. Uncertainty Quantification

Concept: Verification isn't binary. Return credible intervals on trust scores, not just point estimates.

Implementation:

  • File: backend/app/verification/uncertainty.py (NEW)
  • For each verification signal (NLI, lexical overlap, LLM votes), compute confidence distribution
  • Use Bayesian model averaging to get posterior on trust score
  • Return (median, lower_bound, upper_bound) triplet
  • Show calibration curves: "When the system says 0.8±0.1, is the answer actually correct 80% of the time?"

Example API response:

{
  "sentence": "The warranty period is 12 months.",
  "trust_score": 0.88,
  "trust_interval": [0.79, 0.95],  # 90% credible interval
  "confidence_in_estimate": 0.91,
  "nli_score": 0.92,
  "lexical_overlap": 0.84,
  "ensemble_votes": [1.0, 0.8, 0.95]  # 3 LLM votes
}

Expected impact: Shows the system knows what it doesn't know. Crucial for high-stakes domains.

C. Adversarial Robustness Testing

Concept: Red-team the verification. Craft adversarial sentences and document failure modes.

Test suite:

  • Paraphrases: "The notice period is 30 days" vs. "Termination requires a month's notice" (should both verify)
  • Negations: "The cap is NOT unlimited" (should catch as contradictory to "cap is $1M")
  • Partial truths: "The warranty covers defects" (true, but incomplete — also covers materials)
  • Out-of-scope claims: "The CEO approves all terminations" (not mentioned in contract)
  • Temporal shifts: "Was the notice period 30 days?" vs. "Is the notice period 30 days?" (different verification)

Implementation:

  • File: backend/tests/test_adversarial_verification.py (NEW)
  • File: backend/eval/adversarial_suite/ (NEW) — curated adversarial examples
  • Run on each of the 5 benchmark domains
  • Document success rate and failure modes
  • Create a report: docs/ADVERSARIAL_ROBUSTNESS.md

Example report section:

## Adversarial Robustness Results

### Paraphrase Robustness
- Same claim, different wording: 94% consistency
- Example: "30 days" vs. "one month" — both verified ✅

### Negation Detection
- False negatives (missed contradictions): 3%
- False positives (flagged valid negations): 1%
- Example: "NOT unlimited" correctly caught ✅

### Partial Truth Handling
- Partial claims (true but incomplete): 78% flagged as PARTIAL
- Full claims: 96% flagged as SUPPORTED
- Example: "warranty covers defects" (incomplete, doesn't mention materials) — flagged PARTIAL ✅

### Failure Modes
- OOB claims verified as SUPPORTED: 2% (aspirational facts)
- Temporal confusion: 5% (past vs. present tense)

Expected impact: Shows maturity. Honest documentation of limitations is more credible than perfection.

D. Cross-Document Citation

Concept: Claims can be supported by evidence across multiple documents. Weight citations by source credibility.

Implementation:

  • File: backend/app/verification/cross_document_verifier.py (NEW)
  • Track document source_credibility (manual labels or derived from citation count)
  • Allow spans from multiple documents to support a single claim
  • Show contradiction detection: "Document A says X, Document B says Y"
  • API response includes supporting_documents array with weights

Expected impact: Real documents contradict each other; showing this honestly is valuable.

Summary

Technique Files Timeline Impact
Hierarchical hierarchical_verifier.py, schemas 1 week Shows where trust breaks
Uncertainty uncertainty.py, calibration curves 1 week Confidence intervals, not points
Adversarial test suite + report 1.5 weeks Robustness + failure modes
Cross-doc cross_document_verifier.py 0.5 weeks Real-world validity

Total Phase 2 timeline: 4 weeks

Phase 3: Research Paper (ArXiv Publication)

Goal: Publish a peer-reviewed framing of the work. Credibility multiplier.

Why this matters: A paper on ArXiv says "this is research, not a quick hack." It gets cited, discussed, and shows you can communicate precisely.

Paper Structure

Title: "Verifiable Retrieval-Augmented Generation: Mechanistic Citation Verification for Hallucination Detection"

Sections:

  1. Abstract (250 words)

    • Hallucination problem, proposed solution, key results
    • "78% hallucination reduction across 5 domains"

  2. Introduction (2 pages)

    • RAG hallucination epidemic (stats: Stanford 2024, etc.)
    • Why existing solutions fall short
    • Our contributions (span-level verification, self-healing loop, multi-domain validation)
    • Roadmap

  3. Related Work (1.5 pages)

    • FACTUM, HalluGraph, Citation-Grounded Code Comprehension (cite recent ICLR papers)
    • FActScore, RAGAS, other hallucination metrics
    • Distinguish our approach (mechanistic + agentic + production-ready)

  4. Method (3–4 pages)

    • Span-Level Source Tracking — character offsets, raw_text preservation
    • Sentence-Level Citation Engine — NLI entailment, lexical overlap, LLM ensemble
    • Self-Healing Agentic Loop — conditional retry logic, weak claim re-retrieval
    • Verification Uncertainty — Bayesian confidence intervals
    • Architecture diagram (current one from README, enhanced)
    • Pseudocode for key algorithms

  5. Evaluation (3–4 pages)

    • Datasets — LegalBench-RAG (6,858 QA), PubMed-QA (500), SciQA (500), Financial (300), TechDocs (500)
    • Baselines — vanilla RAG, LangChain, RAGFlow, Athena w/o verification
    • Metrics — Precision@1, Recall@64, hallucination rate, mean trust score, latency, cost
    • Results table — leaderboard across all 5 domains
    • Ablation study — contribution of each component (span tracking, NLI, ensemble, self-healing loop)

  6. Analysis (2 pages)

    • Why verification works — signal fusion analysis
    • When it fails — adversarial robustness results, failure modes
    • Calibration — are trust scores actually calibrated? Show curves
    • Cost/accuracy tradeoff — latency overhead, cost per query vs. accuracy gains

  7. Limitations (0.5 pages)

    • Verification doesn't fix bad retrieval (garbage in → garbage out with high confidence)
    • LLM-as-judge is faster than human review but fallible
    • Domain-specific NLI models would improve results (out of scope)
    • Multi-hop reasoning still challenging

  8. Future Work (0.5 pages)

    • Graph-augmented verification (entities + relations)
    • Domain-specific NLI fine-tuning
    • Human-in-the-loop feedback loop
    • Multimodal verification (images + text)

  9. Conclusion (0.5 pages)

    • Verification is the frontier of production-grade RAG
    • Athena is open-source, reproducible, ready for deployment

  10. References (40–50 papers)

Figures & Tables

  • Figure 1: Architecture diagram (span tracking + verification pipeline + self-healing loop)
  • Figure 2: Example verified answer with trust scores and citation highlights
  • Figure 3: Calibration curves (predicted trust vs. empirical accuracy)
  • Figure 4: Ablation study results (bar chart)
  • Table 1: Leaderboard across 5 domains
  • Table 2: Adversarial robustness results
  • Table 3: Hyperparameters and configuration

Submission & Timeline

  • Write: 2 weeks (parallel with Phase 2)
  • Review & revise: 1 week (internal, then one external reviewer)
  • Submit to ArXiv: Week 5
  • Submit to venue: EMNLP 2026 or ICLR 2026 (check deadlines)

Expected Impact

  • ArXiv listing (citable, indexed by Google Scholar)
  • Citation in README + blog posts
  • Discussion on research Slack/Reddit
  • Open calls for collaboration

Phase 4: Production Deployment Case Study

Goal: Real-world validation. Deploy Athena somewhere and document the results.

Why this matters: Benchmarks are important, but a production case study is proof. "We reduced hallucinations by X% in production at Company Y" is worth 100 benchmark points.

Option A: Law Firm / Legal Tech (Recommended)

Partner: Contract review startup, legal tech vendor, or in-house legal team

Setup:

  • Integrate Athena verification into their contract Q&A workflow
  • Run baseline (their current system) vs. Athena verified for 2–4 weeks
  • Measure:
    • Hallucination rate (manual review of 100 outputs)
    • User satisfaction (thumbs up/down)
    • Review time reduction (if applicable)
    • Cost per query
    • Precision and recall on their internal QA set

Expected results:

  • 70–80% reduction in hallucinations
  • 20–40% faster review cycles
  • Cost increase: 15–30% (verification overhead)

Deliverable: Case study document + blog post

  • File: docs/CASE_STUDIES.md
  • Content: Partner name, challenge, implementation, results, lessons learned
  • Length: ~2,000 words
  • Commit to GitHub, cite in README

Option B: Medical / Research

Partner: Health system, research institute, medical publisher

Setup:

  • Clinical Q&A system or research paper summarization
  • Compare clinician satisfaction: baseline system vs. Athena verified
  • Measure accuracy on standard medical benchmarks (MedQA, USMLE-style questions)

Expected results:

  • Similar hallucination reduction (77–79%)
  • High clinician trust (92%+ rate verified answers as acceptable without checking)

Option C: Financial Services

Partner: Financial advisory firm, asset manager, trading desk

Setup:

  • Earnings call Q&A or 10-K document search
  • Measure accuracy on factual questions (numbers, dates, key announcements)
  • Track which queries benefit most from verification

Expected results:

  • 6–8× improvement on financial precision questions
  • Lower adoption for prediction questions (where verification can't help)

Timeline

  • Identify partner: 1–2 weeks
  • Implement integration: 1–2 weeks
  • Run evaluation: 2–4 weeks
  • Write case study: 1 week
  • Publish: Week 8–10

Success Metrics

  • ✅ Deploy to production environment
  • ✅ Document baseline metrics (no Athena)
  • ✅ Show 5–10× improvement on key metric
  • ✅ Publish case study with partner name (or anonymized if NDAs require)
  • ✅ Get testimonial quote from partner

Phase 5: Ecosystem Integration

Goal: Make Athena the default verification layer for the RAG ecosystem.

Why this matters: Distribution. If 100K developers use Athena through LangChain/LlamaIndex, that's 100K people who know your name. Ecosystem integration = reach.

A. LangChain Integration

What: Drop-in VerifiableRAG component for LangChain chains

Implementation:

  • File: integrations/langchain/verifiable_rag.py (NEW)
  • Works with any LangChain retriever + LLM
  • Wraps the output in Athena verification
  • Simple API:
from langchain_athena import VerifiableRAGChain

chain = VerifiableRAGChain.from_retriever(
    retriever=my_retriever,
    llm=ChatAnthropic(),
    verify=True,
    max_iterations=3,
)

result = chain.invoke({"question": "..."})
# result.answer, result.trust_score, result.verified_sentences

Timeline: 1 week Effort: Medium (mostly integration glue)

B. LlamaIndex Integration

What: VerificationQueryEngine for LlamaIndex

Implementation:

  • File: integrations/llamaindex/verification_engine.py (NEW)
  • Works with any LlamaIndex retriever + LLM
  • Drop-in replacement for QueryEngine
  • API:
from llama_index.athena import VerificationQueryEngine

engine = VerificationQueryEngine.from_documents(
    documents=docs,
    verify=True,
    max_iterations=3,
)

response = engine.query("What is...?")
# response.response, response.metadata["trust_score"]

Timeline: 1 week Effort: Medium

C. Hugging Face Hub

What: Package verification pipeline as standalone component

Implementation:

  • File: huggingface_hub/model_card.md (NEW)
  • Create model cards for:
    • athena-nli-verifier — span-level entailment checker
    • athena-lexical-scorer — overlap scoring
    • athena-trust-scorer — confidence aggregation
  • Each component can be used independently
  • Example:
from huggingface_hub import from_pretrained
verifier = from_pretrained("athena-nli-verifier")
score = verifier("The notice period is 30 days.", "30 days. The terminating party...")

Timeline: 1 week Effort: Low (mostly packaging)

D. Vercel AI SDK

What: Pre-built verified RAG patterns for Next.js

Implementation:

  • File: integrations/vercel_ai/verified_rag.ts (NEW)
  • Vercel AI SDK plugin
  • Use case: Next.js docs site with verified search
import { VerifiedRAG } from '@vercel/ai/athena';

const rag = new VerifiedRAG({
  documents: docCollection,
  verify: true,
});

const result = await rag.query("How do I use...?");
// result.answer, result.trustScore, result.verifiedSentences

Timeline: 1 week Effort: Medium (API design is critical)

Timeline & Effort

Integration Timeline Effort Priority
LangChain 1 week Medium High
LlamaIndex 1 week Medium High
HuggingFace 1 week Low Medium
Vercel AI 1 week Medium Medium

Total Phase 5: 4 weeks (can run in parallel, 2 weeks with full parallelization)

Sequencing & Timeline

Critical Path

Phase 1 (Multi-domain) ─────────────────────────────┐
                                                    ├─→ Phase 3 (Paper) ─┐
Phase 2 (Advanced verification) ─────────────────┐ │                    ├─→ Publish + Launch
                                                 │ │                    │
Phase 4 (Case study) ────────────────────────────┘─┘──────────┐        │
                                                              │        │
Phase 5 (Ecosystem) ────────────────────────────────────────┴────────┴─→ Public release

Week-by-Week Breakdown

Week Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
1 Med benchmark Hierarchical Partner match
2 SciQA + Financial Uncertainty Paper intro + method Integration start
3 TechDocs eval Adversarial suite Paper eval Eval running LangChain
4 Leaderboard + blog Cross-doc + wrap Paper write Analysis LlamaIndex
5 Revise + ArXiv submit Case study draft HuggingFace
6 Publish Vercel
7 Polish + docs
8 Launch

Recommended Sequencing

If you have 8 weeks and want maximum impact:

  1. Weeks 1–2: Phase 1 (Multi-domain benchmarks)

    • Run all 5 benchmarks in parallel (write loaders in week 1, run evaluations in week 2)
    • This is your proof point and sells everything else
    • Commit results to docs/BENCHMARKS.md

  2. Weeks 2–4: Phase 2 (Advanced verification) + Phase 3 (Paper)

    • While Phase 1 runs, start Phase 2 in parallel
    • Write paper (weeks 2–4) using Phase 1 results
    • Submit to ArXiv by end of week 4

  3. Weeks 3–5: Phase 4 (Case study)

    • Partner identification (week 3)
    • Implementation + evaluation (weeks 4–5)
    • Write-up (week 5)

  4. Weeks 6–8: Phase 5 (Ecosystem) + Launch

    • Ecosystem integrations in parallel
    • Public launch with all assets (paper, case study, benchmarks, blog posts)

If you have limited time (3–4 weeks): Do Phases 1 + 3 (benchmarks + paper). This is the minimum for "world-class."

Success Metrics & Milestones

Phase 1: Multi-Domain Benchmarks

  • ✅ 5+ domains evaluated
  • ✅ Public leaderboard showing 5–10× improvement
  • ✅ Blog post published
  • ✅ CI/CD integrated (benchmark runs on every PR)

Phase 2: Advanced Verification

  • ✅ Hierarchical verification implemented + tested
  • ✅ Uncertainty quantification with calibration curves
  • ✅ Adversarial robustness test suite (100+ adversarial examples)
  • ✅ Documentation of failure modes

Phase 3: Research Paper

  • ✅ ArXiv submission with DOI
  • ✅ Peer review comments (if venue) or ArXiv visibility
  • ✅ Paper cited in README

Phase 4: Case Study

  • ✅ Production deployment (real data, real queries)
  • ✅ Baseline metrics + Athena metrics
  • ✅ Published case study document
  • ✅ Partner testimonial

Phase 5: Ecosystem

  • ✅ 2+ framework integrations published
  • ✅ >100 downloads/installs in first month
  • ✅ Community contributions (PRs for additional frameworks)

Launch Checklist

  • All 5 benchmarks public and reproducible
  • ArXiv paper submitted
  • Case study published
  • 2+ ecosystem integrations available
  • Blog posts written and published
  • Press list (10+ tech journalists/influencers)
  • Show HN post ready
  • Demo video (<2 min)
  • All code committed + tests passing
  • README updated with leaderboard + case study link

Why This Becomes World-Class

Signal Current Athena World-Class Athena
Proof of concept LegalBench-RAG only 5 domains, 78% avg improvement
Technical depth Single-sentence NLI Hierarchical verification, uncertainty quantification, adversarial testing
Validation Benchmarks only Benchmarks + production case study
Credibility GitHub project ArXiv paper + citations
Ecosystem reach Standalone LangChain + LlamaIndex + HF + Vercel
Hiring signal "Impressive project" "This person understands RAG, verification, production systems, research, AND ecosystem"

Risks & Mitigations

Risk Mitigation
Multi-domain eval takes longer than expected Pre-compute embeddings, reuse infrastructure from Phase 1
Paper gets rejected from venue ArXiv is perfectly fine; focus on visibility there
Can't find production partner Do a small internal case study (e.g., test on 1,000 public questions + manual review of 100)
Ecosystem integrations reveal design issues Start with LangChain (simpler integration) before committing to others
Scope creep (want to add X, Y, Z features) Commit to the 5 phases; ship what's listed, take feature requests for v2.1

Go / No-Go Decision Points

Week 2 (end of Phase 1):

  • If benchmarks show <5× improvement on any domain → recalibrate thresholds or debug
  • If 2+ domains show <75% hallucination reduction → investigate why (domain-specific issues?)
  • Go: ≥5× improvement on ≥4 domains, ≥77% hallucination reduction → proceed to Phase 2 + 3 + 4

Week 5 (end of Phase 2 + 3):

  • Paper submitted to ArXiv
  • Advanced verification code passes test suite
  • Go: Both complete → proceed to Phase 5 launch

Week 8:

  • 2+ ecosystem integrations published
  • Case study documented
  • All benchmarks public + reproducible
  • Launch: Show HN + blog posts + all assets

What You'll Tell Interviewers

"I built Athena, an open-source RAG system that detects and corrects hallucinations at the sentence level using mechanistic verification. We evaluated it on five different domains (legal, medical, research, financial, technical) and achieved a consistent 78% hallucination reduction with 5–10× precision improvements. The system uses NLI entailment checking, lexical overlap scoring, and ensemble LLM judges to produce calibrated trust scores for every sentence, and it automatically retries with refined queries when verification fails. We published a paper on ArXiv, deployed it in production at [partner name], and integrated it into LangChain and LlamaIndex. The code is open-source (MIT license), fully async with pgvector backend, K8s-ready, and every PR is automatically benchmarked."

That sentence is gold.

Next Steps

  1. Review this plan with your target companies / mentors
  2. Start with Phase 1 (2 weeks of work, highest ROI)
  3. Commit to shipping by [DATE]
  4. Weekly progress updates to a WORLDCLASS_PROGRESS.md file
  5. Launch with all assets (benchmarks + paper + case study + ecosystem) at once

You're 2–3 months away from being genuinely unhireable (in a good way).