🧠 CodeAtlas: Autonomous Codebase RAG & Agentic Refactoring Engine

Production-Grade Agentic RAG System featuring Graph RAG (Neo4j), Hybrid Retrieval (Qdrant), Dual LangGraph Agents (QA & Refactor Mode), Isolated WSL Pytest Sandbox Verification, and Multi-Key API Infrastructure.

Empirical Benchmarks • Architecture • Interactive Demo UI • Quick Start • Design Decisions

---

🏆 Empirical Benchmarks & Evaluation Results

CodeAtlas is evaluated over the official FastAPI codebase across QA Mode ($N=60$) and Refactor Mode ($N=20$). All benchmark evaluations execute using full real LLM agent graph invocations, vector search in Qdrant, graph impact analysis in Neo4j, and isolated Pytest execution in WSL. Raw trace logs are exported to /tmp/phase5_eval_raw_traces.jsonl.

Table 1: QA Mode Real-Agent Execution Benchmark ($N=60$)

Evaluating the end-to-end LangGraph QA agent graph on 60 real domain queries across 5 core FastAPI modules (encoders.py, routing.py, applications.py, datastructures.py, dependencies/utils.py). Line-range citations ([fastapi/encoders.py:10-30]) are mechanically verified against actual file boundaries in the workspace.

Metric	Score	Explicit Sample Size ($N$)	Definition & Verification Method
Precision@1 (P@1)	0.3667	$N=60$	Top-1 chunk matches primary target module
Precision@3 (P@3)	0.2167	$N=60$	Average precision across top 3 retrieved chunks
Precision@5 (P@5)	0.1400	$N=60$	Average precision across top 5 retrieved chunks
Hit@5 (File Recall@5)	0.7000	$N=60$	70.00% of queries (42/60) retrieved target module file in top 5
Mean Reciprocal Rank (MRR)	0.5097	$N=60$	Mean Reciprocal Rank ($1/\text{rank}$) of target module
QA Citation Coverage Rate	0.3167	$N=60$	19/60 real agent answers spontaneously included bracketed line-range citations
QA Strict Citation Validity	0.9737	$N=19$	18/19 cited answers (97.37%) pointed to strictly valid workspace file line ranges

Metric Terminology Note: Hit@5 (File Recall@5) evaluates whether the target module file was retrieved within the top 5 candidates ($\text{Recall}@K = \frac{|\text{Retrieved}_K \cap GT|}{|GT|}$ where $|GT|=1$).

---

Table 2: Refactor Mode Real-Agent Execution & Error Histogram ($N=20$)

Evaluating the end-to-end LangGraph Refactor agent across 20 automated code modification tasks over FastAPI with dynamic target symbol context resolution. Includes self-correction repair loops in an isolated WSL Pytest sandbox.

Refactor Metric	Metric Value	Sample Size ($N$)	Definition & Notes
Patch Apply Success Rate	0.2500	$N=20$	5/20 tasks (+5.0% over static snippet) applied Search/Replace blocks cleanly
Test Pass Rate (Non-regression)	0.1500	$N=20$	3/20 tasks passed full Pytest test suite post-refactoring
Mean Repair Iterations	2.55	$N=20$	Average self-correction repair loops spent per task
Line Citation Coverage	N/A	$N=20$	Refactor agent outputs structured Search/Replace patches, not free-form text
Target Header Match Rate	0.6500	$N=20$	13/20 patches specified correct target module in FILE: headers

Refactor Failure Mode Error Histogram (17 Failed Tasks)

pie title Phase 5 Refactor Failure Histogram (17 Failed Tasks)
    "Class A: SEARCH Block String Mismatch (52.9%)" : 9
    "Class B: Malformed / Non-Compliant Patch (35.3%)" : 6
    "Class D: Context Window / HTTP 413 Error (5.9%)" : 1
    "Class C: Post-Patch Pytest Logic Failure (5.9%)" : 1

Loading

• Class A: SEARCH Block Mismatch (52.9%, 9/17): LLM generated Search blocks with minor indentation or whitespace variations relative to source code.
• Class B: Malformed Patch (35.3%, 6/17): LLM generated python code blocks or natural text instead of standard Search/Replace tags.
• Class D: Context Window / HTTP 413 (5.9%, 1/17): Pytest error traceback payload exceeded Groq API prompt token limits.
• Class C: Post-Patch Pytest Failure (5.9%, 1/17): Patch applied cleanly, but introduced a NameError symbol resolution issue.

---

🏛️ System Architecture

CodeAtlas combines a Neo4j Graph Database (structural AST caller/callee relationships) and Qdrant Vector DB (dense semantic search) with dual LangGraph orchestrators.

flowchart TB
    subgraph input["📥 User Request & Router"]
        user[User Query] --> router{Agent Mode Router}
    end
    
    subgraph qa_graph["🔍 QA Agent Graph (LangGraph)"]
        router -- QA Mode --> hyde[HyDE Generator]
        hyde --> qdrant[(Qdrant Vector DB)]
        hyde --> neo4j[(Neo4j Graph DB)]
        qdrant & neo4j --> fuse[Reciprocal Rank Fusion]
        fuse --> answer[LLM Answer + Citation Verifier]
        answer --> cit_check{Mechanical Citation Check}
        cit_check -- Valid --> user_qa[Verified Answer with Citations]
        cit_check -- Invalid --> strip[Strip Unverifiable Citation Note] --> user_qa
    end

    subgraph refactor_graph["🛠️ Refactor Agent Graph (LangGraph)"]
        router -- Refactor Mode --> impact[Impact Analysis Node]
        impact --> neo4j
        impact --> gen_patch[Generate Search/Replace Patch]
        gen_patch --> sandbox[Sandbox Apply in WSL]
        sandbox --> pytest{Run Pytest Suite}
        pytest -- Pass --> approve{Human-in-the-Loop Interrupt}
        approve -- Approve --> commit[Git Commit & Finish]
        pytest -- Fail (Iter < 3) --> repair[Self-Correction Repair Node] --> gen_patch
    end

    subgraph infra["⚡ Infrastructure & Observability"]
        groq[GroqProvider: 4-Key API Rotation & 0s Backoff]
        opik[Opik Prompt Tracing & Cost Tracking]
        sandbox_env[WSL Pytest Isolated Sandbox]
    end

    qa_graph -.-> groq & opik
    refactor_graph -.-> groq & opik & sandbox_env

    style qa_graph fill:#e3f2fd,stroke:#1565c0
    style refactor_graph fill:#fff3e0,stroke:#ef6c00
    style infra fill:#f3e5f5,stroke:#7b1fa2

Loading

---

🖥️ Interactive Demo UI

CodeAtlas includes a single-page interactive web application (codeatlas_demo.html) featuring:

1. Benchmark Ablation Dashboard: Real-time visualization of QA recall curves, MRR metrics, and Refactor error histograms.
2. Interactive QA Agent Simulator: Live query input, HyDE generation visualization, vector/graph fusion progress, and answer rendering with verified bracketed line citations.
3. Refactor Sandbox Preview: Visualizing the 4-step Refactor graph (impact_analysis -> generate_patch -> sandbox_apply -> run_tests), SEARCH/REPLACE diff preview, and self-correction repair loop logs.
4. Graph RAG Explorer: Visual representation of codebase nodes, relationships, and Opik trace logs.

Open codeatlas_demo.html directly in any standard browser to launch the dashboard.

---

⚡ Quick Start

Prerequisites

• Python 3.11
• Docker & Docker Compose (for Qdrant & Neo4j)
• WSL 2 (Linux Environment for Sandbox execution)

1. Environment Setup

# Clone repository
git clone https://github.com/ductaip/codeatlas.git
cd codeatlas

# Install dependencies using Poetry
poetry install

# Configure environment keys in .env
cat <<EOT > .env
GROQ_API_KEY=gsk_your_key_1
GROQ_API_KEY2=gsk_your_key_2
GROQ_API_KEY3=gsk_your_key_3
GROQ_API_KEY4=gsk_your_key_4
GROQ_MODEL_ID=llama-3.3-70b-versatile
SANDBOX_REPO_PATH=$HOME/.cache/codeatlas-eval/fastapi
EOT

2. Prepare Refactor Sandbox (Required for Refactor Mode)

# Clone the FastAPI repo as a sandbox for closed-loop refactoring & pytest verification
git clone https://github.com/tiangolo/fastapi.git ~/.cache/codeatlas-eval/fastapi

Note: The Refactor agent applies patches and runs pytest inside this directory. Without it, Refactor mode will fail with FileNotFoundError. QA mode works without this step.

3. Start Infrastructure Services

# Start Qdrant Vector DB, Neo4j Graph DB, MongoDB, etc.
docker compose up -d

# (Optional) If you have an NVIDIA GPU and want local vLLM:
# docker compose --profile gpu up -d

4. Execute Phase 5 Benchmark Suite

# Run full real-agent evaluation suite (60 QA tasks + 20 Refactor tasks)
python scripts/run_phase5_eval.py

---

📚 Architectural Design Decisions

For full transparency regarding architectural trade-offs, engineering failures, and design decisions, view: 👉 docs/DESIGN_DECISIONS.md

Key decisions documented:

• Decision 1: Unified Diff Failure → Search & Replace Blocks (Fixing 0% patch apply rate caused by LLM line offset calculation errors).
• Decision 2: Per-Function Ablation Noise Filtering (impacted_symbols >= 3).
• Decision 3: Context Truncation Fix & Mechanical Line-Range Citation Verification.
• Decision 4: Refactor Citation Metric vs. Structured Patch Output.

---

📄 License

MIT License. See LICENSE for details.