GraphRAG vs Vector RAG — a multi-agent PoC

A small proof of concept that shows where plain vector RAG falls short and a graph database (Neo4j) does better. A LangGraph agent looks at the query, picks the vector store, the graph, or both, and returns a Pydantic-validated answer.

The domain is software supply-chain risk: services, the libraries they pull in, transitive dependencies between libraries, and the CVEs attached to specific library versions.

Table of Contents

1. Why this exists
2. Architecture
3. Tech stack
4. Observability
5. Project structure
6. Prerequisites
7. Setup
8. Running it
9. The query that breaks vector search
10. Neo4j schema
11. How routing works
12. What's not built yet

Why this exists

Vector RAG retrieves documents by semantic similarity. That works for "what is service X" but it can't follow a chain of relationships. Ask "which production services are exposed to a critical CVE through their transitive dependencies, and what's the path?" and similarity search returns documents that happen to mention services and CVEs near each other — it never walks the dependency graph, so it misses indirect exposure and can't show a path.

A graph query answers that exactly. This PoC runs the same query both ways so you can see the difference side by side.

Architecture

The agent is a bounded ReAct loop on a LangGraph state machine — Reason → Act → Observe, repeating until the LLM has enough evidence:

  user query
      │
      ▼
  ┌─────────┐   reason: LLM reads query + scratchpad, emits a
  │ reason  │◄─┐ ReActStep — Thought + Action (vector|graph|
  └────┬────┘  │ hybrid|finish), Pydantic-validated
       │ act?  │
       ▼       │
  ┌─────────┐  │ act: runs the chosen tool(s) (concurrently
  │   act   │──┘ for "hybrid"), appends an Observation to the
  └────┬────┘    scratchpad, then loops back to reason
       │ finish (or iteration budget hit)
       ▼
  ┌──────────┐  synthesize: one LLM call that consolidates
  │synthesize│  the accumulated tool output into a FinalAnswer
  └────┬─────┘
       ▼
  FinalAnswer (Pydantic-validated)

The loop is bounded four ways in reason_node (graph.py) so it can never run away or wander, even with a weak model: a deterministic finish once the first tool returns non-empty evidence (_has_evidence, no LLM call); a no-progress backstop for the all-empty case (_loop_converged); a hard MAX_ITERATIONS cap; and "no finish before any tool ran". The ReAct recovery still works: if a tool returns nothing, there's no evidence, so the next reason step autonomously switches tools instead of repeating the dead end.

There is no separate "validate" node. Schema validation happens inside every LLM call: the model is asked for JSON matching the target schema, the output is parsed and validated with Pydantic, and on a parse/validation failure it is re-prompted once with the error attached so it can self-correct. Routing is not a single upfront decision: the reason node re-evaluates on every loop iteration using the Observations accumulated in the scratchpad.

Tech stack

Layer	Choice
Orchestration	LangGraph (state graph)
Graph DB	Neo4j 5.20 community (run via Docker Compose)
Vector DB	ChromaDB, local persistent (no container)
LLM	Ollama, any local chat model
Embeddings	sentence-transformers/all-MiniLM-L6-v2
Concurrency	asyncio (vector + graph tools run in parallel)
Validation	Pydantic v2 with a one-shot schema-repair retry
CLI	Typer + Rich
Logging	stdlib logging, centralized in src/logging_config.py

Observability

Logging is configured in one place (src/logging_config.py) so a failure is understandable from the logs alone:

• Correlation — every line carries a short [run_id]; all logs from one query share it, even though the vector and graph tools run concurrently.
• Signal over noise — httpx, huggingface_hub, sentence_transformers, Neo4j notifications etc. are pinned to WARNING so a real error is never buried. Set LOG_LEVEL=DEBUG to unmute them.
• Failures, not bare tracebacks — the agent entrypoint and both tools log logger.exception(...) with the query / route / run-id before re-raising; Neo4j retries log a WARNING per attempt.
• The ReAct trace — run_agent logs agent start / agent done route=… elapsed_ms=…, and at LOG_LEVEL=DEBUG dumps the full Thought/Action/Observation scratchpad — the single most useful artifact for debugging the loop.
• Where logs go — to the console (stderr) and a rotating file at logs/graphrag.log (5 MB × 3 backups, gitignored), so a failure can be inspected after the fact. Path/size are configurable via LOG_DIR / LOG_FILE / LOG_MAX_BYTES / LOG_BACKUP_COUNT.

Project structure

DS/
├── README.md
├── pyproject.toml
├── requirements.txt
├── .env.example
├── docker-compose.yml          # Neo4j only; Chroma is local on disk
├── data/
│   ├── seed_graph.cypher       # Neo4j seed script
│   └── documents.jsonl         # docs for the vector store
├── src/
│   ├── config.py               # Pydantic settings, read from .env
│   ├── schemas.py              # Pydantic v2 output schemas
│   ├── main.py                 # Typer CLI (ask / headtohead / ingest)
│   ├── ingest/
│   │   ├── load_graph.py       # loads seed_graph.cypher into Neo4j
│   │   └── load_vectors.py     # embeds documents.jsonl into Chroma
│   ├── tools/
│   │   ├── graph_tool.py       # async Neo4j tool (CVE blast radius)
│   │   └── vector_tool.py      # async Chroma similarity tool
│   └── agent/
│       ├── llm.py              # Ollama wrapper + structured_call
│       ├── router.py           # route-decision node
│       ├── state.py            # LangGraph state
│       └── graph.py            # compiled graph + run_agent()
├── tests/
│   ├── conftest.py
│   ├── test_schemas.py
│   ├── test_router.py
│   ├── test_llm_parsing.py
│   └── test_breaking_point.py
└── notebooks/
    └── head_to_head_demo.ipynb

Prerequisites

• Python 3.11+
• Docker and Docker Compose (for Neo4j)
• Ollama running locally with at least one chat model pulled
• ~2 GB free disk, plus whatever your Ollama model needs

Setup

1. Install dependencies

git clone <your-repo-url> graphrag-vs-vector-rag
cd graphrag-vs-vector-rag

# uv
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"

# or pip
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Configure

cp .env.example .env

The defaults work for a local Ollama install:

LLM_MODEL=llama3.2:3b
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_FORMAT=json

Pull a model first:

ollama pull llama3.2:3b
# qwen2.5:7b-instruct, llama3.1:8b, mistral:7b, phi3:mini, gemma2:2b also work

llama3.2:3b is the default because it's small and fast. If you have the RAM, a 7B model (e.g. qwen2.5:7b-instruct) is more reliable for structured output — 3B models hit the schema-repair retry in structured_call more often, which is slower. Set LLM_MODEL in .env to switch.

3. Start Neo4j

docker compose up -d neo4j
# Neo4j Browser: http://localhost:7474  (neo4j / graphrag-demo)

4. Seed the data

python -m src.main ingest

This loads data/seed_graph.cypher into Neo4j and embeds data/documents.jsonl into the local Chroma store.

Running it

The CLI is a Typer app with three subcommands (or use graphrag after pip install -e):

# single query through the full agent
python -m src.main ask "Which production services are transitively impacted by CVE-2024-31337?"

# same query, vector-only vs graph-only, side by side with timings
python -m src.main headtohead "Which production services are transitively impacted by CVE-2024-31337?"

# re-run both ingest steps
python -m src.main ingest

Notebook walkthrough:

jupyter lab notebooks/head_to_head_demo.ipynb

Tests:

pytest -v

The query that breaks vector search

"Which production services are transitively impacted by CVE-2024-31337, and what is the dependency path?"

CVE-2024-31337 is a CRITICAL RCE in log-utils 1.4.x. Nothing depends on log-utils directly — the exposure is four hops deep:

payments-api ─DEPENDS_ON→ auth-sdk@2.3.0 ─USES→ jwt-lib@0.9.1 ─USES→ log-utils@1.4.0 ─HAS_VULNERABILITY→ CVE-2024-31337

• Vector RAG returns documents that mention services and CVEs, but it never connects payments-api to log-utils because no single document states that link. No path, and indirect exposure is missed.
• The graph tool walks DEPENDS_ON/USES to the vulnerable library and returns the affected services with the full path.

Neo4j schema

Node properties below match data/seed_graph.cypher exactly.

Nodes:

• (:Service {name, environment}) — environment is production, staging, or dev
• (:Library {id, name, version}) — id is "<name>@<version>" and is the unique key
• (:CVE {id, severity, description, published})
• (:Team {name})

Relationships:

• (:Team)-[:OWNS]->(:Service) — ownership is a relationship, not a property on Service
• (:Service)-[:DEPENDS_ON]->(:Library)
• (:Library)-[:USES]->(:Library) — transitive dependency chain
• (:Library)-[:HAS_VULNERABILITY]->(:CVE)

How routing works

router.py makes one LLM call and returns a RouterOutput (route, reasoning, confidence), all schema-validated:

• definition / summary / single-entity lookup → vector
• multi-hop, transitive, "path", "impacted by", "blast radius" → graph
• needs a traversal plus narrative context → hybrid

act_node runs the selected tool. On hybrid the vector and graph tools run concurrently with asyncio. synthesize_node then takes the raw tool output and produces a FinalAnswer, with the same parse-validate-repair guarantee as the router. The executed route is stamped onto the answer so it always reflects what actually ran.

What's not built yet

This is a PoC. The following are deliberately out of scope and would be the next steps for a production version:

• Cost/latency tracking (e.g. MLflow or LangSmith spans per node)
• A semantic cache (e.g. Redis keyed on the query embedding)
• Scaling: Neo4j read replicas, moving Chroma to pgvector or Qdrant

License

MIT