MedGraph-Multimodal

A demonstration of hybrid vector–graph retrieval and safety-gated LLM orchestration, built around a clinical decision-support scenario.

⚕️ Not a medical device — architecture demonstration only. This is a research/engineering project. It has not been evaluated on real patient data, has undergone no regulatory or clinical review, and is not intended for use in patient care. The evaluation thresholds documented later in this README (e.g. faithfulness ≥ 0.85, answer relevancy ≥ 0.80) are illustrative of the evaluation pattern against synthetic cases — they are not clinically validated benchmarks. Any outputs are engineering artifacts, never clinical advice.

The clinical domain was chosen as a concrete, high-stakes setting to exercise the architecture — hybrid retrieval, Reciprocal Rank Fusion, and structural safety-gating — not because this is a working clinical answering system. The demonstration combines three retrieval signals over one knowledge base:

• a Neo4j property graph of clinical entities (symptoms, diagnoses, procedures, medications) and their curated relationships (contraindications, treatment links);
• OpenCLIP multimodal embeddings of imaging + text stored in a Qdrant vector index; and
• LangGraph orchestration that triages the query, retrieves fused context, runs a mandatory safety gate, and only then synthesizes a plan.

The design goal is not "chatbot over documents" but grounded, auditable reasoning — in this demonstration, every treatment plan must pass a contraindication and cross-reaction check before it can be produced.

---

Table of contents

• Why it's built this way
• Architecture
• The reasoning workflow
• Retrieval: Vector–Graph Fusion
• The safety gate
• Data model
• Quickstart
• Usage
• Configuration
• Project layout
• Development
• Supply-chain posture
• Scope and limitations

---

Why it's built this way

Concern	Approach
Structured clinical knowledge	A Neo4j graph encodes entities and the edges that matter for safety (CONTRAINDICATED_WITH, treatment relations), so conflicts are queryable, not buried in prose.
Semantic + visual recall	OpenCLIP puts imaging and text in a shared embedding space (Qdrant), so a query can retrieve relevant findings across modalities.
Recall and precision	Neither vectors nor graph traversal alone is enough — the hybrid retriever anchors semantic hits into the graph and fuses both rankings with Reciprocal Rank Fusion (RRF).
Safety is non-optional	The workflow graph has no edge from retrieval to plan synthesis that skips the safety node. The gate is structural, not a prompt instruction.
Automated LLM evaluation	Eval thresholds (faithfulness ≥ 0.85, answer relevancy ≥ 0.80, in tests/eval/judge.py) gate the CI suite against synthetic test cases. This demonstrates the pattern of automated LLM-judge evaluation — it does not certify clinical accuracy.
Reproducible, auditable installs	uv with a committed lockfile, wheels-only installs, single index, and CVE scanning in CI.

---

Architecture

                         ┌──────────────────────────────────────────┐
                         │             LangGraph workflow            │
   clinical query  ─────▶│  triage → retrieve → safety → synthesize  │────▶ plan
                         └───────────────┬───────────────┬──────────┘
                                         │               │
                          ┌──────────────▼───┐   ┌───────▼──────────┐
                          │ Hybrid retriever │   │  Safety checker  │
                          │  (Vector⇄Graph   │   │ graph edges +    │
                          │   RRF fusion)    │   │ LLM cross-react  │
                          └───┬──────────┬───┘   └───────┬──────────┘
                              │          │               │
                   ┌──────────▼──┐   ┌───▼───────────┐   │
                   │   Qdrant    │   │    Neo4j      │◀──┘
                   │ OpenCLIP    │   │ property graph│
                   │ vectors     │   │ (+APOC)       │
                   └─────────────┘   └───────────────┘
                          ▲                  ▲
                          └──── ingestion ───┘
                     literature · cases · imaging

Two backing services (defined in compose.yml):

• Neo4j 5.26 (community, APOC enabled) — the property graph. APOC file import/export is disabled and procedures are allow-listed so untrusted graph data can't trigger I/O.
• Qdrant 1.12 — the multimodal vector store.

---

The reasoning workflow

Built in src/medgraph/graph/builder.py as a compiled, cyclic LangGraph StateGraph. A single ClinicalAgentState (see graph/state.py) is threaded through every node; nodes return partial updates that LangGraph merges.

START
  └─▶ triage                         classify intent + extract patient attributes
        ├─ clarify ──▶ END           (insufficient detail to act safely)
        └─▶ retrieve_graph_data      Vector–Graph Fusion → unified context
              └─▶ safety_verification   MANDATORY contraindication / cross-reaction gate
                    ├─ alert_handler ──▶ END     (blocking contraindication: SAFETY HALT)
                    ├─ retrieve_graph_data       (loop: widen traversal, bounded budget)
                    └─▶ synthesize ──▶ END       (cleared: produce the cited plan)

Node	Responsibility
triage	OpenAI structured-output call extracts age/sex/allergies/medications/conditions and an intent label in one shot, so downstream nodes work off typed state.
clarify	Terminal node when the query lacks enough detail to proceed.
retrieve_graph_data	Runs the hybrid retriever, attaches a FusionContext to state.
safety_verification	The gate. Combines curated graph edges, a deterministic allergy check, and an LLM cross-reaction pass.
alert_handler	Emits a SAFETY HALT — no plan — when a blocking conflict is found.
synthesize	Composes a terse, clinician-facing plan, surfacing any non-blocking cautions.

The only path to synthesize is through safety_verification returning no blocking flags. The retrieve ⇄ safety cycle is the refinement loop, bounded by an iteration budget (_MAX_ITERATIONS = 2).

State is checkpointed (MemorySaver by default), so a stable thread_id keeps memory across turns within a session.

---

Retrieval: Vector–Graph Fusion

src/medgraph/retrieval/hybrid_retriever.py implements the core retrieval idea:

1. Vector search over the shared image/text space (Qdrant) → the most semantically relevant points become anchors.
2. Anchor into the graph via entity keys in the point payloads, then run a multi-hop Cypher traversal outward from the anchors.
3. Fuse the two signals with Reciprocal Rank Fusion (RRF_K = 60): a node's score blends its vector rank (semantic relevance) and its graph proximity (structural relevance to the anchors).
4. Return a unified structural context — ranked nodes plus the edges among the survivors — ready for the reasoning/synthesis node.

Anti-leakage: because medical terms overlap heavily in embedding space, an absolute similarity floor can't cleanly separate patient cases. The primary lever is case-scoping — lock anchors to the case of the single most-relevant hit (fusion_scope_to_top_case), with the score floor as a secondary filter.

---

The safety gate

src/medgraph/graph/nodes/safety.py layers three independent checks and fails closed:

1. Curated graph assertions — CONTRAINDICATED_WITH edges already in Neo4j for the medications in the patient profile and fused context.
2. Deterministic allergy cross-check — any medication matching a documented allergy is a blocking, severe flag.
3. LLM cross-reaction reasoning — a structured-output OpenAI call reasons over the medication list for pairwise interactions and drug–condition/allergy conflicts the graph may not yet encode. If this call fails, it emits a non-blocking "requires human review" flag rather than silently passing.

Any blocking flag (severe severity, or an allergy match) routes to the alert handler. The loop re-retrieves only when a patient medication was not represented in the retrieved context (fusion likely missed its neighborhood) and the iteration budget isn't exhausted.

---

Data model

Medical entities are extracted from free-text case reports via OpenAI structured outputs, constrained to the Pydantic schema in src/medgraph/ingestion/schema.py (the field descriptions double as extraction instructions):

• Entities — Symptom, Diagnosis (with ICD-10 + status), Procedure, Medication (dose/route/frequency). Each carries an extraction confidence in [0,1] and a verbatim evidence_span.
• Relationships — Contraindication (medication ✗ entity, with severity + rationale) and TreatmentRelation (intervention → target, with certainty).
• Controlled vocabularies: DiagnosisStatus, Severity, Certainty.

Everything rolls up into a CaseExtraction, which the ingestion graph_writer persists into Neo4j and whose embeddings land in Qdrant.

---

Quickstart

Requires uv, Docker, and Python 3.11/3.12.

cp .env.example .env          # fill in secrets (see Configuration)
docker compose up -d          # start Neo4j (+APOC) and Qdrant
uv sync                       # install from the locked, wheels-only resolution
uv run medgraph               # launch the interactive CLI

Optional extras:

uv sync --extra imaging       # DICOM parsing (pydicom, numpy) for imaging ingestion
uv sync --extra ui            # Streamlit chat UI  →  streamlit run ui/app.py

---

Usage

The medgraph console script (src/medgraph/cli.py) runs in two modes:

# One-shot: run a single query, print the plan (or clarification / safety halt).
uv run medgraph -q "45F, productive cough and fever, on warfarin. Workup?"

# Interactive REPL: graph built once, checkpointer threads turns together.
uv run medgraph

# Show intent / iterations / safety flags after each answer.
uv run medgraph --trace -q "..."

Startup fails fast with a friendly message (not a traceback) when required secrets are missing.

---

Configuration

All runtime config is validated via pydantic-settings (src/medgraph/config.py) from environment variables / .env, using the MEDGRAPH_ prefix. Copy .env.example and fill in:

Variable	Purpose
MEDGRAPH_OPENAI_API_KEY	OpenAI key for triage / safety / synthesis.
OPENAI_API_KEY	Same value; read directly by the eval judge and OpenAI SDK.
MEDGRAPH_NEO4J_URI / _USER / _PASSWORD	Neo4j Bolt connection.
NEO4J_PASSWORD	Consumed by compose.yml to set container auth.
MEDGRAPH_QDRANT_URL / _API_KEY	Qdrant connection.
QDRANT_API_KEY	Consumed by compose.yml.

Tunable defaults (in config.py): reasoning_model / synthesis_model (gpt-4.1), clip_model (ViT-B-32 / laion2b_s34b_b79k), embedding_dim (512), and the fusion knobs fusion_vector_limit, fusion_min_score, fusion_scope_to_top_case.

🔒 Never put a real key in .env.example — it is committed. Secrets go only in your local, git-ignored .env.

---

Project layout

src/medgraph/
├── config.py               # validated env-based settings (pydantic-settings)
├── cli.py                  # console entry point (one-shot + REPL)
├── resilience.py           # token-bucket rate limiter + retry decorator for LLM calls
├── db/                     # Neo4j + Qdrant client factories
│   ├── neo4j_client.py
│   └── vector_client.py
├── embeddings/
│   └── clip_encoder.py     # OpenCLIP image/text encoder
├── ingestion/              # source → graph + vectors
│   ├── schema.py           # Pydantic extraction contract
│   ├── graph_writer.py     # persist entities/edges into Neo4j
│   ├── literature/
│   ├── cases/extractor.py  # LLM entity extraction from case reports
│   └── imaging/embedder.py
├── retrieval/
│   ├── graph_retriever.py  # multi-hop Cypher traversal
│   ├── vector_retriever.py # Qdrant similarity search
│   └── hybrid_retriever.py # Vector–Graph RRF fusion
└── graph/                  # LangGraph orchestration
    ├── state.py            # ClinicalAgentState + typed reducers
    ├── builder.py          # compiled cyclic StateGraph
    ├── dependencies.py     # cached client accessors
    └── nodes/              # triage · retrieve · safety · synthesize
tests/
├── unit/
├── integration/
└── eval/                   # LLM-judge evals + safety-gate tests
ui/                         # optional Streamlit chat UI
compose.yml                 # Neo4j + Qdrant local stack
pyproject.toml · uv.lock    # dependencies, pinned + hash-locked

---

Development

uv sync --group dev          # ruff, mypy, pytest, pytest-cov, pip-audit

uv run pytest                # run the test suite
uv run pytest tests/eval     # safety-gate + LLM-judge evals
uv run ruff check .          # lint
uv run mypy src              # type-check
uv run pip-audit             # CVE scan of the resolved lockfile

Outbound LLM calls are wrapped by resilience.py: a client-side token-bucket rate limiter (stay under account RPM before the API 429s) plus a with_retries decorator that adds bounded, jittered exponential backoff and honors the server's retry-after header — retrying only transient failures (429 / 5xx / connection drops), never client errors.

---

Supply-chain posture

pyproject.toml configures uv to install prebuilt wheels only (no-build = true), so no setup.py / PEP 517 build hooks execute during install — dependencies cannot run arbitrary code at build time. Resolution is pinned to a single index (first-index) and frozen in uv.lock (locked = true). Dev/CI dependencies live in a PEP 735 group that never ships to production, and pip-audit scans the resolved lockfile for CVEs. Flip require-hashes = true once the lockfile is committed to enforce hash verification on every install.

---

Scope and limitations

To be unambiguous about what this project is and is not:

• Not evaluated on real patient data or outcomes. All cases, imaging, and graph content are synthetic or illustrative, used to exercise the pipeline.
• No regulatory pathway. It is not Software as a Medical Device (SaMD), has had no clinical or regulatory review, and is not intended for use in patient care or clinical decision-making.
• The knowledge graph and case data are for demonstration only — they are not a curated, validated medical knowledge base.
• The goal is to demonstrate architecture patterns — hybrid vector–graph retrieval with RRF fusion, a structural (non-bypassable) safety gate, and automated LLM-judge evaluation — not to produce a usable clinical tool.

The engineering claims elsewhere in this README (RRF fusion, the safety gate having no edge that bypasses it, parameterized Cypher, wheels-only installs) are accurate descriptions of the code. The limitations above concern clinical readiness, which this project deliberately does not claim.