Agentic Retrieval on Qdrant: a measurement lab

Nota Bene: Forked from (Dylan's self-correcting-loops-workshop repository)[https://github.com/qdrant-labs/self-correcting-loops-workshop]

A hands-on lab for reasoning about agentic retrieval on Qdrant. It builds a self-correcting retrieval loop from primitives, then measures a purpose-trained retrieval agent (SID-1) against ordinary retrieval and against that loop. The question underneath all of it: how much retrieval quality can be moved between Qdrant configuration and agent effort.

Everything runs on one local Qdrant instance, one MuSiQue-derived mixed workload (single-hop, multi-hop, and unanswerable questions), and a fixed top-3 answer window so ranking quality is what shows up in the score. All embedding and reranking is local (FastEmbed ONNX); only the agent, the stop autorater, and the optional SID-1 arm hit the network.

The notebooks

Read them in this order. Each one is self-contained (it builds retrieval, the gate, and the loops inline from src/ primitives), but they layer conceptually and share one banked row/trace store, so a run survives interruption and an arm is paid for once.

Foundational, "is the agent worth it?" (banks the traces the stands reuse):

Notebook	What it measures
sid_vs_baselines.ipynb	SID-1 vs ordinary retrieval (one hybrid call, one ColBERT rerank) on the pooled test split.
sid_vs_loops.ipynb	SID-1 vs the hand-built self-correcting ladder (IRCoT decompose plus signal-routed tiers), on multi-hop.

The three stands, the exchange rate between Qdrant config and agent effort:

Notebook	What it measures
stand1_configuration.ipynb	SID-1 held fixed; sweep one search knob at a time (tool limit, RRF vs DBSF fusion, prefetch depth). What a config change buys or costs in searches, turns, and tokens at held coverage.
stand2_exotic_tools.ipynb	Hand SID-1 a Qdrant tool it was never trained on (recommend, exclude-ids, ColBERT precise_search); measure adoption, malformed-call rate, and coverage delta.
stand3_quantization.ipynb	The paid quantization run: the calls/tokens curve at flat coverage, and where the agent can no longer absorb the substrate thrift.
sid_quantization.ipynb	Precursor free sweep (dense and hybrid quantization levels) that Stand 3 builds on.

Experimental side-studies live in notebooks/experimental/ with their own README: scaling to 100K and 1M haystacks, realistic filtering with ACORN, and SID-1 per-search forensics.

The SID-1 arms need SID_API_KEY. Without it they skip themselves and the rest still runs.

Prerequisites

• Docker (for Qdrant)
• Python 3.12
• An .env at the repo root (see Setup)

Setup

# 1. Qdrant. v1.19.0 on ports 6343/6344 (non-default ON PURPOSE, see docker-compose.yml)
docker compose up -d
curl -s http://localhost:6343/healthz

# 2. Python env
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip uv
uv pip install -r requirements.lock.txt        # exact pins; or requirements.txt for loose floors

# 3. .env at the repo root
#   ANTHROPIC_API_KEY=...                  required: the agent + stop autorater (via LiteLLM)
#   QDRANT_URL=http://localhost:6343       required: config defaults to :6333, but Qdrant is on :6343
#   SID_API_KEY=...                        optional: enables the SID-1 arms (they skip themselves if unset)
#   SID_API_URL=...                        optional: override the SID endpoint

# 4. Build the two collections from the committed corpus (one-time; ~22.8k docs).
#    First run downloads + caches the FastEmbed models, so it takes a few minutes.
python scripts/setup_collections.py            # musique: dense + miniCOIL
python scripts/setup_colbert.py                # musique_colbert: dense (reused) + ColBERT multivector

# 5. Verify everything is "Ready"
python scripts/check_env.py --llm

check_env.py confirms the libraries import, FastEmbed exposes the models the notebooks use, Qdrant is healthy, both collections are populated, and (with --llm) the Claude agent answers. Then open a notebook and run its Setup cell; it should print Ready.

Large data + artifacts (DVC)

The base corpus and question splits are committed to git, so the setup above and the canonical notebooks work with no extra steps. The large scaling corpora (corpus_wiki100k/1m.jsonl, up to 467 MB) and the benchmark row/trace .jsonl files are tracked with DVC instead of git. The pointer .dvc files are in the repo; the bytes live in a local DVC cache.

No DVC remote is configured yet (local cache only). Once one is added, colleagues run dvc pull to fetch the corpora and banked rows. Until then, the DVC-tracked files are only present on machines that already have the cache.

The scaling rungs (used by the experimental notebooks) build the 1M collection once, then derive the 100K collection from its stored vectors, so there is no re-embedding:

MUSIQUE_VARIANT=wiki1m python scripts/setup_collections.py --stream --parallel 4 --resume
MUSIQUE_VARIANT=wiki100k python scripts/setup_collections.py --stream \
  --from-collection musique_wiki1m

The stack

Layer	Choice
Vector DB	Qdrant v1.19.0 (Docker, ports 6343/6344), collections musique + musique_colbert, named vectors
Dense	BAAI/bge-base-en-v1.5 (768-d, cosine), FastEmbed / local ONNX
Sparse	Qdrant/minicoil-v1 (word-sense-aware sparse, IDF modifier); the baseline fusion sparse
Fusion	Reciprocal Rank Fusion (RRF, k=60), server-side via the Qdrant Query API. No cross-encoder in the baseline; DBSF available as a sharper signal substrate
Tier-2 rerank	answerdotai/answerai-colbert-small-v1 ColBERT late interaction (Qdrant native multivector, MaxSim)
Agent	Claude Sonnet 4.6 via LiteLLM (decompose + answer)
Stop autorater	Claude Haiku 4.5 via LiteLLM (the optional LLM sufficiency check)
Benchmark agent	SID-1 (sid.ai), OpenAI-compatible endpoint, optional (needs SID_API_KEY)
Dataset	MuSiQue, recast as a mixed single-hop / multi-hop / unanswerable workload

Retrieval constants (src/config.py): RETRIEVE_N=50 raw per retriever, TOP_K=10 signal pool, ANSWER_K=3 focused passages the LLM actually reads.

The SID-1 comparison

SID-1 is sid.ai's RL-trained agentic retrieval model. It runs its own multi-turn search loop over your backend and returns ranked doc ids, never an answer. Here its information tools are wired to the same Qdrant collections everything else uses, so it searches the same index and is scored through the same top-3 window. The adapter is the src/sid/ package (runner, tools, prompt, models, trace); it only runs when SID_API_KEY is set, and otherwise those arms skip themselves.

Reproducibility

• data/: the base corpus (corpus.jsonl, 22,808 passages), the question splits (questions.jsonl), the derived mixed workload (questions_mixed.jsonl), and metadata, all committed. The large corpus_wiki{100k,1m}.jsonl variants are DVC-tracked.
• artifacts/mixed_manifest.json: the frozen mixed-workload population (seeded; every question id per split pinned), committed as evidence.
• artifacts/headline_final_v25.json: the precomputed workload-level scorecard, committed frozen. Notebooks cross-check their live numbers against it, so a drifted harness says so.
• artifacts/*_rows.jsonl and *_traces.jsonl: one row per (question, arm, repeat) and one trace per question, appended as a run proceeds and shared across notebooks (an arm is paid for once). DVC-tracked; delete to re-measure from scratch.

To rebuild the dataset from MuSiQue (needs HuggingFace access), run scripts/prepare_data.py, then scripts/audit_single_hop.py, then scripts/prepare_mixed.py.

Repo layout

notebooks/                 # canonical: sid_vs_{baselines,loops}, sid_quantization, stand1/2/3
notebooks/experimental/    # scaling (100K/1M), realistic filtering, SID forensics
src/                       # config, data, retrieval, harness, metrics, questions, filters, stand2
src/sid/                   # the SID-1 tool-loop adapter (runner, tools, prompt, models, trace)
scripts/                   # dataset build, collection setup, env check, benchmarks
data/                      # corpus + splits (base in git; large variants in DVC)
artifacts/                 # frozen manifest/scorecard (git); benchmark rows + figures (DVC/git)
docker-compose.yml         # Qdrant v1.19.0, ports 6343/6344
docs/                      # the working plan (PLAN_article.md), deferred work (TODOS.md), design notes