ragdocket

A retrieval-augmented evaluation toolkit for the legal domain. It runs lm-evaluation-harness tasks with an optional evidence-retrieval step in front of the model under test, and an optional LLM-critic rubric score after it, from either a CLI or a small web console.

ragdocket depends on lm-eval as an ordinary package; it does not fork it. Task YAMLs use lm-eval's own class: seam to plug in evidence injection and rubric scoring, and the critic is just another model in lm-eval's own model registry. See docs/architecture.md for why, and docs/decisions.md for how that was verified.

Install

pip install ragdocket

That gets you the CLI, the web service, and the dummy backend — enough to run --help and --tasks list with no JVM, no GPU, and no heavy ML libraries pulled in. Retrieval and reordering backends are optional extras:

Extra	Adds	Needed for
ragdocket[lexical]	bm25s	--source lexical
ragdocket[dense]	sentence-transformers, numpy	--source dense, --source hybrid
ragdocket[reorder]	sentence-transformers	--reorder --orderer cross_encoder
ragdocket[web]	fastapi, uvicorn, pydantic	ragdocket-web

Install more than one with pip install "ragdocket[lexical,web]". If you use a backend without its extra installed, ragdocket fails with an actionable pip install ragdocket[...] message rather than a bare ImportError.

Quickstart: CLI

# see what's runnable
ragdocket --tasks list

# a bare run against the built-in dummy model
ragdocket --tasks legal_demo_gen --include-path ragdocket/benchmarks/legal_demo \
  --model dummy --limit 2 --output-path out.json

# with evidence retrieval and rubric scoring
ragdocket --tasks legal_demo_gen --include-path ragdocket/benchmarks/legal_demo \
  --model dummy \
  --with-evidence --source lexical --source-opts corpus_path=tests/fixtures/corpus.jsonl --evidence-k 3 \
  --critic dummy \
  --limit 2 --output-path out.json

Adding --reorder --orderer cross_encoder reorders retrieved evidence with a cross-encoder before it's rendered; it needs the reorder extra (sentence-transformers).

--tasks takes a comma-separated list of task names, resolved the same way lm-eval resolves its own built-in tasks, plus any directory passed via --include-path (for a project-local task like the bundled legal_demo_* pair used in this repo's own tests).

Retrieval and reordering backends, and the critic, are named via --source/--orderer/--critic and configured with a comma-separated key=value string via --source-opts/--orderer-opts/--critic-opts (types are inferred: k=3,rerank=true,name=bm25 parses int, bool, and str respectively). --critic accepts any name registered in lm-eval's own model registry (dummy, hf, ...); --critic-device and --critic-sampling mirror --model-args' role for the critic and its generation kwargs.

A failed rubric parse is never silently scored as zero: it is excluded from the rubric_score mean and counted separately in rubric_parse_failures, so a judge that returns malformed JSON shows up as a visible metric rather than a quiet drag on the average.

Flags can also be given defaults from a YAML file with --config path.yaml; any flag also passed on the command line overrides the file.

Quickstart: web console

pip install "ragdocket[web]"
ragdocket-web --data-dir ~/.ragdocket

This serves the built console (a small React app, already built into ragdocket/service/assets/ and shipped with the package) and the /v1 API on http://127.0.0.1:8000 by default. From the console you can launch a run, watch it stream through its stages live, inspect predictions and rubric verdicts, and diff two runs' metrics side by side.

Runs execute one at a time, FIFO, in a background worker thread; POST /v1/runs/{id}/stop cancels an active run at its next checkpoint. Run state lives in a small SQLite file under --data-dir; per-run logs and prediction files live under --results-dir (defaults to <data-dir>/results).

Backend matrix

Kind	Name	Extra	Options
source	lexical	lexical	corpus_path
source	dense	dense	corpus_path, model_name
source	hybrid	dense	rrf_k (default 60)
orderer	cross_encoder	reorder	model_name

GET /v1/meta/sources and GET /v1/meta/orderers list whatever is registered at runtime (the console's launch form reads these directly).

Adding a source, orderer, or task

• A source implements ragdocket.core.sources.DocumentSource: a search(query, k) -> EvidenceBundle method, an accepted_options frozenset, and a from_options(spec) classmethod. Register it with ragdocket.core.plugins.SOURCES.register("name", YourClass).
• An orderer implements ragdocket.core.ordering.Reorderer the same way, with reorder(bundle) -> EvidenceBundle instead of search, registered on ORDERERS.
• A task is a normal lm-eval task YAML with two extra keys: class: !function yourmodule.YourTask (a ragdocket.benchmarks.task.EvidenceTask subclass, or EvidenceTask itself) and query_for/criteria_for (a column name, a {template} string, or a callable) telling the task which document field is the retrieval query and which is the rubric criteria. See ragdocket/benchmarks/legal_demo/ for a complete worked example.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[lexical,dense,reorder,web]"
pytest -m "not slow and not network"   # offline, no model download
ruff check ragdocket tests tools
python tools/originality.py /path/to/LRAGE   # optional, needs the reference tree

The console lives in console/ and is a separate Node project:

cd console
npm install
npm test        # vitest + MSW
npm run build    # writes ragdocket/service/assets/, committed to the repo