Quorum is an extendable LLM-as-a-judge evaluation framework for multilingual quality benchmarking. It measures translation and instruction-following capability across challenger models against a fixed anchor (Gemma-4), combining absolute rubric scoring with statistically-grounded pairwise comparison.

Win-rates are computed via sequential, confidence-interval-driven sampling (Wilson score intervals) rather than fixed-N tallying — concentrating evaluation budget on genuinely close comparisons and stopping once a result is statistically resolved, not arbitrarily. The framework is provider-agnostic (Anthropic, Gemini, OpenAI, extendable to others), deployable at scale on Modal with GPU generation workers, and designed to incorporate human calibration — so every result carries an explicit, queryable validation tier rather than uniform, unearned confidence.

---

Features

• Three-provider judging — Anthropic Claude, Gemini 2.5 Pro, and OpenAI GPT-4o run concurrently and independently, with per-provider results compared for divergence
• Batch inference — 20 items packed into a single API call per provider, reducing judge latency and cost by 20x over item-by-item evaluation
• Pairwise comparison — challenger vs. Gemma-4 anchor with two swap runs per item to cancel position bias; swap disagreements recorded as ties with an explicit reason field
• Sequential sampling — Wilson score interval stop rule halts evaluation once the 95% CI half-width is at or below 0.08; hard cap at 150 items prevents runaway cost on inconclusive comparisons
• FLORES+ benchmark — 26 EU official languages sourced from openlanguagedata/flores_plus; source is English (eng_Latn), targets are BCP-47 config names verified against the dataset
• Multi-judge calibration — per-item ValidationTier assigned automatically from judge agreement: MULTI_JUDGE_CONVERGED, MULTI_JUDGE_DIVERGED_UNRESOLVED, JUDGE_ONLY; human tiers (HUMAN_SPOT_CHECKED, HUMAN_CALIBRATED) reserved for future companion service
• vLLM generation workers — Gemma-4-26B-A4B-it and EuroLLM-22B-Instruct run as Modal GPU workers with memory snapshot support for fast cold start
• Pluggable provider architecture — adding a new judge provider is one file and one registry entry; openframe-core handles lifecycle, health, and structured error propagation
• Rubric-driven prompts — pointwise and pairwise prompts rendered from a single YAML rubric; constraint block injected at the top of every prompt for reliable output format compliance

---

Quick Start

1. Install and authenticate Modal

pip install modal
modal setup

2. Create the Modal secret

modal secret create quorum-secrets \
  ANTHROPIC_API_KEY=your_anthropic_key \
  GEMINI_API_KEY=your_gemini_key \
  OPENAI_API_KEY=your_openai_key \
  HF_TOKEN=your_huggingface_token

3. Deploy the app

modal deploy modal_app.py

4. Download model weights (one-time)

modal run modal_app.py --stage download

5. Run the full pipeline

modal run modal_app.py --stage pipeline --language german

This runs all four stages in sequence: load → generate → judge → report. A run_id is auto-generated and printed at the start.

6. Run pairwise comparison against an existing run

modal run modal_app.py --stage pairwise --run-id <run_id> --model eurollm-22b --language german

Reads existing generation outputs from the volume — no GPU time or new generation needed.

➡️ See individual stage flags below for granular control over each pipeline step.

---

Architecture

Quorum runs as a single Modal app with separate CPU and GPU functions per stage. All outputs are persisted to named Modal volumes between stages.

FLORES+ dataset (CPU)
    └── dataset_{language}_{task}.jsonl → quorum-generations volume

Generation workers (GPU, parallel)
    ├── Gemma4Worker  → gemma-4_{language}_{task}.jsonl
    └── EuroLLMWorker → eurollm-22b_{language}_{task}.jsonl

Pointwise judge (CPU, 4 containers in parallel)
    ├── anthropic-judge × gemma-4
    ├── anthropic-judge × eurollm-22b
    ├── gemini-judge    × gemma-4
    └── gemini-judge    × eurollm-22b
    → {provider}_{model}_{language}_{task}.jsonl → quorum-results volume

Calibration report (CPU)
    └── run_calibration_pass() → per-item ValidationTier → report_{language}_{task}.json

Pairwise judge (CPU, 3 containers in parallel)
    ├── anthropic-judge: sequential sampling, swap runs, Wilson CI
    ├── gemini-judge:    sequential sampling, swap runs, Wilson CI
    └── openai-judge:    sequential sampling, swap runs, Wilson CI
    → pairwise_{provider}_{model}_{language}_{task}.jsonl
    → pairwise_{provider}_{model}_{language}_{task}_summary.json

Generation workers use vLLM with memory snapshots (@modal.enter(snap=True)) so model weights are loaded onto GPU once and restored from a snapshot on subsequent cold starts — bypassing the 30–60s HuggingFace download on every container restart.

Judge containers are CPU-only. Each provider runs in its own container so rate limits and failures are isolated. The starmap.aio() pattern fans all judge tasks into parallel containers without any shared state.

Pairwise sampling runs the sequential controller inside a single long-lived container per provider. Items are drawn from the existing generation volume in batches; each batch gets two swap calls (A=challenger/B=anchor and A=anchor/B=challenger) run concurrently, then reconciled into a win/loss/tie verdict. Results are written incrementally so a crash leaves partial data rather than losing everything.

---

Pipeline Stages

# Full pipeline — auto-generates a run_id
modal run modal_app.py --stage pipeline --language german

# Individual stages — pass --run-id to resume an existing run
modal run modal_app.py --stage load     --run-id <id> --language german
modal run modal_app.py --stage generate --run-id <id> --language german
modal run modal_app.py --stage judge    --run-id <id> --language german
modal run modal_app.py --stage report   --run-id <id> --language german

# Pairwise — reads generation outputs, no GPU needed
modal run modal_app.py --stage pairwise --run-id <id> --model eurollm-22b --language german

# Pairwise with a single provider (skips the others)
modal run modal_app.py --stage pairwise --run-id <id> --model eurollm-22b --provider openai-judge

# Utilities
modal run modal_app.py --stage download    # download model weights to cache volume
modal run modal_app.py --stage smoke-test  # quick generation check — no volume writes

---

Volumes

Volume	Mount	Contents
quorum-generations	/vol/generations	FLORES+ dataset JSONL, model generation outputs
quorum-results	/vol/results	Per-provider judge JSONL, calibration reports, pairwise summaries
quorum-model-cache	/vol/model-cache	HuggingFace weights (safetensors only)
quorum-vllm-compile-cache	/root/.cache/vllm	Triton kernel and torch.compile artifacts

---

Judge Providers

Provider	Model	Env var	Batch max_tokens
anthropic-judge	claude-sonnet-4-6	ANTHROPIC_API_KEY	400 × batch_size
gemini-judge	gemini-2.5-pro	GEMINI_API_KEY	8192 (fixed ceiling)
openai-judge	gpt-4o	OPENAI_API_KEY	400 × batch_size, max 16384

Model names can be overridden at runtime via environment variables (ANTHROPIC_JUDGE_MODEL, GEMINI_JUDGE_MODEL, OPENAI_JUDGE_MODEL) without code changes.

---

Pairwise Output

Each pairwise run writes two files per provider to quorum-results/{run_id}/:

pairwise_{provider}_{model}_{language}_{task}.jsonl — one record per swap run:

{
  "item_id": "flores_123",
  "swap_run": 0,
  "output_a_model": "eurollm-22b",
  "output_b_model": "gemma-4",
  "winner": "B",
  "winner_model": "gemma-4",
  "confidence": "high",
  "reasoning": "Translation B preserves all factual details with natural phrasing.",
  "provider": "anthropic-judge",
  "judge_model": "claude-sonnet-4-6",
  "error": null
}

pairwise_{provider}_{model}_{language}_{task}_summary.json — aggregate statistics:

{
  "win_rate": 0.306,
  "wilson_ci_low": 0.224,
  "wilson_ci_high": 0.403,
  "ties": 52,
  "tie_rate": 0.347,
  "n_sampled": 150,
  "n_wins": 30,
  "n_losses": 68,
  "status": "inconclusive_at_cap",
  "provider": "anthropic-judge",
  "model": "eurollm-22b",
  "anchor_model": "gemma-4",
  "language": "german",
  "task": "translation"
}

status is one of resolved (CI half-width ≤ 0.08) or inconclusive_at_cap (hit the 150-item cap before the CI tightened).

---

Project Structure

├── src/
│   ├── core/
│   │   ├── rubric_loader.py      # pointwise + pairwise prompt rendering (batch and single)
│   │   ├── pairwise.py           # pairwise judge loop: swap runs, reconciliation, incremental writes
│   │   ├── sampling.py           # SequentialSamplingController + SamplingState
│   │   ├── stats.py              # Wilson score CI (z=1.96, ties excluded from denominator)
│   │   ├── json_parser.py        # depth-aware JSON extraction from judge output (dict + list)
│   │   └── ports.py              # JudgeResult dataclass, JudgeProvider Protocol
│   ├── data/
│   │   ├── dataset_loader.py     # FLORES+ loading, render_translation_prompt
│   │   ├── divergence.py         # calibration pass, ValidationTier assignment, print_report
│   │   └── schema.py             # CalibrationItem, JudgeVerdictRecord, ValidationTier enum
│   ├── plugins/
│   │   ├── anthropic.py          # AnthropicJudgePlugin (urllib, evaluate + evaluate_batch)
│   │   ├── gemini.py             # GeminiJudgePlugin (google-genai, thinking token filtering)
│   │   ├── openai.py             # OpenAIJudgePlugin (AsyncOpenAI, json_object mode)
│   │   └── registry.py           # PROVIDER_REGISTRY_MAP, build_registry
│   ├── workers/
│   │   ├── vllm_base.py          # VLLMConfig, VLLMWorker (generate, warmup, sleep, wake_up)
│   │   ├── model_downloader.py   # HuggingFace snapshot_download to cache volume
│   │   └── models/
│   │       ├── gemma.py          # Gemma-4-26B-A4B-it config (chat_template="auto")
│   │       └── eurollm.py        # EuroLLM-22B-Instruct config (chat_template="auto")
│   ├── prompts/
│   │   └── translation.yaml      # rubric: fluency, adequacy, overall; pointwise + pairwise modes
│   ├── main.py                   # run_single_provider, build_report, prepare_dataset
│   └── utils/
│       ├── config.py             # AnthropicSettings, GeminiSettings, OpenAISettings
│       └── const.py              # N_ITEMS, JUDGE_BATCH_SIZE, PAIRWISE_ANCHOR_MODEL, FLORES_TARGET_CODES
├── modal_app.py                  # all Modal functions + pipeline entrypoint
├── modal_common.py               # judge/vllm/downloader images, volumes, EnvConfig, build_*_config
├── tests/                        # unit tests
├── data/                         # local candidate outputs (gitignored)
└── docs/                         # additional documentation

---

Adding a New Judge Provider

1. Create src/plugins/{name}.py with a class that has name, version, capability = "judge-provider", and initialize, shutdown, health, evaluate, evaluate_batch methods — no inheritance required, structural typing only
2. Add a settings class to src/utils/config.py inheriting BaseAdapterSettings
3. Add one entry to PROVIDER_REGISTRY_MAP in src/plugins/registry.py
4. Add the provider's pip package to _JUDGE_PYTHON_PACKAGES in modal_common.py

The provider is automatically available in all pipeline stages and the pairwise system.

---

Adding a New Generation Model

1. Create src/workers/models/{name}.py with a VLLMConfig instance
2. Add a @app.cls worker class in modal_app.py following the Gemma4Worker pattern
3. Add _MODEL_NAME and wire generate_for_run to _generate_and_write
4. Add the model ID to ALL_MODELS for weight pre-downloading

---

Roadmap

Phase	Area	Goal
01 — Multi-language sweep	Evaluation breadth	Run the full pipeline across all 26 FLORES+ EU language pairs in a single modal run invocation, writing per-language results and a cross-language summary report
02 — Sampling manifest	Pipeline orchestration	Drive which models × languages × tasks to evaluate from a YAML manifest file, replacing the current hardcoded per-run CLI flags
03 — Instruction-following tasks	Task coverage	Extend the rubric and dataset layer beyond translation to ArenaHard-EU instruction-following, with a new task YAML and corresponding generation prompt
04 — Human calibration service	Validation tier	Build a companion annotation service that writes human verdicts to the existing CalibrationItem schema, promoting items to HUMAN_SPOT_CHECKED and HUMAN_CALIBRATED tiers without any changes to the judge pipeline
05 — Additional challenger models	Model coverage	Add Mistral, Llama-3, and Phi-4 as vLLM generation workers; each challenger automatically participates in all existing pairwise and pointwise evaluations against the Gemma-4 anchor
06 — Leaderboard report	Reporting	Aggregate per-language pairwise win-rates and Wilson CIs across all challengers into a ranked leaderboard table, written to the results volume and printable as markdown
07 — Resolved pairwise at scale	Sampling tuning	Profile CI convergence rates per language and task; tune target_ci_halfwidth and max_n per stratum so the stop rule fires resolved rather than inconclusive_at_cap on the majority of runs
08 — Cost and latency dashboard	Observability	Track tokens consumed, API call counts, and wall-clock time per provider per run; surface as a JSON cost report alongside the calibration report

---

Contributing

Contributions are welcome. Please open an issue before starting significant work to align on approach.

Getting started

git clone https://github.com/your-org/quorum
cd quorum
pip install modal pyyaml pydantic-settings openframe-core

For local development, set your API keys directly and run the judge engine locally:

export ANTHROPIC_API_KEY=your_key
python -m src.main --language german --model eurollm-22b

Unit tests:

python -m pytest tests/

Guidelines

• One concern per PR. Judge providers, generation workers, the sampling controller, and the calibration layer are intentionally separate modules — keep changes focused.
• Never fabricate a score on failure. All judge providers must raise the appropriate AdapterError subclass on failure. JudgeResult(error=...) is set by the engine's single catch point — never inside a provider.
• Structural typing only. New providers do not inherit from any base class. Satisfying the JudgeProvider Protocol shape is sufficient.
• Batch always, item-by-item never. New judge integrations must implement evaluate_batch alongside evaluate. The pipeline does not call evaluate in a loop.
• Constants in const.py. Any value referenced in more than one file belongs in src/utils/const.py with a one-line comment.
• Prompt constraints at the top. The STRICT OUTPUT RULES block must be injected before dimension definitions in all rendered prompts — models ignore constraints buried after long criterion lists.

---

License

MIT — see LICENSE.