Local-LLM Bake-Off

A frozen, reproducible benchmark for deciding which local models to actually run: measured on your hardware, for your task.

Public leaderboards tell you how a model does on someone else's tasks, on someone else's hardware, at some point-in-time nobody froze. None of that tells you the thing you actually need to know: which of the models that fit on my GPU is best at the specific job I'm going to give it, and how much slower is the good one? This kit answers exactly that. Point it at a local Ollama, hand it a prompt battery you wrote and froze, and it screens a field of models down to a short, role-grouped list you can review by hand, the same way, every time, so next quarter's releases are measured against this quarter's on a fixed yardstick.

It is domain-neutral by design. The bundled example battery is about clock towers, greenhouses, and Kadane's algorithm: nothing to do with any particular product or field. That is the point: you fork it and swap in prompts from whatever you do.

---

What this is (and isn't)

• It's a funnel, not a verdict. The eval screens a broad field down to a ~10-15 shortlist that a human then personally prompts and reviews. A single automated score is enough to exclude a clear loser; it is never enough to anoint a winner. Err toward inclusion at the cut.
• It's frozen. Once a prompt battery has scored anything, you never edit it. Editing silently invalidates every prior comparison. Changes land as a new version. (See the Four Rules below.)
• It's stdlib-only. Three small Python scripts, no pip install, no dependencies beyond Python 3 and a running Ollama.
• It measures three things per model: quality (a 0.0-10.0 LLM-judged rubric score), speed (generation tok/s, prefill tok/s, load time), and fit (VRAM-resident vs. spilled-to-RAM).

---

The Four Rules (the "lock")

The whole method rests on four rules that keep comparisons valid across models, machines, and time. They're spelled out in CHARTER.md; in brief:

1. Immutable tests. The prompt batteries, the harness, and the rubric are frozen once they've scored anything. Fixes and additions ship as a new version directory, never an in-place edit.
2. Pinned grader. Identical prompts graded two different ways are two different benchmarks. Pick one judge model by calibration, pin it (and its seed), and use it for everything you compare, forever.
3. Quarterly cadence. Every 3-4 months, pull the newest releases, run the frozen suite, grade with the pinned grader, and append a dated row to your results ledger. Append-only; you never overwrite a past cycle.
4. Backfill new axes. A new test isn't active until it's been run against the full reference panel, so a new test never arrives with a blank past.

---

Quickstart

Requirements: Python 3 (stdlib only, nothing to install) and a running Ollama. That's the whole dependency list. The scripts target a single consumer GPU (e.g. an RTX 3060 12 GB) but work against any Ollama endpoint via OLLAMA_BASE_URL (default http://localhost:11434).

# 1. Pull a couple of models to compare (any tags Ollama has)
ollama pull qwen2.5:7b
ollama pull gemma4:12b

# 2. Run the whole GENERATION suite (VRAM-fit check + every battery in prompts/)
bash run_suite.sh "qwen2.5:7b,gemma4:12b" ./results

# 3. Grade the raw outputs with a FIXED judge model + seed
python3 scripts/grade.py --report "results/report_*.md" \
  --grader-model qwen2.5:14b --out results/grades.csv

On Windows without bash, call the scripts directly:

python scripts\spillover.py  --models "qwen2.5:7b,gemma4:12b" --out results
python scripts\model_eval.py --prompts prompts\example_battery.md --models "qwen2.5:7b,gemma4:12b" --out results --tag example_battery
python scripts\grade.py      --report "results\report_*.md" --grader-model qwen2.5:14b --out results\grades.csv

What's in the box

Path	What it is
CHARTER.md	the method: the funnel philosophy, the Four Rules, role groups, how to pick + pin a grader, the reference panel, the reproducibility contract
RUBRIC.md	the 0.0-10.0 decimal grading rubric + per-axis anchors + the think:false/think:true protocol
MODELS.md	a suggested model roster + ollama pull commands (with illustrative reference numbers)
prompts/example_battery.md	a neutral 12-prompt starter battery: a starting point, not gospel
scripts/model_eval.py	the core runner: sends each prompt to each model, captures raw answers + speed
scripts/spillover.py	the "does it fit on my GPU" check: VRAM-resident vs. RAM-spilled per model
scripts/grade.py	grades a report on the 0-10 rubric using a pinned, seeded LLM judge
run_suite.sh	one-shot driver: spillover check, then every *.md battery in prompts/

Reading the results

run_suite.sh and the two Python runners write into your --out directory (default ./results, git-ignored):

• report_<tag>_<timestamp>.md, human-readable: every model's raw answer to each prompt, laid out side-by-side under the prompt, with each answer's tok/s. Any reasoning trace is tucked in a collapsible <details> block so it doesn't pollute the graded answer.
• metrics_<tag>_<timestamp>.csv: generation tok/s, prefill tok/s, load time, and token count per (model, prompt).
• spillover.csv: total size vs. VRAM-resident size per model, so you can see what actually fits on your card and what spilled into system RAM (and therefore crawls).
• grades.csv: the 0-10 scores per (model, prompt); grade.py also prints a per-model average, sorted best-first, right in the terminal.

The example battery spans twelve prompts across faithfulness (answer only from the source), summ arization, structured/JSON extraction, plain-list extraction, multi-step reasoning, and coding, plus two harder discriminator prompts (a constraint-scheduling puzzle and a long-context caveat-spotting task) that separate the finalists once the easy prompts saturate near the top.

---

Make it yours

This is the important part. The bundled battery is a demonstration of the format, not a canonical test. Its prompts don't measure whatever you care about. The kit is built to be forked:

1. Fork it, then write your own battery. Copy prompts/example_battery.md, replace the prompts with tasks from your actual work, and keep the format the runner parses:

   ### <id> | <axis>
   <the prompt text sent to the model (one or many lines)>
   > expected: <a note describing a 10/10 answer; NOT sent to the model, only to the grader>
   

   The > expected: line is the answer key: a short, explicit statement of what a correct response must contain, authored when you freeze the prompt. Write it carefully; it's what the grader scores against.

2. Freeze it (Rule 1). The moment a battery has scored a model, it's immutable. Want to fix a prompt or add an axis? That's a new version: copy the directory to benchmark-v2/, change it there, and re-baseline. Editing a scored test in place silently invalidates every number you've ever recorded against it. (A CHECKSUMS.txt over the frozen files, verified before each run, is the cheap way to catch accidental drift.)

3. Pin your own grader (Rule 2). Don't pick the judge by reputation. Pick it by calibration. Have a few candidate judges grade a small reference sample, keep the one that best reproduces a trusted reference and is most self-consistent, and then never change it. CHARTER.md walks through this with a worked example.

Everything else (the role groups, the reference panel, the quarterly cadence) is guidance you adapt. The only hard constraints are the Four Rules.

---

Gotchas baked into the harness (hard-won)

These are the failure modes that quietly corrupt a bake-off. The scripts already account for them; they're called out here so you don't reintroduce them when you fork.

• An output budget that's too small fakes a bad score. num_predict is a cap, not a target; a model still stops at EOS when it's done, but if the cap is stingy a long or thinking answer gets guillotined mid-sentence and grades as a failure it didn't earn. model_eval.py defaults to a generous --num-predict 6144 inside a --num-ctx 16384 window so honest answers finish. Keep it generous; don't tune it down to save a few seconds.
• temperature 0 is not determinism. GPU inference has residual nondeterminism even at temp 0, so a score captured once is a snapshot, not a reproducible baseline. Two defenses ship in the kit: the grader is seeded (GRADER_SEED, default 42) so re-grading the same report reproduces the same numbers; and the intended generation protocol is to run multiple fresh rounds and average (fresh generations, not reruns of one cached output) so the jitter averages out.
• The grader is part of the measurement: grade the answer, not the reasoning trace. A judge that reads a model's scratchpad rewards models for thinking out loud rather than for the deliverable. grade.py strips <details>…</details> and <think>…</think> before scoring, so only the final answer is judged. It also treats a grader-side JSON parse failure as a grader error, not as a model score of 0.0. A broken judge shouldn't masquerade as a broken model.
• Timing means nothing on a shared GPU. tok/s is only comparable if the card is otherwise idle when you measure it. Both runners load one model at a time and unload it (keep_alive: 0) before moving on, so models don't stack in VRAM and skew each other's numbers, but that only holds if you aren't running something else on the GPU during the suite. Run it on an idle card.

---

Comparing two machines

Quality is a property of the model, so it should land within noise on any box. The deltas that matter across machines are speed and VRAM fit, which is exactly the hardware story you want. The one thing you must hold fixed: use the same pinned grader (and seed) for both machines, or grade both machines' reports centrally on one box. If machine A is graded by one judge and machine B by another, you've measured the graders, not the machines.

---

License

MIT. See LICENSE. Copyright (c) 2026 Merrymak3r. Fork it, freeze your own battery, and run your own bake-off.