Local LLM benchmark suite

This repository contains a reproducible benchmark suite for comparing local LLM serving configurations. It was created around a Qwen3.8-27B comparison, but the runners accept any OpenAI-compatible local server and do not depend on a particular username, home directory, model filename, or network topology.

The suite separates model quality from serving performance. Every comparison should keep the model family, prompt fixtures, sampling parameters, server build, KV-cache types, context limit, and hardware fixed unless that variable is the one being studied.

The complete gated sequence is documented in docs/SEQUENCE.md: direct llama-bench baseline, SPEED baseline and draft sweep, growing-context crossover, candidate selection, capacity and boundary gates, RULER, agent coding, repeated compaction, stability, and finally concurrency.

What is included

Area	Workload	Why it exists
Raw target baseline	llama-bench prefill/decode	Establishes direct target-model behavior before speculative decoding.
Serving speed	SPEED-Bench baseline and draft sweep	Compares short qualitative serving throughput, latency, and acceptance.
Draft selection	Growing-context crossover	Selects the draft length to carry into the expensive suite.
Context quality	RULER: multi-key retrieval, variable tracking, common-word extraction, HotpotQA	Measures retrieval and reasoning as context grows from 32K to 240K.
Deterministic QA	20 frozen source-grounded questions	Separates model answering from live web search, retrieval, and embedding variability.
Agent coding	20 small Pi tasks	Tests isolated implementation and debugging behavior with unit-tested outcomes.
Agent coding	5 medium OhMyPi repositories	Tests a larger multi-file coding workflow.
Context continuity	3 frozen repeated-compaction checkpoints	Tests whether an agent can resume from a compacted state and preserve completed work.
Capacity gates	Context-boundary and maximum-capacity probes	Establishes whether the server can actually hold the intended context before quality testing.
Long-run behavior	Mixed one-hour stability workload	Looks for errors, drift, or failures across alternating short and long requests.
Load behavior	Knowledge-QA concurrency at levels 1, 2, and 4	A final tie-breaker for throughput and latency under modest parallel load.

Vision is intentionally not a scored benchmark in this suite. The requirement is that a candidate configuration can enable vision when sufficient VRAM and a compatible projector are available; vision quality is not part of this text-model comparison.

The capacity gates run before RULER. RULER is not a substitute for proving that the server can allocate and serve the target context boundary.

Quick start

The runners use only the Python standard library except for RULER/SPEED fixture generation and the external Pi/OhMyPi harnesses. Create an environment with Python 3.11 or newer, then install the optional dependencies:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-ruler.txt
python -m pip install -r requirements-speed.txt

Start an OpenAI-compatible server separately. The default endpoint is http://127.0.0.1:8080/v1/chat/completions; override it for your setup:

export BENCHMARK_ENDPOINT="http://127.0.0.1:8080/v1/chat/completions"
export BENCHMARK_MODEL="your-model-id"

All output directories are supplied by the caller. After selecting a draft setting, the post-selection suite can be started with:

./scripts/run_suite.sh results/my-run

The full pre-selection and post-selection workflow is in docs/SEQUENCE.md. Capacity probing and server startup remain explicit gates because their commands depend on the chosen server build and hardware. See docs/SETUP.md for the server contract and the individual commands.

Reproducibility rules

For a valid comparison:

1. Keep the frozen fixtures unchanged. Their hashes are written to each run's manifest.
2. Use deterministic sampling: temperature 0, top-k 1, fixed seed, and the same chat-template settings.
3. Keep prompt-cache reuse disabled unless it is the variable being tested.
4. Record the server build, target model, draft model, KV-cache types, context limit, GPU layers, and speculative-decoding settings alongside the run.
5. Treat each quantization or drafter as a new variant; do not mix raw output directories between variants.

The runners record raw per-request output locally for diagnosis. Only compact, sanitized aggregate results should be committed to a public repository.

Published example

The results directory contains the completed baseline versus DFlash2 2-bit comparison. The detailed rationale is in docs/SUITE.md, and the interpretation is in docs/RESULTS.md.

The accompanying working spreadsheet is available here: benchmark results sheet.

Scope and limitations

This is a local serving benchmark, not a leaderboard. Results are hardware-, server-build-, driver-, and configuration-dependent. The published comparison is evidence for the tested setup only; it does not claim that DFlash2 or any other quantization is universally faster or more accurate.