🏆 Lite Benchmarks

Personal LLM Leaderboard & Benchmark Studio

Small, repeatable samples of established benchmarks with 100% programmatic scoring. No LLM-as-judge. Deterministic sampling. Sandboxed code execution.

Warning

Project Status: 🟡 In Development / Work in Progress This repository is currently under active development and may contain experimental, partial, or non-working features.

🌐 Live Leaderboard

lite-benchmarks.netlify.app - interactive leaderboard, charts, and per-benchmark breakdowns, auto-rebuilt from this repo on every push.

✨ Why Lite Benchmarks?

🎯 Programmatic Scoring Every answer is verified by code - regex extraction, unit test execution, exact match. Zero LLM-as-judge bias.	🔒 3-Layer Sandboxed Execution AST scan → hardened subprocess → Windows Job Object. Model code runs isolated with no API keys, no network, no escape.
🌐 Web Dashboard Select models, pick benchmarks, run, and generate reports - all from a local browser UI. No CLI wrangling.	🎲 Deterministic Sampling Fixed seed (42) random sampling means the exact same questions every run. Reproducible by design.
📊 Statistical Rigor Wilson score confidence intervals on every benchmark score. Know exactly how much noise is in the numbers.	💰 Cost & Token Tracking Per-model token breakdown (input/output/thinking), throughput (TPS), latency, and estimated API cost.

⚡ Quick Start

pip install -e .[dev] # install dependencies
py web_app.py # launch dashboard → http://127.0.0.1:8000

Then select models, pick benchmarks, hit Run Benchmarks, and Generate Reports.

📑 Table of Contents

• Live Leaderboard
• Benchmarks
• Leaderboard
• Charts
• Token Usage & Performance
• Architecture
• Methodology
• How to Run
• Adding Models
• Project Structure

📝 Benchmarks

12 established benchmarks, ~50 questions each, grouped into 5 categories:

Benchmark	Category	Full Dataset	Sampled	Verification	Source
BigCodeBench-Hard	Coding	148	50	Python unittest execution (explicit opt-in required)	bigcode/bigcodebench-hard (v0.1.4)
HumanEval+	Coding	164	50	Python test execution (explicit opt-in required)	evalplus/humanevalplus
MBPP+	Coding	378	50	Python test execution (explicit opt-in required)	evalplus/mbppplus
GPQA Diamond	Science	198	50	Multiple choice (4 options)	nichenshun/gpqa_diamond (community mirror of Idavidrein/gpqa)
SciBench	Science	692	50	Numerical / Formula exact match	xw27/scibench
AIME 2024/2025	Math	90	50	Integer exact match (000-999)	AI-MO/aimo-validation-aime
MATH-500	Math	500	50	Exact match / \boxed{} extraction	HuggingFaceH4/MATH-500
MMLU-Pro	Knowledge	12,032	50	Multiple choice (10 options)	TIGER-Lab/MMLU-Pro
IFEval	Instruction	541	50	25 programmatic verifiers (strict)	google/IFEval
SciCode	Coding	65	50	Python code execution & unit test assertions	SciCode1/SciCode
SuperGPQA	Knowledge	26,529 (7,050 hard)	50	Multiple choice (up to 10 options)	m-a-p/SuperGPQA
Tau-Bench (Retail)	Instruction	82	50	Agentic tool-call function & argument matching	amityco/tau-bench-retail-train-next-action

🏅 Leaderboard

Rank	Model	Overall	💻 Coding	🔬 Science	📐 Math	📚 Knowledge	📋 Instruction
🥇	Gemini 3.7 Flash ⚡	73.9%	48.5%	84.0%	97.0%	82.0%	58.0%
🥈	Gemini 3.5 Flash Lite ⚡	67.6%	46.0%	77.0%	88.0%	74.0%	53.0%
🥉	Mimo v2.5 ⚡	62.0%	62.0%	N/A	N/A	N/A	N/A
4	DeepSeek v4 Flash 🧠	56.8%	42.0%	52.0%	77.0%	68.0%	45.0%
5	Qwen 3.8 Flash 🧠	55.3%	40.5%	58.0%	93.0%	41.0%	44.0%
6	DeepSeek v4 Flash Vision Exp 🧠	54.8%	37.0%	70.0%	61.0%	63.0%	43.0%
7	HY3 🧠	44.9%	51.3%	69.0%	88.0%	16.0%	0.0%
8	Laguna S 2.1	N/A	N/A	N/A	N/A	N/A	N/A
9	MiniMax M3	N/A	N/A	N/A	N/A	N/A	N/A
10	Qwen 3.8 Max Free	N/A	N/A	N/A	N/A	N/A	N/A

🧠 thinking observed (reasoning tokens recorded). ⚡ no-thinking: the run recorded zero thinking tokens - the score reflects answers alone (thinking was requested at max, but the endpoint returned no reasoning output).

Per-Benchmark Scores

Model	BigCodeBench-Hard	HumanEval+	MBPP+	GPQA Diamond	SciBench	AIME 2024/2025	MATH-500	MMLU-Pro	IFEval	SciCode	SuperGPQA	Tau-Bench (Retail)
Gemini 3.7 Flash	6% (3.0/50) ±7.1pp	100% (50.0/50) ±3.6pp	88% (44.0/50) ±9.1pp	100% (50.0/50) ±3.6pp	68% (34.0/50) ±12.5pp	98% (49.0/50) ±5.1pp	96% (48.0/50) ±6.2pp	90% (45.0/50) ±8.5pp	92% (46.0/50) ±7.8pp	0% (0.0/50) ±3.6pp	74% (37.0/50) ±11.8pp	24% (12.0/50) ±11.6pp
Gemini 3.5 Flash Lite	4% (2.0/50) ±6.2pp	100% (50.0/50) ±3.6pp	80% (40.0/50) ±10.9pp	88% (44.0/50) ±9.1pp	66% (33.0/50) ±12.7pp	80% (40.0/50) ±10.9pp	96% (48.0/50) ±6.2pp	90% (45.0/50) ±8.5pp	92% (46.0/50) ±7.8pp	0% (0.0/50) ±3.6pp	58% (29.0/50) ±13.2pp	14% (7.0/50) ±9.6pp
Mimo v2.5	12% (6.0/50) ±9.1pp	98% (49.0/50) ±5.1pp	76% (38.0/50) ±11.6pp	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
DeepSeek v4 Flash	2% (1.0/50) ±5.1pp	86% (43.0/50) ±9.6pp	80% (40.0/50) ±10.9pp	68% (34.0/50) ±12.5pp	36% (18.0/50) ±12.9pp	78% (39.0/50) ±11.2pp	76% (38.0/50) ±11.6pp	78% (39.0/50) ±11.2pp	78% (39.0/50) ±11.2pp	0% (0.0/50) ±3.6pp	58% (29.0/50) ±13.2pp	12% (6.0/50) ±9.1pp
Qwen 3.8 Flash	2% (1.0/50) ±5.1pp	90% (45.0/50) ±8.5pp	70% (35.0/50) ±12.3pp	64% (32.0/50) ±12.9pp	52% (26.0/50) ±13.3pp	94% (47.0/50) ±7.1pp	92% (46.0/50) ±7.8pp	82% (41.0/50) ±10.5pp	88% (44.0/50) ±9.1pp	0% (0.0/50) ±3.6pp	0% (0/50) ±3.6pp	0% (0/50) ±3.6pp
DeepSeek v4 Flash Vision Exp	4% (2.0/50) ±6.2pp	68% (34.0/50) ±12.5pp	76% (38.0/50) ±11.6pp	92% (46.0/50) ±7.8pp	48% (24.0/50) ±13.3pp	48% (24.0/50) ±13.3pp	74% (37.0/50) ±11.8pp	64% (32.0/50) ±12.9pp	76% (38.0/50) ±11.6pp	0% (0.0/50) ±3.6pp	62% (31.0/50) ±13.0pp	10% (5.0/50) ±8.5pp
HY3	2% (1.0/50) ±5.1pp	72% (36.0/50) ±12.1pp	80% (40.0/50) ±10.9pp	86% (43.0/50) ±9.6pp	52% (26.0/50) ±13.3pp	82% (41.0/50) ±10.5pp	94% (47.0/50) ±7.1pp	16% (8.0/50) ±10.1pp	0% (0/50) ±3.6pp	N/A	N/A	N/A
Laguna S 2.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
MiniMax M3	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Qwen 3.8 Max Free	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A

±pp indicates 95% Wilson score confidence interval half-width.

📊 Charts

Overall Scores

Horizontal bar chart ranked by overall score (average of all category scores).

Category Breakdown

Grouped bar chart comparing each model across the 5 categories.

Category Radar

Spider chart showing each model's profile across categories. Larger area = stronger overall.

Benchmark Heatmap

Per-benchmark scores for every model. Green = high, red = low.

Token Breakdown

Stacked bar chart of input, thinking, and output tokens per model.

Thinking Effort vs Performance

Scatter plot showing if models that use more thinking tokens achieve higher overall scores.

🪙 Token Usage & Performance

Model	Input	Output	Thinking	Total	Out %	Think %	Avg TPS	Avg Time	Est. Cost
Gemini 3.7 Flash	314,999	204,193	0 ⚡	519,192	39%	-	22.7	14.5s	-
Gemini 3.5 Flash Lite	314,999	202,689	0 ⚡	517,688	39%	-	78.4	6.2s	-
Mimo v2.5	57,723	234,435	0 ⚡	292,158	80%	-	61.4	23.8s	-
DeepSeek v4 Flash	243,326	116,314	1,500,068	1,859,708	6%	81%	89.1	43.7s	-
Qwen 3.8 Flash	98,983	99,178	2,038,234	2,236,395	4%	91%	93.4	69.9s	-
DeepSeek v4 Flash Vision Exp	241,244	69,356	1,663,159	1,973,759	4%	84%	85.0	61.6s	-
HY3	53,307	54,796	1,897,742	2,005,845	3%	95%	103.7	72.3s	-
Laguna S 2.1	0	0	0	0	-	-	-	-	-
MiniMax M3	0	0	0	0	-	-	-	-	-
Qwen 3.8 Max Free	0	0	0	0	-	-	-	-	-

TPS = output tokens/second (cloud APIs only, skipped for local models). Est. Cost calculated via LiteLLM cost tables. 0 ⚡ = no thinking tokens recorded for that run (score reflects answers alone).

🏗️ Architecture

flowchart LR
 A[config.yaml] --> B[datasets.py<br/>HF sampling]
 B --> C[engine.py<br/>concurrent execution]
 C --> D[providers.py<br/>litellm calls]
 D --> E[benchmarks.py<br/>scoring & verification]
 E --> F[results_store.py<br/>schema v2 JSON]
 F --> G[charts.py<br/>matplotlib PNGs]
 F --> H[readme_gen.py<br/>this README]

 C --> I[sandbox.py<br/>3-layer isolation]
 I --> E

Loading

🔬 Methodology

📐 Sampling & Statistical Significance

• ~50 questions are sampled from each benchmark's full dataset
• Sampling uses a fixed seed (42) via random sampling so exact questions are stable across runs
• Samples of n=50 have 95% confidence intervals of roughly ±7–14pp; treat small ranking gaps as noise
• Scoring v2 Notice: Sampling and scoring strictness updated in v0.2.0; results are not directly comparable with pre-v0.2.0 runs

✅ Scoring & Verification

• All scoring in these results is programmatic - no LLM-as-judge was used
• Code benchmarks require explicit opt-in (allow_unsafe_code_execution) and run in a layered sandbox: an AST scan of generated code rejects destructive / escape constructs, the child runs with a scrubbed environment (no API keys, temp working dir), a runtime confinement shim restricts file I/O to the sandbox dir and sockets to loopback, and on Windows it is additionally confined by a Job Object that blocks grandchild processes and UI access. The opt-in gate is enforced at the sandbox layer, so it fails closed even for direct callers. Tasks whose reference solution the sandbox cannot run are filtered before sampling, so every graded task is passable.
• Multiple-choice benchmarks extract the answer letter and compare to ground truth
• Math benchmarks extract boxed/numerical answers (scientific notation included) and evaluate via normalized string, deterministic symbolic (sympy) equivalence, or numerical comparison
• IFEval uses its 25 strict programmatic verifiers (word count, format, keywords, etc.)
• Tau-Bench verifies tool function name AND argument dictionary match

📊 Category & Overall Scores

• Category score = average of its benchmark scores
• 💻 Coding = avg(BigCodeBench-Hard, HumanEval+, MBPP+, SciCode)
• 🔬 Science = avg(GPQA Diamond, SciBench)
• 📐 Math = avg(AIME 2024/2025, MATH-500)
• 📚 Knowledge = avg(MMLU-Pro, SuperGPQA)
• 📋 Instruction = avg(IFEval, Tau-Bench (Retail))
• Overall score = average of completed category scores (equal weight per category)
• Provider failures are excluded and recorded separately; scorer exceptions score 0.0 without retrying provider

⚙️ Inference Settings

• temperature: 0.0
• max_tokens: 16384
• timeout: 300s per request
• retries: transient errors retry with exponential backoff until a good response arrives (no cap); permanent errors (context length, content filter) are never retried

🚀 How to Run

Prerequisites

pip install -e .[dev]

API Keys

Set environment variables for the providers you want to test. litellm picks them up automatically.

Provider	Environment Variable	Get a key
DeepSeek	DEEPSEEK_API_KEY	platform.deepseek.com
Groq	GROQ_API_KEY	console.groq.com
Google Gemini	GEMINI_API_KEY	aistudio.google.com
LM Studio (local)	(none needed)	lmstudio.ai
HuggingFace	HF_TOKEN	huggingface.co/settings/tokens

Running Benchmarks via Web Dashboard

# Launch the local web dashboard (serves on http://127.0.0.1:8000)
py web_app.py

# Launch without automatically opening browser
py web_app.py --no-browser

1. Open the dashboard in your browser.
2. Select models, benchmarks, and settings.
3. Click Run Benchmarks.
4. Click Generate Reports to update README.md and charts/.

➕ Adding Models

Edit config.yaml or add models directly in the Web UI:

models:
- id: anthropic/claude-sonnet-4-20250514
 name: Claude Sonnet 4
 max_tokens: 16384
- id: openai/gpt-4o
 name: GPT-4o
- id: lm_studio/qwen2.5-coder-7b-instruct
 name: Qwen 2.5 Coder 7B (local)

📁 Project Structure

├── config.yaml # Models, benchmarks, categories, settings
├── web_app.py # Web dashboard server
├── windows_sandbox.py # Windows Job-object/restricted-token sandbox
├── README.md # ← this file (auto-generated)
├── web/ # Web dashboard frontend (HTML/CSS/JS)
├── lite_bench/
│ ├── engine.py # Unified execution engine & thread concurrency
│ ├── results_store.py # Results persistence, schema v2, atomic writes
│ ├── metadata.py # Benchmark display metadata & category mapping
│ ├── config.py # Config loading & validation
│ ├── providers.py # litellm wrapper & telemetry
│ ├── datasets.py # Deterministic HuggingFace sampling
│ ├── benchmarks.py # Benchmark implementations & verifiers
│ ├── ifeval_verifiers.py # 25 strict IFEval verifiers
│ ├── sandbox.py # Code-exec sandbox (AST scan + subprocess + Win job)
│ ├── charts.py # matplotlib chart generation
│ └── readme_gen.py # README generator
├── results/ # JSON results per run
│ └── latest.json # Leaderboard results (schema v2)
└── charts/ # Generated PNG charts

---

Auto-generated by lite-benchmarks on 2026-09-04 12:21 UTC · Licensed under MIT · Built with litellm + HuggingFace Datasets

⭐ Star this repo if you find it useful!