A mixed-capability benchmark for evaluating LLMs on NVIDIA DGX Spark (GB10 Grace-Blackwell), tuned for agentic / Hermes-style local serving: tool use, multi-turn workflows, executable code, and real cluster serving (vLLM, llama.cpp, multi-node).
Current methodology: v6.4c · 74 scenarios · 12 domains · 12 agentic workflows
| Scenarios | 74 (base + hard + expert agentic) |
| Domains | 12 |
| Agentic | 12 multi-turn workflows (AG-01…AG-12), including deep chains, tool failure injection, nested schemas, long noisy context |
| TrueScore weights | Quality 55% · Calibration 25% · Reliability 15% · Efficiency 1.5% · Responsiveness 3.5% (speed total 5%) |
| Integrity | Golden-gate grader self-test, endpoint model + tool-call preflight, grader git provenance, one-eval-per-box lock, run heartbeats |
| Output | Long-format CSV (results/spark_bench.csv) + per-run markdown/HTML; live board at wesche.com/dgx |
Scope: This is a DGX Spark operator bench, not a universal LMSYS-style ranking. It answers: which model + recipe should I serve on this cluster for agentic / Hermes work?
Authoritative numbers live on the auto leaderboard (wesche.com/dgx), rendered from results/spark_bench.csv. Do not treat the tables below as current v6.4c scores — they are a historical v5c snapshot (64 scenarios, think-OFF, Q4_K_M era) kept for context.
Models are grouped by size tier. Within each tier, models compete against peers of similar capacity.
TrueScore weights (unchanged through v5c → v6.4c): Quality 55%, Calibration 25%, Reliability 15%, Efficiency 1.5%, Responsiveness 3.5% (speed total: 5%)
Calibration measures prompt injection resistance, robustness, and over-refusal of legitimate requests. Content-refusal scenarios (harmful content requests) are informational only — 0% weight — because for uncensored models, answering everything is a feature.
Agentic (v6.4c): 12 multi-turn scenarios. Expert tier (AG-07…AG-12) adds dependency chains, injected tool failures, nested parameter schemas, and facts buried in long ops briefings. Graders check tool sequences and final-answer correctness.
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemma 4 E4B (4B dense) | 75.9 | 60.9 | 94.0 | 94.9 | 100 | 91.7 | 3/36 ⭐ | llama.cpp, Q4_K_M |
| 2 | Gemma 4 E2B (2B dense) | 75.9 | 64.8 | 86.6 | 91.9 | 100 | 94.0 | 2/36 | llama.cpp, Q4_K_M |
| 3 | Gemma 4 12B (dense) | 74.9 | 63.8 | 81.9 | 98.6 | 100 | 86.5 | 1/36 | llama.cpp, Q4_K_M |
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwopus 27B (Qwen finetune) | 78.5 | 65.3 | 96.7 | 96.3 | 100 | 70.8 | 1/36 | llama.cpp, Q4_K_M |
| 2 | Qwen 35B base (MoE, 3B active) | 78.0 | 64.0 | 94.0 | 97.3 | 100 | 92.1 | 3/36 | llama.cpp, Q4_K_M |
| 3 | Gemma 4 31B (dense) | 77.4 | 64.3 | 93.5 | 99.6 | 100 | 64.5 | 1/36 | llama.cpp, Q4_K_M |
| 4 | Nemotron 30B (MoE, 3B active) | 76.0 | 59.9 | 94.0 | 99.0 | 100 | 91.7 | 2/36 | llama.cpp, Q4_K_M |
| 5 | Bytkim 27B (Qwen finetune) | 75.3 | 58.8 | 94.0 | 98.1 | 100 | 91.8 | 3/36 | llama.cpp, Q4_K_M |
| 6 | Qwen 27B base (dense) | 75.2 | 65.8 | 81.9 | 97.8 | 100 | 68.0 | 1/36 | llama.cpp, Q4_K_M |
| 7 | Gemma 26B-A4B (MoE, 4B active) | 75.2 | 59.5 | 94.0 | 95.2 | 100 | 91.3 | 2/36 | llama.cpp, Q4_K_M |
| 8 | Huihui 35B (MoE, abliterated) | 75.1 | 59.1 | 94.0 | 96.1 | 100 | 91.7 | 2/36 | llama.cpp, Q4_K_M |
| 9 | Qwable 27B (Coder finetune) | 74.3 | 64.3 | 81.9 | 97.3 | 100 | 68.3 | 1/36 | llama.cpp, Q4_K_M |
| 10 | Ornith 35B (MoE, 3B active) | 73.4 | 58.2 | 88.2 | 97.3 | 100 | 91.4 | 2/36 | llama.cpp, Q4_K_M |
| 11 | AEON 7 (NVFP4, DFlash) | 70.7 | 63.2 | 67.7 | 96.3 | 100 | 87.5 | 3/36 | vLLM Docker, NVFP4 |
| 12 | HauhauCS 35B (MoE, uncensored) | 68.9 | 65.3 | 54.3 | 97.8 | 100 | 93.2 | 2/36 | llama.cpp, Q4_K_M |
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepSeek V4 Flash (MoE, dual-node) | 78.9 | 66.6 | 100.0 | 93.5 | 33.6 | 77.2 | 6/36 ⭐ | vLLM, TP=2, RoCE |
For current v6.x / NVFP4 / MTP / DFlash / multi-engine numbers, use the live board and spark_bench.csv (every row has a run_id).
Building in public means admitting mistakes and correcting them:
- 6 agentic scenarios — multi-turn workflows; models must chain tool calls.
- Calibration rewrite — content refusal → informational (0% weight).
- Speed weight cut to 5% so speed cannot wash out quality.
- One model per Spark — multi-tenant GPU runs contaminated speed.
- Token tracking + serving throughput side sweep (non-scoring
tier2rows).
- v6.1 — Safety split: content refusal informational; security behaviour (injection, destructive-cmd refusal) scores as calibration. Reliability no longer gives 100% for consistent failure.
- v6.3 — Standard tool protocol in the agentic harness (
role:tool+ tool_calls retained). - v6.4 — Expert agentic tier (AG-07…AG-12): deep dependency chains, deterministic tool failure injection + retry, nested schemas, long noisy context.
- v6.4b — Structured domain no longer flat-1.0 (value + extra-key checks); agentic difficulty rebalanced.
- v6.4c — Final-answer correctness gates on agentic; harder executable code (CODE-11…14); agentic efficiency bonus removed (it rewarded fast-but-wrong).
- Integrity gates — golden-gate grader self-test, endpoint preflight (right model + parseable tool_calls), grader git provenance (dirty tree aborts), box locks, on-disk run markers (not
pgrep).
Comparable leaderboard rows should stamp methodology v6.4c (and thinking mode, quant, engine, topology on the label/notes).
Forward vLLM policy: new vLLM-backed recipes should start from the newest validated stable vLLM at setup time (record image/wheel/tag/commit in the recipe). Existing leaderboard runs are not re-run for engine-only upgrades unless you intentionally rebenchmark.
See recipes/ for published DGX Spark recipes (DeepSeek V4 Flash, Qwen NVFP4 + MTP/DFlash, Laguna XS, MiniMax, Step, etc.).
| Class | Typical serving | Hardware |
|---|---|---|
| Large MoE / multi-node | vLLM, TP/PP, RoCE | 2–4× DGX Spark |
| NVFP4 + spec decode | vLLM (stock or forks), MTP / DFlash | 1–2× Spark |
| Dense / mid GGUF | llama.cpp, Q4_K_M | 1× Spark |
Counts from the live eval_suite.SCENARIOS bank:
| Domain | Scenarios | Description |
|---|---|---|
| agentic | 12 | Multi-turn workflows (AG-01…AG-06 hard + AG-07…AG-12 expert) |
| code | 14 | Executable Python/SQL graded by running tests (incl. harder CODE-11…14) |
| safety | 11 | Content refusal (informational) + security calibration |
| instruction | 9 | Hard instruction following (constraints, format) |
| tool_use | 6 | Function calling with real tool schemas |
| planning | 5 | Multi-step planning with tool orchestration |
| structured | 5 | JSON / schema validation (value-aware) |
| robustness | 4 | Missing params, malformed input, injection |
| visual | 3 | Animated HTML5 canvas artifacts |
| long_context | 2 | Needle / retrieval under noise |
| composition | 2 | Multi-skill composition |
| classification | 1 | Categorization with structured output |
Difficulty tiers: base (18) · hard (50) · expert (6 agentic).
- Partial credit — 3/5 constraints → 0.60, not 0.00
- Executable code — model Python/SQL run in a process sandbox against tests
- Pixel-based visual grading — HTML artifacts scored by render metrics
- Multi-turn tool calls — full tool sequence + env state, not first response only
- Agentic harness — multi-turn loop with simulated tools; expert scenarios inject failures and bury facts in long briefings; final-answer gates apply
- Consistency — repeated trials; reliability excludes all-error / never-pass rows
- Trial stats — Pass@1, Pass@K, Reliability Gap, score stddev
- Run validity — if >5% of scenarios are transport/timeout errors, the run is marked invalid
TrueScore = 0.55·Quality + 0.25·Calibration + 0.15·Reliability + 0.015·Efficiency + 0.035·Responsiveness
| Component | Weight | What it measures |
|---|---|---|
| Quality | 55% | Task correctness (tools, code, agentic, instruction, structured, …) |
| Calibration | 25% | Injection resistance, robustness, security-relevant refusal / over-refusal (content refusal = 0%) |
| Reliability | 15% | Consistency across repeats (only non-error scenarios that pass at least once) |
| Efficiency | 1.5% | Useful-token ratio (answer vs answer+reasoning) — anti-overthink, not serving tok/s |
| Responsiveness | 3.5% | Latency-derived score from median scenario wall time |
Speed-related components total 5%. Quality dominates. Serving throughput (tok/s, concurrency, context sweep) is recorded separately as non-scoring tier2 metrics under the same
run_id.
| Tier | Range | Hardware | Rationale |
|---|---|---|---|
| Small | ≤12B params | 1× DGX Spark | Lightweight / edge |
| Mid | 13–35B params | 1× DGX Spark | Full single-node |
| Large | 36B+ or 2+ Sparks | 2–4× DGX Spark | Multi-node deployment |
Every full eval (unless explicitly skipped for debug):
- Golden gate — grader reproduces known scores on canned transcripts (
golden_gate.py) - Provenance — grader git short hash; uncommitted grader files abort (or
--allow-dirty) - Preflight — endpoint serves the requested model and emits parseable structured
tool_calls - Box lock — one eval per endpoint host
- Run markers —
.STARTED/.DONEheartbeats (watchers must not usepgrepon job names)
# Full graded eval against any OpenAI-compatible endpoint
python3 spark_bench.py eval \
--label my-model-thinkoff \
--endpoint http://localhost:8000/v1 \
--model my-model \
--thinking off --repeats 2 --temperature 0.3 --tier all \
--notes "clean run, single Spark, v6.4c"Full eval runs automatically append the serving throughput sweep (--throughput-contexts 1024,8192,32768, concurrency 1,2,4,8, gen tokens 512). Use --skip-throughput only for quick/debug runs that should not be compared as full board rows.
Other entry points: tier1 (RoCE interconnect), tier2 (inference throughput), tier3 (light workloads), all (tier2+tier3).
# Offline: prove the grader still works
python3 golden_gate.py
python3 eval_suite.py --selftest- 4× NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified memory each)
- Small/Mid: often llama.cpp (Q4_K_M) or single-node vLLM NVFP4 — one model per Spark for comparable speed
- Large: vLLM tensor/pipeline parallel across Sparks (RoCE)
- Pure stdlib Python for the core harness — no pip required for
spark_bench.py/eval_suite.py
| Path | Role |
|---|---|
spark_bench.py |
CLI harness (tier1/2/3 + eval) |
eval_suite.py |
Scenario bank + graders + TrueScore (v6.4c) |
golden_gate.py |
Offline grader self-check |
html_report.py / render_leaderboard.py |
Per-run HTML + board render |
visual_pixel_grader.py |
Visual domain grading helper |
recipes/ |
Published serve recipes for DGX Spark |
tests/ |
Mock OpenAI server for gate tests |
results/spark_bench.csv |
Long-format score source of truth |
results/runs/ |
Per-run markdown/HTML + markers |
results/artifacts/ |
Generated code/visual artifacts |
archive/ |
One-off drivers, probes, lab scripts (not needed for a normal eval) — see archive/README.md |
MIT licensed. Methodology inspired by tool-eval-bench.
Live leaderboard: wesche.com/dgx