Skip to content

bench: Granite 4.0 H-Small vs gpt-oss-20B benchmarks (CPU + GPU, agentic + context scaling) - #136

Merged
bernardladenthin merged 6 commits into
mainfrom
claude_20260702_00
Jul 8, 2026
Merged

bench: Granite 4.0 H-Small vs gpt-oss-20B benchmarks (CPU + GPU, agentic + context scaling)#136
bernardladenthin merged 6 commits into
mainfrom
claude_20260702_00

Conversation

@bernardladenthin

@bernardladenthin bernardladenthin commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds a Granite 4.0 H-Small (UD-Q4_K_XL) vs gpt-oss-20B head-to-head benchmark (CPU + GPU) answering whether one Granite hybrid model can replace the current two-model setup: quality is on par, but H-Small is ~4x slower on CPU decode (docs/ai-index-benchmark/HSMALL-VS-GPTOSS.md), plus the EXP-Granite-4.0-H-Small aiDefinition used to produce it.
  • Adds a context-scaling sweep showing the gpt-oss vs H-Small prefill crossover (~5,000 input tokens), informing the plugin's size-tiered routing sweet spot.
  • Adds an agentic tool-calling reaction-time benchmark (CPU) plus H-Tiny vs gpt-oss context curves (docs/ai-index-benchmark/AGENTIC-CPU-BENCH.md).
  • Adds GPU (non-CUDA) agentic curves: OpenCL is unusable on the test NVIDIA card (falls back to CPU), Vulkan works at ~CUDA speed.
  • Switches the two benchmark chart images + their plot scripts to English labels (no data changes).
  • Includes the reusable harness scripts/TSVs/plot scripts under docs/ai-index-benchmark/tools/.

All commits are documentation/benchmark-data additions; no changes to plugin source code or its runtime behavior.

Test plan

  • Benchmark-only change; no plugin source touched
  • CI is green on this branch

Related issues / PRs

Checklist

  • My commits follow Conventional Commits
  • No security-sensitive changes

Generated by Claude Code

bernardladenthin and others added 5 commits July 3, 2026 02:15
@bernardladenthin @claude
bench: Granite 4.0 H-Small (UD-Q4_K_XL) vs gpt-oss-20B head-to-head (…
6633aad 
…CPU + GPU)

Adds a focused benchmark answering "can one Granite hybrid replace the two-model
setup?" on the current 24-file config+provider tree (prompt v1, 16k ctx), RTX 3070
8GB + Ryzen 7 5800H, llama 5.0.4.

Findings (docs/ai-index-benchmark/HSMALL-VS-GPTOSS.md):
- Quality: H-Small is a genuine peer to gpt-oss (0 fence / 0 leak both), and leaner
  (config summary 108 vs 168 lines).
- Speed: the "Granite is faster" assumption is FALSE for H-Small on this workload -
  ~9B active vs gpt-oss ~3.6B active -> CPU decode 5 vs 22 tok/s (~4x slower).
  Decode dominates summary generation, so H-Small is slower overall.
- Mamba win is real but narrow: on GPU H-Small prefill is fast + FLAT with context
  (215 tok/s) while gpt-oss attention prefill degrades (154->73); KV/state memory is
  flat (4 of 40 layers grow KV). Helps large inputs/long context, not decode.
- 8GB GPU: neither model fits -> partial offload accelerates prefill but not decode.
- Verdict: H-Small = quality-parity + RAM-light single model, NOT the fast option
  (that stays H-Tiny, 1B active).

Includes: both raw .ai.md output trees, EXP-Granite-4.0-H-Small aiDefinition in the
ai-index-selftest profile (greedy, no thinking - IBM/Unsloth recommended), and a
reusable parameterized calibrate POM (tools/calibrate-bench-pom.xml).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
@bernardladenthin @claude
bench: context-scaling sweep - gpt-oss vs H-Small prefill crossover
00b9aa9 
GPU prefill-vs-input sweep (2k/4k/8k/16k contexts). gpt-oss prefills far faster on
small inputs (~600 tok/s) then COLLAPSES past ~2.5k tokens (O(L^2) attention);
H-Small is flat (~205, Mamba). Prefill crossover ~5,000 input tokens (~15-20 KB).
Total summarization time crosses at ~the same point because decode time per summary
is roughly equal (gpt-oss reasoning_effort=low emits ~4x more tokens, cancelling its
4x faster decode rate). Sweet spot = the plugin's size-tiered routing: cap gpt-oss's
input at ~15 KB, route larger files to a hybrid. Raw data in tools/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
@bernardladenthin @claude
bench: agentic reaction-time (CPU) + h-tiny vs gpt-oss context curves
fc08e9a 
Adds an agentic tool-calling reaction-time benchmark via llama-server --jinja:
- Only granite-h-tiny and gpt-oss emit native tool_calls; Qwen2.5-Coder emits
  them as text, DeepSeek-Coder-7B-v1.5 not at all.
- CPU reaction time (TTFT) vs context for the two agentic-capable models: h-tiny
  wins at every context and the lead GROWS (1.8x -> 2.1x at 16k), because its
  hybrid prefill rises with context (129->366 tok/s) while gpt-oss attention
  prefill peaks then degrades. No speed crossover.
- Chart htiny-vs-gptoss-cpu-context.png (2 panels: TTFT + prefill vs context).
- Fact-check: big dense coders (14B/32B) are async-only on CPU (45-113s even at
  low context); "20-30 tok/s on CPU" claims don't hold (~2-3 tok/s measured).

Includes reusable harness (tools/agentic-*.sh + parse-*.py), plot script, and raw
TSVs. AGENTIC-CPU-BENCH.md documents it; README links it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
@bernardladenthin @claude
bench: GPU (non-CUDA) agentic curves - OpenCL dead on NVIDIA, Vulkan …
d2b220b 
…works

OpenCL is not usable on the RTX 3070: llama.cpp's OpenCL backend detects the card
but drops it ("unsupported GPU") and falls back to CPU (OpenCL targets Adreno/
Qualcomm). Vulkan is the working vendor-neutral non-CUDA path and runs at ~CUDA speed.

CPU-vs-Vulkan context curves (htiny-vs-gptoss-cpu-vs-vulkan.png):
- granite-h-tiny fully fits the 8GB GPU -> TTFT stays <5.2s even at 16k context
  (~3100 tok/s prefill) = the only interactive combo. On CPU it is 44s.
- gpt-oss (11GB) does NOT fit 8GB -> only 6/24 layers offload -> barely faster than
  CPU (80.7s vs 93s at 16k); stays CPU-bound.

Adds the chart, Vulkan raw TSVs, the NGL/DEVICE-parameterized harness, and the
CPU-vs-GPU plot script; AGENTIC-CPU-BENCH.md section 3b documents it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
@bernardladenthin @claude
bench: switch benchmark charts + plot scripts to English labels
b8459b2 
Regenerated htiny-vs-gptoss-cpu-context.png and htiny-vs-gptoss-cpu-vs-vulkan.png
with English titles/axes/legends; updated the plot scripts accordingly. No data
changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
@claude
Fix REUSE compliance: add SPDX headers/license sidecars to benchmark …
3b639ba 
…files

Adds SPDX-FileCopyrightText/SPDX-License-Identifier headers to the benchmark
shell/Python scripts and the calibrate-bench pom, and .license sidecar files
for the two PNG charts and five TSV data files (inline headers would corrupt
their format). reuse lint now reports 684/684 files compliant.
@bernardladenthin
bernardladenthin merged commit 22b5162 into main Jul 8, 2026
8 of 23 checks passed
@bernardladenthin
bernardladenthin deleted the claude_20260702_00 branch July 8, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bernardladenthin @claude