bench: Granite 4.0 H-Small vs gpt-oss-20B benchmarks (CPU + GPU, agentic + context scaling)

docs/ai-index-benchmark/AGENTIC-CPU-BENCH.md

@@ -0,0 +1,100 @@
+<!--
+SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Agentic reaction‑time benchmark (local models, CPU)
+
+Measures **reaction time** (time‑to‑first‑token / prefill), decode speed, and **native
+tool‑calling capability** for an agentic use case (e.g. Continue.dev), via `llama-server`
+with `--jinja` (OpenAI‑compatible `/v1/chat/completions` + `tools`). One tool is defined
+(`run_tests(module)`) and the model must emit a valid `tool_call`.
+
+- **Hardware:** AMD Ryzen 7 5800H (8 threads), CPU‑only unless noted. RTX 3070 8 GB for the GPU note.
+- **Harness:** `tools/agentic-cpu-ctx.sh` (context sweep) and `tools/agentic-bench.sh` (single‑shot),
+  parsers `tools/parse-agentic*.py`, plot `tools/plot-context-curves.py`. Raw data in `tools/*.tsv`.
+
+## 1. Which models can actually do agentic (native tool_calls)?
+
+| Model | tool_call | note |
+|---|---|---|
+| **granite‑4.0‑h‑tiny** (1B‑act hybrid) | ✅ native | + fastest |
+| **gpt‑oss‑20b** (3.6B‑act) | ✅ native | quality, but slow (reasoning) |
+| Qwen2.5‑Coder‑14B / 32B | ⚠️ **text only** | emits `<tool_call>` as prose; llama‑server didn't parse it natively here |
+| DeepSeek‑Coder‑7B‑v1.5 (Q8) | ❌ **none** | 2024 coder, no tool training |
+
+Only **h‑tiny** and **gpt‑oss** produced valid native tool calls out of the box.
+
+## 2. Reaction time vs. context (the agent loads files → context grows)
+
+CPU reaction time (TTFT) and prefill speed vs. context size for the two agentic‑capable models:
+
+![h-tiny vs gpt-oss, CPU, context](htiny-vs-gptoss-cpu-context.png)
+
+| context (tokens) | h‑tiny TTFT | gpt‑oss TTFT | h‑tiny faster by |
+|---|---|---|---|
+| ~700 | 5.7 s | 10.2 s | 1.8× |
+| ~2,200 | 8.2 s | 13.4 s | 1.6× |
+| ~4,200 | 12.8 s | 21.0 s | 1.6× |
+| ~8,200 | 23.2 s | 40.1 s | 1.7× |
+| ~12,000 | 33.4 s | 67.4 s | 2.0× |
+| ~16,000 | 43.9 s | 93.0 s | **2.1×** |
+
+**No speed crossover: h‑tiny wins at every context, and the lead *grows* as context fills.**
+Prefill speed explains it — h‑tiny (hybrid/Mamba) *rises* with context and plateaus high
+(129 → 366 tok/s); gpt‑oss (transformer) peaks (~203) then *degrades* (172 at 16k) as O(L²)
+attention bites.
+
+## 3. Full 5‑model CPU table (reaction time by context)
+
+From `tools/agentic-5models-cpu.tsv` (TTFT / decode tok/s / tool_ok):
+
+| Model | ~700 tok | ~3.3k tok | ~16k tok | decode |
+|---|---|---|---|---|
+| granite‑4.0‑h‑tiny | 6.0 s / ✅ | 19.3 s / ✅ | 46 s / ✅ | ~9 |
+| gpt‑oss‑20b | 30 s* / ✅ | 18.8 s / ✅ | 101 s / ✅ | 3–7 |
+| deepseek‑coder‑7b‑v1.5 | 18.9 s / ❌ | 69.6 s / ❌ | — | 3–5 |
+| Qwen2.5‑Coder‑14B | 32 s / ⚠️ | 108 s / ⚠️ | ctx‑err | 2.5–3 |
+| Qwen2.5‑Coder‑32B | 45 s / ⚠️ | 113 s / ⚠️ | ctx‑err | 1.6–1.9 |
+
+\* cold first request; warm prefill ~160 tok/s.
+
+## 3b. GPU backend: OpenCL vs Vulkan (non‑CUDA), and CPU‑vs‑GPU curves
+
+Goal: a **non‑CUDA** GPU path on the RTX 3070 (8 GB).
+
+- **OpenCL does NOT work on this NVIDIA GPU.** llama.cpp's OpenCL backend finds the card but rejects
+  it: `ggml_opencl: unsupported GPU 'NVIDIA GeForce RTX 3070 Laptop GPU'. drop unsupported device` →
+  `no usable GPU found, --gpu-layers ignored` (falls back to CPU). llama.cpp OpenCL targets
+  Adreno/Qualcomm; desktop NVIDIA/AMD are out.
+- **Vulkan works and is ~CUDA‑fast** (vendor‑neutral). The winget `llama-server` is a Vulkan build and
+  sees the 3070 as `Vulkan1`. Use `--device Vulkan1 -ngl <N>`.
+
+CPU vs Vulkan‑GPU, reaction time + prefill vs context, both models:
+
+![CPU vs Vulkan GPU](htiny-vs-gptoss-cpu-vs-vulkan.png)
+
+| combination | TTFT @16k | prefill |
+|---|---|---|
+| **granite‑h‑tiny · GPU (Vulkan, full)** | **5.2 s** | ~3100 tok/s |
+| granite‑h‑tiny · CPU | 43.9 s | ~366 tok/s |
+| gpt‑oss‑20b · GPU (Vulkan, 6/24 layers) | 80.7 s | ~200 tok/s |
+| gpt‑oss‑20b · CPU | 93.0 s | ~172 tok/s |
+
+**h‑tiny fully fits the 8 GB GPU → the only interactive combo** (TTFT stays < 5.2 s even at 16k).
+**gpt‑oss (11 GB) does not fit → only 6/24 layers offload → barely faster than CPU** (still 8–81 s);
+it stays CPU‑bound. So on this hardware the winning setup is unambiguous: **run the small hybrid on
+the GPU (Vulkan); big models are async/CPU only.**
+
+## 4. Takeaways for interactive/agentic use on this laptop
+
+- **The only responsive agentic model on CPU is `granite‑h‑tiny`** — smallest, hybrid, native
+  tool‑calling, fastest, and its lead grows with context.
+- **`granite‑h‑tiny` fits the 8 GB GPU (3.96 GB)** → measured **165 ms** TTFT + 146 tok/s there
+  (vs 5.7 s on CPU). So run it on the GPU: the CPU curves above are the worst case.
+- **gpt‑oss** = quality/async only (30–100 s reaction).
+- **14B/32B dense coders are async‑only on CPU** (45–113 s even at low context) — "hand it off,
+  get coffee". `RAM fitting ≠ interactive speed` — the CPU memory‑bandwidth bottleneck dominates.
+- Big‑model "20–30 tok/s on CPU" claims do **not** hold here: measured decode is ~2–3 tok/s (14B/32B).
+
+> Temporary experiment scaffold, not production config.

docs/ai-index-benchmark/HSMALL-VS-GPTOSS.md

@@ -0,0 +1,176 @@
+<!--
+SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Granite 4.0 H‑Small vs gpt‑oss‑20B — one model for everything?
+
+A focused head‑to‑head between **IBM Granite 4.0 H‑Small** (Unsloth **UD‑Q4_K_XL**, hybrid
+Mamba‑2 + sparse‑MoE, 32B total / ~9B active) and **OpenAI gpt‑oss‑20B** (UD‑Q4_K_XL, MoE,
+~3.6B active) for the plugin's code‑summarization task, on **CPU and GPU**.
+
+Motivating question: *can a single well‑configured Granite hybrid replace the two‑model setup
+(gpt‑oss for precision, H‑Tiny for speed), given Mamba's long‑context advantage?*
+
+## TL;DR verdict
+
+- **Quality: a genuine peer.** On the identical 24‑file tree, H‑Small matches gpt‑oss's
+  faithfulness and structure, with **zero** code‑fence or reasoning‑leak defects, and is
+  **noticeably leaner** (it does not pad the large config file the way gpt‑oss does).
+- **Speed: the "Granite is faster" assumption is *false* for H‑Small on this workload/hardware.**
+  H‑Small activates ~9B params/token vs gpt‑oss's ~3.6B, so its **decode is ~4× slower on CPU**
+  (5 vs 22 tok/s). Decode dominates summary *generation*, so H‑Small is slower overall.
+- **The Mamba win is real but narrow:** H‑Small's **prefill is faster and stays flat at long
+  context on GPU** (215 tok/s vs gpt‑oss 154→73), and its **KV/state memory is flat** (only 4 of
+  40 layers grow a KV cache). Those advantages help *large inputs / long context*, not the
+  decode‑bound generation of short summaries.
+- **On an 8 GB GPU neither model fits** (H‑Small ~17 GB, gpt‑oss ~12 GB), so both are
+  partial‑offload; that accelerates prefill but **not decode** → little real‑world gain for this task.
+- **They cross at ~5,000 input tokens (~15–20 KB).** gpt‑oss prefills far faster on small inputs
+  (~600 tok/s) then collapses (O(L²)); H‑Small is flat (~205) and overtakes above ~5k. This is the
+  natural **size‑tier boundary** (§3b) — route small files to gpt‑oss, large files to a hybrid.
+- **So:** H‑Small is a **quality‑parity, RAM‑light** option, **not** the fast one. The fast Granite
+  remains **H‑Tiny (1B active)**. "One model for everything" only holds if you accept gpt‑oss‑class
+  (or slower) speed in exchange for a single, permissive, long‑context‑cheap model.
+
+## Setup
+
+- **Scope:** `config` + `provider` subtree, **24 source files → 35 `.ai.md`** (24 file + 10 package
+  + 1 project), prompt **v1**, context **16384**. Both models run on the *same current tree*
+  (the older `ai__gptoss-20b__v1` cell was the stale 12‑file tree and is **not** comparable).
+- **Hardware:** AMD Ryzen 7 5800H (8 threads) · NVIDIA RTX 3070 **8 GB** · `net.ladenthin:llama`
+  5.0.4 (llama.cpp JNI), CUDA 13.3.
+- **Params (from the research pass):**
+  - H‑Small — **greedy** (`temperature=0`, `topP=1`, `topK=0`, `repeatPenalty=1`), **no thinking
+    mode** (Granite 4.0 Instruct has none). IBM+Unsloth's official recommendation.
+  - gpt‑oss — `temperature=0.7`, `topP=1`, `topK=0`, `minP=0.05`, `repeatPenalty=1`,
+    **`reasoning_effort=low`** (the right setting for summarization; corroborated by COMPARISON.md §11).
+- **Outputs:** `outputs/ai__granite4-h-small__v1/` and `outputs/ai__gptoss-20b__v1cur/`.
+
+## 1. Quality (CPU, identical 24 files)
+
+| Metric | **H‑Small UD‑Q4_K_XL** | **gpt‑oss‑20B (c16k)** |
+|---|---|---|
+| files summarized | 24 | 24 |
+| avgLn (per file `.ai.md`) | 61.2 | 62.5 |
+| KB (all 35 `.ai.md`) | 100 | 99 |
+| stray code fences | **0** | **0** |
+| reasoning/harmony leak | **0** | **0** |
+| enum lines (trivial file) | 42 | 43 |
+| **pojo lines (large config)** | **108** | **168** |
+
+Both emit all 9 sections and the identical (deterministic) `<facts>` block. The one real
+difference is verbosity on the large `AiGenerationConfig.java`: gpt‑oss elaborates to 168 lines,
+H‑Small to 108 — **H‑Small is the more concise at equal faithfulness**. Neither hallucinated the
+structural facts (those are prepended deterministically). A tighter (v2‑style) prompt would rein
+in both.
+
+## 2. Speed — CPU (16k context)
+
+| CPU | prefill tok/s | decode tok/s | source |
+|---|---|---|---|
+| gpt‑oss‑20B (~3.6B active) | **42** | **22.2** | real 24‑file run (35 gens) |
+| Granite H‑Small (~9B active) | 23 | **5.0** | calibrate (near‑window 13k) |
+
+gpt‑oss is faster on **both** axes on CPU. Decode — the tokens‑out that dominate summary
+generation — is **~4.4× faster** on gpt‑oss because it activates far fewer params per token, and
+llama.cpp's hybrid/SSM decode kernels are not yet optimized (see ggml‑org/llama.cpp#16454).
+
+## 3. Speed — GPU partial offload (8 GB, neither model fits)
+
+| GPU (partial) | layers on GPU | prefill tok/s | decode tok/s |
+|---|---|---|---|
+| Granite H‑Small | 14 / 40 | **215** (flat) | 5 |
+| gpt‑oss‑20B | 16 / 24 | 154 → **73** (degrades with ctx) | ~15–19 |
+
+- **Prefill:** GPU offload helps H‑Small a lot (23 → 215) and, crucially, its prefill **stays flat**
+  as context grows, whereas gpt‑oss's attention prefill **degrades** (154 → 73 from ~5k → ~10k
+  tokens). This is the Mamba long‑context advantage made visible — but it needs the GPU to appear.
+- **Decode:** unchanged for H‑Small (still 5 tok/s — the 26 CPU‑resident layers gate every token)
+  and no better (arguably worse) for gpt‑oss than pure CPU. **Partial offload does not help the
+  decode‑bound generation step on 8 GB.**
+
+## 3b. Context scaling — where they cross (GPU prefill sweep)
+
+Prefill throughput vs input length (GPU partial offload; raw data in
+`tools/context-scaling-sweep.tsv`):
+
+| input tokens | gpt‑oss prefill tok/s | H‑Small prefill tok/s | winner |
+|---|---|---|---|
+| ~1,000 | ~600 | ~150 | gpt‑oss (4×) |
+| ~2,200 | 592 | ~185 | gpt‑oss |
+| ~2,600 | 287 | ~198 | gpt‑oss |
+| ~4,700 | 226 | 205 | ≈ tie |
+| ~5,400 | 154 | 205 | H‑Small |
+| ~9,700 | 73 | ~212 | H‑Small (3×) |
+| ~13,000 | ~60 | 215 | H‑Small (3.5×) |
+
+- gpt‑oss starts far **ahead** at small inputs (~600 tok/s) then **collapses** past ~2.5k tokens
+  (O(L²) attention prefill). H‑Small is slow‑but‑**flat** (~205, Mamba O(L) prefill).
+- **Prefill crossover ≈ 5,000 input tokens** (~15–20 KB Java, ~250–400 lines).
+- **Total‑time crossover for summarization ≈ the same ~5k tokens:** decode time per summary is
+  roughly equal despite gpt‑oss's ~4× faster decode *rate*, because `reasoning_effort=low` still
+  emits ~4× more tokens (analysis channel) — so both spend ~110 s decoding a summary, and prefill
+  is the tie‑breaker.
+
+**Cure / sweet spot = the plugin's size‑tiered routing.** Cap gpt‑oss's input and route larger
+files to the hybrid: `maxFileSizeBytes ≈ 15000` on the gpt‑oss `<fieldGeneration>`, larger files to
+a Granite hybrid (H‑Small for quality, H‑Tiny for speed), and/or `<onOversize>` `mapReduce` to chunk.
+The "cure" for gpt‑oss's long‑input collapse is simply **not feeding it long inputs**. (Crossover
+value is quant/offload‑dependent on the 8 GB card; the *shape* — gpt‑oss high‑then‑collapsing vs
+hybrid flat — is architectural. On CPU both rates are ~10× lower and gpt‑oss stays competitive
+further out, but H‑Small's slow absolute speed makes it unattractive there regardless.)
+
+## 4. Memory / long context (research‑established)
+
+Only **4 of H‑Small's 40 layers** are attention (grow a KV cache with context); the other 36 are
+Mamba‑2 with a **fixed‑size recurrent state**. llama.cpp implements this split
+(`llama_memory_hybrid`), so H‑Small holds long context in far less RAM than a transformer whose KV
+grows linearly (gpt‑oss mitigates this only via GQA + sliding‑window). This is the durable Granite
+advantage for *very large single files* / long windows — orthogonal to the decode‑speed gap above.
+(Sources: Mamba‑2 arXiv 2405.21060; IBM Granite 4.0 announcement; llama.cpp #13550/#16454/#19264.)
+
+Caveat for this plugin: llama.cpp does **not** yet reuse a shared prompt‑prefix KV for recurrent
+models (#19264), so the per‑file `cache_prompt` optimization (COMPARISON.md §10) does **not** help
+H‑Small — each file recomputes the template prefix.
+
+## 5. Answering the goal
+
+**"Can Granite H‑Small be the single model for everything (with a good config)?"**
+
+- **Yes, on quality and RAM** — it equals gpt‑oss's faithfulness, is leaner in output, permissive
+  (Apache‑2.0), 128K‑capable, and cheap on memory at long context.
+- **No, on speed** — on CPU it is ~2× slower prefill and ~4× slower decode than gpt‑oss; on an 8 GB
+  GPU that doesn't change (decode stays CPU‑gated). It is **not** the fast option the Mamba
+  reputation suggests; that role belongs to **H‑Tiny (1B active)**.
+- **Where H‑Small would win outright:** a **≥24 GB GPU** (full offload → its flat‑state prefill and
+  memory dominate at long context), or workloads dominated by **huge inputs** rather than long
+  generation.
+
+Net: H‑Small is a strong **precision + long‑context + permissive** single model if you can accept
+gpt‑oss‑class (or slower) throughput; it is not a speed upgrade on 8 GB / CPU.
+
+## Reproduce
+
+Quality cells (both on the current tree, prompt v1, 16k):
+
+```bash
+mvn install -DskipTests            # install the plugin locally first
+mvn -o prepare-package -P ai-index-selftest -DskipTests \
+  -Dexp.model=EXP-Granite-4.0-H-Small \
+  -Dexp.filePrompt=file-body-java -Dexp.pkgPrompt=package-body \
+  -Dai.index.output.directory=docs/ai-index-benchmark/outputs/ai__granite4-h-small__v1
+mvn -o prepare-package -P ai-index-selftest -DskipTests \
+  -Dexp.model=gpt-oss-20B-c16k -Dai.gpuLayers=0 \
+  -Dexp.filePrompt=file-body-java -Dexp.pkgPrompt=package-body \
+  -Dai.index.output.directory=docs/ai-index-benchmark/outputs/ai__gptoss-20b__v1cur
+```
+
+Speed (exact prefill/decode tok/s) via a standalone calibrate POM parameterized by
+`bench.model` / `bench.ctx` / `bench.ngl` / `bench.temp` / `bench.effort`, run with
+`mvn ai-index:calibrate -f <pom>`; GPU adds `-Dbench.ngl=<N>
+-Dllama.classifier=cuda13-windows-x86-64` with the CUDA toolkit on `PATH`. The `EXP-Granite-4.0-H-Small`
+`<aiDefinition>` lives in the `ai-index-selftest` profile in `pom.xml`.
+
+> Temporary experiment scaffold (not production config), analogous to the `claude2026_06_26_00`
+> benchmark cells.

docs/ai-index-benchmark/README.md

@@ -4,6 +4,8 @@ This directory captures a controlled comparison of **local GGUF models** and **t
 versions** for the plugin's code-summarization task (`.ai.md` generation), run on CPU.
 
 - **[COMPARISON.md](COMPARISON.md)** — scored results, per-model pros/cons, source deep-dive, recommendation.
+- **[HSMALL-VS-GPTOSS.md](HSMALL-VS-GPTOSS.md)** — focused Granite 4.0 H‑Small (UD‑Q4_K_XL) vs gpt‑oss‑20B head‑to‑head, **CPU + GPU**, on the current 24‑file tree: quality parity, but H‑Small is slower (9B active) — the Mamba win is memory + long‑context prefill, not decode.
+- **[AGENTIC-CPU-BENCH.md](AGENTIC-CPU-BENCH.md)** — agentic **reaction‑time** benchmark (tool‑calling via `llama-server --jinja`): which local models can do native tool_calls, and reaction time vs. context size. h‑tiny (hybrid) beats gpt‑oss at every context and the lead grows; big coders are async‑only on CPU.
 - **[outputs/](outputs/)** — the raw generated `.ai.md` trees, one directory per cell.
 
 ## Bottom line

docs/ai-index-benchmark/htiny-vs-gptoss-cpu-context.png.license

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+
+SPDX-License-Identifier: Apache-2.0

docs/ai-index-benchmark/htiny-vs-gptoss-cpu-vs-vulkan.png.license

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+
+SPDX-License-Identifier: Apache-2.0

docs/ai-index-benchmark/outputs/ai__gptoss-20b__v1cur/main/java/net/ladenthin/maven/llamacpp/aiindex/config/AiCalibration.java.ai.md

@@ -0,0 +1,55 @@
+### AiCalibration.java
+- H: 1.0
+- C: C4C9EE6C
+- D: 2026-07-02T19:31:59Z
+- T: 2026-07-02T22:42:43Z
+- G: 1.0.3-SNAPSHOT
+- A: 0.0.0
+- X: file
+---
+**Facts (exact, whole file):** type declarations: 1; public declarations: 8; TODO/FIXME: 0; @Override: 0; methods (approx): 7; constructors: 1; field declarations (w/ modifier): 3
+
+> Holds per‑model calibration data for AI inference timing and token‑to‑character ratios, used by the estimator to adjust time estimates to the target hardware.
+
+#### Purpose
+- Stores calibration metrics for AI model inference.
+- Supplies values to the estimator when both prefill and decode rates are set.
+
+#### Type
+- `class`  
+  `public`  
+  `@SuppressWarnings({"NullAway.Init","initialization.fields.uninitialized"})`  
+  `@ToString` (Lombok)
+
+#### Input
+- Constructor: no arguments.  
+- Setters:  
+  - `setPrefillTokensPerSecond(double)`  
+  - `setDecodeTokensPerSecond(double)`  
+  - `setCharsPerToken(double)`  
+
+#### Output
+- Getters return the stored calibration values.  
+- `toString()` provides a string representation of all fields.
+
+#### Core logic
+- Simple storage of three double values.  
+- No additional computation or validation performed.
+
+#### Public API
+- `getPrefillTokensPerSecond() -> double` – read prefill rate  
+- `setPrefillTokensPerSecond(double) -> void` – set prefill rate  
+- `getDecodeTokensPerSecond() -> double` – read decode rate  
+- `setDecodeTokensPerSecond(double) -> void` – set decode rate  
+- `getCharsPerToken() -> double` – read chars‑per‑token  
+- `setCharsPerToken(double) -> void` – set chars‑per‑token  
+
+#### Dependencies
+- `lombok.ToString`
+
+#### Exceptions / Errors
+- No checked exceptions.  
+- No null handling; primitive types.
+
+#### Concurrency
+- Not thread‑safe; intended for single‑threaded configuration use.