Benchmark: keep each attempt's draft counts beside the receipt

docs/community-benchmarks.md

@@ -13,7 +13,7 @@ The command starts its own single-request `serve` on a loopback port, disables p
 
 An engine that decodes greedily only refuses the sampled cells, so it cannot complete community-v1; the run prints the server's reason, and `--temperatures 0` gives a local run. A large checkpoint may need its engine's own load limit raised (`GLM_LOAD_LIMIT_GB` for GLM-5.3-Flash) and a longer `--startup-timeout`: without the flag, the run waits up to 600 s plus 6 s per GB of weights past 20 GB, at most 3600 s.
 
-Results are saved locally in `~/.tensorfold/benchmarks/RUN_ID.json`; `--output FILE` chooses another location. Files contain public hardware and model metadata and individual attempts, never prompt text, generated output, machine names, serial numbers, GPU UUIDs, addresses or local model paths. Local model directories can be benchmarked but cannot be published, and a drafter directory is never named. Managed native runs record `python_version` as `"none"`. An attached manifest may attest the same value; an unattested attached run records `"unknown"`.
+Results are saved locally in `~/.tensorfold/benchmarks/RUN_ID.json`; `--output FILE` chooses another location. Files contain public hardware and model metadata and individual attempts, never prompt text, generated output, machine names, serial numbers, GPU UUIDs, addresses or local model paths. Local model directories can be benchmarked but cannot be published, and a drafter directory is never named. Managed native runs record `python_version` as `"none"`. An attached manifest may attest the same value; an unattested attached run records `"unknown"`. Beside each receipt, `RUN_ID.drafts.json` (for `--output FILE.json`, `FILE.drafts.json`) keeps what the server reported for each attempt: its verify rounds, the tokens a drafter proposed and the tokens the target accepted, with their totals (null, and the run says so, when a sum would pass 64 bits) and whether the run passed a drafter. The board's receipt has no field for these counts, so they are never uploaded; they are how a local run shows that its drafter ran, since equal timings do not. When a drafter was passed and the server drafted no tokens, the run says so.
 
 ## Commands
 

docs/receipts/drafter-counts/README.md

@@ -0,0 +1,124 @@
+# Draft counts beside the receipt: gfx1151 runs and host controls
+
+These are the receipts behind [#1045](https://github.com/ashhart/TensorFold/pull/1045): each benchmark attempt's
+verify rounds, drafted tokens and accepted tokens, as the server reported them, kept in `RUN_ID.drafts.json` beside
+the board's receipt. They show that the counts in the file are the server's own: for the two attached runs, every
+sample's counts are the same numbers the server printed for that request in its log.
+
+## Where and when
+
+The GPU runs were measured on 18 October 2026 on a Bosgame M5 (AMD Ryzen AI Max+ 395, Radeon 8060S, `gfx1151`,
+one GPU, Linux, HIP backend; the receipts record it as `rocm`). Times below are UTC, as in the receipts.
+
+| Folder | Measured | Binary SHA-256 (`tensorfold-native`, HIP build) | Source |
+| --- | --- | --- | --- |
+| `out/` | 17:33 to 17:47 | `e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247` | see below |
+| `out-r5b/` | ended 18:18 | `7fcaa74ec93561902ef97675f00db1b6d022392c665a7578960e18c92bcd2407` | `out/` plus the identity fix |
+
+The `out/` binary was built after the merge that `heads.txt` records (`7b84f909`, made on the box and not published)
+of `hip-refuses-drafter` and this branch. The bundle prepared for the run carries main at `76aafcf2`,
+`hip-refuses-drafter` at `920d1348` and this branch at `1b4b2d7b`; the branch's later `d2dfb603` changes comments
+only. The `out-r5b/` binary adds `benchmark-rocm-host-identity` at `624e2ce6` (merge `25a7201f`,
+`out-r5b/heads-r5b.txt`). The second line of `heads.txt` is `git rev-parse` echoing a ref the bundle clone did not
+have (the matching `fatal:` line is in `run.log`).
+
+The checkpoints are `TensorFold/Qwen3.8-27B-MLX-4bit` at `70ae7fac63274ff2eac54152031433374cb80f2f` and
+`TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP` at `f84b054c5677b6e59bdb969ae61915b0983d68bb` (`pull.log`). Every run is
+community-v1: code and chat, temperatures 1 and 0, one warm-up and five repeats per cell, 256 tokens per reply.
+
+## The runs
+
+| Run | How | Drafter passed | Rounds | Drafted | Accepted | Server log |
+| --- | --- | --- | ---: | ---: | ---: | --- |
+| `out/27b-auto` | managed | no | 5030 | 262 | 70 | not kept |
+| `out/35b` | managed | no | 2495 | 4490 | 2615 | not kept |
+| `out/35b-serial` | managed, `serial: true` | no | 5100 | 0 | 0 | not kept |
+| `out/35b-attached` | attached to `out/35b-serve.log`'s server | unknown | 2441 | 4626 | 2668 | `out/35b-serve.log` |
+| `out-r5b/35b-attached` | attached to `out-r5b/35b-serve.log`'s server | unknown | 2473 | 4576 | 2636 | `out-r5b/35b-serve.log` |
+
+Each total is the sum of its 20 samples and the same as the run's own `Draft counts:` line (line 37 of each `.log`).
+A managed run unlinks its server's log as it starts, so only the attached runs can be held against a server log.
+
+## Files
+
+- `out/<run>.json`: the board's receipt, community-v1 fields only. `out/<run>.drafts.json`: the counts file beside
+  it. `out/<run>.log`: the run's console output. The same three for `out-r5b/35b-attached`.
+- `out/35b-serve.log`, `out-r5b/35b-serve.log`: the attached server's log for each attached run.
+- `out/35b-manifest.json`, `out-r5b/35b-manifest.json`: the attached server's benchmark manifest, saved with each run.
+- `out/27b-dflash2.log`, `out/27b-serve-drafter.log`: two refusals of `--drafter` on HIP (below).
+- `run.log`, `out-r5b/run-r5b.log`: the box's run script: merges, build exit, binary SHA-256, each step's exit, GPU
+  memory and busy readings between steps, and the state of the box's other LLM service before and after.
+- `heads.txt`, `out-r5b/heads-r5b.txt`: the merged source. `binary.sha256`: the first binary's SHA-256.
+  `pull.log`: the two checkpoint downloads.
+- `match.py`, `match.txt`: the check of the counts against the server logs, and its output (written for this folder).
+- `host-controls/`: the CPU checks of the overflow fix. `SHA256SUMS`: see the last section.
+
+## Counts against the server's log
+
+Each attached server prints one line per finished request, for example `out/35b-serve.log` line 10:
+`done cmpl-... tokens=256 sha=5d7effdcf06f finish=length rounds=106 accepted=150/246`, where `accepted=A/D` is
+accepted over drafted. The POST line just before it names the route. In both server logs the 24 finished requests are
+lines 8 to 54 (every even line), each after its POST line: one warm-up per cell at lines 8, 20, 32 and 44, then the
+cell's five repeats. Warm-ups are not samples, so the file leaves them out.
+
+`match.py` (Python 3, standard library) reads the sanitized copies here and checks, for every sample of both attached
+runs, that the file's rounds, drafted, accepted and completion tokens are the server's for that request, that the
+receipt's `token_sha` is the server's `sha=`, and that the route is `/v1/completions` for code and
+`/v1/chat/completions` for chat. It prints each sample beside its log line, route and `finish=` (all `length`), since
+the counts file records neither the route nor the finish reason and the receipt's schema stays as it is. For all five
+runs it checks that the totals are the samples' sum and the run's printed line. Its output is `match.txt`. Two
+controls run in the same script: a file with one drafted count off by one, and the server log read one request out of
+step; it refuses both.
+
+```sh
+python3 docs/receipts/drafter-counts/match.py
+```
+
+## What else the GPU runs show
+
+- `out/35b` and `out/35b-serial` load the same checkpoint. With `serial: true` the server drafted no tokens: 255
+  rounds for each 256-token reply. The default run drafted 4490 tokens and accepted 2615 in 2495 rounds.
+- `out/27b-auto.log` says the run passes no drafter, as the HIP engine runs no DFlash2 drafter yet, and the file says
+  `drafter_passed: false`. The server still reported 262 drafted and 70 accepted tokens in 14 of the 20 attempts.
+  These receipts do not show where those drafts came from: the managed run's server log is not kept.
+- `out/27b-dflash2.log` and `out/27b-serve-drafter.log` are the refusals from `hip-refuses-drafter`: the benchmark and
+  `serve` both refuse `--drafter` on HIP. `run.log` records both steps (R2a, R2b) exiting 1, and no `27b-dflash2.json`.
+- `out/35b-manifest.json` and `out-r5b/35b-manifest.json` are the attached server's benchmark manifest, saved with
+  each run; the second carries the identity fix. The counts match their server logs in both runs.
+
+## Host controls
+
+`host-controls/` holds the CPU-only checks, on a MacBook (Apple silicon, macOS, Zig 0.17.0) on 19 October 2026.
+`commands.txt` says what each log ran.
+
+- `host-tests.log`: the host test step on `616d3065`, the overflow fix: 292 of 292 steps succeed. Steps whose inputs
+  did not change since the first run come from the cache, so its summary counts the 958 tests that ran again (957
+  passed, 1 skipped), among them the 433 of 433 in the server's test binary, which holds the one new unit test and
+  the one new case in the existing run test.
+- `host-tests-full.log`: the first, uncached run of the same step, before the commit, on a tree that differs from
+  `616d3065` only in one comment's wording: 292 of 292 steps, 4315 of 4331 tests passed, 16 skipped.
+- `negative-control.log`: `zig/src/server/benchmark_drafts.zig` put back as `d2dfb603` has it, the unchecked sums,
+  with the new run test kept. That test aborts with `panic: integer overflow` at `t.rounds += d.rounds`
+  (`benchmark_drafts.zig:18`, from `totals`), in both binaries that hold the server's tests: 431 passed and 1
+  crashed of 432 in each. The summary: 288 of 292 steps, 2 failed.
+- `lean.txt`: the lean check finds 503 problems on `616d3065`, on main (`b7ce8121`) and on the merge base
+  (`76aafcf2`), none new. `fmt.txt`: `zig fmt --check` passes.
+
+In a passing log, a `failed command:` line is the build runner printing a test binary's stderr (expected warnings
+from the tests) under its command; the `Build Summary` line is the result.
+
+The GPU runs predate the overflow fix. For sums that fit in 64 bits the fix writes the same file and the same line.
+
+## Sanitized copies
+
+These are sanitized copies of the raw files. Changed: `/home/<user>` became `~` in `binary.sha256`, `run.log`,
+`pull.log`, `out-r5b/run-r5b.log` and the `Receipt saved:` and `Draft counts:` lines of every run's `.log`;
+trailing spaces were removed from line 1 of both `35b-serve.log` files; the host logs' local paths became `~` and
+their trailing spaces were removed. Every receipt, counts file and manifest is byte for byte the raw file, so its hash
+in `SHA256SUMS` is the original's. Counts, hashes, run IDs, times and the binary SHA-256 values above are unchanged.
+
+The hashes of the raw files that were sanitized are not published. Such a file differs from its published copy only in
+the replaced paths and the removed spaces, so its raw hash would let anyone who guesses a replaced path check the
+guess. `SHA256SUMS` lists the published bytes of every file here except itself; run `shasum -a 256 -c SHA256SUMS` in
+this folder. Not copied: an empty `build.log` (the build's exit, 0, is in `run.log`) and two scripts that check
+receipts against the board's backend list, which is not about draft counts.

docs/receipts/drafter-counts/SHA256SUMS

@@ -0,0 +1,36 @@
+5c6540f8fdc7b7131a0576ad27d44f4f8b045dbc9f7c996f5c9ec39e5cc7ca09  README.md
+c8d46b5b5e8f9361422854deb9ec085dddcf4cbe1024963ac8d48ec0570e9b34  binary.sha256
+f9ef9a708c394481e9541c28d9852536dee07d9a5e644e17daa4f6542dc6cb9e  heads.txt
+5518e54903ed0183d01a4818af45a15a8b3fd3da4311216f1daf83f0ccea04c4  host-controls/commands.txt
+c1b5e4c5afe80315680e2389c046fe88950f61f9d95926dd7fd809ed2c59e255  host-controls/fmt.txt
+8a6e05131f8d36349a7f4411c335d184ff5cab6673ea2de0a9f889b2ddf0f189  host-controls/host-tests-full.log
+dc5719f3986c65a850849ba148826c75978bfeb74d364762f8415ccb58461ef7  host-controls/host-tests.log
+502144bc91d20a4ec26cfa5ef5aab36b266913b3b9f9b876e75b72daf20f4c65  host-controls/lean.txt
+73f79a91aa5c7133b5b6b4e5138f6e1b00b624f340fc0e0446ec2948afbdba0a  host-controls/negative-control.log
+384d539232f442966c80368c1c194711b8299de92bce67d540976e65a8145d91  match.py
+79a9859f54a31680e21d8acccb7a0dc8cd254340e062d5d22736e4d6ebbeb642  match.txt
+ed60c8674304318863468c3db31f7ef3148c0d53a24f9b15b325ce01b2e181ce  out-r5b/35b-attached.drafts.json
+9646641b849fbed3d9f27311b36add30dcddfe68036a755cca10fd67fd1af457  out-r5b/35b-attached.json
+e267ac9084bc4927e40e625f579736237bf7930ab279d2a51c01f86e9f2e4f0e  out-r5b/35b-attached.log
+09a0007d8ce0b59842642ebb8b95f5995e94006d3194b567ac04b6733dab3d11  out-r5b/35b-manifest.json
+1f946eab3ea5eeb58a250dbeed91b4ad4aec61cf888bd54e01d91f5324430012  out-r5b/35b-serve.log
+3bbe4a9b12f87d63e4e6fed9ab5095923c748d09e209138521b281ee6b1e4a31  out-r5b/heads-r5b.txt
+70cfeffeb0d1e6c9cfc28e8cd3636823bbc03cbace8514ddcf0097a613bd3261  out-r5b/run-r5b.log
+689ee610811cc72e505ddded6e982bdbdf337f7d6aafcb2c2bd227b78acb8870  out/27b-auto.drafts.json
+95ae8bad28ac5702aa66c326f96ac7b6816a4a5834f4f779c35ed117bb5c7a6e  out/27b-auto.json
+32ed01a245b52ad8823c332a4ef30ae381503d754abdb1a3efa8bf98d2baad66  out/27b-auto.log
+d0c05b5d100079881ccb6a5cf05060629cc529c87893eac3c625fda50439e8c2  out/27b-dflash2.log
+c53992d45e846da1a8d97e0864ea658c256624bc891621b558d1de5f9aae5a9e  out/27b-serve-drafter.log
+b96cd7d5b770026988bf13a1d5b7a6301fda14de94350d5744ac26fd2c9e2807  out/35b-attached.drafts.json
+097812b10a5665c85289b2d4142dd79385855f23cac533ae9b8098c9b4dd677c  out/35b-attached.json
+6f0cb0ba3dc3a742be06063aa5bb20ab54b506534f9319858ed0e1b29229bbb1  out/35b-attached.log
+376497f0f00b1b5277e641ccf66abc79b1ded09f947f0016e87040f80ae276f0  out/35b-manifest.json
+92fe121f484b1682a0ed911e8fc025a580343a496b362464dc24c089d7715a62  out/35b-serial.drafts.json
+c61da6ee47d938af90a55fea95153c06cfceccfbd6272d50dbfa08aff0d2b08c  out/35b-serial.json
+7e6d8dfd93e5b2ee7a516fc7b6e12bd7179a6b867f3113bb7118d94db4d54111  out/35b-serial.log
+86c2555de69aa7699a5d6e0d9c660120d5cede3d65a1d6528ba335c348824050  out/35b-serve.log
+9bb8baa0274cf8e4a1658dbc77ecf8fa22764e79ab3cec1bac093bbda47e6470  out/35b.drafts.json
+c895bc792123d2542555d71a17d5c563879ff4434b585daba38bec920026d70e  out/35b.json
+a4af1cfe5c5b51d751ce6d4f560f106bb2faa84315bf9055896634839548ffb8  out/35b.log
+8c9fe07b7b34c04f534b69eab48a9e2261072e40fcee63d5aedab132935ae4e8  pull.log
+e7118e8cc40d5469c81eca4215abfd7e2c7ed03ada41153ce8ddc4728dbc6d9c  run.log

docs/receipts/drafter-counts/binary.sha256

@@ -0,0 +1 @@
+e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247  ~/tf-1001/zig-out/native-hip/bin/tensorfold-native

docs/receipts/drafter-counts/fresh-2026-10-19/README.md

@@ -0,0 +1,66 @@
+# A fresh gfx1151 run with all three #1001 branches
+
+The run offered on [#1001](https://github.com/ashhart/TensorFold/issues/1001): GPU, CPU and checkpoint identity and
+each request's drafted and accepted tokens, measured after the fixes in #1043, #1044 and #1045, with all three merged
+into one binary. The 18 October receipts in `../out/` and `../out-r5b/` were made with earlier heads of these
+branches, before the visible-device commits `82237515` and `fb8eaf85` (#1043) and the last fixes of #1044 and #1045.
+
+## Where and when
+
+A Bosgame M5, 19 October 2026, 17:31 to 17:47 UTC (the `t=` stamps in `run.log`). The receipts record the CPU
+(`AMD RYZEN AI MAX+ 395 with Radeon 8060S`), one GPU (`AMD Radeon 8060S Graphics`), Linux and the `rocm` backend; the
+binary was built for `gfx1151` (`prep.sh`). The checkpoints are the 18 October ones, offline:
+`TensorFold/Qwen3.8-27B-MLX-4bit` at `70ae7fac63274ff2eac54152031433374cb80f2f` and
+`TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP` at `f84b054c5677b6e59bdb969ae61915b0983d68bb`.
+
+Source (`heads.txt`, `prep.log`): main `76aafcf2` with `benchmark-rocm-host-identity` at `c949cc98` (#1043),
+`hip-refuses-drafter` at `f2c78eeb` (#1044) and `benchmark-drafter-counts` at `134a5f43` (#1045), merged on the box
+(`7f4db1cf`, not published; git merged the three without a conflict) and built with `-Doptimize=ReleaseFast`
+(`binary.sha256`). `prep.sh` merged and built; `run.sh` ran R1 to R6 on the box. `match.py` and `check.mjs` were run
+afterwards on a Mac.
+
+## The runs
+
+Each benchmark ran 20 attempts, all `ok`; the counts are summed over the 20 and every attempt reported them.
+
+| Run | What it is | Exit | Accepted of drafted, rounds |
+| --- | --- | --- | --- |
+| R1 `27b-auto` | 27B, default drafting, no drafter passed | 0 | 70 of 262, 5030 |
+| R2a `27b-dflash2` | `benchmark --drafter z-lab/Qwen3.8-27B-DFlash2`: refused, no receipt | 1 | none |
+| R2b `27b-serve-drafter` | `serve --drafter z-lab/Qwen3.8-27B-DFlash2`: refused | 1 | none |
+| R3 `35b` | 35B-A3B-MTP, MTP drafts | 0 | 2466 of 4251, 2642 |
+| R4 `35b-serial` | the same, `--serial` (plain decode) | 0 | 0 of 0, 5100 |
+| R5 `35b-attached` | the same served, benchmarked attached (`35b-manifest.json`) | 0 | 2626 of 4537, 2483 |
+| R6 `35b-hipvis0` | R3 with `HIP_VISIBLE_DEVICES=0` set by `run.sh` | 0 | 2601 of 4500, 2507 |
+
+R1's counts are the same as 18 October's (`../out/27b-auto`): 70 of 262 over 5030 rounds. This checkpoint has no
+MTP tensors (its file list in `../pull.log` has no `mtp-4bit.safetensors`), so these are not MTP drafts; a managed run
+keeps no server log, so the receipts do not show where they come from.
+
+## What the receipts show
+
+- Identity: every receipt names the CPU, the one GPU (40 compute units, 128,849,018,880 bytes), unified memory with
+  source `detected`, and the checkpoint's `revision`, `config_sha256` and `tokenizer_sha256` with `local` false. R5's
+  attached receipt takes them from the server's manifest, which carries no `family` or `quantization`, so those two are
+  null there, as in `../out-r5b/`.
+- Per-request counts: `match.py` is `../match.py` with `RUNS` set to these five runs and its two controls pointed at
+  R5. It finds every count matching: each file's samples sum to its totals and to the run's printed line, and each of
+  R5's 20 samples has the counts the server printed for that request (`35b-serve.log`; the four warm-ups are left out).
+  A drafted count off by one and the server log read one request out of step are both refused (`match.txt`).
+- R6 names the same GPU as R3. The receipts do not record the environment; `run.sh` sets the variable for R6 only.
+  On a box with one GPU this shows that the visible-device changes keep the GPU when
+  `HIP_VISIBLE_DEVICES` is set; it cannot tell the old probe from the new one. The host tests and negative controls in
+  #1043 (`docs/receipts/rocm-host-identity/host-controls/`) do that.
+- The board: `board-validator.log` runs the board's `protocol.js` (the same file, by sha256, as on 18 October) through
+  `check.mjs` from #1043. Each receipt is refused as `rocm` ("Unsupported benchmark runtime.") and accepted when only
+  `runtime.backend` is changed to `cuda`, in memory.
+
+The box's own `llm-server` was stopped for the run and restarted by `run.sh`'s exit trap (stopped by 17:31:37 UTC,
+answering again at 17:50:29, in `run.log`).
+
+## Sanitizing
+
+Home paths are written as `~` in the text files (`*.log`, `binary.sha256`), and the trailing spaces of line 1 of
+`35b-serve.log` were removed, as in `../out/`. The receipts, draft-count files and the manifest are as the run wrote
+them. `SHA256SUMS` covers every file in this folder except itself
+(`shasum -a 256 -c SHA256SUMS` from this folder).