Repository navigation
Benchmark: ROCm receipts name their GPU, CPU and checkpoint the way the board checks them - #1043
ThinkOffApp wants to merge 11 commits into
Conversation
A ROCm receipt recorded no GPU, so a Strix Halo run could not say what it ran on. The probe now reads rocminfo's GPU agents: vendor and marketing name, first global pool and compute units, kept by HIP_VISIBLE_DEVICES (else ROCR_VISIBLE_DEVICES) indices. An APU named "... Graphics" records unified memory, a discrete card dedicated. The in-kernel amdgpu driver has no version, so the driver stays unknown. Publishing ROCm is unchanged: still refused.
Strix Halo's /proc/cpuinfo names its CPU "AMD RYZEN AI MAX+ 395 w/ Radeon 8060S". The "/" is outside the board's name pattern, so the CPU was dropped and the receipt was partial. The name probe now writes a "w/" word as "with", as AMD's other APU names spell it, and the receipt names the CPU. "/" stays outside the pattern, so a path in a name is still refused.
…ed run records A managed run takes the checkpoint revision from the Hugging Face cache snapshot it loads; a server's manifest took it only from TF_CHECKPOINT_REVISION, so an attached run of the same snapshot could name another revision or none, and the manifest dropped the GPU core counts the managed probe records. Both now name one checkpoint and one GPU list: the manifest attests the loaded snapshot's name (TF_CHECKPOINT_REVISION stands in only for a directory that is not a snapshot, and one that disagrees with the snapshot leaves the revision out), and it serves and reads GPU cores within the board's range. A rocm receipt from a cached snapshot with the rocminfo probe now passes every publish rule but the backend.
The board ranks a complete run only with its GPU hardware, checkpoint revision and configuration digest (docs/community-benchmarks.md). A managed receipt that lacked one, such as a ROCm run before rocminfo named its GPU, was saved and could be published with nothing said. The run table, inspect and publish now name what a public managed receipt lacks, before any upload, so the client says what the board will say.
The gfx1151 run showed it: a HIP server's /v1/benchmark/manifest said backend "hip", the engine's name, while a receipt and the board say "rocm". The client refuses a manifest for another backend, so an attached rocm run stayed unranked with unknown hardware, and the server probed no GPU, because the rocminfo probe is keyed to "rocm". Metal's "metal" against "mlx" was the same. The manifest now names the receipt's backend (hip -> rocm, metal -> mlx, others unchanged) and probes the hardware for it, as a managed run does.
|
Thanks @ThinkOffApp; we've reviewed this draft early. The name normalization, snapshot revision and HIP/Metal backend names address useful gaps. One case needs a fix: filtering an unreportable GPU name before applying visible-device indices can report a different GPU. Please preserve agent positions and add a test with an invalid first name and a selected second agent, plus selection of the invalid first agent. Please attach the host controls and GPU receipts mentioned in the body. We'll carry it onto the 1.1.2 line ourselves, keeping community-v2's publication refusal and the chat-only docs, after that fix and our gates. Posted by Ash's AI Assistant Synthena |
The rocm probe dropped a rocminfo GPU agent whose name fails the allowlist before it applied HIP_VISIBLE_DEVICES (else ROCR_VISIBLE_DEVICES), so the indices shifted: with an unreportable first agent, selecting 0 recorded the second card and selecting 1 recorded none. The probe now keeps every GPU agent in rocminfo's order, applies the indices, and only then leaves out the agents with no reportable name. Selecting such an agent records no GPU for it, and gpu_count stays the length of the GPU list. A test covers an unreportable first agent with no selection, 0, 1 and 0,1 under both variables, and the receipt's gpu_count and save rules for each.
docs/receipts/rocm-host-identity/ holds the GPU receipts from 18 Oct on a Bosgame M5 (Radeon 8060S, gfx1151): the managed 27B and 35B runs, the first attached run whose manifest said hip with no GPUs, and the attached rerun with commit 5, whose manifest and receipt name rocm, the GPU and the revision. With them are the source heads, the binary sha256 values as recorded, and the board validator's re-run on each receipt. host-controls/ has the full host suite on main (4305 of 4321, 16 skipped) and on this branch (4317 of 4333, 16 skipped), the lean and fmt checks, and one negative control per fix: each undoes one fix on 8223751, and the test that fix explains fails. The new agent-position control fails on exactly the four selections that name one position. The files are sanitized copies: home paths became ~ and trailing spaces were stripped. provenance.tsv gives each file's raw and published sha256, and SHA256SUMS covers the folder.
The rocm probe applied ROCR_VISIBLE_DEVICES to the GPU agents rocminfo lists, but ROCr has already applied it there. BuildTopology surfaces only the GPUs that variable names, in its order, and hsa_iterate_agents, which rocminfo walks, lists only those (ROCm/ROCR-Runtime 547bc139, runtime/hsa-runtime/core/runtime/amd_topology.cpp:401-459, runtime.cpp: 236-245 and 312-329). With ROCR_VISIBLE_DEVICES=1 on two cards the probe indexed 1 again and recorded no GPU; with 1,2 on three cards it kept only the third while a one-rank HIP server ran on the second. The probe no longer reads ROCR_VISIBLE_DEVICES. It applies the list HIP applies on top: HIP_VISIBLE_DEVICES, else CUDA_VISIBLE_DEVICES (ROCm/clr dfa6202d, rocclr/device/rocm/rocdevice.cpp:390-393), each read as clr reads a string flag: a variable set to nothing is " ", and one pair of double quotes around the value is dropped (rocclr/utils/flags.cpp:26-61, 121-124, 187). The list is parsed as rocdevice.cpp:393-440 parses it. An entry is GPU- text found in an agent's Uuid, or an index that std::to_string(atoi(entry)) writes back unchanged, so " 1", "01" and "+1" are invalid. The first invalid or out-of-range entry ends the list, and a repeated index is skipped. Before, "0,3,2" named two cards where HIP sees one, "0,0" named one card twice and "2,0,2" named three. rocminfo prints every agent's Uuid (ROCm/rocm-systems df181d33, projects/rocminfo/rocminfo.cc:450-453, 665), so a GPU- entry is matched against it as clr matches it, as a case-sensitive substring. The Uuid is compared and never recorded. When a GPU HIP keeps has no reportable name, or a GPU- entry cannot be matched because rocminfo printed no Uuid line, the hardware source is now partial, not detected: the list is shorter than what the server sees. gpu_count stays the length of the GPU list, as the save rules require. benchmark_rocm_test.zig builds rocminfo text as ROCr leaves it and applies HIP's list on top (7 cases), and checks HIP's list rules (15 cases); the agent-position test's ROCR_VISIBLE_DEVICES rows become CUDA_VISIBLE_DEVICES rows, and each row names its source. Every case also passes the save rules.
…ce fix The receipts folder gains the full suite on fb8eaf8 (4321 of 4337 tests passed, 16 skipped, 292/292 steps; the server test binary 436 of 436, run twice), a fresh lint (503 lean problems on main 76aafcf and on this branch, 0 new), and one negative control per part of fb8eaf8. Each ran in a scratch copy of that commit with only that part undone: ROCR_VISIBLE_DEVICES applied twice (2 tests fail, 7 rows), the list parsed by the old select (1 test, 12 rows), and no partial source (2 tests, 3 rows). The README names the ROCm source lines the fix follows (ROCR-Runtime 547bc139, clr dfa6202d, rocm-systems df181d33) and fixes two wordings: R1, R3, R4 and R5 each ran 20 attempts, and heads.txt line 1 is a merge commit. provenance.tsv withholds the raw hashes of the sanitized logs, and SHA256SUMS is recomputed.
…ite iterateAgentCallback's lines HIP falls back to CUDA_VISIBLE_DEVICES when HIP_VISIBLE_DEVICES is unset or only a pair of quotes, and an empty value hides every GPU only for the variable in use. The receipts README cites rocdevice.cpp 327-349 for iterateAgentCallback.
|
Thanks for the early review @ashhart ! Changes since your comment:
Left out on purpose: the CUDA probe has the same pattern (it drops nvidia-smi rows with an unreportable name before CUDA_VISIBLE_DEVICES applies). We can send that as a separate PR if you want it. |
… branches merged The run offered on ashhart#1001, made after the fixes: main 76aafcf with ashhart#1043 (c949cc9), ashhart#1044 (f2c78ee) and this branch (134a5f4) merged on the box and built for gfx1151. R1 to R5 repeat the 18 October set; R6 repeats R3 with HIP_VISIBLE_DEVICES=0. Every receipt names the CPU, the one GPU and the checkpoint; every run's counts match its printed line, and R5's 20 samples match the server's own log lines, while a count off by one and a log read one request out of step are refused. Both --drafter calls are refused before any weights load. The board's validator refuses each receipt as rocm and accepts it as cuda, as on 18 October. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c949cc9 to
19430d8
Compare
… branches merged The run offered on ashhart#1001, made after the fixes: main 76aafcf with ashhart#1043 (c949cc9), ashhart#1044 (f2c78ee) and this branch (134a5f4) merged on the box and built for gfx1151. R1 to R5 repeat the 18 October set; R6 repeats R3 with HIP_VISIBLE_DEVICES=0. Every receipt names the CPU, the one GPU and the checkpoint; every run's counts match its printed line, and R5's 20 samples match the server's own log lines, while a count off by one and a log read one request out of step are refused. Both --drafter calls are refused before any weights load. The board's validator refuses each receipt as rocm and accepts it as cuda, as on 18 October.
…ches merged The run offered on #1001, made after the fixes: main 76aafcf with #1043 (c949cc9), #1044 (f2c78ee) and this branch (134a5f4) merged on the box and built for gfx1151. R1 to R5 repeat the 18 October set; R6 repeats R3 with HIP_VISIBLE_DEVICES=0. Every receipt names the CPU, the one GPU and the checkpoint; every run's counts match its printed line, and R5's 20 samples match the server's own log lines, while a count off by one and a log read one request out of step are refused. Both --drafter calls are refused before any weights load. The board's validator refuses each receipt as rocm and accepts it as cuda, as on 18 October.
…dits The 1.1.2 section had no entries for the contributed pull requests carried into this release, so neither their changes nor their authors reached the release notes. Add an entry for each, written from its commits: Flash Next's CUDA prompt tiles and structured output (#1046, #1062), --thinking-reserve, the long-loop guard and the --vision template diagnostic (#1064, #1075, #1076), GLM's Metal eh_proj tile and CUDA waiting ranks (#1066, #1053), M1's simdgroup cap (#1055), Nemotron's CUDA image and shared-window fixes (#1057, #1058), Qwen's live-row expert routing (#1047), the shared CUDA operators and contract fixture (#1056, #1068, #1070, #1071, #1073), the ROCm receipt, drafter and draft-count work (#1043, #1044, #1045), the ModelOpt recipe numbers (#1042), the Windows host build (#1034) and the session store's allocation-free rollback (#637). Thank #643's and #1060's reporters as the other entries thank theirs.
An audit of all 192 commits in the release against the 1.1.2 section found user-visible changes with no entry: the HTTP framing and private key-file checks, the private request log, research redirect credentials, model-pull paths and sizes, checkpoint index and write checks, the shared EOS resolver, Kolibri's admission edge, GLM's snapshot pages (#993), uneven TP=3 KDA shares (#1067) and unread indexer scores (#1054), Linux video loading (#1107), the 27B's paired Metal tiles, Flash Next's Metal cleanup, scheduling and commit-storage changes, the prompt-cache restore choice, the prompt profiler and benchmark timing, prepared proposal vocabularies, the README quick start and the app's checks and #884 fixes. Add an entry for each and complete the #1117, #1046 and #1043 entries; thank the reporters the commits name.
|
The ROCm receipt identity work is included in 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2), with your original authorship retained. Local receipts record the detected GPU and CPU alongside checkpoint identity; ROCm board publication remains disabled pending matching receiver and client support. Closing this PR as carried, and thank you. It landed as 11 commits, cfad959 through f7896cb. - Synthema, Ash's AI Assistant. |
|
Thank you! Agreed on keeping ROCm board publication off until the receiver and client support it. |
For #1001 (1.1.1). Branch
ThinkOffApp:benchmark-rocm-host-identity, onmainat76aafcf2.What and why
Issue #1001 asks that the client and the board agree on GPU/CPU identity and checkpoint identity before anything is published. The three earlier gfx1151 receipts did not: they named no GPU, they dropped the CPU because its name has a
/, and the checkpoint was a local copy, so it had no public name or revision. This PR makes a ROCm run record those facts in the form the board's validator accepts, and makes the client say what the board would say.The board's rules matched here are in
https://tensorfold.dev/benchmarks/protocol.js(fetched 18 Oct 2026, sha2560252160a...9cc4b):NAMEpattern for CPU and GPU names (no/);repo_idmust be a publicowner/name,localmust be false,revision40 or 64 lowercase hex,config_sha256andtokenizer_sha25664 lowercase hex;gpu_countequal to the GPU list.The client's copy of these rules is
zig/src/server/benchmark_check.zig(model()at line 85,hardware()at line 99,name()at line 232). Beyond the validator,docs/community-benchmarks.mdsays the board ranks a complete run only when it names a checkpoint revision, a configuration digest and its GPU hardware.Five commits, one subject:
Name ROCm GPUs from rocminfo. The probe reads
rocminfo's GPU agents: the vendor and marketing name (AMD Radeon 8060S Graphics), the first global memory pool and the compute units, filtered byHIP_VISIBLE_DEVICES(elseROCR_VISIBLE_DEVICES). An APU, whose GPU name ends in "Graphics", is recorded asunifiedmemory; discrete cards asdedicated. The in-kernel amdgpu driver has no version, so the driver stays unknown. Withoutrocminfo, the GPUs stay unknown, as before.Write AMD's
w/aswithin CPU names./proc/cpuinfonames Strix HaloAMD RYZEN AI MAX+ 395 w/ Radeon 8060S. The name probe now writes the wordw/aswith, so the name fits the board's pattern. A/anywhere else is still refused, so a path in a name cannot get through.An attached server attests the checkpoint and GPUs a managed run records. A managed run takes the revision from the Hugging Face cache snapshot it loads. The server's manifest took it only from
TF_CHECKPOINT_REVISION, so an attached run of the same snapshot could name a different revision or none. The manifest now attests the loaded snapshot's name.TF_CHECKPOINT_REVISIONis used only when the server's directory is not a snapshot, and if it names a different commit than the loaded snapshot, the manifest leaves the revision out. The manifest also kept GPU core counts out (it always sentcores: null); it now sends them, and the client reads them within the board's range.Say which facts the board ranks by a receipt does not name. If a public managed receipt has no GPU, no checkpoint revision or no configuration digest, the run table,
inspectandpublishnow say so before anything is uploaded. Before rocminfo, every ROCm receipt would have triggered this.A HIP or Metal server's manifest names the receipt's backend. The gfx1151 run found this (below). A HIP server's manifest said
backend: "hip", the engine's name, but receipts and the board sayrocm. The client refuses a manifest whose backend does not match, so an attached rocm run stayed unranked with unknown hardware. The server also probed no GPU, because the rocminfo probe only runs forrocm. Metal had the same problem (metalagainstmlx). The manifest now names the receipt's backend (hipbecomesrocm,metalbecomesmlx, the rest stay as they are) and probes the hardware for it, the same way a managed run does.Publishing ROCm is unchanged: the board takes
mlxandcuda, so--publishwith ROCm is still refused before the run. Once the board takesrocm, these receipts pass every other rule (a test checks this, below).Tests
Host tests on an M-series Mac, CPU only, Zig 0.17.0,
zig build test -Dcpu=apple_m1 --summary all:zig/src/server/root.zig)main76aafcf2624e2ce6The server binary runs twice in
zig build test(intestand intest-cuda-host), so the full count on this branch is 4315 of 4331 with the same 16 skips. All three PRs for #1001 also build and pass together on one merged tree.New or changed tests, in
zig/src/server/benchmark_facts_test.zig:rocminfogives one GPU,AMD Radeon 8060S Graphics, 40 compute units and 120 GiB,unified; the/proc/cpuinfoname becomesAMD RYZEN AI MAX+ 395 with Radeon 8060Sand passes the board'sNAMEpattern. A receipt with that CPU, that GPU and a cached-snapshot checkpoint is refused for publishing only because the board does not take rocm yet. With only the backend changed, it passes every publish rule. The ranking note is empty for it, saysGPUwhen rocminfo is missing, and sayscheckpoint revision, configuration digestwhen those are missing.HIP_VISIBLE_DEVICESselection, malformed selections, and no rocminfo output.hip,metalandcudaservers attestrocm,mlxandcuda. The client reads each manifest back, and it has the same GPU list as a managed probe for that backend.AMD w/home/exampleandAMD Ryzen w/ /home.Negative controls. Each fix was undone in turn on this branch and ran the suite again. The new tests failed, and each failure was the one that fix explains:
expected 40, found null);w/kept as is: the rocm probe test fails, because the CPU is dropped;GPUline.With the fixes back, all tests pass.
GPU receipts
Measured on 18 Oct 2026 (17:31 to 17:47 UTC) on a Bosgame M5: AMD Ryzen AI MAX+ 395, Radeon 8060S (
gfx1151), with 131,209,781,248 bytes of system memory shared with the GPU, on Ubuntu with HIP from/usr/bin/hipccand LLVM 21 device bitcode. The machine's own LLM service was stopped for the run, and the GPU was idle (0% busy, 1.1 GB of GPU memory in use) before it started.main76aafcf2, with this branch at5906a3f7and the other two Benchmarks: accept ROCm receipts on the board (Strix Halo) #1001 branches (920d1348,1b4b2d7b) merged into it (merge7b84f909). The fifth commit (624e2ce6) came after the run; see the attached run below.zig build native-hip -Dhip-arch=gfx1151 -Dhipcc=/usr/bin/hipcc -Dhip-include=/usr/include -Dhip-device-lib=/usr/lib/llvm-21/lib/clang/21/amdgcn/bitcode -Doptimize=ReleaseFast -j16, Zig 0.17.0, exit 0.tensorfold-nativesha256e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247. There was no-Dversion, so the receipts say1.0.12, the version inbuild.zig.zon.TensorFold/Qwen3.8-27B-MLX-4bit@70ae7fac63274ff2eac54152031433374cb80f2fandTensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@f84b054c5677b6e59bdb969ae61915b0983d68bb.Managed runs, community-v1 at defaults (256 tokens, 5 repeats, temperatures 1 and 0). All 20 attempts in every run were
ok:hardwaremodel--drafter autocpu_modelAMD RYZEN AI MAX+ 395 with Radeon 8060S;gpus[{"name": "AMD Radeon 8060S Graphics", "memory_bytes": 128849018880, "cores": 40}],gpu_count1;memory_typeunified;sourcedetectedrepo_idTensorFold/Qwen3.8-27B-MLX-4bit,revision70ae7fac...,config_sha25614b65a0e...,tokenizer_sha25649d8f5dc...,localfalserepo_idTensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP,revisionf84b054c...,config_sha256a822a9e4...,tokenizer_sha2564f990165...,localfalse--serialNone of the managed runs printed the "board ranks" note.
The board's own validator:
protocol.js(sha2560252160a...9cc4b, the same as served on 18 Oct 2026) was run with node on each receipt. Asrocm, every receipt is refused with "Unsupported benchmark runtime.", as expected today. As a schema check only, copies withruntime.backendchanged tocudapass every other rule (27B, 35B, 35B serial, 35B attached). Those copies are test-only and must never be submitted: the real receipts stayrocm, and the board has to acceptrocmbefore they can be published. That is the maintainer's decision, so this gate stays open.Attached run (found a bug, fixed in commit 5, rerun on the GPU). The same 35B snapshot was served and the benchmark run against it. The manifest attested the right revision (
f84b054c..., with noTF_CHECKPOINT_REVISIONset) and the same config and tokenizer hashes as the managed receipt. But it said"backend": "hip", and"memory_type": "unknown"with"gpus": []. The client therefore refused the manifest, and the attached receipt stayed unranked with unknown hardware. Commit 5 fixes both. Even after the fix, the manifest names no rank count unlessTF_BENCHMARK_RANK_COUNTis set, so an attached run stays unranked bymain's own rule. Attached rerun with commit 5 (18 Oct 2026, ~18:30 UTC, same Bosgame, binary sha2567fcaa74e...92bcd2407): the attached 35B receipt now carries the full identity: revisionf84b054c...,localfalse, backendrocm, CPUAMD RYZEN AI MAX+ 395 with Radeon 8060S, GPUAMD Radeon 8060S Graphicswith 40 cores and 128849018880 bytes, unified memory, sourcedetected. Run exit 0; receipts in the receipts folder (out-r5b).Full local receipts: the receipts, the
*.drafts.jsonfiles, the run logs, the manifest, the build hash and the heads, with SHA-256 sums (the 27B receipt is95ae8bad...). They can be attached on request to this PR.