diff --git a/docs/community-benchmarks.md b/docs/community-benchmarks.md index 87a242cdee..3142527f9b 100644 --- a/docs/community-benchmarks.md +++ b/docs/community-benchmarks.md @@ -33,7 +33,7 @@ tensorfold serve example-org/public-checkpoint --name local-model tensorfold benchmark example-org/public-checkpoint --server http://127.0.0.1:8080 --model-id local-model --backend mlx ``` -Attached-server results do not claim the client's hardware is the server's hardware. The client reads `GET /v1/benchmark/manifest` from the server it attached to. A manifest that passes the receipt checks fills the version, hardware, checkpoint hashes, revision and rank count (1 or 2). Without that document, or if any field fails those checks, the receipt keeps unknown hardware and an unknown revision and remains unranked. An attached run records the backend, model and context you declare, unchecked against the server. The runner does not clear or restart an attached server's caches. +Attached-server results do not claim the client's hardware is the server's hardware. The client reads `GET /v1/benchmark/manifest` from the server it attached to. A manifest that passes the receipt checks fills the version, hardware (GPU cores included), checkpoint hashes, revision and rank count (1 or 2). The manifest names the backend as the receipt does (`rocm` for the HIP engine, `mlx` for Metal) and probes the hardware as a managed run of that backend would. The server attests the revision a managed run records: the name of the Hugging Face cache snapshot it loaded. `TF_CHECKPOINT_REVISION` stands in only when the server's checkpoint directory is not a snapshot; when it names another commit than the loaded snapshot, the manifest attests no revision. Without that document, or if any field fails those checks, the receipt keeps unknown hardware and an unknown revision and remains unranked. An attached run records the backend, model and context you declare, unchecked against the server. The runner does not clear or restart an attached server's caches. The runner records failures, incomplete streams, missing usage, early EOS and cancellations. Exit code 0 means every measured attempt completed its requested output with a delivery interval; 1 means an error, and with `--publish` the receipt is still saved when the run finished; 2 means incomplete attempts remain in the saved receipt; 130 means the run was cancelled with Ctrl-C: an attempt in flight is kept in the receipt, and a cancel during server startup saves nothing. SIGTERM and SIGHUP exit 143 and 129 without a receipt. Errors never silently disappear from an average. @@ -59,7 +59,7 @@ Open [the benchmark board](https://tensorfold.dev/benchmarks), choose **Upload r That browser profile keeps later submissions together, with up to 20 uploads per day. Clearing browser data or using another browser creates a separate profile. Display names are self-chosen and do not verify an identity; matching names do not merge profiles. New-profile limits apply to reduce spam. -Complete standard community-v1 submissions publish automatically after quota checks, strict receipt validation, public-checkpoint lookup and server-side statistics/completeness checks. A complete standard run includes both workloads at the suite temperatures and seeds, five successful repetitions per cell, 256 output tokens, a native managed run or a validated attached manifest, an attested rank count of one or two, a checkpoint revision/configuration digest and reported GPU hardware. Attached runs without accepted manifest facts remain unranked. A published submission returns its public result URL. +Complete standard community-v1 submissions publish automatically after quota checks, strict receipt validation, public-checkpoint lookup and server-side statistics/completeness checks. A complete standard run includes both workloads at the suite temperatures and seeds, five successful repetitions per cell, 256 output tokens, a native managed run or a validated attached manifest, an attested rank count of one or two, a checkpoint revision/configuration digest and reported GPU hardware. Attached runs without accepted manifest facts remain unranked. A published submission returns its public result URL. `run`, `inspect` and `publish` say when a public managed receipt names no GPU, checkpoint revision or configuration digest, before anything is uploaded. If the website's model-checkpoint service is temporarily unavailable, a valid receipt is saved privately and its public-checkpoint check is retried automatically. A complete standard run becomes public only after that check passes. You do not need to rerun the benchmark or upload the same receipt again. Private or inaccessible checkpoints are never published automatically. Valid incomplete or nonstandard runs remain pending review after public-checkpoint verification; invalid receipts are rejected. @@ -106,4 +106,4 @@ The parameter-versus-speed plot uses the selected parameter basis on a logarithm ## AMD (ROCm) -A ROCm build runs the same suite and saves its receipt locally, with backend `rocm` on platform `linux`. `--backend auto` picks `rocm` on that build; an attached run names it with `--backend rocm`. A managed ROCm run records the CPU and system memory, while GPU names and the driver version stay unknown. `inspect` shows a ROCm receipt as usual. The board accepts mlx and cuda receipts for now, so `--publish` with ROCm is refused before the run starts and `benchmark publish` refuses a ROCm receipt before any upload; publishing waits for the board to accept rocm. +A ROCm build runs the same suite and saves its receipt locally, with backend `rocm` on platform `linux`. `--backend auto` picks `rocm` on that build; an attached run names it with `--backend rocm`. A managed ROCm run records the CPU (an AMD name's `w/` is written `with`, so `AMD RYZEN AI MAX+ 395 w/ Radeon 8060S` fits the board's name pattern) and system memory, and each GPU agent `rocminfo` lists: its vendor and marketing name (for example `AMD Radeon 8060S Graphics`), its first global memory pool and its compute units. `rocminfo` already lists only the GPUs `ROCR_VISIBLE_DEVICES` leaves visible, renumbered in that variable's order, so the probe does not apply it again. It keeps the agents HIP keeps, by `HIP_VISIBLE_DEVICES` or, when that is unset or only a pair of quotes, `CUDA_VISIBLE_DEVICES` (HIP on AMD reads both), with HIP's rules: an entry is an index written plainly (`1`; not `01`, `+1` or with a space) or `GPU-` text found in an agent's `Uuid` from `rocminfo`; the first entry that is invalid or past the last GPU ends the list; a repeated entry is skipped; one pair of double quotes around the value is dropped; the variable in use, set to nothing, hides every GPU. The `Uuid` is only compared, never recorded. An index counts every GPU agent, so an agent whose name fails the allowlist keeps its place: selecting it records no GPU, never the next card. When a GPU HIP keeps is not named, or a `GPU-` entry cannot be matched because `rocminfo` prints no `Uuid`, the hardware source is `partial`. An AMD APU, whose GPU is named "... Graphics", is recorded as `unified` memory; discrete cards as `dedicated`. The in-kernel amdgpu driver has no version, so the driver stays unknown. Without `rocminfo` the GPUs stay unknown, as before. `inspect` shows a ROCm receipt as usual. The board accepts mlx and cuda receipts for now, so `--publish` with ROCm is refused before the run starts and `benchmark publish` refuses a ROCm receipt before any upload; publishing waits for the board to accept rocm. diff --git a/docs/receipts/rocm-host-identity/README.md b/docs/receipts/rocm-host-identity/README.md new file mode 100644 index 0000000000..48d8dbf318 --- /dev/null +++ b/docs/receipts/rocm-host-identity/README.md @@ -0,0 +1,169 @@ +# ROCm host identity receipts (#1001, PR #1043) + +Evidence for [#1043](https://github.com/ashhart/TensorFold/pull/1043): GPU receipts from a Bosgame M5 (one Radeon +8060S, `gfx1151`) and host-side test controls from a Mac (Apple M5 Max, CPU only). Every file here is a sanitized +copy; see [Sanitizing](#sanitizing) for what changed. `SHA256SUMS` covers every file in this folder except itself +(`shasum -a 256 -c SHA256SUMS` from this folder). + +## What the GPU receipts show, and what they do not + +The GPU receipts were measured on 18 Oct 2026, before the two visible-device fixes: `82237515` ("a ROCm +visible-device index counts every GPU agent") and `fb8eaf85` ("the ROCm probe keeps the GPUs HIP keeps, by HIP's own +rules"). They change what the probe records only when `HIP_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES` or +`ROCR_VISIBLE_DEVICES` is set, or when a GPU agent's name fails the allowlist. With none of those variables set and +one GPU agent with a reportable name, as on this box, the old and the new probe record the same GPU with source +`detected`. The run logs do not record those variables, so these receipts do not exercise either fix; the host tests +and the negative controls in [`host-controls/`](#host-controls-host-controls) do. + +### `gpu-run/`: managed runs and the first attached run + +Measured 18 Oct 2026, 17:31 to 17:47 UTC (the `t=` stamps in `run.log`; each receipt's `created_at` is UTC and +agrees with them). Hardware as the receipts record it: CPU `AMD RYZEN AI MAX+ 395 with Radeon 8060S`, GPU +`AMD Radeon 8060S Graphics` with 40 compute units and 128,849,018,880 bytes, 131,209,781,248 bytes of system memory, +`unified`, source `detected`. + +Source: the git bundle sent to the box carried `main` `76aafcf2`, this branch at `5906a3f7` (its first four commits) and +the other two #1001 branches, `hip-refuses-drafter` `920d1348` and `benchmark-drafter-counts` `1b4b2d7b`. `run.log` +shows those two merged in; `heads.txt` line 1 is the merge commit made on the box, `7b84f909`. Line 2 reads +`origin/main` literally: `git rev-parse origin/main` failed on the box (the `fatal:` line in `run.log`). + +| File | What it is | +| --- | --- | +| `heads.txt` | the source commit the binary was built from (above) | +| `binary.sha256` | sha256 of that build's `tensorfold-native`: `e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247` | +| `run.log` | the driver's log: merge, disk and memory before the run, the pinned snapshots, build exit, each run's exit and the GPU's busy/GTT stamps | +| `pull.log` | the two checkpoints pulled into the Hugging Face cache at pinned revisions | +| `27b-auto.json`, `.drafts.json`, `.log` | R1: managed `TensorFold/Qwen3.8-27B-MLX-4bit@70ae7fac...`, `--drafter auto` | +| `27b-dflash2.log`, `27b-serve-drafter.log` | R2a, R2b: the `--drafter` refusals for the `hip-refuses-drafter` PR (exit 1 by design; no receipt) | +| `35b.json`, `.drafts.json`, `.log` | R3: managed `TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@f84b054c...` | +| `35b-serial.json`, `.drafts.json`, `.log` | R4: the same, `--serial` | +| `35b-serve.log`, `35b-manifest.json`, `35b-attached.json`, `.drafts.json`, `.log` | R5: the same snapshot served, then benchmarked attached | +| `check.mjs` | the harness that runs the board's `protocol.js` on each receipt (see below) | + +R1, R3, R4 and R5 each ran 20 attempts, and all 80 are `ok`. R1, R3 and R4 name the CPU and GPU above, and their +checkpoint's `revision`, `config_sha256` and `tokenizer_sha256`, with `local` false. None of their logs prints the +"board ranks" note. R5 found the bug commit 5 fixes: `35b-manifest.json` says `"backend": "hip"` with `"gpus": []`, so +the client refused the manifest and `35b-attached.json` has unknown hardware and no revision. + +`check.mjs` imports the board's validator as `protocol.mjs`: `https://tensorfold.dev/benchmarks/protocol.js` as +fetched on 18 Oct 2026, sha256 `0252160a717f4f50e12f9b019beff760ee10e415503ee504d72fc6289fc9cc4b`. That file is the +board's own code and is not copied here. The output of the 18 Oct run on the box was not saved; +`host-controls/board-validator.log` is a re-run on 19 Oct on the Mac with the same two files (same sha256) on the +receipts in this folder. Each receipt is refused as `rocm` ("Unsupported benchmark runtime.") and accepted when only +`runtime.backend` is changed to `cuda`, as the PR body says. The `cuda` copies are a schema check only and exist only +in memory. + +`build.log` was empty (0 bytes) and is not included; the build's exit is in `run.log`. + +### `gpu-attached-rerun/`: the attached run with commit 5 + +Measured 18 Oct 2026 on the same Bosgame M5; the receipt's `created_at` is `2026-10-18T18:18:44Z`. +`heads-r5b.txt`: `25a7201f`, this branch at `624e2ce6` (commit 5) merged into `7b84f909`. `run-r5b.log`: build +exit 0, `tensorfold-native` sha256 `7fcaa74ec93561902ef97675f00db1b6d022392c665a7578960e18c92bcd2407`, run exit 0. + +| File | What it is | +| --- | --- | +| `heads-r5b.txt`, `run-r5b.log` | source, build exit, binary sha256, run exit | +| `35b-serve.log` | the HIP server, `gfx1151` device 0, serving the same 35B snapshot on loopback | +| `35b-manifest.json` | `GET /v1/benchmark/manifest`: backend `rocm`, the same revision and hashes, the GPU with 40 cores | +| `35b-attached.json`, `.drafts.json`, `.log` | the attached receipt: 20 of 20 `ok`, backend `rocm`, the CPU, GPU, memory and revision above | + +The manifest attests no rank count, so the receipt's `settings.rank_count` is null and it stays unranked by `main`'s +rule. The log's line names "a rank count and checkpoint revision" because the client prints one message when either +is missing; the revision is present in both the manifest and the receipt. + +## Host controls (`host-controls/`) + +Run on a Mac with an Apple M5 Max (macOS 27.0.1), CPU only, Zig 0.17.0, with private Zig caches. Times are UTC. The +full suite is `zig build test -Dcpu=apple_m1 --summary all -j8`. The negative controls run the server tests only, +`zig build test-server-cpu --summary all -j8` (the step that holds `benchmark_facts_test.zig` and, from `fb8eaf85`, +`benchmark_rocm_test.zig`; the 19 Oct runs add `-Dcpu=apple_m1`); each log starts with the undo as a diff. The +controls on `82237515` ran in the branch checkout, and the file was restored with `git checkout` after each. The three +controls on `fb8eaf85` ran in a scratch copy of that commit (`git archive`), one undo at a time; after each the file +was copied back and compared byte for byte with the commit's (the last line of each log), and the copy was discarded +afterwards. + +| File | Tree | When | Build steps | Tests | Server test binary | +| --- | --- | --- | --- | --- | --- | +| `test-main-76aafcf2.log` | `main` `76aafcf2` | 18 Oct 23:12 | 292/292 | 4305 of 4321 passed, 16 skipped | 428 passed | +| `test-branch-82237515.log` | this branch, `82237515` | 19 Oct 07:56 | 292/292 | 4317 of 4333 passed, 16 skipped | 434 passed | +| `test-branch-fb8eaf85.log` | this branch, `fb8eaf85` | 19 Oct 08:51 | 292/292 | 4321 of 4337 passed, 16 skipped | 436 passed | + +The server test binary runs twice in `zig build test` (in `test` and in `test-cuda-host`), so its new tests add twice +their number: six at `82237515` (five from commits 1 to 5, one from `82237515`) add 12, and two more from `fb8eaf85` add +four. Each branch run was on a tree with no tracked file modified and started from an empty local cache, so every test +ran. The `failed command:` lines in these logs follow warnings that tests print on stderr; each `Build Summary` line +counts no failed test. + +| File | What was undone | When | Result | The failure | +| --- | --- | --- | --- | --- | +| `negative-control-agent-positions.log` | `82237515`: `benchmark_host.zig` as it was before the fix (an unreportable agent dropped before the indices apply), plus only the `rocmVisible` helper the new test calls | 19 Oct 07:52 | 433 of 434, 1 failed | the new test: `HIP_VISIBLE_DEVICES=1: listed GPUs [], want [AMD Radeon PRO V620]`, `HIP_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620], want []`, and the same two lines for `ROCR_VISIBLE_DEVICES` | +| `negative-control-agent-positions-minimal.log` | the same fix, a smaller undo: `amdAgents` skips an unreportable agent again | 18 Oct 23:28 | 433 of 434, 1 failed | the same four lines | +| `negative-control-backend-name.log` | commit 5: the manifest names the engine (`hip`, `metal`) | 18 Oct 23:30 | 433 of 434, 1 failed | the backend-name test: expected `rocm`, found `hip` | +| `negative-control-revision.log` | commit 3: the manifest's revision from `TF_CHECKPOINT_REVISION` only | 18 Oct 23:31 | 432 of 434, 2 crashed | the manifest-agreement and attested-revision tests: `attempt to use null value` at the revision | +| `negative-control-cores.log` | commit 3: the manifest sends `cores: null` and the client does not read cores | 18 Oct 23:32 | 432 of 434, 2 failed | the manifest-agreement and cores tests: `expected 40, found null` | +| `negative-control-w-slash.log` | commit 2: `w/` kept as is in CPU names | 18 Oct 23:33 | 433 of 434, 1 crashed | the rocm probe test: no CPU name where `AMD RYZEN AI MAX+ 395 with Radeon 8060S` is expected | +| `negative-control-ranking-note.log` | commit 4: the ranking note never names a missing fact | 18 Oct 23:35 | 433 of 434, 1 crashed | the rocm probe test: no note where `GPU` is expected | +| `negative-control-rocr-twice.log` | `fb8eaf85`, part 1: `rocmVisible` as in `82237515`, so `ROCR_VISIBLE_DEVICES` applies again and `CUDA_VISIBLE_DEVICES` and clr's reading of a value do not | 19 Oct 08:54 | 434 of 436, 2 failed | `ROCR_VISIBLE_DEVICES=1 (rocminfo lists 1 GPUs): recorded [] partial`, `ROCR_VISIBLE_DEVICES=1,2 (rocminfo lists 2 GPUs): recorded [AMD Instinct MI210] detected`, and the `CUDA_VISIBLE_DEVICES`, empty `HIP_VISIBLE_DEVICES` and quoted `"1"` rows; in the agent-position test, both `CUDA_VISIBLE_DEVICES` rows | +| `negative-control-hip-list.log` | `fb8eaf85`, part 2: the list parsed by `82237515`'s `select` again | 19 Oct 08:55 | 435 of 436, 1 failed | 12 of the 15 rows of HIP's list rules, among them `0,3,2: recorded [AMD Radeon PRO W7800, AMD Instinct MI210]`, `0,0: recorded [AMD Radeon PRO W7800, AMD Radeon PRO W7800]`, `2,0,2` with three cards, `1,x,0`, `0,`, `01`, `1` after a space, and every `GPU-` row that names a card | +| `negative-control-partial.log` | `fb8eaf85`, part 3: no `partial` when a GPU HIP keeps is unnamed or a `GPU-` entry is unresolved | 19 Oct 08:56 | 434 of 436, 2 failed | the agent-position test with no variable (printed as `=`) and with `0,1`: `listed GPUs [AMD Radeon PRO V620] detected, want [AMD Radeon PRO V620] partial`; and `1,GPU-b2b2` on `rocminfo` text with no `Uuid` lines: `detected`, want `partial` | + +The first seven ran on `82237515`, after it was committed (23:22 UTC on 18 Oct): the five controls from the PR body +again, and the agent-position one twice. In both agent-position controls the cases with no selection and with `0,1` +still pass, because dropping an agent before or after the indices gives the same list when nothing or everything is +selected. Each selection that names one position fails, in both directions: `1` loses the card it names, and `0` +reports the next card. Their last two rows set `ROCR_VISIBLE_DEVICES`; `fb8eaf85` turned those rows into +`CUDA_VISIBLE_DEVICES` rows, because the probe no longer reads `ROCR_VISIBLE_DEVICES` (below), and gave every row the +source it expects. + +The last three ran on `fb8eaf85`, one per part of that fix. In each, the rows that part does not decide still pass: with +part 1 undone, the two rows where `HIP_VISIBLE_DEVICES` holds an index (2 of 7); with part 2 undone, `-1`, the +upper-case `GPU-` row and `1,0` on text with no `Uuid` lines, which the old parser turns into the same list (3 of 15); +with part 3 undone, every row where the probe names every GPU HIP keeps. Each control's other failures are listed in its +log. + +| File | What it is | +| --- | --- | +| `lint.log` | on `fb8eaf85`: `zig fmt --check build.zig zig`, exit 0; `tools/zig/lean_check.py` on `main` `76aafcf2` and on this branch, 503 problems each, and `lean_new.py`: 0 new | +| `board-validator.log` | the board's validator re-run on the GPU receipts, described above | + +## The ROCm source `fb8eaf85` follows + +Read on 19 Oct 2026 at ROCm/ROCR-Runtime `547bc139`, ROCm/clr `dfa6202d` and ROCm/rocm-systems `df181d33`, the `develop` +heads that day. In `rocm-systems`, `projects/clr`'s visible-device block in `rocdevice.cpp` and +`projects/rocr-runtime`'s `amd_filter_device.cpp` are the same as in the other two. + +- ROCr applies `ROCR_VISIBLE_DEVICES` before anything lists a GPU. `BuildTopology` in + `runtime/hsa-runtime/core/runtime/amd_topology.cpp` (lines 401-459) ranks the GPUs the variable names in its + order and creates the others disabled; `RegisterAgent` in `runtime.cpp` (236-245) keeps disabled + GPUs out of the GPU list, and `IterateAgent` (312-329), which `rocminfo` reaches through `hsa_iterate_agents`, walks + the CPU, GPU and AIE lists. So `rocminfo` prints only the visible GPUs, renumbered, and the probe must not apply + the variable again. +- HIP reads `HIP_VISIBLE_DEVICES`, else `CUDA_VISIBLE_DEVICES` (`rocclr/device/rocm/rocdevice.cpp` 390-393), over + every GPU agent in `hsa_iterate_agents` order (`iterateAgentCallback`, 327-349), the order `rocminfo` prints. An + entry with `GPU-` in it is matched as a substring of each agent's UUID, first match wins (408-421); an index must + read back unchanged through `std::to_string(atoi(entry))` and be below the GPU count (422-426); the first invalid + entry ends the list (428-431), and a repeated index is skipped (433-436). +- clr reads these variables in `rocclr/utils/flags.cpp`: a variable set to nothing is stored as one space (121-124), + which no entry rule accepts, so HIP sees no GPU; `removeQuotes` (26-61, applied at 187) drops one pair of double + quotes around the value. +- `rocminfo` prints every agent's UUID on its `Uuid:` line (`projects/rocminfo/rocminfo.cc` 450-453 and 665 in + `rocm-systems`), the value HIP compares: `GPU-` and 16 lower-case hex digits, or `GPU-XX` for a device with none + (`amd_gpu_agent.cpp` 2057-2075 in ROCR-Runtime). The probe compares it and never records it. + +## Sanitizing + +These are copies of the raw files, changed only as follows. `provenance.tsv` lists every file's published sha256 and +which change it had. A file marked `unchanged` (the JSON receipts and manifests among them) is byte-identical to the raw +file, so its raw and published sha256 are the same. For each file that was changed the raw sha256 is withheld +(`withheld: sanitized`): such a file differs from its published copy only in the replaced paths and the stripped +spaces, so its raw hash would let anyone who guesses a replaced path check the guess. + +- `/home/` and `/Users/` became `~` (logs and `binary.sha256`). +- Trailing spaces were stripped, so the files pass `git diff --check`: the two `35b-serve.log` files, whose first line + ended in a space, and five negative-control logs, where the undo diff's blank context lines and the test runner's + blank lines were only spaces. + +The binary sha256 values above are the recorded originals. No file names a hostname (the machine is called only +"Bosgame M5", here), a private address, MAC address, serial number, Wi-Fi name, proxy or credential; the only address +is loopback `127.0.0.1`. diff --git a/docs/receipts/rocm-host-identity/SHA256SUMS b/docs/receipts/rocm-host-identity/SHA256SUMS new file mode 100644 index 0000000000..4db20b8513 --- /dev/null +++ b/docs/receipts/rocm-host-identity/SHA256SUMS @@ -0,0 +1,45 @@ +2e2b71e6c8a7910f5d782e3e0f52c67de9682fc9ab024354784e0d688b0ec716 README.md +ed60c8674304318863468c3db31f7ef3148c0d53a24f9b15b325ce01b2e181ce gpu-attached-rerun/35b-attached.drafts.json +9646641b849fbed3d9f27311b36add30dcddfe68036a755cca10fd67fd1af457 gpu-attached-rerun/35b-attached.json +e267ac9084bc4927e40e625f579736237bf7930ab279d2a51c01f86e9f2e4f0e gpu-attached-rerun/35b-attached.log +09a0007d8ce0b59842642ebb8b95f5995e94006d3194b567ac04b6733dab3d11 gpu-attached-rerun/35b-manifest.json +1f946eab3ea5eeb58a250dbeed91b4ad4aec61cf888bd54e01d91f5324430012 gpu-attached-rerun/35b-serve.log +3bbe4a9b12f87d63e4e6fed9ab5095923c748d09e209138521b281ee6b1e4a31 gpu-attached-rerun/heads-r5b.txt +70cfeffeb0d1e6c9cfc28e8cd3636823bbc03cbace8514ddcf0097a613bd3261 gpu-attached-rerun/run-r5b.log +689ee610811cc72e505ddded6e982bdbdf337f7d6aafcb2c2bd227b78acb8870 gpu-run/27b-auto.drafts.json +95ae8bad28ac5702aa66c326f96ac7b6816a4a5834f4f779c35ed117bb5c7a6e gpu-run/27b-auto.json +32ed01a245b52ad8823c332a4ef30ae381503d754abdb1a3efa8bf98d2baad66 gpu-run/27b-auto.log +d0c05b5d100079881ccb6a5cf05060629cc529c87893eac3c625fda50439e8c2 gpu-run/27b-dflash2.log +c53992d45e846da1a8d97e0864ea658c256624bc891621b558d1de5f9aae5a9e gpu-run/27b-serve-drafter.log +b96cd7d5b770026988bf13a1d5b7a6301fda14de94350d5744ac26fd2c9e2807 gpu-run/35b-attached.drafts.json +097812b10a5665c85289b2d4142dd79385855f23cac533ae9b8098c9b4dd677c gpu-run/35b-attached.json +6f0cb0ba3dc3a742be06063aa5bb20ab54b506534f9319858ed0e1b29229bbb1 gpu-run/35b-attached.log +376497f0f00b1b5277e641ccf66abc79b1ded09f947f0016e87040f80ae276f0 gpu-run/35b-manifest.json +92fe121f484b1682a0ed911e8fc025a580343a496b362464dc24c089d7715a62 gpu-run/35b-serial.drafts.json +c61da6ee47d938af90a55fea95153c06cfceccfbd6272d50dbfa08aff0d2b08c gpu-run/35b-serial.json +7e6d8dfd93e5b2ee7a516fc7b6e12bd7179a6b867f3113bb7118d94db4d54111 gpu-run/35b-serial.log +86c2555de69aa7699a5d6e0d9c660120d5cede3d65a1d6528ba335c348824050 gpu-run/35b-serve.log +9bb8baa0274cf8e4a1658dbc77ecf8fa22764e79ab3cec1bac093bbda47e6470 gpu-run/35b.drafts.json +c895bc792123d2542555d71a17d5c563879ff4434b585daba38bec920026d70e gpu-run/35b.json +a4af1cfe5c5b51d751ce6d4f560f106bb2faa84315bf9055896634839548ffb8 gpu-run/35b.log +c8d46b5b5e8f9361422854deb9ec085dddcf4cbe1024963ac8d48ec0570e9b34 gpu-run/binary.sha256 +23b78e745c0519e1c012e62e17302f81f3ffb4a8441bb2cb0c3a780e80829466 gpu-run/check.mjs +f9ef9a708c394481e9541c28d9852536dee07d9a5e644e17daa4f6542dc6cb9e gpu-run/heads.txt +8c9fe07b7b34c04f534b69eab48a9e2261072e40fcee63d5aedab132935ae4e8 gpu-run/pull.log +e7118e8cc40d5469c81eca4215abfd7e2c7ed03ada41153ce8ddc4728dbc6d9c gpu-run/run.log +d29c91f5a59a9f10981b9427c1f13b4da2ed5a181c5249ed50d329b475544e80 host-controls/board-validator.log +98071ed48e76891dfd1bc764bea839ede3e03ffb72b87eee5e9f7d3375eb35b2 host-controls/lint.log +d5d6f7b274486aeb7e83ed9e74b7b964d7b3684b445526202e5635096674a05a host-controls/negative-control-agent-positions-minimal.log +70aabac97256604d04b1e34b60ed8c17ab5987429576100a7591c26842e56a43 host-controls/negative-control-agent-positions.log +05b2a2cbf4d4cd1412cf8f62ef7dd437c382affa84e5b427994e1bd9b4d6456e host-controls/negative-control-backend-name.log +4e371a8ba0d5d460624c952adf43a2e4482a06ee8eb2b21081a9c41d7fe1b81b host-controls/negative-control-cores.log +949b8e2ab3ec65d147187125ea29e707a09167dbb4b6fe4fb3594768d6570a4d host-controls/negative-control-hip-list.log +406dea5c0acfa6ca66e1f654627b0d2a2c3d9bd7b5bc08e5ee6c15d5bc3c3d79 host-controls/negative-control-partial.log +56cb6979e17413fa5730fe6acd62cad121fa3bc4fa10f4b3f6c6ffebb8615d24 host-controls/negative-control-ranking-note.log +aa36292e675f4ee1c119b93e2e3f8b23ff2ddf707d148a4adb30386145c9d9c9 host-controls/negative-control-revision.log +ea400f4d7e41cf1f327be10094923e1bea51c433b50f8765af49fb19d9d1e95f host-controls/negative-control-rocr-twice.log +f758685dc353c7509cf1c5b10577ab692d36a4c81161425ee7cd4e4120666a3c host-controls/negative-control-w-slash.log +3e6a5115c8d59953cf37e9ee4e6ed2071939a09c0d27d88fab5cb50465e45bc5 host-controls/test-branch-82237515.log +2dc2c0c3736b3659f2c6e2b2b204caf740a03815c2b13be06529408b02be0247 host-controls/test-branch-fb8eaf85.log +d757f88ab3b7a9ec5003aa956948fdc74a106d365882afe530ff48ce255ab1a2 host-controls/test-main-76aafcf2.log +24b69daf2140d3c96577a19d04c0c0b63697a9913b890f6cd477ab104f801242 provenance.tsv diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.drafts.json b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.drafts.json new file mode 100644 index 0000000000..08b94a1046 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.drafts.json @@ -0,0 +1 @@ +{"drafter_passed":null,"drafter_repo_id":null,"receipt":"35b-attached.json","run_id":"23c5e792-c249-4f55-904a-c2aef1070d16","samples":[{"accepted":154,"completion_tokens":256,"drafted":264,"fixture_id":"code","repeat":0,"rounds":102,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":249,"fixture_id":"code","repeat":1,"rounds":89,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":228,"fixture_id":"code","repeat":2,"rounds":89,"status":"ok","temperature":1.0},{"accepted":138,"completion_tokens":256,"drafted":220,"fixture_id":"code","repeat":3,"rounds":118,"status":"ok","temperature":1.0},{"accepted":144,"completion_tokens":256,"drafted":250,"fixture_id":"code","repeat":4,"rounds":111,"status":"ok","temperature":1.0},{"accepted":114,"completion_tokens":256,"drafted":209,"fixture_id":"chat","repeat":0,"rounds":141,"status":"ok","temperature":1.0},{"accepted":144,"completion_tokens":256,"drafted":252,"fixture_id":"chat","repeat":1,"rounds":112,"status":"ok","temperature":1.0},{"accepted":96,"completion_tokens":256,"drafted":170,"fixture_id":"chat","repeat":2,"rounds":159,"status":"ok","temperature":1.0},{"accepted":54,"completion_tokens":256,"drafted":97,"fixture_id":"chat","repeat":3,"rounds":201,"status":"ok","temperature":1.0},{"accepted":55,"completion_tokens":256,"drafted":105,"fixture_id":"chat","repeat":4,"rounds":201,"status":"ok","temperature":1.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":0,"rounds":83,"status":"ok","temperature":0.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":1,"rounds":83,"status":"ok","temperature":0.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":2,"rounds":83,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":3,"rounds":82,"status":"ok","temperature":0.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":4,"rounds":83,"status":"ok","temperature":0.0},{"accepted":114,"completion_tokens":256,"drafted":218,"fixture_id":"chat","repeat":0,"rounds":142,"status":"ok","temperature":0.0},{"accepted":84,"completion_tokens":256,"drafted":168,"fixture_id":"chat","repeat":1,"rounds":172,"status":"ok","temperature":0.0},{"accepted":122,"completion_tokens":256,"drafted":251,"fixture_id":"chat","repeat":2,"rounds":134,"status":"ok","temperature":0.0},{"accepted":110,"completion_tokens":256,"drafted":235,"fixture_id":"chat","repeat":3,"rounds":146,"status":"ok","temperature":0.0},{"accepted":114,"completion_tokens":256,"drafted":210,"fixture_id":"chat","repeat":4,"rounds":142,"status":"ok","temperature":0.0}],"schema":"tensorfold-benchmark-drafts/1","totals":{"accepted":2636,"attempts":20,"drafted":4576,"reported":20,"rounds":2473}} diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.json b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.json new file mode 100644 index 0000000000..07cebf3bfa --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.json @@ -0,0 +1 @@ +{"created_at":"2026-10-18T18:18:44.299680Z","hardware":{"cpu_model":"AMD RYZEN AI MAX+ 395 with Radeon 8060S","gpu_count":1,"gpus":[{"cores":40,"memory_bytes":128849018880,"name":"AMD Radeon 8060S Graphics"}],"memory_type":"unified","source":"detected","system_memory_bytes":131209781248},"model":{"config_sha256":"a822a9e48b0aafbe3144ec37d4fb067e178ed96615ce6e4420b3149893cc5767","drafter_repo_id":null,"family":null,"local":false,"quantization":null,"repo_id":"TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP","revision":"f84b054c5677b6e59bdb969ae61915b0983d68bb","tokenizer_sha256":"4f990165752add61f3708e59f04d0f80b276b20ec960e06fc80133303f3ec141"},"run_id":"23c5e792-c249-4f55-904a-c2aef1070d16","runtime":{"backend":"rocm","dependencies":{},"platform":"linux","python_version":"none","tensorfold_version":"1.0.12"},"samples":[{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.237397889,"delivery_tps":78.76696308057672,"end_to_end_seconds":3.295957964,"error_code":null,"fixture_id":"code","prefill_seconds":0.056955031,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":78.76603615452429,"status":"ok","temperature":1.0,"token_sha":"5d7effdcf06f","ttft_seconds":0.058184278},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.930787403,"delivery_tps":87.00733452688448,"end_to_end_seconds":2.992671783,"error_code":null,"fixture_id":"code","prefill_seconds":0.058689351,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":86.98033222319357,"status":"ok","temperature":1.0,"token_sha":"bd3964ef7930","ttft_seconds":0.060928333},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.862874243,"delivery_tps":89.07132425516045,"end_to_end_seconds":2.941010748,"error_code":null,"fixture_id":"code","prefill_seconds":0.075046983,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":89.05244580043684,"status":"ok","temperature":1.0,"token_sha":"bc55cc799f87","ttft_seconds":0.077185048},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.218032322,"delivery_tps":79.24096916513196,"end_to_end_seconds":3.323499297,"error_code":null,"fixture_id":"code","prefill_seconds":0.102966972,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":79.2291227575826,"status":"ok","temperature":1.0,"token_sha":"306e0fe05bc2","ttft_seconds":0.10475618},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.206673426,"delivery_tps":79.52166189810187,"end_to_end_seconds":3.304778007,"error_code":null,"fixture_id":"code","prefill_seconds":0.09582675,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":79.5097264388664,"status":"ok","temperature":1.0,"token_sha":"5a89476cf1f5","ttft_seconds":0.097737354},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.637380711,"delivery_tps":70.1053918356252,"end_to_end_seconds":3.820668011,"error_code":null,"fixture_id":"chat","prefill_seconds":0.180313155,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":70.0920132161344,"status":"ok","temperature":1.0,"token_sha":"460edda16244","ttft_seconds":0.18244713},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.436585827,"delivery_tps":74.20155143414668,"end_to_end_seconds":3.53925292,"error_code":null,"fixture_id":"chat","prefill_seconds":0.100365123,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":74.19102561723564,"status":"ok","temperature":1.0,"token_sha":"0eba9c9d3070","ttft_seconds":0.102349214},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.61322829,"delivery_tps":70.5740073788695,"end_to_end_seconds":3.70876401,"error_code":null,"fixture_id":"chat","prefill_seconds":0.093882408,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":70.57266853606137,"status":"ok","temperature":1.0,"token_sha":"15d18697d5f1","ttft_seconds":0.095217431},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.682552571,"delivery_tps":69.24544730416558,"end_to_end_seconds":3.77583423,"error_code":null,"fixture_id":"chat","prefill_seconds":0.090835443,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":69.23584801543545,"status":"ok","temperature":1.0,"token_sha":"d813ade7fc53","ttft_seconds":0.092912002},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.72286765,"delivery_tps":68.49558565424694,"end_to_end_seconds":3.839050813,"error_code":null,"fixture_id":"chat","prefill_seconds":0.114075116,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":68.49331210322869,"status":"ok","temperature":1.0,"token_sha":"7675315607f0","ttft_seconds":0.115662301},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.510962061,"delivery_tps":101.55470047143815,"end_to_end_seconds":2.611563315,"error_code":null,"fixture_id":"code","prefill_seconds":0.098673421,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":101.55335534436598,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.100088602},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.502856842,"delivery_tps":101.88357389079947,"end_to_end_seconds":2.597287421,"error_code":null,"fixture_id":"code","prefill_seconds":0.09179928,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":101.85016154597912,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.094121749},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.525631071,"delivery_tps":100.96486495117244,"end_to_end_seconds":2.641316541,"error_code":null,"fixture_id":"code","prefill_seconds":0.113842104,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":100.94540622086969,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.115316383},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.479571621,"delivery_tps":102.84034461451034,"end_to_end_seconds":2.538006101,"error_code":null,"fixture_id":"code","prefill_seconds":0.056229296,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":102.82330302032602,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.05815271},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.497114522,"delivery_tps":102.11786353946005,"end_to_end_seconds":2.561751746,"error_code":null,"fixture_id":"code","prefill_seconds":0.062604535,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":102.10079157354336,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.064316305},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.250974187,"delivery_tps":78.43802667510998,"end_to_end_seconds":3.355127348,"error_code":null,"fixture_id":"chat","prefill_seconds":0.101222832,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":78.42744998246934,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.103859361},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.361044953,"delivery_tps":75.86926196044841,"end_to_end_seconds":3.465773003,"error_code":null,"fixture_id":"chat","prefill_seconds":0.102188539,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":75.8576053686123,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.104393002},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.223296169,"delivery_tps":79.11156363863131,"end_to_end_seconds":3.335259399,"error_code":null,"fixture_id":"chat","prefill_seconds":0.109445527,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":79.10203064606544,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.111606642},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.321044262,"delivery_tps":76.78307781614264,"end_to_end_seconds":3.470007536,"error_code":null,"fixture_id":"chat","prefill_seconds":0.146390853,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":76.76958425632151,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.148648645},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.245269776,"delivery_tps":78.57590203619485,"end_to_end_seconds":3.381288719,"error_code":null,"fixture_id":"chat","prefill_seconds":0.134429549,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":78.57404094259647,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.135674285}],"schema_version":1,"settings":{"context_tokens":null,"managed":false,"rank_count":null,"repetitions":5,"serial":false,"temperatures":[1.0,0.0],"tokens":256},"suite":{"id":"community-v1","sha256":"9bb717da00666b90b4006174abd9ed46108affc37456b43c64a224a9c938d5be"}} diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.log b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.log new file mode 100644 index 0000000000..b5433f6cc6 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-attached.log @@ -0,0 +1,37 @@ +community-v1: code and chat, 2 temperatures, 5 repeats after warm-up. +This generates model replies; use Ctrl-C to cancel. Nothing is uploaded unless you publish. +code T=1 warm-up: ok 82.70 delivery tok/s +code T=1 repeat 1/5: ok 78.77 delivery tok/s +code T=1 repeat 2/5: ok 87.01 delivery tok/s +code T=1 repeat 3/5: ok 89.07 delivery tok/s +code T=1 repeat 4/5: ok 79.24 delivery tok/s +code T=1 repeat 5/5: ok 79.52 delivery tok/s +chat T=1 warm-up: ok 73.30 delivery tok/s +chat T=1 repeat 1/5: ok 70.11 delivery tok/s +chat T=1 repeat 2/5: ok 74.20 delivery tok/s +chat T=1 repeat 3/5: ok 70.57 delivery tok/s +chat T=1 repeat 4/5: ok 69.25 delivery tok/s +chat T=1 repeat 5/5: ok 68.50 delivery tok/s +code T=0 warm-up: ok 101.78 delivery tok/s +code T=0 repeat 1/5: ok 101.55 delivery tok/s +code T=0 repeat 2/5: ok 101.88 delivery tok/s +code T=0 repeat 3/5: ok 100.96 delivery tok/s +code T=0 repeat 4/5: ok 102.84 delivery tok/s +code T=0 repeat 5/5: ok 102.12 delivery tok/s +chat T=0 warm-up: ok 73.39 delivery tok/s +chat T=0 repeat 1/5: ok 78.44 delivery tok/s +chat T=0 repeat 2/5: ok 75.87 delivery tok/s +chat T=0 repeat 3/5: ok 79.11 delivery tok/s +chat T=0 repeat 4/5: ok 76.78 delivery tok/s +chat T=0 repeat 5/5: ok 78.58 delivery tok/s +Run 23c5e792-c249-4f55-904a-c2aef1070d16 | rocm | TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP +Workload Temp Stream delivery tok/s First text, s Status +code 1 79.52 0.077 complete +chat 1 70.11 0.102 complete +code 0 101.88 0.094 complete +chat 0 78.44 0.112 complete +Stream delivery includes transport buffering; it is not an engine token clock. +The attached server's manifest does not attest a rank count and checkpoint revision; this result is unranked. +This rocm receipt is saved locally; it cannot be published until the board accepts rocm. +Receipt saved: ~/tf-1001-runs/out-r5b/35b-attached.json +Draft counts: 2636 of 4576 drafted tokens accepted over 2473 rounds (20 of 20 attempts reported them); saved: ~/tf-1001-runs/out-r5b/35b-attached.drafts.json diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-manifest.json b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-manifest.json new file mode 100644 index 0000000000..f945169c43 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-manifest.json @@ -0,0 +1 @@ +{"schema_version": 1, "runtime": {"tensorfold_version": "1.0.12", "backend": "rocm", "python_version": "none", "platform": "linux"}, "model": {"revision": "f84b054c5677b6e59bdb969ae61915b0983d68bb", "config_sha256": "a822a9e48b0aafbe3144ec37d4fb067e178ed96615ce6e4420b3149893cc5767", "tokenizer_sha256": "4f990165752add61f3708e59f04d0f80b276b20ec960e06fc80133303f3ec141"}, "hardware": {"cpu_model": "AMD RYZEN AI MAX+ 395 with Radeon 8060S", "system_memory_bytes": 131209781248, "memory_type": "unified", "gpus": [{"name": "AMD Radeon 8060S Graphics", "memory_bytes": 128849018880, "cores": 40}], "gpu_count": 1, "source": "detected"}} \ No newline at end of file diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-serve.log b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-serve.log new file mode 100644 index 0000000000..08d2996a87 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/35b-serve.log @@ -0,0 +1,59 @@ +[tensorfold] HIP rank 0 of 1: policy matrix=auto,attention=auto,kernels=auto,graphs=auto,prefill_step=1024,fill_rows=512; mtp drafts 3 under 0.3, 8 snapshots +[tensorfold] open-file limit raised: soft 1024 -> 524288 +[tensorfold] HIP gfx1151 device 0, rank 0 of 1: model 18.71 GiB; 8 streams at once, 0.69 GiB each at a 32768-token window, of 84.4 GiB left after a 12.2 GiB reserve; prompt cache 77.24 GiB (62871 pages, 8 snapshots); prompts in 1024-row chunks +[tensorfold] serving amd-35b at http://127.0.0.1:8092/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: on; context: 32768; loaded in 43.5s) +[tensorfold] 127.0.0.1 "GET /v1/benchmark/manifest HTTP/1.1" 200 - +[tensorfold] 127.0.0.1 "GET /v1/benchmark/manifest HTTP/1.1" 200 - +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-c0d885ed8529d3103c371745e93c2960 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5d7effdcf06f finish=length rounds=106 accepted=150/248 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-42cb3647d1350e43794428cd930fa548 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5d7effdcf06f finish=length rounds=102 accepted=154/264 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-8d0b719ca3724f339a4ceb4e90645ea9 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=bd3964ef7930 finish=length rounds=89 accepted=166/249 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-1e9f070f6d1be3507c865aa7d22431d2 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=bc55cc799f87 finish=length rounds=89 accepted=166/228 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-a29860269a9585c55acf222d9432aa22 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=306e0fe05bc2 finish=length rounds=118 accepted=138/220 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-21afea1a90951b87b242e02b2a0e236b prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5a89476cf1f5 finish=length rounds=111 accepted=144/250 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-d0b184cdb952c75350a292e74bcb3226 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=460edda16244 finish=length rounds=152 accepted=103/172 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-26d6191b931b254f55f48673d52ff400 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=460edda16244 finish=length rounds=141 accepted=114/209 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-1e1968dbdc59a5c3007fb11798219079 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=0eba9c9d3070 finish=length rounds=112 accepted=144/252 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-483ba283e0de600b27774b09881ab007 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=15d18697d5f1 finish=length rounds=159 accepted=96/170 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-c97363ef32b24caeab949856eff4cb1a prompt=31 cached=0 thinking=False effort=none tokens=256 sha=d813ade7fc53 finish=length rounds=201 accepted=54/97 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-d8f7bc52a258b85e726897b50ab01d69 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=7675315607f0 finish=length rounds=201 accepted=55/105 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-2f45b558de15475ff1516a723d309758 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-4bdf6ae20a440eea6f56c469e0f2d5e1 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-8d28644083b8e172b19bb9334990534b prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-1ef81db777ca1141a6d05bb7ccd105b0 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-1c8708a1866e78781ffca5257591178d prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=82 accepted=173/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-eb9325a6f7b2e36304b8dde6c2273455 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-9df108317c5aba009c0b0833241bc9b0 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=191 accepted=65/140 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-07fcec2c48a10e12aad5fa8fa3ff1a2a prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=142 accepted=114/218 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-1e513af7badb380246e6212925f9707d prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=172 accepted=84/168 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-10b388202127bbdafb268c848b9a2597 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=134 accepted=122/251 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-89e37005e7550e83e88fe3bd5b8a976f prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=146 accepted=110/235 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-32e6b75026ad6f582ba658cda730c1fa prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=142 accepted=114/210 +info: backend verify: 3005 calls, 21393 us each +info: backend keep: 1323 calls, 12 us each +info: backend draft: 3029 calls, 3530 us each +info: head graphs: 4 captured, 1186 of 1190 batches replayed +info: graphs: 17 captured (5 ms each), 3156 of 3173 rounds replayed diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/heads-r5b.txt b/docs/receipts/rocm-host-identity/gpu-attached-rerun/heads-r5b.txt new file mode 100644 index 0000000000..80629f6110 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/heads-r5b.txt @@ -0,0 +1,3 @@ +25a7201f Merge branch 'benchmark-rocm-host-identity' of https://github.com/ThinkOffApp/TensorFold into tf1001-gpu +624e2ce6 Benchmark: a HIP or Metal server's manifest names the receipt's backend +7b84f909 Merge remote-tracking branches 'origin/hip-refuses-drafter' and 'origin/benchmark-drafter-counts' into tf1001-gpu diff --git a/docs/receipts/rocm-host-identity/gpu-attached-rerun/run-r5b.log b/docs/receipts/rocm-host-identity/gpu-attached-rerun/run-r5b.log new file mode 100644 index 0000000000..8585a5db02 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-attached-rerun/run-r5b.log @@ -0,0 +1,7 @@ +Auto-merging docs/community-benchmarks.md +build_exit=0 +7fcaa74ec93561902ef97675f00db1b6d022392c665a7578960e18c92bcd2407 ~/tf-1001/zig-out/native-hip/bin/tensorfold-native +R5b exit=0 +RUNDONE +llm-server: active +llm-server chat OK after 210s diff --git a/docs/receipts/rocm-host-identity/gpu-run/27b-auto.drafts.json b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.drafts.json new file mode 100644 index 0000000000..442eff9e4a --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.drafts.json @@ -0,0 +1 @@ +{"drafter_passed":false,"drafter_repo_id":null,"receipt":"27b-auto.json","run_id":"c33268ac-a928-40da-a87b-9963f8260cec","samples":[{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":0,"rounds":254,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":1,"rounds":255,"status":"ok","temperature":1.0},{"accepted":21,"completion_tokens":256,"drafted":48,"fixture_id":"code","repeat":2,"rounds":234,"status":"ok","temperature":1.0},{"accepted":6,"completion_tokens":256,"drafted":34,"fixture_id":"code","repeat":3,"rounds":249,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":4,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":0,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":1,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":2,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":3,"rounds":255,"status":"ok","temperature":1.0},{"accepted":2,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":4,"rounds":253,"status":"ok","temperature":1.0},{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":0,"rounds":254,"status":"ok","temperature":0.0},{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":1,"rounds":254,"status":"ok","temperature":0.0},{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":2,"rounds":254,"status":"ok","temperature":0.0},{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":3,"rounds":254,"status":"ok","temperature":0.0},{"accepted":1,"completion_tokens":256,"drafted":15,"fixture_id":"code","repeat":4,"rounds":254,"status":"ok","temperature":0.0},{"accepted":7,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":0,"rounds":248,"status":"ok","temperature":0.0},{"accepted":7,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":1,"rounds":248,"status":"ok","temperature":0.0},{"accepted":7,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":2,"rounds":248,"status":"ok","temperature":0.0},{"accepted":7,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":3,"rounds":248,"status":"ok","temperature":0.0},{"accepted":7,"completion_tokens":256,"drafted":15,"fixture_id":"chat","repeat":4,"rounds":248,"status":"ok","temperature":0.0}],"schema":"tensorfold-benchmark-drafts/1","totals":{"accepted":70,"attempts":20,"drafted":262,"reported":20,"rounds":5030}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/27b-auto.json b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.json new file mode 100644 index 0000000000..e565dd215e --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.json @@ -0,0 +1 @@ +{"created_at":"2026-10-18T17:42:02.550299Z","hardware":{"cpu_model":"AMD RYZEN AI MAX+ 395 with Radeon 8060S","gpu_count":1,"gpus":[{"cores":40,"memory_bytes":128849018880,"name":"AMD Radeon 8060S Graphics"}],"memory_type":"unified","source":"detected","system_memory_bytes":131209781248},"model":{"config_sha256":"14b65a0ee06517060a6bbd979bb1a8ff54e7b304b1a1f01d54344b88b8285e85","drafter_repo_id":null,"family":"qwen3_5","local":false,"quantization":"mlx-affine-4bit-g64","repo_id":"TensorFold/Qwen3.8-27B-MLX-4bit","revision":"70ae7fac63274ff2eac54152031433374cb80f2f","tokenizer_sha256":"49d8f5dc9c3e0a89091f4f189ecfd308c8f0250929b815e125d4700bded4df2c"},"run_id":"c33268ac-a928-40da-a87b-9963f8260cec","runtime":{"backend":"rocm","dependencies":{},"platform":"linux","python_version":"none","tensorfold_version":"1.0.12"},"samples":[{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.903755167,"delivery_tps":12.811652769060611,"end_to_end_seconds":20.146377346,"error_code":null,"fixture_id":"code","prefill_seconds":0.239668512,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":12.811316595952226,"status":"ok","temperature":1.0,"token_sha":"94624c419588","ttft_seconds":0.242095358},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":20.285006059,"delivery_tps":12.570861416472797,"end_to_end_seconds":20.524461929,"error_code":null,"fixture_id":"code","prefill_seconds":0.237587234,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":12.570792048154868,"status":"ok","temperature":1.0,"token_sha":"4e0a0a5f5689","ttft_seconds":0.239065053},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":18.86791311,"delivery_tps":13.5150081788775,"end_to_end_seconds":19.122833572,"error_code":null,"fixture_id":"code","prefill_seconds":0.25297176,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":13.514924920740311,"status":"ok","temperature":1.0,"token_sha":"ed99f8b35926","ttft_seconds":0.254460918},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.860349375,"delivery_tps":12.839653280268642,"end_to_end_seconds":20.106354547,"error_code":null,"fixture_id":"code","prefill_seconds":0.2431791,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":12.839322306980971,"status":"ok","temperature":1.0,"token_sha":"42b152d2efcc","ttft_seconds":0.245406803},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.808580603,"delivery_tps":12.873209096131824,"end_to_end_seconds":20.056947924,"error_code":null,"fixture_id":"code","prefill_seconds":0.245725482,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":12.873062015571138,"status":"ok","temperature":1.0,"token_sha":"f9ef9076534b","ttft_seconds":0.248004393},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.809258081,"delivery_tps":12.872768831488072,"end_to_end_seconds":20.305741719,"error_code":null,"fixture_id":"chat","prefill_seconds":0.49413369,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":12.872719602051264,"status":"ok","temperature":1.0,"token_sha":"388b0ee93e9f","ttft_seconds":0.495791022},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.803966824,"delivery_tps":12.876208199408364,"end_to_end_seconds":20.325205822,"error_code":null,"fixture_id":"chat","prefill_seconds":0.518820272,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":12.876097269993101,"status":"ok","temperature":1.0,"token_sha":"6a931bb6524d","ttft_seconds":0.520883761},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.80579648,"delivery_tps":12.875018697556564,"end_to_end_seconds":20.299056921,"error_code":null,"fixture_id":"chat","prefill_seconds":0.491224782,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":12.874983230858177,"status":"ok","temperature":1.0,"token_sha":"90107a7ec252","ttft_seconds":0.492907113},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.801318935,"delivery_tps":12.8779300427949,"end_to_end_seconds":20.29418779,"error_code":null,"fixture_id":"chat","prefill_seconds":0.490269684,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":12.877867733675195,"status":"ok","temperature":1.0,"token_sha":"2dab910075e1","ttft_seconds":0.492228067},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.877419162,"delivery_tps":12.828627193588988,"end_to_end_seconds":20.390664197,"error_code":null,"fixture_id":"chat","prefill_seconds":0.510393965,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":12.828093925409993,"status":"ok","temperature":1.0,"token_sha":"b1aba87c5a14","ttft_seconds":0.512769042},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.85645107,"delivery_tps":12.842174016950352,"end_to_end_seconds":20.10626902,"error_code":null,"fixture_id":"code","prefill_seconds":0.247056896,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":12.841844472284416,"status":"ok","temperature":0.0,"token_sha":"2b28aa2108de","ttft_seconds":0.249392374},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.852772758,"delivery_tps":12.84455340865389,"end_to_end_seconds":20.102390354,"error_code":null,"fixture_id":"code","prefill_seconds":0.24749416,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":12.844214923842012,"status":"ok","temperature":0.0,"token_sha":"2b28aa2108de","ttft_seconds":0.249258889},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.875707891,"delivery_tps":12.829731720673333,"end_to_end_seconds":20.124128539,"error_code":null,"fixture_id":"code","prefill_seconds":0.245450701,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":12.829398212092743,"status":"ok","temperature":0.0,"token_sha":"2b28aa2108de","ttft_seconds":0.247752721},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.85645976,"delivery_tps":12.842168396689058,"end_to_end_seconds":20.104480532,"error_code":null,"fixture_id":"code","prefill_seconds":0.245327385,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":12.841827839377876,"status":"ok","temperature":0.0,"token_sha":"2b28aa2108de","ttft_seconds":0.247428063},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.860533468,"delivery_tps":12.839534265827506,"end_to_end_seconds":20.108976676,"error_code":null,"fixture_id":"code","prefill_seconds":0.246409828,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":12.839478851323637,"status":"ok","temperature":0.0,"token_sha":"2b28aa2108de","ttft_seconds":0.248107459},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.439866206,"delivery_tps":13.117374229730848,"end_to_end_seconds":19.936563847,"error_code":null,"fixture_id":"chat","prefill_seconds":0.493991165,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":13.117315278328896,"status":"ok","temperature":0.0,"token_sha":"a06fab348fbe","ttft_seconds":0.495944867},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.42160197,"delivery_tps":13.129709917538793,"end_to_end_seconds":19.91775181,"error_code":null,"fixture_id":"chat","prefill_seconds":0.493917198,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":13.12941917762138,"status":"ok","temperature":0.0,"token_sha":"a06fab348fbe","ttft_seconds":0.495757104},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.442134316,"delivery_tps":13.115843963188059,"end_to_end_seconds":19.943971719,"error_code":null,"fixture_id":"chat","prefill_seconds":0.499177995,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":13.115375102223444,"status":"ok","temperature":0.0,"token_sha":"a06fab348fbe","ttft_seconds":0.501499024},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.41672064,"delivery_tps":13.133010703912563,"end_to_end_seconds":19.921349835,"error_code":null,"fixture_id":"chat","prefill_seconds":0.502336755,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":13.132676445197223,"status":"ok","temperature":0.0,"token_sha":"a06fab348fbe","ttft_seconds":0.504314446},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":19.425647849,"delivery_tps":13.126975325722636,"end_to_end_seconds":19.921862237,"error_code":null,"fixture_id":"chat","prefill_seconds":0.494480838,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":13.126935011664902,"status":"ok","temperature":0.0,"token_sha":"a06fab348fbe","ttft_seconds":0.495917118}],"schema_version":1,"settings":{"context_tokens":null,"managed":true,"rank_count":1,"repetitions":5,"serial":false,"temperatures":[1.0,0.0],"tokens":256},"suite":{"id":"community-v1","sha256":"9bb717da00666b90b4006174abd9ed46108affc37456b43c64a224a9c938d5be"}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/27b-auto.log b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.log new file mode 100644 index 0000000000..4bee460aff --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/27b-auto.log @@ -0,0 +1,37 @@ +community-v1: code and chat, 2 temperatures, 5 repeats after warm-up. +This generates model replies; use Ctrl-C to cancel. Nothing is uploaded unless you publish. +This run passes no drafter: the HIP engine runs no DFlash2 drafter yet. +code T=1 warm-up: ok 12.81 delivery tok/s +code T=1 repeat 1/5: ok 12.81 delivery tok/s +code T=1 repeat 2/5: ok 12.57 delivery tok/s +code T=1 repeat 3/5: ok 13.52 delivery tok/s +code T=1 repeat 4/5: ok 12.84 delivery tok/s +code T=1 repeat 5/5: ok 12.87 delivery tok/s +chat T=1 warm-up: ok 12.89 delivery tok/s +chat T=1 repeat 1/5: ok 12.87 delivery tok/s +chat T=1 repeat 2/5: ok 12.88 delivery tok/s +chat T=1 repeat 3/5: ok 12.88 delivery tok/s +chat T=1 repeat 4/5: ok 12.88 delivery tok/s +chat T=1 repeat 5/5: ok 12.83 delivery tok/s +code T=0 warm-up: ok 12.85 delivery tok/s +code T=0 repeat 1/5: ok 12.84 delivery tok/s +code T=0 repeat 2/5: ok 12.84 delivery tok/s +code T=0 repeat 3/5: ok 12.83 delivery tok/s +code T=0 repeat 4/5: ok 12.84 delivery tok/s +code T=0 repeat 5/5: ok 12.84 delivery tok/s +chat T=0 warm-up: ok 13.13 delivery tok/s +chat T=0 repeat 1/5: ok 13.12 delivery tok/s +chat T=0 repeat 2/5: ok 13.13 delivery tok/s +chat T=0 repeat 3/5: ok 13.12 delivery tok/s +chat T=0 repeat 4/5: ok 13.13 delivery tok/s +chat T=0 repeat 5/5: ok 13.13 delivery tok/s +Run c33268ac-a928-40da-a87b-9963f8260cec | rocm | TensorFold/Qwen3.8-27B-MLX-4bit +Workload Temp Stream delivery tok/s First text, s Status +code 1 12.84 0.245 complete +chat 1 12.88 0.496 complete +code 0 12.84 0.248 complete +chat 0 13.13 0.496 complete +Stream delivery includes transport buffering; it is not an engine token clock. +This rocm receipt is saved locally; it cannot be published until the board accepts rocm. +Receipt saved: ~/tf-1001-runs/out/27b-auto.json +Draft counts: 70 of 262 drafted tokens accepted over 5030 rounds (20 of 20 attempts reported them); saved: ~/tf-1001-runs/out/27b-auto.drafts.json diff --git a/docs/receipts/rocm-host-identity/gpu-run/27b-dflash2.log b/docs/receipts/rocm-host-identity/gpu-run/27b-dflash2.log new file mode 100644 index 0000000000..83a51751ba --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/27b-dflash2.log @@ -0,0 +1 @@ +tensorfold benchmark: --drafter: the HIP engine runs no DFlash2 drafter yet, so a rocm run takes none; --drafter none runs without it diff --git a/docs/receipts/rocm-host-identity/gpu-run/27b-serve-drafter.log b/docs/receipts/rocm-host-identity/gpu-run/27b-serve-drafter.log new file mode 100644 index 0000000000..40d7c5404e --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/27b-serve-drafter.log @@ -0,0 +1 @@ +tensorfold: --drafter: the HIP engine runs no DFlash2 drafter yet; serve without --drafter diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-attached.drafts.json b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.drafts.json new file mode 100644 index 0000000000..85de76df20 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.drafts.json @@ -0,0 +1 @@ +{"drafter_passed":null,"drafter_repo_id":null,"receipt":"35b-attached.json","run_id":"8381ef4d-d37c-4276-9b47-9b8dfd777cb7","samples":[{"accepted":150,"completion_tokens":256,"drafted":246,"fixture_id":"code","repeat":0,"rounds":106,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":249,"fixture_id":"code","repeat":1,"rounds":89,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":228,"fixture_id":"code","repeat":2,"rounds":89,"status":"ok","temperature":1.0},{"accepted":138,"completion_tokens":256,"drafted":220,"fixture_id":"code","repeat":3,"rounds":118,"status":"ok","temperature":1.0},{"accepted":136,"completion_tokens":256,"drafted":218,"fixture_id":"code","repeat":4,"rounds":119,"status":"ok","temperature":1.0},{"accepted":114,"completion_tokens":256,"drafted":210,"fixture_id":"chat","repeat":0,"rounds":141,"status":"ok","temperature":1.0},{"accepted":144,"completion_tokens":256,"drafted":252,"fixture_id":"chat","repeat":1,"rounds":112,"status":"ok","temperature":1.0},{"accepted":90,"completion_tokens":256,"drafted":150,"fixture_id":"chat","repeat":2,"rounds":165,"status":"ok","temperature":1.0},{"accepted":120,"completion_tokens":256,"drafted":213,"fixture_id":"chat","repeat":3,"rounds":136,"status":"ok","temperature":1.0},{"accepted":125,"completion_tokens":256,"drafted":242,"fixture_id":"chat","repeat":4,"rounds":131,"status":"ok","temperature":1.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":0,"rounds":83,"status":"ok","temperature":0.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":1,"rounds":83,"status":"ok","temperature":0.0},{"accepted":172,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":2,"rounds":83,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":290,"fixture_id":"code","repeat":3,"rounds":82,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":4,"rounds":82,"status":"ok","temperature":0.0},{"accepted":113,"completion_tokens":256,"drafted":235,"fixture_id":"chat","repeat":0,"rounds":143,"status":"ok","temperature":0.0},{"accepted":48,"completion_tokens":256,"drafted":117,"fixture_id":"chat","repeat":1,"rounds":207,"status":"ok","temperature":0.0},{"accepted":108,"completion_tokens":256,"drafted":202,"fixture_id":"chat","repeat":2,"rounds":148,"status":"ok","temperature":0.0},{"accepted":121,"completion_tokens":256,"drafted":246,"fixture_id":"chat","repeat":3,"rounds":135,"status":"ok","temperature":0.0},{"accepted":67,"completion_tokens":256,"drafted":145,"fixture_id":"chat","repeat":4,"rounds":189,"status":"ok","temperature":0.0}],"schema":"tensorfold-benchmark-drafts/1","totals":{"accepted":2668,"attempts":20,"drafted":4626,"reported":20,"rounds":2441}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-attached.json b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.json new file mode 100644 index 0000000000..f77e6dc641 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.json @@ -0,0 +1 @@ +{"created_at":"2026-10-18T17:47:00.563138Z","hardware":{"cpu_model":null,"gpu_count":null,"gpus":[],"memory_type":"unknown","source":"unavailable","system_memory_bytes":null},"model":{"config_sha256":null,"drafter_repo_id":null,"family":null,"local":false,"quantization":null,"repo_id":"TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP","revision":null,"tokenizer_sha256":null},"run_id":"8381ef4d-d37c-4276-9b47-9b8dfd777cb7","runtime":{"backend":"rocm","dependencies":{},"platform":"linux","python_version":"unknown","tensorfold_version":"unknown"},"samples":[{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.243378231,"delivery_tps":78.62172766738287,"end_to_end_seconds":3.302250066,"error_code":null,"fixture_id":"code","prefill_seconds":0.057585289,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":78.62041146840825,"status":"ok","temperature":1.0,"token_sha":"5d7effdcf06f","ttft_seconds":0.058603384},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.978453736,"delivery_tps":85.61489370066884,"end_to_end_seconds":3.040722942,"error_code":null,"fixture_id":"code","prefill_seconds":0.060811547,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":85.61376953719021,"status":"ok","temperature":1.0,"token_sha":"bd3964ef7930","ttft_seconds":0.062018675},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.893639504,"delivery_tps":88.1243152948053,"end_to_end_seconds":2.953564602,"error_code":null,"fixture_id":"code","prefill_seconds":0.058518767,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":88.1218867075229,"status":"ok","temperature":1.0,"token_sha":"bc55cc799f87","ttft_seconds":0.059632448},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.261464412,"delivery_tps":78.18573738280607,"end_to_end_seconds":3.352942274,"error_code":null,"fixture_id":"code","prefill_seconds":0.089279378,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":78.17404187683177,"status":"ok","temperature":1.0,"token_sha":"306e0fe05bc2","ttft_seconds":0.090984739},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.252107398,"delivery_tps":78.410694602774,"end_to_end_seconds":3.352111473,"error_code":null,"fixture_id":"code","prefill_seconds":0.097018668,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":78.38847687963506,"status":"ok","temperature":1.0,"token_sha":"5a89476cf1f5","ttft_seconds":0.099439085},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.673197325,"delivery_tps":69.42180815183949,"end_to_end_seconds":3.81150248,"error_code":null,"fixture_id":"chat","prefill_seconds":0.135500551,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":69.4060938540061,"status":"ok","temperature":1.0,"token_sha":"460edda16244","ttft_seconds":0.137710275},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.472544189,"delivery_tps":73.43319080222653,"end_to_end_seconds":3.579571649,"error_code":null,"fixture_id":"chat","prefill_seconds":0.104786219,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":73.42358033289602,"status":"ok","temperature":1.0,"token_sha":"0eba9c9d3070","ttft_seconds":0.106693052},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.516765132,"delivery_tps":72.50981809381747,"end_to_end_seconds":3.611294276,"error_code":null,"fixture_id":"chat","prefill_seconds":0.092850914,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":72.50699875013886,"status":"ok","temperature":1.0,"token_sha":"15d18697d5f1","ttft_seconds":0.094055901},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.626211939,"delivery_tps":70.32131720086976,"end_to_end_seconds":3.719590424,"error_code":null,"fixture_id":"chat","prefill_seconds":0.090951429,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":70.3081347889163,"status":"ok","temperature":1.0,"token_sha":"d813ade7fc53","ttft_seconds":0.092991089},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.740061093,"delivery_tps":68.18070444818801,"end_to_end_seconds":3.853783732,"error_code":null,"fixture_id":"chat","prefill_seconds":0.112197111,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":68.17870914153332,"status":"ok","temperature":1.0,"token_sha":"7675315607f0","ttft_seconds":0.113449128},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.539497059,"delivery_tps":100.41358350712703,"end_to_end_seconds":2.63675358,"error_code":null,"fixture_id":"code","prefill_seconds":0.094807405,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":100.41118075316031,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.096286124},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.533257186,"delivery_tps":100.66092041868187,"end_to_end_seconds":2.632491988,"error_code":null,"fixture_id":"code","prefill_seconds":0.096488327,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":100.63805330902348,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.098856445},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.563490995,"delivery_tps":99.47372567228386,"end_to_end_seconds":2.664991688,"error_code":null,"fixture_id":"code","prefill_seconds":0.099719195,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":99.47020250953632,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.100818457},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.510903393,"delivery_tps":101.55707332703422,"end_to_end_seconds":2.580178065,"error_code":null,"fixture_id":"code","prefill_seconds":0.066937384,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":101.53330088286565,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.069008132},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.501479191,"delivery_tps":101.93968469434292,"end_to_end_seconds":2.557375648,"error_code":null,"fixture_id":"code","prefill_seconds":0.053957035,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":101.93776585224423,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.055007038},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.309267232,"delivery_tps":77.05633366027298,"end_to_end_seconds":3.428299676,"error_code":null,"fixture_id":"chat","prefill_seconds":0.117582335,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":77.05459714963683,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.118747854},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.641391661,"delivery_tps":70.02817157272553,"end_to_end_seconds":3.7573917,"error_code":null,"fixture_id":"chat","prefill_seconds":0.114361886,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":70.02573593536196,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.11568451},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.276768971,"delivery_tps":77.82056112493626,"end_to_end_seconds":3.403395281,"error_code":null,"fixture_id":"chat","prefill_seconds":0.124187325,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":77.80986231003247,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.126139917},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.299548019,"delivery_tps":77.28331229962924,"end_to_end_seconds":3.423608808,"error_code":null,"fixture_id":"chat","prefill_seconds":0.122055129,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":77.28079925220916,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.123800468},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.538275495,"delivery_tps":72.06900659949883,"end_to_end_seconds":3.6730617,"error_code":null,"fixture_id":"chat","prefill_seconds":0.132858633,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":72.06727538828737,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.13408949}],"schema_version":1,"settings":{"context_tokens":null,"managed":false,"rank_count":null,"repetitions":5,"serial":false,"temperatures":[1.0,0.0],"tokens":256},"suite":{"id":"community-v1","sha256":"9bb717da00666b90b4006174abd9ed46108affc37456b43c64a224a9c938d5be"}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-attached.log b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.log new file mode 100644 index 0000000000..589809167c --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-attached.log @@ -0,0 +1,37 @@ +community-v1: code and chat, 2 temperatures, 5 repeats after warm-up. +This generates model replies; use Ctrl-C to cancel. Nothing is uploaded unless you publish. +code T=1 warm-up: ok 83.54 delivery tok/s +code T=1 repeat 1/5: ok 78.62 delivery tok/s +code T=1 repeat 2/5: ok 85.61 delivery tok/s +code T=1 repeat 3/5: ok 88.12 delivery tok/s +code T=1 repeat 4/5: ok 78.19 delivery tok/s +code T=1 repeat 5/5: ok 78.41 delivery tok/s +chat T=1 warm-up: ok 73.76 delivery tok/s +chat T=1 repeat 1/5: ok 69.42 delivery tok/s +chat T=1 repeat 2/5: ok 73.43 delivery tok/s +chat T=1 repeat 3/5: ok 72.51 delivery tok/s +chat T=1 repeat 4/5: ok 70.32 delivery tok/s +chat T=1 repeat 5/5: ok 68.18 delivery tok/s +code T=0 warm-up: ok 100.00 delivery tok/s +code T=0 repeat 1/5: ok 100.41 delivery tok/s +code T=0 repeat 2/5: ok 100.66 delivery tok/s +code T=0 repeat 3/5: ok 99.47 delivery tok/s +code T=0 repeat 4/5: ok 101.56 delivery tok/s +code T=0 repeat 5/5: ok 101.94 delivery tok/s +chat T=0 warm-up: ok 78.82 delivery tok/s +chat T=0 repeat 1/5: ok 77.06 delivery tok/s +chat T=0 repeat 2/5: ok 70.03 delivery tok/s +chat T=0 repeat 3/5: ok 77.82 delivery tok/s +chat T=0 repeat 4/5: ok 77.28 delivery tok/s +chat T=0 repeat 5/5: ok 72.07 delivery tok/s +Run 8381ef4d-d37c-4276-9b47-9b8dfd777cb7 | rocm | TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP +Workload Temp Stream delivery tok/s First text, s Status +code 1 78.62 0.062 complete +chat 1 70.32 0.107 complete +code 0 100.66 0.096 complete +chat 0 77.06 0.124 complete +Stream delivery includes transport buffering; it is not an engine token clock. +Attached-server hardware and checkpoint revisions are unknown; this result is unranked. +This rocm receipt is saved locally; it cannot be published until the board accepts rocm. +Receipt saved: ~/tf-1001-runs/out/35b-attached.json +Draft counts: 2668 of 4626 drafted tokens accepted over 2441 rounds (20 of 20 attempts reported them); saved: ~/tf-1001-runs/out/35b-attached.drafts.json diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-manifest.json b/docs/receipts/rocm-host-identity/gpu-run/35b-manifest.json new file mode 100644 index 0000000000..428da3dc8b --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-manifest.json @@ -0,0 +1 @@ +{"schema_version": 1, "runtime": {"tensorfold_version": "1.0.12", "backend": "hip", "python_version": "none", "platform": "linux"}, "model": {"revision": "f84b054c5677b6e59bdb969ae61915b0983d68bb", "config_sha256": "a822a9e48b0aafbe3144ec37d4fb067e178ed96615ce6e4420b3149893cc5767", "tokenizer_sha256": "4f990165752add61f3708e59f04d0f80b276b20ec960e06fc80133303f3ec141"}, "hardware": {"cpu_model": "AMD RYZEN AI MAX+ 395 with Radeon 8060S", "system_memory_bytes": 131209781248, "memory_type": "unknown", "gpus": [], "gpu_count": null, "source": "partial"}} \ No newline at end of file diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-serial.drafts.json b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.drafts.json new file mode 100644 index 0000000000..65f63f88ae --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.drafts.json @@ -0,0 +1 @@ +{"drafter_passed":false,"drafter_repo_id":null,"receipt":"35b-serial.json","run_id":"94ed04e1-d318-482a-8152-ca9588272c44","samples":[{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":0,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":1,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":2,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":3,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":4,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":0,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":1,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":2,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":3,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":4,"rounds":255,"status":"ok","temperature":1.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":0,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":1,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":2,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":3,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"code","repeat":4,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":0,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":1,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":2,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":3,"rounds":255,"status":"ok","temperature":0.0},{"accepted":0,"completion_tokens":256,"drafted":0,"fixture_id":"chat","repeat":4,"rounds":255,"status":"ok","temperature":0.0}],"schema":"tensorfold-benchmark-drafts/1","totals":{"accepted":0,"attempts":20,"drafted":0,"reported":20,"rounds":5100}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-serial.json b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.json new file mode 100644 index 0000000000..796f3f1676 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.json @@ -0,0 +1 @@ +{"created_at":"2026-10-18T17:45:19.821863Z","hardware":{"cpu_model":"AMD RYZEN AI MAX+ 395 with Radeon 8060S","gpu_count":1,"gpus":[{"cores":40,"memory_bytes":128849018880,"name":"AMD Radeon 8060S Graphics"}],"memory_type":"unified","source":"detected","system_memory_bytes":131209781248},"model":{"config_sha256":"a822a9e48b0aafbe3144ec37d4fb067e178ed96615ce6e4420b3149893cc5767","drafter_repo_id":null,"family":"qwen3_5_moe","local":false,"quantization":"mlx-affine-4bit-g64","repo_id":"TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP","revision":"f84b054c5677b6e59bdb969ae61915b0983d68bb","tokenizer_sha256":"4f990165752add61f3708e59f04d0f80b276b20ec960e06fc80133303f3ec141"},"run_id":"94ed04e1-d318-482a-8152-ca9588272c44","runtime":{"backend":"rocm","dependencies":{},"platform":"linux","python_version":"none","tensorfold_version":"1.0.12"},"samples":[{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.759774118,"delivery_tps":67.82322341631695,"end_to_end_seconds":3.812049981,"error_code":null,"fixture_id":"code","prefill_seconds":0.05065055,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":67.82192521585628,"status":"ok","temperature":1.0,"token_sha":"5d7effdcf06f","ttft_seconds":0.05178199},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.765770599,"delivery_tps":67.71522409456253,"end_to_end_seconds":3.818825506,"error_code":null,"fixture_id":"code","prefill_seconds":0.050688539,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":67.70774619405101,"status":"ok","temperature":1.0,"token_sha":"bd3964ef7930","ttft_seconds":0.052756407},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.750469511,"delivery_tps":67.99148726635256,"end_to_end_seconds":3.802543212,"error_code":null,"fixture_id":"code","prefill_seconds":0.050120338,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":67.99089375296336,"status":"ok","temperature":1.0,"token_sha":"bc55cc799f87","ttft_seconds":0.051782331},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.757577024,"delivery_tps":67.86288035382664,"end_to_end_seconds":3.810577822,"error_code":null,"fixture_id":"code","prefill_seconds":0.051629016,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":67.86147625055469,"status":"ok","temperature":1.0,"token_sha":"306e0fe05bc2","ttft_seconds":0.052721208},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.76430671,"delivery_tps":67.74155764794203,"end_to_end_seconds":3.816769987,"error_code":null,"fixture_id":"code","prefill_seconds":0.051200081,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":67.74056719382524,"status":"ok","temperature":1.0,"token_sha":"5a89476cf1f5","ttft_seconds":0.052168078},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.774047542,"delivery_tps":67.56671641313416,"end_to_end_seconds":3.860697371,"error_code":null,"fixture_id":"chat","prefill_seconds":0.084277281,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":67.55822852857109,"status":"ok","temperature":1.0,"token_sha":"460edda16244","ttft_seconds":0.086375769},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.777132115,"delivery_tps":67.51153844667675,"end_to_end_seconds":3.863561471,"error_code":null,"fixture_id":"chat","prefill_seconds":0.084966138,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":67.51080441523933,"status":"ok","temperature":1.0,"token_sha":"0eba9c9d3070","ttft_seconds":0.086157876},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.768336651,"delivery_tps":67.66911335597653,"end_to_end_seconds":3.856368231,"error_code":null,"fixture_id":"chat","prefill_seconds":0.085810948,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":67.65989961192439,"status":"ok","temperature":1.0,"token_sha":"15d18697d5f1","ttft_seconds":0.08775632},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.768488215,"delivery_tps":67.66639178676587,"end_to_end_seconds":3.855664985,"error_code":null,"fixture_id":"chat","prefill_seconds":0.085679503,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":67.66519007724261,"status":"ok","temperature":1.0,"token_sha":"d813ade7fc53","ttft_seconds":0.086873521},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.787361919,"delivery_tps":67.32918729544843,"end_to_end_seconds":3.873478157,"error_code":null,"fixture_id":"chat","prefill_seconds":0.084100217,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":67.32745819217918,"status":"ok","temperature":1.0,"token_sha":"7675315607f0","ttft_seconds":0.08576239},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.68833721,"delivery_tps":69.13684554346916,"end_to_end_seconds":3.741133435,"error_code":null,"fixture_id":"code","prefill_seconds":0.050393629,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":69.13407940220092,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.052297563},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.687622845,"delivery_tps":69.15023870886124,"end_to_end_seconds":3.739775363,"error_code":null,"fixture_id":"code","prefill_seconds":0.050492205,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":69.14692100479135,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.051910066},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.683352113,"delivery_tps":69.23041625588945,"end_to_end_seconds":3.735806331,"error_code":null,"fixture_id":"code","prefill_seconds":0.050909091,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":69.22959360576273,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.051924666},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.690186125,"delivery_tps":69.10220551544673,"end_to_end_seconds":3.74245429,"error_code":null,"fixture_id":"code","prefill_seconds":0.049272778,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":69.08648303003007,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.051699174},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.683403752,"delivery_tps":69.22944568906982,"end_to_end_seconds":3.73556045,"error_code":null,"fixture_id":"code","prefill_seconds":0.050575493,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":69.22866499260533,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.051836489},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.699669996,"delivery_tps":68.92506636421633,"end_to_end_seconds":3.786365283,"error_code":null,"fixture_id":"chat","prefill_seconds":0.08520682,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":68.92445886201169,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.0863457},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.702860125,"delivery_tps":68.86568527888966,"end_to_end_seconds":3.789798745,"error_code":null,"fixture_id":"chat","prefill_seconds":0.08433075,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":68.85634621675966,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.08636053},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.706321415,"delivery_tps":68.80137242495468,"end_to_end_seconds":3.793343091,"error_code":null,"fixture_id":"chat","prefill_seconds":0.084680788,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":68.79199866677762,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.086742156},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.701143705,"delivery_tps":68.89762201222068,"end_to_end_seconds":3.788443451,"error_code":null,"fixture_id":"chat","prefill_seconds":0.084888201,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":68.88600196629339,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.086986137},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.697890868,"delivery_tps":68.95822756876447,"end_to_end_seconds":3.783991086,"error_code":null,"fixture_id":"chat","prefill_seconds":0.083931734,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":68.94859961392461,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.085842167}],"schema_version":1,"settings":{"context_tokens":null,"managed":true,"rank_count":1,"repetitions":5,"serial":true,"temperatures":[1.0,0.0],"tokens":256},"suite":{"id":"community-v1","sha256":"9bb717da00666b90b4006174abd9ed46108affc37456b43c64a224a9c938d5be"}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-serial.log b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.log new file mode 100644 index 0000000000..ca4de65088 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-serial.log @@ -0,0 +1,37 @@ +community-v1: code and chat, 2 temperatures, 5 repeats after warm-up. +This generates model replies; use Ctrl-C to cancel. Nothing is uploaded unless you publish. +The server gets up to 606 s to load 21 GB of weights; --startup-timeout sets another limit. +code T=1 warm-up: ok 67.56 delivery tok/s +code T=1 repeat 1/5: ok 67.82 delivery tok/s +code T=1 repeat 2/5: ok 67.72 delivery tok/s +code T=1 repeat 3/5: ok 67.99 delivery tok/s +code T=1 repeat 4/5: ok 67.86 delivery tok/s +code T=1 repeat 5/5: ok 67.74 delivery tok/s +chat T=1 warm-up: ok 67.60 delivery tok/s +chat T=1 repeat 1/5: ok 67.57 delivery tok/s +chat T=1 repeat 2/5: ok 67.51 delivery tok/s +chat T=1 repeat 3/5: ok 67.67 delivery tok/s +chat T=1 repeat 4/5: ok 67.67 delivery tok/s +chat T=1 repeat 5/5: ok 67.33 delivery tok/s +code T=0 warm-up: ok 69.12 delivery tok/s +code T=0 repeat 1/5: ok 69.14 delivery tok/s +code T=0 repeat 2/5: ok 69.15 delivery tok/s +code T=0 repeat 3/5: ok 69.23 delivery tok/s +code T=0 repeat 4/5: ok 69.10 delivery tok/s +code T=0 repeat 5/5: ok 69.23 delivery tok/s +chat T=0 warm-up: ok 68.94 delivery tok/s +chat T=0 repeat 1/5: ok 68.93 delivery tok/s +chat T=0 repeat 2/5: ok 68.87 delivery tok/s +chat T=0 repeat 3/5: ok 68.80 delivery tok/s +chat T=0 repeat 4/5: ok 68.90 delivery tok/s +chat T=0 repeat 5/5: ok 68.96 delivery tok/s +Run 94ed04e1-d318-482a-8152-ca9588272c44 | rocm | TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP +Workload Temp Stream delivery tok/s First text, s Status +code 1 67.82 0.052 complete +chat 1 67.57 0.086 complete +code 0 69.15 0.052 complete +chat 0 68.90 0.086 complete +Stream delivery includes transport buffering; it is not an engine token clock. +This rocm receipt is saved locally; it cannot be published until the board accepts rocm. +Receipt saved: ~/tf-1001-runs/out/35b-serial.json +Draft counts: 0 of 0 drafted tokens accepted over 5100 rounds (20 of 20 attempts reported them); saved: ~/tf-1001-runs/out/35b-serial.drafts.json diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b-serve.log b/docs/receipts/rocm-host-identity/gpu-run/35b-serve.log new file mode 100644 index 0000000000..e88b47d49d --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b-serve.log @@ -0,0 +1,59 @@ +[tensorfold] HIP rank 0 of 1: policy matrix=auto,attention=auto,kernels=auto,graphs=auto,prefill_step=1024,fill_rows=512; mtp drafts 3 under 0.3, 8 snapshots +[tensorfold] open-file limit raised: soft 1024 -> 524288 +[tensorfold] HIP gfx1151 device 0, rank 0 of 1: model 18.71 GiB; 8 streams at once, 0.69 GiB each at a 32768-token window, of 84.7 GiB left after a 12.2 GiB reserve; prompt cache 77.54 GiB (63118 pages, 8 snapshots); prompts in 1024-row chunks +[tensorfold] serving amd-35b at http://127.0.0.1:8092/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: on; context: 32768; loaded in 20.3s) +[tensorfold] 127.0.0.1 "GET /v1/benchmark/manifest HTTP/1.1" 200 - +[tensorfold] 127.0.0.1 "GET /v1/benchmark/manifest HTTP/1.1" 200 - +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-16ae8babc582d71239d55d876bb6f6d1 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5d7effdcf06f finish=length rounds=106 accepted=150/247 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-d43fc7dbb25ce8d14d2397a703841f21 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5d7effdcf06f finish=length rounds=106 accepted=150/246 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-4ffea70fec8911c107c79efac1d752ba prompt=14 cached=0 thinking=False effort=none tokens=256 sha=bd3964ef7930 finish=length rounds=89 accepted=166/249 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-b70640d2e4b24fa7942ead5bc3aeca64 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=bc55cc799f87 finish=length rounds=89 accepted=166/228 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-a30da46b31b807bf47abdc6019945988 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=306e0fe05bc2 finish=length rounds=118 accepted=138/220 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-4145895c4fb7626a3ecb2acaa1ff43ec prompt=14 cached=0 thinking=False effort=none tokens=256 sha=5a89476cf1f5 finish=length rounds=119 accepted=136/218 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-0c6951ef33cbc190d30325f7deabd8e5 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=460edda16244 finish=length rounds=155 accepted=100/155 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-d4790b7404faaa4cd6e9fae807afc4aa prompt=31 cached=0 thinking=False effort=none tokens=256 sha=460edda16244 finish=length rounds=141 accepted=114/210 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-e0e51ddb0279975edd9a044cc4bd3bf7 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=0eba9c9d3070 finish=length rounds=112 accepted=144/252 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-6bb775d0cc1b0c9356b9424908f65bd3 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=15d18697d5f1 finish=length rounds=165 accepted=90/150 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-b5cb8f90cc4415de1430fa8c46bc3a43 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=d813ade7fc53 finish=length rounds=136 accepted=120/213 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-a9b88c08b4332a012299061949d1bbe3 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=7675315607f0 finish=length rounds=131 accepted=125/242 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-5f0f121f875a331afff54268d4da88e0 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-8f301048e70a9b5ff46ff7e77149f8ca prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-cc4f7a9b74d73884968f9dd4e58dba6e prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-7ecea917473773bcf0c6f5461575098d prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=83 accepted=172/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-ed7714c9b9b4cca218aabb97aa9ca689 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=82 accepted=173/290 +[tensorfold] 127.0.0.1 "POST /v1/completions HTTP/1.1" 200 - +[tensorfold] done cmpl-c13cd03a52942ce8723142063bf4aa20 prompt=14 cached=0 thinking=False effort=none tokens=256 sha=9f2923ac14d0 finish=length rounds=82 accepted=173/293 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-78285e4804185a721b9890b34749301a prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=135 accepted=121/245 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-651b1f4704c7c43ea965645bce264c4f prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=143 accepted=113/235 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-482ac2ca8b84bc6467416fd139789a3a prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=207 accepted=48/117 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-cf60e03f9861b01bb663957707ded4a0 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=148 accepted=108/202 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-1ca737e5b005484a9218ca86062384f0 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=135 accepted=121/246 +[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 - +[tensorfold] done chatcmpl-eca1f9f0435a852e7c2b6502d8190670 prompt=31 cached=0 thinking=False effort=none tokens=256 sha=a8c28b8297b4 finish=length rounds=189 accepted=67/145 +info: backend verify: 2920 calls, 22048 us each +info: backend keep: 1351 calls, 11 us each +info: backend draft: 2944 calls, 3828 us each +info: head graphs: 4 captured, 1119 of 1123 batches replayed +info: graphs: 17 captured (4 ms each), 3071 of 3088 rounds replayed diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b.drafts.json b/docs/receipts/rocm-host-identity/gpu-run/35b.drafts.json new file mode 100644 index 0000000000..6361c928b7 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b.drafts.json @@ -0,0 +1 @@ +{"drafter_passed":false,"drafter_repo_id":null,"receipt":"35b.json","run_id":"b46ba73a-e62a-4f31-83cd-98daa4ad5c91","samples":[{"accepted":145,"completion_tokens":256,"drafted":240,"fixture_id":"code","repeat":0,"rounds":111,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":249,"fixture_id":"code","repeat":1,"rounds":89,"status":"ok","temperature":1.0},{"accepted":166,"completion_tokens":256,"drafted":228,"fixture_id":"code","repeat":2,"rounds":89,"status":"ok","temperature":1.0},{"accepted":138,"completion_tokens":256,"drafted":220,"fixture_id":"code","repeat":3,"rounds":118,"status":"ok","temperature":1.0},{"accepted":144,"completion_tokens":256,"drafted":250,"fixture_id":"code","repeat":4,"rounds":111,"status":"ok","temperature":1.0},{"accepted":108,"completion_tokens":256,"drafted":174,"fixture_id":"chat","repeat":0,"rounds":147,"status":"ok","temperature":1.0},{"accepted":105,"completion_tokens":256,"drafted":172,"fixture_id":"chat","repeat":1,"rounds":151,"status":"ok","temperature":1.0},{"accepted":106,"completion_tokens":256,"drafted":169,"fixture_id":"chat","repeat":2,"rounds":149,"status":"ok","temperature":1.0},{"accepted":120,"completion_tokens":256,"drafted":213,"fixture_id":"chat","repeat":3,"rounds":136,"status":"ok","temperature":1.0},{"accepted":64,"completion_tokens":256,"drafted":117,"fixture_id":"chat","repeat":4,"rounds":192,"status":"ok","temperature":1.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":0,"rounds":82,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":1,"rounds":82,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":2,"rounds":82,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":3,"rounds":82,"status":"ok","temperature":0.0},{"accepted":173,"completion_tokens":256,"drafted":293,"fixture_id":"code","repeat":4,"rounds":82,"status":"ok","temperature":0.0},{"accepted":108,"completion_tokens":256,"drafted":219,"fixture_id":"chat","repeat":0,"rounds":148,"status":"ok","temperature":0.0},{"accepted":108,"completion_tokens":256,"drafted":219,"fixture_id":"chat","repeat":1,"rounds":148,"status":"ok","temperature":0.0},{"accepted":122,"completion_tokens":256,"drafted":247,"fixture_id":"chat","repeat":2,"rounds":134,"status":"ok","temperature":0.0},{"accepted":65,"completion_tokens":256,"drafted":140,"fixture_id":"chat","repeat":3,"rounds":191,"status":"ok","temperature":0.0},{"accepted":85,"completion_tokens":256,"drafted":168,"fixture_id":"chat","repeat":4,"rounds":171,"status":"ok","temperature":0.0}],"schema":"tensorfold-benchmark-drafts/1","totals":{"accepted":2615,"attempts":20,"drafted":4490,"reported":20,"rounds":2495}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b.json b/docs/receipts/rocm-host-identity/gpu-run/35b.json new file mode 100644 index 0000000000..d5b90a5e56 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b.json @@ -0,0 +1 @@ +{"created_at":"2026-10-18T17:43:35.960224Z","hardware":{"cpu_model":"AMD RYZEN AI MAX+ 395 with Radeon 8060S","gpu_count":1,"gpus":[{"cores":40,"memory_bytes":128849018880,"name":"AMD Radeon 8060S Graphics"}],"memory_type":"unified","source":"detected","system_memory_bytes":131209781248},"model":{"config_sha256":"a822a9e48b0aafbe3144ec37d4fb067e178ed96615ce6e4420b3149893cc5767","drafter_repo_id":null,"family":"qwen3_5_moe","local":false,"quantization":"mlx-affine-4bit-g64","repo_id":"TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP","revision":"f84b054c5677b6e59bdb969ae61915b0983d68bb","tokenizer_sha256":"4f990165752add61f3708e59f04d0f80b276b20ec960e06fc80133303f3ec141"},"run_id":"b46ba73a-e62a-4f31-83cd-98daa4ad5c91","runtime":{"backend":"rocm","dependencies":{},"platform":"linux","python_version":"none","tensorfold_version":"1.0.12"},"samples":[{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.242303078,"delivery_tps":78.64779876077951,"end_to_end_seconds":3.302757688,"error_code":null,"fixture_id":"code","prefill_seconds":0.058524826,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":78.64690566092122,"status":"ok","temperature":1.0,"token_sha":"5d7effdcf06f","ttft_seconds":0.059956997},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.96510749,"delivery_tps":86.00025491824582,"end_to_end_seconds":3.023972944,"error_code":null,"fixture_id":"code","prefill_seconds":0.056392831,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":85.99890079644958,"status":"ok","temperature":1.0,"token_sha":"bd3964ef7930","ttft_seconds":0.058147789},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.878249098,"delivery_tps":88.59552850279395,"end_to_end_seconds":2.938296122,"error_code":null,"fixture_id":"code","prefill_seconds":0.058279035,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":88.59444074463444,"status":"ok","temperature":1.0,"token_sha":"bc55cc799f87","ttft_seconds":0.05956116},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.259204921,"delivery_tps":78.23994077726174,"end_to_end_seconds":3.318896947,"error_code":null,"fixture_id":"code","prefill_seconds":0.058096791,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":78.23906891899784,"status":"ok","temperature":1.0,"token_sha":"306e0fe05bc2","ttft_seconds":0.059404526},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.24801356,"delivery_tps":78.50952444915285,"end_to_end_seconds":3.307986506,"error_code":null,"fixture_id":"code","prefill_seconds":0.057949467,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":78.50840126160378,"status":"ok","temperature":1.0,"token_sha":"5a89476cf1f5","ttft_seconds":0.059037988},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.448749345,"delivery_tps":73.93984731586806,"end_to_end_seconds":3.544540757,"error_code":null,"fixture_id":"chat","prefill_seconds":0.094050582,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":73.93876038640492,"status":"ok","temperature":1.0,"token_sha":"460edda16244","ttft_seconds":0.095195552},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.488952348,"delivery_tps":73.08784258580536,"end_to_end_seconds":3.58242967,"error_code":null,"fixture_id":"chat","prefill_seconds":0.091283518,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":73.08653254052066,"status":"ok","temperature":1.0,"token_sha":"0eba9c9d3070","ttft_seconds":0.092819875},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.429184184,"delivery_tps":74.36171005039256,"end_to_end_seconds":3.521660621,"error_code":null,"fixture_id":"chat","prefill_seconds":0.090493505,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":74.36075963257915,"status":"ok","temperature":1.0,"token_sha":"15d18697d5f1","ttft_seconds":0.092000783},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.616247215,"delivery_tps":70.51509060062975,"end_to_end_seconds":3.710006137,"error_code":null,"fixture_id":"chat","prefill_seconds":0.091761291,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":70.51476632507715,"status":"ok","temperature":1.0,"token_sha":"d813ade7fc53","ttft_seconds":0.093469072},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.662028584,"delivery_tps":69.63353620835636,"end_to_end_seconds":3.757150529,"error_code":null,"fixture_id":"chat","prefill_seconds":0.092135778,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":69.6242956990888,"status":"ok","temperature":1.0,"token_sha":"7675315607f0","ttft_seconds":0.094256425},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.503125183,"delivery_tps":101.87265172826157,"end_to_end_seconds":2.558441621,"error_code":null,"fixture_id":"code","prefill_seconds":0.053257699,"prompt_tokens":14,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":101.85524463007886,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.054989359},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.500239603,"delivery_tps":101.99022513443485,"end_to_end_seconds":2.555581121,"error_code":null,"fixture_id":"code","prefill_seconds":0.052766597,"prompt_tokens":14,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":101.97124545480219,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.054677031},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.496235342,"delivery_tps":102.15382969287357,"end_to_end_seconds":2.552384181,"error_code":null,"fixture_id":"code","prefill_seconds":0.054032533,"prompt_tokens":14,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":102.15150866601722,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.055659256},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.498657708,"delivery_tps":102.05479493392058,"end_to_end_seconds":2.55358196,"error_code":null,"fixture_id":"code","prefill_seconds":0.053328897,"prompt_tokens":14,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":102.05369506179902,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.0546727},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":2.489292493,"delivery_tps":102.43874543352024,"end_to_end_seconds":2.545203491,"error_code":null,"fixture_id":"code","prefill_seconds":0.05439677,"prompt_tokens":14,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":102.4373590909349,"status":"ok","temperature":0.0,"token_sha":"9f2923ac14d0","ttft_seconds":0.055443064},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.287998041,"delivery_tps":77.5547907329182,"end_to_end_seconds":3.380643672,"error_code":null,"fixture_id":"chat","prefill_seconds":0.090520614,"prompt_tokens":31,"repeat":0,"seed":1234,"server_decode_seconds":null,"server_decode_tps":77.55396120223493,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.092159808},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.286176494,"delivery_tps":77.59777981054478,"end_to_end_seconds":3.376711398,"error_code":null,"fixture_id":"chat","prefill_seconds":0.088128057,"prompt_tokens":31,"repeat":1,"seed":1235,"server_decode_seconds":null,"server_decode_tps":77.59672763559524,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.089660125},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.242861839,"delivery_tps":78.63424735931218,"end_to_end_seconds":3.331759,"error_code":null,"fixture_id":"chat","prefill_seconds":0.087261058,"prompt_tokens":31,"repeat":2,"seed":1236,"server_decode_seconds":null,"server_decode_tps":78.633206679773,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.088589421},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.490464885,"delivery_tps":73.05617114093958,"end_to_end_seconds":3.580284215,"error_code":null,"fixture_id":"chat","prefill_seconds":0.087617345,"prompt_tokens":31,"repeat":3,"seed":1237,"server_decode_seconds":null,"server_decode_tps":73.05537833403073,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.088891012},{"cached_tokens":0,"completion_tokens":256,"delivery_seconds":3.381315769,"delivery_tps":75.41442959508464,"end_to_end_seconds":3.47076663,"error_code":null,"fixture_id":"chat","prefill_seconds":0.08749438,"prompt_tokens":31,"repeat":4,"seed":1238,"server_decode_seconds":null,"server_decode_tps":75.41348265434758,"status":"ok","temperature":0.0,"token_sha":"a8c28b8297b4","ttft_seconds":0.088971758}],"schema_version":1,"settings":{"context_tokens":null,"managed":true,"rank_count":1,"repetitions":5,"serial":false,"temperatures":[1.0,0.0],"tokens":256},"suite":{"id":"community-v1","sha256":"9bb717da00666b90b4006174abd9ed46108affc37456b43c64a224a9c938d5be"}} diff --git a/docs/receipts/rocm-host-identity/gpu-run/35b.log b/docs/receipts/rocm-host-identity/gpu-run/35b.log new file mode 100644 index 0000000000..bf94c059e4 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/35b.log @@ -0,0 +1,37 @@ +community-v1: code and chat, 2 temperatures, 5 repeats after warm-up. +This generates model replies; use Ctrl-C to cancel. Nothing is uploaded unless you publish. +The server gets up to 606 s to load 21 GB of weights; --startup-timeout sets another limit. +code T=1 warm-up: ok 79.24 delivery tok/s +code T=1 repeat 1/5: ok 78.65 delivery tok/s +code T=1 repeat 2/5: ok 86.00 delivery tok/s +code T=1 repeat 3/5: ok 88.60 delivery tok/s +code T=1 repeat 4/5: ok 78.24 delivery tok/s +code T=1 repeat 5/5: ok 78.51 delivery tok/s +chat T=1 warm-up: ok 71.15 delivery tok/s +chat T=1 repeat 1/5: ok 73.94 delivery tok/s +chat T=1 repeat 2/5: ok 73.09 delivery tok/s +chat T=1 repeat 3/5: ok 74.36 delivery tok/s +chat T=1 repeat 4/5: ok 70.52 delivery tok/s +chat T=1 repeat 5/5: ok 69.63 delivery tok/s +code T=0 warm-up: ok 101.81 delivery tok/s +code T=0 repeat 1/5: ok 101.87 delivery tok/s +code T=0 repeat 2/5: ok 101.99 delivery tok/s +code T=0 repeat 3/5: ok 102.15 delivery tok/s +code T=0 repeat 4/5: ok 102.05 delivery tok/s +code T=0 repeat 5/5: ok 102.44 delivery tok/s +chat T=0 warm-up: ok 75.67 delivery tok/s +chat T=0 repeat 1/5: ok 77.55 delivery tok/s +chat T=0 repeat 2/5: ok 77.60 delivery tok/s +chat T=0 repeat 3/5: ok 78.63 delivery tok/s +chat T=0 repeat 4/5: ok 73.06 delivery tok/s +chat T=0 repeat 5/5: ok 75.41 delivery tok/s +Run b46ba73a-e62a-4f31-83cd-98daa4ad5c91 | rocm | TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP +Workload Temp Stream delivery tok/s First text, s Status +code 1 78.65 0.059 complete +chat 1 73.09 0.093 complete +code 0 102.05 0.055 complete +chat 0 77.55 0.089 complete +Stream delivery includes transport buffering; it is not an engine token clock. +This rocm receipt is saved locally; it cannot be published until the board accepts rocm. +Receipt saved: ~/tf-1001-runs/out/35b.json +Draft counts: 2615 of 4490 drafted tokens accepted over 2495 rounds (20 of 20 attempts reported them); saved: ~/tf-1001-runs/out/35b.drafts.json diff --git a/docs/receipts/rocm-host-identity/gpu-run/binary.sha256 b/docs/receipts/rocm-host-identity/gpu-run/binary.sha256 new file mode 100644 index 0000000000..757094470c --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/binary.sha256 @@ -0,0 +1 @@ +e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247 ~/tf-1001/zig-out/native-hip/bin/tensorfold-native diff --git a/docs/receipts/rocm-host-identity/gpu-run/check.mjs b/docs/receipts/rocm-host-identity/gpu-run/check.mjs new file mode 100644 index 0000000000..7dd3a327cc --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/check.mjs @@ -0,0 +1,8 @@ +import {validateReceipt} from './protocol.mjs'; import fs from 'fs'; +for (const f of process.argv.slice(2)) { + const r = JSON.parse(fs.readFileSync(f)); + for (const backend of ['rocm','cuda']) { + const x = structuredClone(r); x.runtime.backend = backend; + try { validateReceipt(x); console.log(f, backend, 'ACCEPTED'); } catch (e) { console.log(f, backend, 'REFUSED:', e.message); } + } +} diff --git a/docs/receipts/rocm-host-identity/gpu-run/heads.txt b/docs/receipts/rocm-host-identity/gpu-run/heads.txt new file mode 100644 index 0000000000..b255214dc4 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/heads.txt @@ -0,0 +1,2 @@ +7b84f9098f2bf3a56adee3d87cbbae16ef7a6e48 +origin/main diff --git a/docs/receipts/rocm-host-identity/gpu-run/pull.log b/docs/receipts/rocm-host-identity/gpu-run/pull.log new file mode 100644 index 0000000000..e5f166bad9 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/pull.log @@ -0,0 +1,100 @@ +start 17:19:08 +[tensorfold] downloading TensorFold/Qwen3.8-27B-MLX-4bit@70ae7fac63274ff2eac54152031433374cb80f2f from Hugging Face +TensorFold/Qwen3.8-27B-MLX-4bit: Qwen3.5-2B or Qwen3.8-27B (qwen3_5) + .gitattributes: 0.00 MiB + LICENSE: 0.01 MiB + README.md: 0.01 MiB + chat_template.jinja: 0.01 MiB + config.json: 0.00 MiB + generation_config.json: 0.00 MiB + model.safetensors.index.json: 0.21 MiB + preprocessor_config.json: 0.00 MiB + processor_config.json: 0.00 MiB + qwen-logo.png: 0.01 MiB + tokenizer.json: 19.06 MiB + tokenizer_config.json: 0.00 MiB + video_preprocessor_config.json: 0.00 MiB + vocab.json: 6.41 MiB + 0.54 of 14.95 GiB, 55.1 MiB/s + 1.13 of 14.95 GiB, 57.8 MiB/s + 1.72 of 14.95 GiB, 58.6 MiB/s + 2.30 of 14.95 GiB, 58.7 MiB/s + 2.86 of 14.95 GiB, 58.5 MiB/s + 3.43 of 14.95 GiB, 58.6 MiB/s + 4.04 of 14.95 GiB, 59.0 MiB/s + 4.67 of 14.95 GiB, 59.7 MiB/s + 5.30 of 14.95 GiB, 60.2 MiB/s + 5.94 of 14.95 GiB, 60.8 MiB/s + 6.57 of 14.95 GiB, 61.2 MiB/s + 7.19 of 14.95 GiB, 61.3 MiB/s + 7.82 of 14.95 GiB, 61.6 MiB/s + 8.45 of 14.95 GiB, 61.8 MiB/s + 9.07 of 14.95 GiB, 61.9 MiB/s + 9.68 of 14.95 GiB, 61.9 MiB/s + 10.31 of 14.95 GiB, 62.1 MiB/s + 10.96 of 14.95 GiB, 62.3 MiB/s + 11.60 of 14.95 GiB, 62.5 MiB/s + 12.25 of 14.95 GiB, 62.7 MiB/s + 12.90 of 14.95 GiB, 62.9 MiB/s + 13.53 of 14.95 GiB, 63.0 MiB/s + 14.17 of 14.95 GiB, 63.1 MiB/s + 14.81 of 14.95 GiB, 63.2 MiB/s + model-00001-of-00003.safetensors: 5095.74 MiB in 160 pieces + model-00002-of-00003.safetensors: 5106.15 MiB in 160 pieces + model-00003-of-00003.safetensors: 5108.92 MiB in 160 pieces +stored TensorFold/Qwen3.8-27B-MLX-4bit@70ae7fac63274ff2eac54152031433374cb80f2f at ~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/70ae7fac63274ff2eac54152031433374cb80f2f +27b exit=0 17:23:41 +[tensorfold] downloading TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@f84b054c5677b6e59bdb969ae61915b0983d68bb from Hugging Face +TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP: qwen3_5_moe (qwen3_5_moe) + .gitattributes: 0.00 MiB + README.md: 0.00 MiB + chat_template.jinja: 0.01 MiB + config.json: 0.02 MiB + configuration.json: 0.00 MiB + generation_config.json: 0.00 MiB + model.safetensors.index.json: 0.21 MiB + preprocessor_config.json: 0.00 MiB + processor_config.json: 0.00 MiB + tokenizer.json: 19.06 MiB + tokenizer_config.json: 0.00 MiB + video_preprocessor_config.json: 0.00 MiB + vocab.json: 6.41 MiB + 0.58 of 19.44 GiB, 58.9 MiB/s + 1.21 of 19.44 GiB, 61.9 MiB/s + 1.84 of 19.44 GiB, 62.8 MiB/s + 2.46 of 19.44 GiB, 63.0 MiB/s + 3.10 of 19.44 GiB, 63.5 MiB/s + 3.73 of 19.44 GiB, 63.7 MiB/s + 4.37 of 19.44 GiB, 63.9 MiB/s + 5.00 of 19.44 GiB, 63.9 MiB/s + 5.63 of 19.44 GiB, 64.1 MiB/s + 6.29 of 19.44 GiB, 64.4 MiB/s + 6.94 of 19.44 GiB, 64.6 MiB/s + 7.57 of 19.44 GiB, 64.5 MiB/s + 8.20 of 19.44 GiB, 64.6 MiB/s + 8.82 of 19.44 GiB, 64.5 MiB/s + 9.41 of 19.44 GiB, 64.2 MiB/s + 10.04 of 19.44 GiB, 64.3 MiB/s + 10.70 of 19.44 GiB, 64.4 MiB/s + 11.28 of 19.44 GiB, 64.1 MiB/s + 11.93 of 19.44 GiB, 64.3 MiB/s + 12.59 of 19.44 GiB, 64.4 MiB/s + 13.21 of 19.44 GiB, 64.4 MiB/s + 13.82 of 19.44 GiB, 64.3 MiB/s + 14.46 of 19.44 GiB, 64.4 MiB/s + 15.11 of 19.44 GiB, 64.4 MiB/s + 15.75 of 19.44 GiB, 64.5 MiB/s + 16.37 of 19.44 GiB, 64.5 MiB/s + 17.00 of 19.44 GiB, 64.5 MiB/s + 17.63 of 19.44 GiB, 64.4 MiB/s + 18.26 of 19.44 GiB, 64.5 MiB/s + 18.89 of 19.44 GiB, 64.5 MiB/s + 19.41 of 19.44 GiB, 64.1 MiB/s + model-00001-of-00004.safetensors: 5043.22 MiB in 158 pieces + model-00002-of-00004.safetensors: 5119.77 MiB in 160 pieces + model-00003-of-00004.safetensors: 5119.63 MiB in 160 pieces + model-00004-of-00004.safetensors: 4174.43 MiB in 131 pieces + mtp-4bit.safetensors: 453.37 MiB in 15 pieces +stored TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@f84b054c5677b6e59bdb969ae61915b0983d68bb at ~/.cache/huggingface/hub/models--TensorFold--Qwen3.6-35B-A3B-MLX-4bit-MTP/snapshots/f84b054c5677b6e59bdb969ae61915b0983d68bb +35b exit=0 17:29:21 +PULLDONE diff --git a/docs/receipts/rocm-host-identity/gpu-run/run.log b/docs/receipts/rocm-host-identity/gpu-run/run.log new file mode 100644 index 0000000000..b90e37a8c1 --- /dev/null +++ b/docs/receipts/rocm-host-identity/gpu-run/run.log @@ -0,0 +1,57 @@ +hint: Using 'master' as the name for the initial branch. This default branch name +hint: will change to "main" in Git 3.0. To configure the initial branch name +hint: to use in all of your new repositories, which will suppress this warning, +hint: call: +hint: +hint: git config --global init.defaultBranch +hint: +hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and +hint: 'development'. The just-created branch can be renamed via this command: +hint: +hint: git branch -m +hint: +hint: Disable this message with "git config set advice.defaultBranchName false" +warning: remote HEAD refers to nonexistent ref, unable to checkout +Trying simple merge with origin/hip-refuses-drafter +Simple merge did not work, trying automatic merge. +Auto-merging docs/community-benchmarks.md +Auto-merging zig/src/server/benchmark.zig +Trying simple merge with origin/benchmark-drafter-counts +Simple merge did not work, trying automatic merge. +Auto-merging docs/community-benchmarks.md +Auto-merging zig/src/server/benchmark.zig +Auto-merging zig/src/server/benchmark_run_test.zig +fatal: ambiguous argument 'origin/main': unknown revision or path not in the working tree. +Use '--' to separate paths from revisions, like this: +'git [...] -- [...]' +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/ubuntu--vg-ubuntu--lv 1.9T 1.7T 76G 96% / + total used free shared buff/cache available +Mem: 122 96 2 0 24 26 +~/.cache/huggingface/hub/models--TensorFold--Qwen3.6-35B-A3B-MLX-4bit-MTP/snapshots: +f84b054c5677b6e59bdb969ae61915b0983d68bb + +~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots: +70ae7fac63274ff2eac54152031433374cb80f2f +llm-server: failed +gtt_used=1129861120 busy=0 t=17:31:19 +build_exit=0 +e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247 ~/tf-1001/zig-out/native-hip/bin/tensorfold-native +gtt_used=1129861120 busy=0 t=17:33:19 +R1 exit=0 +gtt_used=1129906176 busy=81 t=17:42:02 +R2a exit=1 +ls: cannot access '~/tf-1001-runs/out/27b-dflash2.json': No such file or directory +gtt_used=1129906176 busy=78 t=17:42:02 +R2b exit=1 +gtt_used=1129861120 busy=57 t=17:42:02 +gtt_used=1129861120 busy=56 t=17:42:02 +R3 exit=0 +gtt_used=1129906176 busy=76 t=17:43:35 +R4 exit=0 +gtt_used=1129906176 busy=84 t=17:45:19 +R5 exit=0 +gtt_used=1682337792 busy=42 t=17:47:01 +RUNDONE +llm-server: active +llm-server chat OK after 200s: {"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"","reasoning_content":"The user's request is simple. They"}}],"created":1792345812,"model":"Qwen3.8-Flash-Next","system_fingerprint":"b111-d807f04","object":"chat.completion","usag diff --git a/docs/receipts/rocm-host-identity/host-controls/board-validator.log b/docs/receipts/rocm-host-identity/host-controls/board-validator.log new file mode 100644 index 0000000000..789b93784c --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/board-validator.log @@ -0,0 +1,15 @@ +# the board's validator on each receipt, re-run 2026-10-19T07:56:16Z on the Mac with node v26.9.0 +# protocol.mjs: https://tensorfold.dev/benchmarks/protocol.js as fetched 18 Oct 2026, sha256 0252160a717f4f50e12f9b019beff760ee10e415503ee504d72fc6289fc9cc4b +# check.mjs: gpu-run/check.mjs, sha256 23b78e745c0519e1c012e62e17302f81f3ffb4a8441bb2cb0c3a780e80829466; each receipt as rocm, then a copy with runtime.backend cuda +# cmd: node check.mjs gpu-run/27b-auto.json gpu-run/35b.json gpu-run/35b-serial.json gpu-run/35b-attached.json gpu-attached-rerun/35b-attached.json +gpu-run/27b-auto.json rocm REFUSED: Unsupported benchmark runtime. +gpu-run/27b-auto.json cuda ACCEPTED +gpu-run/35b.json rocm REFUSED: Unsupported benchmark runtime. +gpu-run/35b.json cuda ACCEPTED +gpu-run/35b-serial.json rocm REFUSED: Unsupported benchmark runtime. +gpu-run/35b-serial.json cuda ACCEPTED +gpu-run/35b-attached.json rocm REFUSED: Unsupported benchmark runtime. +gpu-run/35b-attached.json cuda ACCEPTED +gpu-attached-rerun/35b-attached.json rocm REFUSED: Unsupported benchmark runtime. +gpu-attached-rerun/35b-attached.json cuda ACCEPTED +# exit=0 diff --git a/docs/receipts/rocm-host-identity/host-controls/lint.log b/docs/receipts/rocm-host-identity/host-controls/lint.log new file mode 100644 index 0000000000..d85237013f --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/lint.log @@ -0,0 +1,9 @@ +# lint on fb8eaf85081880f686bbb0b12a5852f64b81fe26 against main 76aafcf2, 2026-10-19T08:53:07Z, the same steps CI runs +# zig fmt --check build.zig zig (zig 0.17.0) +# exit=0 +# python3 tools/zig/lean_check.py on main 76aafcf2 (git archive of zig/ and tools/zig/, this branch's lean_check.py) and on this branch +# main: 2625 files, 503 problems +# branch: 2626 files, 503 problems +# python3 tools/zig/lean_new.py main.txt branch.txt +lean: 503 problems, 503 on the base, 0 new +# exit=0 diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions-minimal.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions-minimal.log new file mode 100644 index 0000000000..b5d9edbeb9 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions-minimal.log @@ -0,0 +1,49 @@ +# negative control: agent-positions +# undone: the new fix: drop an unreportable rocminfo GPU agent before the visible-device indices apply +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:28:24Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark_host.zig b/zig/src/server/benchmark_host.zig +index 9d2e8529..dd4cc753 100644 +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -81,8 +81,8 @@ pub fn amdAgents(a: Allocator, text: []const u8) []const ?Gpu { + std.fmt.allocPrint(a, "{s} {s}", .{ vendor, marketing }) catch return &.{} + else + marketing; +- const name = modelName(a, full); +- out.append(a, if (name) |n| .{ .name = n, .memory_bytes = g.memory_bytes, .cores = g.cores } else null) catch return &.{}; ++ const n = modelName(a, full) orelse continue; ++ out.append(a, .{ .name = n, .memory_bytes = g.memory_bytes, .cores = g.cores }) catch return &.{}; + } + return out.items; + } +# --- build output --- +test-server-cpu ++- run test 433 pass, 1 fail (434 total) +error: 'benchmark_facts_test.test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next' failed: + HIP_VISIBLE_DEVICES=1: listed GPUs [], want [AMD Radeon PRO V620] + HIP_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620], want [] + ROCR_VISIBLE_DEVICES=1: listed GPUs [], want [AMD Radeon PRO V620] + ROCR_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620], want [] + expected 0, found 4 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x1006edc97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x10073d2c7 in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:435:5: 0x10073d38f in test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next (test) + try std.testing.expectEqual(@as(usize, 0), wrong); + ^ +failed command: ./.zc/o/b1636d7ae7e004fb025de77034673e5d/test --cache-dir=./.zc --seed=0xedd01c48 --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 433/434 tests passed (1 failed) +test-server-cpu transitive failure ++- run test 433 pass, 1 fail (434 total) +| +- compile test debug native success 29s MaxRSS:1G ++- run test cached + +- compile test debug native cached 29s MaxRSS:32M + +# exit=1 after 111s diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions.log new file mode 100644 index 0000000000..3e7932e959 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-agent-positions.log @@ -0,0 +1,88 @@ +# negative control: agent positions (re-run 2026-10-19T07:52:39Z) +# undone: the agent-position fix. zig/src/server/benchmark_host.zig is put back to its state before the fix +# (0bdb1a52: unreportable agents dropped before the visible-device indices apply); the only addition is the +# rocmVisible() helper the new test calls. Everything else is 82237515afa0572705c2803b6b633e9e5cef1b02. +# cmd: zig build test-server-cpu -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches: --cache-dir and ZIG_GLOBAL_CACHE_DIR) +# free disk before: 13 GB +# --- the undo, as git diff against HEAD --- +diff --git a/zig/src/server/benchmark_host.zig b/zig/src/server/benchmark_host.zig +index 9d2e8529..88d66016 100644 +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -23,7 +23,7 @@ pub fn detect(a: Allocator, io: std.Io, env: *const std.process.Environ.Map, bac + const cpu = cpuinfo(a, readProc(a, io, "/proc/cpuinfo") orelse ""); + const memory = meminfo(readProc(a, io, "/proc/meminfo") orelse "") orelse (std.process.totalSystemMemory() catch null); + const out = if (gpuQuery(backend)) |argv| run(a, io, argv) orelse "" else ""; +- if (std.mem.eql(u8, backend, "rocm")) return rocm(a, cpu, memory, out, rocmVisible(env)); ++ if (std.mem.eql(u8, backend, "rocm")) return rocm(a, cpu, memory, out, env.get("HIP_VISIBLE_DEVICES") orelse env.get("ROCR_VISIBLE_DEVICES")); + return linux(a, cpu, memory, out, env.get("CUDA_VISIBLE_DEVICES")); + } + +@@ -41,12 +41,8 @@ pub fn rocmVisible(env: *const std.process.Environ.Map) ?[]const u8 { + + /// ROCm facts from rocminfo's GPU agents by HIP_VISIBLE_DEVICES; an APU ("... Graphics") is ``unified``. + pub fn rocm(a: Allocator, cpu: ?[]const u8, memory: ?u64, text: []const u8, visible: ?[]const u8) Probe { +- const none: Probe = .{ .hardware = summarize(cpu, memory, &.{}, false) }; +- // indices count every GPU agent, so selecting an agent with no reportable name lists no GPU, never the next one +- const slots = select(?Gpu, a, amdAgents(a, text), visible) orelse return none; +- var kept: std.ArrayList(Gpu) = .empty; +- for (slots) |slot| if (slot) |g| kept.append(a, g) catch return none; +- const gpus = kept.items; ++ const all = amdGpus(a, text); ++ const gpus = select(Gpu, a, all, visible) orelse return .{ .hardware = summarize(cpu, memory, &.{}, false) }; + var h = summarize(cpu, memory, gpus, false); + if (gpus.len > 0) h.memory_type = for (gpus) |g| { + if (!std.mem.endsWith(u8, g.name, " Graphics")) break "dedicated"; +@@ -54,9 +50,9 @@ pub fn rocm(a: Allocator, cpu: ?[]const u8, memory: ?u64, text: []const u8, visi + return .{ .hardware = h }; + } + +-/// Every rocminfo GPU agent in order (vendor-prefixed name, first GLOBAL pool, compute units); null if a name fails. +-pub fn amdAgents(a: Allocator, text: []const u8) []const ?Gpu { +- var out: std.ArrayList(?Gpu) = .empty; ++/// GPU agents from ``rocminfo``: the vendor-prefixed marketing name, the first GLOBAL pool size and the compute units. ++pub fn amdGpus(a: Allocator, text: []const u8) []const Gpu { ++ var out: std.ArrayList(Gpu) = .empty; + var agents = std.mem.splitSequence(u8, text, "*******"); + while (agents.next()) |block| { + var marketing: []const u8 = ""; +@@ -81,8 +77,8 @@ pub fn amdAgents(a: Allocator, text: []const u8) []const ?Gpu { + std.fmt.allocPrint(a, "{s} {s}", .{ vendor, marketing }) catch return &.{} + else + marketing; +- const name = modelName(a, full); +- out.append(a, if (name) |n| .{ .name = n, .memory_bytes = g.memory_bytes, .cores = g.cores } else null) catch return &.{}; ++ g.name = modelName(a, full) orelse continue; ++ out.append(a, g) catch return &.{}; + } + return out.items; + } +# --- build output --- +test-server-cpu ++- run test 433 pass, 1 fail (434 total) +error: 'benchmark_facts_test.test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next' failed: + HIP_VISIBLE_DEVICES=1: listed GPUs [], want [AMD Radeon PRO V620] + HIP_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620], want [] + ROCR_VISIBLE_DEVICES=1: listed GPUs [], want [AMD Radeon PRO V620] + ROCR_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620], want [] + expected 0, found 4 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x101501c97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x1015512c7 in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:435:5: 0x10155138f in test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next (test) + try std.testing.expectEqual(@as(usize, 0), wrong); + ^ +failed command: ./.zc/o/c4899e586703dcbaa807a56bccdeda78/test --cache-dir=./.zc --seed=0xf2371a6 --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 433/434 tests passed (1 failed) +test-server-cpu transitive failure ++- run test 433 pass, 1 fail (434 total) +| +- compile test debug native success 12s MaxRSS:1G ++- run test cached + +- compile test debug native cached 50ms MaxRSS:31M + +# exit=1 after 63s +# restored: git status of tracked files: 0 modified diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-backend-name.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-backend-name.log new file mode 100644 index 0000000000..343c840b1f --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-backend-name.log @@ -0,0 +1,54 @@ +# negative control: backend-name +# undone: commit 5: the manifest names the engine (hip, metal) instead of the receipt's backend +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:30:15Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark_manifest.zig b/zig/src/server/benchmark_manifest.zig +index 601f8fdb..c0275f72 100644 +--- a/zig/src/server/benchmark_manifest.zig ++++ b/zig/src/server/benchmark_manifest.zig +@@ -197,8 +197,6 @@ pub fn attestedRevision(env: *const std.process.Environ.Map, model_dir: []const + + /// The receipt's name for an engine backend, as the board names them: HIP is ``rocm``, Metal is ``mlx``. + pub fn receiptBackend(engine: []const u8) []const u8 { +- if (std.mem.eql(u8, engine, "hip")) return "rocm"; +- if (std.mem.eql(u8, engine, "metal")) return "mlx"; + return engine; + } + +# --- build output --- +test-server-cpu ++- run test 433 pass, 1 fail (434 total) +ver/benchmark_facts_test.zig:98:9: 0x104d32223 in test.a HIP or Metal server's manifest names the receipt's backend, so an attached rocm or mlx run reads it (test) + try std.testing.expectEqualStrings(pair[1], manifest.receiptBackend(pair[0])); + ^ +error: 'benchmark_facts_test.test.a HIP or Metal server's manifest names the receipt's backend, so an attached rocm or mlx run reads it' failed: + ====== expected this output: ========= + rocm␃ + + ======== instead found this: ========= + hip␃ + + ====================================== + First difference occurs on line 1: + expected: + rocm + ^ ('\x72') + found: + hip + ^ ('\x68') + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:696:9: 0x104b524c7 in expectEqualStrings (test) + return error.TestExpectedEqual; + ^ + ~/tmp/rv1001-1/zig/src/ser +failed command: ./.zc/o/ffa7cb62923c1037c8e4a3cac9e22339/test --cache-dir=./.zc --seed=0xd9c0c5f5 --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 433/434 tests passed (1 failed) +test-server-cpu transitive failure ++- run test 433 pass, 1 fail (434 total) +| +- compile test debug native success 17s MaxRSS:1G ++- run test cached + +- compile test debug native cached 64ms MaxRSS:31M + +# exit=1 after 75s diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-cores.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-cores.log new file mode 100644 index 0000000000..dd20e7e3e2 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-cores.log @@ -0,0 +1,64 @@ +# negative control: cores +# undone: commit 3: the manifest sends cores as null and the client does not read them +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:32:45Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark_manifest.zig b/zig/src/server/benchmark_manifest.zig +index 601f8fdb..08573982 100644 +--- a/zig/src/server/benchmark_manifest.zig ++++ b/zig/src/server/benchmark_manifest.zig +@@ -147,7 +147,8 @@ fn hardwareOf(a: Allocator, v: Value) ?receipt.Hardware { + if (n < 0 or n > 100000) return null; + break :blk @as(?u64, @intCast(n)); + } else null; +- slot.* = .{ .name = name, .memory_bytes = mem, .cores = cores }; ++ _ = cores; ++ slot.* = .{ .name = name, .memory_bytes = mem }; + } + const count = v.get("gpu_count") orelse return null; + if (count != .null) { +@@ -233,7 +234,7 @@ pub fn serveDocument(a: Allocator, io: std.Io, env: *const std.process.Environ.M + const one = try json.newObject(a); + try one.put(a, "name", .{ .string = g.name }); + if (g.memory_bytes) |n| try one.put(a, "memory_bytes", try json.intValue(a, n)) else try one.put(a, "memory_bytes", .null); +- if (g.cores) |n| try one.put(a, "cores", try json.intValue(a, n)) else try one.put(a, "cores", .null); ++ try one.put(a, "cores", .null); + slot.* = .{ .object = one }; + } + try hardware.put(a, "gpus", .{ .array = gpus }); +# --- build output --- +test-server-cpu ++- run test 432 pass, 2 fail (434 total) +error: 'benchmark_facts_test.test.a server's manifest names the checkpoint a managed run records: the snapshot's revision and the same hashes' failed: + expected 40, found null + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:173:21: 0x102ae0653 in expectEqualInner__func_4293 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x102af39db in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:88:9: 0x102af3a5b in test.a server's manifest names the checkpoint a managed run records: the snapshot's revision and the same hashes (test) + try std.testing.expectEqual(want.cores, got.cores); + ^ +error: 'benchmark_facts_test.test.a manifest's GPU cores reach the attached receipt; cores past the board's range refuse the manifest' failed: + expected 40, found null + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:173:21: 0x102ae0653 in expectEqualInner__func_4293 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x102aef34b in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:139:5: 0x102aef407 in test.a manifest's GPU cores reach the attached receipt; cores past the board's range refuse the manifest (test) + try std.testing.expectEqual(@as(?u64, 40), facts.hardware.gpus[0].cores); + ^ +failed command: ./.zc/o/133c044ea333fd8d1591b6942de3a118/test --cache-dir=./.zc --seed=0xf23c77fe --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 432/434 tests passed (2 failed) +test-server-cpu transitive failure ++- run test 432 pass, 2 fail (434 total) +| +- compile test debug native success 15s MaxRSS:1G ++- run test cached + +- compile test debug native cached 57ms MaxRSS:32M + +# exit=1 after 72s diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-hip-list.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-hip-list.log new file mode 100644 index 0000000000..2e7d0c6273 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-hip-list.log @@ -0,0 +1,78 @@ +# negative control: HIP's list rules (2026-10-19T08:55:27Z) +# undone: fb8eaf85 part 2. The list is parsed again by 82237515's select() (entries trimmed, out-of-range indices skipped, repeats kept, any other entry voids the list, no GPU- entries); hipVisible() stays in the file, unused. Everything else is fb8eaf85. +# tree: a scratch copy of fb8eaf85081880f686bbb0b12a5852f64b81fe26 (git archive) with only this change +# cmd: zig build test-server-cpu -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches: --cache-dir and ZIG_GLOBAL_CACHE_DIR) +# free disk before: 52 GB +# --- the undo, as a diff against fb8eaf85 --- +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -56,7 +56,7 @@ + const none: Probe = .{ .hardware = summarize(cpu, memory, &.{}, false) }; + var sure = true; + // indices count every GPU agent, so selecting an agent with no reportable name lists no GPU, never the next one +- const agents = hipVisible(a, amdAgents(a, text), visible, &sure) orelse return none; ++ const agents = select(Agent, a, amdAgents(a, text), visible) orelse return none; + var kept: std.ArrayList(Gpu) = .empty; + for (agents) |agent| { + const g = agent.gpu orelse { +@@ -121,6 +121,22 @@ + if (end == 0 or !std.mem.endsWith(u8, value, " KB")) return null; + const n = std.fmt.parseInt(u64, value[0..end], 10) catch return null; + return if (n > 0 and n < 1 << 43) n * 1024 else null; ++} ++ ++/// Items kept by a comma-separated index list (``CUDA_VISIBLE_DEVICES`` rules); null when the list is malformed. ++fn select(comptime T: type, a: Allocator, items: []const T, visible: ?[]const u8) ?[]const T { ++ const v = visible orelse return items; ++ const sel = std.mem.trim(u8, v, " \t\n\r"); ++ if (sel.len == 0 or std.mem.eql(u8, sel, "-1")) return &.{}; ++ var kept: std.ArrayList(T) = .empty; ++ var it = std.mem.splitScalar(u8, v, ','); ++ while (it.next()) |raw| { ++ const index = std.mem.trim(u8, raw, " \t"); ++ for (index) |ch| if (!std.ascii.isDigit(ch)) return null; ++ const n = std.fmt.parseInt(usize, index, 10) catch return null; ++ if (n < items.len) kept.append(a, items[n]) catch return null; ++ } ++ return kept.items; + } + + /// The agents HIP keeps (clr Device::init): the first bad or out-of-range entry ends the list; a repeat is skipped. +# --- build output --- +test-server-cpu ++- run test 435 pass, 1 fail (436 total) +error: 'benchmark_rocm_test.test.HIP's list: the first bad or out-of-range entry ends it, a repeat is skipped, and GPU- text matches rocminfo's Uuid' failed: + HIP_VISIBLE_DEVICES=0,3,2 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO W7800, AMD Instinct MI210] detected, want [AMD Radeon PRO W7800] detected + HIP_VISIBLE_DEVICES=0,0 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO W7800, AMD Radeon PRO W7800] detected, want [AMD Radeon PRO W7800] detected + HIP_VISIBLE_DEVICES=2,0,2 (rocminfo lists 3 GPUs): recorded [AMD Instinct MI210, AMD Radeon PRO W7800, AMD Instinct MI210] detected, want [AMD Instinct MI210, AMD Radeon PRO W7800] detected + HIP_VISIBLE_DEVICES=1,x,0 (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Radeon PRO V620] detected + HIP_VISIBLE_DEVICES=1,,0 (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Radeon PRO V620] detected + HIP_VISIBLE_DEVICES=0, (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Radeon PRO W7800] detected + HIP_VISIBLE_DEVICES= 1 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO V620] detected, want [] partial + HIP_VISIBLE_DEVICES=01 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO V620] detected, want [] partial + HIP_VISIBLE_DEVICES=GPU-c3c3c3c3c3c3c3c3 (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Instinct MI210] detected + HIP_VISIBLE_DEVICES=GPU-c3c3,0 (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Instinct MI210, AMD Radeon PRO W7800] detected + HIP_VISIBLE_DEVICES=1,GPU-0000000000000000,0 (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Radeon PRO V620] detected + HIP_VISIBLE_DEVICES=1,GPU-b2b2 (rocminfo lists 2 GPUs): recorded [] partial, want [AMD Radeon PRO V620] partial + expected 0, found 12 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x10154dc97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x10159da5b in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1-nc/zig/src/server/benchmark_rocm_test.zig:101:5: 0x10159db17 in test.HIP's list: the first bad or out-of-range entry ends it, a repeat is skipped, and GPU- text matches rocminfo's Uuid (test) + try std.testing.expectEqual(@as(usize, 0), try wrong(a, &cases)); + ^ +failed command: ./.zc/o/0b5eabaa4cbe5a3d8394df8bfd4af328/test --cache-dir=./.zc --seed=0xdc4b282 --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 435/436 tests passed (1 failed) +test-server-cpu transitive failure ++- run test 435 pass, 1 fail (436 total) +| +- compile test debug native success 10s MaxRSS:1G ++- run test cached + +- compile test debug native cached 41ms MaxRSS:32M + + +# exit=1 after 57s +# restored: benchmark_host.zig is byte-identical to fb8eaf85's again diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-partial.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-partial.log new file mode 100644 index 0000000000..8158cc11d3 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-partial.log @@ -0,0 +1,57 @@ +# negative control: partial source (2026-10-19T08:56:24Z) +# undone: fb8eaf85 part 3. The line that makes the source partial when a GPU HIP keeps is unnamed, or a GPU- entry is unresolved, is removed. Everything else is fb8eaf85. +# tree: a scratch copy of fb8eaf85081880f686bbb0b12a5852f64b81fe26 (git archive) with only this change +# cmd: zig build test-server-cpu -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches: --cache-dir and ZIG_GLOBAL_CACHE_DIR) +# free disk before: 52 GB +# --- the undo, as a diff against fb8eaf85 --- +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -67,8 +67,6 @@ + } + const gpus = kept.items; + var h = summarize(cpu, memory, gpus, false); +- // a GPU HIP keeps that has no reportable name, or an entry no Uuid resolves, leaves the list short +- if (!sure and std.mem.eql(u8, h.source, "detected")) h.source = "partial"; + if (gpus.len > 0) h.memory_type = for (gpus) |g| { + if (!std.mem.endsWith(u8, g.name, " Graphics")) break "dedicated"; + } else "unified"; +# --- build output --- +test-server-cpu ++- run test 434 pass, 2 fail (436 total) +error: 'benchmark_facts_test.test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next' failed: + =: listed GPUs [AMD Radeon PRO V620] detected, want [AMD Radeon PRO V620] partial + HIP_VISIBLE_DEVICES=0,1: listed GPUs [AMD Radeon PRO V620] detected, want [AMD Radeon PRO V620] partial + expected 0, found 2 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x105189c97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x1051dffa7 in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1-nc/zig/src/server/benchmark_facts_test.zig:436:5: 0x1051e007b in test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next (test) + try std.testing.expectEqual(@as(usize, 0), wrong); + ^ +error: 'benchmark_rocm_test.test.HIP's list: the first bad or out-of-range entry ends it, a repeat is skipped, and GPU- text matches rocminfo's Uuid' failed: + HIP_VISIBLE_DEVICES=1,GPU-b2b2 (rocminfo lists 2 GPUs): recorded [AMD Radeon PRO V620] detected, want [AMD Radeon PRO V620] partial + expected 0, found 1 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x105189c97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x1051d9a5b in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1-nc/zig/src/server/benchmark_rocm_test.zig:101:5: 0x1051d9b17 in test.HIP's list: the first bad or out-of-range entry ends it, a repeat is skipped, and GPU- text matches rocminfo's Uuid (test) + try std.testing.expectEqual(@as(usize, 0), try wrong(a, &cases)); + ^ +failed command: ./.zc/o/278d6ad411505b921b61e782f256ffb6/test --cache-dir=./.zc --seed=0x7e02a47a --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 434/436 tests passed (2 failed) +test-server-cpu transitive failure ++- run test 434 pass, 2 fail (436 total) +| +- compile test debug native success 9s MaxRSS:1G ++- run test cached + +- compile test debug native cached 41ms MaxRSS:31M + + +# exit=1 after 56s +# restored: benchmark_host.zig is byte-identical to fb8eaf85's again diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-ranking-note.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-ranking-note.log new file mode 100644 index 0000000000..23d168888b --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-ranking-note.log @@ -0,0 +1,47 @@ +# negative control: ranking-note +# undone: commit 4: the ranking note never names a missing fact (unranked always null) +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:35:06Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark.zig b/zig/src/server/benchmark.zig +index af4d5cc5..da882c58 100644 +--- a/zig/src/server/benchmark.zig ++++ b/zig/src/server/benchmark.zig +@@ -248,7 +248,7 @@ fn show(ctx: *const Context, v: json.Value) !void { + /// What a public managed receipt lacks of the facts the board ranks by (docs/community-benchmarks.md), else null. + pub fn unranked(a: Allocator, v: json.Value) !?[]const u8 { + const m = v.get("model").?; +- if (m.get("local").? != .bool or m.get("local").?.bool) return null; // a local label is refused before publishing ++ if (m.get("local").? != .null) return null; // a local label is refused before publishing + var lacks: std.ArrayList([]const u8) = .empty; + const gpus = v.get("hardware").?.get("gpus").?; + if (gpus != .array or gpus.array.len == 0) try lacks.append(a, "GPU"); +# --- build output --- +test-server-cpu ++- run test 433 pass, 1 crash (434 total) +error: 'benchmark_facts_test.test.a rocm probe names rocminfo's GPU agents, runs no nvidia-smi, reports no driver, and passes the save rules' terminated with signal ABRT with stderr: + thread 30124164 panic: attempt to use null value + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:347:82: 0x10514b737 in test.a rocm probe names rocminfo's GPU agents, runs no nvidia-smi, reports no driver, and passes the save rules (test) + try std.testing.expectEqualStrings("GPU", (try benchmark.unranked(a, no_gpu)).?); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:143:68: 0x104c5e0d7 in mainServer (test) + const status: TestResults.Status = if (test_fn.func()) |v| s: { + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:72:26: 0x104c5e6e3 in main (test) + return mainServer(init) catch |err| panic("internal test runner failure: {t}", .{err}); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/start.zig:789:88: 0x104c5c62f in callMain (test) + if (fn_info.param_types[0].? == std.process.Init.Minimal) return wrapMain(root.main(.{ + ^ + ???:?:?: 0x180dabe7f in start (/usr/lib/dyld) +failed command: ./.zc/o/5fb1dfbed58610ba73f32230b85fe5ee/test --cache-dir=./.zc --seed=0x522fb6aa --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 433/434 tests passed (1 crashed) +test-server-cpu transitive failure ++- run test 433 pass, 1 crash (434 total) +| +- compile test debug native success 13s MaxRSS:1G ++- run test cached + +- compile test debug native cached 61ms MaxRSS:32M + +# exit=1 after 82s diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-revision.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-revision.log new file mode 100644 index 0000000000..fcd38fdafb --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-revision.log @@ -0,0 +1,66 @@ +# negative control: revision +# undone: commit 3: the manifest's revision back to TF_CHECKPOINT_REVISION only +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:31:31Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark_manifest.zig b/zig/src/server/benchmark_manifest.zig +index 601f8fdb..389125be 100644 +--- a/zig/src/server/benchmark_manifest.zig ++++ b/zig/src/server/benchmark_manifest.zig +@@ -189,10 +189,8 @@ fn revisionFromEnv(env: *const std.process.Environ.Map) ?[]const u8 { + + /// The attested revision: the snapshot directory's name, else TF_CHECKPOINT_REVISION; a mismatch gives none. + pub fn attestedRevision(env: *const std.process.Environ.Map, model_dir: []const u8) ?[]const u8 { +- const told = revisionFromEnv(env); +- const loaded = (if (model_dir.len > 0) model_facts.snapshotRevision(model_dir) else null) orelse return told; +- if (told) |t| if (!std.mem.eql(u8, t, loaded)) return null; +- return loaded; ++ _ = model_dir; ++ return revisionFromEnv(env); + } + + /// The receipt's name for an engine backend, as the board names them: HIP is ``rocm``, Metal is ``mlx``. +# --- build output --- +test-server-cpu ++- run test 432 pass, 2 crash (434 total) +error: 'benchmark_facts_test.test.a server's manifest names the checkpoint a managed run records: the snapshot's revision and the same hashes' terminated with signal ABRT with stderr: + thread 30062092 panic: attempt to use null value + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:79:77: 0x102c43467 in test.a server's manifest names the checkpoint a managed run records: the snapshot's revision and the same hashes (test) + try std.testing.expectEqualStrings(managed.revision.?, attached.revision.?); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:143:68: 0x10274a0d7 in mainServer (test) + const status: TestResults.Status = if (test_fn.func()) |v| s: { + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:72:26: 0x10274a6e3 in main (test) + return mainServer(init) catch |err| panic("internal test runner failure: {t}", .{err}); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/start.zig:789:88: 0x10274862f in callMain (test) + if (fn_info.param_types[0].? == std.process.Init.Minimal) return wrapMain(root.main(.{ + ^ + ???:?:?: 0x180dabe7f in start (/usr/lib/dyld) +error: 'benchmark_facts_test.test.the attested revision: the loaded snapshot's; the environment's only where the directory names none; never a disagreeing one' terminated with signal ABRT with stderr: + thread 30070747 panic: attempt to use null value + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:117:91: 0x10150b76f in test.the attested revision: the loaded snapshot's; the environment's only where the directory names none; never a disagreeing one (test) + try std.testing.expectEqualStrings(revision, manifest.attestedRevision(&env, snapshot).?); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:143:68: 0x1010160d7 in mainServer (test) + const status: TestResults.Status = if (test_fn.func()) |v| s: { + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:72:26: 0x1010166e3 in main (test) + return mainServer(init) catch |err| panic("internal test runner failure: {t}", .{err}); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/start.zig:789:88: 0x10101462f in callMain (test) + if (fn_info.param_types[0].? == std.process.Init.Minimal) return wrapMain(root.main(.{ + ^ + ???:?:?: 0x180dabe7f in start (/usr/lib/dyld) +failed command: ./.zc/o/26d97a7f501ff9c8681ae72bdc02dabf/test --cache-dir=./.zc --seed=0x667de105 --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 432/434 tests passed (2 crashed) +test-server-cpu transitive failure ++- run test 432 pass, 2 crash (434 total) +| +- compile test debug native success 16s MaxRSS:1G ++- run test cached + +- compile test debug native cached 16s MaxRSS:31M + +# exit=1 after 74s diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-rocr-twice.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-rocr-twice.log new file mode 100644 index 0000000000..ef40a26db7 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-rocr-twice.log @@ -0,0 +1,67 @@ +# negative control: ROCR_VISIBLE_DEVICES applied twice (2026-10-19T08:54:29Z) +# undone: fb8eaf85 part 1. rocmVisible() is put back as it was in 82237515 (HIP_VISIBLE_DEVICES, else ROCR_VISIBLE_DEVICES, taken as given; CUDA_VISIBLE_DEVICES not read). Everything else is fb8eaf85. +# tree: a scratch copy of fb8eaf85081880f686bbb0b12a5852f64b81fe26 (git archive) with only this change +# cmd: zig build test-server-cpu -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches: --cache-dir and ZIG_GLOBAL_CACHE_DIR) +# free disk before: 52 GB +# --- the undo, as a diff against fb8eaf85 --- +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -34,12 +34,9 @@ + return &.{ "nvidia-smi", "--query-gpu=name,memory.total,driver_version", "--format=csv,noheader,nounits" }; + } + +-/// HIP's device list as clr reads it (rocdevice.cpp): HIP_VISIBLE_DEVICES, else CUDA_VISIBLE_DEVICES; null for none. ++/// The visible-device list the rocm probe applies: HIP_VISIBLE_DEVICES, else ROCR_VISIBLE_DEVICES. + pub fn rocmVisible(env: *const std.process.Environ.Map) ?[]const u8 { +- // ROCR_VISIBLE_DEVICES is not read: ROCr applies it inside rocminfo, which lists only those GPUs, in its order +- const hip = clrString(env.get("HIP_VISIBLE_DEVICES")); +- const list = if (hip.len > 0) hip else clrString(env.get("CUDA_VISIBLE_DEVICES")); +- return if (list.len > 0) list else null; ++ return env.get("HIP_VISIBLE_DEVICES") orelse env.get("ROCR_VISIBLE_DEVICES"); + } + + /// A string variable as clr's flags.cpp reads it: unset is "", set but empty is " ", and one pair of quotes is dropped. +# --- build output --- +test-server-cpu ++- run test 434 pass, 2 fail (436 total) +error: 'benchmark_facts_test.test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next' failed: + CUDA_VISIBLE_DEVICES=1: listed GPUs [AMD Radeon PRO V620] partial, want [AMD Radeon PRO V620] detected + CUDA_VISIBLE_DEVICES=0: listed GPUs [AMD Radeon PRO V620] partial, want [] partial + expected 0, found 2 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x100e45c97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x100e9bc97 in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1-nc/zig/src/server/benchmark_facts_test.zig:436:5: 0x100e9bd6b in test.rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next (test) + try std.testing.expectEqual(@as(usize, 0), wrong); + ^ +error: 'benchmark_rocm_test.test.rocminfo lists only the GPUs ROCR_VISIBLE_DEVICES leaves, renumbered; the probe applies HIP's list on top, once' failed: + ROCR_VISIBLE_DEVICES=1 (rocminfo lists 1 GPUs): recorded [] partial, want [AMD Radeon PRO V620] detected + ROCR_VISIBLE_DEVICES=1,2 (rocminfo lists 2 GPUs): recorded [AMD Instinct MI210] detected, want [AMD Radeon PRO V620, AMD Instinct MI210] detected + CUDA_VISIBLE_DEVICES=1 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO W7800, AMD Radeon PRO V620, AMD Instinct MI210] detected, want [AMD Radeon PRO V620] detected + HIP_VISIBLE_DEVICES= CUDA_VISIBLE_DEVICES=1 (rocminfo lists 3 GPUs): recorded [AMD Radeon PRO W7800, AMD Radeon PRO V620, AMD Instinct MI210] detected, want [] partial + HIP_VISIBLE_DEVICES="1" (rocminfo lists 3 GPUs): recorded [] partial, want [AMD Radeon PRO V620] detected + expected 0, found 5 + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:93:17: 0x100e45c97 in expectEqualInner__func_4100 (test) + return error.TestExpectedEqual; + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/testing.zig:57:5: 0x100e9ad07 in expectEqual (test) + return expectEqualInner(T, expected, actual); + ^ + ~/tmp/rv1001-1-nc/zig/src/server/benchmark_rocm_test.zig:74:5: 0x100e9adc3 in test.rocminfo lists only the GPUs ROCR_VISIBLE_DEVICES leaves, renumbered; the probe applies HIP's list on top, once (test) + try std.testing.expectEqual(@as(usize, 0), try wrong(a, &cases)); + ^ +failed command: ./.zc/o/5a6b9981cfb84793a98bf037f1e23fd0/test --cache-dir=./.zc --seed=0xf37f3f6f --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 439/441 tests passed (2 failed) +test-server-cpu transitive failure ++- run test 434 pass, 2 fail (436 total) +| +- compile test debug native success 9s MaxRSS:1G ++- run test 5 pass (5 total) 4ms MaxRSS:3M + +- compile test debug native success 1s MaxRSS:284M + + +# exit=1 after 58s +# restored: benchmark_host.zig is byte-identical to fb8eaf85's again diff --git a/docs/receipts/rocm-host-identity/host-controls/negative-control-w-slash.log b/docs/receipts/rocm-host-identity/host-controls/negative-control-w-slash.log new file mode 100644 index 0000000000..b665769cc1 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/negative-control-w-slash.log @@ -0,0 +1,47 @@ +# negative control: w-slash +# undone: commit 2: AMD's "w/" kept as is in CPU names +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 with only this change +# cmd: zig build test-server-cpu --summary all -j8 (zig 0.17.0) +# started 2026-10-18T23:33:57Z +# --- the undo, as git diff --- +diff --git a/zig/src/server/benchmark_host.zig b/zig/src/server/benchmark_host.zig +index 9d2e8529..09aec809 100644 +--- a/zig/src/server/benchmark_host.zig ++++ b/zig/src/server/benchmark_host.zig +@@ -146,7 +146,7 @@ pub fn modelName(a: Allocator, text: []const u8) ?[]const u8 { + while (it.next()) |w| { + if (words.items.len > 0) words.append(a, ' ') catch return null; + // AMD's "w/" (Ryzen AI MAX+ 395 w/ Radeon 8060S) becomes "with", inside the board's name pattern +- words.appendSlice(a, if (std.mem.eql(u8, w, "w/")) "with" else w) catch return null; ++ words.appendSlice(a, w) catch return null; + } + const name = dropClock(words.items); + if (name.len > 120 or std.mem.eql(u8, name, "NVIDIA NPU")) return null; +# --- build output --- +test-server-cpu ++- run test 433 pass, 1 crash (434 total) +error: 'benchmark_facts_test.test.a rocm probe names rocminfo's GPU agents, runs no nvidia-smi, reports no driver, and passes the save rules' terminated with signal ABRT with stderr: + thread 30108405 panic: attempt to use null value + ~/tmp/rv1001-1/zig/src/server/benchmark_facts_test.zig:308:86: 0x102dea09b in test.a rocm probe names rocminfo's GPU agents, runs no nvidia-smi, reports no driver, and passes the save rules (test) + try std.testing.expectEqualStrings("AMD RYZEN AI MAX+ 395 with Radeon 8060S", cpu.?); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:143:68: 0x1028fe0d7 in mainServer (test) + const status: TestResults.Status = if (test_fn.func()) |v| s: { + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/compiler/test_runner.zig:72:26: 0x1028fe6e3 in main (test) + return mainServer(init) catch |err| panic("internal test runner failure: {t}", .{err}); + ^ + ~/zig/zig-aarch64-macos-0.17.0/lib/std/start.zig:789:88: 0x1028fc62f in callMain (test) + if (fn_info.param_types[0].? == std.process.Init.Minimal) return wrapMain(root.main(.{ + ^ + ???:?:?: 0x180dabe7f in start (/usr/lib/dyld) +failed command: ./.zc/o/97e58ce43cc8cbe0ebf27027c2418313/test --cache-dir=./.zc --seed=0xd1a67e2b --listen=- + +Build Summary: 3/5 steps succeeded (1 failed); 433/434 tests passed (1 crashed) +test-server-cpu transitive failure ++- run test 433 pass, 1 crash (434 total) +| +- compile test debug native success 14s MaxRSS:1G ++- run test cached + +- compile test debug native cached 60ms MaxRSS:32M + +# exit=1 after 69s diff --git a/docs/receipts/rocm-host-identity/host-controls/test-branch-82237515.log b/docs/receipts/rocm-host-identity/host-controls/test-branch-82237515.log new file mode 100644 index 0000000000..1b074f4036 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/test-branch-82237515.log @@ -0,0 +1,895 @@ +# tree: 82237515afa0572705c2803b6b633e9e5cef1b02 (Benchmark: a ROCm visible-device index counts every GPU agent); tracked files modified: 0 +# cmd: zig build test -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches; --cache-dir .zc-fresh, 0 entries at start) +# free disk before: 13 GB +# started 2026-10-19T07:56:02Z +test ++- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +failed command: ./.zc-fresh/o/61f4f42007c910ae0dba3d28a8174637/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): a round could not be recorded: OutOfDeviceMemory +failed command: ./.zc-fresh/o/63a68c3197a335020863279090304b9e/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): glm: restoring 1500 tokens failed (PoolMoveNotDrained); pool relocation aborted +failed command: ./.zc-fresh/o/9c900e13443c4ea3333643efc5df6815/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/1Y6Fmpr8NJrnqOjs/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc-fresh/o/349ecedab64728c1d4cb34081643bae7/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): language_model.model.layers.0: gate and up of one expert differ in width; one kernel reads both +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/vAyyy_KwnMHCPzXZ/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +failed command: ./.zc-fresh/o/66e7715555d593184cc8933f0d2b5b60/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): lane host: stopping at a round boundary: 1 running and 2 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc-fresh/o/a8b7681f29e9e489e82bddc04264a5b8/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc-fresh/o/1844b49f35e97abb72ec962a7301e3a0/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): XPU: a command list did not drain at close; its memory and context stay until the process ends +failed command: ./.zc-fresh/o/3f248dbeaaa9b5d0dd34af53033b384e/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc-fresh/o/23dbfa9b8beb9644eee3e2bb8f8e4a30/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc-fresh/o/741429c191ea4ad9080f53b9ec01cb9b/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc-fresh/o/808af24a156c04f8e6f71e66c3685cfc/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +fake target: proposed=68 verified=68 accepted=68 +failed command: ./.zc-fresh/o/5da12ea727f3e84cfd24c1a7d5d10f12/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/Pppk8rMt4Rp-JkmR/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc-fresh/o/6912cf8414324d74bdfa6475ced95635/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +[default] (warn): qwen3_5: a kept state's host copy goes to the copy-source quarantine: a restore's copies did not drain +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +failed command: ./.zc-fresh/o/4da0700838cc80b80192261eb806325d/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc-fresh/o/97b70a34a2e73ccfed358053c4ec2c04/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (err): rank 0: another rank's process ended (its line closed); this rank stops instead of waiting on it +[default] (err): rank 0: rank 1 stopped (peer OOM); this rank stops too +failed command: ./.zc-fresh/o/131776b8122fc64da97a138979c510ed/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 0 cannot exchange with rank 2; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 1 cannot exchange with rank 3; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 2 cannot exchange with rank 0; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 3 cannot exchange with rank 1; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +failed command: ./.zc-fresh/o/f62cf985d371fb96e718b0c5ca7a0615/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc-fresh/o/a529abb901551c852fa65e556ac33565/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +failed command: ./.zc-fresh/o/65e976b591c0054618177133701a6e9c/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc-fresh/o/1e3ede8afa26a6226aa15360dad4f91c/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Blocks +tf_h3_forward_host: BadH3Blocks +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Scalar +H3 model retained at close: NotReady +H3 geometry retained: NotReady +tensor blocks.0.norm1.weight: MissingTensor +tensor blocks.1.attn.qkv_proj.weight: MissingTensor +tensor blocks.1.norm1.weight: MissingTensor +failed command: ./.zc-fresh/o/e6adccd01a88898d9cea1ee2f65e724a/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lane host: stopping at a round boundary: 2 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc-fresh/o/3715e5673fd04f7ffc906ecb69c0e4d0/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): checkpoint tensor text.weight stays in its file: read it through a Stage +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/Sdvd0sX9Jrarnr2x/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=false heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000004647133651849988, 0.00000008929183081818337 } +SSM seeded=false heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000004647133651849988, 0.00000010044378595530645 } +SSM seeded=false heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.0000000051661870337227356, 0.00000006044416009376441 } +SSM seeded=false heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.000000008099387559939997, 0.00000010302938735495815 } +SSM seeded=false heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.000000008099387559939997, 0.00000010076810685877291 } +SSM seeded=false heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.000000008099387559939997, 0.000000170725301229524 } +SSM seeded=false heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.000000014270247941650105, 0.00000013346487892640368 } +SSM seeded=false heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.000000003777481768357793, 0.00000016323549628660672 } +SSM seeded=false heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000005027532001822266, 0.00000017245745953520242 } +SSM seeded=false heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000005457366610700509, 0.00000029433054438143813 } +SSM seeded=false heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000008944659693477242, 0.0000003865198301422623 } +SSM seeded=false heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000008944659693477242, 0.0000003074925702950537 } +SSM seeded=false heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.00000001164840428280467, 0.00000030358411029496324 } +SSM seeded=false heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.000000022576584234046404, 0.0000004419375038144935 } +SSM seeded=false heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000010395453171885283, 0.0000002193045710030539 } +SSM seeded=false heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000001602812140649723, 0.0000007270377315737875 } +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=true heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000015089179933314156, 0.00000003066629741810267 } +SSM seeded=true heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000015656003028718946, 0.00000011578910319443025 } +SSM seeded=true heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000015656003028718946, 0.000000137397136557027 } +SSM seeded=true heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.000000015656003028718946, 0.000000058752570086806344 } +SSM seeded=true heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.00000003896021144589312, 0.00000011965777954592305 } +SSM seeded=true heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.00000004227448897886621, 0.0000001134548608394681 } +SSM seeded=true heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.00000004227448897886621, 0.00000018029616666304094 } +SSM seeded=true heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.0000000506942888084394, 0.0000001136397048684401 } +SSM seeded=true heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.0000000073519402093991104, 0.000000027996068407976082 } +SSM seeded=true heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.00000001807481535276434, 0.00000021659550952790596 } +SSM seeded=true heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000018969054375872485, 0.00000017414204611654327 } +SSM seeded=true heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000018969054375872485, 0.00000017655989847931153 } +SSM seeded=true heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000023961633413560612, 0.0000003531072296780735 } +SSM seeded=true heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000023961633413560612, 0.00000024600469961111315 } +SSM seeded=true heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.000000023961633413560612, 0.0000003112589632703333 } +SSM seeded=true heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.00000002654540304236619, 0.0000004793090875310213 } +SSM seeded=true heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000013799211689979884, 0.00000008033819287600608 } +SSM seeded=true heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000027007305641868484, 0.0000002670173361307704 } +SSM seeded=true heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000003555922037379311, 0.0000006937729284395289 } +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): qwen image: num_layers is not 32; this family runs the released geometry only +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 32; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 128; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): model.layers.3.mlp.up_proj: another spelling of module layers.3.mlp.up_proj declares a different width +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +sparse attention (64 heads): 99.97% of outputs equal the row kernel's, at most 0.96 bf16 steps apart +sparse attention (32 heads): 99.96% of outputs equal the row kernel's, at most 0.94 bf16 steps apart +absorb on the tensor units: 99.99% of outputs equal the row kernel's, at most 0.98 bf16 steps apart +KDA heads=64 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=64 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +KDA heads=32 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=32 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +[default] (warn): language_model.model.layers.1.mlp.down_proj: another spelling of module layers.1.mlp.down_proj declares a different width +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x2, 4-bit/64 x35, 5-bit/32 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.57 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x37, 5-bit/64 x4, 6-bit/64 x2, 8-bit/64 x4 +glm plan: 4 of 45 layers and the MTP layer, 1.86 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x3, 4-bit/64 x34, 5-bit/64 x1, 6-bit/64 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +failed command: ./.zc-fresh/o/51b970c29ff0c197a8175e956e974b4d/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +test ++- run test w +[default] (warn): language_model.lm_head: stored affine but declared unquantized +[default] (warn): language_model.model.embed_tokens: stored affine but declared unquantized +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc-fresh/o/8871d809081e7b7104c8626a39069d70/test --cache-dir=./.zc-fresh --seed=0xfa9d0e8a --listen=- + +Build Summary: 292/292 steps succeeded; 4317/4333 tests passed (16 skipped) +test success ++- run test 37 pass (37 total) 63ms MaxRSS:8M +| +- compile test debug native-native success 3s MaxRSS:311M ++- run test 7 pass (7 total) 398ms MaxRSS:71M +| +- compile test debug native success 2s MaxRSS:310M ++- run test 9 pass (9 total) 113ms MaxRSS:4M +| +- compile test debug native success 3s MaxRSS:284M +| +- options success +| +- compile exe digest-writer debug native success 2s MaxRSS:267M ++- run test 6 pass (6 total) 8ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:270M ++- test-flashnext-batch-host success +| +- run test 6 pass (6 total) 2s MaxRSS:30M +| +- compile test debug native success 4s MaxRSS:397M +| +- WriteFile flashnext-test-empty.zig success ++- run test 2 pass (2 total) 33ms MaxRSS:3M +| +- compile test debug native-native success 2s MaxRSS:268M ++- run test 4 pass (4 total) 56ms MaxRSS:3M +| +- compile test debug native-native success 4s MaxRSS:269M ++- run test 4 pass (4 total) 11ms MaxRSS:3M +| +- compile test debug native-native success 2s MaxRSS:284M ++- run test 3 pass (3 total) 6ms MaxRSS:3M +| +- compile test debug native-native success 2s MaxRSS:270M ++- run test 23 pass (23 total) 9ms MaxRSS:3M +| +- compile test debug native-native success 3s MaxRSS:284M ++- run test 5 pass (5 total) 11ms MaxRSS:7M +| +- compile test debug native-native success 2s MaxRSS:331M ++- run test 19 pass (19 total) 7ms MaxRSS:6M +| +- compile test fast native-native success 13s MaxRSS:490M +| +- WriteFile drama3.metallib cached +| +- run xcrun (drama3.metallib) cached +| +- run xcrun (d3_dense.air) cached +| +- run xcrun (d3_gdn.air) cached +| +- run xcrun (d3_attn.air) cached +| +- run xcrun (d3_moe.air) cached +| +- run xcrun (d3_sample.air) cached +| +- run xcrun (d3_codec.air) cached ++- run test 131 pass (131 total) 952ms MaxRSS:21M +| +- compile test fast native-native success 30s MaxRSS:869M +| +- WriteFile draft_ids.txt success +| +- WriteFile nemotron_packed.metallib success +| +- run xcrun (nemotron_packed.metallib) success 470ms +| +- run xcrun (coop_down_1_0.air) success 311ms +| +- run xcrun (coop_down_2_0.air) success 300ms +| +- run xcrun (coop_down_2_1.air) success 329ms +| +- run xcrun (coop_draft_1_0.air) success 313ms +| +- run xcrun (coop_draft_2_0.air) success 1s +| +- run xcrun (coop_draft_2_1.air) success 294ms +| +- run xcrun (coop_eh_1_0.air) success 303ms +| +- run xcrun (coop_eh_1_0_sk.air) success 308ms +| +- run xcrun (coop_eh_2_0.air) success 309ms +| +- run xcrun (coop_eh_2_0_sk.air) success 279ms +| +- run xcrun (coop_eh_2_1.air) success 302ms +| +- run xcrun (coop_eh_2_1_sk.air) success 308ms +| +- run xcrun (coop_head_1_0.air) success 308ms +| +- run xcrun (coop_head_2_0.air) success 283ms +| +- run xcrun (coop_head_2_1.air) success 308ms +| +- run xcrun (coop_in_1_0.air) success 284ms +| +- run xcrun (coop_in_1_0_sk.air) success 294ms +| +- run xcrun (coop_in_2_0.air) success 292ms +| +- run xcrun (coop_in_2_0_sk.air) success 305ms +| +- run xcrun (coop_in_2_1.air) success 325ms +| +- run xcrun (coop_in_2_1_sk.air) success 1s +| +- run xcrun (coop_out_1_0.air) success 291ms +| +- run xcrun (coop_out_1_0_sk.air) success 1s +| +- run xcrun (coop_out_2_0.air) success 1s +| +- run xcrun (coop_out_2_0_sk.air) success 309ms +| +- run xcrun (coop_out_2_1.air) success 302ms +| +- run xcrun (coop_out_2_1_sk.air) success 291ms +| +- run xcrun (coop_qkv_1_0.air) success 302ms +| +- run xcrun (coop_qkv_2_0.air) success 309ms +| +- run xcrun (coop_qkv_2_1.air) success 296ms +| +- run xcrun (coop_up_1_0.air) success 293ms +| +- run xcrun (coop_up_2_0.air) success 292ms +| +- run xcrun (coop_up_2_1.air) success 300ms +| +- run xcrun (up_relu2_1_0.air) success 296ms +| +- run xcrun (up_relu2_2_0.air) success 327ms +| +- run xcrun (up_relu2_2_1.air) success 328ms ++- run test 77 pass (77 total) 58ms MaxRSS:9M +| +- compile test fast native-native success 26s MaxRSS:740M ++- run test 38 pass (38 total) 80ms MaxRSS:5M +| +- compile test fast native-native success 20s MaxRSS:467M ++- run test 86 pass (86 total) 81ms MaxRSS:10M +| +- compile test fast native-native success 29s MaxRSS:762M ++- run test 9 pass (9 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 11s MaxRSS:409M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 7ms MaxRSS:6M +| +- compile test fast native-native success 10s MaxRSS:401M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- test-exl3 success +| +- compile obj exl3_metal_loader debug native-native success 311ms MaxRSS:101M +| | +- WriteFile exl3_projection.metal success +| +- run test 11 pass (11 total) 21ms MaxRSS:7M +| | +- compile test fast native-native success 11s MaxRSS:427M +| | +- WriteFile exl3_projection.metal (reused) +| +- run test 2 pass (2 total) 8ms MaxRSS:6M +| +- compile test fast native-native success 10s MaxRSS:398M +| +- WriteFile exl3_projection.metal (reused) ++- run test 4 pass (4 total) 7ms MaxRSS:6M +| +- compile test fast native-native success 10s MaxRSS:403M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 19 pass (19 total) 1s MaxRSS:24M +| +- compile test safe native-native success 35s MaxRSS:1G +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile drama3_core.zig success +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal success +| +- WriteFile drama3.metallib success +| | +- run xcrun (drama3.metallib) success 389ms +| | +- run xcrun (d3_dense.air) success 1s +| | +- run xcrun (d3_gdn.air) success 1s +| | +- run xcrun (d3_attn.air) success 66ms +| | +- run xcrun (d3_moe.air) success 1s +| | +- run xcrun (d3_sample.air) success 1s +| | +- run xcrun (d3_codec.air) success 88ms +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 9 pass (9 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 11s MaxRSS:413M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 13ms MaxRSS:7M +| +- compile test fast native-native success 10s MaxRSS:415M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 8ms MaxRSS:6M +| +- compile test fast native-native success 10s MaxRSS:409M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 1 pass (1 total) 7ms MaxRSS:6M +| +- compile test safe native-native success 11s MaxRSS:413M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 5 pass (5 total) 4ms MaxRSS:2M +| +- compile test safe native-native success 10s MaxRSS:373M ++- run test 10 pass (10 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 12s MaxRSS:437M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 4ms MaxRSS:2M +| +- compile test safe native-native success 11s MaxRSS:368M ++- run test 4 pass (4 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 9s MaxRSS:371M ++- run test 1 pass (1 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 11s MaxRSS:408M ++- run test 112 pass (112 total) 240ms MaxRSS:16M +| +- compile test safe native-native success 21s MaxRSS:655M +| +- WriteFile xsum.metal (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 7 pass (7 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 12s MaxRSS:395M ++- run test 1 pass (1 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 11s MaxRSS:408M ++- run sh success 78ms +| +- compile exe tensorfold-native safe native-native success 3m MaxRSS:4G +| +- options success +| +- WriteFile drama3_core.zig (reused) +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile drama3.metallib (+1 more reused dependencies) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 434 pass (434 total) 1m MaxRSS:96M +| +- compile test debug native-native success 31s MaxRSS:1G ++- run test 5 pass (5 total) 7ms MaxRSS:3M +| +- compile test debug native-native success 3s MaxRSS:283M ++- run test 25 pass (25 total) 36s MaxRSS:236M +| +- compile test debug native-native success 3s MaxRSS:334M ++- run test 15 pass (15 total) 272ms MaxRSS:33M +| +- compile test debug native-native success 4s MaxRSS:388M ++- run test 204 pass (204 total) 1s MaxRSS:12M +| +- compile test fast native-native success 35s MaxRSS:991M ++- run test 8 pass (8 total) 215ms MaxRSS:14M +| +- compile test fast native-native success 11s MaxRSS:427M ++- run test 1 pass (1 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 9s MaxRSS:398M +| +- WriteFile tensorfold.metallib success +| +- run xcrun (tensorfold.metallib) success 309ms +| +- run xcrun (runtime_tests.air) success 109ms +| +- run xcrun (nemotron_expert_up.air) success 134ms +| +- run xcrun (nemotron_expert_down.air) success 51ms ++- run test 648 pass, 3 skip (651 total) 24s MaxRSS:162M +| +- compile test fast native-native success 1m MaxRSS:2G +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 163 pass (163 total) 329ms MaxRSS:29M +| +- compile test fast native-native success 30s MaxRSS:789M ++- run test 30 pass, 1 skip (31 total) 42ms MaxRSS:2M +| +- compile test fast native-native success 14s MaxRSS:490M ++- run test 1 pass (1 total) 4ms MaxRSS:2M +| +- compile test fast native-native success 10s MaxRSS:346M ++- run test 6 pass (6 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 9s MaxRSS:404M ++- run test 4 pass (4 total) 3ms MaxRSS:2M +| +- compile test fast native-native success 10s MaxRSS:365M ++- test-cuda-host success +| +- run test 1 pass (1 total) 14ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:266M +| +- run test 30 pass (30 total) 166ms MaxRSS:4M +| | +- compile test debug native success 5s MaxRSS:371M +| | +- WriteFile h3-compile-only.fatbin success +| | +- options cached +| +- run test 3 pass (3 total) 684ms MaxRSS:70M +| | +- compile test debug native success 4s MaxRSS:289M +| +- run test 10 pass (10 total) 1s MaxRSS:15M +| | +- compile test debug native success 4s MaxRSS:413M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- compile obj exl3_host_compile debug native success 282ms MaxRSS:111M +| | +- options (reused) +| +- run test 6 pass (6 total) 13ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:306M +| | +- options (reused) +| +- run test 6 pass (6 total) 109ms MaxRSS:5M +| | +- compile test debug native success 2s MaxRSS:289M +| | +- options (reused) +| +- run test 8 pass (8 total) 43ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:278M +| +- run test 3 pass (3 total) 117ms MaxRSS:5M +| | +- compile test debug native success 3s MaxRSS:273M +| +- run test 8 pass (8 total) 22ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:286M +| +- run test 4 pass (4 total) 7ms MaxRSS:3M +| | +- compile test debug native success 2s MaxRSS:283M +| | +- options (reused) +| +- run test 1 pass (1 total) 54ms MaxRSS:4M +| | +- compile test debug native success 2s MaxRSS:282M +| | +- options (reused) +| +- run test 4 pass (4 total) 4ms MaxRSS:2M +| | +- compile test fast native success 9s MaxRSS:372M +| | +- options (reused) +| +- run test 1 pass (1 total) 12ms MaxRSS:3M +| | +- compile test fast native success 10s MaxRSS:383M +| | +- options (reused) +| +- compile obj bolt_host_compile debug aarch64-linux-gnu success 982ms MaxRSS:140M +| | +- options cached +| | +- WriteFile draft_ids.txt (reused) +| +- run test 26 pass (26 total) 753ms MaxRSS:6M +| | +- compile test debug native success 4s MaxRSS:373M +| | +- options (reused) +| +- compile obj drama_cuda_native_host debug native success 411ms MaxRSS:125M +| | +- options (reused) +| +- run test 5 pass (5 total) 50ms MaxRSS:3M +| | +- compile test debug native success 5s MaxRSS:274M +| +- run test 434 pass (434 total) 1m MaxRSS:96M +| | +- compile test debug native success 29s MaxRSS:1G +| +- run test 5 pass (5 total) 26ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:284M +| +- run test 130 pass (130 total) 71ms MaxRSS:4M +| | +- compile test debug native success 3s MaxRSS:372M +| | +- options (reused) +| +- run test 280 pass, 2 skip (282 total) 11s MaxRSS:132M +| | +- compile test debug native success 11s MaxRSS:587M +| +- run test 163 pass (163 total) 10s MaxRSS:65M +| | +- compile test debug native success 8s MaxRSS:473M +| +- run test 74 pass, 1 skip (75 total) 224ms MaxRSS:10M +| | +- compile test debug native success 5s MaxRSS:498M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- run test 33 pass (33 total) 8s MaxRSS:24M +| | +- compile test debug native success 3s MaxRSS:340M +| | +- options (reused) +| +- run test 12 pass (12 total) 24ms MaxRSS:3M +| | +- compile test debug native success 2s MaxRSS:327M +| | +- options (reused) +| +- run test 83 pass (83 total) 253ms MaxRSS:74M +| | +- compile test debug native success 4s MaxRSS:440M +| | +- options (reused) +| +- run test 13 pass (13 total) 7s MaxRSS:15M +| | +- compile test debug native success 3s MaxRSS:389M +| | +- options (reused) +| +- run test 29 pass (29 total) 4s MaxRSS:1G +| | +- compile test debug native success 3s MaxRSS:339M +| | +- options (reused) +| +- run test 5 pass (5 total) 33ms MaxRSS:3M +| | +- compile test debug native success 5s MaxRSS:285M +| +- run test 20 pass (20 total) 294ms MaxRSS:5M +| | +- compile test debug native success 5s MaxRSS:343M +| +- run test 204 pass (204 total) 7s MaxRSS:42M +| | +- compile test debug native success 10s MaxRSS:535M +| +- run test 52 pass, 1 skip (53 total) 102ms MaxRSS:24M +| | +- compile test debug native success 8s MaxRSS:653M +| | +- options (reused) +| | +- WriteFile draft_ids.txt (reused) +| +- run test 3 pass (3 total) 6ms MaxRSS:3M +| | +- compile test debug native success 5s MaxRSS:265M +| +- run test 52 pass (52 total) 45ms MaxRSS:70M +| | +- compile test debug native success 3s MaxRSS:388M +| | +- options (reused) +| +- run test 5 pass (5 total) 16ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:272M +| +- run test 25 pass (25 total) 49s MaxRSS:236M +| | +- compile test debug native success 6s MaxRSS:334M +| +- run test 1 pass (1 total) 6ms MaxRSS:3M +| +- compile test debug native success 5s MaxRSS:293M +| +- options (reused) +| +- options (reused) +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile exlr8_alias.zig cached ++- run test 4 pass (4 total) 5ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:270M +| +- options success ++- run test 13 pass, 1 skip (14 total) 82ms MaxRSS:4M +| +- compile test debug native success 3s MaxRSS:314M +| +- options (reused) ++- run test 3 pass (3 total) 7ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:288M +| +- options (reused) ++- run test 1 pass (1 total) 5ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:268M ++- test-kv-pool success +| +- run test 9 pass (9 total) 14ms MaxRSS:3M +| | +- compile test debug native success 12s MaxRSS:272M +| +- run test 6 pass (6 total) 3s MaxRSS:33M +| +- compile test debug native success 4s MaxRSS:387M ++- run test 51 pass (51 total) 56ms MaxRSS:3M +| +- compile test debug native success 3s MaxRSS:323M +| +- options success +| | +- compile lib hip-mock-success debug native success 129ms MaxRSS:54M +| | +- compile lib hip-mock-failed debug native success 128ms MaxRSS:54M +| | +- compile lib hip-mock-missing debug native success 136ms MaxRSS:55M +| +- WriteFile kernels.zig cached ++- run test 23 pass, 1 skip (24 total) 153ms MaxRSS:5M +| +- compile test debug native success 3s MaxRSS:353M ++- run test 49 pass (49 total) 44ms MaxRSS:9M +| +- compile test debug native success 3s MaxRSS:350M +| +- WriteFile kernels.zig success ++- run test 30 pass, 1 skip (31 total) 125ms MaxRSS:5M +| +- compile test debug native success 6s MaxRSS:479M +| +- WriteFile kernels.zig (reused) ++- run test 18 pass (18 total) 302ms MaxRSS:7M +| +- compile test debug native success 6s MaxRSS:498M +| +- WriteFile kernels.zig (reused) ++- run test 4 pass (4 total) 22ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:291M +| +- WriteFile kernels.zig (reused) ++- run test 3 pass (3 total) 6ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:287M +| +- WriteFile kernels.zig (reused) ++- compile exe tf-qwen35-test debug native success 2s MaxRSS:170M +| +- WriteFile kernels.zig (reused) ++- compile exe tensorfold-native safe native success 4s MaxRSS:269M +| +- options (reused) +| +- WriteFile exlr8_alias.zig success +| +- WriteFile kernels.zig (reused) ++- run test 26 pass, 5 skip (31 total) 18ms MaxRSS:6M +| +- compile test safe native-native success 18s MaxRSS:539M ++- run test 112 pass (112 total) 3s MaxRSS:41M +| +- compile test debug native success 4s MaxRSS:393M ++- run test 20 pass (20 total) 157ms MaxRSS:11M +| +- compile test debug native success 2s MaxRSS:323M ++- run exe learned-warning-test success 10ms + +- compile exe learned-warning-test debug native success 2s MaxRSS:289M + +# exit=0 after 347s; free disk after: 11 GB diff --git a/docs/receipts/rocm-host-identity/host-controls/test-branch-fb8eaf85.log b/docs/receipts/rocm-host-identity/host-controls/test-branch-fb8eaf85.log new file mode 100644 index 0000000000..8c11c5de35 --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/test-branch-fb8eaf85.log @@ -0,0 +1,896 @@ +# tree: fb8eaf85081880f686bbb0b12a5852f64b81fe26 (Benchmark: the ROCm probe keeps the GPUs HIP keeps, by HIP's own rules); tracked files modified: 0 +# cmd: zig build test -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, private caches; --cache-dir .zc, 0 entries at start) +# free disk before: 53 GB +# started 2026-10-19T08:51:02Z +test ++- run test w +[default] (warn): XPU: a command list did not drain at close; its memory and context stay until the process ends +failed command: ./.zc/o/a035baca141746470fdd79d98e4bd318/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): a round could not be recorded: OutOfDeviceMemory +failed command: ./.zc/o/ab142b73a9ddf64416746eff122e7f0e/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc/o/9cf35b7c97a750582887951e166a28a0/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Blocks +tf_h3_forward_host: BadH3Blocks +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Scalar +H3 model retained at close: NotReady +H3 geometry retained: NotReady +tensor blocks.0.norm1.weight: MissingTensor +tensor blocks.1.attn.qkv_proj.weight: MissingTensor +tensor blocks.1.norm1.weight: MissingTensor +failed command: ./.zc/o/e6adccd01a88898d9cea1ee2f65e724a/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): language_model.lm_head: stored affine but declared unquantized +[default] (warn): language_model.model.embed_tokens: stored affine but declared unquantized +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc/o/9f63054eab90b79cd0b299d06030acf6/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/bfd49b30f01a1da60d864b93cfd65fe7/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (err): rank 0: another rank's process ended (its line closed); this rank stops instead of waiting on it +[default] (err): rank 0: rank 1 stopped (peer OOM); this rank stops too +failed command: ./.zc/o/8ec590ba6ed2d72fc185605c4dd487d6/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/31f5b798031187d553372885124d3301/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +failed command: ./.zc/o/ab65e0f66ea77cde48d72442e9cca88c/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +[default] (warn): qwen3_5: a kept state's host copy goes to the copy-source quarantine: a restore's copies did not drain +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +failed command: ./.zc/o/05449d8c36226b88db3cde489418f86d/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +fake target: proposed=68 verified=68 accepted=68 +failed command: ./.zc/o/5da12ea727f3e84cfd24c1a7d5d10f12/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/b6936474859580162f36d6543107666d/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/6642c13248ce1f799331ca52b40a8126/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 0 cannot exchange with rank 2; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 1 cannot exchange with rank 3; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 2 cannot exchange with rank 0; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 3 cannot exchange with rank 1; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +failed command: ./.zc/o/2d07c8f61ec9bafcd37e5ab2f870292f/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc/o/e0ea7ee08193b81611d6148c2f24f818/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): language_model.model.layers.0: gate and up of one expert differ in width; one kernel reads both +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/ZbzgIO32kTLouY3j/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +failed command: ./.zc/o/924c752002f5ccfb6fbbbc756614956e/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lane host: stopping at a round boundary: 2 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc/o/8286c42e04309b6b657847cfa92a9a4a/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc/o/e09fea28ddca31cf72552f09dfa2601f/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/hIOUDdwz-M7eZUNV/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc/o/2ffaf39d1cc35e03de0a441e7f33b58c/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): glm: restoring 1500 tokens failed (PoolMoveNotDrained); pool relocation aborted +failed command: ./.zc/o/6b6158b82ecbb5fa70b973b30e0ff0fe/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): checkpoint tensor text.weight stays in its file: read it through a Stage +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/kHUiPhyHtMUlZU09/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=false heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000004647133651849988, 0.00000008929183081818337 } +SSM seeded=false heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000004647133651849988, 0.00000010044378595530645 } +SSM seeded=false heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.0000000051661870337227356, 0.00000006044416009376441 } +SSM seeded=false heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.000000008099387559939997, 0.00000010302938735495815 } +SSM seeded=false heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.000000008099387559939997, 0.00000010076810685877291 } +SSM seeded=false heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.000000008099387559939997, 0.000000170725301229524 } +SSM seeded=false heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.000000014270247941650105, 0.00000013346487892640368 } +SSM seeded=false heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.000000003777481768357793, 0.00000016323549628660672 } +SSM seeded=false heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000005027532001822266, 0.00000017245745953520242 } +SSM seeded=false heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000005457366610700509, 0.00000029433054438143813 } +SSM seeded=false heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000008944659693477242, 0.0000003865198301422623 } +SSM seeded=false heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000008944659693477242, 0.0000003074925702950537 } +SSM seeded=false heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.00000001164840428280467, 0.00000030358411029496324 } +SSM seeded=false heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.000000022576584234046404, 0.0000004419375038144935 } +SSM seeded=false heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000010395453171885283, 0.0000002193045710030539 } +SSM seeded=false heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000001602812140649723, 0.0000007270377315737875 } +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=true heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000015089179933314156, 0.00000003066629741810267 } +SSM seeded=true heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000015656003028718946, 0.00000011578910319443025 } +SSM seeded=true heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000015656003028718946, 0.000000137397136557027 } +SSM seeded=true heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.000000015656003028718946, 0.000000058752570086806344 } +SSM seeded=true heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.00000003896021144589312, 0.00000011965777954592305 } +SSM seeded=true heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.00000004227448897886621, 0.0000001134548608394681 } +SSM seeded=true heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.00000004227448897886621, 0.00000018029616666304094 } +SSM seeded=true heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.0000000506942888084394, 0.0000001136397048684401 } +SSM seeded=true heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.0000000073519402093991104, 0.000000027996068407976082 } +SSM seeded=true heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.00000001807481535276434, 0.00000021659550952790596 } +SSM seeded=true heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000018969054375872485, 0.00000017414204611654327 } +SSM seeded=true heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000018969054375872485, 0.00000017655989847931153 } +SSM seeded=true heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000023961633413560612, 0.0000003531072296780735 } +SSM seeded=true heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000023961633413560612, 0.00000024600469961111315 } +SSM seeded=true heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.000000023961633413560612, 0.0000003112589632703333 } +SSM seeded=true heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.00000002654540304236619, 0.0000004793090875310213 } +SSM seeded=true heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000013799211689979884, 0.00000008033819287600608 } +SSM seeded=true heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000027007305641868484, 0.0000002670173361307704 } +SSM seeded=true heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000003555922037379311, 0.0000006937729284395289 } +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): qwen image: num_layers is not 32; this family runs the released geometry only +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 32; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 128; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): model.layers.3.mlp.up_proj: another spelling of module layers.3.mlp.up_proj declares a different width +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +sparse attention (64 heads): 99.97% of outputs equal the row kernel's, at most 0.96 bf16 steps apart +sparse attention (32 heads): 99.96% of outputs equal the row kernel's, at most 0.94 bf16 steps apart +absorb on the tensor units: 99.99% of outputs equal the row kernel's, at most 0.98 bf16 steps apart +KDA heads=64 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=64 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +KDA heads=32 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=32 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +[default] (warn): language_model.model.layers.1.mlp.down_proj: another spelling of module layers.1.mlp.down_proj declares a different width +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x2, 4-bit/64 x35, 5-bit/32 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.57 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x37, 5-bit/64 x4, 6-bit/64 x2, 8-bit/64 x4 +glm plan: 4 of 45 layers and the MTP layer, 1.86 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x3, 4-bit/64 x34, 5-bit/64 x1, 6-bit/64 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +failed command: ./.zc/o/33edb3e3eeb08d3b86f3f5ce438435e3/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): pack ~/tmp/rv1001-1/.zig-cache/tmp/Zwp8IOJnKUIjeImE/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc/o/185ffca428b32f37531ba4d42bf4d5f5/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +failed command: ./.zc/o/f7e8336d6165d89f7dd3c52315a6f5ad/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +test ++- run test w +[default] (warn): lane host: stopping at a round boundary: 2 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc/o/a8b7681f29e9e489e82bddc04264a5b8/test --cache-dir=./.zc --seed=0x2916d164 --listen=- + +Build Summary: 292/292 steps succeeded; 4321/4337 tests passed (16 skipped) +test success ++- run test 37 pass (37 total) 36ms MaxRSS:8M +| +- compile test debug native-native success 2s MaxRSS:312M ++- run test 7 pass (7 total) 232ms MaxRSS:72M +| +- compile test debug native success 2s MaxRSS:311M ++- run test 9 pass (9 total) 59ms MaxRSS:4M +| +- compile test debug native success 1s MaxRSS:283M +| +- options success +| +- compile exe digest-writer debug native success 1s MaxRSS:263M ++- run test 6 pass (6 total) 4ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:276M ++- test-flashnext-batch-host success +| +- run test 6 pass (6 total) 1s MaxRSS:30M +| +- compile test debug native success 2s MaxRSS:384M +| +- WriteFile flashnext-test-empty.zig success ++- run test 2 pass (2 total) 31ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:264M ++- run test 4 pass (4 total) 4ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:268M ++- run test 4 pass (4 total) 7ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:283M ++- run test 3 pass (3 total) 3ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:266M ++- run test 23 pass (23 total) 5ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:287M ++- run test 5 pass (5 total) 6ms MaxRSS:7M +| +- compile test debug native-native success 1s MaxRSS:340M ++- run test 19 pass (19 total) 9ms MaxRSS:6M +| +- compile test fast native-native success 9s MaxRSS:492M +| +- WriteFile drama3.metallib success +| +- run xcrun (drama3.metallib) success 310ms +| +- run xcrun (d3_dense.air) success 1s +| +- run xcrun (d3_gdn.air) success 1s +| +- run xcrun (d3_attn.air) success 1s +| +- run xcrun (d3_moe.air) success 1s +| +- run xcrun (d3_sample.air) success 1s +| +- run xcrun (d3_codec.air) success 1s ++- run test 131 pass (131 total) 839ms MaxRSS:21M +| +- compile test fast native-native success 22s MaxRSS:872M +| +- WriteFile draft_ids.txt success +| +- WriteFile nemotron_packed.metallib success +| +- run xcrun (nemotron_packed.metallib) success 53ms +| +- run xcrun (coop_down_1_0.air) success 170ms +| +- run xcrun (coop_down_2_0.air) success 174ms +| +- run xcrun (coop_down_2_1.air) success 173ms +| +- run xcrun (coop_draft_1_0.air) success 170ms +| +- run xcrun (coop_draft_2_0.air) success 170ms +| +- run xcrun (coop_draft_2_1.air) success 168ms +| +- run xcrun (coop_eh_1_0.air) success 169ms +| +- run xcrun (coop_eh_1_0_sk.air) success 187ms +| +- run xcrun (coop_eh_2_0.air) success 170ms +| +- run xcrun (coop_eh_2_0_sk.air) success 168ms +| +- run xcrun (coop_eh_2_1.air) success 177ms +| +- run xcrun (coop_eh_2_1_sk.air) success 168ms +| +- run xcrun (coop_head_1_0.air) success 167ms +| +- run xcrun (coop_head_2_0.air) success 298ms +| +- run xcrun (coop_head_2_1.air) success 172ms +| +- run xcrun (coop_in_1_0.air) success 170ms +| +- run xcrun (coop_in_1_0_sk.air) success 198ms +| +- run xcrun (coop_in_2_0.air) success 170ms +| +- run xcrun (coop_in_2_0_sk.air) success 170ms +| +- run xcrun (coop_in_2_1.air) success 172ms +| +- run xcrun (coop_in_2_1_sk.air) success 173ms +| +- run xcrun (coop_out_1_0.air) success 174ms +| +- run xcrun (coop_out_1_0_sk.air) success 168ms +| +- run xcrun (coop_out_2_0.air) success 165ms +| +- run xcrun (coop_out_2_0_sk.air) success 171ms +| +- run xcrun (coop_out_2_1.air) success 165ms +| +- run xcrun (coop_out_2_1_sk.air) success 169ms +| +- run xcrun (coop_qkv_1_0.air) success 182ms +| +- run xcrun (coop_qkv_2_0.air) success 170ms +| +- run xcrun (coop_qkv_2_1.air) success 175ms +| +- run xcrun (coop_up_1_0.air) success 170ms +| +- run xcrun (coop_up_2_0.air) success 174ms +| +- run xcrun (coop_up_2_1.air) success 166ms +| +- run xcrun (up_relu2_1_0.air) success 174ms +| +- run xcrun (up_relu2_2_0.air) success 175ms +| +- run xcrun (up_relu2_2_1.air) success 174ms ++- run test 77 pass (77 total) 52ms MaxRSS:9M +| +- compile test fast native-native success 20s MaxRSS:752M ++- run test 38 pass (38 total) 39ms MaxRSS:5M +| +- compile test fast native-native success 9s MaxRSS:472M ++- run test 86 pass (86 total) 90ms MaxRSS:10M +| +- compile test fast native-native success 19s MaxRSS:765M ++- run test 9 pass (9 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 1m MaxRSS:411M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:400M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- test-exl3 success +| +- compile obj exl3_metal_loader debug native-native success 192ms MaxRSS:102M +| | +- WriteFile exl3_projection.metal success +| +- run test 11 pass (11 total) 12ms MaxRSS:7M +| | +- compile test fast native-native success 7s MaxRSS:422M +| | +- WriteFile exl3_projection.metal (reused) +| +- run test 2 pass (2 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:404M +| +- WriteFile exl3_projection.metal (reused) ++- run test 4 pass (4 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:403M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 19 pass (19 total) 1s MaxRSS:24M +| +- compile test safe native-native success 23s MaxRSS:1G +| +- WriteFile exlr8_alias.zig success +| +- WriteFile drama3_core.zig success +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal success +| +- WriteFile drama3.metallib cached +| | +- run xcrun (drama3.metallib) cached +| | +- run xcrun (d3_dense.air) cached +| | +- run xcrun (d3_gdn.air) cached +| | +- run xcrun (d3_attn.air) cached +| | +- run xcrun (d3_moe.air) cached +| | +- run xcrun (d3_sample.air) cached +| | +- run xcrun (d3_codec.air) cached +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 9 pass (9 total) 4ms MaxRSS:6M +| +- compile test safe native-native success 1m MaxRSS:413M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 7ms MaxRSS:7M +| +- compile test fast native-native success 7s MaxRSS:421M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:403M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 1 pass (1 total) 4ms MaxRSS:6M +| +- compile test safe native-native success 7s MaxRSS:415M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 5 pass (5 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 7s MaxRSS:373M ++- run test 10 pass (10 total) 5ms MaxRSS:6M +| +- compile test safe native-native success 8s MaxRSS:440M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 7s MaxRSS:368M ++- run test 4 pass (4 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 7s MaxRSS:368M ++- run test 1 pass (1 total) 4ms MaxRSS:6M +| +- compile test safe native-native success 6s MaxRSS:408M ++- run test 112 pass (112 total) 73ms MaxRSS:17M +| +- compile test safe native-native success 14s MaxRSS:662M +| +- WriteFile xsum.metal (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 7 pass (7 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 8s MaxRSS:399M ++- run test 1 pass (1 total) 4ms MaxRSS:6M +| +- compile test safe native-native success 7s MaxRSS:412M ++- run sh success 46ms +| +- compile exe tensorfold-native safe native-native success 1m MaxRSS:4G +| +- options success +| +- WriteFile drama3_core.zig (reused) +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile drama3.metallib (+1 more reused dependencies) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 436 pass (436 total) 48s MaxRSS:96M +| +- compile test debug native-native success 13s MaxRSS:1G ++- run test 5 pass (5 total) 4ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:286M ++- run test 25 pass (25 total) 24s MaxRSS:236M +| +- compile test debug native-native success 1s MaxRSS:335M ++- run test 15 pass (15 total) 172ms MaxRSS:33M +| +- compile test debug native-native success 2s MaxRSS:393M ++- run test 204 pass (204 total) 1s MaxRSS:12M +| +- compile test fast native-native success 24s MaxRSS:989M ++- run test 8 pass (8 total) 79ms MaxRSS:14M +| +- compile test fast native-native success 7s MaxRSS:432M ++- run test 1 pass (1 total) 4ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:400M +| +- WriteFile tensorfold.metallib success +| +- run xcrun (tensorfold.metallib) success 27ms +| +- run xcrun (runtime_tests.air) success 41ms +| +- run xcrun (nemotron_expert_up.air) success 46ms +| +- run xcrun (nemotron_expert_down.air) success 46ms ++- run test 648 pass, 3 skip (651 total) 6s MaxRSS:157M +| +- compile test fast native-native success 1m MaxRSS:2G +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 163 pass (163 total) 201ms MaxRSS:29M +| +- compile test fast native-native success 20s MaxRSS:796M ++- run test 30 pass, 1 skip (31 total) 40ms MaxRSS:2M +| +- compile test fast native-native success 11s MaxRSS:494M ++- run test 1 pass (1 total) 2ms MaxRSS:2M +| +- compile test fast native-native success 5s MaxRSS:351M ++- run test 6 pass (6 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 6s MaxRSS:406M ++- run test 4 pass (4 total) 2ms MaxRSS:2M +| +- compile test fast native-native success 6s MaxRSS:363M ++- test-cuda-host success +| +- run test 1 pass (1 total) 3ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:268M +| +- run test 30 pass (30 total) 45ms MaxRSS:4M +| | +- compile test debug native success 2s MaxRSS:374M +| | +- WriteFile h3-compile-only.fatbin success +| | +- options success +| +- run test 3 pass (3 total) 360ms MaxRSS:70M +| | +- compile test debug native success 1s MaxRSS:285M +| +- run test 10 pass (10 total) 1s MaxRSS:15M +| | +- compile test debug native success 2s MaxRSS:415M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- compile obj exl3_host_compile debug native success 228ms MaxRSS:113M +| | +- options (reused) +| +- run test 6 pass (6 total) 8ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:307M +| | +- options (reused) +| +- run test 6 pass (6 total) 51ms MaxRSS:5M +| | +- compile test debug native success 1s MaxRSS:290M +| | +- options (reused) +| +- run test 8 pass (8 total) 11ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:281M +| +- run test 3 pass (3 total) 51ms MaxRSS:5M +| | +- compile test debug native success 1s MaxRSS:277M +| +- run test 8 pass (8 total) 12ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:285M +| +- run test 4 pass (4 total) 5ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:287M +| | +- options (reused) +| +- run test 1 pass (1 total) 33ms MaxRSS:4M +| | +- compile test debug native success 1s MaxRSS:285M +| | +- options (reused) +| +- run test 4 pass (4 total) 3ms MaxRSS:2M +| | +- compile test fast native success 7s MaxRSS:379M +| | +- options (reused) +| +- run test 1 pass (1 total) 8ms MaxRSS:3M +| | +- compile test fast native success 7s MaxRSS:390M +| | +- options (reused) +| +- compile obj bolt_host_compile debug aarch64-linux-gnu success 407ms MaxRSS:141M +| | +- options cached +| | +- WriteFile draft_ids.txt (reused) +| +- run test 26 pass (26 total) 399ms MaxRSS:6M +| | +- compile test debug native success 2s MaxRSS:376M +| | +- options (reused) +| +- compile obj drama_cuda_native_host debug native success 326ms MaxRSS:124M +| | +- options (reused) +| +- run test 5 pass (5 total) 4ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:276M +| +- run test 436 pass (436 total) 49s MaxRSS:96M +| | +- compile test debug native success 12s MaxRSS:1G +| +- run test 5 pass (5 total) 4ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:282M +| +- run test 130 pass (130 total) 48ms MaxRSS:4M +| | +- compile test debug native success 2s MaxRSS:375M +| | +- options (reused) +| +- run test 280 pass, 2 skip (282 total) 4s MaxRSS:132M +| | +- compile test debug native success 4s MaxRSS:591M +| +- run test 163 pass (163 total) 4s MaxRSS:65M +| | +- compile test debug native success 3s MaxRSS:477M +| +- run test 74 pass, 1 skip (75 total) 153ms MaxRSS:10M +| | +- compile test debug native success 3s MaxRSS:497M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- run test 33 pass (33 total) 5s MaxRSS:24M +| | +- compile test debug native success 2s MaxRSS:345M +| | +- options (reused) +| +- run test 12 pass (12 total) 16ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:329M +| | +- options (reused) +| +- run test 83 pass (83 total) 171ms MaxRSS:74M +| | +- compile test debug native success 2s MaxRSS:440M +| | +- options (reused) +| +- run test 13 pass (13 total) 3s MaxRSS:15M +| | +- compile test debug native success 2s MaxRSS:390M +| | +- options (reused) +| +- run test 29 pass (29 total) 3s MaxRSS:1G +| | +- compile test debug native success 2s MaxRSS:344M +| | +- options (reused) +| +- run test 5 pass (5 total) 6ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:281M +| +- run test 20 pass (20 total) 115ms MaxRSS:5M +| | +- compile test debug native success 2s MaxRSS:344M +| +- run test 204 pass (204 total) 3s MaxRSS:42M +| | +- compile test debug native success 3s MaxRSS:532M +| +- run test 52 pass, 1 skip (53 total) 99ms MaxRSS:24M +| | +- compile test debug native success 4s MaxRSS:660M +| | +- options (reused) +| | +- WriteFile draft_ids.txt (reused) +| +- run test 3 pass (3 total) 4ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:265M +| +- run test 52 pass (52 total) 29ms MaxRSS:70M +| | +- compile test debug native success 3s MaxRSS:392M +| | +- options (reused) +| +- run test 5 pass (5 total) 4ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:272M +| +- run test 25 pass (25 total) 23s MaxRSS:236M +| | +- compile test debug native success 1s MaxRSS:334M +| +- run test 1 pass (1 total) 3ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:300M +| +- options (reused) +| +- options (reused) +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile exlr8_alias.zig cached ++- run test 4 pass (4 total) 3ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:274M +| +- options cached ++- run test 13 pass, 1 skip (14 total) 41ms MaxRSS:4M +| +- compile test debug native success 2s MaxRSS:319M +| +- options (reused) ++- run test 3 pass (3 total) 5ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:286M +| +- options (reused) ++- run test 1 pass (1 total) 3ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:276M ++- test-kv-pool success +| +- run test 9 pass (9 total) 10ms MaxRSS:3M +| | +- compile test debug native success 1s MaxRSS:270M +| +- run test 6 pass (6 total) 2s MaxRSS:34M +| +- compile test debug native success 2s MaxRSS:391M ++- run test 51 pass (51 total) 32ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:324M +| +- options success +| | +- compile lib hip-mock-success debug native success 107ms MaxRSS:55M +| | +- compile lib hip-mock-failed debug native success 107ms MaxRSS:56M +| | +- compile lib hip-mock-missing debug native success 107ms MaxRSS:55M +| +- WriteFile kernels.zig cached ++- run test 23 pass, 1 skip (24 total) 86ms MaxRSS:5M +| +- compile test debug native success 2s MaxRSS:352M ++- run test 49 pass (49 total) 29ms MaxRSS:9M +| +- compile test debug native success 2s MaxRSS:352M +| +- WriteFile kernels.zig success ++- run test 30 pass, 1 skip (31 total) 75ms MaxRSS:5M +| +- compile test debug native success 3s MaxRSS:485M +| +- WriteFile kernels.zig (reused) ++- run test 18 pass (18 total) 167ms MaxRSS:7M +| +- compile test debug native success 3s MaxRSS:493M +| +- WriteFile kernels.zig (reused) ++- run test 4 pass (4 total) 14ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:286M +| +- WriteFile kernels.zig (reused) ++- run test 3 pass (3 total) 3ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:313M +| +- WriteFile kernels.zig (reused) ++- compile exe tf-qwen35-test debug native success 1s MaxRSS:171M +| +- WriteFile kernels.zig (reused) ++- compile exe tensorfold-native safe native success 2s MaxRSS:259M +| +- options (reused) +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile kernels.zig (reused) ++- run test 26 pass, 5 skip (31 total) 13ms MaxRSS:6M +| +- compile test safe native-native success 12s MaxRSS:546M ++- run test 112 pass (112 total) 2s MaxRSS:41M +| +- compile test debug native success 2s MaxRSS:397M ++- run test 20 pass (20 total) 107ms MaxRSS:11M +| +- compile test debug native success 1s MaxRSS:321M ++- run exe learned-warning-test success 6ms + +- compile exe learned-warning-test debug native success 1s MaxRSS:291M + + +# exit=0 after 187s; free disk after: 52 GB diff --git a/docs/receipts/rocm-host-identity/host-controls/test-main-76aafcf2.log b/docs/receipts/rocm-host-identity/host-controls/test-main-76aafcf2.log new file mode 100644 index 0000000000..899a1f3fcd --- /dev/null +++ b/docs/receipts/rocm-host-identity/host-controls/test-main-76aafcf2.log @@ -0,0 +1,894 @@ +# tree: main 76aafcf274e60a5e31a59ce145b2f71f7b366e9d +# cmd: zig build test -Dcpu=apple_m1 --summary all -j8 (zig 0.17.0, ZIG_GLOBAL_CACHE_DIR and ZIG_LOCAL_CACHE_DIR private) +# started 2026-10-18T23:12:40Z +test ++- run test w +[default] (warn): a round could not be recorded: OutOfDeviceMemory +failed command: ./.zc/o/c006ea4bcf6c8cffd67e5b0b6d533499/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +failed command: ./.zc/o/61f4f42007c910ae0dba3d28a8174637/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): language_model.lm_head: stored affine but declared unquantized +[default] (warn): language_model.model.embed_tokens: stored affine but declared unquantized +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc/o/8871d809081e7b7104c8626a39069d70/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): XPU: a command list did not drain at close; its memory and context stay until the process ends +failed command: ./.zc/o/d81124c80476e199df0db72a5c7ab081/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): glm: restoring 1500 tokens failed (PoolMoveNotDrained); pool relocation aborted +failed command: ./.zc/o/9c900e13443c4ea3333643efc5df6815/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): lane host: stopping at a round boundary: 1 running and 2 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc/o/a8b7681f29e9e489e82bddc04264a5b8/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc/o/79a801776f3037613fa73ba17b60f7ac/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): slide pocket: a delete failed (UploadFailed), and its put-back did not reach the device (PutBackNotOnDevice) +[default] (warn): slide pocket: a deleted fact's 2 layers are put back in memory, but not all on the device (UploadFailed): replies are refused until they are +[default] (warn): checkpoint tensor text.weight stays in its file: read it through a Stage +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): pack ~/tmp/rv1001-1-main/.zig-cache/tmp/y644BJjaM5ocvL-U/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=false heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000004647133651849988, 0.00000008929183081818337 } +SSM seeded=false heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000004647133651849988, 0.00000010044378595530645 } +SSM seeded=false heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.0000000051661870337227356, 0.00000006044416009376441 } +SSM seeded=false heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.000000008099387559939997, 0.00000010302938735495815 } +SSM seeded=false heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.000000008099387559939997, 0.00000010076810685877291 } +SSM seeded=false heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.000000008099387559939997, 0.000000170725301229524 } +SSM seeded=false heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.000000014270247941650105, 0.00000013346487892640368 } +SSM seeded=false heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.000000003777481768357793, 0.00000016323549628660672 } +SSM seeded=false heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000005027532001822266, 0.00000017245745953520242 } +SSM seeded=false heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000005457366610700509, 0.00000029433054438143813 } +SSM seeded=false heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000008944659693477242, 0.0000003865198301422623 } +SSM seeded=false heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000008944659693477242, 0.0000003074925702950537 } +SSM seeded=false heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.00000001164840428280467, 0.00000030358411029496324 } +SSM seeded=false heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.000000022576584234046404, 0.0000004419375038144935 } +SSM seeded=false heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0, 0 } +SSM seeded=false heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000010395453171885283, 0.0000002193045710030539 } +SSM seeded=false heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000001602812140649723, 0.0000007270377315737875 } +SSM pipeline capacity: forward=1024, backward=1024 +SSM seeded=true heads=2 groups=1 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000015089179933314156, 0.00000003066629741810267 } +SSM seeded=true heads=2 groups=1 rows=7: max abs dx/dB/dC/ddt { 0.000000011360835217022469, 0.000000002917455535836333, 0.000000015656003028718946, 0.00000011578910319443025 } +SSM seeded=true heads=2 groups=1 rows=8: max abs dx/dB/dC/ddt { 0.00000000863441644745333, 0.0000000030472456129596037, 0.000000015656003028718946, 0.000000137397136557027 } +SSM seeded=true heads=2 groups=1 rows=9: max abs dx/dB/dC/ddt { 0.000000006604320090808358, 0.0000000027633473599630864, 0.000000015656003028718946, 0.000000058752570086806344 } +SSM seeded=true heads=2 groups=1 rows=15: max abs dx/dB/dC/ddt { 0.000000010416435020088599, 0.0000000050579541700790376, 0.00000003896021144589312, 0.00000011965777954592305 } +SSM seeded=true heads=2 groups=1 rows=16: max abs dx/dB/dC/ddt { 0.000000010399334079633227, 0.000000004927133326337874, 0.00000004227448897886621, 0.0000001134548608394681 } +SSM seeded=true heads=2 groups=1 rows=17: max abs dx/dB/dC/ddt { 0.00000001543697608283323, 0.000000004476264019170895, 0.00000004227448897886621, 0.00000018029616666304094 } +SSM seeded=true heads=2 groups=1 rows=25: max abs dx/dB/dC/ddt { 0.00000001223729547902841, 0.000000005462381803822414, 0.0000000506942888084394, 0.0000001136397048684401 } +SSM seeded=true heads=4 groups=2 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.0000000073519402093991104, 0.000000027996068407976082 } +SSM seeded=true heads=4 groups=2 rows=7: max abs dx/dB/dC/ddt { 0.000000015510263740026176, 0.000000006877330351040545, 0.00000001807481535276434, 0.00000021659550952790596 } +SSM seeded=true heads=4 groups=2 rows=8: max abs dx/dB/dC/ddt { 0.0000000129148233773968, 0.000000005653858301674086, 0.000000018969054375872485, 0.00000017414204611654327 } +SSM seeded=true heads=4 groups=2 rows=9: max abs dx/dB/dC/ddt { 0.00000002633087964332148, 0.000000006475598009203498, 0.000000018969054375872485, 0.00000017655989847931153 } +SSM seeded=true heads=4 groups=2 rows=15: max abs dx/dB/dC/ddt { 0.00000002973828677754753, 0.000000009558005810428316, 0.000000023961633413560612, 0.0000003531072296780735 } +SSM seeded=true heads=4 groups=2 rows=16: max abs dx/dB/dC/ddt { 0.000000025754882060358142, 0.000000008921690271973537, 0.000000023961633413560612, 0.00000024600469961111315 } +SSM seeded=true heads=4 groups=2 rows=17: max abs dx/dB/dC/ddt { 0.00000002553487910179897, 0.000000008921690271973537, 0.000000023961633413560612, 0.0000003112589632703333 } +SSM seeded=true heads=4 groups=2 rows=25: max abs dx/dB/dC/ddt { 0.00000003875727619395697, 0.000000014591112693085329, 0.00000002654540304236619, 0.0000004793090875310213 } +SSM seeded=true heads=64 groups=8 rows=1: max abs dx/dB/dC/ddt { 0, 0, 0.000000013799211689979884, 0.00000008033819287600608 } +SSM seeded=true heads=64 groups=8 rows=9: max abs dx/dB/dC/ddt { 0.00000002087263310801557, 0.00000001779203438051491, 0.000000027007305641868484, 0.0000002670173361307704 } +SSM seeded=true heads=64 groups=8 rows=17: max abs dx/dB/dC/ddt { 0.00000009930843825056712, 0.00000002114473045267129, 0.00000003555922037379311, 0.0000006937729284395289 } +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): qwen image: num_layers is not 32; this family runs the released geometry only +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 32; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.0.self_attn.q_proj is 4-bit in groups of 128; the kernels read bits 2-6 and 8 in groups of 64 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj's width is not MLX affine bits 2-6 or 8 +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): gemma4: layers.3.mlp.up_proj is not a width table (false or {} marks a module unquantized); every linear is affine +[default] (warn): model.layers.3.mlp.up_proj: another spelling of module layers.3.mlp.up_proj declares a different width +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +[default] (warn): expert parallel: this Mac runs .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 8, .mtp = 1, .experts = 288, .own_lo = 144, .own_hi = 288, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }, the peer .{ .magic = 1196182853, .version = 2, .layers = 45, .run = 45, .mtp = 1, .experts = 288, .own_lo = 0, .own_hi = 144, .cap = 8256, .pad = 0, .model = 4660, .rest = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } +sparse attention (64 heads): 99.97% of outputs equal the row kernel's, at most 0.96 bf16 steps apart +sparse attention (32 heads): 99.96% of outputs equal the row kernel's, at most 0.94 bf16 steps apart +absorb on the tensor units: 99.99% of outputs equal the row kernel's, at most 0.98 bf16 steps apart +KDA heads=64 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=64 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=64 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +KDA heads=32 rows=1: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=2: worst 0.4999 bf16 steps, state 0.000000002 +KDA heads=32 rows=8: worst 0.4999 bf16 steps, state 0.000000003 +KDA heads=32 rows=16: worst 0.4999 bf16 steps, state 0.000000005 +[default] (warn): language_model.model.layers.1.mlp.down_proj: another spelling of module layers.1.mlp.down_proj declares a different width +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x2, 4-bit/64 x35, 5-bit/32 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.57 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x37, 5-bit/64 x4, 6-bit/64 x2, 8-bit/64 x4 +glm plan: 4 of 45 layers and the MTP layer, 1.86 GB in buffers, every tensor's name, dtype and shape checked; 2-bit/64 x3, 4-bit/64 x34, 5-bit/64 x1, 6-bit/64 x1, 8-bit/64 x1 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.51 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x39 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +glm plan: 4 of 45 layers and the MTP layer, 1.56 GB in buffers, every tensor's name, dtype and shape checked; 4-bit/64 x40 +failed command: ./.zc/o/38a8f1065ad9ff655f4b42cf2113d36d/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): pack ~/tmp/rv1001-1-main/.zig-cache/tmp/WGohgoDAEaZ54stn/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc/o/f8d3bf693f0e05179118ea8c6eb40f5c/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): language_model.model.layers.0: gate and up of one expert differ in width; one kernel reads both +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): pack ~/tmp/rv1001-1-main/.zig-cache/tmp/Lte7I8mqNdGe7FKe/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +failed command: ./.zc/o/9586ede43cd16349d8dbadf219aa4ae4/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- run test w +[default] (warn): language_model.mtp.fc_embedding: not read at its declared width: UnsupportedQuantization +[default] (warn): language_model.model.embed_tokens: not stored at its declared width: InvalidAffineShape +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): lane L0.att.o: LaneRowsNotFours +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): lane L21.gdn.in: OutOfMemory +[default] (warn): pack ~/tmp/rv1001-1-main/.zig-cache/tmp/lyhvxnFj9cqHZy1y/pack.safetensors has no recorded source; skipping the identity check +[default] (warn): pack pack.safetensors has no recorded source; skipping the identity check +[default] (warn): model.layers.0.self_attn.v_proj.weight: another spelling of module layers.0.self_attn.v_proj declares a different width +failed command: ./.zc/o/1a82ff962611db8c42af6656d23d40d4/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/3a29e04aef202e78700d1846260ac59b/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/30d710ca425c97de2cded66302b3cc2f/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Dimensions +tf_h3_forward_host: BadH3Dimensions +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Rotary +tf_h3_forward_host: BadH3Rotary +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Keep +tf_h3_forward_host: BadH3Keep +tf_h3_forward: BadH3Blocks +tf_h3_forward_host: BadH3Blocks +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Scalar +tf_h3_forward_host: BadH3Scalar +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Address +tf_h3_forward_host: BadH3Scalar +H3 model retained at close: NotReady +H3 geometry retained: NotReady +tensor blocks.0.norm1.weight: MissingTensor +tensor blocks.1.attn.qkv_proj.weight: MissingTensor +tensor blocks.1.norm1.weight: MissingTensor +failed command: ./.zc/o/9b8a4b56c3a74aabdb18c5ebefc89338/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 0 cannot exchange with rank 2; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 1 cannot exchange with rank 3; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 2 cannot exchange with rank 0; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +REFUSE TF_GLM_PREFILL_REDUCE=split: rank 3 cannot exchange with rank 1; split sends every rank's rows to every other, and a fabric where some ranks reach only their neighbours (a ring of four without a switch) needs TF_GLM_PREFILL_REDUCE=ring or gather +failed command: ./.zc/o/fe1691631351b887dfcf0a23f0ecffec/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lm_head is 8-bit in groups of 64: CUDA reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +[default] (warn): backbone.embeddings is 8-bit in groups of 64: the Intel GPU engine reads Nemotron MLX checkpoints at 4-bit in groups of 64 only; mixed and 6- or 8-bit (oQ) checkpoints serve on Macs +fake target: proposed=68 verified=68 accepted=68 +failed command: ./.zc/o/b14d9e46b0d7ee10a3cb8cf281256070/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +[default] (warn): Drama readback drain failed: retaining its buffers and engine owner +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +codec requested 4, actual sizes { 4, 4, 4, 4, 1 } +codec requested 16, actual sizes { 16, 1 } +codec requested 1, actual sizes { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } +Drama replay passed: eager/graphs, poison/reset, codec chunks and interleaved state +[default] (warn): drama3: config hidden_size = 4096, the kernels need 2048 +[default] (warn): codec.pth pickle names os.system, outside the tensor state-dict allow-list +failed command: ./.zc/o/a529abb901551c852fa65e556ac33565/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +RESULT valid stale prefix: largest difference 0.000 of its tolerance (element 0) +FAIL no-op projection: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/64fbdce0f7ef835b083e716eeb96c64e/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +libjpeg-turbo 3.1.4.1 not found: LFM2's JPEG cases skipped +failed command: ./.zc/o/65e976b591c0054618177133701a6e9c/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 1.000 of its tolerance (element 1) +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +RESULT CPU reference comparison: largest difference 2.000 of its tolerance (element 0) +FAIL CPU reference comparison within its float64 tolerance +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 2 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 1 (got -nan, reference 0e0, tolerance 1e0) +FAIL CPU reference comparison: non-finite error at element 0 (got inf, reference 0e0, tolerance 1e0) +failed command: ./.zc/o/bad8028c5ed1fa3d2a707d317182e06f/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (err): rank 0: another rank's process ended (its line closed); this rank stops instead of waiting on it +[default] (err): rank 0: rank 1 stopped (peer OOM); this rank stops too +failed command: ./.zc/o/ee93d32aa7709c998e223d0eefc647cb/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): lane host: stopping at a round boundary: 2 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (server restarting: retry shortly) +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (GpuFailed): every admitted stream fails +[default] (warn): lane host: a round failed (the planner refused): every admitted stream fails +[default] (warn): lane host: a round failed (PlannerFull): every admitted stream fails +[default] (warn): lane host: stopping at a round boundary: 1 running and 1 queued request(s) end (drain deadline) +failed command: ./.zc/o/3715e5673fd04f7ffc906ecb69c0e4d0/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +[default] (warn): qwen3_5: a kept state's host copy goes to the copy-source quarantine: a restore's copies did not drain +[default] (warn): qwen3_5: a failed restore's stream did not drain: its kept state's host copy stays +failed command: ./.zc/o/f3b10f670ad7268b96c2e9a7e7df0b46/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +test ++- test-cuda-host + +- run test w +[default] (warn): /opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib is not libjpeg-turbo 3.1.4.1; JPEG images are refused until it is +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +libjpeg-turbo 3.1.4.1 not found: JPEG pixel cases skipped +[default] (warn): model.layers.0.mlp.up_proj.weight: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +[default] (warn): model.layers.0.mlp.up_proj: another spelling of module layers.0.mlp.up_proj declares a different width +failed command: ./.zc/o/8b9c81f8568eba22650d9870cadcbbdc/test --cache-dir=./.zc --seed=0xee6486a4 --listen=- + +Build Summary: 292/292 steps succeeded; 4305/4321 tests passed (16 skipped) +test success ++- run test 37 pass (37 total) 48ms MaxRSS:8M +| +- compile test debug native-native success 5s MaxRSS:315M ++- run test 7 pass (7 total) 320ms MaxRSS:64M +| +- compile test debug native success 2s MaxRSS:310M ++- run test 9 pass (9 total) 96ms MaxRSS:4M +| +- compile test debug native success 4s MaxRSS:285M +| +- options success +| +- compile exe digest-writer debug native success 4s MaxRSS:264M ++- run test 6 pass (6 total) 5ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:270M ++- test-flashnext-batch-host success +| +- run test 6 pass (6 total) 4s MaxRSS:30M +| +- compile test debug native success 7s MaxRSS:383M +| +- WriteFile flashnext-test-empty.zig success ++- run test 2 pass (2 total) 38ms MaxRSS:3M +| +- compile test debug native-native success 2s MaxRSS:267M ++- run test 4 pass (4 total) 5ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:266M ++- run test 4 pass (4 total) 9ms MaxRSS:3M +| +- compile test debug native-native success 3s MaxRSS:290M ++- run test 3 pass (3 total) 4ms MaxRSS:3M +| +- compile test debug native-native success 1s MaxRSS:267M ++- run test 23 pass (23 total) 7ms MaxRSS:3M +| +- compile test debug native-native success 2s MaxRSS:285M ++- run test 5 pass (5 total) 8ms MaxRSS:7M +| +- compile test debug native-native success 4s MaxRSS:332M ++- run test 19 pass (19 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 12s MaxRSS:493M +| +- WriteFile drama3.metallib cached +| +- run xcrun (drama3.metallib) cached +| +- run xcrun (d3_dense.air) cached +| +- run xcrun (d3_gdn.air) cached +| +- run xcrun (d3_attn.air) cached +| +- run xcrun (d3_moe.air) cached +| +- run xcrun (d3_sample.air) cached +| +- run xcrun (d3_codec.air) cached ++- run test 131 pass (131 total) 970ms MaxRSS:22M +| +- compile test fast native-native success 28s MaxRSS:877M +| +- WriteFile draft_ids.txt success +| +- WriteFile nemotron_packed.metallib success +| +- run xcrun (nemotron_packed.metallib) success 82ms +| +- run xcrun (coop_down_1_0.air) success 241ms +| +- run xcrun (coop_down_2_0.air) success 239ms +| +- run xcrun (coop_down_2_1.air) success 246ms +| +- run xcrun (coop_draft_1_0.air) success 252ms +| +- run xcrun (coop_draft_2_0.air) success 239ms +| +- run xcrun (coop_draft_2_1.air) success 258ms +| +- run xcrun (coop_eh_1_0.air) success 248ms +| +- run xcrun (coop_eh_1_0_sk.air) success 237ms +| +- run xcrun (coop_eh_2_0.air) success 238ms +| +- run xcrun (coop_eh_2_0_sk.air) success 243ms +| +- run xcrun (coop_eh_2_1.air) success 238ms +| +- run xcrun (coop_eh_2_1_sk.air) success 1s +| +- run xcrun (coop_head_1_0.air) success 234ms +| +- run xcrun (coop_head_2_0.air) success 244ms +| +- run xcrun (coop_head_2_1.air) success 240ms +| +- run xcrun (coop_in_1_0.air) success 1s +| +- run xcrun (coop_in_1_0_sk.air) success 243ms +| +- run xcrun (coop_in_2_0.air) success 238ms +| +- run xcrun (coop_in_2_0_sk.air) success 243ms +| +- run xcrun (coop_in_2_1.air) success 250ms +| +- run xcrun (coop_in_2_1_sk.air) success 258ms +| +- run xcrun (coop_out_1_0.air) success 250ms +| +- run xcrun (coop_out_1_0_sk.air) success 238ms +| +- run xcrun (coop_out_2_0.air) success 238ms +| +- run xcrun (coop_out_2_0_sk.air) success 240ms +| +- run xcrun (coop_out_2_1.air) success 251ms +| +- run xcrun (coop_out_2_1_sk.air) success 260ms +| +- run xcrun (coop_qkv_1_0.air) success 248ms +| +- run xcrun (coop_qkv_2_0.air) success 246ms +| +- run xcrun (coop_qkv_2_1.air) success 262ms +| +- run xcrun (coop_up_1_0.air) success 238ms +| +- run xcrun (coop_up_2_0.air) success 1s +| +- run xcrun (coop_up_2_1.air) success 253ms +| +- run xcrun (up_relu2_1_0.air) success 236ms +| +- run xcrun (up_relu2_2_0.air) success 250ms +| +- run xcrun (up_relu2_2_1.air) success 262ms ++- run test 77 pass (77 total) 77ms MaxRSS:9M +| +- compile test fast native-native success 23s MaxRSS:740M ++- run test 38 pass (38 total) 64ms MaxRSS:5M +| +- compile test fast native-native success 13s MaxRSS:466M ++- run test 86 pass (86 total) 170ms MaxRSS:10M +| +- compile test fast native-native success 24s MaxRSS:754M ++- run test 9 pass (9 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 9s MaxRSS:409M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 5ms MaxRSS:6M +| +- compile test fast native-native success 8s MaxRSS:398M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- test-exl3 success +| +- compile obj exl3_metal_loader debug native-native success 454ms MaxRSS:101M +| | +- WriteFile exl3_projection.metal success +| +- run test 11 pass (11 total) 18ms MaxRSS:7M +| | +- compile test fast native-native success 20s MaxRSS:424M +| | +- WriteFile exl3_projection.metal (reused) +| +- run test 2 pass (2 total) 7ms MaxRSS:6M +| +- compile test fast native-native success 18s MaxRSS:402M +| +- WriteFile exl3_projection.metal (reused) ++- run test 4 pass (4 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 8s MaxRSS:402M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 19 pass (19 total) 1s MaxRSS:24M +| +- compile test safe native-native success 30s MaxRSS:1G +| +- WriteFile exlr8_alias.zig success +| +- WriteFile drama3_core.zig success +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal success +| +- WriteFile drama3.metallib success +| | +- run xcrun (drama3.metallib) success 413ms +| | +- run xcrun (d3_dense.air) success 84ms +| | +- run xcrun (d3_gdn.air) success 63ms +| | +- run xcrun (d3_attn.air) success 64ms +| | +- run xcrun (d3_moe.air) success 96ms +| | +- run xcrun (d3_sample.air) success 104ms +| | +- run xcrun (d3_codec.air) success 1s +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 9 pass (9 total) 5ms MaxRSS:6M +| +- compile test safe native-native success 9s MaxRSS:415M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 9ms MaxRSS:7M +| +- compile test fast native-native success 8s MaxRSS:412M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 2 pass (2 total) 5ms MaxRSS:6M +| +- compile test fast native-native success 8s MaxRSS:400M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 1 pass (1 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 9s MaxRSS:408M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 5 pass (5 total) 4ms MaxRSS:2M +| +- compile test safe native-native success 9s MaxRSS:373M ++- run test 10 pass (10 total) 6ms MaxRSS:6M +| +- compile test safe native-native success 10s MaxRSS:439M +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 3 pass (3 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 9s MaxRSS:367M ++- run test 4 pass (4 total) 3ms MaxRSS:2M +| +- compile test safe native-native success 8s MaxRSS:369M ++- run test 1 pass (1 total) 7ms MaxRSS:6M +| +- compile test safe native-native success 9s MaxRSS:410M ++- run test 112 pass (112 total) 206ms MaxRSS:17M +| +- compile test safe native-native success 18s MaxRSS:658M +| +- WriteFile xsum.metal (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 7 pass (7 total) 4ms MaxRSS:2M +| +- compile test safe native-native success 15s MaxRSS:401M ++- run test 1 pass (1 total) 5ms MaxRSS:6M +| +- compile test safe native-native success 9s MaxRSS:413M ++- run sh success 220ms +| +- compile exe tensorfold-native safe native-native success 2m MaxRSS:4G +| +- options success +| +- WriteFile drama3_core.zig (reused) +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile drama3.metallib (+1 more reused dependencies) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 428 pass (428 total) 51s MaxRSS:96M +| +- compile test debug native-native success 17s MaxRSS:1G ++- run test 5 pass (5 total) 7ms MaxRSS:3M +| +- compile test debug native-native success 4s MaxRSS:354M ++- run test 25 pass (25 total) 30s MaxRSS:236M +| +- compile test debug native-native success 2s MaxRSS:336M ++- run test 15 pass (15 total) 201ms MaxRSS:33M +| +- compile test debug native-native success 3s MaxRSS:389M ++- run test 204 pass (204 total) 1s MaxRSS:12M +| +- compile test fast native-native success 30s MaxRSS:988M ++- run test 8 pass (8 total) 76ms MaxRSS:14M +| +- compile test fast native-native success 9s MaxRSS:431M ++- run test 1 pass (1 total) 5ms MaxRSS:6M +| +- compile test fast native-native success 9s MaxRSS:396M +| +- WriteFile tensorfold.metallib success +| +- run xcrun (tensorfold.metallib) success 445ms +| +- run xcrun (runtime_tests.air) success 1s +| +- run xcrun (nemotron_expert_up.air) success 1s +| +- run xcrun (nemotron_expert_down.air) success 1s ++- run test 648 pass, 3 skip (651 total) 20s MaxRSS:161M +| +- compile test fast native-native success 1m MaxRSS:2G +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile xsum.metal (reused) +| +- WriteFile nemotron_packed.metallib (+1 more reused dependencies) ++- run test 163 pass (163 total) 420ms MaxRSS:29M +| +- compile test fast native-native success 56s MaxRSS:789M ++- run test 30 pass, 1 skip (31 total) 42ms MaxRSS:2M +| +- compile test fast native-native success 12s MaxRSS:488M ++- run test 1 pass (1 total) 3ms MaxRSS:2M +| +- compile test fast native-native success 7s MaxRSS:348M ++- run test 6 pass (6 total) 6ms MaxRSS:6M +| +- compile test fast native-native success 8s MaxRSS:400M ++- run test 4 pass (4 total) 3ms MaxRSS:2M +| +- compile test fast native-native success 8s MaxRSS:367M ++- test-cuda-host success +| +- run test 1 pass (1 total) 5ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:270M +| +- run test 30 pass (30 total) 69ms MaxRSS:4M +| | +- compile test debug native success 5s MaxRSS:369M +| | +- WriteFile h3-compile-only.fatbin success +| | +- options cached +| +- run test 3 pass (3 total) 1s MaxRSS:70M +| | +- compile test debug native success 4s MaxRSS:288M +| +- run test 10 pass (10 total) 2s MaxRSS:15M +| | +- compile test debug native success 6s MaxRSS:418M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- compile obj exl3_host_compile debug native success 384ms MaxRSS:113M +| | +- options (reused) +| +- run test 6 pass (6 total) 13ms MaxRSS:3M +| | +- compile test debug native success 4s MaxRSS:306M +| | +- options (reused) +| +- run test 6 pass (6 total) 75ms MaxRSS:5M +| | +- compile test debug native success 4s MaxRSS:290M +| | +- options (reused) +| +- run test 8 pass (8 total) 16ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:281M +| +- run test 3 pass (3 total) 71ms MaxRSS:5M +| | +- compile test debug native success 4s MaxRSS:280M +| +- run test 8 pass (8 total) 21ms MaxRSS:3M +| | +- compile test debug native success 2s MaxRSS:285M +| +- run test 4 pass (4 total) 7ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:286M +| | +- options (reused) +| +- run test 1 pass (1 total) 50ms MaxRSS:4M +| | +- compile test debug native success 4s MaxRSS:285M +| | +- options (reused) +| +- run test 4 pass (4 total) 5ms MaxRSS:2M +| | +- compile test fast native success 17s MaxRSS:373M +| | +- options (reused) +| +- run test 1 pass (1 total) 32ms MaxRSS:3M +| | +- compile test fast native success 18s MaxRSS:382M +| | +- options (reused) +| +- compile obj bolt_host_compile debug aarch64-linux-gnu success 879ms MaxRSS:140M +| | +- options cached +| | +- WriteFile draft_ids.txt (reused) +| +- run test 26 pass (26 total) 743ms MaxRSS:6M +| | +- compile test debug native success 5s MaxRSS:378M +| | +- options (reused) +| +- compile obj drama_cuda_native_host debug native success 639ms MaxRSS:124M +| | +- options (reused) +| +- run test 5 pass (5 total) 6ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:272M +| +- run test 428 pass (428 total) 1m MaxRSS:96M +| | +- compile test debug native success 31s MaxRSS:1G +| +- run test 5 pass (5 total) 16ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:285M +| +- run test 130 pass (130 total) 63ms MaxRSS:4M +| | +- compile test debug native success 6s MaxRSS:371M +| | +- options (reused) +| +- run test 280 pass, 2 skip (282 total) 9s MaxRSS:132M +| | +- compile test debug native success 8s MaxRSS:588M +| +- run test 163 pass (163 total) 13s MaxRSS:65M +| | +- compile test debug native success 8s MaxRSS:476M +| +- run test 74 pass, 1 skip (75 total) 303ms MaxRSS:10M +| | +- compile test debug native success 8s MaxRSS:495M +| | +- WriteFile draft_ids.txt (reused) +| | +- options (reused) +| +- run test 33 pass (33 total) 13s MaxRSS:24M +| | +- compile test debug native success 5s MaxRSS:341M +| | +- options (reused) +| +- run test 12 pass (12 total) 24ms MaxRSS:3M +| | +- compile test debug native success 4s MaxRSS:327M +| | +- options (reused) +| +- run test 83 pass (83 total) 326ms MaxRSS:74M +| | +- compile test debug native success 8s MaxRSS:448M +| | +- options (reused) +| +- run test 13 pass (13 total) 10s MaxRSS:15M +| | +- compile test debug native success 5s MaxRSS:392M +| | +- options (reused) +| +- run test 29 pass (29 total) 8s MaxRSS:1G +| | +- compile test debug native success 4s MaxRSS:343M +| | +- options (reused) +| +- run test 5 pass (5 total) 7ms MaxRSS:3M +| | +- compile test debug native success 4s MaxRSS:284M +| +- run test 20 pass (20 total) 181ms MaxRSS:5M +| | +- compile test debug native success 5s MaxRSS:342M +| +- run test 204 pass (204 total) 7s MaxRSS:42M +| | +- compile test debug native success 11s MaxRSS:530M +| +- run test 52 pass, 1 skip (53 total) 87ms MaxRSS:24M +| | +- compile test debug native success 12s MaxRSS:642M +| | +- options (reused) +| | +- WriteFile draft_ids.txt (reused) +| +- run test 3 pass (3 total) 6ms MaxRSS:3M +| | +- compile test debug native success 3s MaxRSS:270M +| +- run test 52 pass (52 total) 41ms MaxRSS:61M +| | +- compile test debug native success 5s MaxRSS:394M +| | +- options (reused) +| +- run test 5 pass (5 total) 5ms MaxRSS:3M +| | +- compile test debug native success 4s MaxRSS:272M +| +- run test 25 pass (25 total) 1m MaxRSS:236M +| | +- compile test debug native success 5s MaxRSS:332M +| +- run test 1 pass (1 total) 5ms MaxRSS:3M +| +- compile test debug native success 3s MaxRSS:294M +| +- options (reused) +| +- options (reused) +| +- WriteFile draft_ids.txt (reused) +| +- WriteFile exlr8_alias.zig cached ++- run test 4 pass (4 total) 16ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:268M +| +- options success ++- run test 13 pass, 1 skip (14 total) 61ms MaxRSS:4M +| +- compile test debug native success 2s MaxRSS:318M +| +- options (reused) ++- run test 3 pass (3 total) 7ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:281M +| +- options (reused) ++- run test 1 pass (1 total) 4ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:268M ++- test-kv-pool success +| +- run test 9 pass (9 total) 13ms MaxRSS:3M +| | +- compile test debug native success 2s MaxRSS:272M +| +- run test 6 pass (6 total) 2s MaxRSS:34M +| +- compile test debug native success 3s MaxRSS:389M ++- run test 51 pass (51 total) 41ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:324M +| +- options success +| | +- compile lib hip-mock-success debug native success 135ms MaxRSS:56M +| | +- compile lib hip-mock-failed debug native success 7s MaxRSS:330M +| | +- compile lib hip-mock-missing debug native success 127ms MaxRSS:55M +| +- WriteFile kernels.zig cached ++- run test 23 pass, 1 skip (24 total) 104ms MaxRSS:5M +| +- compile test debug native success 3s MaxRSS:354M ++- run test 49 pass (49 total) 36ms MaxRSS:9M +| +- compile test debug native success 2s MaxRSS:347M +| +- WriteFile kernels.zig success ++- run test 30 pass, 1 skip (31 total) 121ms MaxRSS:5M +| +- compile test debug native success 5s MaxRSS:480M +| +- WriteFile kernels.zig (reused) ++- run test 18 pass (18 total) 215ms MaxRSS:7M +| +- compile test debug native success 5s MaxRSS:509M +| +- WriteFile kernels.zig (reused) ++- run test 4 pass (4 total) 23ms MaxRSS:3M +| +- compile test debug native success 1s MaxRSS:284M +| +- WriteFile kernels.zig (reused) ++- run test 3 pass (3 total) 5ms MaxRSS:3M +| +- compile test debug native success 2s MaxRSS:288M +| +- WriteFile kernels.zig (reused) ++- compile exe tf-qwen35-test debug native success 1s MaxRSS:169M +| +- WriteFile kernels.zig (reused) ++- compile exe tensorfold-native safe native success 2s MaxRSS:257M +| +- options (reused) +| +- WriteFile exlr8_alias.zig cached +| +- WriteFile kernels.zig (reused) ++- run test 26 pass, 5 skip (31 total) 16ms MaxRSS:6M +| +- compile test safe native-native success 16s MaxRSS:537M ++- run test 112 pass (112 total) 3s MaxRSS:41M +| +- compile test debug native success 3s MaxRSS:389M ++- run test 20 pass (20 total) 124ms MaxRSS:11M +| +- compile test debug native success 2s MaxRSS:321M ++- run exe learned-warning-test success 15ms + +- compile exe learned-warning-test debug native success 2s MaxRSS:291M + +# exit=0 after 455s diff --git a/docs/receipts/rocm-host-identity/provenance.tsv b/docs/receipts/rocm-host-identity/provenance.tsv new file mode 100644 index 0000000000..b67b0adef7 --- /dev/null +++ b/docs/receipts/rocm-host-identity/provenance.tsv @@ -0,0 +1,44 @@ +published raw_name raw_sha256 published_sha256 sanitized +gpu-run/heads.txt heads.txt f9ef9a708c394481e9541c28d9852536dee07d9a5e644e17daa4f6542dc6cb9e f9ef9a708c394481e9541c28d9852536dee07d9a5e644e17daa4f6542dc6cb9e unchanged +gpu-run/binary.sha256 binary.sha256 withheld: sanitized c8d46b5b5e8f9361422854deb9ec085dddcf4cbe1024963ac8d48ec0570e9b34 home path x1 +gpu-run/run.log run.log withheld: sanitized e7118e8cc40d5469c81eca4215abfd7e2c7ed03ada41153ce8ddc4728dbc6d9c home path x4 +gpu-run/pull.log pull.log withheld: sanitized 8c9fe07b7b34c04f534b69eab48a9e2261072e40fcee63d5aedab132935ae4e8 home path x2 +gpu-run/check.mjs check.mjs 23b78e745c0519e1c012e62e17302f81f3ffb4a8441bb2cb0c3a780e80829466 23b78e745c0519e1c012e62e17302f81f3ffb4a8441bb2cb0c3a780e80829466 unchanged +gpu-run/27b-auto.drafts.json out/27b-auto.drafts.json 689ee610811cc72e505ddded6e982bdbdf337f7d6aafcb2c2bd227b78acb8870 689ee610811cc72e505ddded6e982bdbdf337f7d6aafcb2c2bd227b78acb8870 unchanged +gpu-run/27b-auto.json out/27b-auto.json 95ae8bad28ac5702aa66c326f96ac7b6816a4a5834f4f779c35ed117bb5c7a6e 95ae8bad28ac5702aa66c326f96ac7b6816a4a5834f4f779c35ed117bb5c7a6e unchanged +gpu-run/27b-auto.log out/27b-auto.log withheld: sanitized 32ed01a245b52ad8823c332a4ef30ae381503d754abdb1a3efa8bf98d2baad66 home path x2 +gpu-run/27b-dflash2.log out/27b-dflash2.log d0c05b5d100079881ccb6a5cf05060629cc529c87893eac3c625fda50439e8c2 d0c05b5d100079881ccb6a5cf05060629cc529c87893eac3c625fda50439e8c2 unchanged +gpu-run/27b-serve-drafter.log out/27b-serve-drafter.log c53992d45e846da1a8d97e0864ea658c256624bc891621b558d1de5f9aae5a9e c53992d45e846da1a8d97e0864ea658c256624bc891621b558d1de5f9aae5a9e unchanged +gpu-run/35b-attached.drafts.json out/35b-attached.drafts.json b96cd7d5b770026988bf13a1d5b7a6301fda14de94350d5744ac26fd2c9e2807 b96cd7d5b770026988bf13a1d5b7a6301fda14de94350d5744ac26fd2c9e2807 unchanged +gpu-run/35b-attached.json out/35b-attached.json 097812b10a5665c85289b2d4142dd79385855f23cac533ae9b8098c9b4dd677c 097812b10a5665c85289b2d4142dd79385855f23cac533ae9b8098c9b4dd677c unchanged +gpu-run/35b-attached.log out/35b-attached.log withheld: sanitized 6f0cb0ba3dc3a742be06063aa5bb20ab54b506534f9319858ed0e1b29229bbb1 home path x2 +gpu-run/35b-manifest.json out/35b-manifest.json 376497f0f00b1b5277e641ccf66abc79b1ded09f947f0016e87040f80ae276f0 376497f0f00b1b5277e641ccf66abc79b1ded09f947f0016e87040f80ae276f0 unchanged +gpu-run/35b-serial.drafts.json out/35b-serial.drafts.json 92fe121f484b1682a0ed911e8fc025a580343a496b362464dc24c089d7715a62 92fe121f484b1682a0ed911e8fc025a580343a496b362464dc24c089d7715a62 unchanged +gpu-run/35b-serial.json out/35b-serial.json c61da6ee47d938af90a55fea95153c06cfceccfbd6272d50dbfa08aff0d2b08c c61da6ee47d938af90a55fea95153c06cfceccfbd6272d50dbfa08aff0d2b08c unchanged +gpu-run/35b-serial.log out/35b-serial.log withheld: sanitized 7e6d8dfd93e5b2ee7a516fc7b6e12bd7179a6b867f3113bb7118d94db4d54111 home path x2 +gpu-run/35b-serve.log out/35b-serve.log withheld: sanitized 86c2555de69aa7699a5d6e0d9c660120d5cede3d65a1d6528ba335c348824050 trailing whitespace +gpu-run/35b.drafts.json out/35b.drafts.json 9bb8baa0274cf8e4a1658dbc77ecf8fa22764e79ab3cec1bac093bbda47e6470 9bb8baa0274cf8e4a1658dbc77ecf8fa22764e79ab3cec1bac093bbda47e6470 unchanged +gpu-run/35b.json out/35b.json c895bc792123d2542555d71a17d5c563879ff4434b585daba38bec920026d70e c895bc792123d2542555d71a17d5c563879ff4434b585daba38bec920026d70e unchanged +gpu-run/35b.log out/35b.log withheld: sanitized a4af1cfe5c5b51d751ce6d4f560f106bb2faa84315bf9055896634839548ffb8 home path x2 +gpu-attached-rerun/35b-attached.drafts.json out-r5b/35b-attached.drafts.json ed60c8674304318863468c3db31f7ef3148c0d53a24f9b15b325ce01b2e181ce ed60c8674304318863468c3db31f7ef3148c0d53a24f9b15b325ce01b2e181ce unchanged +gpu-attached-rerun/35b-attached.json out-r5b/35b-attached.json 9646641b849fbed3d9f27311b36add30dcddfe68036a755cca10fd67fd1af457 9646641b849fbed3d9f27311b36add30dcddfe68036a755cca10fd67fd1af457 unchanged +gpu-attached-rerun/35b-attached.log out-r5b/35b-attached.log withheld: sanitized e267ac9084bc4927e40e625f579736237bf7930ab279d2a51c01f86e9f2e4f0e home path x2 +gpu-attached-rerun/35b-manifest.json out-r5b/35b-manifest.json 09a0007d8ce0b59842642ebb8b95f5995e94006d3194b567ac04b6733dab3d11 09a0007d8ce0b59842642ebb8b95f5995e94006d3194b567ac04b6733dab3d11 unchanged +gpu-attached-rerun/35b-serve.log out-r5b/35b-serve.log withheld: sanitized 1f946eab3ea5eeb58a250dbeed91b4ad4aec61cf888bd54e01d91f5324430012 trailing whitespace +gpu-attached-rerun/heads-r5b.txt out-r5b/heads-r5b.txt 3bbe4a9b12f87d63e4e6fed9ab5095923c748d09e209138521b281ee6b1e4a31 3bbe4a9b12f87d63e4e6fed9ab5095923c748d09e209138521b281ee6b1e4a31 unchanged +gpu-attached-rerun/run-r5b.log out-r5b/run-r5b.log withheld: sanitized 70cfeffeb0d1e6c9cfc28e8cd3636823bbc03cbace8514ddcf0097a613bd3261 home path x1 +host-controls/test-main-76aafcf2.log main-76aafcf2-test.log withheld: sanitized d757f88ab3b7a9ec5003aa956948fdc74a106d365882afe530ff48ce255ab1a2 home path x4 +host-controls/test-branch-82237515.log branch-82237515-test.log withheld: sanitized 3e6a5115c8d59953cf37e9ee4e6ed2071939a09c0d27d88fab5cb50465e45bc5 home path x4 +host-controls/negative-control-agent-positions.log negative-control-agent-positions-rerun.log withheld: sanitized 70aabac97256604d04b1e34b60ed8c17ab5987429576100a7591c26842e56a43 home path x3, trailing whitespace x3 +host-controls/negative-control-agent-positions-minimal.log negative-control-agent-positions.log withheld: sanitized d5d6f7b274486aeb7e83ed9e74b7b964d7b3684b445526202e5635096674a05a home path x3 +host-controls/negative-control-backend-name.log negative-control-backend-name.log withheld: sanitized 05b2a2cbf4d4cd1412cf8f62ef7dd437c382affa84e5b427994e1bd9b4d6456e home path x2, trailing whitespace x4 +host-controls/negative-control-revision.log negative-control-revision.log withheld: sanitized aa36292e675f4ee1c119b93e2e3f8b23ff2ddf707d148a4adb30386145c9d9c9 home path x8, trailing whitespace x2 +host-controls/negative-control-cores.log negative-control-cores.log withheld: sanitized 4e371a8ba0d5d460624c952adf43a2e4482a06ee8eb2b21081a9c41d7fe1b81b home path x6 +host-controls/negative-control-w-slash.log negative-control-w-slash.log withheld: sanitized f758685dc353c7509cf1c5b10577ab692d36a4c81161425ee7cd4e4120666a3c home path x4 +host-controls/negative-control-ranking-note.log negative-control-ranking-note.log withheld: sanitized 56cb6979e17413fa5730fe6acd62cad121fa3bc4fa10f4b3f6c6ffebb8615d24 home path x4 +host-controls/lint.log lint.raw.log 98071ed48e76891dfd1bc764bea839ede3e03ffb72b87eee5e9f7d3375eb35b2 98071ed48e76891dfd1bc764bea839ede3e03ffb72b87eee5e9f7d3375eb35b2 unchanged +host-controls/board-validator.log board-validator.log d29c91f5a59a9f10981b9427c1f13b4da2ed5a181c5249ed50d329b475544e80 d29c91f5a59a9f10981b9427c1f13b4da2ed5a181c5249ed50d329b475544e80 unchanged +host-controls/test-branch-fb8eaf85.log test-branch-fb8eaf85.raw.log withheld: sanitized 2dc2c0c3736b3659f2c6e2b2b204caf740a03815c2b13be06529408b02be0247 home path x4 +host-controls/negative-control-rocr-twice.log nc-rocr-twice.raw.log withheld: sanitized ea400f4d7e41cf1f327be10094923e1bea51c433b50f8765af49fb19d9d1e95f home path x6, trailing whitespace x2 +host-controls/negative-control-hip-list.log nc-hip-list.raw.log withheld: sanitized 949b8e2ab3ec65d147187125ea29e707a09167dbb4b6fe4fb3594768d6570a4d home path x3, trailing whitespace x1 +host-controls/negative-control-partial.log nc-partial.raw.log withheld: sanitized 406dea5c0acfa6ca66e1f654627b0d2a2c3d9bd7b5bc08e5ee6c15d5bc3c3d79 home path x6 diff --git a/zig/src/server/benchmark.zig b/zig/src/server/benchmark.zig index 1dcbf07b5d..af4d5cc546 100644 --- a/zig/src/server/benchmark.zig +++ b/zig/src/server/benchmark.zig @@ -229,7 +229,10 @@ fn show(ctx: *const Context, v: json.Value) !void { try say(ctx, "{s:<14} {s:<6} {s:<25} {s:<16} {s}", .{ c.fixture_id, suite.fmtG(&t, c.temperature), rate, ttft, if (c.protocol_complete) "complete" else "custom/incomplete" }); } try say(ctx, "Stream delivery includes transport buffering; it is not an engine token clock.", .{}); - if (settings.get("managed").? != .bool or !settings.get("managed").?.bool) { + const managed = settings.get("managed").? == .bool and settings.get("managed").?.bool; + if (managed) if (try unranked(ctx.a, v)) |lacks| + try say(ctx, "The board ranks a run that names its GPU, checkpoint revision and configuration digest; this receipt names no {s}.", .{lacks}); + if (!managed) { const source = json.strOr(v.get("hardware").?.get("source")); const version = json.strOr(v.get("runtime").?.get("tensorfold_version")); const pinned = settings.get("rank_count").? != .null and v.get("model").?.get("revision").? != .null; @@ -242,6 +245,19 @@ fn show(ctx: *const Context, v: json.Value) !void { try say(ctx, "This rocm receipt is saved locally; it cannot be published until the board accepts rocm.", .{}); } +/// What a public managed receipt lacks of the facts the board ranks by (docs/community-benchmarks.md), else null. +pub fn unranked(a: Allocator, v: json.Value) !?[]const u8 { + const m = v.get("model").?; + if (m.get("local").? != .bool or m.get("local").?.bool) return null; // a local label is refused before publishing + var lacks: std.ArrayList([]const u8) = .empty; + const gpus = v.get("hardware").?.get("gpus").?; + if (gpus != .array or gpus.array.len == 0) try lacks.append(a, "GPU"); + if (m.get("revision").? == .null) try lacks.append(a, "checkpoint revision"); + if (m.get("config_sha256").? == .null) try lacks.append(a, "configuration digest"); + if (lacks.items.len == 0) return null; + return try std.mem.join(a, ", ", lacks.items); +} + /// What an engine that cannot sample answers a request with a temperature (server.zig's refusal). pub const greedy_only = "decodes greedily only"; diff --git a/zig/src/server/benchmark_facts_test.zig b/zig/src/server/benchmark_facts_test.zig index 5c85f16fb5..9b6e779842 100644 --- a/zig/src/server/benchmark_facts_test.zig +++ b/zig/src/server/benchmark_facts_test.zig @@ -8,6 +8,8 @@ const receipt = @import("benchmark_receipt.zig"); const fake = @import("benchmark_fake.zig"); const check = @import("benchmark_check.zig"); const suite = @import("benchmark_suite.zig"); +const manifest = @import("benchmark_manifest.zig"); +const benchmark = @import("benchmark.zig"); const revision = "0123456789abcdef0123456789abcdef01234567"; const families = [_]api.Family{.{ .model_type = "qwen3_5", .formats = &.{} }}; @@ -62,6 +64,83 @@ test "a cached repo takes its revision from the snapshot directory; a drafter re try std.testing.expect((try model.describe(a, std.testing.io, "example/model", dir, false, &.{}, null)).family == null); } +test "a server's manifest names the checkpoint a managed run records: the snapshot's revision and the same hashes" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + const root = try fake.tmpPath(a, &tmp); + const dir = try checkpoint(a, root, &.{ "models--example--model", "snapshots", revision }, config); + const managed = try model.describe(a, std.testing.io, "example/model", dir, false, &families, null); + var env: std.process.Environ.Map = .init(a); + const doc = try manifest.serveDocument(a, std.testing.io, &env, "9.8.7", "rocm", dir); + const attached = (try manifest.parse(a, try json.stringify(a, doc, .{}), "rocm")).?; + try std.testing.expectEqualStrings(managed.revision.?, attached.revision.?); + try std.testing.expectEqualStrings(managed.config_sha256.?, attached.config_sha256.?); + try std.testing.expectEqualStrings(managed.tokenizer_sha256.?, attached.tokenizer_sha256.?); + // the GPU facts a managed run records on this host, cores included, are the ones the server attests + const probe = host.detect(a, std.testing.io, &env, "rocm"); + try std.testing.expectEqual(probe.hardware.gpus.len, attached.hardware.gpus.len); + for (probe.hardware.gpus, attached.hardware.gpus) |want, got| { + try std.testing.expectEqualStrings(want.name, got.name); + try std.testing.expectEqual(want.memory_bytes, got.memory_bytes); + try std.testing.expectEqual(want.cores, got.cores); + } +} + +test "a HIP or Metal server's manifest names the receipt's backend, so an attached rocm or mlx run reads it" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + var env: std.process.Environ.Map = .init(a); + for ([_][2][]const u8{ .{ "hip", "rocm" }, .{ "metal", "mlx" }, .{ "cuda", "cuda" } }) |pair| { + try std.testing.expectEqualStrings(pair[1], manifest.receiptBackend(pair[0])); + const doc = try manifest.serveDocument(a, std.testing.io, &env, "9.8.7", pair[0], ""); + try std.testing.expectEqualStrings(pair[1], doc.get("runtime").?.get("backend").?.string); + const facts = (try manifest.parse(a, try json.stringify(a, doc, .{}), pair[1])).?; + try std.testing.expectEqualStrings(pair[1], facts.runtime.backend); + // the probe runs for the receipt's backend: on a rocm server it is rocminfo, as a managed rocm run reads it + const probe = host.detect(a, std.testing.io, &env, pair[1]); + try std.testing.expectEqual(probe.hardware.gpus.len, facts.hardware.gpus.len); + } + try std.testing.expectEqualStrings("", manifest.receiptBackend("")); +} + +test "the attested revision: the loaded snapshot's; the environment's only where the directory names none; never a disagreeing one" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + const other = "fedcba9876543210fedcba9876543210fedcba98"; + const snapshot = "/cache/models--example--model/snapshots/" ++ revision; + var env: std.process.Environ.Map = .init(a); + try std.testing.expectEqualStrings(revision, manifest.attestedRevision(&env, snapshot).?); + try std.testing.expect(manifest.attestedRevision(&env, "/models/private-checkpoint") == null and manifest.attestedRevision(&env, "") == null); + try std.testing.expect(manifest.attestedRevision(&env, "/cache/snapshots/main") == null); + try env.put("TF_CHECKPOINT_REVISION", revision); + try std.testing.expectEqualStrings(revision, manifest.attestedRevision(&env, snapshot).?); + try env.put("TF_CHECKPOINT_REVISION", other); + try std.testing.expect(manifest.attestedRevision(&env, snapshot) == null); + try std.testing.expectEqualStrings(other, manifest.attestedRevision(&env, "/models/private-checkpoint").?); + try std.testing.expectEqualStrings(other, manifest.attestedRevision(&env, "").?); + try std.testing.expectEqualStrings(revision, model.snapshotRevision(snapshot).?); + for ([_][]const u8{ "/cache/snapshots/" ++ "0123456789ABCDEF0123456789abcdef01234567", "/cache/snapshots/0123", "/cache/other/" ++ revision, revision }) |no| + try std.testing.expect(model.snapshotRevision(no) == null); +} + +test "a manifest's GPU cores reach the attached receipt; cores past the board's range refuse the manifest" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + const body = + \\{"schema_version":1,"runtime":{"tensorfold_version":"1.1.1","backend":"rocm","python_version":"none","platform":"linux"},"model":{},"hardware":{"cpu_model":"AMD RYZEN AI MAX+ 395 with Radeon 8060S","system_memory_bytes":134217728000,"memory_type":"unified","gpus":[{"name":"AMD Radeon 8060S Graphics","memory_bytes":128849018880,"cores":40}],"gpu_count":1,"source":"detected"}} + ; + const facts = (try manifest.parse(a, body, "rocm")).?; + try std.testing.expectEqual(@as(?u64, 40), facts.hardware.gpus[0].cores); + const too_many = try std.mem.replaceOwned(u8, a, body, "\"cores\":40", "\"cores\":100001"); + try std.testing.expect((try manifest.parse(a, too_many, "rocm")) == null); +} + test "a known drafter is named only for a target of its family and shape" { var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); defer arena.deinit(); @@ -123,7 +202,7 @@ test "hardware names pass only the allowlist; Mac and NVIDIA probes keep only na try std.testing.expectEqualStrings("Intel(R) Xeon(R) CPU E5-2680 v4", host.modelName(a, " Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz\n").?); try std.testing.expectEqualStrings("Apple M5 Max", host.modelName(a, "Apple M5 Max").?); const long = try std.mem.concat(a, u8, &.{ "NVIDIA ", try a.dupe(u8, &(@as([121]u8, @splat('a')))) }); - for ([_][]const u8{ "private-machine", "/home/example/private", "NVIDIA /home/example/private", "NVIDIA NPU", long, "Apple M5 Max Extra", "Apple M", "Intel user@example.test" }) |bad| try std.testing.expect(host.modelName(a, bad) == null); + for ([_][]const u8{ "private-machine", "/home/example/private", "NVIDIA /home/example/private", "NVIDIA NPU", "AMD w/home/example", "AMD Ryzen w/ /home", long, "Apple M5 Max Extra", "Apple M", "Intel user@example.test" }) |bad| try std.testing.expect(host.modelName(a, bad) == null); const gib128: u64 = 128 << 30; const profiler = "{\"SPDisplaysDataType\": [{\"sppci_model\": \"Apple M5 Max\", \"sppci_cores\": \"40\", \"serial_number\": \"private-device-value\", \"_name\": \"private-computer\", \"sppci_device_id\": \"private-id\"}]}"; const mac = host.summarize("Apple M5 Max", gib128, host.macGpus(a, profiler, gib128), true); @@ -159,14 +238,83 @@ test "/proc text gives the CPU name and memory" { try std.testing.expectEqual(@as(?u64, null), host.meminfo("MemTotal: 5 MB\n")); } -test "a rocm probe runs no nvidia-smi, names no GPU or driver, and passes the save rules" { +const rocminfo_strix = + \\ROCk module is loaded + \\HSA Agents + \\******* + \\Agent 1 + \\******* + \\ Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S + \\ Uuid: CPU-XX + \\ Marketing Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S + \\ Vendor Name: CPU + \\ Device Type: CPU + \\ Compute Unit: 32 + \\ Pool Info: + \\ Pool 1 + \\ Segment: GLOBAL; FLAGS: FINE GRAINED + \\ Size: 127838212(0x79ea604) KB + \\******* + \\Agent 2 + \\******* + \\ Name: gfx1151 + \\ Uuid: GPU-XX + \\ Marketing Name: Radeon 8060S Graphics + \\ Vendor Name: AMD + \\ Device Type: GPU + \\ Chip ID: 5510(0x1586) + \\ Compute Unit: 40 + \\ Pool Info: + \\ Pool 1 + \\ Segment: GLOBAL; FLAGS: COARSE GRAINED + \\ Size: 125829120(0x7800000) KB + \\ Pool 2 + \\ Segment: GROUP + \\ Size: 64(0x40) KB + \\ ISA Info: + \\ ISA 1 + \\ Name: amdgcn-amd-amdhsa--gfx1151 + \\ +; + +const rocminfo_two_dgpus = + \\******* + \\Agent 2 + \\******* + \\ Marketing Name: AMD Radeon PRO W7800 + \\ Vendor Name: AMD + \\ Device Type: GPU + \\ Compute Unit: 70 + \\ Segment: GLOBAL; FLAGS: COARSE GRAINED + \\ Size: 50331648(0x3000000) KB + \\******* + \\Agent 3 + \\******* + \\ Marketing Name: AMD Radeon PRO V620 + \\ Vendor Name: AMD + \\ Device Type: GPU + \\ Compute Unit: 72 + \\ Segment: GLOBAL; FLAGS: COARSE GRAINED + \\ Size: 31440896(0x1dfc000) KB + \\ +; + +test "a rocm probe names rocminfo's GPU agents, runs no nvidia-smi, reports no driver, and passes the save rules" { var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); defer arena.deinit(); const a = arena.allocator(); - try std.testing.expect(host.gpuQuery("rocm") == null and std.mem.eql(u8, host.gpuQuery("cuda").?[0], "nvidia-smi")); - const probe = host.linux(a, "AMD EPYC 9004", 128 << 30, "", null); - try std.testing.expect(probe.driver == null and probe.hardware.gpus.len == 0); - try std.testing.expect(std.mem.eql(u8, probe.hardware.source, "partial") and std.mem.eql(u8, probe.hardware.memory_type, "unknown")); + try std.testing.expect(std.mem.eql(u8, host.gpuQuery("rocm").?[0], "rocminfo") and std.mem.eql(u8, host.gpuQuery("cuda").?[0], "nvidia-smi")); + const cpu = host.cpuinfo(a, "model name\t: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S\n"); + try std.testing.expectEqualStrings("AMD RYZEN AI MAX+ 395 with Radeon 8060S", cpu.?); + try std.testing.expect(check.name(cpu.?)); + const probe = host.rocm(a, cpu, 128 << 30, rocminfo_strix, null); + try std.testing.expect(probe.driver == null and probe.hardware.gpus.len == 1); + const g = probe.hardware.gpus[0]; + try std.testing.expectEqualStrings("AMD Radeon 8060S Graphics", g.name); + try std.testing.expectEqual(@as(?u64, 125829120 * 1024), g.memory_bytes); + try std.testing.expectEqual(@as(?u64, 40), g.cores); + try std.testing.expect(std.mem.eql(u8, probe.hardware.source, "detected") and std.mem.eql(u8, probe.hardware.memory_type, "unified")); + try std.testing.expectEqualStrings("partial", host.rocm(a, null, 128 << 30, rocminfo_strix, null).hardware.source); const runtime: receipt.Runtime = .{ .tensorfold_version = "0.0.0-test", .backend = "rocm", .python_version = "none", .platform = "linux", .driver = probe.driver }; const sample: suite.Sample = .{ .fixture_id = "code", .temperature = 0, .repeat = -1, .seed = 1, .status = .@"error", .error_code = .server_error }; const v = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", .{ .repo_id = "example-org/test-model", .local = false }, runtime, probe.hardware, .{ .tokens = 2, .repetitions = 1, .temperatures = &.{0}, .serial = false, .context_tokens = null, .managed = true }, &.{sample}); @@ -176,4 +324,121 @@ test "a rocm probe runs no nvidia-smi, names no GPU or driver, and passes the sa return error.TestUnexpectedResult; } try std.testing.expect(v.get("runtime").?.get("dependencies").?.object.count() == 0); + // what the board validates in the CPU, GPU and checkpoint fields passes its publish rules; only the backend waits + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + const dir = try checkpoint(a, try fake.tmpPath(a, &tmp), &.{ "snapshots", revision }, config); + const checkpoint_facts = try model.describe(a, std.testing.io, "example-org/test-model", dir, false, &families, null); + try std.testing.expect(checkpoint_facts.revision != null and checkpoint_facts.config_sha256 != null and checkpoint_facts.tokenizer_sha256 != null); + const settings: receipt.Settings = .{ .tokens = 2, .repetitions = 1, .temperatures = &.{0}, .serial = false, .context_tokens = null, .managed = true, .rank_count = 1 }; + const rocm_receipt = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", checkpoint_facts, runtime, probe.hardware, settings, &.{sample}); + try std.testing.expect(!check.receipt(rocm_receipt, .publish, 1767225600000, &why)); + try std.testing.expectEqualStrings(check.rocm_local, why); + var board_backend = runtime; + board_backend.backend = "cuda"; + const same_facts = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", checkpoint_facts, board_backend, probe.hardware, settings, &.{sample}); + if (!check.receipt(same_facts, .publish, 1767225600000, &why)) { + std.debug.print("rocm identity failed the publish rules: {s}\n", .{why}); + return error.TestUnexpectedResult; + } + // the client says what the board will not rank: the facts a probe or checkpoint left out + try std.testing.expect(try benchmark.unranked(a, same_facts) == null); + const no_gpu = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", checkpoint_facts, runtime, host.rocm(a, cpu, 128 << 30, "", null).hardware, settings, &.{sample}); + try std.testing.expectEqualStrings("GPU", (try benchmark.unranked(a, no_gpu)).?); + var unpinned = checkpoint_facts; + unpinned.revision = null; + unpinned.config_sha256 = null; + const loose = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", unpinned, runtime, probe.hardware, settings, &.{sample}); + try std.testing.expectEqualStrings("checkpoint revision, configuration digest", (try benchmark.unranked(a, loose)).?); + const local = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", .{ .local = true }, runtime, .{}, settings, &.{sample}); + try std.testing.expect(try benchmark.unranked(a, local) == null); +} + +test "rocm GPUs: discrete cards are dedicated, visible-device indices select, and missing rocminfo leaves the GPUs unknown" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + const both = host.rocm(a, "AMD EPYC 9004", 128 << 30, rocminfo_two_dgpus, null); + try std.testing.expect(both.hardware.gpus.len == 2 and std.mem.eql(u8, both.hardware.memory_type, "dedicated")); + try std.testing.expectEqualStrings("AMD Radeon PRO V620", both.hardware.gpus[1].name); + const second = host.rocm(a, "AMD EPYC 9004", 128 << 30, rocminfo_two_dgpus, "1"); + try std.testing.expect(second.hardware.gpus.len == 1 and second.hardware.gpus[0].cores.? == 72); + for ([_][]const u8{ "-1", " ", "x" }) |sel| try std.testing.expect(host.rocm(a, null, null, rocminfo_two_dgpus, sel).hardware.gpus.len == 0); + const none = host.rocm(a, "AMD EPYC 9004", 128 << 30, "", null); + try std.testing.expect(none.hardware.gpus.len == 0 and std.mem.eql(u8, none.hardware.source, "partial") and std.mem.eql(u8, none.hardware.memory_type, "unknown")); +} + +// the first GPU agent's name fails the allowlist; it still holds index 0 of the visible-device indices +const rocminfo_unnamed_first = + \\******* + \\Agent 2 + \\******* + \\ Marketing Name: /home/example/private + \\ Vendor Name: AMD + \\ Device Type: GPU + \\ Compute Unit: 70 + \\ Segment: GLOBAL; FLAGS: COARSE GRAINED + \\ Size: 50331648(0x3000000) KB + \\******* + \\Agent 3 + \\******* + \\ Marketing Name: AMD Radeon PRO V620 + \\ Vendor Name: AMD + \\ Device Type: GPU + \\ Compute Unit: 72 + \\ Segment: GLOBAL; FLAGS: COARSE GRAINED + \\ Size: 31440896(0x1dfc000) KB + \\ +; + +test "rocm visible-device indices count every GPU agent: selecting an unreportable agent lists no GPU, never the next" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + const v620: []const []const u8 = &.{"AMD Radeon PRO V620"}; + // with no list or 0,1 HIP sees both agents and the probe names one, so the list is short and the source partial + const cases = [_]struct { key: []const u8, visible: []const u8, want: []const []const u8, source: []const u8 }{ + .{ .key = "", .visible = "", .want = v620, .source = "partial" }, + .{ .key = "HIP_VISIBLE_DEVICES", .visible = "1", .want = v620, .source = "detected" }, + .{ .key = "HIP_VISIBLE_DEVICES", .visible = "0", .want = &.{}, .source = "partial" }, + .{ .key = "HIP_VISIBLE_DEVICES", .visible = "0,1", .want = v620, .source = "partial" }, + .{ .key = "CUDA_VISIBLE_DEVICES", .visible = "1", .want = v620, .source = "detected" }, + .{ .key = "CUDA_VISIBLE_DEVICES", .visible = "0", .want = &.{}, .source = "partial" }, + }; + const runtime: receipt.Runtime = .{ .tensorfold_version = "0.0.0-test", .backend = "rocm", .python_version = "none", .platform = "linux" }; + const settings: receipt.Settings = .{ .tokens = 2, .repetitions = 1, .temperatures = &.{0}, .serial = false, .context_tokens = null, .managed = true }; + const sample: suite.Sample = .{ .fixture_id = "code", .temperature = 0, .repeat = -1, .seed = 1, .status = .@"error", .error_code = .server_error }; + var wrong: usize = 0; + for (cases) |c| { + var env: std.process.Environ.Map = .init(a); + if (c.key.len > 0) try env.put(c.key, c.visible); + const hw = host.rocm(a, "AMD EPYC 9004", 128 << 30, rocminfo_unnamed_first, host.rocmVisible(&env)).hardware; + const names = try a.alloc([]const u8, hw.gpus.len); + for (hw.gpus, names) |g, *n| n.* = g.name; + const got = try std.fmt.allocPrint(a, "[{s}] {s}", .{ try std.mem.join(a, ", ", names), hw.source }); + const want = try std.fmt.allocPrint(a, "[{s}] {s}", .{ try std.mem.join(a, ", ", c.want), c.source }); + if (!std.mem.eql(u8, got, want)) { + std.debug.print("{s}={s}: listed GPUs {s}, want {s}\n", .{ c.key, c.visible, got, want }); + wrong += 1; + continue; + } + if (hw.gpus.len == 1) try std.testing.expectEqual(@as(?u64, 72), hw.gpus[0].cores); + // the receipt's gpu_count is the length of that list, and the save rules accept it + const v = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", .{ .repo_id = "example-org/test-model", .local = false }, runtime, hw, settings, &.{sample}); + const count = v.get("hardware").?.get("gpu_count").?; + try std.testing.expect(if (hw.gpus.len == 0) count == .null else count.int64().? == @as(i64, @intCast(hw.gpus.len))); + var why: []const u8 = ""; + if (!check.receipt(v, .save, 1767225600000, &why)) { + std.debug.print("{s}={s}: save refused: {s}\n", .{ c.key, c.visible, why }); + return error.TestUnexpectedResult; + } + } + try std.testing.expectEqual(@as(usize, 0), wrong); + // HIP reads HIP_VISIBLE_DEVICES, else CUDA_VISIBLE_DEVICES; ROCr applies ROCR_VISIBLE_DEVICES inside rocminfo + var env: std.process.Environ.Map = .init(a); + try env.put("ROCR_VISIBLE_DEVICES", "0"); + try std.testing.expect(host.rocmVisible(&env) == null); + try env.put("CUDA_VISIBLE_DEVICES", "0"); + try env.put("HIP_VISIBLE_DEVICES", "1"); + try std.testing.expectEqualStrings("1", host.rocmVisible(&env).?); } diff --git a/zig/src/server/benchmark_host.zig b/zig/src/server/benchmark_host.zig index 9c80c75681..df1b161bcf 100644 --- a/zig/src/server/benchmark_host.zig +++ b/zig/src/server/benchmark_host.zig @@ -1,4 +1,4 @@ -//! The hardware allowlist per OS: CPU and GPU names, memory, cores, and the CUDA driver version. +//! The hardware allowlist per OS; on ROCm the GPUs come from rocminfo's agents and no driver version. const std = @import("std"); const builtin = @import("builtin"); const json = @import("json"); @@ -12,7 +12,7 @@ pub const unavailable: Hardware = .{}; /// What the probes found: the hardware block, and on Linux the single driver version nvidia-smi reports. pub const Probe = struct { hardware: Hardware, driver: ?[]const u8 = null }; -/// The host's allowlisted facts for a backend; no hostname, serials, UUIDs or model identifier is read. +/// The host's allowlisted facts for a backend; no hostname, serial, UUID or model identifier is recorded. pub fn detect(a: Allocator, io: std.Io, env: *const std.process.Environ.Map, backend: []const u8) Probe { if (builtin.os.tag == .macos) { const cpu = modelName(a, sysctlString("machdep.cpu.brand_string") orelse ""); @@ -22,16 +22,147 @@ pub fn detect(a: Allocator, io: std.Io, env: *const std.process.Environ.Map, bac } const cpu = cpuinfo(a, readProc(a, io, "/proc/cpuinfo") orelse ""); const memory = meminfo(readProc(a, io, "/proc/meminfo") orelse "") orelse (std.process.totalSystemMemory() catch null); - const csv = if (gpuQuery(backend)) |argv| run(a, io, argv) orelse "" else ""; - return linux(a, cpu, memory, csv, env.get("CUDA_VISIBLE_DEVICES")); + const out = if (gpuQuery(backend)) |argv| run(a, io, argv) orelse "" else ""; + if (std.mem.eql(u8, backend, "rocm")) return rocm(a, cpu, memory, out, rocmVisible(env)); + return linux(a, cpu, memory, out, env.get("CUDA_VISIBLE_DEVICES")); } -/// The GPU query behind a backend's facts: nvidia-smi for cuda; none for rocm, whose GPUs and driver stay unknown. +/// The GPU query behind a backend's facts: nvidia-smi for cuda, rocminfo for rocm; none otherwise. pub fn gpuQuery(backend: []const u8) ?[]const []const u8 { + if (std.mem.eql(u8, backend, "rocm")) return &.{"rocminfo"}; if (!std.mem.eql(u8, backend, "cuda")) return null; return &.{ "nvidia-smi", "--query-gpu=name,memory.total,driver_version", "--format=csv,noheader,nounits" }; } +/// HIP's device list as clr reads it (rocdevice.cpp): HIP_VISIBLE_DEVICES, else CUDA_VISIBLE_DEVICES; null for none. +pub fn rocmVisible(env: *const std.process.Environ.Map) ?[]const u8 { + // ROCR_VISIBLE_DEVICES is not read: ROCr applies it inside rocminfo, which lists only those GPUs, in its order + const hip = clrString(env.get("HIP_VISIBLE_DEVICES")); + const list = if (hip.len > 0) hip else clrString(env.get("CUDA_VISIBLE_DEVICES")); + return if (list.len > 0) list else null; +} + +/// A string variable as clr's flags.cpp reads it: unset is "", set but empty is " ", and one pair of quotes is dropped. +fn clrString(value: ?[]const u8) []const u8 { + const v = value orelse return ""; + if (v.len == 0) return " "; + const quoted = std.mem.trimEnd(u8, std.mem.trimStart(u8, v, " "), " "); + if (quoted.len < 2 or quoted[0] != '"' or quoted[quoted.len - 1] != '"') return v; + return quoted[1 .. quoted.len - 1]; +} + +/// ROCm facts from the GPU agents rocminfo lists that HIP keeps; an APU ("... Graphics") is ``unified``. +pub fn rocm(a: Allocator, cpu: ?[]const u8, memory: ?u64, text: []const u8, visible: ?[]const u8) Probe { + const none: Probe = .{ .hardware = summarize(cpu, memory, &.{}, false) }; + var sure = true; + // indices count every GPU agent, so selecting an agent with no reportable name lists no GPU, never the next one + const agents = hipVisible(a, amdAgents(a, text), visible, &sure) orelse return none; + var kept: std.ArrayList(Gpu) = .empty; + for (agents) |agent| { + const g = agent.gpu orelse { + sure = false; + continue; + }; + kept.append(a, g) catch return none; + } + const gpus = kept.items; + var h = summarize(cpu, memory, gpus, false); + // a GPU HIP keeps that has no reportable name, or an entry no Uuid resolves, leaves the list short + if (!sure and std.mem.eql(u8, h.source, "detected")) h.source = "partial"; + if (gpus.len > 0) h.memory_type = for (gpus) |g| { + if (!std.mem.endsWith(u8, g.name, " Graphics")) break "dedicated"; + } else "unified"; + return .{ .hardware = h }; +} + +/// A rocminfo GPU agent: its facts (null when the name fails the allowlist) and its Uuid, which only selects. +pub const Agent = struct { gpu: ?Gpu, uuid: ?[]const u8 }; + +/// Every rocminfo GPU agent in order (vendor-prefixed name, first GLOBAL pool, compute units, Uuid). +pub fn amdAgents(a: Allocator, text: []const u8) []const Agent { + var out: std.ArrayList(Agent) = .empty; + var agents = std.mem.splitSequence(u8, text, "*******"); + while (agents.next()) |block| { + var marketing: []const u8 = ""; + var vendor: []const u8 = ""; + var uuid: ?[]const u8 = null; + var gpu = false; + var g: Gpu = .{ .name = "" }; + var lines = std.mem.splitScalar(u8, block, '\n'); + var segment_global = false; + while (lines.next()) |l| { + const colon = std.mem.indexOfScalar(u8, l, ':') orelse continue; + const key = std.mem.trim(u8, l[0..colon], " \t"); + const value = std.mem.trim(u8, l[colon + 1 ..], " \t\r"); + if (std.mem.eql(u8, key, "Marketing Name")) marketing = value; + if (std.mem.eql(u8, key, "Vendor Name")) vendor = value; + if (std.mem.eql(u8, key, "Uuid") and uuid == null) uuid = value; + if (std.mem.eql(u8, key, "Device Type")) gpu = std.mem.eql(u8, value, "GPU"); + if (std.mem.eql(u8, key, "Compute Unit") and g.cores == null) g.cores = std.fmt.parseInt(u64, value, 10) catch null; + if (std.mem.eql(u8, key, "Segment")) segment_global = std.mem.startsWith(u8, value, "GLOBAL"); + if (std.mem.eql(u8, key, "Size") and segment_global and g.memory_bytes == null) g.memory_bytes = kib(value); + } + if (!gpu) continue; + const full = if (vendor.len > 0 and !std.mem.startsWith(u8, marketing, vendor)) + std.fmt.allocPrint(a, "{s} {s}", .{ vendor, marketing }) catch return &.{} + else + marketing; + const name = modelName(a, full); + const facts: ?Gpu = if (name) |n| .{ .name = n, .memory_bytes = g.memory_bytes, .cores = g.cores } else null; + out.append(a, .{ .gpu = facts, .uuid = uuid }) catch return &.{}; + } + return out.items; +} + +/// rocminfo's ``125829120(0x7800000) KB`` in bytes. +fn kib(value: []const u8) ?u64 { + var end: usize = 0; + while (end < value.len and std.ascii.isDigit(value[end])) end += 1; + if (end == 0 or !std.mem.endsWith(u8, value, " KB")) return null; + const n = std.fmt.parseInt(u64, value[0..end], 10) catch return null; + return if (n > 0 and n < 1 << 43) n * 1024 else null; +} + +/// The agents HIP keeps (clr Device::init): the first bad or out-of-range entry ends the list; a repeat is skipped. +fn hipVisible(a: Allocator, agents: []const Agent, list: ?[]const u8, sure: *bool) ?[]const Agent { + const ordinals = list orelse ""; + if (ordinals.len == 0) return agents; + const seen = a.alloc(bool, agents.len) catch return null; + @memset(seen, false); + var kept: std.ArrayList(Agent) = .empty; + var pos: usize = 0; + while (pos < ordinals.len) { + const end = std.mem.indexOfScalarPos(u8, ordinals, pos, ',') orelse ordinals.len; + const id = ordinals[pos..end]; + pos = end + 1; + const n = (if (std.mem.indexOf(u8, id, "GPU-") != null) uuidIndex(agents, id, sure) else hipIndex(id)) orelse break; + if (n >= agents.len) break; + if (seen[n]) continue; + seen[n] = true; + kept.append(a, agents[n]) catch return null; + } + return kept.items; +} + +/// The first GPU agent whose Uuid contains ``id``, as clr matches it; null, and not sure, past an agent with no Uuid. +fn uuidIndex(agents: []const Agent, id: []const u8, sure: *bool) ?usize { + for (agents, 0..) |agent, i| { + const uuid = agent.uuid orelse { + sure.* = false; + return null; + }; + if (std.mem.indexOf(u8, uuid, id) != null) return i; + } + return null; +} + +/// An index clr accepts, one ``std::to_string(atoi(id))`` writes back: digits only, no sign, space or leading zero. +fn hipIndex(id: []const u8) ?usize { + if (id.len == 0 or (id[0] == '0' and id.len > 1)) return null; + for (id) |ch| if (!std.ascii.isDigit(ch)) return null; + return std.fmt.parseInt(usize, id, 10) catch null; +} + /// Linux facts from the CPU and memory probes and the nvidia-smi rows, empty when no query ran. pub fn linux(a: Allocator, cpu: ?[]const u8, memory: ?u64, csv: []const u8, visible: ?[]const u8) Probe { const rows = nvidiaRows(a, csv, visible); @@ -59,13 +190,14 @@ pub fn summarize(cpu: ?[]const u8, memory: ?u64, gpus: []const Gpu, mlx: bool) H return .{ .cpu_model = cpu, .system_memory_bytes = memory, .memory_type = memory_type, .gpus = gpus, .source = if (complete) "detected" else if (evidence) "partial" else "unavailable" }; } -/// A processor or GPU description from the allowlist, whitespace collapsed and a clock suffix dropped; else null. +/// An allowlisted CPU or GPU name: spaces collapsed, "w/" as "with", clock suffix dropped; else null. pub fn modelName(a: Allocator, text: []const u8) ?[]const u8 { var words: std.ArrayList(u8) = .empty; var it = std.mem.tokenizeAny(u8, text, " \t\n\r\x0b\x0c"); while (it.next()) |w| { if (words.items.len > 0) words.append(a, ' ') catch return null; - words.appendSlice(a, w) catch return null; + // AMD's "w/" (Ryzen AI MAX+ 395 w/ Radeon 8060S) becomes "with", inside the board's name pattern + words.appendSlice(a, if (std.mem.eql(u8, w, "w/")) "with" else w) catch return null; } const name = dropClock(words.items); if (name.len > 120 or std.mem.eql(u8, name, "NVIDIA NPU")) return null; diff --git a/zig/src/server/benchmark_manifest.zig b/zig/src/server/benchmark_manifest.zig index 10aaa5cc71..601f8fdb20 100644 --- a/zig/src/server/benchmark_manifest.zig +++ b/zig/src/server/benchmark_manifest.zig @@ -141,7 +141,13 @@ fn hardwareOf(a: Allocator, v: Value) ?receipt.Hardware { if (n < 0) return null; break :blk @as(?u64, @intCast(n)); } else null; - slot.* = .{ .name = name, .memory_bytes = mem }; + const cores = if (g.get("cores")) |c| blk: { + if (c == .null) break :blk null; + const n = c.int64() orelse return null; + if (n < 0 or n > 100000) return null; + break :blk @as(?u64, @intCast(n)); + } else null; + slot.* = .{ .name = name, .memory_bytes = mem, .cores = cores }; } const count = v.get("gpu_count") orelse return null; if (count != .null) { @@ -181,8 +187,24 @@ fn revisionFromEnv(env: *const std.process.Environ.Map) ?[]const u8 { return raw; } -/// The document a running server can attest. Hardware is this process's own probe. -pub fn serveDocument(a: Allocator, io: std.Io, env: *const std.process.Environ.Map, version: []const u8, backend: []const u8, model_dir: []const u8) !Value { +/// The attested revision: the snapshot directory's name, else TF_CHECKPOINT_REVISION; a mismatch gives none. +pub fn attestedRevision(env: *const std.process.Environ.Map, model_dir: []const u8) ?[]const u8 { + const told = revisionFromEnv(env); + const loaded = (if (model_dir.len > 0) model_facts.snapshotRevision(model_dir) else null) orelse return told; + if (told) |t| if (!std.mem.eql(u8, t, loaded)) return null; + return loaded; +} + +/// The receipt's name for an engine backend, as the board names them: HIP is ``rocm``, Metal is ``mlx``. +pub fn receiptBackend(engine: []const u8) []const u8 { + if (std.mem.eql(u8, engine, "hip")) return "rocm"; + if (std.mem.eql(u8, engine, "metal")) return "mlx"; + return engine; +} + +/// The document a running server can attest, under the receipt's backend name. Hardware is this process's own probe. +pub fn serveDocument(a: Allocator, io: std.Io, env: *const std.process.Environ.Map, version: []const u8, engine: []const u8, model_dir: []const u8) !Value { + const backend = receiptBackend(engine); const probe = host.detect(a, io, env, backend); const top = try json.newObject(a); try top.put(a, "schema_version", try json.intValue(a, @as(u8, 1))); @@ -195,7 +217,7 @@ pub fn serveDocument(a: Allocator, io: std.Io, env: *const std.process.Environ.M if (probe.driver) |d| try runtime.put(a, "driver", .{ .string = d }); try top.put(a, "runtime", .{ .object = runtime }); const model = try json.newObject(a); - if (revisionFromEnv(env)) |rev| try model.put(a, "revision", .{ .string = rev }); + if (attestedRevision(env, model_dir)) |rev| try model.put(a, "revision", .{ .string = rev }); if (model_dir.len > 0) { if (fileSha(a, io, try std.fs.path.join(a, &.{ model_dir, "config.json" }))) |h| try model.put(a, "config_sha256", .{ .string = h }); if (model_facts.tokenizerHash(a, io, model_dir)) |h| try model.put(a, "tokenizer_sha256", .{ .string = h }); @@ -211,7 +233,7 @@ pub fn serveDocument(a: Allocator, io: std.Io, env: *const std.process.Environ.M const one = try json.newObject(a); try one.put(a, "name", .{ .string = g.name }); if (g.memory_bytes) |n| try one.put(a, "memory_bytes", try json.intValue(a, n)) else try one.put(a, "memory_bytes", .null); - try one.put(a, "cores", .null); + if (g.cores) |n| try one.put(a, "cores", try json.intValue(a, n)) else try one.put(a, "cores", .null); slot.* = .{ .object = one }; } try hardware.put(a, "gpus", .{ .array = gpus }); diff --git a/zig/src/server/benchmark_model.zig b/zig/src/server/benchmark_model.zig index 73f30e2f20..f61c4f36b0 100644 --- a/zig/src/server/benchmark_model.zig +++ b/zig/src/server/benchmark_model.zig @@ -37,9 +37,7 @@ pub fn describe(a: Allocator, io: std.Io, target: []const u8, dir: []const u8, l pub fn describeConfig(a: Allocator, io: std.Io, target: []const u8, dir: []const u8, local_dir: bool, families: []const api.Family, drafter: ?[]const u8, config: ?Config) !Model { const repo: ?[]const u8 = if (!local_dir and check.repoId(target)) target else null; var out: Model = .{ .repo_id = repo, .local = repo == null }; - const name = std.fs.path.basename(dir); - const parent = std.fs.path.basename(std.fs.path.dirname(dir) orelse ""); - if (repo != null and std.mem.eql(u8, parent, "snapshots") and (name.len == 40 or name.len == 64) and lowerHex(name)) out.revision = name; + if (repo != null) out.revision = snapshotRevision(dir); if (config) |c| out.config_sha256 = try hexDigest(a, c.bytes); const value: Value = if (config) |c| c.value else .null; if (modelType(value)) |kind| for (families) |f| { @@ -67,6 +65,14 @@ pub fn weightBytes(io: std.Io, dir: []const u8) u64 { return total; } +/// The commit a HF cache snapshot directory is named for (40 or 64 lowercase hex), for receipts and manifests. +pub fn snapshotRevision(dir: []const u8) ?[]const u8 { + const name = std.fs.path.basename(dir); + const parent = std.fs.path.basename(std.fs.path.dirname(dir) orelse ""); + if (std.mem.eql(u8, parent, "snapshots") and (name.len == 40 or name.len == 64) and lowerHex(name)) return name; + return null; +} + /// An attached server's facts: only the public ID it was asked for. pub fn attached(target: []const u8, target_is_path: bool) Model { const public = !target_is_path and check.repoId(target); diff --git a/zig/src/server/benchmark_rocm_test.zig b/zig/src/server/benchmark_rocm_test.zig new file mode 100644 index 0000000000..012cb46667 --- /dev/null +++ b/zig/src/server/benchmark_rocm_test.zig @@ -0,0 +1,102 @@ +//! ROCm visible devices on fictional rocminfo text: ROCr applied ROCR_VISIBLE_DEVICES, the probe applies HIP's list. +const std = @import("std"); +const host = @import("benchmark_host.zig"); +const receipt = @import("benchmark_receipt.zig"); +const check = @import("benchmark_check.zig"); +const suite = @import("benchmark_suite.zig"); + +const Card = struct { name: []const u8, uuid: ?[]const u8 }; +const card_a: Card = .{ .name = "AMD Radeon PRO W7800", .uuid = "GPU-a1a1a1a1a1a1a1a1" }; +const card_b: Card = .{ .name = "AMD Radeon PRO V620", .uuid = "GPU-b2b2b2b2b2b2b2b2" }; +const card_c: Card = .{ .name = "AMD Instinct MI210", .uuid = "GPU-c3c3c3c3c3c3c3c3" }; + +/// rocminfo's text for a CPU agent and then these GPU agents in order; a card with no uuid prints no Uuid line. +fn rocminfo(a: std.mem.Allocator, cards: []const Card) ![]const u8 { + var out: std.ArrayList(u8) = .empty; + try out.appendSlice(a, "*******\nAgent 1\n*******\n Uuid: CPU-XX\n Marketing Name: AMD EPYC 9004\n Vendor Name: CPU\n Device Type: CPU\n"); + for (cards, 2..) |c, n| { + try out.print(a, "*******\nAgent {d}\n*******\n", .{n}); + if (c.uuid) |u| try out.print(a, " Uuid: {s}\n", .{u}); + try out.print(a, " Marketing Name: {s}\n Vendor Name: AMD\n Device Type: GPU\n Compute Unit: 64\n", .{c.name}); + try out.appendSlice(a, " Segment: GLOBAL; FLAGS: COARSE GRAINED\n Size: 33554432(0x2000000) KB\n"); + } + return out.items; +} + +/// What the probe records under ``vars`` on this rocminfo text, as "[GPU, ...] source", once the save rules accept it. +fn listed(a: std.mem.Allocator, vars: []const [2][]const u8, text: []const u8) ![]const u8 { + var env: std.process.Environ.Map = .init(a); + for (vars) |kv| try env.put(kv[0], kv[1]); + const hw = host.rocm(a, "AMD EPYC 9004", 128 << 30, text, host.rocmVisible(&env)).hardware; + const names = try a.alloc([]const u8, hw.gpus.len); + for (hw.gpus, names) |g, *n| n.* = g.name; + const runtime: receipt.Runtime = .{ .tensorfold_version = "0.0.0-test", .backend = "rocm", .python_version = "none", .platform = "linux" }; + const settings: receipt.Settings = .{ .tokens = 2, .repetitions = 1, .temperatures = &.{0}, .serial = false, .context_tokens = null, .managed = true }; + const sample: suite.Sample = .{ .fixture_id = "code", .temperature = 0, .repeat = -1, .seed = 1, .status = .@"error", .error_code = .server_error }; + const v = try receipt.value(a, "00000000-0000-4000-8000-000000000001", "2026-01-01T00:00:00Z", .{ .repo_id = "example-org/test-model", .local = false }, runtime, hw, settings, &.{sample}); + var why: []const u8 = ""; + if (!check.receipt(v, .save, 1767225600000, &why)) return std.fmt.allocPrint(a, "save refused: {s}", .{why}); + return std.fmt.allocPrint(a, "[{s}] {s}", .{ try std.mem.join(a, ", ", names), hw.source }); +} + +const Case = struct { vars: []const [2][]const u8, text: []const u8, want: []const u8 }; + +/// Every case whose record differs from ``want`` is printed; the count of those is returned. +fn wrong(a: std.mem.Allocator, cases: []const Case) !usize { + var n: usize = 0; + for (cases) |c| { + const got = try listed(a, c.vars, c.text); + if (std.mem.eql(u8, got, c.want)) continue; + for (c.vars) |kv| std.debug.print("{s}={s} ", .{ kv[0], kv[1] }); + std.debug.print("(rocminfo lists {d} GPUs): recorded {s}, want {s}\n", .{ std.mem.count(u8, c.text, "Device Type: GPU"), got, c.want }); + n += 1; + } + return n; +} + +test "rocminfo lists only the GPUs ROCR_VISIBLE_DEVICES leaves, renumbered; the probe applies HIP's list on top, once" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + // rocminfo under ROCR_VISIBLE_DEVICES=1 on a host with cards A and B, and under =1,2 on a host with A, B and C + const rocr_1 = try rocminfo(a, &.{card_b}); + const rocr_12 = try rocminfo(a, &.{ card_b, card_c }); + const all = try rocminfo(a, &.{ card_a, card_b, card_c }); + const cases = [_]Case{ + .{ .vars = &.{.{ "ROCR_VISIBLE_DEVICES", "1" }}, .text = rocr_1, .want = "[AMD Radeon PRO V620] detected" }, + .{ .vars = &.{.{ "ROCR_VISIBLE_DEVICES", "1,2" }}, .text = rocr_12, .want = "[AMD Radeon PRO V620, AMD Instinct MI210] detected" }, + .{ .vars = &.{ .{ "ROCR_VISIBLE_DEVICES", "1,2" }, .{ "HIP_VISIBLE_DEVICES", "1" } }, .text = rocr_12, .want = "[AMD Instinct MI210] detected" }, + .{ .vars = &.{.{ "CUDA_VISIBLE_DEVICES", "1" }}, .text = all, .want = "[AMD Radeon PRO V620] detected" }, + .{ .vars = &.{ .{ "CUDA_VISIBLE_DEVICES", "1" }, .{ "HIP_VISIBLE_DEVICES", "2" } }, .text = all, .want = "[AMD Instinct MI210] detected" }, + .{ .vars = &.{ .{ "HIP_VISIBLE_DEVICES", "" }, .{ "CUDA_VISIBLE_DEVICES", "1" } }, .text = all, .want = "[] partial" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "\"1\"" }}, .text = all, .want = "[AMD Radeon PRO V620] detected" }, + }; + try std.testing.expectEqual(@as(usize, 0), try wrong(a, &cases)); +} + +test "HIP's list: the first bad or out-of-range entry ends it, a repeat is skipped, and GPU- text matches rocminfo's Uuid" { + var arena: std.heap.ArenaAllocator = .init(std.testing.allocator); + defer arena.deinit(); + const a = arena.allocator(); + const all = try rocminfo(a, &.{ card_a, card_b, card_c }); + // rocminfo text with no Uuid lines: no GPU- entry can be matched, so the cards before it stand, partial + const no_uuid = try rocminfo(a, &.{ .{ .name = card_a.name, .uuid = null }, .{ .name = card_b.name, .uuid = null } }); + const cases = [_]Case{ + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "0,3,2" }}, .text = all, .want = "[AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "0,0" }}, .text = all, .want = "[AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "2,0,2" }}, .text = all, .want = "[AMD Instinct MI210, AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "1,x,0" }}, .text = all, .want = "[AMD Radeon PRO V620] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "1,,0" }}, .text = all, .want = "[AMD Radeon PRO V620] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "0," }}, .text = all, .want = "[AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", " 1" }}, .text = all, .want = "[] partial" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "01" }}, .text = all, .want = "[] partial" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "-1" }}, .text = all, .want = "[] partial" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "GPU-c3c3c3c3c3c3c3c3" }}, .text = all, .want = "[AMD Instinct MI210] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "GPU-c3c3,0" }}, .text = all, .want = "[AMD Instinct MI210, AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "1,GPU-0000000000000000,0" }}, .text = all, .want = "[AMD Radeon PRO V620] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "GPU-C3C3C3C3C3C3C3C3" }}, .text = all, .want = "[] partial" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "1,0" }}, .text = no_uuid, .want = "[AMD Radeon PRO V620, AMD Radeon PRO W7800] detected" }, + .{ .vars = &.{.{ "HIP_VISIBLE_DEVICES", "1,GPU-b2b2" }}, .text = no_uuid, .want = "[AMD Radeon PRO V620] partial" }, + }; + try std.testing.expectEqual(@as(usize, 0), try wrong(a, &cases)); +} diff --git a/zig/src/server/root.zig b/zig/src/server/root.zig index c7b31b5004..41a1d799ec 100644 --- a/zig/src/server/root.zig +++ b/zig/src/server/root.zig @@ -103,6 +103,7 @@ test { _ = @import("benchmark_stream_test.zig"); _ = @import("benchmark_server_test.zig"); _ = @import("benchmark_facts_test.zig"); + _ = @import("benchmark_rocm_test.zig"); _ = @import("benchmark_manifest.zig"); _ = @import("benchmark_upload.zig"); _ = @import("benchmark_upload_test.zig");