Repository navigation
Conversation
|
Thanks @akol1, the same-bytes dual-width measurement is useful. Before we carry this with #827 on 1.1.2, please fix Posted by Ash's AI Assistant Synthena |
Round-2 receipts on
|
| artifact | bytes | sha256 |
|---|---|---|
reference-fp32-logits.f32 |
4,136,960 | 9698a8aaf3640869fc69fe539819845519cb1c0f94c72a5d5dedc1a94ced93ab |
reference-fp64-logits.f64 |
8,273,920 | 696de82fca773faa8329e4efca66e3caff4fc1d62c0d5d8ae596aee69176a037 |
precision-fp32-vs-fp64.json |
347 | 0f34cc96ee47781504ee10473b18369eb4b36f0e956720e2dabd023244576143 |
residency.json |
460 | 6cc7717fe5735f8bfa69223bda5c1188686163a3bc35c521af114c84d51095a3 |
| run log (stderr) | 5,020 | 2a8d8eb823ad8c1fd042689373faa9a0ec7e777ece2915d3dc8cdb98dd97f303 |
The logit arrays are byte-identical to the arrays produced on 53b790082 before the repair (same sha256 values), so the repair did not move any FP32 or FP64 model output. Input IDs [0, 671, 6102, 294, 8760, 344, 11111, 16], sha256 14da70af68c78cbf320dbcfac88e126563ebe7b8a212e7bb95f774b611ae0bf3; the eight tokens' argmax are 5, 41079, 294, 270, 344, 11111, 16, 660 at both widths with 0 greedy disagreements. Pack, Engram-tables, source and token-map pins are in the PR body.
4. Logs. Host: Apple M3 Ultra (80-core GPU, 512 GB unified memory), macOS 26.6.2, Zig 0.17.0.
zig build test-deepseek-v41-reference→Build Summary: 43/43 steps succeeded; 32/32 tests passed; the compiled test binary printsAll 32 tests passed.zig build test test-golden --summary all(cold cache outside the worktree) →Build Summary: 311/311 steps succeeded; 4442/4457 tests passed (15 skipped), exit 0 (the log'sfailed command:lines are expected negative-path output from tests that simulate backend failures; every build step succeeded).python3 tools/zig/lean_check.py→2696 files, 501 problems; the parent53b790082tree reports2696 files, 501 problems, so 0 new. (The first draft of these tests had twelve multi-line-comment problems; they were folded to one-line comments and the count returned to baseline.)zig fmt --check zig→ clean.
Scope is unchanged from the body: this is the offline diagnostic mode. The 1% whole-model gate and the long-history checks remain separate, and DeepSeek serving stays held until the whole-model gate passes.
|
Combined carry for the 1.1.2 line is up as a review artifact: akol1#3, based on Your asks are covered there: the One layout adaptation: the Engram export names the module where the 1.1.2 carry keeps it, On that carry: 43/43 reference steps and 23/23 tests, the full test plan completed with exit 0, |
|
CI on the combined carry: akol1#3's The one red job is No new ask; the carry artifact now has its own CI. |
|
The review for the whole stack is on #825. If serving is the lane to keep, this is its head: the dual-width oracle and the NaN narrowing repair are useful. Bug 3 there (the frozen projection test) is this PR's, and it inherits bugs 1 and 2. #1051's extra Engram work isn't in this head. - Synthema, Ash's AI Assistant. |
|
Both bugs fixed on this head. Bug 2 ( Bug 3 ( The frozen baseline is regenerated, because the old values were only reachable through the undefined shift. 8 of 16 Worth knowing: Full write-up of all three findings is on #825. |
|
Head updated to carry bug 1 from the stack review.
All four now pass a value: the loop at Verified on this head: This is a fast-forward, no rebase and no history change. The PR remains |
…he release's tf-dsv41-token-map derives the compressed key map from the original's tokenizer.json and inference config with no Python step: decode each id, fall back to the raw token when the decoder emits U+FFFD, normalize (NFKD, strip category-M marks, lowercase, the whitespace replaces, strip — the Rust chain NFKC, NFD, StripAccents, Lowercase, replaces, strip composes to this), dedupe first-seen. The multipliers come from numpy's PCG64 exactly: SeedSequence's hashmix pool, the XSL-RR output and Lemire's bounded draw, checked against the release's recorded words. The generated nfkd_data.zig comes from unicodedata 16.0.0 (the version unicode_data.zig pins); the generator stays in the fork. The golden test is the release's own 517,680-byte engram-token-map.bin, reproduced byte for byte.
… staging Config parsing, the Engram hash layout, table plans over the original shards' embed weight and scale, and the EXL3 resident weight staging behind paced reads. Host tests cover exact integer boundaries, table selection and the staged bytes. On a 512 GiB Mac the resident payload is about 385 GiB; freeBytes counts reclaimable speculative pages so the load admission sees real room. No kernels; the served engine comes with the lane backend.
… qualification runner The Engram hash/commit, sparse and compressed attention (CSA2), the hyperconnection mix, routed and shared experts, the cache and rotary glue as source Metal kernels, with zig/tests/deepseek_v41_reference as the independent CPU oracle: plain reductions in fp64, no native statistics, no injected coefficients. tf-dsv41-run drives admission, teacher-forced logits, chunk seams, same-input operator replays and profiling on the resident checkpoint. The retained whole-model gate stays failed at 4.0961% NRMS against the unchanged 1% limit; operator checks are evidence, not qualification.
…-Flash's MIT inference
…ps to it Rank-split packs (jayleaton's 2.8bpw) duplicate whole-load tensors in native.safetensors beside the per-layer shards; the index's weight map decides each name's owner file, so the plan no longer refuses the pack as duplicates.
The released checkpoints store the gate f16 and its bias f16; rank-split repacks store the weight bf16 and the bias f32. The dense kernel's weight dtype and the route kernel's bias dtype already decode all three, so the loader accepts the range and the encoders follow the stored tensor.
The gate kernel decodes its q/k weights by dtype, the Gate descriptor carries it, and the loader accepts the rank-split packs' bf16 projections beside the released f32 ones.
Jayleaton's 2.8bpw stores the compressor and indexer matrices as a bf16 weight beside the rotation scales, and prefers it over the trellis copy it also ships. The exl3 module gains the dense form: a plain product kernel over the rotated rows writing the sums region the rotate_out stage reads, so the stage composition and its bit boundaries stay the ones the reference defines. The loader rides the pack's dtypes for the router gate, bias, Engram gate weights and the indexer's weights projection; the reference takes the union and names the dense apply it still owes. The 2.8bpw plan passes: 406.6 GB, 44 shards.
The fp64 oracle's dense path mirrors the quantized one's boundaries: the input rotation keeps its f16 rounding, the product is an FP64 dot over the f16-rounded rotated row and the bf16 weight, and the output stages are the rotate_out contract. The reference errors until now named the missing half; the union is complete.
…ument, and the duplicate-tensor case stops logging at error level d58f281f grew Plan.add an owned-index argument and left the in-file tests calling the old shape; the duplicate-tensor path also logged at error level, which the test runner treats as a failure even where the test expects the error.
…indow kernel's brace
…nuated Ported from the mhc worktree: the hyperconnection mix's per-stage fp64 harness (operator_hc_fp64.zig) and the comptime-Real reference parameterization in math.zig. Verdicts and the 1-ULP attenuation receipts live in the fork.
…se-head boundaries Carries the in-progress edits: comptime Real-style math (wide/d64/b64, rmsOf/rmsOfValues, poolOf, sparseHeadOf + KeyTable) with the FP32 instances kept as the source contract. Two repairs to that draft: model.zig re-exported kv as a duplicate member (compile error), and rmsOfValues captured the unused row index in its f32/f64 loops.
…l-faithful key addressing
…lection agreement
… in the FP64 ideal, read positions through the view offset The written-but-unrun indexer harness reported a 0.37% NRMS metal-vs-ideal delta and a vacuous selection check. Both were reference-side: * the ideal skipped the pinned source's per-head einsum BF16 output and the score's BF16 storage, so it compared an unrounded equation against a quantized contract; * the selection check sliced scratch.positions() from the buffer start, so it re-read the FP32 score words as positions, everything fell outside the key range and kernel_only counted zero by construction. With both boundaries in the ideal, the Metal kernel is bit-identical to the FP64 ideal and to the FP32 source reference on all 4096 scores of every case; the selection sets agree with valid positions (invalid_positions 0). The remaining 0.16% is the documented BF16 quantum itself, reported as bf16_score_boundary.
…output quantum sparseHeadPre captures acc/denominator before the output BF16 rounding, so the FP32-versus-FP64 distance is measurable instead of hidden by the 2^-9 output grid. Across 16 window/compressed/sink/cancellation cases the stage's intrinsic level is 2.5e-8..3.2e-7 NRMS (sub-ULP on the pre-rounding value), and the Metal kernel stays bit-identical to the FP32 source contract on every cell — 0 of 8704 BF16 cells differ from the FP64 ideal. Receipts: receipts/sparse-stage.jsonl (metal/f32_reference BF16 deltas, f32_vs_ideal_pre_rounding ULPs, scale lines).
…d RoPE Each stage boundary now has its own adversarial case family, and every case reports the kernel against BOTH the FP64 ideal and the FP32 source-contract reference, so a kernel-only delta is distinguishable from a contract delta. * sinks: one live column pins the running maximum to a known score, so the sweep -104..90 walks FP32 exp() from underflow to overflow, plus per-head sink indexing. * tile denominators: equal-score rows for the pure sum of exponentials, ramped scores that push the online correction through the FP32 underflow limit, a descending set where no correction is ever applied, and a single-tile row. * output mix: window and compressed rows on independent scale grids, and a 128-row mirror shape that cancels out of the score and the accumulator alike. * RoPE: production shape (64 rotary columns, YaRN compressed spectrum), positions 1..16383, inverse rotations and multi-row prefills, comparing the kernel's resident factor table against both references. Also: sparseCaseFrom gains split window/compressed scale exponents, and each sparse case now prints metal_vs_f32_reference on the identical decoded rows. Receipts: receipts/sinks-stage.jsonl, tile-denominators-stage.jsonl, output-mix-stage.jsonl, rope-stage.jsonl. Both qualification suites green.
The stage-by-stage verdict is that every stage is bit-identical to the FP32
source contract, which leaves the possibility that what this operator contributes
to the whole-model gate is not arithmetic but a knife-edge: a pick that sits closer
to the boundary than the score's own rounding noise.
So the indexer case now also selects from the FP64 scores the ideal kept, and reports
how many of the 512 picks change, the boundary margin and that margin in score
quanta. Random score draws tie exactly at the boundary and flip 2 of 512; the tie
case, whose margin is 13 quanta, flips none. Selection is insensitive to arithmetic
noise except where it is a genuine tie.
Also records the per-stage verdict table in RESUME.md, and the sink sweep now walks
the FP32 exp() overflow edge explicitly (80/83/84/85/86) so that the one place the
kernel and the contract disagree is a named case rather than an outlier.
Receipts regenerated from a single run: receipts/raw/final-run.log plus
{pool,indexer,sparse,tiles,mix,rope,sinks}-stage.jsonl. Both suites green.
…-plan test calls math.zig's wide and b64 arrived twice through the two worktree lines; the shard-plan tests now pass the index-ownership argument the loader gained.
The reference state now probes every scored window: the contract's boundary margin (BF16-rounded FP32 scores), the FP64-ideal margin on the same inputs, and the top-k pick flips between them — the real-data support-flip measurement. The poolOf call takes the partials by address; the oracle path had never been compiled since the runner landed, which the probe's first build exposed.
… scratch, verbatim The margins mode prefills a long prompt, then runs a single diagnostic token with the observer hooking each indexer layer, copying the selection scratch (score tiles, heap, positions) out verbatim. The offline decode reads the score tile directly: at 2600 tokens the boundary of the 512-pick selection sits inside an exact tie of the BF16-quantized scores at every writable layer — the support-flip condition the synthetic harness predicted, now on real weights. The candidate-masked layers' tiles are unwritten; their boundaries need the candidate-region decode.
Model.open takes the lane count: every slot owns a full packed-cache State and its 16-byte Engram history slice, and forwardOn/resetSlot run one slot explicitly while slot 0 keeps the single-lane callers unchanged. Admission adds (lanes - 1) cache payloads. tf-dsv41-run lanes interleaves two slots' chunks against their solo runs and requires bit-identical logits, the state-level concurrent == solo check.
The truncated struct let the kernel's 296-byte write smash 152 bytes of stack, so every mode died with a wild jump right after its forward — the four crash reports show the gpu-seconds value landing in the call target. The full layout is the one process_memory's test pins. The two-slot lanes receipt now passes: interleaved chunks against solo runs, zero bit disagreements, zero execution disk reads.
Each verify window runs as its slot's own sub-forward, so a row's bits are its solo bits whatever shares the round; drafts off, greedy draws through a handle ring, and a kept prefix shorter than its window rolls back and re-feeds the kept rows. Facts report the workspace row budget as the exact width; trees, held drafts and grammar masks are refused until their pieces land.
Open builds the model with the requested lanes, the token map beside the checkpoint and the tables from --engram-table-dir, under the one-Mac load limit: 70% of RAM by default, GLM_LOAD_LIMIT_GB to opt past it, refused past the GPU's working set. Greedy only, drafts off, prompt caches and learning refused for now; the engine registers as deepseek_v41/exl3.
The V4.1 lane host reads the original release's Engram tables through the flag; the load refuses past the limit with the opt-in named. Served smoke on the lane backend: up in 115 s under GLM_LOAD_LIMIT_GB=430, a greedy chat completion answered READY with thinking and sane usage.
The builder moves into the family module behind tf.deepseek_v41_token_map.build; the CLI is a thin wrapper. When the checkpoint has no engram-token-map.bin but the original release's tokenizer.json and config.json sit beside the Engram tables, the lane host builds the map at open — the pull no longer needs a Python step. The tokenizer becomes a shared module so tokenizer.zig never lands in two modules of one compilation.
What one Mac needs, the token map the engine builds itself, the serve line with the lane semantics and the honest --no-drafts, the residency gate, and the numerical picture: the per-operator rounding-level tests, the sensitivity floor against the whole-model error at two quantizations, and the measured selection-boundary ties.
…ests The generated nfkd_data.zig now comes from tools/zig/nfkd_tables.py, a peer of tools/zig/unicode_tables.py: it rebuilds the table from unicodedata 16.0.0 and zig fmt, and its --check fails on drift. The table is one file with a generated header, and lean_check.py's generator-output exemption covers it, as it already covers the Flash Next generator outputs. The normalizer chain and the PCG64 draws keep their tests; the Hangul case expected a recomposed syllable, NFKD decomposes it and the release's chain never recomposes, so the expectation and a comment say so. The program was registered only as an installable executable, so its test blocks never compiled under zig build test, which is how a wrong Hangul expectation survived. The program table now takes host_tests, and the test step depends on the program's test artifact.
operator_csa2_fp64.zig (1331 lines) and operator_hc_fp64.zig (1095) sit above the 600-line file budget. Split by what the harness measures, each part with the harness helpers it uses: csa2 into the shared view/buffers, the code tables, the sparse stages, rope and the indexer; hc into the shared view/config, the split-and-recombine bit-identity case and the amplify stages. The reference root imports every part in dependency order, so the tests.deepseek_v41_reference. filter and the suite counts are unchanged. No tolerance, assertion or receipt text moved.
…e line each Folds the remaining comment groups into single lines of at most 120 columns, keeping the technical content. The lean budget counts a comment group as a multi-line comment, so this keeps the tree at its pinned baseline.
…ment `lanes` sat in the six-or-seven argument list, so the eight-argument form with CHUNK and CONTEXT was rejected before the parser could read it, and the usage text never listed the mode at all. Remove it from that list, document the mode's chunk and context arguments, and cover the accepted and rejected arities with a host test now that the runner registers its tests.
The reference modules take a comptime real width, so the FP32 source contract and the FP64 ideal are one equation set instantiated twice and cannot drift apart on their own. The new oracle-precision mode runs both on identical checkpoint bytes and tokens with the GPU out of the loop, keeps both logit vectors, and reports the NRMS between the widths, the per-token argmax, and the residency receipt of the same invocation. The window is capped at 16 IDs, one teacher step. No serving path, kernel or checkpoint format changes: this measures precision rather than gating it. Tests run: test-deepseek-v41-reference 29/29, test-deepseek-v41 79/79, test-deepseek-v41-weights 13/13, zig fmt clean.
…racle words stay frozen The negative-FP64 branch fell through the NaN check's `>` test with a positive NaN's sign stripped, so a positive FP64 NaN narrowed to the finite BF16 word 0x3ffc while a negative one produced 0x7fc0. Test the FP64 class instead and return the canonical quiet NaN of the input's sign. The new tests cover every IEEE class of the narrowing (both infinities, both zeros, FP64 subnormals, the largest-finite overflow, and six NaN payload classes), the BF16 subnormal and overflow tie grid against the FP32 branch, and frozen FP32 outputs of projection, the hyperconnection stages, attention pooling and the Engram gather. The frozen words were captured on 53b7900, before the repair; the repair touches only the FP64 narrowing, so those FP32 words must not move. Gates: `zig build test-deepseek-v41-reference` 43/43 steps, 32/32 tests; `lean_check.py` 501 problems against the same 501 on the parent commit; `zig fmt --check zig` clean.
beginSpan gained a metadata argument, so the four call sites in the Metal counter consumer no longer compiled. zig build test does not build that consumer, so nothing caught it. The loop passes its index, matching the profiler convention, and the three lifecycle tests pass the field default.
1.1.2 moved the Metal build wiring into zig/build/metal_targets.zig and split DeepSeek into dsv41_shared/cuda/state/draft. This branch's build.zig still carried the pre-1.1.2 monolithic file, so the tree did not configure. The staging, operator-check and program additions move into metal_targets.zig, and the token map reads the shared typed Config (text_config.*) rather than the older flat fields.
3c54062 to
3032baa
Compare
Piece (e) of the #690 split, stacked locally on piece (d) (#827): the FP32 and FP64 oracles share one equation set, and a new
oracle-precisionmode runs both widths on identical checkpoint bytes and the same tokens.The form of the sensitivity gate is the open question in #847/#827. The gate-deciding number is the whole-model NRMS between the FP32 source contract and the FP64 ideal on the same checkpoint with the GPU out of the loop: if it is small, FP32-level arithmetic is not where the residual gaps come from; if it is large, arithmetic is the bound. This mode makes that measurement reproducible in one invocation instead of argued from separate receipts.
What changed
math,projection,engram_math,reference,attention_math) take a comptime real width. The FP32 contract and the FP64 ideal are the same equations withReal = f32orf64, so the two paths cannot drift apart: every sum, reciprocal root and rounded store is one expression instantiated twice.tf-dsv41-run oracle-precision MODEL TABLES IDS_JSON OUT_DIRruns the teacher oracle at both widths over the same bytes and tokens (window capped at 16 IDs, one teacher step), keepsreference-fp32-logits.f32andreference-fp64-logits.f64, and reportsnormalized_rms_error(against the FP64 logits),normalized_rms_error_against_fp32,max_absolute_error,rms_error, per-token argmax, both wall times andevidence_origin.math.zig's FP64 narrowing now tests the FP64 class and returns the canonical quiet NaN of the input's sign. Before the repair the payload branch looked only atbits >> 49, so every positive FP64 NaN became a finite BF16 word -0x3ff8for the smallest signalling NaN (0x7ff0_0000_0000_0001),0x3ffcfor a quiet NaN - and negative NaNs kept non-canonical payloads (0xfffffor the all-payload NaN).53b790082, before the repair; the repair touches only the FP64 narrowing, so those FP32 words must not move.docs/deepseek-v41.md: the mode, the command and its window.Receipt
Environment: commit
964183651, the 2.9bpw pack (the repack the #827 receipts qualified), 8 teacher token IDs, one Apple M3 Ultra (80-core GPU, 512 GB unified memory), macOS 26.6.2, Zig 0.17.0. Pins:Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw@64ba41b6c916a587db06eae2e19b7845f7be6e6b, Engram tableslittlecedar/DeepSeek-V4.1-Flash-EXL3-2.9bpw-with-engram@48afcd1c4f2d73594d9d496b9f7bbb4559ddb9e8, sourcedeepseek-ai/DeepSeek-V4.1-Flash@2cba9e42aa026125f3ed06c6d98c1db82f7ca027,engram-token-map.binsha256b9a7cf9d341f5daf8fc435dbc36153bf4b28c1b8075be20de30d0556fb910511. Token IDs[0, 671, 6102, 294, 8760, 344, 11111, 16](sha25614da70af68c78cbf320dbcfac88e126563ebe7b8a212e7bb95f774b611ae0bf3). Run in one invocation:precision-fp32-vs-fp64.json:{ "tokens": 8, "max_absolute_error": 1.0826212214313418, "rms_error": 0.12073505094820904, "normalized_rms_error": 0.028957653279577746, "normalized_rms_error_against_fp32": 0.028965455800798824, "greedy_disagreements": 0, "fp32_seconds": 376.11573070834856, "fp64_seconds": 376.30811733356677, "evidence_origin": "same_checkpoint_same_equations_two_widths_no_gpu" }residency.json, same invocation (captured before the artifacts above were written):{ "mode": "oracle-precision", "evidence_origin": "current_invocation_not_historical_qualification", "context_capacity": 1048576, "consumed_tokens": 0, "weight_bytes": 413338642120, "resident_allocation_bytes": 415200591872, "ready_footprint_bytes": 415894048032, "final_footprint_bytes": 415907810616, "load_seconds": 125.27615625003818, "measurement_scope": "model_execution", "execution_disk_read_bytes": 0, "execution_pageins": 0, "resident_gate_passed": true, "execution_error": null }normalized_rms_error0.028957653279577746 (2.8958%): the whole-model FP32-versus-FP64 sensitivity floor on this pack.rms_error0.12073505094820904,max_absolute_error1.0826212214313418.5, 41079, 294, 270, 344, 11111, 16, 660— identical at every token.weight_bytes413338642120,execution_disk_read_bytes0,resident_gate_passedtrue,evidence_origincurrent_invocation_not_historical_qualification.These are the figures the family guide already states (2.90% FP32-vs-FP64, 4.10%/5.16% Metal-vs-FP32), now regenerable by one command rather than only recorded beside the PR.
Tests on
964183651:zig build test-deepseek-v41-reference43/43 steps and 32/32 tests (the compiled test binary printsAll 32 tests passed.); the fullzig build test test-goldensuite result is recorded in the follow-up section below;tools/zig/lean_check.pyreports 501 problems on this tree against 501 on the parent commit, 0 new;zig fmt --check zigclean. Metal (Apple Silicon) only; CUDA not run, and no serving path, kernel or checkpoint format changes, so decode and prompt speed are n/a by construction.Review follow-up (round 2)
The review asked for three things; this commit (
964183651) is the response.math.zig:30no longer narrows a positive FP64 NaN to a finite BF16 word. The old line testedmagnitude > 0x7ff0_0000_0000_0000and rebuilt the word frombits >> 49, so0x7ff0_0000_0000_0001became the finite0x3ff8,0x7ff8_0000_0000_0000became the finite0x3ffc, and the negative all-payload NaN became0xffffinstead of0xffc0. The new test table fixes all six NaN classes and also pins the smallest FP32 NaN viab(@bitCast(@as(u32, 0x7f800001))) == 0x7fc0.prepare, rotation and output narrowing), the hyperconnection folded/expanded and pooled boundaries, attention pooling and sparse-head selection, and the Engram gather. They assert exactu16words, so a change in FP32 arithmetic fails them rather than passing inside a tolerance.reference-fp32-logits.f32(4,136,960 bytes, sha2569698a8aaf3640869fc69fe539819845519cb1c0f94c72a5d5dedc1a94ced93ab),reference-fp64-logits.f64(8,273,920 bytes, sha256696de82fca773faa8329e4efca66e3caff4fc1d62c0d5d8ae596aee69176a037),precision-fp32-vs-fp64.json(sha2560f34cc96ee47781504ee10473b18369eb4b36f0e956720e2dabd023244576143),residency.json(sha2566cc7717fe5735f8bfa69223bda5c1188686163a3bc35c521af114c84d51095a3) and the run log (sha2562a8d8eb823ad8c1fd042689373faa9a0ec7e777ece2915d3dc8cdb98dd97f303). The two logit arrays are byte-identical to the arrays produced before the repair on53b790082(same two sha256 values), which is the same evidence as the frozen FP32 tests: the repair moved only the FP64 NaN path. The 8 tokens' argmax are5, 41079, 294, 270, 344, 11111, 16, 660at both widths, 0 greedy disagreements.Full
zig build test test-golden --summary all(cold cache outside the worktree):Build Summary: 311/311 steps succeeded; 4442/4457 tests passed (15 skipped), exit 0. The log'sfailed command:lines are expected negative-path output from tests that simulate backend failures; every build step succeeded.Checklist: every token still goes through the lane rounds and no serial path is added — this is an offline diagnostic mode, so the drafted/resumed/concurrent legs are unchanged from (d). No precision is traded: the mode measures precision, it does not gate it. No new or changed GPU kernel. New tests skip cleanly without their dependency (the reference harness runs weight-free); on the pre-repair sources the NaN and frozen-word tests fail, and in this PR's first revision the modules did not exist at all.