test: record A once, replay B against it, as one rank or several - #20
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c1c58a681
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fn bytes(&self, from: &Scratch, len: usize) -> Result<Vec<u8>> { | ||
| Ok(self.rt.read_scratch(from, len)?) | ||
| Ok(self.rt(0).read_scratch(from, len)?) |
There was a problem hiding this comment.
For any multi-rank run, snapshots allocated with rt(rank).scratch(...) on ranks above zero are still read through rank 0's runtime here. Runtime::read_scratch binds that runtime's CUDA context and uses its stream, so default fuzzing will attempt to read rank 1+ device pointers from GPU 0; the final-state comparison does the same even when fuzzing is disabled. Pass the rank through Side::bytes and dispatch to rt(rank).
Useful? React with 👍 / 👎.
| } else { | ||
| findings.push(Finding::of(&run.program, &label, n, &c)); | ||
| } | ||
| res.entry(run.program.clone()).or_default().entry(n.clone()).or_default().push((label.clone(), c)); |
There was a problem hiding this comment.
In a multi-rank run where A's nondeterminism differs by rank, this stores every rank's noise under the same program/buffer key (and the state path similarly omits the rank). The later within_noise calculation compares global maxima and membership using those collapsed keys, so noise observed only on rank 0 can authorize an unrelated B mismatch on rank 1 and produce a false PASS when fuzz/logit evidence does not take precedence. Include the rank in both noise and local matching keys.
Useful? React with 👍 / 👎.
Real run: DSv4.1 Flash EP4, paged → fused, 4 ranks on one GB300 tray (2026-09-15)A =
Fixed on the way, each with a fixture: once-program outputs are load-time constants, not span inputs (200 GB of rope-table scratch otherwise); a buffer declared differently on the two sides is not compared; a changed Follow-ups in Archive: |
|
17f9604 — the logits oracle is KL per row. Recomputed from the tray18 run's archived rows (
Under the new rule that run reads: FAIL at 🤖 Generated with Claude Code |
|
104a3c1 — the FAIL on a flip beyond the KL limit is now guarded by A's own end-to-end band (noise runs A's workload twice: KL within the limit, no flips), not by the span-level noise floor. A reference that is noisy at some span but reproduces its distribution can be failed; without a noise pass the guard stays open as before. 23 harness fixtures, CI green. |
比较下设备 + KL 判据：DSv4.1 Flash EP4 复跑（tray18，2026-09-15）
同一对 manifest、同一 seed、同一 tray 复跑（
计数和 perf 逐项与首跑一致：local 0/2312、noise 492/984 clean、fuzz 1045/5904、decode_batch TPOT 11.41 → 10.57 ms（−7.4%）、round 12.97 → 11.93、prefill 22.5 → 19.0 ms、120 spans −46.7%。 判定变了：FAIL。noise 底线 "A against itself end to end: 4036 rows · max KL 0.00e0 · 0 argmax flips"，所以翻转全是 B 的：3960/4036 argmax 一致，76 次翻转全部超过 还剩的时间全在 record：323 s 里 workload 只有 1.3 s，怀疑对象是 84 GB 的 state 镜像（按整块分配尺寸拷）、每个输入一次 🤖 Generated with Claude Code |
A and B were loaded side by side and run in lockstep, so a model that
fills its GPUs could not be tested at all: DSv4.1 Flash EP4 puts 119 GB
of weights on each rank. The harness now records A first (every program
run's state image, every span's frontier inputs and reference outputs,
kept on the device; A's noise floor, A's outputs under fuzz, A's
timings), drops A, loads B and replays: B starts every run from A's
image and every span from A's inputs, then free-runs the workload for
the end-to-end logits. The recording's device handles (`Side::Buf`)
outlive the side that made them; the two loads share the primary
context.
A side may be several ranks. `Side` names the rank on everything that
moves bytes and runs everything on all ranks at once (a collective in a
span needs its peers issuing); recording and comparison are rank-local,
findings say `rank q`. In kern-run the side is `Ranks`: one runtime per
GPU with its place in the topology, `{ep}` weight shards, peers
exported and imported, `once` programs run, a thread per rank for
run/time/capture, the slowest rank's time, a hung rank named after
600 s. `--gpu` takes a list, or one ordinal to start the ranks at.
In the static diff a `peer` buffer stands for the exported buffer or
state it holds the group's addresses of: the kernel given every rank's
copy reads and writes the target on this rank too, so the target is on
the span's frontier; the address array is a load-time constant.
The verdict is unchanged. The fake side gained ranks (`Fixture::ranks`,
a `scale_rank1_wrong` entry) and a peer fixture for the diff; the same
seed on the single-rank fixtures reports the same counts as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…nput A span's frontier inputs excluded weights but not what the `once` programs make of them: packed expert weights, rope tables. On DSv4.1 Flash every attention span reads the 268 MB `rope.window.interleaved` carry, so recording 120 spans over 9 runs on 4 ranks asked for 200 GB of device scratch per rank and died out of memory right after loading A. These buffers are each side's own, like the weights they come from (B may pack them in another layout), so `constants(m, once)` names them and the recording hands over neither. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
CI's hawk lint: seven items were pub(crate) with no use outside the module. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
A driver error out of a four-rank recording said only CUDA_ERROR_INVALID_VALUE. Every byte-moving call of the real side now adds which rank and which buffer or state, and the recording adds which span it was saving or reading. A frontier input with no live bytes at the run's vars is skipped rather than allocated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
A scratch allocated on rank 2 read through rank 0's runtime is a pointer from another device: CUDA_ERROR_INVALID_VALUE, first seen when fuzz took A's tapped inputs on a four-rank recording. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
DSv4.1's fused decode keeps the name `attention_raw` but stores it as fp8 where the paged kernel wrote bf16: same name, half the bytes, no common layout to compare. Such a buffer is neither handed from A to B as a span input nor compared as its output; it is listed with the one-sided writes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
DSv4.1 writes `decode.target_head.logits`; only a name starting with `logits` was an oracle, so the first four-rank run fell back to span identity with next_token bit-identical on every rank and no verdict. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…d program DSv4.1's B packs its attention weights in `load`, so `load` has spans; it ran at load on both sides and is nothing the workload could drive. The first four-rank run ended INCONCLUSIVE on it alone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…rage The logits limit was counted in ulps of the row's storage dtype. On DSv4.1 the logits happen to be f32 while the pipeline is fp8/bf16, so a 2.6 move read as 1.7e7 ulp and `--logit-ulp 4` decided nothing; the storage of the head's output is not the precision the model ran at. Now every row is measured on the distribution: KL(A‖B) in nats against `--logit-kl` (default 0.01), whatever dtype the row is kept in. A flip within the limit is a tie that broke the other way (0.55/0.45 swapping already costs 5e-3); a flip beyond it is the FAIL, NaN on B's side included. The near-tie rule "A's margin ≤ B's Δ" is gone: it explained B's flips by B's own error, so the step-5 verify flip on tray18 with margin 2.69 passed as a tie under a Δ of 14.5. The report follows A's top-20 through B (overlap, where A's token ranks in B) as a diagnostic, not a gate: with a margin rule it is a subset of KL, without one the tail's ties drown it. Noise runs A's workload once more and reports A against itself end to end, the band any KL verdict sits in. Recomputed on the tray18 rows: pre-divergence 0.09–0.33, step-5 verify 4–12. Rows after a diverged draft are not comparable under any metric; teacher forcing of integer decisions is in the roadmap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…end to end The FAIL was guarded by the span-level noise floor: any span A could not repeat blocked the conclusion. A reference noisy at `o_lowrank` may still land on the same distribution every run, and that, not the spans, is what says whether B's flip is B's. The guard is now A's own end-to-end band from the noise pass (KL within the limit, no flips); without a noise pass it stays open as before. The INCONCLUSIVE text names A's band when it is the reason. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
Three harness primitives over two byte ranges on the device — `At::{
Buffer, State, Scratch}` — each one launch and a few bytes back:
`compare` (element-wise counts as kern-test's `compare` defines them),
`changed` (one bit per 64-byte block that differs, so a state's
write-set is found against an image of it without reading either), and
`logits` (one block per row: argmax, A's top two, KL, where A's argmax
ranks in B, the overlap of the top sets). The kernels are `compare.cu`,
bundled as PTX like `profile.ptx` (CUDA 13.0, compute_80) and loaded on
first use; they reproduce the host definitions exactly for counts and
maxima (dtype decoding, ulp keys and `total_cmp` ordering included) and
to rounding for the sums. `scratch` no longer zeroes: a harness fills
it before it reads it. `read_scratch` takes a range.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
`Side` gains `compare`, `changed` and `logits`, and the harness compares where the bytes are: A's reference outputs, its logits rows and its outputs under fuzz stay in device scratch; B's tap, the noise floor, the fuzz rounds and the end-to-end rows are kernel launches that return counts. A span's state write-set is the 64-byte blocks that changed against a device image of the state (`changed_blocks` is the definition), so the two whole-state readbacks and the byte loop per kept span are gone — on DSv4.1 EP4 that was some 190 GB through pageable host memory per record. The bytes that still come to the host are the write-set itself, the inputs fuzz perturbs, integer outputs with a domain and the end-to-end output buffers. The fake side implements the three primitives with the host definitions; `kern-run/tests/device_compare.rs` checks the kernels against them on a GPU (ignored in CI). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
Same manifests, seed and tray as the first run; every count and perf number repeats, tap 54 s → 0.9 s and fuzz 288 s → 7 s once nothing is compared on the host. Verdict is now FAIL under logit_kl 0.01: A reproduces itself end to end (max KL 0, no flips) and B moves prefill chunk 0 by KL 0.127. Record still takes 323 s with a 1.3 s workload; that cost is not yet attributed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…d outputs Fuzz perturbed every kept span's float inputs under six distributions and replayed both sides. It answered a question nobody asked: with no float domains in the manifest (a decision), a NaN under x4 scale is a report line, not a verdict; shuffle and resample break the tensors' semantics (q no longer matches its KV), so the differences say nothing; jitter repeats what the noise floor already says. Every bug an op swap produces has another stage on it (layout and edge rows: tap; shapes: the perf sweep; determinism: noise; time: the multi-step workload). In two real cases and dozens of fixture runs it never changed a verdict, and it cost five of the seven minutes of a DSv4.1 EP4 kern test: A's side decoded, perturbed and re-encoded 21 G bf16 elements on one CPU thread (measured 16 ns each; the state images and cuMemAllocAsync, suspected before, are milliseconds). Out-of-distribution coverage is another --seed of the workload: end to end, KL-judged, every row explicable. What stays is fuzz's post-condition: an output B writes end to end must lie in its declared domain, or the verdict is FAIL. Side::write and Options.fuzz go with it; the kern-test suite is 22 fixtures. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
…econds Same manifests, seed and tray; verdict, counts and perf unchanged. The state-image roadmap item goes: master's pool sizes states by the capacity's tokens, so the images are the live 12 MB per rank per run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
record 的 323 s 归因，fuzz 删除，第三跑 74.5 s上一条评论里"84 GB 镜像 / 两万次 cuMemAlloc"的怀疑不成立，实测排除：
同 tray 第三跑（
判定、计数、perf 逐项与前两跑相同（FAIL：prefill chunk 0 A 271 → B 201，KL 0.127）。state 镜像 84 GB → 339.5 MB 是 master 的 pool 改按 capacity 分配 state 的结果（rebase 带进来的），roadmap 里那一项已删。分支已 rebase 到 master，CI 绿，仍不合。 🤖 Generated with Claude Code |
…tion first Review findings, each confirmed on the fixtures: - 0/0 compared was PASS: bit-identical, ahead of the logits oracle. A B that declares every compared buffer differently passed no matter what it computed. Nothing compared is not everything identical; the oracle decides (new fixture). - logits buffers, end-to-end outputs and shared states paired A with B by name alone while spans used dtype and shape; a B that re-declares its logits or repacks a state read as garbage or an opaque range error from a rank thread. One `alike` now decides all of them; states must declare the same, or they join the same report line. - a frontier input the two sides declare differently was dropped without a word, so B's span read its own stale copy and the span was no longer span-local. It is now named on the report line with the writes that are not compared. - B's tap ran spans from the run image only; docs/test.md promised A's pre-image before and A's post-image after on both sides. A state two spans of one run touch made B's second span read its own first. The tap now restores like the noise floor does. - the FAIL names the flip with the largest KL, as the report line does. - logit_rows declared 49,664 B of static shared memory, over the 48 KiB limit ptxas enforces for every arch (the resident driver's JIT let it through). 256 threads per block, 12 KiB; the PTX is regenerated and assembles for sm_80 and sm_100. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
Review cleanups. `Side::alloc` existed to feed `save` / `save_state` at every one of its five call sites; the two now return the device handle (`Runtime::save_buffer` / `save_state` likewise, `scratch` is private). `capture` was always followed by `time_captured`: one `time_graph`. A's noise pass and B's free run were the same loop: one `free_run`. `logit_row`, the `Bytes` alias, the unreachable E8M0 arm of the kernel's `ulp_key`, `Logits.runs` (printed nowhere) and the `ranges_of` re-export go. kern-run's `cmp_of` / `logit_of` are the one place the device counts become kern-test's types, and the GPU oracle test now goes through them instead of a copy. The GPU test also covers what the kernel is most likely to get wrong: one column, a row of one value everywhere on both sides (every tie-break by index), a NaN in A, a signed zero, and `changed` over an operand three bytes off the 16-byte alignment. 3/3 on GB300 with the 256-thread `logit_rows`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi
审查修复（
|
1 participant
What
kern testno longer loads A and B side by side. It records A (state image per program run, frontier inputs and reference outputs per span, all kept on the device; A's noise floor, fuzz outputs and timings), drops A, loads B and replays: B starts every run from A's image and every span from A's inputs, then free-runs the workload for the end-to-end logits. The two sides are never co-resident, so a model that fills its GPUs (DSv4.1 Flash EP4, 119 GB of weights per rank) can be tested at the price of loading twice.A side may be several ranks.
Sidenames the rank on everything that moves bytes and runs everything on all ranks at once (a collective inside a span needs its peers issuing). Recording and comparison are rank-local; findings sayrank q. In kern-run the side isRanks: one runtime per GPU with its place in the topology,{ep}weight shards, peers exported/imported,onceprograms run, one thread per rank for run/time/capture, the slowest rank's time reported, a hung rank named after 600 s.--gputakes a list, or one ordinal to start the ranks at.In the static diff a
peerbuffer stands for the exported buffer or state it holds the group's addresses of (the kernel given every rank's copy reads and writes this rank's target too), so the target is on the span's frontier and the address array is not.The end-to-end oracle is KL, not ulps (commit 17f9604)
The logits limit used to be counted in ulps of the row's storage dtype; DSv4.1 keeps its logits in f32 over an fp8/bf16 pipeline, so the real run read
1.7e7 ulpagainst a limit of 4 and decided nothing. Every row is now measured on the distribution: KL(A‖B) in nats against--logit-kl(default 0.01,kern.toml[test] logit_kl). A flip within the limit is a tie that broke the other way; a flip beyond it is the FAIL (NaN on B's side included). The old near-tie rule (A's margin ≤ B's Δ) is gone: it excused flips by the size of B's own error. The report follows A's top-20 through B (overlap, A's token's rank in B) as a diagnostic, not a gate, andnoiseruns A's workload once more to report A against itself end to end.--logit-ulpis deleted.Tests
cargo test -p kern-test: 22 harness fixtures (record→replay, a two-rank fixture whose findings all sayrank 1, a rank-count mismatch, declared-differently / once / logits-name rules; the KL verdict on rounding, drift, wide flip and NaN), peer-buffer and once-constant diff tests, compare/workload property tests (logit row: KL, top overlap, rank in B).cargo clippy --all-targets -D warnings,cargo fmtclean.Docs
docs/test.md: the design section now describes record → replay and a new 多 rank chapter;docs/roadmap.mdupdated.Not to be merged yet (per request).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Y5ggq6AWjBmMxNdeAgxqVi