[akol1]

Native lane projections can preserve FP32 output until a caller combines residuals or tensor-rank partials. Extend the existing Lane.matmulOut API to all packed modes and retain matmulF32 as a compatibility helper. Keep the shape-driven split-K plan and scratch contract, with required fatbin instantiations and output reduction. Rebuild the fatbins with the runtime.

Depends on #907, retained as separately reviewable commits. The implementation preserves upstream's FP4 FP32-output path and adds BF16, tensor-scaled FP8, block-scaled FP8 and MXFP8 output paths. Shared numerical contracts cover non-finite inputs, repeats and buffer guards, served/tile-edge shapes, and mirrored integer stress against independent FP64 products, separately for clustered and explicit reductions. All 28 added FP32-output launches have compiler-derived resource records.

Review follow-up:

The fixture correction from 68bbde58d9b18d7a3b8f1e6e5ea1a40160932cae is preserved with its original author. Each F32 and BF16 dispatch now poisons its destination and reduction scratch, retaining the strided-input, FP64, row/reduction-invariance and direct BF16-rounding checks.

The shared Rig.close fix and CPU regressions from #933 are preserved through the #907 dependency. An additional CPU control calls this fixture's actual poison-and-dispatch path with no-op CUDA launches after a valid stale prefix. It checks both output formats, every fixture row count and the fused/cluster/explicit-reduction choices in Debug and ReleaseFast. It requires fresh poison in both buffers before launch and requires the real comparison to reject the unwritten result.

The previously reported resource discrepancy is already corrected by #945, inherited here; no audit tolerance is relaxed. The source update changes tests only. At 4544c5c5da7c5a9b6ffa0cfa46275e32cb011397 against main b090e3b19014bfa1eb2b7f38ea3cc4d39abec9be, the macOS host/golden/contract suites pass 222/222 build steps with 3,025 tests passed and ten platform/library skips; golden results are 309/319 equal with zero unexpected differences. The current No GPU workflow commands, including the reader fuzz target, pass. CUDA-enabled Linux cross compilation passes 61/61 steps using explicitly fake compile-only image placeholders; those outputs were never executed or deployed. Whole-tree format and whitespace checks pass; lean has 517 inherited findings and zero new findings. A device rerun of the strengthened fixture remains pending; previous device results are bound to the earlier head and are not qualification of this changed fixture. No model/TP parity or performance result is claimed for this update.

The separate native macOS build, zig build native -Dcpu=apple_m1 -Dversion=1.0.0, also passes 45/45 build steps. The resulting executable was built only, not launched.

Prior device evidence, explicitly historical:

At 787064fe1b962221c29dd1b738042ba6ba4e5627, the exact-source Linux build passed 104/104 steps with real NVCC 13.3.73 fatbins for SM 80/86/89/120/121. The complete five-architecture limits audit exited 0; actual SM 120 execution on RTX PRO 6000 Blackwell reported:

SUMMARY 22 kernels: 22 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 143572 ms, limits 17 ms

Those kernels, runtime sources and launch records are byte-identical in this update, but the changed fixture has not been rerun on a device. SM 80/86/89/121 hardware remains unrun. The macOS host workflow includes its existing Metal tests; no Metal equivalent of these CUDA fixture results or full-model qualification is claimed. No full-model, collective or performance qualification is claimed. Current-head GitHub CI is green: desktop, mobile, Zig format, and No GPU host/contract/golden/fuzz/lean checks (verified 2026-10-16 21:37 UTC).

[ashhart]

Thank you, @akol1, for exposing FP32 projection results through the existing kernel template while keeping the BF16 entry point intact.

We checked f6a0055 on #907. Lane.matmul still selects the original BF16 kernels and reducer; matmulF32 selects the new output instances with the same accumulation and split plan. The shared API fits #847's residual and rank-combination work.

We'll rebuild the fatbin and run tf-cuda-test projection-f32 across all five formats, strided rows and the fused/cluster/explicit reduction paths, checking FP64 error and rounding back to the existing BF16 result. At family integration, we'll compare the actual residual and collective calls against NVIDIA's default math, preserving each rounding point and providing separate scratch for overlapping streams.

We'll handle small integration fixes at landing and keep the existing-format and full-token lane checks alongside the new FP32 cases.

Posted by Ash's AI Assistant Synthena

[cwschroeder]

I found a gap in projection_f32_tests.zig: the output buffers are poisoned only once, then the first rows=256 fused dispatch fills every row. Later dispatches can reuse that prefix and pass the FP64, invariance and rounding checks even if they write nothing. Please poison the destination and scratch before each F32 and BF16 dispatch.

The fixture-only fix is 68bbde58, which can be cherry-picked onto this head.

I also prepared an integration on current main, head 0aa7fc93, base 46a08c18, retaining both contributor commits. Landing now needs the upstream Out/matmulOut API preserved and duplicate FP4/F32 kernel and F32 reducer instantiations removed.

Validation of 0aa7fc93:

• Zig 0.17.0: all four No GPU workflow build commands, whole-tree formatting and lean comparison pass. Lean has 0 new findings; frozen answers have 0 unexpected differences.
• Linux/aarch64 runtime and test executable cross-build passes without embedded fatbins.
• Separate CPU-only NVCC 13.2.78 build of the integrated qmmf.cu passes for SM120 and SM121. cuobjdump shows all 70 kernel and 2 reducer names expected by the loader in each image. Fatbin SHA256: 4543bf83e7df85f7111463acbfef8e48a2009fe206019c2fc95be76b59a29fdd.

These are compilation and host checks. GPU execution of this integration, model/TP parity and performance remain untested. The projection fixture exercises SK=2; SK=4/8 coverage remains separate.

[cwschroeder]

Reviewed 37578da9 against current main 46a08c18. The merge now preserves the upstream output API and avoids duplicate CUDA definitions. The new numerical contracts add useful SK=4/8 and served-shape coverage.

The original fixture issue remains in projection_f32_tests.zig:83–85,110–122: after the full 256-row result, later dispatches can reuse its prefix if they write nothing. The new contract adapters use contiguous inputs; this older gate still supplies the strided-input and direct BF16-rounding checks. Please retain those checks and poison destination and scratch before every dispatch. The fixture-only fix still applies to that unchanged file.

I also compiled this head's qmmf.cu with NVCC 13.2.78 for SM120 and SM121. Both images contain exactly the 70 kernel and 2 reducer symbols expected by the loader. Fatbin SHA256: 6c3637784d11732b519a5c87ec6b1ac80a604bbe3b3f89403806c3d1c364c253. This was a bounded CPU-only compile on GB10, with no GPU execution. The four existing current-head CI checks are green.

[cwschroeder]

A separate issue in this fixture's shared checker is now addressed by #933: Rig.close can accept [2, NaN, 0] against zero/unit tolerances because the NaN erases the previous maximum and the final finite value replaces it.

The CPU regression runs the actual comparison with only synchronization/readback replaced. This complements the per-dispatch poison fix: fresh poison makes unwritten values visible, and #933 prevents a later finite value from hiding their NaNs. Local whole-workflow checks and source review pass; remote CI is running. This is a test-checker defect, with no device arithmetic failure established.

[cwschroeder]

Checked the update through 5bb739249784564c9b8ec591c94b8c4aa8fbd5ad. Relative to 37578da9, only the 36 resource registrations change. zig build test-kernel-contract passes all 18 CPU tests.

I also compared the actual resource records against our earlier independently compiled QMMF fatbin. The kernel source and header are unchanged; the artifact was built with NVCC 13.2.78 for SM120 and SM121. This uses the repository's contract.cubin parser, including its reserved-shared-memory adjustment, and opens no CUDA context.

Scope	Result
All 36 new entries, both architectures	72/72 usage comparisons match
Existing FP4 F32 32-row cluster entry	Table says 63 registers; compiled image says 61 on both architectures
Whole QMMF registry	88 comparisons, 2 mismatches, so the overall audit fails

The CPU reproducer, build command and output pin the source and fatbin hashes. Could you check that older entry against the compiler used for the landing build? It is a compiler-build-specific metadata discrepancy; both counts remain within the launch limit.

This does not establish a device arithmetic fault. The audit covers registers, static shared memory, launch bounds and local-memory metadata for these two images. It does not validate the other architecture columns, dynamic shared-memory sizes, numerical outputs or performance. Astra checked the interpretation and scope. The numerical fixture is unchanged by these commits, so the earlier fixture findings remain separate.

[akol1]

The fixture correction from 68bbde58d9b18d7a3b8f1e6e5ea1a40160932cae is preserved with its original author. Each F32 and BF16 dispatch now poisons its destination and reduction scratch, retaining the strided-input, FP64, row/reduction-invariance and direct BF16-rounding checks.

The shared Rig.close fix and CPU regressions from #933 are preserved through the #907 dependency. An additional CPU control calls this fixture's actual poison-and-dispatch path with no-op CUDA launches after a valid stale prefix. It checks both output formats, every fixture row count and the fused/cluster/explicit-reduction choices in Debug and ReleaseFast. It requires fresh poison in both buffers before launch and requires the real comparison to reject the unwritten result.

The previously reported resource discrepancy is already corrected by #945, inherited here; no audit tolerance is relaxed. The source update changes tests only. At 4544c5c5da7c5a9b6ffa0cfa46275e32cb011397 against main b090e3b19014bfa1eb2b7f38ea3cc4d39abec9be, the macOS host/golden/contract suites pass 222/222 build steps with 3,025 tests passed and ten platform/library skips; golden results are 309/319 equal with zero unexpected differences. The current No GPU workflow commands, including the reader fuzz target, pass. CUDA-enabled Linux cross compilation passes 61/61 steps using explicitly fake compile-only image placeholders; those outputs were never executed or deployed. Whole-tree format and whitespace checks pass; lean has 517 inherited findings and zero new findings. A device rerun of the strengthened fixture remains pending; previous device results are bound to the earlier head and are not qualification of this changed fixture. No model/TP parity or performance result is claimed for this update.

The separate native macOS build, zig build native -Dcpu=apple_m1 -Dversion=1.0.0, also passes 45/45 build steps. The resulting executable was built only, not launched.

[cwschroeder]

Scoped adoption recheck of 4544c5c5: my poison/checker findings are closed in source and CPU tests. Astra found no further material issue in this delta.

zig build test-cuda-reference test-cuda-projection-fixture --summary all passes 11/11 steps and 10/10 test executions with Zig 0.17.0 on macOS arm64. These are the configured Debug/Fast fixture and checker roots; their imported CUDA/core modules remain Debug. The host controls use virtual allocation/readback/launch callbacks and open no device.

In separate temporary mutations, removing either destination poison or scratch poison makes both projection-fixture runs fail (expected 2, found 1). Removing the early finite-error guard makes the three NaN regressions fail in each reference root because Rig.close returns success instead of TestFailed. The unchanged tracked worktree stays clean.

Both real F32/BF16 dispatch sites use the helper. The GPU fixture retains strided input, five weight formats, FP64 references, direct BF16 rounding and row/reduction invariance. The CPU no-op control itself uses FP4 weights, both output formats and all configured row/path choices; it does not numerically qualify the five formats. Whole-tree format, whitespace and lean pass (517/517 against current main 6ee828c2, zero new); all four current-head CI checks are green.

The current-head device rerun remains open, as your description records. This adds no CUDA arithmetic, model/TP parity or performance qualification.

[ashhart]

Thank you, @akol1, for FP32 projection outputs and the no-op control that catches stale-output false passes.

We merged your work for 1.1.0 in f0f7503, after #907. Destination and reduction scratch are poisoned before each F32/BF16 dispatch, and non-finite error cannot erase a failed comparison.

The existing output API and unique launch registrations are retained, with compiler-specific resource records.

NVIDIA GB10/RTX PRO 6000 projection checks passed; nvcc 13.0/13.2 release limits had zero stale records.

Posted by Ash's AI Assistant Synthena