Skip to content

DeepSeek V4.1: the precision and Engram reference work, combined on the 1.1.2 family line (#1052 + #1051) - #3

Open
akol1 wants to merge 7 commits into
pr-dsv41-bc-family-11from
pr-dsv41-ef-combined-11
Open

akol1 wants to merge 7 commits into
pr-dsv41-bc-family-11from
pr-dsv41-ef-combined-11

Conversation

@akol1

@akol1 akol1 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

What this is

Review artifact for carrying ashhart#1052 and ashhart#1051 together on
the 1.1.2 family line: both changes kept, both reference import sets retained.

  • Base: pr-dsv41-bc-family-11 @ 4f69fea10 (= d1018b318 plus 12 family commits)
  • Head: pr-dsv41-ef-combined-11 @ b800c7205
  • 5 commits, original authorship preserved, plus one carry-adaptation commit
  • 10 files, +2,119 / −853

Commits

  1. 08203429d the FP32 and FP64 oracles run side by side (DeepSeek V4.1: the FP32 and FP64 oracles run side by side ashhart/TensorFold#1052)
  2. b25c4a8ce the FP64 narrowing keeps NaN, and the FP32 oracle words stay frozen (DeepSeek V4.1: the FP32 and FP64 oracles run side by side ashhart/TensorFold#1052)
  3. dd8ce6d72 fp64 tests for the Engram hash, gather and gate (DeepSeek V4.1: fp64 reference tests for the Engram hash, gather and gate ashhart/TensorFold#1051)
  4. 9b440f904 hold the Engram gate to the registered kernel contract (DeepSeek V4.1: fp64 reference tests for the Engram hash, gather and gate ashhart/TensorFold#1051)
  5. b800c7205 the Engram export names the relocated module (carry adaptation)

Adaptations required by the 1.1.2 layout

Verification

check result
zig build test-deepseek-v41-reference --summary all 43/43 steps, 23/23 tests
zig fmt --check zig clean (exit 0)
python3 tools/zig/lean_check.py 444 problems, equal to the main baseline (0 new)
600-line file cap max touched file 563 lines (math.zig); others 535, 501, 332 or less
zig build test --summary all rerun: 360/360 steps succeeded, exit 0 (605/606 tests, 1 skipped; cached steps are not re-counted — the first full pass ran 5,054 tests and is described below)
zig build test-golden, test-kernel-contract, fuzz-cortheon green in PR #3's zig CI job (run 39318219955), with the served-Flash-Next regressions and the lean check also green
zig-windows fails before reaching these changes: zig/src/core/digest_key.zig:34 and zig/src/server/speech.zig:279 are unchanged 1.1.2 base files, the same two errors every PR based on current main hits; upstream ashhart#1034 carries the fix

desktop, mobile and fmt also pass on the PR. Full-suite note: the first zig build test attempt reported a failure in the server's
cortheon fuzz: every reader takes linear time on long inputs assertion (2 ms on 16 KiB versus
65 ms on 128 KiB) while the build itself saturated every core with parallel compilation. That
test compares wall-clock scaling and is unrelated to these changes. Re-running the same
compiled test binary by itself passed all 506 tests, including the fuzz test, and the rerun of
the full plan above succeeded.

ENG64 receipts from the reference run (4 families; the assertion is the declared contract
bound, which holds in every case):

  • hash masked-chunk-and-history-carry: 480 cells, 0 Metal-vs-contract differences
  • gather fp8-row-grid: 24,576 cells, 0 differences; fp8-interleaved-264-byte-rows:
    24,576 cells, 0 differences
  • gather vs FP64 ideal: 281/24,576 BF16 differences, NRMS 9.124456e-73, max |Δ| 2.633107e-36
  • gate unit: 2/40,960 differences, NRMS 2.8335300764278995e-8, max |Δ| 3.814697265625e-6,
    worst bound ratio 0.9919194591194878
  • gate wide: 1/40,960, NRMS 1.120183e-7, max |Δ| 2.441406e-4, worst bound ratio
    0.9923596285716317
  • gate narrow: 2/40,960, NRMS 1.147014e-7, max |Δ| 1.192093e-7, worst bound ratio
    0.9911134443853961
  • gate cancellation: 0/40,960 differences, 0 bound failures
  • gate all families: 5/163,840 Metal-vs-contract differences, 0 bound failures, worst ratio
    0.9935598582076476
  • gate dead mask: 20,480 cells, 0 differences

Bit-level Metal-versus-contract differences are reported per cell as the test has always done;
none exceeds the registered envelope.

Still open

akol1 and others added 5 commits October 19, 2026 13:09
The reference modules take a comptime real width, so the FP32 source
contract and the FP64 ideal are one equation set instantiated twice and
cannot drift apart on their own. The new oracle-precision mode runs both
on identical checkpoint bytes and tokens with the GPU out of the loop,
keeps both logit vectors, and reports the NRMS between the widths, the
per-token argmax, and the residency receipt of the same invocation.

The window is capped at 16 IDs, one teacher step. No serving path,
kernel or checkpoint format changes: this measures precision rather
than gating it.

Tests run: test-deepseek-v41-reference 29/29, test-deepseek-v41 79/79,
test-deepseek-v41-weights 13/13, zig fmt clean.

(cherry picked from commit 53b7900)
…racle words stay frozen

The negative-FP64 branch fell through the NaN check's `>` test with a positive
NaN's sign stripped, so a positive FP64 NaN narrowed to the finite BF16 word
0x3ffc while a negative one produced 0x7fc0. Test the FP64 class instead and
return the canonical quiet NaN of the input's sign.

The new tests cover every IEEE class of the narrowing (both infinities, both
zeros, FP64 subnormals, the largest-finite overflow, and six NaN payload
classes), the BF16 subnormal and overflow tie grid against the FP32 branch,
and frozen FP32 outputs of projection, the hyperconnection stages, attention
pooling and the Engram gather. The frozen words were captured on 53b7900,
before the repair; the repair touches only the FP64 narrowing, so those FP32
words must not move.

Gates: `zig build test-deepseek-v41-reference` 43/43 steps, 32/32 tests;
`lean_check.py` 501 problems against the same 501 on the parent commit;
`zig fmt --check zig` clean.

(cherry picked from commit 9641836)
The split in 7665649 left the Engram stage with its behavioral Metal tests
and its FP32 equations but no three-way harness. This drives the real hash,
gather and gate kernels against the FP32 source contract and an FP64 widening
of the same equations.

The family module is private, so tensorfold.zig exports it for the reference
tree; the tables, scales and family API are unchanged.

(cherry picked from commit 87aeaac)
…l contract

The gate test now carries the contract.Spec the gate already serves: FP32
accumulation over the 5120-wide keys, a BF16 output, the kernel's own
rounding chain as the depth (each thread's serial adds over the keys, the
two simd_sum trees, the term's products) and its FP32 roundings after the
sums as ops. Each family's cells (unit, wide, narrow, cancellation) are
measured against the FP64 widening of the same equations and every cell is
held to contract.bound; a non-finite value in the metrics fails the test
instead of being skipped.

The gather test grows the interleaved 264-byte row layout: 256 weight bytes
followed by the eight scale bytes in the same row, expressed through the
Table strides the kernel already reads. Both layouts match the encoded FP32
contract on all 24,576 cells.

tensorfold.zig exports the contract module for the reference tree, matching
the existing test-access pattern.

(cherry picked from commit 075940d)
The 1.1.2 family line keeps the resident Engram at
families/deepseek_v41/metal/engram.zig, while ashhart#1051's export assumed the
old line's families/deepseek_v41/engram.zig. The module body is unchanged
apart from its relative hash import, so the operator test still exercises
the same arithmetic.
… name

Exporting core/contract/contract.zig from tensorfold.zig made that file reachable through two module roots, which Zig refuses: tf-kernel-contract and test-deepseek-v41-reference both failed to compile with "file exists in modules core and contract". The reference check now imports the contract module the way zig/tests/contract/metal_main.zig does, and the engine root keeps no contract import, as upstream main has none.
@akol1

akol1 commented Oct 20, 2026

Copy link
Copy Markdown
Owner Author

Found and fixed a build break in this branch: tf-kernel-contract and test-deepseek-v41-reference did not compile.

Two steps failed with the same message:

zig/src/core/contract/contract.zig:1:1: error: file exists in modules 'core' and 'contract'
error: files must belong to only one module

zig/src/tensorfold.zig exported the contract as @import("core/contract/contract.zig"). That leaf file is also the root of the separate contract module (zig/build/kernel_contract.zig:6), so it ended up belonging to two modules, which Zig refuses. The reference check then read it as tf.contract.

The fix follows what upstream already does. zig/tests/contract/metal_main.zig:5 takes the contract by name (@import("contract")), and upstream zig/src/tensorfold.zig carries no contract import at all. So:

  • zig/src/tensorfold.zig drops the export (2 lines).
  • zig/build/deepseek_v41.zig gives the reference test a named contract module, the same way metal() wires its own targets.
  • operator_engram_fp64.zig reads @import("contract").spec.

Three files, +2/−3.

Verified on M3 Ultra (80-core GPU, 512 GB unified memory), Zig 0.17.0, Xcode 27.0:

Check Result
zig build test --summary all 360/360 steps succeeded; 99/100 tests passed (1 skipped)
zig build test-deepseek-v41-reference pass, ENG64 receipts emit
zig build tf-kernel-contract -Dcpu=apple_m1 compiles (did not compile before this commit)
zig build test-golden 306/319 equal to server goldens, 0 unexpected differences
python3 tools/zig/lean_check.py 3396 files, 444 problems — the recorded baseline, none new

Stashing just these three files restores the file exists in modules error, so the repair is attributable to this commit and not to a cache.

Two checks I could not clear, both environment prerequisites rather than results of this change, and both present before it:

  • kernel-contract stops at UnrecordedRotaryCompiler — it wants tf-rotary-sweep output passed as --rotary-accuracy FILE. Passing --no-limits does not bypass it.
  • With --no-limits --no-gates it stops at TensorUnitsRequired (Gemma coverage on a GPU without tensor units).

So the contract executable now builds and starts; the gates behind those two prerequisites are unrun here, not passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant