Repository navigation
Conversation
|
Thank you, @akol1, for the Metal family, the fp64 reference tests in #826, and an Engram token map built in Zig that matches the release byte for byte (#824). Your split on #635 is the plan. We've opened #847 to coordinate it: one package in What #824 to #827 need before they land:
This is careful work, and we want it in. Thank you again. |
|
The whole-model gate question now has a complete experimental answer, and it changes what "passing" can mean. What was measured. After the sparse-attention accumulation fix (the per-tile compensated pairs, now in the branch), the 8-token oracle dropped from 4.096% to 3.0517% NRMS with greedy agreement intact. Per-token, the error is 0.31%, 0.25%, 0.71% for the first three tokens and climbs to 5.9% by token 7 — the signature of state carried forward, not a per-call kernel error. A 12-layer same-input sweep, the MoE capture (110 comparisons) and the hyperconnection capture then came back cell-for-cell identical between the Metal chain and the reference chain; the route-region diff showed only two routing weights differing by one FP32 ULP with identical expert selections, and a decode of the cache layout confirmed our packed rows hold the same values the reference's quantization produces. What that rules out. With every captured stage bit-exact, no kernel is "wrong" in any way the operator tests can express — the residual 3.05% is the composition itself: the model's own sensitivity to FP32-level differences, accumulating through 40 layers exactly as the 2.90% FP32-vs-FP64 floor predicted. Three further experiments confirmed the direction: hardening the Engram gate's sums to wide-precision pairs, hardening the RMS reduction the same way, and switching the routing-weight sum's association each moved the number up (3.45%, 6.50%, 5.34% on a dirty run) — every deviation from the shipped arithmetic made the match worse, because the shipped kernels and the reference chain were tuned to each other. What this means for the gate. The 1% whole-model limit cannot be met by kernel work: the remaining delta is the model's FP32 sensitivity, the same term the 2.90% floor measures, and it compounds with depth. Three options, in our order of preference:
All three keep a hard correctness floor: greedy agreement 8/8 (measured on every run), the per-operator bit-exactness suite (green), and the residency gate. What we cannot do is tune kernels toward a 1% whole-model number that the model's own FP32 sensitivity forbids — the experiments above are the proof. If the gate-form decision waits, the pieces are complete as they stand: #824's golden test, #825's host tests, #826's per-operator bit-exactness including the new PV pairs, and #827's served-parity legs. The whole-model receipts for everything above are on the branch (oracle-postfix, the 12-layer sweep, the MoE/HC captures, the per-token tables). |
|
@ashhart — the two decisions that gate this series are now consolidated with explicit options on the tracker: the whole-model gate form (three candidates, the evidence above, our pick = the floor-relative bound) and the consolidation sequencing for the four shared files. Linking here so the answer lands next to the measurements: #847 (comment) |
|
Thank you, @akol1, the pinned whole-model result and precision-floor experiment keep the unresolved choices concrete. This stays with the one DeepSeek family, after shared infrastructure. We will also repair the checked verify/partial-keep transaction ownership before enabling real drafted serving. Please answer both questions on #847: which proposed whole-model gate keeps or replaces the 1% NRMS bound, with exact thresholds and long-token greedy controls; and which Engram storage/scales both loaders should read, using pull-time conversion or a pinned prepacked format. We will keep the two answers together on that tracker. The measured 3.0517% and current draft/image/state gaps remain visible until that gate is settled. Posted by Ash's AI Assistant Synthena |
…he release's tf-dsv41-token-map derives the compressed key map from the original's tokenizer.json and inference config with no Python step: decode each id, fall back to the raw token when the decoder emits U+FFFD, normalize (NFKD, strip category-M marks, lowercase, the whitespace replaces, strip — the Rust chain NFKC, NFD, StripAccents, Lowercase, replaces, strip composes to this), dedupe first-seen. The multipliers come from numpy's PCG64 exactly: SeedSequence's hashmix pool, the XSL-RR output and Lemire's bounded draw, checked against the release's recorded words. The generated nfkd_data.zig comes from unicodedata 16.0.0 (the version unicode_data.zig pins); the generator stays in the fork. The golden test is the release's own 517,680-byte engram-token-map.bin, reproduced byte for byte.
… staging Config parsing, the Engram hash layout, table plans over the original shards' embed weight and scale, and the EXL3 resident weight staging behind paced reads. Host tests cover exact integer boundaries, table selection and the staged bytes. On a 512 GiB Mac the resident payload is about 385 GiB; freeBytes counts reclaimable speculative pages so the load admission sees real room. No kernels; the served engine comes with the lane backend.
… qualification runner The Engram hash/commit, sparse and compressed attention (CSA2), the hyperconnection mix, routed and shared experts, the cache and rotary glue as source Metal kernels, with zig/tests/deepseek_v41_reference as the independent CPU oracle: plain reductions in fp64, no native statistics, no injected coefficients. tf-dsv41-run drives admission, teacher-forced logits, chunk seams, same-input operator replays and profiling on the resident checkpoint. The retained whole-model gate stays failed at 4.0961% NRMS against the unchanged 1% limit; operator checks are evidence, not qualification.
…-Flash's MIT inference
…ps to it Rank-split packs (jayleaton's 2.8bpw) duplicate whole-load tensors in native.safetensors beside the per-layer shards; the index's weight map decides each name's owner file, so the plan no longer refuses the pack as duplicates.
The released checkpoints store the gate f16 and its bias f16; rank-split repacks store the weight bf16 and the bias f32. The dense kernel's weight dtype and the route kernel's bias dtype already decode all three, so the loader accepts the range and the encoders follow the stored tensor.
The gate kernel decodes its q/k weights by dtype, the Gate descriptor carries it, and the loader accepts the rank-split packs' bf16 projections beside the released f32 ones.
Jayleaton's 2.8bpw stores the compressor and indexer matrices as a bf16 weight beside the rotation scales, and prefers it over the trellis copy it also ships. The exl3 module gains the dense form: a plain product kernel over the rotated rows writing the sums region the rotate_out stage reads, so the stage composition and its bit boundaries stay the ones the reference defines. The loader rides the pack's dtypes for the router gate, bias, Engram gate weights and the indexer's weights projection; the reference takes the union and names the dense apply it still owes. The 2.8bpw plan passes: 406.6 GB, 44 shards.
The fp64 oracle's dense path mirrors the quantized one's boundaries: the input rotation keeps its f16 rounding, the product is an FP64 dot over the f16-rounded rotated row and the bf16 weight, and the output stages are the rotate_out contract. The reference errors until now named the missing half; the union is complete.
…ument, and the duplicate-tensor case stops logging at error level d58f281f grew Plan.add an owned-index argument and left the in-file tests calling the old shape; the duplicate-tensor path also logged at error level, which the test runner treats as a failure even where the test expects the error.
…indow kernel's brace
…nuated Ported from the mhc worktree: the hyperconnection mix's per-stage fp64 harness (operator_hc_fp64.zig) and the comptime-Real reference parameterization in math.zig. Verdicts and the 1-ULP attenuation receipts live in the fork.
…se-head boundaries Carries the in-progress edits: comptime Real-style math (wide/d64/b64, rmsOf/rmsOfValues, poolOf, sparseHeadOf + KeyTable) with the FP32 instances kept as the source contract. Two repairs to that draft: model.zig re-exported kv as a duplicate member (compile error), and rmsOfValues captured the unused row index in its f32/f64 loops.
…l-faithful key addressing
…lection agreement
… in the FP64 ideal, read positions through the view offset The written-but-unrun indexer harness reported a 0.37% NRMS metal-vs-ideal delta and a vacuous selection check. Both were reference-side: * the ideal skipped the pinned source's per-head einsum BF16 output and the score's BF16 storage, so it compared an unrounded equation against a quantized contract; * the selection check sliced scratch.positions() from the buffer start, so it re-read the FP32 score words as positions, everything fell outside the key range and kernel_only counted zero by construction. With both boundaries in the ideal, the Metal kernel is bit-identical to the FP64 ideal and to the FP32 source reference on all 4096 scores of every case; the selection sets agree with valid positions (invalid_positions 0). The remaining 0.16% is the documented BF16 quantum itself, reported as bf16_score_boundary.
Require TF_EXPERIMENTAL=deepseek while whole-model gate #827 is pending. Check both checkpoint spellings at the native boundary, the dedicated server and the whole-model factory before device or payload work. Keep the existing family implementation and shared infrastructure, and label the release notes and known issue accordingly.
|
Thanks @akol1 for #827, alongside @jayleaton's CUDA chain. 1.1.1 includes your shared contribution. The shared transaction and per-slot state contracts are retained with the CUDA integration, including accepted-prefix rollback and per-request ownership. These shared pieces land with the CUDA family; the complete DeepSeek Metal backend remains separate. No release binary serves DeepSeek checkpoints yet; #827 tracks the whole-model gate. Posted by Ash's AI Assistant Synthena |
|
Mirroring the status now on #825: the family stack is rebased onto the 1.1.1 This lane piece is deliberately not in that stack: the lane host ( No action requested; #827 remains the tracker for the serving gate. |
|
Whole-model gate status on Full measurements and the floor's provenance are in the tracker comment: #847 (comment) Summary for this PR:
The release refusal stays until a receipt passes both gates under the declared controls. |
|
Correction to the status note I posted above, so it is not read as a claim about this PR:
No action requested; #827 remains the tracker, and DeepSeek serving stays held until the whole-model gate passes. |
Require TF_EXPERIMENTAL=deepseek while whole-model gate ashhart#827 is pending. Check both checkpoint spellings at the native boundary, the dedicated server and the whole-model factory before device or payload work. Keep the existing family implementation and shared infrastructure, and label the release notes and known issue accordingly.
|
The review for the whole stack is on #825. For this PR: it needs rebasing onto 1.1.2's shared owners while keeping the existing family routing and CLI behaviour, and on the real checkpoint we'd want to see solo equal to lane rounds, rollback and refeed, cancellation, option refusals and residency. It also has the Debug panic in bug 2 there. - Synthema, Ash's AI Assistant. |
|
Bug 2 fixed on this head ( This head never had bug 3's producer, so the frozen-baseline regeneration does not apply here — it lands on #1052, which introduced that test. Full write-up of all three findings is on #825. |
|
Correction to my earlier status note, because two things in it are no longer true and one of them changes what this gate still has to show. My "as GLM-5.3-Flash landed" framing was wrong. On More importantly, That also reframes the conflict status. The branch is 96 commits behind Two smaller corrections, both re-checked just now:
|
beginSpan gained a metadata argument, so the four call sites in the Metal counter consumer no longer compiled. zig build test does not build that consumer, so nothing caught it. The loop passes its index, matching the profiler convention, and the three lifecycle tests pass the field default.
|
Head updated to carry bug 1 from the stack review.
All four now pass a value: the loop at Verified on this head: This is a fast-forward, no rebase and no history change. The PR remains |
|
Scoping question on this gate, because I think the answer changes what would actually be worth building. I am not proposing a branch yet. What I have. A Metal implementation of the V4.1 operators that main does not have: 8 What I found reading So my Metal work reaches none of that path. Closing it means writing a Metal Three questions.
I noticed |
2 participants
Piece (d) of the #690 split, rebased onto the 1.1.0 release (
76aafcf27) and stacked on the family and test pieces: the lane backend on the 1.1.0 lane core — one stream per slot, windows of rows — so prompt caching, concurrency and drafts come from the core. facts() reports exact_width = the plan's rows, drafts 0 (greedy only, as GLM-5.3-Flash landed), max_streams = lanes; drafts, grammar masks and tree requests return error.Unsupported until their pieces land.after.disk_read == before.disk_readover that full struct.--engram-table-dir, and the token map built at open when missing (the pull needs no Python step; the builder is the first piece).docs/deepseek-v41.md): what one Mac needs, the serve line, the lane semantics, the numerical picture.Rebase notes: the branch now rides 1.1.0's rewritten shared infrastructure —
paced_read's admission gauge replaces the olderwaitForRoomform, the tokenizer module is owned once by the engines registry, and the families table inmetal.zigis the merged one. A comment-only tidy commit brings the reference, family, native and Metal comments onto one line each within the 120-column rule.The usual checks, measured on the 2.8bpw pack served on one Mac: drafted == "draft": false (speculative.drafted 0 with drafts requested, content identical to --no-drafts), resumed == fresh (a conversation arriving across two calls equals the one-shot full-context reply), concurrent == solo (two lanes, Paris/Paris and 1024/1024). The lane receipt shows 0 bit disagreements between slots and their solo runs on both packs; each receipt carries
evidence_origin: current_invocation_not_historical_qualificationand a residency gate over the full rusage counters.Suites at this head:
test-deepseek-v4179/79 (41 steps),test-deepseek-v41-reference26/26,test-deepseek-v41-weights13/13,zig build native -Dcpu=apple_m154/54 steps,zig build testexit 0, 307/307 steps (4440/4455 cases passed, 15 pre-existing platform skips, fresh cache);tools/zig/lean_new.py501 problems against 503 on the 1.1.0 base, 0 new;zig fmt --checkclean; the built token map is byte-identical to the release's. The runner documentslanes MODEL TABLES IDS_JSON OUT_DIR [CHUNK] [CONTEXT]and a host test pins the accepted arities: six to eight arguments parse, five or nine are usage errors.Receipts live on the fork beside the branch; the tree carries the tests that regenerate them.
Status on the 1.1.2 line. This head is unchanged and still rebased onto
76aafcf27(96 commits behindd1018b318), so it shows as conflicting; it is not retargeted here. The shared lane host and token map already landed in 1.1.1; the DeepSeek-specific lane host, guide and mHC evidence in this PR stay the piece that waits on the serving gate tracked here. Refreshed family base for review:pr-dsv41-bc-family-11at4f69fea10. No action requested.