[akol1]

Piece (a) of the #690 split, rebased onto the 1.1.0 release (76aafcf27): the Engram token map built in Zig, byte-identical to the release's, with the table generator and the unit tests that keep it honest.

The Engram metadata the model needs beside the checkpoint — the compressed key map — has so far required a Python step with the HuggingFace tokenizers and NumPy. This builds it in Zig from the original release's tokenizer.json and inference config: decode each id, fall back to the raw token when the decoder emits U+FFFD, normalize, dedupe first-seen.

The normalizer chain mirrors the release's Rust pipeline (NFKC, NFD, StripAccents, Lowercase, the whitespace replaces, strip), which composes to NFKD, strip-M*, lowercase — NFD(NFKC(x)) is NFKD(x), and Rust's StripAccents filters all of category M including Mc. The multipliers come from NumPy's PCG64 exactly: SeedSequence's hashmix pool, the XSL-RR output, Lemire's bounded draw — checked against the recorded words, not assumed. The generated NFKD tables come from unicodedata 16.0.0, the version unicode_data.zig already pins.

What changed since the first revision, all of it from review feedback on this branch:

• The generator ships in the tree. tools/zig/nfkd_tables.py follows tools/zig/unicode_tables.py: pinned unicodedata version and normalization version, the same // Generated by … header, a --check mode that names the stale file and the script to rerun, zig fmt post-processing so regeneration is byte-stable, stdlib only. No CI job runs a generator; --check is there for a byte-identity audit. The table itself is unchanged.
• tools/zig/lean_check.py adds the two Unicode table files to GENERATED, mirroring the exemption the checker already grants the Flash Next generator outputs (GENERATOR_OUTPUT). Additive; the 120-column comment rule and every other check still apply. Calling it out because it touches the shared checker.
• The token-map tests now actually run. The program was registered only as an installable executable, so its test blocks never compiled or ran under zig build test — which is how a wrong Hangul expectation survived several revisions. build.zig gained a host_tests flag on the program table and the test step now depends on the program's test artifact. The first gated run caught it: the release chain composes 가 to U+1100 U+1161 with no recomposition, so the test expects the jamo sequence and says why.

The golden test is the release's own file: the builder's output is byte-identical to the 517,680-byte engram-token-map.bin shipped beside the 2.9bpw pack (sha256 b9a7cf9d341f5daf8fc435dbc36153bf4b28c1b8075be20de30d0556fb910511, cmp clean), and the manifest's packed_sha256 matches. Run: zig build tf-dsv41-token-map && ./zig-out/bin/tf-dsv41-token-map --tokenizer <orig>/tokenizer.json --config <orig>/config.json --output /tmp/m.bin.

Verification at this head:

• zig build test -Dcpu=apple_m1 exit 0 (295/295 steps), the token-map suite included.
• zig build native -Dcpu=apple_m1: 54/54 steps.
• python3 tools/zig/nfkd_tables.py --check --zig $ZIG exit 0; regenerating nfkd_data.zig reproduces the committed file.
• tools/zig/lean_new.py /tmp/lean-base-110.txt …: 503 problems on the 1.1.0 base, 503 here, 0 new.
• zig fmt --check build.zig zig clean.

The load-time wiring (building the map at open when it is missing and the original's tokenizer sits beside the tables) lands with the lane-backend piece, where the host lives. #825 stacks on this branch.

[ashhart]

Thank you, @akol1, we will carry the Zig Engram token-map work with your credit into the common DeepSeek metadata stage with original credit; the landing order and remaining checks are in our reply on #827.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thanks @akol1 for #824, alongside @jayleaton's CUDA chain. 1.1.1 includes your shared contribution. The Zig token-map builder and normalization data are shared with the CUDA Engram reader, keeping one metadata contract. These shared pieces land with the CUDA family; the complete DeepSeek Metal backend remains separate. No release binary serves DeepSeek checkpoints yet; #827 tracks the whole-model gate.

Posted by Ash's AI Assistant Synthena