Inkling (Thinking Machines 975B MoE): new engine, o200k tokenizer, bf16->int4 converter

[nbeerbower]

What

Support for Thinking Machines Inkling (975B total / 41B active MoE, Apache 2.0) — a second full architecture for colibri, following the olmoe.c standalone-engine precedent:

• c/inkling.c — Stage-A engine: GQA with interleaved sliding-window (512, 16 KV heads) / global (8 KV heads) attention at 5:1, learned relative-position bias banks (no RoPE), log-length tau scaling past 128k, four depthwise causal short-convs per layer (fp32, residual inside, cached conv state for decode), dense + MoE layers, sigmoid router with loss-free bias and jointly-normalized routed+shared weights, mup logit scaling. Routed experts stream per-expert from the fused 3D tensors (LRU cache, colibri int4/int8 container or f32). -p "prompt" streaming greedy mode; ref-oracle harness mode.
• c/tok.h — o200k pre-tokenizer support (case-aware Split regex), auto-detected from tokenizer.json so the GLM/cl100k path is byte-identical to before. New tok_unicode_o200k.h range tables.
• c/tools/convert_inkling_int4.py — bf16 checkpoint → colibri container (int4 experts + .qs per-row scales, bf16 residents, f32 norms/convs/router). Threaded quantization, shard-resumable, --watch converts while the download is still running. Note for reviewers: TML's fused w13 tensors store gate/up rows interleaved; the converter de-interleaves exactly per transformers' conversion mapping.
• c/tools/make_tiny_inkling.py — tiny random-init oracle generator (HF transformers ≥ 5.14).
• Zen-4-friendly kernels in inkling.c: bf16 residents dotted via AVX512-BF16 vdpbf16ps (halves resident RAM vs f32 — required: f32 residents would be ~172 GB), int8 expert dot via AVX-512 VNNI (IDOT family; IDOT=0 scalar path preserved). Scalar fallbacks keep the dependency-free default CPU path intact.

Validation

• Token-exact oracle (the colibri standard): tiny random-init Inkling via HF transformers — 36/36 teacher-forced argmax, 24/24 greedy, in f32, int4-container+VNNI, IDOT=0 scalar, and bf16-resident modes.

• Tokenizer: 357/357 strings encode identically to HF tokenizers (case transitions, contractions, CJK, combining marks, emoji+modifiers, 300 mixed-charset fuzz cases).

• Converter e2e (--selftest-e2e): fabricates a TML-named checkpoint from the tiny oracle (inverse mapping, re-fused, re-interleaved), converts back, engine reproduces the oracle exactly in both passthrough and int4 modes.

• Real checkpoint (975B, int4 experts + bf16 residents, 469 GiB snapshot):

  $ SNAP=~/Models/inkling_i4 ./inkling -p "The capital of France is" -n 24
  The capital of France is Paris.
  The capital of the UK is London.
  The capital of Russia is Moscow.
  The capital of Italy
  

  Converted weights published at nbeerbower/Inkling-colibri-int4.

Perf status (honest numbers)

Ryzen 9 7900 (24t), 187 GB RAM, NVMe snapshot, commit b60f881, make CUDA=0, single run, cold cache: RSS 90.5 GB, prefill 150 s (5 tok), decode 0.06 tok/s at cache 16/layer — expert streaming bound, as expected for Stage A with a plain LRU. The architecture is unusually kind to long context (55/66 layers have a fixed 512-token window; 1M-token KV ≈ 46 GB bf16, RAM-resident), so the usual roadmap applies and should pay off more than it did for GLM: hot-store/pinning + prefetch (tier.h), then a GQA CUDA tier — the attention here needs no MLA-style custom kernels.

make check passes (62 tests), -Wall -Wextra clean, GLM oracle path untouched (tok.h dispatch is additive).

Happy to split this into smaller PRs if you prefer (tokenizer / engine / converter).

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[nbeerbower]

Pushed a Stage-B performance commit (3ef6775) — same architecture, the expert path grew the glm.c playbook:

• cache slots keep container rows packed (int4 stays 4-bit in RAM, ~28 MB/expert → double the cache per GB); new matmul_q4 unpacks nibbles in-register, numerically identical to the int8 path
• moe() routes everything first, then fills all missing experts in one parallel burst (prefill batches the whole sequence's misses, so the NVMe finally sees queue depth), then computes
• usage-history pinning (SNAP/.coli_usage, top PIN_N/layer non-evictable) + RAM-sized auto cap

Measured on the 975B checkpoint (Ryzen 9 7900, NVMe, cap auto=64/layer, 1024 pinned):

	Stage A	Stage B cold	Stage B pinned+warm
prefill (5 tok)	150.2 s	31.5 s	21.7 s
decode	0.06 tok/s	0.21 tok/s	0.25 tok/s
cache hit	~0%	72.0%	81.5%

All four oracle modes (f32, int4-container VNNI, IDOT=0 scalar, bf16 residents) remain token-exact, make check 68/68, 0 warnings.

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[nbeerbower]

Final push for this PR — it now includes the CUDA resident tier and cache warming, bringing inkling to feature parity with what glm already has (no concepts new to colibri):

• CUDA tier (backend_cuda_ink.cu, CUDA=1): bf16 residents (~36 GB) live in VRAM on a warp-per-row kernel with the same numeric contract as the CPU vdpbf16ps path; freed RAM goes to the expert cache. Pure-CPU default build unchanged and dependency-free.
• Cache warming (glm's .coli_usage idea): per-expert usage counts persist and accumulate; top PIN_N/layer pinned at startup. Fully toggleable (PIN=off, PIN_N, USAGE_SAVE=0). The -f promptfile mode exists to build the history for a binary with no serve mode yet — happy to drop it if it reads as scope.
• docs/inkling.md: quickstart, env reference, benchmark methodology.

Measured on the 975B (Ryzen 9 7900 / A6000 / NVMe, commit 87981c1, greedy):

Configuration	Prefill	Decode	Hit
Stage A	150.2 s	0.06 tok/s	~0%
CPU perf pass	21.1 s	0.25 tok/s	81.5%
+ CUDA residents	18.4 s	0.32 tok/s	83.6%
trained-prompt pins (best case)	1.9 s	2.51 tok/s	100%
steady state (novel prompt, diverse history)	35.4 s	0.25 tok/s	82.2%

The honest number is the last row; it grows toward the best case as usage history accumulates (counts persist across runs). Phase timers show the remaining time is ~90% CPU expert matmul at high hit rates — GPU expert compute is the natural follow-up PR, not included here.

All modes remain token-exact vs the transformers oracle (f32, int4-container VNNI + IDOT=0, bf16 CPU, bf16 CUDA); make check 68/68, 0 warnings on both builds.

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[JustVugg]

Thank you for this — real work, and I want to be straight with you about the bar rather than let it sit open for another week.

The maintainer's rule, and it's the right one: a new engine lands only if he can run it on his own machine first. Not "the CI compiles it" — run it. Right now neither of us can: there's no checkpoint here, so the only thing that could be verified is that gcc doesn't complain. That's not verification, that's a spell-check.

Why the bar is that high, concretely. This week, on the one engine we can run:

• EXPERT_BUDGET shipped on numbers nobody checked and had to be quarantined (EXPERT_BUDGET (merged in #254, now on main): measured empty operating window — needs a quality gate and an MTP-contract decision #303) — measured 0.13 tok/s against a 0.30 baseline while claiming a speedup, and loading more experts per token than not using it at all;
• three separate users are getting incoherent output and the root cause is still unconfirmed ([Bug]: python coli chat doesn't end #307, Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225);
• a CI check went in that could never fail;
• two tests I wrote myself "found" bugs that didn't exist.

That's the state of the engine that gets daily attention. A second engine that nobody here can execute doesn't get daily attention — it gets zero, until the month it breaks and the person holding it is the maintainer, who never had the weights.

What would change the answer, and I think it's reachable:

1. A model he can actually run, or a tiny one the CI can execute end-to-end. This repo already has the pattern: make_glm_oracle.py builds a glm_tiny/ and the engine is scored token-exact against it. An engine with an oracle test is an engine that can't silently rot.
2. A named maintainer — you, on the record, for that file.
3. Isolation from glm.c's path. This is the part I'd check hardest, and it's specific per PR (see below).

Until then the honest home is a fork, linked from the README. That's not a brush-off: it means people can find and use your work, and it doesn't hand a maintainer 10,000 lines he can't test. If it gets traction and someone can run it in CI, it comes back.

I'd rather tell you this now than leave it open for a month and close it then.

[JustVugg]

Specific to this one — and it's a more favourable read than #165.

You have a track record here: #232 landed today after I merged and tested it (PROF=1 works, the instrumentation costs ~48µs/token against tokens that take 3-30 seconds, so it's free in practice). That matters. This isn't a first PR from someone we've never seen.

inkling.c (+1,255) is isolated and that's the right shape — a separate file, like olmoe.c. backend_cuda_ink.{cu,h} likewise. If the whole thing were that, my only objection would be "nobody can run it".

The part I'd check hardest is c/tok.h (+115/-1) — 118 changed lines in the tokenizer glm.c depends on. That's the one place this PR can break the flagship engine, and it's the reason I'd want a real review rather than a wave-through. If the o200k support is purely additive (new tables, new branches, glm's path untouched byte-for-byte), it's defensible. If it refactors anything shared, that needs to be visible.

Concretely, that's testable without an Inkling checkpoint: if the tokenizer change is additive, the GLM oracle still scores token-exact against glm_tiny/, and tests/test_stops.c (which loads a synthetic tokenizer and checks the special flags) still passes. If you can show those green with your tok.h, you've retired the only risk I can actually name.

The remaining one is the maintainer's rule: he needs to run Inkling here before it lands. 975B is not a small ask, so if there's a smaller Thinking Machines checkpoint — or a way to build a tiny one with convert_inkling_int4.py, the way make_glm_oracle.py builds glm_tiny/ — that's the path, and it would also give the engine an oracle test so it can't rot silently.

[nbeerbower]

Totally get wanting to try it yourself before merging. Just wanted to get pen on paper mostly.

Made a separate PR #330 for just the tokenizer bit.

[JustVugg]

The model is interesting and there are many things we can try to bring it into the hummingbird catalog. Now with kimi 3 it would also be interesting. The open world is going wild!

[JustVugg]

@nbeerbower this is a serious piece of work and a second full engine is exactly where the project wants to go — thank you. One thing before merge, and it's the same bar the GLM engine holds itself to: a tiny test model + oracle check we can run in CI and locally.

For GLM we keep glm_tiny (a few-MB randomly-initialized model in the real container format) and the engine's TF=1 teacher-forcing mode prints N/32 vs oracle — token-exactness against a transformers reference. That's what lets us merge refactors all day without fear. Since none of the maintainers can run the real Inkling 975B, without a tiny model your engine would be effectively untestable by anyone but you — and the first refactor that touches shared code (tok.h, Makefile) could silently break it.

I see you already ship tools/make_tiny_inkling.py — could you: (1) generate + commit (or attach) a tiny Inkling snapshot, (2) wire a TF=1-style oracle check (32-token teacher-forcing vs HF transformers on the same tiny model), and (3) add the one-liner to CI? With that in place this merges as-is — the code is additive and CI is already green. Happy to help with the CI wiring.

[JustVugg]

Status check, and sorry for the wait. This conflicts with dev now — the engine was renamed and split in #391 (glm.c → colibri.c plus header modules) and c/Makefile and c/tok.h have both moved since. Are you still working on it?

On the substance, the structural choice here is right and worth saying out loud: inkling.c as a separate engine alongside colibri.c, sharing the container and tokenizer infrastructure but not entangling the GLM path, is exactly how a second model family should arrive. It's the same shape as the DeepSeek V4 work in #165, and it's why both are reviewable at all at their size.

Two things before I can take it seriously as a merge candidate:

1. A tiny-model oracle. make_tiny_inkling.py is there, but what settles a new engine in this project is a token-exact check against a transformers reference on a tiny checkpoint that lives in the repo — the way glm_tiny does. If tools/make_tiny_inkling.py already produces that and there's a teacher-forcing comparison, point me at the command; if not, that's the piece I'd want most.
2. CI must build it. A second engine that isn't compiled by make check rots within weeks. Add DeepSeek V4 Flash CPU inference with NVMe expert streaming #165 wires its build and tiny-oracle into the check target; the same approach would work here.

If you're still on it, rebase and tell me about the oracle. If you've moved on, say so and I'll close it as groundwork with credit rather than leaving it open — no hard feelings either way, and the design note above stands regardless.

[nbeerbower]

Still on it — rebased onto dev and pushed, so it's mergeable again. The glm.c→colibri.c split was a clean rebase: inkling.c / backend_cuda_ink.* are isolated new files, and the tokenizer commit dropped out entirely since o200k is already upstream (that was #330), so tok.h is now byte-identical to dev. The only real re-apply was the Makefile inkling target onto the new $(GPUCC) build.

On the two asks — both are in the PR now and green in CI:

1. Tiny-model oracle. It's the glm_tiny shape exactly. tools/make_tiny_inkling.py builds a small random-init InklingForCausalLM via HF transformers (8 layers, hitting every branch: sliding + global attention, log-length tau, dense + sparse MLP, 2 shared experts, unpadded vocab) and writes ref_inkling.json. The engine reproduces it token-for-token:

python3 tools/make_tiny_inkling.py tiny_inkling
SNAP=tiny_inkling ./inkling 8 0 tiny_inkling/ref_inkling.json
#  teacher-forced argmax: 36/36 match
#  Matching tokens: 24/24

bits=0 keeps experts f32 so the forward is bit-exact; the greedy pass exits non-zero on any mismatch, so it's a real gate, not a print. Green in f32, int4-container+VNNI, IDOT=0 scalar, and bf16-resident modes.

2. CI builds it. The Engine (Linux, CPU) job now make colibri inkling, the CUDA-syntax job also nvcc -c backend_cuda_ink.cu, and there's a new inkling-oracle job that pip-installs CPU torch + transformers>=5.14 and runs exactly the two commands above. All three pass on this push. I kept the oracle out of the dependency-free make check gate on purpose — that one runs on Windows/macOS where I haven't confirmed the engine builds yet, so it's a Linux CI job instead. Happy to fold it into make check cross-OS as a follow-up once I've verified the Windows/macOS builds.

One honest scope note: the engine has grown well past this PR in my fork (Schneewolf-Labs/sabrewing) — GPU expert tier, int8 residents, MTP draft/verify, LoRA serving, per-phase profiling. I'd suggest landing this Stage-A→CUDA-warming version + oracle + CI first (reviewable at size, the shape you described), and I'll bring the perf/CUDA/MTP work as follow-up PRs rather than dropping another 10k lines at once. Sound right?