Add opt-in per-expert causal-ablation harness (ABLATE_SCORE) for moe()

[jeswr]

An off-by-default research instrument that measures the causal effect of individual experts on the model's real output — a complement to the routing-affinity Expert Atlas (#175): the Atlas shows what routes where; this shows what an expert actually does.

What it adds

• c/abl.h (new, ~100 lines, self-contained, zero deps): a per-item ablation config with three modes + a fail-closed per-item reset. Holds no cross-item state.
• c/colibri.c — four small guarded insertions:
  • #include "abl.h" + static Abl g_abl = {0};
  • FASE A (after the TOPP block, before the eusage counters + norm_topk): mode 2 route-around (drop the ablated cell so survivors renormalise to routed_scale) and mode 3 module-swap (remap the slot's expert id at the same routed weight) — both act before counters/norm_topk/routed_scale/ROUTE_TRACE, so downstream accounting reflects the post-ablation routing.
  • FASE C (CPU accumulate, after if(!nr) continue;): mode 1 contribution — skip this expert's matmul+accumulate so its weighted output is zero (routing/counters/trace unchanged).
  • run_ablate_score() — a teacher-forced path reached via ABLATE_SCORE=<manifest> that, per item, resets g_abl, configures the ablated (layer,expert) cells, runs one prefill, and reads out the final logits at every target position to ABLATE_OUT (JSONL, coli-ablate/1: exact per-token NLL, gold-vs-best logit margin, argmax correctness, top-32 for an approximate next-token KL). Dispatched just before SCORE. If ROUTE_TRACE is set, the existing tracer dumps the post-ablation routing for free.
• c/Makefile: abl.h added to the colibri prerequisite list; a standalone tests/test_ablate target in TEST_BINS.
• c/tests/test_ablate.c (new): a no-model unit gate that drives a faithful miniature of moe()'s routing+accumulate against the same abl.h the engine links.

Motivation

colibri already reasons about which experts matter — EXPERT_BUDGET (#254), the Expert Atlas (#175, routing affinity only), CACHE_ROUTE. What's missing is a causal instrument: "if this specific expert is dropped / zeroed / substituted for this token, how do the final logits move?" — the evidence needed to tell a safe-to-drop expert from a load-bearing one, i.e. to give expert-pruning/budgeting an empirical basis rather than a blind gate-weight cap. The harness produces the evidence; it doesn't act on it (no pruning claim bundled).

Testing (against HEAD 81f08a0)

• make colibri compiles clean under the project's -O3 -march=native -Wall -Wextra (0 warnings from the patch); ABLATE_SCORE/ABLATE_OUT/coli-ablate/1 present in the binary.
• make tests/test_ablate → ALL PASS (18/18): P1 mode-0 byte-identical to the un-hooked reference (inert when off); P2 ablating an unused cell is a no-op in all 3 modes with the (layer,expert) key respected; P3 abl_reset() restores OFF (no spec leaks to the next item); S1/S2/S3 each mode has exactly its intended arithmetic. Added to TEST_BINS so the inertness guarantee can be a permanent CI gate.

Caveats / scope

• Inert by default. All hooks are guarded by g_abl.mode, only ever set on the ABLATE_SCORE path; with it unset the forward is unchanged.
• mode 1 (contribution) is CPU-path only. The zero-contribution skip is in the CPU expert loop; under COLI_CUDA/COLI_METAL an expert may compute on the GPU before that skip (matches the deterministic-CPU-forward discipline). Modes 2/3 edit the selected set in FASE A and are backend-agnostic. If you'd prefer mode 1 backend-agnostic too, an alternative is zeroing ws[] for the ablated cell right after routed_scale — one line, all backends honour it — at the cost of the route trace showing weight 0 for that cell. Flagged for your call.
• Research instrument, not a hot-path feature — teacher-forced, one prefill per item, final-logit read-out; not wired into serve/generate.
• Prior art (full disclosure). Per-expert causal MoE ablation via router-masking + gate renormalisation is a published method (e.g. arXiv:2606.25092, comparable frontier MoE, with released tooling). The upstreamable value here is a clean, engine-native, inert-by-default implementation for colibri specifically — no per-expert causal instrument exists in the repo today (checked Call for probes: building the GLM-5.2 Expert Atlas — measuring what each of the 19,456 experts actually does #175, EXPERT_BUDGET=N: miss-aware cap on distinct experts/layer (+75% decode tok/s, 6x prefill on low-RAM hosts) #254). Stating this plainly so it isn't sold novelty it doesn't have.

Prepared for review — happy to adjust scope, in particular (a) whether mode-1 should be made backend-agnostic (see caveat), and (b) whether you'd like tests/test_ablate in the default make test gate (it is, via TEST_BINS). Heads-up: there may be a delay before I can respond to review comments.

[JustVugg]

Nice harness — the design is right: the three moe() hooks are `if(g_abl.mode==...)` guards that short-circuit when `mode==0`, so an un-ablated build runs the exact original path (byte-identical by construction), the ablation is per-item with an explicit reset (no leak), and it's fully opt-in via `ABLATE_SCORE`. The standalone `test_ablate.c` faithful-miniature is a good call, and all 9 CI checks that ran are green.

One thing before I merge, because this touches the hottest loop in the engine and I don't merge core-path changes on inspection alone: the token-exact oracle didn't run on this PR (it's absent from the checks — a CI-trigger quirk on your branch, not a failure), and that's exactly the check that proves moe() output is unchanged with ablation off.

Could you confirm byte-identity empirically? Either:

• push a trivial commit to re-trigger CI so the "Inkling oracle (token-exact vs transformers)" job runs and goes green, or
• run the GLM teacher-forcing oracle locally with `ABLATE_SCORE` unset and paste the result (should be token-exact vs the committed reference).

A one-line note that decode tok/s is unchanged with the harness off (the added branches are predicted-not-taken, so I expect zero regression) would close it. Do that and this merges — the code itself already looks correct.