pilot: multi-worker PILOT_REAL prefetch (PILOT_WORKERS) — byte-identical base for the #441 hardware A/B

[cdhdt]

Draft / not a merge request (yet). This is the byte-identical base for the good-hardware
A/B you offered to flag on #441 — opened as a PR because you asked for exactly that ("open the
PILOT_WORKERS branch as a PR with default=1 and the eviction guard added, I'll flag it to the
enterprise-NVMe / big-RAM folks"). Please don't merge it until that A/B shows a win — on every
box I can measure it doesn't. Default behavior is unchanged (PILOT_WORKERS=1).

What this is

Part-1 of #441: multi-thread the blocking PILOT_REAL prefetcher so speculative loads run at
queue-depth N instead of 1. SPMC ring (pilot_ring_claim, CAS claim, read-payload-before-claim),
PILOT_WORKERS env (default 1 = today's exact behavior), the URING path's visible -(eid+2)
reservation ported to the blocking loader, and the LFRU eviction guard re-applied on top of the new
SPMC slot-selection — in its corrected #497 form since the 2026-07-26 rebase. Rebased onto
current dev.

Why it's a draft, not a merge request

On every drive I can test it regresses or no-ops, and your own gate on #441 is the right one:
multi-worker needs (a) LFRU-aware eviction (merged as #474, corrected in #497) and (b) a good-hardware A/B
before it ships even opt-in. This PR is (a)-done + the vehicle for (b). It should merge only if
the enterprise-NVMe / big-RAM A/B shows a real win.

What I measured (and why it isn't a win here)

Updated 2026-07-26, on the rebased base. The numbers first posted here were taken with my
original #474 guard, which #497 showed was over-dropping
(~100% of speculations once the cache filled, the #490 regression). That old table is a broken
baseline — its guard ON arm sat at PILOT-off parity, which in hindsight is the #490 signature.
Everything below is re-measured on current dev with the corrected guard.

Controlled 2×2 on my QLC / 62 GB box — PILOT_WORKERS {1,8} × PILOT_EVICT_GUARD {0,1}, frozen
.coli_usage restored before every run, CAP_RAISE=0 (cap 8/layer everywhere), one native build,
interleaved, 2 reps in reversed order, DIRECT=1, greedy, 12-token decode. Output byte-identical
across all 11 runs (sha256 of the generated text):

arm	expert hit	MB/token	felt wait	decode wall
PILOT off	44.1 / 44.2 %	5662 / 5654	25.0 / 25.0 s	57.2 / 57.4 s
workers=1, guard ON	49.3 / 49.7 %	5887 / 5865	24.8 / 23.8 s	78.2 / 57.2 s
workers=8, guard ON	49.0 / 49.8 %	5953 / 5868	26.3 / 24.8 s	80.1 / 56.0 s
workers=1, guard OFF	67.5 / 67.8 %	7322 / 7265	18.6 / 17.5 s	86.6 / 55.7 s
workers=8, guard OFF	67.6 / 68.1 %	7416 / 7311	29.6 / 32.5 s	66.9 / 65.9 s

(rep1 / rep2. Wall clock is drift-dominated in rep 1 and should not be read across reps; hit rate,
MB/token and felt-wait reproduce tightly and are the trustworthy columns.)

workers=8 still doesn't pay here. Guard ON it costs +1.0/+1.5 s felt wait for +0-1% bytes;
guard OFF it costs +11/+15 s. There are no cold slots at cap 8/layer for extra queue depth to
fill, so this box structurally can't show the upside — that needs a host with a real cache
(big RAM) + a drive with a QD8-32 sweet spot / idle bandwidth during compute.

Output-preserving

Cache placement + prefetch scheduling only — never a value. The moe() per-layer barrier fences
every layer against in-flight pilot loads; reservations (-(eid+2)) and loading slots (eid=-1)
are never matmul'd; a dropped/late speculation is just demand-loaded. sha256-identical across
PILOT off / workers 1 / workers 8 / guard on / guard off on the full GLM-5.2 744B model (11/11 runs
on the rebased base, 9/9 before it).

What's in it

• SPMC pilot ring + PILOT_WORKERS (default 1), visible -(eid+2) reservations on the blocking path
• tests/test_pilot_ring.c (N-thread ring stress: each item claimed exactly once, no loss/dup/torn)
• LFRU eviction guard (pilot: LFRU-aware eviction guard — a speculative load never evicts a warm demand expert (#441) #474, corrected form from fix(pilot): narrow LFRU eviction guard so speculations don't over-drop (#490) #497) re-applied into the SPMC slot-selection
  (PILOT_EVICT_GUARD, default on)
• Vetted by 3 adversarial subagent reviews for the barrier + byte-identity; the rss_guard
  eviction UAF this surfaced already shipped separately as rss_guard: free the evicted expert slab under g_pilot_mx (fixes a use-after-free with the pilot prefetcher) #449.

How to run the A/B (single binary)

# baseline arm = today's behavior
PILOT_REAL=1 PILOT_WORKERS=1 PILOT_EVICT_GUARD=1 DIRECT=1 ./coli run ...
# deeper queue
PILOT_REAL=1 PILOT_WORKERS=8 PILOT_EVICT_GUARD=1 DIRECT=1 ./coli run ...

Report PROF=1's byte counter (MB/token) + felt-wait (and tok/s). PILOT_EVICT_GUARD=0 isolates
the eviction-thrash component.

Validation

• make -C c check — 111/111, native + portable, 0 warnings
• Byte-identical output across all four quadrants + PILOT-off (sha256, 11/11 runs on the
  rebased base)
• Throughput A/B on a QD-scaling / big-RAM host — the missing datapoint this draft exists for

Compatibility

• Default PILOT_WORKERS=1 = current behavior; no default-config user is affected
• Default CPU build stays dependency-free; no model files / binaries / artifacts

---

Implemented, reviewed, and measured with AI assistance (Claude), per the project's AI-contribution
precedent (#428, #398). Kept as a draft on purpose — it should not merge without the good-hardware
win it's built to let you measure.

[JustVugg]

This is the exact base we want for the #441 question. To move it out of draft and merge: keep PILOT_WORKERS default=1 (today's behaviour, byte-identical — you've already proven sha-identical output across 1/8), pair it with #474's LFRU eviction guard (merged) so the extra workers can't thrash a warm demand expert, and then the missing piece is a good-hardware A/B: on a box with a real QD8-32 sweet spot + idle bandwidth (enterprise NVMe, or a big-RAM host where the drive is idle during compute), does multi-worker net positive? Your QLC result says it loses there for a measurable reason (eviction + contention); the enterprise datapoint is what proves whether it pays. Post that A/B (or flag the folks on #389/#161 to run it on your byte-identical branch) and it lands as opt-in. Thanks @cdhdt.

[cdhdt]

Thanks @JustVugg — that's the plan exactly. State of the branch for whoever runs the A/B:

• PILOT_WORKERS=1 (default) is today's behaviour, and output is sha256-identical across PILOT off / workers 1 / workers 8 / guard on / guard off on the full 744B (9/9 runs) — so no arm can change tokens, only timing.
• The merged pilot: LFRU-aware eviction guard — a speculative load never evicts a warm demand expert (#441) #474 guard is in and on by default (PILOT_EVICT_GUARD=1); CI is green on linux / macOS / windows.

Single-binary A/B for the enterprise-NVMe / big-RAM folks (#389 / #161), baseline arm first:

PILOT_REAL=1 PILOT_WORKERS=1 PILOT_EVICT_GUARD=1 DIRECT=1 PROF=1 ./coli run …   # today's behaviour
PILOT_REAL=1 PILOT_WORKERS=8 PILOT_EVICT_GUARD=1 DIRECT=1 PROF=1 ./coli run …   # deeper queue

Report the [PROF] expert I/O byte counter + felt-wait (and tok/s). On my QLC box workers=8 loses — guard-OFF reads +44% bytes (eviction thrash), guard-ON removes the thrash but there's no idle bandwidth to convert, so it's a wash. The open question is exactly yours: on a drive with a QD8-32 sweet spot + idle bandwidth during compute, does the deeper queue net positive? Happy for #389 / #161 to run it on this byte-identical branch — I'll fold their numbers in here.

Written with AI assistance (Claude).

[JustVugg]

Status check on this one — it's marked draft, and it conflicts with dev now (colibri.c moved today in #532). Are you still working on it?

The framing in the title is the part I want to keep: byte-identical base for the #441 hardware A/B. A prefetch change that can't alter output is exactly the kind that can be evaluated on speed alone, and that's the only honest way to compare a prefetch strategy — otherwise a quality difference hides inside a throughput number.

Two things worth knowing before you spend more time on it, because the ground moved under this feature recently:

1. The pilot eviction guard was fixed in fix(pilot): narrow LFRU eviction guard so speculations don't over-drop (#490) #497 — it was dropping ~100% of speculations once the cache filled, which collapsed PILOT_REAL to a hint-only path. Any measurement taken before that fix was measuring a broken baseline, so if your numbers predate it, they're worth re-taking.
2. The RAM budget now enforces itself against measured RSS ([Bug]: generation RSS overshoots the projected peak by ~40 GB on GB10 (Grace 20-core, 121 GB unified) — three kernel OOM kills; a low --cap contains it, --ram does not #403), and extra prefetch workers mean extra in-flight buffers. Worth checking that PILOT_WORKERS>1 doesn't push the guard into evicting the very experts the prefetch just fetched — that failure mode would look like "prefetch doesn't help" rather than like a bug.

If you rebase and it's ready to leave draft, I'll approve the workflows and review it. If you'd rather park it until the #441 A/B has hardware behind it, that's reasonable too — just say which, so it isn't sitting here ambiguous.

[cdhdt]

Still on it — thanks for the #497/#490 heads-up. You're right that my original guard over-dropped, so any numbers from before that fix sit on a broken baseline. I'll rebase this onto current dev (so it's the corrected #497 guard under test, not my old one) and re-confirm byte-identity, then settle the draft-vs-park question here — I'll get to it tomorrow. Not abandoned; picking it up fresh. Thanks for flagging the RSS/#403 interaction too, I'll watch for it.

[cdhdt]

Rebased onto current dev (165f5a2), conflicts gone, CI green — and the answer to your question is: park it. Details below, including a correction to my own earlier numbers.

The rebase

The branch now sits on the corrected #497 guard, not my broken #474 one. Both speculative sites (pilot_realload, pilot_uring_batch) use dev's narrowed test — protect the victim only when it is genuinely warm (eheat >= 2) and clearly hotter than the speculation. Two things I fixed while resolving:

• The guard stays inside the eviction branch in pilot_uring_batch. My pre-rebase branch had moved it out of that branch (down past the slot<0 check), so it also ran on the grow path — and fresh ecache slots are calloc'd/memset, i.e. eid == 0, which the guard reads as "this slot holds expert 0". It would then score a slot that holds nothing and could drop a good speculation for it. Only ever present in my branch, never in dev, but worth naming: eid == 0 being a valid expert id is a sharp edge anywhere a zeroed ESlot can be reached.
• Dead isnew removed: with visible -(eid+2) reservations the slot is published at claim time, not after the pread.

Re-measured on the corrected base — and a correction to my July-21 table

Same protocol as before: frozen .coli_usage restored before every run, CAP_RAISE=0 (cap 8/layer everywhere), one native build, interleaved, 2 reps in reversed order, DIRECT=1, greedy, 12-token decode, i7-12700H / 62 GB / Crucial P3 Plus (QLC, DRAM-less).

Output is byte-identical across all 11 runs (sha256 of the generated text: PILOT off ×2 / workers 1|8 × guard on|off × 2 reps). Placement-only contract holds on the new base.

arm	expert hit	MB/token	felt wait	decode wall
PILOT off	44.1 / 44.2 %	5662 / 5654	25.0 / 25.0 s	57.2 / 57.4 s
workers=1, guard ON	49.3 / 49.7 %	5887 / 5865	24.8 / 23.8 s	78.2 / 57.2 s
workers=8, guard ON	49.0 / 49.8 %	5953 / 5868	26.3 / 24.8 s	80.1 / 56.0 s
workers=1, guard OFF	67.5 / 67.8 %	7322 / 7265	18.6 / 17.5 s	86.6 / 55.7 s
workers=8, guard OFF	67.6 / 68.1 %	7416 / 7311	29.6 / 32.5 s	66.9 / 65.9 s

(rep1 / rep2. Wall clock is the untrustworthy column here: the rep-1 runs spread 66.9–86.6 s with no ordering that matches any arm, i.e. session drift dominates it. Hit rate, MB/token and felt-wait reproduce tightly across the two reps — those are the numbers I'd base anything on. The rep-2 runs and the two PILOT-off runs all landed in the same tight 55.7–57.4 s band, which is the only stretch where wall clock says anything at all.)

The correction. My July-21 table on #441 had workers=1, guard ON at hit 36.5% against PILOT-off's 37.0% — i.e. the guard-ON arm was indistinguishable from no pilot at all. I read that as "the guard removes the eviction thrash". It wasn't: that is the #490 signature, on my own box, in my own data, and I missed it. On the corrected guard the same arm now sits at 49.3–49.7% vs 44.1% for PILOT off — the pilot actually loads. So #497 reproduces as a fix here too (Linux / CPU / QLC), independently of the sm_120 report.

One more thing the table shows, offered as tuning feedback on #497 rather than a complaint: even corrected, the guard still costs a lot of hit rate on a small cache — 49.7% guard-ON vs 67.8% guard-OFF at workers=1, and felt-wait 23.8 s vs 17.5 s, both reproducible across reps. At cap 8/layer essentially every resident is "warm", so eheat >= 2 protects nearly all of them and the correctly-predicted experts never get in. It does not convert to throughput here (55.7 s vs 57.2 s wall, and this box is only ~32% I/O-bound), so I'm not claiming a win — but on a host that really is I/O-bound the remaining hysteresis may be leaving something on the table, and it's cheap to check with PILOT_EVICT_GUARD=0.

Why "park"

On the PR's actual question — does queue depth > 1 pay? — the answer here is still no, on a corrected baseline:

• guard ON: workers=8 costs +1.0 / +1.5 s felt wait vs workers=1 (both reps), for +0 to +1% bytes. Flat-to-slightly-negative.
• guard OFF: workers=8 costs +11.0 / +15.0 s felt wait (both reps), and in the one stretch where wall clock is readable it is the single outlier — 65.9 s against 55.7–57.4 s for every other arm including PILOT off.

And the thing this box cannot show hasn't changed: at cap 8/layer there are no genuinely cold slots for extra queue depth to fill. That still needs enterprise NVMe with a QD8–32 sweet spot, or a big-RAM host where the drive is idle during compute.

So: I'm not asking for review or merge, and I don't want it sitting in your queue re-conflicting every week. It's rebased, green, and byte-identical, so it's a ready-to-build A/B vehicle for whoever has the hardware (#389 / #161). Keep it as a draft if that's useful to you; if you'd rather close it, close it — the branch stays on my fork, #441 links to it, and I'll rebase on demand the day someone volunteers a box. Either way it shouldn't block anything.

On your #403 point

Checked it, and it's cleaner than I expected: every prefetch buffer is slot-owned (s->slab / s->fslab, allocated inside expert_load on the ESlot the worker reserved), so N workers hold N ecache slots in flight, not N extra allocations. RSS stays bounded by ecap regardless of PILOT_WORKERS, and rss_guard already skips eid < 0, which covers the -(eid+2) reservations. The failure mode you described would need per-worker scratch buffers, and there aren't any. Worth knowing independently of this PR.

---

Rebased, reviewed, and measured with AI assistance (Claude). The July-21 correction above is mine, not the tool's — I had the #490 evidence in my own table and read it wrong.

[cdhdt]

One more reason for "park" that I left out above, and it's the one that actually decides it for me — I owe it to you since you offered to review if I took it out of draft.

PILOT_WORKERS=1 is behaviour-identical, not code-identical. Even at the default, the pilot thread goes through the new path: pilot_ring_claim (CAS) instead of the old plain pilot_r store, and the restructured pilot_realload slot selection — visible -(eid+2) reservations, ecn bumped at claim time instead of after the pread, the used = UINT64_MAX loading sentinel, free-slot reuse. The 11/11 sha256 runs prove the output is unchanged; they don't make it the same code.

So merging this doesn't ship a dormant knob — it puts new concurrency code on the path of every PILOT_REAL user, to enable a mode that nobody has yet measured a win from, on any hardware. That trade only becomes right once there's a datapoint on the other side of it. I'd rather not hand you a second "this can't regress" patch so soon after #490 taught me the difference between can't-regress-output and can't-regress-throughput.

The counter-argument is real and I'll state it fairly: merged with default 1, anyone with the right box could run the A/B on a stock build instead of compiling my branch, which is exactly the friction currently blocking this. If you judge that discoverability is worth the code-path change, say so and I'll take it out of draft the same day — you know the maintenance cost of this tree better than I do. Left to me, it stays parked until a number exists.

---

Written with AI assistance (Claude).