Repository navigation
lanes: a queued decision takes a free slot beside the live streams, as a new stream's prompt pass would - #1192
grantoverton wants to merge 21 commits into
Conversation
… 32-over-2 attention WIP: the text path matches an fp32-routing reference at every depth on the 4-bit checkpoint (KL 0.0023 at 48 layers; decode top-1 98-100%); host registration, DFlash2 and images follow.
…t taps and a block of 16 (Ornith's)
…routed block (route, experts) and TF_PROFILE_PROMPT
…sor units' product
…d five of Ornith's eight); selector options by env
…erts' kernel (core: indexed_extra row kernel)
… the reference config and the CUDA family read it
…ne thread Ornith serves /v1/decisions and /v1/systemone while ordinary text submits keep running: info.decides stays off and the hooks appear only when a decider is set. A lane host's runner is owned by its engine thread, so HTTP threads cannot run the Metal passes themselves. engine_api gains a DecisionTask/Decider pair; the host queues a Work per score/decide call, the round and generic driver loops run the oldest once no stream is admitted, filling or active, and shutdown or a stale device fails every waiter instead of sleeping past the thread's end. qwen27's passes run one prompt each on the existing LM head path: score reads the labels' logits at the row after its prompt plus the vocabulary logsumexp, and decide keeps the trunk's slot state once, restoring it per branch so every branch's last-row label logits stand on the same trunk. Pictures and requests past the context window are refused.
… at their final-token positions
…sitions, derived from the render
…s a new stream's prompt pass would A score/decide call no longer waits for an idle engine: the decider's `ready` hook says when a slot is free, and the queued pass runs beside the live rounds — its prompt prefilled in the backend's usual chunks, its label rows read, its slot unbound without a drawn token. A queued request a lane is free for still goes first; a slot that will not fit yet leaves the pass queued (error.Busy) for the next gap. An idle engine runs a decision exactly as before, and a decider without `ready` keeps the idle-only rule. qwen27 binds a synthetic stream (id "decision", max_new 0, drafts off) to a free slot for the pass's length — the same bind an admission makes: buffers grown where needed, held drafts dropped, runner slot reset, drafter context cleared — and `unbind` frees it, promoting a live stream into slot 0 when the pass held the drafter's. The pass never overlaps a planned round, so draft-slot ownership and acceptance are untouched.
ashhart
left a comment
ashhart
left a comment
There was a problem hiding this comment.
Thanks @grantoverton. This sits on the unmerged #1125 stack and on #1189, which was sent back, and it conflicts in 9 files on current main. It keeps #1189's second decision queue (decisions/decided) instead of main's lane_host_score.zig, sets the decider on every qwen27 host (so /v1/decisions and /v1/systemone answer by default), and its wait has no cancel or timeout. Here also the lone driver now yields whenever a decision is queued (changing default scheduling), and `unbind` moves a live stream into slot 0 (the drafter's slot) mid-reply. Once #1189 is reworked on main's score queue as an opt-in, please rebase this onto it; we'd then check that drafted equals plain and concurrent equals solo with a decision in flight.
|
Thanks @grantoverton for the free-slot decision lane. We're closing this with #1189, as our 26 Oct review hasn't had a reply. A version that steps decisions one max_rows piece at a time through main's score queue would be very welcome as a fresh PR. |
Part of #1190. Stacked on #1125, #1189 and #1191; base for review: the #1191 head 55b51c0 — this branch carries its parents' commits until they merge. Review the last two commits: df7f0af (the free-slot decision lane), 7f82411 (lean-check comments).
#1189 ran decisions only at idle points, so under load they waited behind generation (~12 s at 4-way). This is the preemption point it deferred: a queued decision binds a free lane slot as a synthetic stream — its prompt rows join the rounds exactly as a newly admitted stream's do, and it never draws a token — so a decision answers beside live traffic instead of behind it.
What changes
core/engine_api.zig: theDecidervtable growsready— the backend says a free slot exists — andrunmay fail witherror.Busy, which leaves the task at the queue's head for the next gap.core/lane_host_decide.zig:headreturns the queue's front when the engine is idle as before, or when a slot is free and no queued request is waiting for that turn (a decision takes the same turn a new stream would — requests keep priority).waitingreports a decision a free slot could start now, so the lone driver hands its stream to the lane core rather than hold the engine through the whole reply.native/qwen27_served.zig:decideRunmay returnerror.Busy, and the host wires thedecideReadygate intodecider.ready.core/lane_host.zig/lane_rounds.zig: comment lines follow the wider rule (no code change).native/qwen27_decide.zig:readyisb.freeSlot() != null;bindadmits the synthetic stream the way a new stream's admission binds a slot — buffers grown where needed, held drafts dropped, runner state reset — anderror.Busycovers "no slot" and "slot's buffers do not fit yet".unbindfrees it on the caller's defer.native/qwen27_lanes.zig/qwen27_slots.zig/lane_host_lone.zig/lane_rounds.zig: the slot accounting and the lone-driver handoff.core/lane_host_decide_test.zig: coverage for the free-slot path, the busy retry, and request priority.Checks
zig build native -Dcpu=apple_m1 -Dversion=1.0.0passes clean on the head.zig build test test-golden -Dcpu=apple_m1: test-golden 0 unexpected differences (306/319 + 26/26 equal). Theteststep's device-dependent binaries fail on this machine while its GPU is serving a model — the same test binaries fail identically on the base head 30e5dcb (run for comparison), so the failure is inherited, not introduced by this diff.tools/zig/lean_check.py: 444 findings, as on the base (zero added).Receipts (Apple M3 Ultra, 256 GB, macOS 26.4.1, zig 0.17.0; local Ornith-1.5-122B A1-affine 4-bit + DFlash2 drafter)
Not run
As #1189: no new GPU jobs for this cut. The wall-time figure is from the recorded production serve; a CONTRIBUTING-style bench_concurrent receipt on the exact head would be produced at file time if wanted.