Repository navigation
qwen27: a JointHead side-head scores options on post-norm hidden rows at their final-token positions - #1191
qwen27: a JointHead side-head scores options on post-norm hidden rows at their final-token positions#1191grantoverton wants to merge 19 commits into
Conversation
… 32-over-2 attention WIP: the text path matches an fp32-routing reference at every depth on the 4-bit checkpoint (KL 0.0023 at 48 layers; decode top-1 98-100%); host registration, DFlash2 and images follow.
…t taps and a block of 16 (Ornith's)
…routed block (route, experts) and TF_PROFILE_PROMPT
…sor units' product
…d five of Ornith's eight); selector options by env
…erts' kernel (core: indexed_extra row kernel)
… the reference config and the CUDA family read it
…ne thread Ornith serves /v1/decisions and /v1/systemone while ordinary text submits keep running: info.decides stays off and the hooks appear only when a decider is set. A lane host's runner is owned by its engine thread, so HTTP threads cannot run the Metal passes themselves. engine_api gains a DecisionTask/Decider pair; the host queues a Work per score/decide call, the round and generic driver loops run the oldest once no stream is admitted, filling or active, and shutdown or a stale device fails every waiter instead of sleeping past the thread's end. qwen27's passes run one prompt each on the existing LM head path: score reads the labels' logits at the row after its prompt plus the vocabulary logsumexp, and decide keeps the trunk's slot state once, restoring it per branch so every branch's last-row label logits stand on the same trunk. Pictures and requests past the context window are refused.
… at their final-token positions
…sitions, derived from the render
ashhart
left a comment
There was a problem hiding this comment.
Thanks @grantoverton. This sits on the unmerged #1125 stack and on #1189, which was sent back, and it conflicts in 9 files on current main. It keeps #1189's second decision queue (decisions/decided) instead of main's lane_host_score.zig, sets the decider on every qwen27 host (so /v1/decisions and /v1/systemone answer by default), and its wait has no cancel or timeout. Once #1189 is reworked on main's score queue as an opt-in, please rebase this onto it; we'd then check that drafted equals plain and concurrent equals solo with a decision in flight.
|
Thanks @grantoverton for the JointHead idea. We're closing this with #1189, as our 26 Oct review hasn't had a reply. If you pick it up again, a qwen27 implementation of #1361's hidden call, with the head in the server, would be the best fit. |
Part of #1190. Stacked on #1125 and #1189 (the
score/decidequeue); base for review: the #1189 head 227b961 — this branch carries its parents' commits until they merge. Review the last three commits: ee2ed50 (the head format and the decide-pass read), 9a2f2ef (option positions derived from the render), 55b51c0 (lean-check comments).A
--joint-head <dir>serve flag loads a small trained side head and thedecide/scorepasses read its option scores off the trunk's post-final-norm hidden rows — each option scored at its own final-token position instead of label logits. #1154 scores labels off the prompt's last row; this PR's option-final map is a different position contract, derived from the render (prompt_format_versionsemantics) — flagged here so the convention settles in review.What changes
native/qwen27_joint.zig(new):joint_head_config.json(formatbf16-safetensors-v1) gives the head's shape — the trunk's hiddendim, the head'swidth, and its Linear→GELU residuallayers. Weights:proj(dim→width), the residual blocks,out(width→1); bf16 words read from the safetensors file.native/metal.zig: the open path passes--joint-headthrough to the qwen27 backend, and the module's tests load the head.native/qwen27_served.zig: the host loads and owns the head (itsdimchecked against the checkpoint), closes it, and hands it to the decide pass.native/qwen27_decide.zig: with a head loaded,labelsare option-final token positions rather than label ids.forwardprefills the ids in chunks and scores eachpositionsentry on that row's hidden state — forscore, positions inside the prompt; fordecide, each branch's option-final positions as absolute offsets into the whole sequence (the trunk's length on a branch's rows). Without a head the pass is unchanged (label logits at row 0).server/decisions.zig:optionPositionsderives each option's final-token position from the render — options sit one a row betweeninputand the closing insidecontent.server/model_text.zig:indexAtmaps a text span's end offset to the covering token's index.server/systemone.zig: a side head's readout — one group an option, its line's last token's index based on the trunk's length; every option rendered (the template's lone-description case gets its bare answer words).server/cli.zig:--joint-headflag.Checks
zig build native -Dcpu=apple_m1 -Dversion=1.0.0passes clean on the head.zig build test test-golden -Dcpu=apple_m1: test-golden 0 unexpected differences (306/319 + 26/26 equal). Theteststep's device-dependent binaries fail on this machine while its GPU is serving a model — the same test binaries fail identically on the base head 30e5dcb (run for comparison), so the failure is inherited, not introduced by this diff.tools/zig/lean_check.py: 444 findings, as on the base (zero added).Receipts (serving validation B3 with the head loaded,
receipts/DECIDE-RECEIPTS.md; Apple M3 Ultra, 256 GB, macOS 26.4.1, zig 0.17.0; local Ornith-1.5-122B A1-affine 4-bit + DFlash2 drafter; stub head dim 3072 / width 1024 / 4 layers)/v1/decisionsreturns real scores with the stub head on a live serve:choice+yes_no+scoreall answered,label_mass1.0, on a 270-token prompt at 4-way and 8-way load./v1/systemoneshares the scorer and the position map but has no recorded run on this stack.Known limits
decidebranch re-prefills the shared trunk per question on the no-head path; with a head the trunk state is kept once and restored per branch (see qwen27: score and decide passes queued onto the lane host's idle engine thread #1189's note — trunk KV reuse is future work either way).d.pictures.len > 0) — image-aware scoring is not claimed.Not run
As #1189: no new GPU jobs for this cut; the figures are from the recorded validation of the stack tip. Cold-prefill tables and bench_concurrent comparisons would be re-run at file time on the exact head if wanted.