Skip to content

qwen27: a JointHead side-head scores options on post-norm hidden rows at their final-token positions - #1191

Closed
grantoverton wants to merge 19 commits into
ashhart:mainfrom
grantoverton:ornith-pr2-jointhead-v2
Closed

grantoverton wants to merge 19 commits into
ashhart:mainfrom
grantoverton:ornith-pr2-jointhead-v2

Conversation

@grantoverton

Copy link
Copy Markdown
Contributor

Part of #1190. Stacked on #1125 and #1189 (the score/decide queue); base for review: the #1189 head 227b961 — this branch carries its parents' commits until they merge. Review the last three commits: ee2ed50 (the head format and the decide-pass read), 9a2f2ef (option positions derived from the render), 55b51c0 (lean-check comments).

A --joint-head <dir> serve flag loads a small trained side head and the decide/score passes read its option scores off the trunk's post-final-norm hidden rows — each option scored at its own final-token position instead of label logits. #1154 scores labels off the prompt's last row; this PR's option-final map is a different position contract, derived from the render (prompt_format_version semantics) — flagged here so the convention settles in review.

What changes

  • native/qwen27_joint.zig (new): joint_head_config.json (format bf16-safetensors-v1) gives the head's shape — the trunk's hidden dim, the head's width, and its Linear→GELU residual layers. Weights: proj (dim→width), the residual blocks, out (width→1); bf16 words read from the safetensors file.
  • native/metal.zig: the open path passes --joint-head through to the qwen27 backend, and the module's tests load the head.
  • native/qwen27_served.zig: the host loads and owns the head (its dim checked against the checkpoint), closes it, and hands it to the decide pass.
  • native/qwen27_decide.zig: with a head loaded, labels are option-final token positions rather than label ids. forward prefills the ids in chunks and scores each positions entry on that row's hidden state — for score, positions inside the prompt; for decide, each branch's option-final positions as absolute offsets into the whole sequence (the trunk's length on a branch's rows). Without a head the pass is unchanged (label logits at row 0).
  • server/decisions.zig: optionPositions derives each option's final-token position from the render — options sit one a row between input and the closing inside content.
  • server/model_text.zig: indexAt maps a text span's end offset to the covering token's index.
  • server/systemone.zig: a side head's readout — one group an option, its line's last token's index based on the trunk's length; every option rendered (the template's lone-description case gets its bare answer words).
  • server/cli.zig: --joint-head flag.

Checks

  • zig build native -Dcpu=apple_m1 -Dversion=1.0.0 passes clean on the head.
  • zig build test test-golden -Dcpu=apple_m1: test-golden 0 unexpected differences (306/319 + 26/26 equal). The test step's device-dependent binaries fail on this machine while its GPU is serving a model — the same test binaries fail identically on the base head 30e5dcb (run for comparison), so the failure is inherited, not introduced by this diff.
  • tools/zig/lean_check.py: 444 findings, as on the base (zero added).

Receipts (serving validation B3 with the head loaded, receipts/DECIDE-RECEIPTS.md; Apple M3 Ultra, 256 GB, macOS 26.4.1, zig 0.17.0; local Ornith-1.5-122B A1-affine 4-bit + DFlash2 drafter; stub head dim 3072 / width 1024 / 4 layers)

  • /v1/decisions returns real scores with the stub head on a live serve: choice + yes_no + score all answered, label_mass 1.0, on a 270-token prompt at 4-way and 8-way load. /v1/systemone shares the scorer and the position map but has no recorded run on this stack.
  • Chat traffic completes normally with the head loaded (B3: drafted chats at the recorded rates, ~0.55 acceptance). The head reads only during decision passes, and a decision landing mid-drafted-round leaves the token stream unchanged (qwen27: score and decide passes queued onto the lane host's idle engine thread #1189's integrity receipt); no with/without-head token-sha comparison was run.
  • A trained head of the same format drops in without code changes (position-invariant joint head; option order shuffled, pick ordering stable).

Known limits

Not run

As #1189: no new GPU jobs for this cut; the figures are from the recorded validation of the stack tip. Cold-prefill tables and bench_concurrent comparisons would be re-run at file time on the exact head if wanted.

BobClawblaw and others added 19 commits October 20, 2026 12:36
… 32-over-2 attention

WIP: the text path matches an fp32-routing reference at every depth on the 4-bit checkpoint
(KL 0.0023 at 48 layers; decode top-1 98-100%); host registration, DFlash2 and images follow.
…routed block (route, experts) and TF_PROFILE_PROMPT
…d five of Ornith's eight); selector options by env
…erts' kernel (core: indexed_extra row kernel)
… the reference config and the CUDA family read it
…ne thread

Ornith serves /v1/decisions and /v1/systemone while ordinary text submits keep
running: info.decides stays off and the hooks appear only when a decider is set.

A lane host's runner is owned by its engine thread, so HTTP threads cannot run
the Metal passes themselves. engine_api gains a DecisionTask/Decider pair; the
host queues a Work per score/decide call, the round and generic driver loops run
the oldest once no stream is admitted, filling or active, and shutdown or a
stale device fails every waiter instead of sleeping past the thread's end.

qwen27's passes run one prompt each on the existing LM head path: score reads
the labels' logits at the row after its prompt plus the vocabulary logsumexp,
and decide keeps the trunk's slot state once, restoring it per branch so every
branch's last-row label logits stand on the same trunk. Pictures and requests
past the context window are refused.

@ashhart ashhart left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @grantoverton. This sits on the unmerged #1125 stack and on #1189, which was sent back, and it conflicts in 9 files on current main. It keeps #1189's second decision queue (decisions/decided) instead of main's lane_host_score.zig, sets the decider on every qwen27 host (so /v1/decisions and /v1/systemone answer by default), and its wait has no cancel or timeout. Once #1189 is reworked on main's score queue as an opt-in, please rebase this onto it; we'd then check that drafted equals plain and concurrent equals solo with a decision in flight.

@ashhart

ashhart commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Review is on #1189, covering the whole stack #1189–#1215.

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks @grantoverton for the JointHead idea. We're closing this with #1189, as our 26 Oct review hasn't had a reply. If you pick it up again, a qwen27 implementation of #1361's hidden call, with the head in the server, would be the best fit.

@ashhart ashhart closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants