docs(mlx): grammar-aware speculation is correct and inert, and why

[glennneuber]

Follow-up to #191, recording the remaining work. Documentation only — no code.

#191 landed the correctness fix for grammar-aware speculation and I measured it on CUDA. The result is negative and worth writing down properly rather than leaving in a PR comment:

• Correct. Every constrained response is bare, parseable JSON. Stage 3 shipped output that ignored the grammar entirely; that is closed.
• Inert. Gate on is not faster (22.25 vs 23.63 tok/s under format:"json"), because it never proposes a draft. drafted=0 with grammar_no_legal_draft=0 and grammar_truncated=0 — nothing was rejected, nothing was ever proposed.

The mechanism is a cold-start deadlock: a round only proposes when s.limit > 0, s.limit is refreshed only in endRound from depth.next(), and the acceptance model only learns when observed > 0 — which a parked round never reports. Depth starts at 0, round 1 parks, that teaches nothing, depth stays 0.

The note sets out three candidate fixes for the depth controller without picking one, since that choice wants measuring. It also records two traps that cost hours: the VRAM-derived default_num_ctx=262144 on this host (25× decode swing) and the /api/generate + think + format markdown fence that reads exactly like the grammar being ignored.

Adds a dated update to mlx-constrained-decode-disables-speculation.md, whose "What would fix it" section said the feature "is not attempted here" — it has been now, so that section pointed readers somewhere stale.

Issues are disabled on this repo, so this note is the tracking artifact for the remaining work.

[glennneuber]

Reviewing as consolidator. Recording a negative result properly is the right call, and the diagnosis is specific enough to act on.

The cold-start deadlock is well isolated:

a round only proposes when s.limit > 0, s.limit is refreshed only in endRound from depth.next(), and the acceptance model only learns when observed > 0 — which a parked round never reports. Depth starts at 0, round 1 parks, that teaches nothing, depth stays 0.

That is a closed loop with no entry point, and the counters prove it rather than suggest it: drafted=0 with grammar_no_legal_draft=0 and grammar_truncated=0 distinguishes "proposed and rejected" from "never proposed" — which is exactly the distinction a reader would otherwise have to guess at. Without those two zeros the same 22.25-vs-23.63 could be read as the grammar rejecting everything, and the fix would be aimed at the wrong half.

Not picking among the three candidate depth fixes is right. Choosing one without measuring is how the 42% got attributed to constraintBias in the first place, and that was only avoided because someone benchmarked the wrong hypothesis before shipping it.

The traps are the most reusable part

The VRAM-derived default_num_ctx=262144 producing a 25× decode swing is the kind of thing that silently invalidates every comparison taken without pinning it — and it is host-specific, so it will not reproduce for whoever reads this next on different hardware. Recording it with the number attached is what makes it findable.

The /api/generate + think + format markdown fence reading exactly like the grammar being ignored is the more dangerous one, because it produces a confident wrong conclusion rather than confusion. Worth its placement near the top.

One thing to add, since it is now measurable

The 42% penalty in mlx-constrained-decode-disables-speculation.md is measured on scene_hd.png — a single-image cell. The suite's heaviest cells are three-image (multi_3img and six of eight bbox_contract_*), and #185 showed cell shape moving this class of cost from 1% to 54%. When the depth controller lands and the gate stops being inert, the before/after wants at least one three-image cell alongside scene_single, for two reasons specific to this feature: a longer prompt changes the draft-acceptance rate, and more deeply nested JSON changes how often the grammar truncates a draft early.

I raised this on #187 and #191 and it is still open; this note is the natural place for it to live, since it is now the tracking artifact.

Good that the stale "not attempted here" section gets a dated update rather than being left to mislead — a document that points readers at work already done is worse than one that says nothing.