Repository navigation
A map of my open PRs: groups, dependencies and a suggested order #645
Description
Hi @youssofal,
I have 46 pull requests open on MTPLX, and I can imagine that is hard to keep track of. This issue adds nothing new to review. It is a map: which PRs belong together, which ones build on another, where two PRs touch the same lines, and an order that keeps the merging simple. At the end I ask how you would like to receive this work from now on, so I can adapt to that.
How to read this
- Opt-in means the change sits behind an environment variable that is off by default. Without that variable the server behaves exactly as it does now.
- Builds on #N means the branch contains the commits of #N. Once #N is merged, the GitHub diff of the later PR shows only its own part.
- Overlaps with #N means both PRs change the same lines. Each merges cleanly on its own, but the second one needs a rebase after the first is in. I will do those rebases; just tell me which one you merge first.
- Sizes are lines added, including tests.
1. Small and independent: quick to review
Each of these stands on its own, is small, and fixes something specific.
| PR | What it does | Size |
|---|---|---|
| #576 | Reports the serial scheduler's queue wait as lock_wait_time_s |
+53 |
| #632 | Gemma 4 without MTP no longer crashes on the first forward (shared_kv TypeError) |
+77 |
| #596 | SSD incremental encode: a fingerprint that cannot miss two sign flips in a block | +111 |
| #637 | Session bank: the end-of-turn save keeps the prompt's recurrent checkpoints | +147 |
| #606 | Takes Claude Code's X-Claude-Code-Session-Id header as the session id |
+205 |
| #588 | Recovers an unclosed reasoning block as content when tools are declared | +258 |
| #570 | Scoped reasoning history: keeps the in-round postcommit | +390 |
| #575 | Forge: stamps the verified depth and a served id for forge-local packs | +224 |
| #578 | FR-Spec: binds the pruned draft head on the configured route | +180 |
| #644 | The test suite drops the operator's MTPLX_* switches before collection, so a tuned shell no longer fails over a hundred tests |
+46 |
#570, #575 and #578 were opened before 2.12.1. They only conflict with main in CHANGELOG.md;
2. Session cache and SSD tier
These are about not losing conversations and finding them back quickly.
| PR | What it does | Size | Relation |
|---|---|---|---|
| #608 | An SSD restore builds the cache layout the request asks for | +165 | base for #610 and #630 |
| #609 | Restores an evicted session from SSD, and releases near-prefix candidates the lane does not keep | +1090 | overlaps with #618 and #620 (cold_tier.py), with #629 (generation.py) |
| #618 | Writes an idle conversation to SSD before admission releases it | +519 | overlaps with #609 |
| #620 | A WARNING pressure trim keeps an idle conversation that is not on disk yet | +804 | builds on #618 |
| #628 | One shared common_prefix_len and one reader per session-prefix setting (cleanup, no behaviour change) |
+395 | overlaps with #629, #610, #630, #639 (session_bank.py) |
| #629 | Cuts per-turn host cost of prefix lookups | +528 | overlaps with #628, #609, #610, #630, #639 |
| #631 | Says why a request found nothing to reuse, instead of labelling normal misses as an SSD fault | +340 | dashboard part overlaps with group 7 |
| #646 | Evicts entries this server can never restore first, and removes them after 7 days unused | +949 | overlaps with #622 (cold_tier.py stats) |
| #647 | Telemetry: where restores spend their time, where they land, and how much the cache writes | +1878 | overlaps with #609, #610, #630, #639 (cold_tier.py), #629 (session_bank.py) |
| #648 | Reads SSD restores in parallel into one buffer per tensor: cold restores 2.3 to 2.6x faster | +753 | overlaps with #647 (codec.py) and #646 (docs/server.md) |
| #651 | The streaming spill uses the incremental-encode memo too (it never did, so the memo never hit for long sessions) | +339 | overlaps with #610, #630, #639, #647 (codec.py, cold_tier.py) |
Suggested order: #608, #609, then #618 and #620, then #628 and #629; #646, #647, #648 and #651 at any point, with #651 after #647 (they count the same writes). #647, #648, #651 and #610 touch the same codec and cold-tier functions; whichever goes in first, I rebase the others.
3. Segmented KV cache
A follow-up turn, a fork or a restore continues on the existing history instead of copying it. This is the largest piece of work, opt-in with MTPLX_SEGMENTED_KV.
| PR | What it does | Size | Relation |
|---|---|---|---|
| #610 | Segmented KV cache on Macs with tensor units (M5) | +5050 | builds on #608 |
| #639 | The segment kernel follows the quarter split and the wide verify route | +5590 on its own; small after #610 and #594 | builds on #610 and #594 |
| #630 | Segmented KV on Macs without tensor units (M1 to M4): per-segment attention with a logsumexp merge | +5103 on its own; small after #610 | shares all but the last three commits of #610 |
Suggested order: #610, then #639 (after group 4's #593 and #594), then #630.
4. Verify speed: kernels and scheduling (all opt-in)
These make the verify pass of speculative decoding faster. Each has its own switch.
| PR | What it does | Size | Relation |
|---|---|---|---|
| #593 | Wide verify attention: a flash kernel for 9 to 32-row windows (MTPLX_NAX_FLASH_WIDE) |
+225 | base for #594 |
| #594 | Quarter head-dim split for 2 to 5-row windows (MTPLX_NAX_FLASH_DSPLIT4) |
+439 | builds on #593 |
| #595 | Multi-row 4-bit matmuls for 5 to 32 rows (MTPLX_MULTIROW_QMM) |
+637 | |
| #579 | Eager verify submits the layer graph in chunks | +257 | opened before 2.12.1, CHANGELOG.md conflict only |
| #597 | Prefill merges a final chunk below the fused-SDPA threshold (MTPLX_PREFILL_MIN_CHUNK_ROWS) |
+108 | |
| #607 | Prefill rungs carry the GDN states: lower memory peak on hybrid models (MTPLX_PREFILL_ASYNC_RUNGS_STATES) |
+258 | |
| #642 | GDN prefill on MLX's chunked gated_delta_update (MTPLX_GDN_FAST_PREFILL) |
+545 |
Suggested order: #593, #594, then #595; the other four in any order. (#641 closed 21 Oct: MLX after 0.32.4 covers it, ml-explore/mlx#4682.)
5. Drafting without an MTP head, and FR-Spec
| PR | What it does | Size | Relation |
|---|---|---|---|
| #599 | Context-copy for runtimes without an MTP head (MTPLX_CONTEXT_COPY_AR) |
+721 | |
| #580 | mtplx frspec build ranks a draft vocabulary on your own text |
+1446 | opened before 2.12.1, CHANGELOG.md conflict only |
6. Batch-invariant lane for MoE models (Qwen3.5 and 3.6 MoE)
A chain of four PRs, opt-in, only active on MoE models. All four were opened before 2.12.1 and now conflict with main in generation.py and server/openai.py, so they need a rebase in any case.
| PR | What it does | Size | Relation |
|---|---|---|---|
| #549 | Batch-invariant prefill for MoE models | +1339 | base of the chain |
| #550 | GDN boundaries inside the forward on the invariant lane | +1910 | builds on #549 |
| #558 | One-kernel MoE combine in prefill | +1854 | builds on #549 |
| #559 | Session bank anchors a GDN boundary at the end of the prompt head | +2882 | builds on #550 |
Tell me when you want to look at this chain, and I will rebase it onto main first.
7. Dashboard
| PR | What it does | Size |
|---|---|---|
| #590 | Colours the verify-cycle waterfall bars and labels every row | +161 |
| #598 | Plots live decode samples over the full 5-minute window | +126 |
| #600 | Shows refused requests as refusals in the request log | +271 |
| #617 | Shows the waits before a request's own work, in the flight log and the dashboard | +182 |
| #622 | Shows what is cached in RAM and on SSD instead of empty slot cards | +974 |
| #623 | The Settings tab shows what the server really runs with | +1501 |
| #626 | Time and tok/s at the cursor on the decode chart | +108 |
| #652 | A cancelled request keeps its cache outcome and is shown as cancelled, not as a miss (server and dashboard) | +581 |
Every dashboard PR also commits the rebuilt bundle in mtplx/dashboard/_static/. Those files have a hash in their name, so any two dashboard PRs conflict on them, even when the source changes do not touch each other. After each merge I rebase the next one and rebuild the bundle, so each one applies cleanly. Apart from the bundle, #652 shares lines in RequestLogTable.tsx with #600 and #631.
Suggested overall order
- Group 1, in any order.
- Groups 2 and 4, in the order given there.
- Group 3.
- Groups 5 and 7.
- Group 6.
How would you like to receive this?
I would like to make this easier for you, and I can see two ways:
- Keep the PRs as they are. I rebase each one in the order you merge, so every next one applies cleanly.
- One integration branch. Everything merged on top of main, with the full test suite green, to take in the way you did for 2.12.1.
Are there PRs you do not want at all? Then I will close them, so the list only holds what you are considering.
Thanks for MTPLX. I run it every day, and I would rather help than add to your queue.
Jeroen
Activity