Repository navigation
[Perf] Reduce Flash-Next MTP4 verifier batch inflation on V100 - #1211
Draft
yangzhuxinyzx wants to merge 21 commits into
Draft
yangzhuxinyzx wants to merge 21 commits into
yangzhuxinyzx wants to merge 21 commits into
Conversation
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Add opt-in E4M3 target hot storage with native and protected-reader support, preserving FP16 attention operands and the default cache policy. Qualify 32K mean verifier bounds, ordinary decode, quality and near-256K capacity on TP4 V100 16GB. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…-20261014-065933 Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…-20261014-065933 Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com> Assisted-by: Codex
Combine the qualified prefill and decode dependencies in an owned local deployment with MTP4, prefix caching, tools, encoded target hot KV and independent draft capacity. Record the model revision, direct mirror download contract and focused GPU qualification. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Record measured prefill and pure decode, near-256K task completion, prefix hit counters and public API validation. Retain the measured prefix admission limitation and transparent proxy routing correction. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Expose the existing host-indexer cache setting in the launcher while preserving its default. Record destination-native GPU checks, memory qualification, near-256K task completion, matched prefill, prefix reuse and public authentication/stream evidence. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse existing SM70 kernels in bounded verifier tiles, compact recurrent state banks, and score replicated host indexer history once per TP group. Keep acceleration options off by default and retain graph-safe fallbacks. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Bound host indexer staging under its engine owner and reuse original dense/expert decoders for qualified batches. Keep opt-ins off by default, preserve legacy native and prepared-operand interfaces, and document whole-verifier distributions and rejected experiments. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Preserve legacy operators and default policies while adding bounded forty-row cache leases, grouped original down projections, and the full-mesh HC batch path. Fix mixed dense batch fusion and qualify graph/state ownership and paired task behavior. Keep C2 on retained HC tiles after the ordinary-worker acceptance regression; document whole-verifier curves, emitted-token evidence, and the unpromoted shared-expert experiment separately. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse the qualified small-HC peer-load schedule and the unused early lifetime of the existing LoRA shared tile. Preserve canonical arithmetic, peer epochs, old entry points, and retained fallback admission. Qualify the normal CMake artifact with four-rank changed-input graphs and payload/state coverage. Remove the shared-expert experiment after model verification failed to improve whole-batch latency; retain negative evidence and unresolved acceptance variance in the report. Whole-model qualification of this HC update is pending. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Retain protected staging by default and add versioned opt-in banks for tracked local TP4 host indexer writers. Route collisions through a refreshed shared bank and the retained paged reader; reset tags on cache rebinding and stamp writes even when MTP skips selection. Preserve old native schemas. Qualify the normal CMake artifact with 48 GPU, 29 policy/cache and six writer-ordering cases. Whole-model speed and memory gates remain pending; document the latest HC whole-batch curve and failed two-hop screen. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Preserve the resident grid, peer epochs, buffers and reduction order while increasing parallel exchange, row reduction and copy work. Retain prior runtime admission and old native schemas. Normal CMake passes44 policy cases, four-rank payload/state and changed-input graphs. Record the completed versioned-indexer whole-model gates and enhanced real-page/rebind tests, plus the rejected dense row-chunk screen. Full-model qualification of this HC schedule remains pending. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Keep the existing shared-key and GGUF execution families and old operator schemas. Use one grid for uniform banked MTP4 indexer groups and eight independent Q8 output warps per CTA, bound at weight initialization with retained fallback. Remove the legacy MMQ 255-expert cutoff discovered by the 65-row boundary gate; valid high expert IDs previously left output rows uninitialized. Preserve negative padding, numerical order and the MMQ threshold. Normal CMake artifacts pass 90 numerical, changed-graph, state, fallback and fake/AOT tests. Separate normal-artifact screens confirm batch amortization. Whole-model MTP4 qualification is pending; C2/C4/C8 verifier targets remain unmet and no default is promoted. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Compute the retained per-row reduction once and publish its FP32 inverse in unused padded down-projection columns. Reset row epochs after producer writes, before the existing grid rendezvous; pair GPU release and acquire around coherent payload loads. Keep allocations, peer protocol, arithmetic and M17..40 admission unchanged. Normal artifacts pass 45 interface cases and four-rank compiled payload/state tests, including changed widths, graph inputs and wrapped local barrier counters. Reject the two-tile HC and GDN tuning screens after their benefit gates fail. Record the completed b303 full MTP4 matrix and outstanding targets; whole-model qualification of this HC update is pending. Explicitly reject non-byte Q8 packed layouts. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse tagged LL words and dead LoRA padding to halve FP16 partial payload bytes while retaining FP32 sums for secondary inputs. Preserve mixed per-rank input forms, old native entry points and old-binary fallback; bind the optional new operation at initialization. Keep the existing provider, allocations, epochs and canonical arithmetic. Requalify M9..40 execution only for an initialized capacity-forty owner after fixing the512-expert MMQ boundary. Keep small startup owners and fused producers on retained tiles. Normal artifacts pass changed-input graphs, unaligned half views, FP32 overflow/cancellation boundaries, fake registration and four-rank compiled state checks. Whole-model and C2 acceptance gates are pending; all verifier targets remain open except the approximate C1 baseline. Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Reduce whole MTP4 verification cost as TP4 V100-16GB concurrency grows while retaining official sampling, prefix caching, thinking/tools, and 80% GPU / 20% host expert placement. Reuse the existing SM70 attention, GGUF, HC, and resource owners. New policies remain default off; existing native names and schemas remain available.
The branch combines bounded root-only indexer staging, versioned per-layer key reuse, compact recurrent state, forty-row expert leases, mixed dense fusion, grouped expert projections, and whole-batch HC. Later changes batch banked MTP4 indexer groups, group Q8 draft output warps, compute canonical HC RMS statistics once, and losslessly pack FP16 peer partials. A discovered MMQ expert-ID ceiling is fixed: valid IDs above255 previously left512-expert prefill outputs uninitialized.
Integration:
onecat/mainat6f3557c973b4602905037e086fd4acba9b0d867f, dependency composition699c864d3c3a841b8626b657a1c8f236457ab1b5(#1204/#1205/#1206). Current source:99efc51b0eb2356e4a24a211ae4519f108510e5b. Draft; AI assistance: Codex.Test Plan
Build normal extensions from owned source; inspect dependencies in fresh processes and actual TP workers; check numerical, graph, state, padding and fallback boundaries. Measure complete TP4 verifier rounds separately from draft and ordinary emitted-token throughput. MTP4 uses T0.6/p0.95/k20 and natural EOS. Configured capacity is256K; the matched measured case is32K shared-prefix, capacity8, 512-token prefill chunks. Actual256K and production8192-chunk prefill performance are not established.
Test Result
Latest complete model matrix at99ef (whole verifier = aligned per-ordinal TP4 maximum of target forward + sampling + state commit; fenced diagnostic, no division by concurrency):
There are187/210/203/190/193 complete rounds and zero incomplete rounds. The C2/C4/C8 goals of18/22/28ms remain unmet. Relative to a7cf, C2 improves24.508→23.167ms and C8 improves48.937→48.297ms, but C4 regresses31.777→32.618ms. C2 acceptance is2.190 in both ordinary repeats; C8 acceptance is2.335/2.258. Emitted-token throughput does not establish a general win. Renewed C2 native admission is not promoted on kernel timings alone; explicit C2 concurrent task checks and fresh C2/C8 node attribution remain pending.
Ordinary C1/C8 32K retrieval/arithmetic, executable coding, tool-call/reply and natural-EOS checks pass. Interface/GPU cases cover512-expert M0..65 boundaries, E256/257/512/513 IDs, changed-input/routing graphs, fake/AOT, persistent-key ownership, ragged groups and missing-op fallback. HC checks include M5..41 owners, mixed optional secondary inputs across ranks, two-byte-aligned half views, FP32 local sums beyond65504 that cancel before the FP16 boundary, compiled state and local barrier-counter wrap. Peer-epoch wrap remains unqualified. Full changed-source hooks pass.
Normal core/stable/indexer SHA256 prefixes are
4fab324f/ac770d73/ae483cb4. Fresh-process and actual-worker inspection show the normal packaged artifacts without research sidecars or private preload dependencies. Allocation-inclusive kernel evidence remains distinct: Q8 C8-aligned down1.45505→0.34477ms; shared-prefix banked scoring0.17455→0.11694ms per indexer; paired no-secondary C8 HC0.091793→0.071998ms. Independent-prefix scoring is essentially unchanged. Thirteen persistent banks use about52.4MiB on the root. Sampled root allocation16078MiB/free67MiB does not prove transient or256K boundaries.Rejected experiments include concurrent HC MMA tiles (<1% gain), GDN schedule tuning (best6.8%, below its15% gate), compact down, shared-expert batch dispatch, two-hop HC and other paths documented with reasons. New grouped gate/up compaction/reuse remains private research, excluded from services and performance claims.
The campaign report records commands, ownership, coverage and rejected paths. Latest raw artifacts and machine-readable matrix:
/home/ymzx/桌面/1cat-vllm/reports/flashnext-mtp-batch-20261014/hcx-half-packets-model-summary.json. No default or production deployment is promoted.