Repository navigation
EXL3 Qwen3.8-Flash-Next on upstream v1.1.0: offload, tp=2, 200k x 2, graphs (draft, to be split) - #50
Closed
BlivionIaG wants to merge 3864 commits into
Closed
BlivionIaG wants to merge 3864 commits into
BlivionIaG wants to merge 3864 commits into
Conversation
…ts serving pin updated
…st 1.0.12 on an M5 Ultra
…m the digest cache, with a GB10 pair's two starts
…bedding, PLE and expert widths and the PLE scale ashhart#606's shared rounds called the embedding, the PLE lookup and the routed experts in their recorded 6-bit, group-32 forms and left out the PLE weight scale, where the single-reply forward reads the widths the loader installs and applies the scale. A plain request never takes the lone driver, so on a checkpoint that is not uniform 6-bit in groups of 32, plain requests and every concurrent stream ran other arithmetic than a lone drafted request. The shared forward now makes the same width-aware calls; a null width keeps the recorded kernel. A host test runs lane rounds through the shared verifier and compares their picks with the single-reply forward on a recording Metal boundary: 4-bit in groups of 64, the recorded 6/32 and mixed routed and shared widths, alone and beside a neighbour, plain and drafted.
…d what its sessions hold
… one 2D copy each way Each call to the common module's experts reads every expert's weights, 1.73 GB a MoE layer a rank for GLM-5.3-Flash at 4 bpw, and GLM's 2048-row prompt pass went through it 64 rows at a time: 32 reads a pass. The adapter also moved each row's picks in and slot outputs out with its own copy, 128 copies a call. Calls now take up to 1024 rows and one cuMemcpy2DAsync moves each side. The module's split-K plan does not depend on the row count, so each row keeps its bits; tf-cuda-test glm-exl3 now runs a 1030-row prompt and checks the rows either side of the calls' boundary against the same row alone. The scratch for GLM-5.3-Flash's prompt pass grows from 34.6 MB to 554 MB a rank.
… for all its picks, with full-K accumulation The common EXL3 expert module now runs prompt passes for verified uniform 4-bit MCG layers with contiguous gate and up projections, model widths divisible by 256 and expert widths by 128; other books, widths and layouts keep the path they had. When gate and up share their input scales, one fp32 Hadamard rotation per input row, rounded to fp16, serves every pick of that row; otherwise each pick keeps its own rotations. Transposed gate/up and down kernels accumulate the whole K in fp32 over bf16 activations, with no split-K partials, so GLM's prompt rows run on the common module with no family kernels of their own. Decode keeps its split-K launches. The three new launches stay pending in the kernel contract until their records are measured. The design follows ashhart#976's shared-row rotations and fused prompt path, and the transposed prompt kernels it credits. Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com> Co-authored-by: MiaAI-Lab <83042094+MiaAI-Lab@users.noreply.github.com>
…e most the grouped router counts The entry's sizes and run took up to 4096 experts, while the grouped router's plan kernels count each expert's pairs in a shared array of 1024 (experts.cu's EMAX), so a pick of expert 1024 or more would index past it. The entry now refuses past grouped.max_experts before any launch; a test takes 1024 at 1, 1024 and 2048 rows and refuses 1025. GLM's scratch sizing shapes its layer at that bound instead of 4096. GLM has 288 experts, so no checkpoint changes path. The serving pin takes grouped.zig's one added line.
…13.0 and 13.2 from every SM's image tf_exl3_prompt_rotate, tf_exl3_prompt_gu and tf_exl3_prompt_down join the measured records with what kernel-contract-limits read from the release compiler's fatbins on a GB10: the rotation 24 registers (20 on sm_120 and sm_121), the gate/up product 128, 126, 126, 124 and 124 across sm_80 to sm_121, the down product 126 on every SM, both products 256 threads with 43,264 static shared bytes. nvcc 13.2's set gives both products 127 on sm_120 and sm_121. The pending list they sat in is gone, so the live audit covers them as it does every other launch.
…-Flash run with a 1M-token prompt saw them
…, with the 27B's cancellation failure and what stays unqualified
…'s shared-expert plans and pair ids
…event, where it waited 20 ms
…ized calls, a thread an expert and 32 columns a warp, same bits Five files took both sides. The contract's coverage paragraph is ashhart#999's, with this line's section on records per nvcc release after it. The registry keeps tf_nemo_topk_wide beside tf_super_topk_few. The host top-k reference is ashhart#999's pickRow, sized by the config's 1,024 experts and 32 picks, which covers the 512 the wide top-k takes. The contract tests import contract_ck beside this line's kernels, load nemotron_latent and ck_experts among this line's images, and add the ck experts after this line's Flash Next, H3 and Drama 3 kernels. The glue tests' rig is ashhart#999's glue_rig.zig, the rig this line had, and the summary names the latent top-k and the paged attention both.
…ds one message, not 16 rows of padding (ashhart#1000) At TP=2 a Super decode step sends each MoE layer a job of one row, but the first message was sized for 16 rows: 165,440 bytes, of which a one-row job uses 10,560. With small_rows = 1 the first message is those 10,560 bytes, and longer jobs still send their rest in a second message. Arithmetic is unchanged: drafted, serial and concurrent replies keep their token hashes.
…oaded at run time, on the shared picture decoder docs/vision.md took both sides: ashhart#1004's WebP paragraph after the JPEG one, and this line's sizing sentence, which names Gemma 4's processor beside GLM5-Next's.
….cu, nemotron_latent.cu and ck.zig
…4 up experts at decode-sized calls kernel-contract-limits read 13.2's fatbins on an x86_64 box: tf_super_topk_few takes 32 registers on sm_120 and sm_121 where 13.0's record has 30, and tf_ck4_up_relu2_few 124 where it has 125; every other SM and the down experts are as recorded. Both join 13.2's set, and the contract's notes say what that set now holds.
…n 4096-row rings, so 16 streams fit a 40,960-token window The branch builds on ashhart#855's later commits, which come in with it. One file took both sides: cuda_forward.zig takes the shared-ready signature and the MTP ring admission, and keeps this line's copy-source owner at both transfers.
… cursor, so a rejected proposal rolls back into held rows The rings restarted at the current pool's first row, which for MTP could lie past the rows a rejected proposal rolls back to. The rebase now starts at the earliest pool such a rollback may revisit. A host test drives the state's MTP admission, rebases and restores through recorded CUDA copies at each ring boundary, fresh and restored, with rewinds of every length.
…bits at load, a cap on RoCE gather shards, and NCCL's cuMem pools off Three files took both sides. cuda_engine.zig keeps this line's load owner, tiled-setting agreement and checkpoint digest, and packs through the tile callback; cuda_ranks.zig takes the union of the width, media, ring and tiled fingerprint fields as layout 9; the tiler moves from cuda_weights.zig to cuda_tile.zig and keeps the load owner's drain before any free. The rank test of a packed linear takes the checkpoint digest as this line's fingerprint does, hashed.
… any allocation, and the packed launches are in the contract TF_ROCE_MAX_KB was read without bounds. A value that does not parse, is zero or passes the RDMA length field is now refused before the ranks allocate. The 22 matmul, tile, prompt and reduction launches of the packed linears are registered with every resource pending, and a host test checks the catalog against each dispatch's shared memory. The engine's state descriptions and options move unchanged to cuda_engine_types.zig, which keeps cuda_engine.zig under 600 lines, and a compile check builds the Linux open body with empty GPU images.
… their chances against measured window costs, 4-bit linears by default Four files took both sides. The CUDA build keeps this line's kernel catalog and adds the drafter's image to cuda_kernels.zig; cuda_engine.zig joins the taps and the drafter with the packed table, the media fingerprint, the load owner, ring forwarding and saved bases; cuda_native.zig adds the drafter's ring bytes beside the image stream bytes and keeps the parsed RoCE cap and native vision limits; the drafter's options go through this line's generic option adapter in native/cuda.zig.
…oad frees what it allocated, and its launches are in the contract Device storage, host allocation and candidate readback each sized the gathered candidates their own way; they now share one layout for 1 to 16 ranks. The loader's tail coordinator owns both books, the scratch and every host table until the load succeeds, so a failed allocation or book read at any point frees the rest. Weight-loading exits drain before device memory is freed, and the engine stays the only image owner when a later drafter step fails. The 11 loaded drafter launches are registered pending their records. The serving pin takes kernels.zig's drafter image.
…ge chats by frame, and enforces structured output on shared sessions (ashhart#986) The branch's eight commits come in as this one. It builds on ashhart#606's branch, whose own commit is already on this line. Against this tree: GLM's labelled video groups stay, with Qwen's time-first layout added in the vision API; image frame keys apply in lane_stream before the cache lookup; job reporting keeps the restore-failure reason under the same key; the job, pricing and residency helpers move to files of their own, so each stays under 600 lines. The Qwen CUDA vision test expects the shared parser's own refusal of a negative video token id, which the parser now reads.
…y and sampled draws, and the vision launches are in the contract A grammar mask could leave a verify row whose allowed logits were all NaN or negative infinity, and the draw then returned a forbidden token. Such a row is now refused; greedy ties keep the lowest allowed id, and a sampled pick is checked before it returns. The two vision pipelines are registered pending their measured limits, with probes in the Metal kernel contract for the mask bits, M-RoPE rows at 1, 7 and 128 and the scalar references. A host test shows that an image's content under another frame cannot resume a cached state.
…past the trained window, and image refusals name the decoder's problem (ashhart#1005) The branch's two commits after ashhart#986 come in as this one. Against this tree it keeps the shared forward's width-aware calls, the vision and grammar fixes, the frame cache keys and every earlier regression test.
…ngs, so a long or failed open cannot change another's The long window's capacity and rotary settings were process-wide, so an ordinary open after a long one, or after a failed long load, kept the enlarged setting. Each run now validates and owns its settings and passes them to Engine.loadConfigured, and allocations, selectors, histories, metadata, admission and both rotary buffers read that engine's own. The compatibility loaders take the trained, unscaled default, and a long engine applies its x4 table to every request it serves. The five changed rotary launches are registered pending their records, with probes in the Metal kernel contract.
The cluster-node test certificate cleared only the sign bit of its random 16-byte serial, so about one serial in 256 began with a zero byte that DER forbids, and OpenSSL refused the certificate with illegal padding. bounds.test.ts failed that way on CI at b188c8e. The first byte is now always 0x40 to 0x7f: 0 of 20,000 generated certificates refused, against 9 of 4,000 before.
… token of thread spawns gone) What: Ngram.rows runs an EXL3 table's gather on the calling thread up to 1024 rows (it spawned items/4 threads: 4 for one token, 12 for a 3-row window, ~150 us each); the EXL3 row decoder unpacks each row's values once and builds every state from them (same bits as `state`, host test over widths 2..8). flashnext-exl3-graph gains `prof` (cuProfilerStart/Stop window for nsys) and `ngram` (gather timing) modes, and graph stats count host prepare time. Why: the gather is on the decode critical path (a graphed step prepares it before launch; an eager step joins it at layer 1 after only layer 0 of GPU work): 3.1 ms a graphed step measured. Tested on par1-cs13: ngram rows of 1 token 0.58-0.69 -> 0.05-0.13 ms, of 3 tokens 2.35 -> 0.15-0.36 ms; tp=2 graphed decode (prof, idle cards): rounds 22.3 -> 20.4 ms, verify windows of 3 rows 29.4 -> 24.5 ms; host prepare 3.1 -> 0.25 ms a step. Output bits unchanged (same rows).
… before sleeping on enqueue What: TF_FLASHNEXT_NCCL_BLOCKING=1 initializes the two-rank communicator blocking (NCCL enqueues on the calling thread); Collective.wait spins ~4096 polls before its 50 us sleeps. Why: profile of the drafted tp=2 server (nsys on both ranks over one 128-token request): almost every kernel ran eagerly (verify windows rarely repeat a graph key), and each eager allreduce cost ~294 us of host time (6801 x cudaStreamWaitEvent = 2.0 s on rank 0); the GPU idled ~half the time. Captured allreduce: 20 us; eager nonblocking: 109 us; eager blocking: 20.6 us (48 in a row: 5.24 -> 0.98 ms). Collective results are the same calls on the same buffers: same bits. Tested on par1-cs13, tp=2 serve gate (two BENCH runs of 128 tokens, ms/token; DRAFTED==UNDRAFTED, BENCH sha 2c5d814cfa80 in every run): eager nonblocking drafts off 27.9/26.8 on 13.2/13.0 eager blocking drafts off 24.0/21.8 on 12.4/11.8 graphs nonblocking drafts off 24.3/22.0 on 13.8/12.8 graphs blocking drafts off 24.1/21.7 on 13.0/12.3 graphs blocking NCCL_PROTO=LL drafts off 23.4/21.1 on 13.2/12.3 Opt-in: a blocking init has no deadline if the peer never joins.
…ojection parts on parallel streams What: (1) cuda_exl3_prefill.chunk: restore `if (pg != .eager) dp.red_n = 0` lost in the exl3-c-main merge of the tp=2 deep prefill: without it a captured layer waited on the reduction events inside the capture (CUDA_ERROR_STREAM_CAPTURE_ISOLATION, graphs then turned off for the process). (2) Prefill layer graphs are opt-in (TF_FLASHNEXT_EXL3_GRAPH_PREFILL=1): bit-equal but 10% slower at tp=2 (a graph waits for every sub-block's all-reduce before the layer, losing the overlap): 8192 rows 1473.6 -> 1324.5 rows/s. (3) cuda_exl3_fork.zig: a multi-part EXL3 projection's parts after the first run on side streams with their own Dense scratch, joined by events (parallel branches in a graph); TF_FLASHNEXT_EXL3_FORK=0 off. Tested on par1-cs13 tp=2: flashnext-exl3-graph fork (3 golden prompts, 32 rounds + windows 3,1,4,2,8,16 in graphs): logits bit-equal serial vs parallel; ms/token 19.51->19.48, 19.92->19.73, 24.24->23.60 (GPU busy per round 19.6 -> 18.8 ms in nsys); prefill 8192 rows eager/graph bit-equal on both ranks.
… state set in the key What: (1) graph keys include the sequence's DeltaNet state set (a drafting stream swaps it with its window's spare set, Draft.settle; a cached graph could otherwise read a swapped-out set once buffers are reused); (2) a window's keep is hashed by contents, not by where the Draft struct sits (Lane values move inside the lanes hashmap: 64 of 95 window steps looked new); (3) a same-shape graph is updated in place only when stale or the cache is full (two live state variants of a shape no longer evict each other); (4) capture at the 2nd sighting; (5) cuda_exl3_recycle.zig: with graphs on, stream capacities round up to 4096 tokens and finished streams' sequence and draft buffers are kept (zeroed on reuse as new ones are) for the next stream, on both ranks; (6) coverage counters by kind/rows (TF_FLASHNEXT_EXL3_GRAPH_LOG) and a key-diff debug log. Why: with drafts on, verify windows almost never replayed (coverage after 96 steps: win2 4 replays/30 eager, win3..7 none). Tested on par1-cs13 tp=2 serve gate, GRAPH=1 NCCL_BLOCKING=1: DRAFTED==UNDRAFTED, sha 2c5d814cfa80; drafts off 23.9/21.4, drafts on 15.2/11.8 ms/token (first bench still pays instantiations; second 11.8 vs eager 11.6). Coverage: round1 312/4/4, win2 20/6/6 (replay/ eager/instantiated).
…experts on the device planner What: cuda_exl3_chain.zig: with graphs on and greedy drafts, Ops.draft's chain (fn_row_best, host read of 4 floats, head step per draft) is captured once per shape: each pick's record stays on the device, fn_exl3_draft_next turns it into the next step's token and upstream's confidence rule into a decision that drives an IF conditional node (cuGraphAddNode + cuStreamBeginCaptureTo- Graph, entry points from libcuda) around the next head step; the host reads the records and decisions once and applies the same rule. The draft head's (MTP layer's) pool joins the device planner (registered at attach), and every head step of up to 16 rows plans on the device (forwardHead hook), so the head's pool no longer takes the host cut. Also: sequences from the KV-placement path are recycled (the placement branch ran before the recycle hook), and Collective.wait spins 3 ms before sleeping (50 us sleeps cost up to ~100 us a wait). Why: drafts-on served round on rank 0 spent ~10 ms between verify windows on the head chain (4 host syncs, ~280 eager launches, host-cut MoE), and the GPU idled there. Tested on par1-cs13 tp=2 serve gate (GRAPH=1 NCCL_BLOCKING=1): DRAFTED==UNDRAFTED, sha 2c5d814cfa80, accepted 90/113 (the same drafts as eager); drafts on 15.1/11.5 ms/token (first bench pays captures), drafts off 23.5/21.2; chains 56 replays / 5 captures.
What: a newly captured decode/window graph (cuda_exl3_graph.run) or draft-chain graph (cuda_exl3_chain.run) is instantiated on a worker thread (context made current there) while that step runs eagerly; later sightings replay once it landed (TF_FLASHNEXT_EXL3_GRAPH_ASYNC=0: inline). Pending entries are joined before eviction or reuse. Why: instantiations (15-35 ms each) sat in the request path; the first drafted bench paid them. Tested on par1-cs13 tp=2 serve gate, same build, BENCH 128 tokens x2, ms/token: GRAPH=1 NCCL_BLOCKING=1: drafts off 23.4/21.1, drafts on 13.7/11.5 (chains 56 replays, 5 eager) NCCL_BLOCKING=1 (eager): drafts off 23.3/21.2, drafts on 12.2/11.6 DRAFTED==UNDRAFTED and sha 2c5d814cfa80 in both; accepted 90/113 in both.
What was built, how it was verified, test measurements, switches, known gaps and next steps.
Record draft PR #50 and distinguish verified TP=2 resident-KV results from the still-unverified paged-KV plus graphs target. Clarify partial graph coverage, performance gates, long-context reference-parity failure, and limitations in the current verification scripts.
BlivionIaG
added a commit
that referenced
this pull request
Oct 19, 2026
Record draft PR #50 and distinguish verified TP=2 resident-KV results from the still-unverified paged-KV plus graphs target. Clarify partial graph coverage, performance gates, long-context reference-parity failure, and limitations in the current verification scripts. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0179oxN37AvEBF1nsVXmybqg
Record the current pushed report commit separately from the runtime code head so PR status remains unambiguous.
Keep the PR handover stable by identifying the tested runtime code head rather than a report commit that changes with each documentation update.
State exact tested prompt lengths, separate resident-KV concurrency from paging, and flag prefill-duration records that do not reconcile to the saved measurements.
Capture expert-parallel reduction event waits as external graph nodes so each ready sub-block can start without waiting for later reductions. Clear prefill graphs before their event handles are destroyed, and key cached graphs by their external wait dependencies. - What: add external event waits for TP2 prefill graph capture and replay; update graph performance notes and architecture report - Why: the prior graph capture serialized TP2 reductions and reduced prefill throughput - Tested: remote Zig CUDA and native builds; TP2 deep-prefill graph gate at 1024/2048/4096-row chunks, bit-identical logits and post-prefill decode; 8192-row chunk OOM fallback excluded
Route host-indexer sequences through the BF16 pager when requested, and calculate allocation and stream admission from one placement-aware layout. Account for the actual engine chunk and one-row cache placeholders; reject unsupported quantized and split-head combinations. Record the remaining graph, overflow, snapshot and concurrent-TP2 gates. Validated: CUDA-fatbin test build and native TP2 server build on par1-cs13; focused host regressions pass; short eager TP2 paging smoke on both cards loaded BF16 pages and drafted output equaled undrafted output. Full graph and long-context concurrency gates remain open.
Update the EXL3 graph and final-report plans with the host-indexer paging fix, the short TP2 BF16 eager smoke, and the remaining graph, overflow, snapshot and concurrency gates. Record the local verify-tail A/B as correctness-preserving but without a demonstrated timing benefit. Validated: Markdown diff check; remote TP2 graph A/B and BF16 paging smoke completed.
Add rank-synchronized graph readiness and sequence-scoped cache cleanup, fail closed for unsupported paged graph/snapshot paths, and validate BF16 paging through deep TP2 prefill and decode. Update the paging report with hardware results and snapshot constraints. Tested: 52/52 host tests; CUDA tests and native build on par1-cs13; TP2 16k BF16-paged prefill/decode graph parity on both GPUs; git diff --cached --check.
Implement pager-aware snapshots that preserve BF16 host history, cached device windows, attention mappings, recurrent/indexer/PLE state, logits, and settled MTP state. Restore validates and commits across TP2 ranks before execution resumes; graph state is invalidated safely. Add strict HTTP gates for two-lane warm, branch, and rewind cache reuse, plus CUDA lifecycle/graph coverage. Update the final report and architecture visualization to distinguish ordinary-context correctness from the deferred 200k x 2 capacity target. Tested: TP2 two-GPU snapshot/graph gate (exact state/logits; 72 prefill and 5 decode replays per rank); TP2 HTTP BF16 paging + MTP + graphs cache gate (4,122/4,139 cached tokens and exact response hashes); snapshot gate unit tests (2/2); Python compile; git diff --check. Paged-acceptance unit tests could not bind localhost in the sandbox; local Zig formatter unavailable. Remote native/CUDA build and device gates passed before documentation-only updates.
Log prefill graph replay, eager, capture, update, and cumulative capture time with the existing decode graph coverage. Extend the strict two-lane snapshot gate with per-request prefill, decode and elapsed timings. Record the 12k BF16-paged TP2 cache/graph gate and the alternating DPLAN-vs-graph result; update the acceptance order to optimize graph speed and cache lifecycle before the deferred 200k x 2 capacity gate. Tested: remote native build; Zig formatting check; Flash Next EXL3 host suite (62/62); Python cache and paging acceptance suites (24/24); two-GPU TP2 graph/cache runs with exact cold, warm, branch and rewind hashes and per-rank prefill replay telemetry. Paired 4k DPLAN/graph timings showed no repeatable speed win.
Expose existing decode prepare and capture timers in per-rank graph coverage logs. Record threshold A/B results for the 12k two-lane BF16-paged cache gate: default capture-after=2 reduces construction overhead relative to capture-after=1, but neither threshold yields a repeatable speedup over DPLAN. Keep the 200k x 2 capacity run deferred until graph speed and cache reuse are optimized. Tested: remote native build; Zig formatting check; two-GPU TP2 BF16-paged 12k chat-history gate with MTP/graphs; exact cold/warm/branch/rewind hashes; observed prefill and decode graph telemetry.
Document the TP=2 16k BF16-paged graph run, expected default-threshold replay behavior, cache parity, and the DPLAN comparison. Keep graph and paging ownership keys unchanged until attention-key churn is profiled. Tested: 16k TP=2 HTTP graph and DPLAN gates on par1-cs13; exact cache hashes, 72 replays per rank, no measured prefill speedup. git diff --check.
Add the TP=2 16k BF16-paged graph replay counters and DPLAN timing comparison. Document that 12k zero replay was expected from the second-sighting capture threshold, and preserve graph ownership keys pending attention-key profiling. Tested: 16k TP=2 HTTP graph and DPLAN gates on par1-cs13; exact cache hashes and 72 replays per rank; no measured speedup. git diff --check.
Keep exact replay keys intact while normalizing only dynamic position and paging-window-origin fields for prefill capture sightings and checked executable-update candidates. Preserve sequence, scratch, paging allocations, shape and event identities; CUDA rejects incompatible topologies and the existing instantiate fallback remains in place. Update both graph reports with the 16k TP=2 gate results. Add a host test proving dynamic origin tolerance and ownership-key strictness. Tested: test-flashnext-exl3-host (62/62); native CUDA build; two-lane 16,040-token TP=2 DPLAN/graph gate with exact cache hashes, 24 graph updates per rank. No speedup measured. git diff --check.
Add a bounded Nsight profiler gate for warmed BF16-paged TP=2 prefill and record the paired eager/graph findings. The results preserve logits parity but do not establish a graph speedup; document the expert planning and transfer boundary as the next optimization target.
Integrate upstream main into the EXL3 draft branch and resolve the Flash Next hook overlaps. Rename the tp=2 head-weight capture to avoid shadowing the half helper, initialize the new paging field in the image-release host fixture, and apply Zig 0.17 formatting to the reported files. Validation: Zig 0.17 native CUDA build passed on par1-cs13; test-cuda-host passed after the fixture fix; full Zig format check passed. The lean checker reports 1,317 new findings against current upstream/main and remains an explicit draft blocker.
BlivionIaG
force-pushed
the
exl3-on-upstream-wip
branch
from
October 20, 2026 07:34
40c6eca to
d3865ec
Compare
BlivionIaG
pushed a commit
that referenced
this pull request
Oct 20, 2026
- The conversation spill is documented in one-line docstrings and comments, and the shutdown save keeps the spill directory within --spill-gib. - The GLM non-fit admission test reads no host memory, so it holds on a DGX Spark too. - CHANGELOG and version. Fixes #38, #45, #50, #52, ashhart#55, ashhart#60.
BlivionIaG
added a commit
that referenced
this pull request
Oct 20, 2026
Record draft PR #50 and distinguish verified TP=2 resident-KV results from the still-unverified paged-KV plus graphs target. Clarify partial graph coverage, performance gates, long-context reference-parity failure, and limitations in the current verification scripts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, not for merge as one piece. This is the working line for serving the EXL3 Qwen3.8-Flash-Next pack on two RTX 5060 Ti 16 GiB cards, built on upstream v1.1.0. It will be split into reviewable PRs later. Full report:
docs/plans/exl3-on-upstream-final-report.md.What it adds
EXL3 weights for upstream's Flash Next CUDA family and server, with hooks in upstream files kept to one line each:
Verified (par1-cs13, idle cards, test measurements)
Known gaps (please read)
zig build test-cuda-hostfails on clean upstream v1.1.0 on macOS too;test-flashnext-exl3-hostpasses. Upstream PR Flash Next CUDA: prompt processing 1.4-1.8x faster on GB10, bit for bit with 1.1.0 ashhart/TensorFold#1046 touches our hook points and was not reviewed.🤖 Generated with Claude Code
https://claude.ai/code/session_0179oxN37AvEBF1nsVXmybqg