Repository navigation
Serve Ripple-65B-Preview with its DFlash2 drafter - #729
Draft
vvsotnikov wants to merge 52 commits into
Draft
vvsotnikov wants to merge 52 commits into
vvsotnikov wants to merge 52 commits into
Conversation
Step 1 of the Ripple-65B support plan: the production draft attention and dynamic convolution with their geometry as compile-time parameters, built into a variant metallib and driven by a sweep that checks every case against the draft attention test's CPU reference and times each dispatch. Findings (dev/benchmarks/ripple/STEP1-NOTES.md, artifacts under splash-ripple/step1-draft-kernels): the one-80-row group is refused by the device's 32 KB threadgroup limit, the two-40-row-group design checks and runs at per-head efficiency at or above the shipped kernels, MPP's matmul2d needs M or N to be a multiple of 16 (the eight current scores become fp32 dot products), and the 4-tap convolution costs 14% more than the 2-tap one.
The draft attention, convolution, q/k/v preparation, reorder and context commit take their geometry (query heads, the head groups sharing a KV head, splits, ring window, convolution taps) as template parameters. The shipped drafts' instantiations keep their pipeline names and their matmul current scores; Ripple's draft geometry (80 query heads, 4096-slot ring, four-tap convolutions, head groups of five query heads) runs the suffix-named instantiations, whose eight-current-score tiles are fp32 dot products where the MPP matmul's multiple-of-sixteen shape rule needs it. The static threadgroup workspace is compile-time checked against the device's 32 KB.
The draft attention ops match a model's draft shape against the compiled draft variants (metal/abi/DraftAttention.h), carry the matched variant in the plan, and name, size and grid every draft dispatch from it: the kernel suffix, the ring and context-window lengths, the convolution taps' weights, the head groups' split and reduce grids and the partials behind the grouped queries. The shipped drafts' plans are unchanged, and the prototype sweep now drives the production metallib's kernels, whose outputs match the prototype's bit for bit.
Ripple (ricursive; model type ricursive, GGUF architecture ricursive) is a Qwen-hybrid decoder of 80 four-norm blocks: 80 query heads over 8 KV heads of 128 with the Qwen3.5 output gate and per-head q/k norms, 64 of the layers sliding over windows of 1024 keys under RoPE and the other 16 full without RoPE, a value residual that mixes the scaled embeddings into each block's v input by a learned per-layer lambda, a dense SwiGLU of 40960 whose activations clamp at 20, a norm after each FFN, and token embeddings the square root of the hidden size scales. Its GGUF metadata and sliding-window-pattern array are checked, its planner is to land, and loading refuses until its target does. The layer masks its window and RoPE patterns need are wider than a word: they are kv::LayerMask now, two words where a family's layers pass sixty-four, which Ripple's eighty do.
Ripple joins kFamilies: its flat config's sizes and its GGUF metadata's keys are held to the family's values (the two million tokens its config claims against the window the family serves among them), its family values are in the golden, and its fixtures are its release's config and draft configs and its GGUF header's scalars. Its draft's selector keeps more candidates than this build's kernels, which refuse it at the installer's check until they serve them. A DFlash2 draft's layout names the compiled draft variant its config states - query heads, ring window, convolution taps - which its state layout keeps as the ring's slots and its checks read instead of the shipped drafts' constants; the engine sizes by the longest compiled ring. A draft window longer than a prefill chunk's rows rebuilds through the chunk's buffers in chunks, and the model's draft window is the runtime's to name (model::Runtime::draftWindow).
The ricursive planner plans each block's sections around its mixer and FFN: the packed queries holding each query head's output gate after them, which the GGUF keeps beside the queries and one projection takes (Builder's quantizedPair, of one format), the key and value projections, the per-head norms, the block's value-residual lambda and its four norms, and the dense SwiGLU; the head keeps the output norm beside the untied logits. The metadata the planner checks gained the scale, the clamp and the window, and the sliding pattern it reads from the array the installer's copy leaves out. The draft checkpoint plans a draft's convolutions by its layout's taps. The engine tests plan a small ricursive GGUF whose queries and gate are Q4_K and the rest Q8_0, as the release's files mix them, and name each metadata mismatch, the layer patterns' among the arrays; the fixture's tensor table is the release file's own.
The target attention splits and reduces instantiate Ripple's geometry - 80 query heads over 8 KV heads of 128, over the whole history and over a window of the 1024 keys before a row, in both KV formats - with its KV store kernels, and the compiled-attention table serves the geometry; the gate applies as the Qwen3.5's does, in the gate kernel after the attention. The plan tests resolve the geometry's kernels and grids.
Ripple loads through the shared Qwen target loader: each block's image reads its input norm, its mixer (whose fused queries carry the gate rows as the Qwen3.5's do), its post-attention norm, its value-residual lambda - an F32 section the mixer's v input mixes by - its dense SwiGLU and the norms after it; the block sections' image order follows the read. The loader's embedding scale lands at the gather: a gathered row is written multiplied by the family's embedding scale, the shipped families gathering at one, so every reader of the table - the target's input and a draft's mask rows with it - sees the scaled rows the reference computes. The draft checkpoint plans a convolution by its layout's taps, which the preparation and loading tests hold against their goldens.
The residual splits across the fused mixer projection: its v rows scale by sigmoid of the block's lambda at load, and the scaled embeddings project through those rows alone, weighted by the rest, adding where the attention prepares its q, k and v.
The ricursive architecture derives the release's flat config from its header - every size, the FFN's width, the window, the embeddings' scale, the SwiGLU clamp, the RoPE base, the sliding layers as the release's layer_types names them, and the tokens that begin and pad a turn, which is what the header states (its end-of-turn token rides the tokenizer's configuration) - and its tokenizer by the dbrx profile, the splits the DBRX converter reused (llama-3's), which the release's own tokenizer.json splits text by before its byte-level BPE without a normalizer. The tensor screen holds every block to its four norms, its gate beside its queries, its value-residual lambda, and its dense FFN. The install tests hold the derivation to the release: its config's keys and values, each derived value's header key, a sliding pattern of other layers, the derived tokenizer's splits (digit runs of three, a contraction whatever its case, the chat tags at their real IDs), and the tensor screen against the real AD-Q4_K_M file's own tensor table, the fixture's.
The llama.cpp port's build_arch_graph (KLD 0 against the vendor reference) is the block spec: a post-norm architecture whose branch outputs the post-attention and post-FFN norms apply before their residual adds, the residual's norm before the FFN, the value residual mixing sigmoid(lambda) of the normed input with the scaled embeddings the same rows every block reads, the gate reading the normed input, and the clamp per layer. The loader and planner read the four norms by name and stand; the target composes its own blocks over the shared step machinery.
The mixed v input is materialized by an elementwise kernel and the v projection reads it, writing the packed row's v columns through the decode params' output offset the staged kernels already apply: the fused projection carries the queries, their gate and the keys, the v projection the value rows alone, the attention's prepare reads the packed row as every family's does, and the weight bytes stay the same. The mixer's reading becomes the family's own, as the reader's FFN hook already is.
The generic block reader takes the family's mixer reader as it takes its FFN reader. Ripple's reads the fused projection as the queries, their gate and the keys, and the value rows as a projection of their own, which the mixed input projects through; every other family's reader is the shared one, whose fused projection carries the value rows with the queries'.
The mix kernel lerps a block's normed input to the scaled token embeddings by the sigmoid of its learned lambda, one fp32 mix rounded once to bf16, and the host dispatch binds the block's lambda section beside the two inputs. The placed projection wraps a projection's segments at an offset of a wider destination row - the packed row's value columns - whose declared output is the destination's width, so the decode dispatch's stride and the segment's offset write them exactly where the fused projection leaves them. A test holds both against direct CPU references: the mix element for element, and a value projection placed at a packed row beside rows another projection wrote, whose value columns the CPU reference re-projects.
The clamped gated product - silu of the clamped gate times the clamped up, whose clamp commutes with the sums' bf16 rounding as the threshold is exactly representable - is an epilogue of its own (EpClampedUpWithGate), whose threshold the decode and prefill params carry to the kernels and the dispatch fills from the family's value, infinity clamping nothing. The decode staged, register and prefill up kernels instantiate it, and the one-pass gate/up pair runs the unclamped pair alone, whose two-pass fallback the clamped pair takes. The workload checks hold the clamped pair to the decode side and the clamped up pass to the gate scratch, as the unclamped ones' are.
Ripple's target composes the block graph the port states: the attention branch reads the norm of the block input, whose mixer runs the family's split value (the fused projection's rows, the mix of the normed input and the scaled embeddings the target gathers and keeps, and the value projection at the packed rows' tails), and writes its output projection without a residual; the block norms the attention output and adds the block's input; the FFN reads the norm of that sum, whose activations clamp, and its output's norm adds back to the sum; the block's output rows are the draft's captures. The gather writes the scaled rows and copies them to the block input, which the family's own scratch holds across the layers and the prefill passes. The one-pass gate/up pair and the Neural Engine FFN split do not serve the family yet, which the target refuses at the prefill; the loader reads the four norms, whose pre-FFN one the struct had lost.
The proposal samples from the softmax over the candidates the kernels keep, and the acceptance looks the proposed token's probability up in the same list, so a draft trained to keep more candidates serves exactly over the subset the compiled selection holds - the rejection sampling preserves the target's distribution whatever the proposal's reach, which the acceptance rate pays for. The config's candidate count must hold the kernels' (fewer could not select their list), and the draft's geometry guard goes: the variant table serves Ripple's draft attention, whose selector kernels read the shapes their checkpoint's projections give them. The fixture's draft descriptor is valid now, with the Ripple draft's taps, window and query heads held.
Ripple's vocabulary keeps none of the standard tags as tokens: the think, call and close tags the server's machinery matches are text its tokenizer splits. The engine stands without them; families.py opens when the server's detection reads the tags' text.
The post-norm block's pieces are individually tested; the composition's ordering is code review against the pinned graph. The parity harness — a tiny model through the full loading and prefill against a CPU reference — is the numerical gate, which the block-level test's float-path limitation motivated.
A 2-layer tiny model at the family's attention geometry (80 query heads over 8 KV heads of 128) and hidden 256, with a 1-layer draft at the shipped variant's geometry, loads through the full pipeline: the descriptor constructs from the layout, the planner plans the GGUF's images, the preparation writes them, the loader reads them into the target's and the draft's weights, and the target constructs. The fixture's builder writes the GGUF and the checkpoint at the test's dimensions, whose tensors carry the real names and formats the planner and the loader read.
The attention branch's output norm adds the block's input, the FFN's norm reads the sum, the FFN output's norm adds back to the sum, and the block's output is the result — the chain the target's composition encodes, whose adds and norms the test runs through the real dispatches against a CPU reference of the same chain, element for element. The FFN runs as an identity, which isolates the chain's ordering from the projection numerics the quantized kernels' own tests hold.
The parity harness: the tiny Ripple model's prefill runs through the engine's real machinery - the loaded weights, the prefill arena's buffers, a paged K/V page, the target's dispatches and the head - and its logits meet an independent fp64 reference of the pinned block graph at all four positions, within the pipeline's accumulated bf16 rounding (worst 0.008 of a row's largest logit, measured). The reference reads the fixture's GGUF itself, so a loader bug cannot hide in it. The gate caught two defects the per-kernel tests had missed: the compiled attention baked a window of 1024 keys before a row where the architecture's sliding window of 1024 includes its own (the port masks at p1 - p0 >= n_swa), so kernels and the compiled entry now hold 1023; and the geometry's q/k/v prepare and output-gate kernels were never instantiated, the variants test having exercised only the smaller geometry's. The compiled entry's suffix is the bare geometry again, the window riding the variant as the split kernels' names build it. withPrefillArena hands its caller the whole prefill buffer set now, which the ANE FFN's timing and preparation take apart as before.
The real Ripple-65B-Preview (the release's AD-Q4_K_M GGUF and DFlash2 draft) loads through the engine's real machinery, a real prompt prefills, and tokens generate by prefilling the grown sequence again - one pass over the weights per token, which measures the single-token path's cost directly. The request's token ids come from the release tokenizer through the preparation script, which links the HF cache's files into a model root without copying them.
The real Ripple-65B-Preview (the release's AD-Q4_K_M GGUF and DFlash2 draft) loads in four seconds and generates coherent text greedily - 'The capital of France is Paris. The capital of Germany is Berlin' - at about 0.1 s per prefill of a dozen tokens on this machine. The request surfaced four defects the tiny fixture could not, because its blocks' contributions drowned under the residual (its embedding scale put embedding-only logits within tolerance, which the gate now removes: the fixture's scale is 1, and the gate sees the blocks): - the draft loader sized a layer's convolution section with the shipped variant's taps (the ABI's) rather than the layout's, so a four-tap draft's every later section read from mid-data; - the packed q held the queries and then the gates, where the prepare kernel reads each head's gate after its queries: the pair's tensors now interleave per head, as the Qwen3.5 GGUF's fused rows already do; - the fused projection and the placed value projection spanned their own widths, which the dispatches take as the destination's row stride - rows past the first were written at 21504- and 1024-element strides into 22528-element packed rows. Both projections span the packed rows' whole width now, their segments at their column offsets; - the q/k/v prepare and output-gate kernels of the geometry, and the verify-side store, were never instantiated. The variants test now also runs the Ripple geometry end to end (its splits loops take the geometry's tile parts and fused-row strides, and its model of the packed rows holds the gated layout), which is what cleared the attention kernels of the row-1 divergence the parity caught: the kernels were right; the projections' strides were not.
The real model runs through the model::Runtime the engine serves through: its prefill chunks take the request's tokens and page table, and its decode cycles run the draft's proposal and the target's verification end to end. A 32-token greedy request serves at 44.8 tok/s of decode - 2.4x the single-token weight-read ceiling of about 19 tok/s on this machine - with the draft proposing 42 tokens of which the target accepted 25 (59.5%). The harness names each command's page list with the revision the runtime's incremental table writes read.
Ripple joins families.py, the release it serves, with its chat format: its think and call tags are text the tokenizer splits, which startup now checks by encoding each one and decoding it back unchanged, and the output's machinery reads by text where a format's tags are text: - the reasoning's tokens count through the decoded stream, from the opener's text the model writes itself (or the output's start where the prompt opened the block) to the close's first character, or a call's opening where one may follow; - a thinking-end text ends the decoded text with the close; - a thinking budget has no engine cutoff token for such a family, so the reasoning runs to the model's own close and the budget's tokens count; - a grammar's optional think block takes the opener and the close as text literals, and its call rule quotes the call tags where the family's are text (the Qwen3-Coder's, whose parameter tags are text but whose call tags are tokens, keep the bare token form). Ripple's call syntax lays the arguments out as one JSON value between the name's close and the arguments' close, which the grammars and the projector take up next; the syntax carries the mode so that commit's shapes have a home. Its stop tokens are the engine's: <|im_end|> and <|endofprompt|>.
A tool grammar for the JSON-arguments syntax: a strict tool's arguments are one JSON value of its schema, a loose one's any JSON value, and the projector reads the arguments as a JSON value streamed as it is written, the name's close before it and the call's close tag after it framing it. A thinking block's terminal is a suffix rule, whose lexeme runs to the close's first spelling and ends with it - a TEXT terminal and a close literal would not do, for the lexer commits the close's earlier tokens to the TEXT and the close's last token then satisfies nothing - and the TEXT rule of a text-tags format excludes nothing, whose bytes would prune the shared lexer's path through the close, which the suffix's completion needs; the lexer prefers the structural literals over TEXT regardless. A grammar that generates a call, takes the reasoning before it, streams the arguments whole and in pieces and keeps a cut call's arguments, all with the fixture tokenizer whose tags are text, passes.
The mask's countBelow read a layer past the first word's bits as count minus the word's suffix, which holds only when no windowed layer sits at or above the layer: the first family whose window runs past bit 64 read every later layer's region wrong, and the pool's trim moved its pages by the wrong regions, which a page of the Slim runway refused at its windowed layers' K/V. The count walks the whole words below the layer's, then its word's bits below its bit. The one-layer staging layout of a spilled history drops its windowed mask, whose regions a one-region page cannot hold; and the engine's memory plan carries the family's own prefill scratch (the scaled embedding rows the blocks mix back in), which the audit counted against the plan it did not know.
The bench mode splits the served path's costs: prefill chunks of rising length and decode cycles, each timed alone. Long prefill is compute-bound at about 55 effective TFLOPS (428 tok/s at 512 tokens and 420 at 2048); the short chunk's 0.127 s sits 1.7x over the memory floor of one pass over the 42.8 GB of weights. A decode cycle runs 0.097-0.111 s at 386-441 GB/s effective - 70-80% of the machine's bandwidth - and the draft's acceptance (78-88% over the runs) is the bigger lever: every accepted row is bandwidth the target did not spend. The server's earlier TTFT figures were the HTTP path's overheads, not the runtime's.
The bench reads its descriptor from the model root (inspectModelRoot), so the shipped DFlash2 path runs the same protocol: the Qwen3.8-27B assembly prefills at about 60 effective TFLOPS and decodes at 200 tok/s - 0.040 s a cycle, 368 GB/s effective, 8 tokens a cycle with every proposal accepted under greedy. Ripple's kernels reach the same per-byte efficiency (466 GB/s at its best cycle); the served rate differs where the models and drafts differ - 2.9x the weights a pass, a 8.1 GB draft, and a 16-candidate subset proposal whose acceptance caps below the trained selector's.
…tor's per its candidates The draft variant carries the proposal it was trained for - the block's rows a cycle drafts and the selector's kept candidates - matched against the config's block_size and selector_top_k as the variant's keys, whose check replaces the subset's. The draft's kernels instantiate per rows and the selector's per candidates: the shipped drafts' 8-row, 16-candidate kernels keep their names, and Ripple's draft runs 16-row proposals of 32 candidates beside them. The proposal length's host consumers and the decode arena's sizing follow.
The FFN's clamped epilogues plan and dispatch beside the gate/up pair's, whose one-pass form stays unclamped; a family whose SwiGLU clamps takes no Neural Engine FFN split, which computes an unclamped product. The tests allocate under a claim's scope, as every governed allocation now does.
The checkpoint interval, the junction gain and the capture rules take the served model's own draft ring: two of its windows, whole KV pages, within the compiled rings' ceiling, unless a start names an interval - which keeps the shipped drafts' spacing and gives Ripple's 4096-slot ring its own. The gather instantiates an unscaled and a scaled form per format: a multiply flushes the subnormals fp32 keeps, which a bf16 table stores, so a family whose embeddings the hidden size's root scales gathers through the scaled one and every other table through its own. The installer screens Ripple's metadata the way it screens every family's, and its tests keep what the current installer checks. The server's chat format stays without Ripple's text tags, whose reading is the remaining server work: install does not offer a serve that would fail at startup.
The step-1 geometry sweep, its variant metallib and the two real-model debug probes served the prototype; the engine's own tests (the parity and real-model harnesses among them) carry what they validated.
… the ring The server reads Ripple's output in its chat format, whose think and call tags are text the tokenizer splits: the reasoning counts through the decoded stream, a call's arguments are one JSON value between the tags as text, the grammars generate them there, and startup checks each tag by encoding and decoding it back unchanged. The installer derives the family's configuration from its GGUF header and offers the family, whose sampling defaults are Qwen's numbers (the card states none). The option report keeps the options the family's models can use. A restored lane rebuilds its draft ring from up to the draft's own window's rows, which a variant of a longer ring (Ripple's 4096-slot one) holds past the prefill chunk budget: the ring replay's tensors size by the served model's ring, whose window the rebuild's tables take as their bound. The serving eval caught it: an agentic conversation's state restore rebuilt a 2720-row ring against a 2048-row table.
…ling prefill range A restored lane's ring rebuild and a long conversation's checkpoint captures run the draft's context path at up to the draft's own window's rows, which a variant of a longer ring (Ripple's 4096-slot one) holds past the prefill chunk budget: the context, its keys and values and the draft RoPE tables size by the served model's ring, whose window the rebuild's tables take as their bound. The serving eval caught the rebuild reading a 2720-row ring against a 2048-row table. What still fails: a prefill chunk past the first 8192 tokens of a long conversation computes a non-finite final row, which the engine refuses (the request ends; the engine lives). The engine now names the chunk's range; a per-stage finiteness probe in the real-model harness is the next step.
The engine's prefill reports the failing chunk's logical range; a layer cap (SPLASH_PREFILL_LAYER_CAP) and a draft-capture toggle (SPLASH_NO_DRAFT_CONTEXT) bisect it; the checkpoint interval takes SPLASH_CHECKPOINT_INTERVAL. With them the serving eval narrowed its non-finite prefill row to the orchestration after the first checkpoint capture: the target's own chunks are finite alone (the parity-style probe, ripple-long-prefill-probe, runs the failing conversation's every chunk finite), the draft's context capture is not the enterer (SPLASH_NO_DRAFT_CONTEXT still fails), and moving the checkpoint interval past the conversation (SPLASH_CHECKPOINT_INTERVAL) passes. The capture machinery's write path is the remaining dig.
The serving eval's non-finite prefill row enters with the KV window trim on and leaves with it off: the trim at the first checkpoint boundary (a prefill group's edge, at the served model's own interval) corrupts the pages the next command reads. The option reaches the engine for the dig; with the trim off the served conversation passes.
The serving eval isolated its non-finite prefill row to the KV window trim: with the trim on, the first checkpoint boundary's capture corrupts the pages the next prefill command reads (a long conversation's rows 8192..10240, deterministically); with it off the conversation serves. The option reaches the engine through the server, the launcher offers it and the desktop app lists it, and its parsers' and fixtures' tests follow. The trim's region math for an interleaved window mask's second word is the remaining dig.
The draft's 4-bit context window was sized by SPLASH_DRAFT_SLIDING_WINDOW (2048 slots, the shipped drafts'), while the store kernel writes at (start + row) % variant.window — for Ripple's 4096-slot ring, the first conversation whose captured rows wrap past slot 2048 wrote the window's second half past the buffer. The state's lane and cached buffers, the window groups' bound and the oracle's samples size by the ring now. The serving eval caught it: an agentic conversation's 45K-token turn failed its prefill row deterministically, and passes 3x with the fix.
The draft attention plan sizes its workspace and grids by SPLASH_DRAFT_QUERY_ROWS (8, the host's cycle), while the Ripple draft's kernels were instantiated at the trained proposal length (16 rows): the split kernel wrote twice the partials the plan allocated, out of bounds on every draft decode cycle — the arithmetic is exact (heads per group 2 x rows: kernel 32, plan 16). The served acceptance collapsed with conversation depth, which the serving eval measured at 2.2 accepted tokens a cycle. The draft's forward for 8 rows is the trained forward's first 8 rows (the attention is causal over the ring, the convolutions and FFN are per-row), so the kernels run the host's cycle until the 16-row host chain lands; the variant's config rows stay 16.
The served model's draft block (StateLayout draft rows, from the compiled variant's rows) becomes the single source of the rows a cycle drafts and the target verifies: Model grows draftRows/draftProposalTokens/ targetVerifyRows beside draftWindow, and the verify-input and draft-mask kernels take their rows per dispatch (VerifyInputBatchParams and DraftMaskRowsParams carry them) instead of reading the ABI's 8-row constant. The verify sub-pass architecture this enables: the target verifies a 16-row proposal as two 8-row passes through its layers - the second attends the first's rows through their page slots (every verify row is stored before attention) and folds its tape rows as pending - so no target kernel needs a second instantiation and the shipped drafts' cycles are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Serves Ripple-65B-Preview (
ricursiveAI/Ripple-65B-Preview) with its own DFlash2 drafter, end to end through the engine and the server:RippleTarget): the post-norm block graph (each branch's output normed before its residual add, which the Qwen block helpers do not express), the value residual each mixer's v input mixes from scaled embeddings by a learned per-block lambda, the packed q holding each head's output gate beside its queries (a per-head quantized-pair repack), the SwiGLU clamp compiled into the gate/up epilogues, and the 4+1 sliding pattern whose window reads the 1023 keys before a row.ricursive): the metadata the installer derives the config from (embedding_scale,swiglu_clamp, the sliding pattern), the token table's IQ4_XS rows and the byte-level tokenizer whose dbrx-style splits the installer reuses; every tensor the loader does not read is refused by name (problems.addUnread), which caught the release's extras.block_sizeandselector_top_k. The shipped drafts' 8-row, 16-candidate kernels keep their names; Ripple's draft selects over 32 candidates beside them, in head groups of two query heads (five 32-row threadgroups per KV head).ripple-parity): the tiny fixture model's prefill runs through the engine and through an fp64 CPU reference that reads the GGUF itself, every position's logits compared within the pipeline's measured bf16 rounding. It caught two defects the per-kernel tests missed: the sliding window's off-by-one (1023 keys, not 1024), and the geometry's q/k/v prepare kernels never instantiated.model::Runtime(the engine's own path): the draft proposes, the target verifies.families.pyoffers the family, and tool calls, the reasoning channel, and long multi-turn conversations serve end to end.Measurements (M5 Max 128 GB, AD-Q4_K_M target + DFlash2 draft, 42.8 GB of weights)
An agentic serving eval (Terminal-Bench 2.0 through Harbor, terminus-2, the family's default sampling, no agent-side temperature override) holds a 3.4 mean accepted tokens/cycle across ~270 requests → 19.5 tok/s on long generations, and a task solves under the served model (verifier reward 1.0). That eval found three real engine bugs, all fixed here (below); the first fix alone moved acceptance 2.2 → 3.4.
A 160-token reasoning turn opens its think block, reasons and answers short factual prompts exactly ("What is the capital of France?" → "Paris." +
<|im_end|>); long greedy reasoning runs decay into repetition in this harness on both this branch and its base, which the server's sampling defaults are the known fix for. The same harness on the shipped family (Qwen3.8-27B, UD-Q3_K_XL): 200 tok/s decode, every greedy proposal accepted, 368 GB/s effective per cycle — the two families' per-byte kernel efficiency matches (368–466 GB/s), and the decode rate differs where the models differ (2.9× the weights per pass).Serving-found engine fixes
SPLASH_DRAFT_SLIDING_WINDOWwhile the store writesslot = (start + row) % variant.window— the second half of a wrapped ring wrote past its tensors, surfacing as a non-finite prefill row at logical 8192…10240. Both sizings now derive from the served ring's window (draft_context_codes_bytes/draft_context_window_bytes,DraftAttention::contextWindowBytes, the kernel'sWindow).heads-per-group × rowspartials; Ripple's head groups hold two query heads, so 16-row kernels wrote twice the partials an 8-row plan sized — out of bounds on every draft decode cycle, which is what held the eval's acceptance at 2.2. The kernels now run the host's 8-row cycle (the trained forward's first 8 rows: the attention is causal over the ring, convolutions and FFN are per-row) until the host chain below lands.What the suite covers
make test-engine-metal(all kernel suites, including the GGUF projection, preparation, token-gather and draft tests): green. The token gather gains an unscaled and a scaled form per format — a multiply flushes the subnormals fp32 keeps, which a bf16 table stores.make test-python(1614 server/installer tests, including the Ripple family, tokenizer round-trip, and flat-metadata fixtures): green.test-engine-cpu: green except 10enginecheckpoint/replay tests (named below), plus the regenerated goldens (kernel_selectiongains Ripple's rows;family_valuesgains the family's).ripple-real,ripple-serve,ripple-parity,ripple-load,ripple-chain) are opt-in binaries that take the model root and a prompt's tokens;prepare_ripple_real.pylinks the HF cache's files into a model root. A 67,843-token multi-turn conversation replays cleanly with--kv-window-trim.Known remaining
enginecheckpoint/replay tests fail (testSkippedCheckpointKeepsPreviousRecoveryPoint,testPinnedCheckpointSkipsReplacementButNotOrdinaryState,testRepeatedRetriesRollTheRestoredCheckpoint, …): this branch's draft-window economics globalized Ripple's 4096-slot ring (kPrefillCheckpointTokensand the junction gain), which moved every shipped draft's checkpoint spacing; the interval now derives from the served model's own ring, and these ten fixtures need the same treatment. The decode path is unaffected (verified above).Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.