[gilby]

Part of #1360. Stacked on #1361 (Clef-Flash: the record encoder, joint head and --clef endpoint); the last commit is new here.

Cloudflare Clef for Qwen3.8-27B ships as a merged checkpoint. Folding the adapter into a 4-bit pack re-quantizes the touched projections and loses most of it, so this applies Clef's adapter at runtime instead, on the base checkpoint the server already serves for chat:

• qwen3_5/cuda_lora.zig: loads adapter.safetensors (the rank-256 / rank-48 LoRA pairs on layers 40–63, checked against each projection's widths) and computes y = bf16(y + bf16(x A^T) B^T) with fp32 sums and a bf16 add. cuda_lora_lane.zig: on a base with the lane kernels (ModelOpt / compressed-tensors: NVFP4, FP8, bf16) both products run on the same bf16 lane GEMM (tensor cores) the base's projections use; each factor pair is packed at startup (A as a (rank, k) projection, B as an (n, rank) one; a rank below a whole 64-group is zero-padded, which adds exact zeros) and the products' scratch for one max_rows piece is reserved with it. Other bases keep the DenseRows path. Either way a row's bits do not depend on the row count or the chunking (tf-cuda-test modelopt: the lane at the adapter's shapes, 600 rows at once byte-equal to pieces of 1–255).
• cuda_forward.zig: six hooks after the adapted projections, no-ops unless a decision pass enables the adapter. The decision pass runs outside the captured CUDA graphs; chat and decode run exactly as before.
• --clef DIR with DIR/adapter.safetensors opens the adapter in the family's open, beside clef: Cloudflare Clef decisions on /v1/systemone through its joint schema head (--clef DIR) #1361's reserved decision cache, so the memory budget counts it; only decision pieces turn it on. It refuses to run inside a stream capture (error.QwenLoraDuringCapture). cuda_lora.zig's test runs on the host from test-qwen35-adapter-host. One server answers chat with the base weights and /v1/systemone with Clef.

Adapter weights

Cloudflare publishes Clef only as merged weights. Build the adapter directory from the public releases with #1366's tools/clef_adapter.py --merged <Cloudflare/clef> --base <Qwen/Qwen3.8-27B> --out <dir> (rank-256 LoRA on the 186 projections of layers 40–63, Cloudflare's joint head files, lm_head_rows.safetensors, a manifest with both revisions) and pass the folder to --clef. Base + adapter vs the merged release: mean KL 3.9e-7, top choice 100%.

Receipts (one DGX Spark, GB10 128 GB, driver 580.178, CUDA 13.0; this branch, re-run on head 472497a5d for the NVFP4 row)

Cloudflare's 60 records / 557 questions against Cloudflare's own bf16 merged-model output; the gate is mean KL ≤ 0.05 and top choice ≥ 95%. Chat is a greedy reply sent before and after the decisions.

base checkpoint	mean KL	top choice	≤1K / 1–4K / 4–10K / >10K top	chat before == after	gate
RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead	0.0085	95.69%	100 / 96.4 / 97.1 / 92.7%	yes	pass
TensorFold/Qwen3.8-27B-MLX-4bit (earlier tree)	0.0139	94.43%	95.5 / 95.2 / 96.3 / 91.1%	yes	top choice just under
merged Clef checkpoint, 4-bit re-quantized (no runtime adapter, for comparison)	0.0947	92.82%		—	fail

The quality depends on the 4-bit base: on NVIDIA's NVFP4 weights the runtime adapter matches the Python engine's Clef (KL 0.0048–0.0074, top choice 95.7–95.9% on EXL3 4bpw / NVFP4 bases).

• Four concurrent decisions equal decisions alone (60/60 records). bench_concurrent --tokens 128 --reps 3 --temperatures 0 --levels 1,4 --alone --serial: 0 unequal, 0 failed.

Decision latency (same 60 records, one at a time, --parallel 4 --context 32768, NVFP4 base)

The first version ran the two products on DenseRows, a CUDA-core kernel; nsys on a 16K-token decision showed it taking 80% of the kernel time. On the base's tensor-core GEMM the adapter's share is small:

tokens	DenseRows p50 / p95 (s)	lane GEMM p50 / p95 (s)
≤1K (22)	1.42 / 2.68	0.63 / 1.07
1–4K (15)	5.80 / 10.94	2.24 / 4.19
4–10K (15)	21.71 / 27.82	8.14 / 10.80
>10K (8)	42.51 / 44.91	16.43 / 17.67

Both builds pass the gate (DenseRows 0.0085 / 95.87%, lane 0.0085 / 95.69%); their probabilities differ in the last bits because the sums run in another order. The packed adapter and its scratch take 0.29 GiB more than the DenseRows copy, counted in the startup line (model 24.33 → 24.62 GiB). Chat replies are the same with both builds.

• zig build test --summary all: 502/502 steps; zig fmt clean; lean_check.py adds no problems over main.

[ashhart]

Thanks @gilby. Chat code and launches are untouched, good. On top of #1361's points: Lora.open loads lazily on the engine thread (an estimated 1-2 GB) outside the memory budget; please open and reserve it at startup. Add a guard that the LoRA never runs during stream capture, and reach cuda_lora.zig's test from a host step. Please point the docs at #1366's rebuild tool rather than a personal re-upload of the adapter.

[gilby]

Thanks. Rebased on #1361's new head, and the adapter commit was reworked (6a97715d6 → 36b0c0079):

• Open at startup, inside the budget: the adapter no longer travels with each request. --clef DIR sets decision_adapter in the engine open request when DIR/adapter.safetensors exists. The qwen3_5 CUDA family's open then loads it (Lora.open) next to clef: Cloudflare Clef decisions on /v1/systemone through its joint schema head (--clef DIR) #1361's reserved decision cache, so the native host's model measurement counts both. Nothing loads on the engine thread afterwards. hiddenFn turns the adapter on for its own piece only (e.lora = self.lora, reset by defer).
• Never inside a capture: cuda_lora.after() returns error.QwenLoraDuringCapture when cuda.graph.captureStatus(e.stream) is not .none.
• Host step for the test: cuda_lora.zig's test is renamed "Qwen adapter tensors: …" and runs from a new test-qwen35-adapter-host step, which zig build test depends on (I broke it once and saw it fail).
• Docs: the body now points at tools: clef_adapter.py rebuilds a Clef runtime adapter from the merged release and its stock base #1366's tools/clef_adapter.py to rebuild the adapter from Cloudflare/clef and Qwen/Qwen3.8-27B. The re-upload is no longer the way in.

Gate on this head (one DGX Spark, GB10, CUDA 13.0, -Dsm=121; RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead + the published adapter, which #1366's tool reproduces, Cloudflare's 60 records):

• Mean KL 0.0085 and top choice 95.87%, the same as before the rework. Gate passes.
• A greedy chat reply is bit-identical before and after the decisions.
• bench_concurrent --tokens 256 --reps 3 --temperatures 0 --levels 1,2 --alone --serial: 0 unequal, 0 failed.
• zig build test: 502/502 steps, and lean adds no problems.

[ashhart]

Review against main e9b69ed (shared notes on #1361). Scoping is right: the adapter is set only for the decision pass's fresh cache and cleared after, and chat never sees it. The blocker is exactness. The runtime computes BF16(y_base + B·BF16(A·x)), with two extra BF16 rounding points (qwen3_5/cuda_lora.zig:87), from factors that are rank-truncated and rounded, so it isn't the merged release's projection; your receipt shows it too (95.87% top-choice agreement on NVFP4, 94.43% on MLX). TensorFold serves a model's own arithmetic rather than an approximation of it, so we can't take this path as is. What would work: serve the merged release's decision weights directly (a second resident copy of the projections the decision pass uses), or an exact representation with a proof of the rounding. The loader should also check the full adapter inventory, base identity and hashes (cuda_lora.zig:42, :56) and use the shared bounded reader (:153, :181).

[gilby]

The adapter's two products now run on the base's bf16 lane GEMM (tensor cores) instead of DenseRows (472497a): on the NVFP4 base, decisions p50 ≤1K 1.42 → 0.63 s, 1–4K 5.80 → 2.24 s, 4–10K 21.71 → 8.14 s, >10K 42.51 → 16.43 s; gate still passes (KL 0.0085, top choice 95.69%), chat unchanged, 0.29 GiB more at startup. On the 9B of #1372 that brings decisions from about 4× to 1.2× the merged release served alone. Tables in the description.

[gilby]

Thanks for the review. Agreed: a rank-truncated bf16 adapter can't reproduce the merged release's arithmetic, and a second resident copy of the decision projections is the opposite of what this PR was for (one copy of the base weights). So I'm closing this rather than turning it into the merged-weights path. #1361 stays the way to serve a Clef release upstream. The lane-GEMM speedup and numbers above stay here for reference.