Repository navigation
Opt-in CUDA trunk with Intel Xe2 lm_head split for one Qwen3.5 dense model (existing CUDA and XPU backends) #1421
Description
Proposal: an opt-in split of one Qwen3.5 dense model across an NVIDIA CUDA card and an Intel Xe2 card, using the existing CUDA family engine and the existing XPU backend. No new backend and no change to default behavior.
Scope: the CUDA trunk (embedding, layers, final norm) stays where it is. Only the last projection (the Q4 g64 lm_head) runs on the Intel card through the existing Level Zero runtime and the existing w4_decode GEMM kernel. The head input rows go to the Intel card, the logits come back into the CUDA logits buffer, and the existing sampler reads them unchanged. Off by default, enabled by one environment option. Unsupported checkpoint formats (EXL3, ModelOpt, non-affine, non-4-bit head, hidden size not a multiple of 64) fail at load with a named error. There is no silent fallback to the CUDA head.
Why this shape: it is the smallest real split. It reuses zig/src/xpu (loader, GEMM, FP64 reference helpers) and zig/src/families/qwen3_5/cuda_*, needs no new transport, and keeps CUDA graph capture untouched because the cross-device copy sits outside the captured graph. It is not tensor parallel and not a layer split. It is also not a claim of a speedup.
Validation I intend to attach, per the repository rules:
- Pinned Zig 0.17.0 host tests with no GPU: head packing round trip, bf16 and fp16 conversion edge values, format refusal cases, and an FP64 row-dot-product reference with a stated bound.
- On real hardware, one same-model run with CUDA head against Arc head on identical ids: top-1 must match on every row, relative L2 of the logits within 1 percent, plus one baseline benchmark with the existing HTTP benchmark client and the same checkpoint, reporting transfer-inclusive tokens per second and the output digest. Exact checkpoint, driver and compiler versions in the receipt.
- Until that run exists the branch is untested on hardware and I will say so in the PR.
Overlap I have already checked with atc who: the files this touches are busy. zig/build/xpu.zig is changed by #1330 and #1331. zig/src/families/qwen3_5/cuda_engine.zig and cuda_forward.zig are changed by several open PRs (for example #1137, #1361, #1373, #1355 to #1357, #1418). I will rebase on current main, keep my changes to a new file plus a few lines at the existing head call sites, and re-run atc overlap before any PR. If you would rather have this land after those PRs, say so here.
What I am not proposing here: mixed-vendor tensor parallel (see #1236), the Intel EXL3 work (#1328), or the Intel Qwen MoE work (#1329).
[Joshua agent]
Activity
joshuaswarren commentedon Oct 26, 2026
/atc claim
joshuaswarren commentedon Oct 26, 2026
/atc presence ffc3d83f324aad17a94ab831def843ed25ae1488411b4aa994e873060a3fca84
joshuaswarren commentedon Oct 27, 2026
Status update on the CUDA trunk plus Intel head split (this issue), with what has run and what has not.
Hardware results so far, all on one Linux host with an NVIDIA RTX 5060 Ti and an Intel Arc Pro B60 behind a Thunderbolt dock:
arc: ?*Adapter = nulldefault never ran, and with the Arc head off the field held stale memory thatforward()then used. The fix assignse.arc = null(ande.routed = .{}, the same class on dense models) ininitWithArcbefore anything reads them. A host test guards the order only; it cannot prove behavior, because an Engine needs CUDA.zeInit(ZE_RESULT_ERROR_UNINITIALIZED) after printing "Resizable BAR not detected" for the B60. The kernel log shows the B60 as a small-BAR device (256 MiB CPU-visible of 24 GiB). Intel's compute-runtime tracker (intel/compute-runtime Living Weights: delete one learned fact, recheck the rest, put it back on failure #872, internal GSD-13433) states that on Battlemage with the xe driver the compute runtime is intentionally not enabled in a small-BAR configuration, and that bypassing the check is not enough. The B60 shares the Battlemage G21 die with the card in that report; I did not read the runtime source.So the split as proposed here (head on the Intel card through the existing XPU/Level Zero backend) cannot initialize on this hardware unless the BAR is enlarged. The CUDA-side fix stands on its own. The next design under evaluation is an Intel head through a Vulkan backend (Vulkan on Intel tolerated the small BAR in earlier llama.cpp runs), which would depend on the native Vulkan work in #1420. No PR yet; nothing here is a claim of a working split.
[Joshua agent]
joshuaswarren commentedon Oct 27, 2026
Follow-up with the source facts behind the Level Zero refusal (intel/compute-runtime master, read through the GitHub API; I did not run anything for this and did not read the xe kernel driver's resize logic):
shared/source/os_interface/linux/memory_info.cpp(lines 37 to 40):smallBarDetectedis true when any local-memory region hascpuVisibleSize != 0 && cpuVisibleSize < probedSize. Small BAR therefore means the CPU-visible window is smaller than the region's total VRAM, not smaller than a fixed size. For a 24 GiB card an 8 GiB or 16 GiB BAR would still count as small.shared/source/os_interface/linux/drm_neo.cpp(lines 665 to 669): whenisSmallBarDetected()is true the runtime prints "WARNING: Resizable BAR not detected for device ..." and returns -1 unlessioctlHelper->isSmallBarConfigAllowed()is true. The -1 is thezeInitfailure (ZE_RESULT_ERROR_UNINITIALIZED).shared/source/os_interface/linux/xe/ioctl_helper_xe.h(line 155):IoctlHelperXe::isSmallBarConfigAllowed()returnsfalse; the i915 helper inioctl_helper.h(line 347) returnstrue. In the lines I read no environment variable or debug flag changes this for xe, which matches the tracker reports thatNEO_LOCAL_MEMORY_ALLOCATION_MODEhas no effect and that Intel says bypassing the check is not enough.Consequence for this issue: with the xe driver the Level Zero route needs a CPU-visible BAR as large as the whole VRAM region. On the card used here the Resizable BAR capability lists sizes up to 32 GB, but the host side of this dock limits the window to 8 GiB, so the condition cannot be met on this hardware. The Intel head will be designed against a Vulkan backend instead (Vulkan has no such check; Vulkan on the same card worked on the same small BAR in earlier runs). Everything above concerns master as of today; an older runtime may differ.
[Joshua agent]
ashhart commentedon Oct 27, 2026
Thanks, @joshuaswarren, for running this down to the runtime's small-BAR check. As Level Zero won't start on that card under xe, and the Intel head is moving to a Vulkan design, we're closing this in favour of #1420. Once the Vulkan backend is in, please open the head split as a fresh proposal: it will need to leave the CUDA-only run's tokens unchanged, and to say what it buys, room for a larger model or speed. If the uninitialised-field fix you found applies to main's own engine set-up, a small separate PR with it is welcome.