Skip to content

ROCm GGUF: split native JIT by operation family - #38

Draft
nekomario28 wants to merge 10 commits into
work/rocm-qwen35moe-plus-pr131-cpu-20260826from
work/rocm-gguf-split-jit-20260826
Draft

ROCm GGUF: split native JIT by operation family#38
nekomario28 wants to merge 10 commits into
work/rocm-qwen35moe-plus-pr131-cpu-20260826from
work/rocm-gguf-split-jit-20260826

Conversation

@nekomario28

Copy link
Copy Markdown
Owner

Internal review/delivery carrier for the gfx1101-validated ROCm GGUF compile fix. Base is the existing ROCm + qwen35moe integration branch. CUDA keeps the original monolithic GGUF extension; ROCm lazily builds dequant, MMVQ, MMQ, MoE-vector and grouped-MoE modules. Physical RX 7800 XT evidence is recorded separately; this PR also includes the latest upstream FlashML-org#131 GGUF CPU/hybrid reachability commit with original author history preserved and a cheap Python routing contract test. Do not merge to fork main as a product release; use this as a review/delivery surface.

Copy link
Copy Markdown
Owner Author

Physical evidence snapshot (RX 7800 XT / gfx1101, torch 2.11.0+rocm7.2, HIP 7.2.26015):

  • the pre-split monolithic native GGUF TU HIPified with 0 unsupported CUDA calls but remained in active AMD-clang compilation beyond the bounded build window;
  • production split dequant: cold JIT 16.006s, Q4_0 pure-Torch oracle PASS;
  • production split MMVQ: cold JIT 16.426s, Q4_0 cosine 0.99995995 / NRMSE 0.008951;
  • production split MMQ: cold JIT 115.677s, Q4_0 cosine 0.99993253 / NRMSE 0.011621;
  • after replacing a HIPify-unsafe wrapper macro with the explicit switch in 93cc7ec, affected-only run 32940588112 / job 98090595262 passed: MoE-vector cold JIT 16.040s, Q4_0 cosine 0.99994171 / NRMSE 0.010843, IQ3_S cosine 0.99998331 / NRMSE 0.005793, grouped-MoE Q4_0 cosine 0.99997103 / NRMSE 0.007668;
  • HIPify reported 0 unsupported CUDA calls in the successful follow-up.

Claim boundary: this proves the five native GGUF operation families compile/link/run numerically on physical gfx1101 through the split ROCm path. It does not yet claim full qwen35moe/Ornith model serving, all quant types numerically, or CUDA performance parity. CUDA remains on the existing monolithic path.

Persistent receipt: project-incubator research/2026-08-26-freetoken-rocm-gguf-operation-split-receipt.md.

@nekomario28
nekomario28 changed the base branch from work/rocm-qwen35moe-integration-20260826 to work/rocm-qwen35moe-plus-pr131-cpu-20260826 August 26, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant