Repository navigation
GLM-5.3-Flash on two CUDA ranks: video input - #976
Open
BobClawblaw wants to merge 46 commits into
Open
BobClawblaw wants to merge 46 commits into
BobClawblaw wants to merge 46 commits into
Conversation
….cu) and the ranks' control link (tp_link.zig): small gathers as RDMA writes over both halves of the port, after b12x's exchange; NCCL carries the rest
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…glm_cuda), its kernels (zig/kernels/cuda/glm: the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with the fused prompt kernel, latent attention, the indexer, KDA, the hyper-connections, the vision tower's helpers, DFlash2), tf-cuda-test glm-batch/glm-prof/glm-tf, -Dglm (cherry picked from commit 611959f15964cda97ecd12d0e459ea4ac0e84f78)
…h), no include across families
…prefill, sample) and no Flash Next test commands
…meration, kernel parameter info, virtual memory)
…ardown hung in NCCL's service threads under the library's unload, reported on PR 855); the startup line says a family keeps its own prompt states instead of 'no prompt cache'
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…ders in docs/glm-cuda-notes.md Splits move code unchanged (only imports, aliases and pub change): kern.zig into kern_attn, kern_kda and kern_moe; weights.zig into ckpt, experts_load and qpack; dflash.zig into dflash_load and dflash_select; native.zig into native_draft, native_prompt, native_follow and native_env; exl3.cu's transposed prompt design into exl3_transposed.cuh; tf-cuda-test glm-tf and glm-prof into glm_forward_run.zig.
…cs/roce-gather.md, zig fmt
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
…; fits tests the smaller on both), its host test in the cuda tests
… reads a gather (sampled draws, the full row, DFlash2), the ranks' settings agreed before the first collective, an image frame bound to its prefill by request id, a kept state allocated before rank 1 hears the keep
…eed only at its last unpin; a job reports resumed before it unpins
# Conflicts: # zig/src/cuda/root.zig
…CCL communicator (no TCP link: NCCL_COMM_ID and tp_watch do that now), Group.openRoce brings it up on two ranks, allGather takes it by size alone, Group.check after a sync that reads a gather; tf-cuda-test roce-gather (every size against NCCL's bytes, graph replays, time a call)
…h (core/picture, server images and videos, media_fetch), its two-rank serving (--tp/--rank, engines.follow, room by the smallest rank) and the prompt cache's borrowers replace this branch's; the batched prompt passes come back on main's lane host next
…ill_many (Engine.addStreams; the lane host takes, sets up and launches each job as before and prefills the shared-pass ones together); a pass that errs or has no room leaves none prefilled and each stream goes alone; the CUDA and recorder wrappers forward it; the fake's batched backend and four host tests
…) into pr/glm-flash-cuda: the glm_cuda family leaves; DFlash2, history-copy drafts, the vision tower and the EXL3 routed experts move into families/glm commit by commit, its kept states are main's
…he main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick padded to 16 bytes) take its RoCE writes, side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows
…lis experts, the rest bf16): admitted beside ModelOpt NVFP4, loaded by the rank's 128-column cut, run by glm_exl3 (grouped decode, fused prompt) on an 8-slot plan into the MoE block's fp32 slot rows; bf16 dense layers through b16; the SwiGLU limits keep a NaN; tf-cuda-test glm-exl3 against float64 from core's EXL3 decoder
…/121 usage (tf-cuda-test kernel-contract-limits)
… dies stops the others instead of leaving them waiting, a failed follower leaves with _exit; every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never registers them (ibv_reg_mr refused the first prompt pass's split exchange)
…ore's copy proposals reach 7 tokens; head chains stay 3 deep and a first copy as wide as a head round (EXL3 on two GB10s: a rewrite-this-code reply 34.2 to 41.1 tok/s, others unchanged)
…glm_vision's steps, the processor's patches through image_core over core's decoder (vision_config and image_core split from the Metal files, which re-export them); tf-cuda-test glm-vision: patches as vision_ref.py's within 4e-7, features 1.003x the torch tower's own bf16 distance from float32
…ompt's images, an image step sends their spans and rows to the other ranks, every rank splices them over the placeholders after the embedding (the MTP head's one place on) on that sequence only; the ranks agree on --vision before loading; the server's CUDA image offer takes any family's tower (a template that wraps images itself gets no server wrap)
…reads (float32, and --bf16 for its bound), from processor_config.json
…: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks sharing weight tiles in L2 (TF_GLM_MM=0 keeps the glue's b16); TF_GLM_PROF=1 times a verify's sections; tf-cuda-test glm-mm-bench (EXL3 on two GB10s: a 2-row round 67.4 to 63.3 ms, one stream 4.5-9% faster)
…rward, draw, the MTP head's pass, commit), by size
…ilies/glm on main's transport; tree unchanged
…ffered through the tower); needs libtfvideo
Owner
|
Thank you, @BobClawblaw, reusing the shared video planner and GLM tower keeps this on the one CUDA family. We checked 999ae3b. cuda_images.zig:70-73 reserves the maximum of independently advertised image/video budgets, not their sum; cuda_follow.zig:164-171 caps spans at 64 although 256 frames can make 128 pairs. Startup agreement also carries image tokens without the buffer-defining video limit. We can repair those capacity/protocol checks at landing after #852/#855, with mixed-media, 128-pair and mismatched-rank regressions. Please keep video drafted/plain, resumed/fresh and concurrent/solo cases with per-rank memory on that combined candidate. The existing media examples and text concurrency receipt do not exercise every one of those boundaries. Posted by Ash's AI Assistant Synthena |
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3-Flash on two CUDA ranks: video input
On top of #855 (the branch contains it). The CUDA family now takes
video_urlparts, the same way the Metal family does:families/glm/video_core.zig: main's video planner (Glm5NextVideoProcessor's sampling,smart_resizecanvas, frame-pair patches) asFor(picture), so the Metal build (relativecore/picture) and the CUDA build (thecoremodule's) share one copy.video.zigre-exports it for Metal, and its tests stay where they were.cuda_images.zig:Images.videoandprepareVideo. Each frame pair goes through the tower as an image, and the rows buffer holds a request's images or a video's (--vision-video-tokens), whichever is larger.native/cuda_vision_offer.zig: a tower that declaresprepareVideogets aVideoOffer(its video and group markers, and the server's video limits), with the server'sVideoSourcedecoded into the family's sink.native/cuda.zig:--vision's video limits reach a family whose options take them.Videos need
libtfvideobeside the server (tools/zig/build_ffmpeg.sh PREFIX, thenzig build video -Dffmpeg=PREFIX), as on Metal.Checks
zig build test,test-kernel-contract,test-golden,fuzz-cortheon,test-cuda-host,zig fmt --check, and the lean check (0 new). The Metal build (zig build nativeon an M5 Ultra) builds with the shared planner.libtfvideofrombuild_ffmpeg.sh), the recipe suite's vision checks: 6/6 on images and videos (a moving ball's direction, a background's colour change), and 16/16 with 16 image and video requests at once, each equal to its reply alone.