You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Are there any plans to implement a safe interface for the CUDA Graph API? Maybe DeviceOperation could be used for some graph-like stuff, but it doesn't seem like a full replacement
The shared cuda-async already has a graph module built on stream capture. Stream capture means: run your launches once while the stream is recording, get a graph back, replay the graph as one unit. The per-launch CPU cost is paid once instead of every iteration (feat(cuda-core): RAII CUDA graph capture and replay #574 measured about 1 us saved per kernel launch).
On DeviceOperation: it is not a replacement, it is the input. A DeviceOperation chain is what gets recorded.
// sketch of the shared cuda_async::cuda_graph API once oxide launchers plug inlet graph = CudaGraph::scope(&stream, |s| {
s.record(kernels.gemm_async(cfg,&x,&w0,&mut h))?;// recorded, not run
s.record(kernels.relu_async(cfg,&mut h))})?;for batch in inputs {
graph.update(memcpy_into(&mut x,&batch))?;// new input, same buffer
graph.launch().sync_on(&stream)?;// replay as one launch}
What is left on my side: oxide's launchers implement a different trait (same job) from the one record accepts today, so the shared crate needs a small bridge for them, plus an oxide example with buffers allocated outside the capture. That pair is what this issue waits on.
Which workload do you want to replay (decode-step loop, iterative solver, something else), so the example matches it?
Thanks for the update. A common base for cuda-oxide and cutile-rs sounds great.
I am more likely to use explicitly defined graphs than captured graphs. No particular workload in mind currently, but I've previously encoded the dependencies between compartmental model components in a CUDA graph allowing for parallel model updates
We are running the dense block of a DeepSeek-V2-Lite decode step (33 launches, one token) as one captured graph on an RTX 3090 with cuda-oxide launchers, MoE blocks next, so this thread is the one we depend on. Two things, one small.
The small one. The only capture entry points today are on the Tile-side cuda_core::runtime::Stream (begin_capture/end_capture, runtime.rs); the simt::CudaStream that #[cuda_module] launchers take has none, so our graph wrapper calls cuStreamBeginCapture_v2 / cuStreamEndCapture through cuda_core::sys on the raw handle. We have the parity patch on a branch — begin_capture / end_capture / is_capturing on simt::CudaStream, calling the same cudarc_shim::stream helpers runtime::Stream uses, with a GPU test (records a host function, checks the status transitions, one node in the graph, end_capture without begin_capture is a driver error, the default stream refuses):
It stops at the CUgraph, exactly where runtime::Stream stops; nothing about instantiation or launch. Before opening a PR: is this a shape you want at all, given the bridge you describe (oxide launchers → cuda_async::CudaGraph::record)? If the plan is for oxide code to capture only through cuda-async, the patch is redundant and we will drop it. If it is useful as the low-level half, does it belong in a PR of its own, or in cutile-rs#257 (which edits the same simt/stream.rs; #270 is the graph half in cuda-async, disjoint from this)?
The data point, in case it helps the API design. On the 3090, one captured graph with touch (empty) kernels costs 0.76 µs per node; our block-0 FFN half went from 8 nodes at 104 µs to 4 fused nodes at 84.5 µs, and only 2.9 µs of that 19.7 µs was node overhead — the rest was small-kernel bodies (norm, quantize, swiglu, residual add) that stopped occupying the GPU alone. So for us the graph is worth having for the per-launch CPU cost, not for the GPU-side gap; the fusion decisions happen elsewhere. What did shape the code around the graph is that capture freezes launch scalars: the KV row index, the live key count, and the MoE expert ids picked by the router all had to move into device buffers read by the kernels so one graph can be replayed at every position and with every routing (update_with() in #270 covers the input-buffer case; the "a kernel reads its own index from a device buffer" case needs nothing from the API, but an example of it would have saved us a day).
All numbers are from our repo's gates on the 3090 under a machine lease; happy to share the harness. The patch and this note were drafted with Claude; the measurements and the review are ours.
Activity
nihalpasham commentedon Sep 3, 2026
Short answer: yes, and it arrives through the shared host crates, not oxide's own.
cuda-bindings/cuda-core/cuda-asyncare being swapped for the crates cutile-rs publishes (Use the shared cutile-rs host crates in cuda-oxide #1102, draft, CI builds pass). That is why feat: fp and cuda-graph support #346 and feat(cuda-core): safe CUDA graph capture and replay #585 were closed rather than merged: a graph layer in the old crates would be deleted with them.cuda-asyncalready has a graph module built on stream capture. Stream capture means: run your launches once while the stream is recording, get a graph back, replay the graph as one unit. The per-launch CPU cost is paid once instead of every iteration (feat(cuda-core): RAII CUDA graph capture and replay #574 measured about 1 us saved per kernel launch).DeviceOperation: it is not a replacement, it is the input. ADeviceOperationchain is what gets recorded.recordaccepts today, so the shared crate needs a small bridge for them, plus an oxide example with buffers allocated outside the capture. That pair is what this issue waits on.Which workload do you want to replay (decode-step loop, iterative solver, something else), so the example matches it?
mtrsl commentedon Sep 6, 2026
Thanks for the update. A common base for
cuda-oxideandcutile-rssounds great.I am more likely to use explicitly defined graphs than captured graphs. No particular workload in mind currently, but I've previously encoded the dependencies between compartmental model components in a CUDA graph allowing for parallel model updates
midagedev commentedon Sep 21, 2026
We are running the dense block of a DeepSeek-V2-Lite decode step (33 launches, one token) as one captured graph on an RTX 3090 with cuda-oxide launchers, MoE blocks next, so this thread is the one we depend on. Two things, one small.
The small one. The only capture entry points today are on the Tile-side
cuda_core::runtime::Stream(begin_capture/end_capture, runtime.rs); thesimt::CudaStreamthat#[cuda_module]launchers take has none, so our graph wrapper callscuStreamBeginCapture_v2/cuStreamEndCapturethroughcuda_core::syson the raw handle. We have the parity patch on a branch —begin_capture/end_capture/is_capturingonsimt::CudaStream, calling the samecudarc_shim::streamhelpersruntime::Streamuses, with a GPU test (records a host function, checks the status transitions, one node in the graph,end_capturewithoutbegin_captureis a driver error, the default stream refuses):NVlabs/cutile-rs@main...midagedev:cutile-rs:feat/simt-stream-capture
It stops at the
CUgraph, exactly whereruntime::Streamstops; nothing about instantiation or launch. Before opening a PR: is this a shape you want at all, given the bridge you describe (oxide launchers →cuda_async::CudaGraph::record)? If the plan is for oxide code to capture only throughcuda-async, the patch is redundant and we will drop it. If it is useful as the low-level half, does it belong in a PR of its own, or in cutile-rs#257 (which edits the samesimt/stream.rs; #270 is the graph half incuda-async, disjoint from this)?The data point, in case it helps the API design. On the 3090, one captured graph with
touch(empty) kernels costs 0.76 µs per node; our block-0 FFN half went from 8 nodes at 104 µs to 4 fused nodes at 84.5 µs, and only 2.9 µs of that 19.7 µs was node overhead — the rest was small-kernel bodies (norm, quantize, swiglu, residual add) that stopped occupying the GPU alone. So for us the graph is worth having for the per-launch CPU cost, not for the GPU-side gap; the fusion decisions happen elsewhere. What did shape the code around the graph is that capture freezes launch scalars: the KV row index, the live key count, and the MoE expert ids picked by the router all had to move into device buffers read by the kernels so one graph can be replayed at every position and with every routing (update_with()in #270 covers the input-buffer case; the "a kernel reads its own index from a device buffer" case needs nothing from the API, but an example of it would have saved us a day).All numbers are from our repo's gates on the 3090 under a machine lease; happy to share the harness. The patch and this note were drafted with Claude; the measurements and the review are ours.
billylindeman commentedon Sep 28, 2026
Is there a timeline for this?