Wgpu Backend - #56
Open
KimHenrikOtte wants to merge 678 commits into
Open
Conversation
Owner
|
@KimHenrikOtte thanks for the PR. This is super exciting, please let me know when you are ready for review. |
EricLBuehler
marked this pull request as ready for review
March 23, 2025 13:33
Owner
|
@KimHenrikOtte is this ready for an initial review? |
* Add varbuilder get_unchecked methods * Add set_device and set_dtype
…ngface#3164) * Add command status semaphore used to ensure metal backend is send/sync * Update metal backend to use encoders directly instead of command buffer for send/sync correctness * Update metal candle-nn ops to use encoders directly instead of command buffer for send/sync correctness * Clippy
* Add sqrt2 as constant for gelu_erf * fix formatting * Use a better erf function
…huggingface#3175) * add initial pool implementation * update implementation to fix breaking tests * add pool based tests to main test suite * fix ordering types to avoid race conditions * add in flight processing * improve error handling and add wait flush test * ensure flush state is returned from entry * rename vars for clarity * address pr comments * update error mapping * update to select entry with max compute count * update tests and set default pool size
…generation (huggingface#3143) * add concat cache; use in qwen3 * update tradeoff desc; resolve unused var warning in concatKV test * update kv-cache concat method description * quant-qwen leverage concatKV; add 8_0 to example main * format 8_0 load * remove trailing , * trailing line * removed unnecessary contiguous calls * Update candle-nn/src/kv_cache.rs remove verbose kv-cache description Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com> * Update candle-nn/src/kv_cache.rs remove verbose kv-cache description Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com> * Update candle-nn/src/kv_cache.rs remove verbose kv-cache description Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com> * Update candle-nn/src/kv_cache.rs consolidate tests Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com> * Update candle-transformers/src/models/quantized_qwen3.rs Large improvements for kv_cache append quantized tensors when in contiguous layout * Update candle-nn/src/kv_cache.rs Since always using contiguous * Update candle-nn/src/kv_cache.rs after contiguous * Update candle-nn/src/kv_cache.rs after contiguous * Update candle-transformers/src/models/quantized_qwen3.rs contiguous called inside append * Update candle-transformers/src/models/quantized_qwen3.rs improves some devices but doesn't hurt others * make k and v continguous post repeat in qwen3 --------- Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
- Corrected errors for the `wgpu_debug` feature - Improved performance measurement documentation and debug file information
* Add cublas_handle api, update safetensors * Add more quantized apis * Make .vscode a .gitignore
* Update CI * I have no clue what was going on with this maturin file, but I don't like it * update cuda container options * Add compute cap to cuda wf * Fix rust toolchain call * update cuda ci runner and bindgen_cuda
…ngface#3200) * fix(cuda): fix integer reduction initialization Replace hardcoded INFINITY/-INFINITY values with type-safe template functions for reduction initialization. Using floating-point infinity values with integer types causes undefined behavior and crashes on newer GPU architectures like Blackwell. The new template specializations use appropriate numeric_limits values for integer types while preserving the original behavior for floating-point types. * fix(cuda): replace limits import with cuda std equivalents --------- Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
* make qwen3 vl config public
- Fixed Clippy warning related to black_box usage - Updated build.rs: functions inside shader files now preserve their source order instead of using a hash-based ordering - fixed errors in wgsl, the shaders now use the wgsl select function, instead of defining its own
* Fix clippy warnings for Rust 1.97 `cargo clippy --fix` for lints promoted to errors on stable 1.97 (useless_borrows_in_formatting, for_kv_map, manual_filter). * Preserve NaN handling in LogitsProcessor::new f64 is only PartialOrd, so clippy's manual_filter suggestion drops NaN where the original comparison kept it. Keep the original expression and allow the lint instead.
Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…uggingface#3771) * feat(metal): register external buffers in the device residency set * Apply suggestions from code review * Address PR review --------- Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…ingface#3736) conv2d_im2col_gemm copies a non-contiguous kernel into a fresh contiguous buffer that starts at index 0, but it built the matmul layout using the original strided kernel's start offset. For a kernel with a non-zero start offset, such as a channel slice of a larger pointwise weight, the matmul read past the materialized buffer and produced incorrect forward values and gradients. Use offset 0 so the layout matches the materialized buffer. Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…uggingface#3586) * fix(qwen3): build causal mask batch-independently (huggingface#3582) `Model::causal_mask` built the additive mask buffer with `tgt*(tgt+offset)` elements (independent of the batch) but shaped it `(b, 1, tgt, tgt+offset)`. For `b > 1` the tensor claims b× the elements actually present, so every batch row but the first reads past the buffer and is masked incorrectly, producing wrong output for batched forwards (the bug disappears at b=1). This is hit on the standard matmul attention path (e.g. Metal), as reported in huggingface#3582. Extract the mask construction into a `build_causal_mask` free function that shapes the mask `(1, 1, tgt, tgt+offset)` and relies on the existing `broadcast_add` to apply it across the batch. Add a CPU regression test asserting the mask is batch-independent and broadcasts to a causal, per-row-identical mask. The same `(b, 1, tgt, ...)` pattern exists in several sibling models (qwen3_moe, quantized_qwen3{,_moe}, glm4_new, quantized_glm4, smollm3, z_image/text_encoder); happy to fix those in this PR or a follow-up. * fix(qwen3): use usize arithmetic for sliding-window mask check Address review feedback on huggingface#3586: the sliding-window check computed `(i + offset) as i64 - j as i64 <= w as i64`, casting to signed to allow a negative result when `j > i + offset`. Rearranged to `j + w >= i + offset` — equivalent, but stays in `usize` with no signed casts and no subtraction underflow. Add a sliding-window regression test; the prior tests only covered the no-window path. * docs(qwen3): trim causal mask comments per review
Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…pendently (huggingface#3879) Follow-up to huggingface#3586 / huggingface#3582, extending that fix to every remaining model with the same pattern and moving the shared logic into `crate::utils` alongside `build_causal_mask`, as huggingface#3437 did for the rectangular mask. Eight models filled the additive mask buffer with `tgt * (tgt + offset)` elements (independent of the batch) but shaped the tensor `(b, 1, tgt, tgt + offset)`. `Tensor::from_slice` does not validate the element count against the shape, since the blanket `impl<S: Into<Shape>> ShapeWithOneHole` discards `el_count`, so the oversized tensor is built silently and only misbehaves at use: - CPU: panics with `range end index N out of range for slice of length M` once the mask is read. - Metal/CUDA: reads past the buffer, so every batch row after the first is masked with garbage and the model returns wrong output. That is what huggingface#3582 reported. Affected: qwen3_moe, quantized_qwen3, quantized_qwen3_moe, glm4_new, quantized_glm4, smol/smollm3, smol/quantized_smollm3 and z_image/text_encoder. Unlike qwen3, where the mask path was gated to CPU-only under the `flash-attn` feature, these gate only on `l == 1`, so the broken mask was built on every multi-token forward on every backend. `utils::build_additive_causal_mask` returns `(1, 1, tgt, tgt + offset)` and is broadcast over the batch by the existing `broadcast_add`. All nine models now call it, including qwen3, which drops the per-model copy added in huggingface#3586. The sliding-window check uses `usize` arithmetic (`j + w >= i + offset`) as in huggingface#3586, avoiding the signed casts. Tests move with the helper: mask shape and batch broadcast, the KV-cache offset case, and the sliding window.
…gface#3845) Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…ce#3855) UgIOp1::metal_fwd calls set_output_buffer and dispatch_threads directly on the CommandsGuard, which only forwards pipeline and label calls since the concurrent-dispatch rework, so --features ug,metal has not compiled since 0.11. Deref to the underlying ComputeCommandEncoder, matching how the other metal call sites use the guard. Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
…huggingface#3862) * fix(metal): surface command buffer errors that occur during execution Re-check the status after waiting and report it. --------- Co-authored-by: ivarflakstad <69173633+ivarflakstad@users.noreply.github.com>
implement load_safetensor function for handling BF16 tensors in WGPU
…ng (huggingface#3892) * fix(flash-attn-v3): allocate the tile count semaphore The causal/local/split path selects DynamicPersistentTileScheduler, which atomicAdds params.tile_count_semaphore. run_mha_v3 memsets the params and never set it, so the kernel dereferenced a null pointer. * fix(flash-attn-v3): launch on the caller's stream run_mha_v3 hardcoded the legacy default stream. candle allocates its streams with cudaStreamNonBlocking, which do not synchronize with it, so the kernel could race the surrounding ops. Matches candle-flash-attn. * fix(flash-attn-v3): size softmax_lse to b*h*seqlen_q The buffer was over-allocated 128x. get_lse_gmem_layout is (b, h, m) and the epilogue bounds its stores by actual_seq_len, so the extra factor was never written. At 16k tokens this reclaims ~3 GB per forward. * fix(flash-attn-v3): stop clobbering the varlen window sizes FlashAttnVarLen overwrote window_size_left and window_size_right with max_seqlen_k whenever they were smaller, so the -1/0 sentinels never survived and is_causal was always 0. Every varlen call ran bidirectional, silently corrupting output for causal models, and sliding windows were widened to the full sequence the same way. * fix(flash-attn-v3): keep varlen off the GQA-packed launchers run_mha_fwd_gqa_ hardcodes FixedSeqLenTraits for every head dim, with no SEQLEN_SWITCH, so it ignores cu_seqlens and indexes a packed batch by seqlen_q and the batch stride. With use_gqa_packing set, varlen therefore read the wrong rows for every sequence past the first.
fixed a native crash in wgpu tests caused by wasm-only logging being used on non-wasm targets made test logging target-aware so wasm uses browser console and native uses normal stdout fixed wgpu test device setup so wasm uses async device creation while native uses the safe native path improved generated wasm test async handling so await inside macro arguments is detected correctly fixed flaky unary_grad expectations by applying lower precision only for wgpu, while keeping strict checks for other backends
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.