diff --git a/README.md b/README.md index a023ebf..406bf17 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,107 @@ Verify it engaged: `RUST_LOG=info linux-broadcast` logs `ONNX Runtime: CUDA execution provider registered`. A working NVIDIA driver (`nvidia-smi`) supporting CUDA 13 is required. +### New NVIDIA architectures: build a custom CUDA provider + +The prebuilt ONNX Runtime CUDA provider is compiled only for a fixed set of +GPU architectures. If your GPU is newer than that set, session creation can +succeed but the first inference can fail with +`cudaErrorNoKernelImageForDevice`. LinuxBroadcast then shows **GPU unavailable +· CPU** and continues on CPU. Updating only the NVIDIA driver or CUDA runtime +cannot add a kernel image that was omitted from the provider. + +This happened, for example, with Blackwell consumer GPUs (RTX 50 series, +compute capability / SM 120). The following procedure makes a local package +for a newer architecture. It is intentionally an advanced build: the custom +ONNX Runtime static library, its CUDA provider libraries, and the +LinuxBroadcast binary must all come from the same ONNX Runtime version and +build configuration. + +1. Identify the required CUDA architecture. Check the GPU name with + `nvidia-smi`, then look up its compute capability in NVIDIA's CUDA GPU table. + For an RTX 50 card use `120`. If one package must support several GPU + generations, provide a semicolon-separated list such as `89;120`; a build + for only `120` is not a universal replacement for older GPUs. + +2. Find the ONNX Runtime version used by this checkout. Build it once normally, + then inspect `ort-sys`'s build output: + + ```bash + cargo build --release -p linux-broadcast --features cuda + find target -path '*/build/ort-sys-*/output' -type f \ + -exec grep -H 'downloading from' {} \; + ``` + + The URL includes the ONNX Runtime version and CUDA major version. Use an + `ort-artifacts` revision that supports that exact ONNX Runtime release; do + not assume the newest artifact-builder revision is compatible with an older + release. + +3. Build ONNX Runtime from that release with the same CUDA major version and + cuDNN major version as the prebuilt provider, but add your architecture. + The `pykeio/ort-artifacts` static-build wrapper is a convenient way to + produce the layout required by the Rust `ort` crate. Its essential CMake + option is: + + ```bash + -Donnxruntime_USE_CUDA=ON \ + -DCMAKE_CUDA_ARCHITECTURES=120 + ``` + + Start from the wrapper's normal CUDA build options for that release and + change only `CMAKE_CUDA_ARCHITECTURES`. Use an explicit value rather than + `native`, especially in a container: a build container usually cannot query + the host GPU. The resulting install directory must contain all three files: + + ```text + libonnxruntime.a + libonnxruntime_providers_cuda.so + libonnxruntime_providers_shared.so + ``` + +4. Point `ort-sys` at that directory and rebuild LinuxBroadcast. Cleaning + `ort-sys` matters because Cargo otherwise reuses the downloaded prebuilt + ONNX Runtime library: + + ```bash + export ORT_LIB_PATH=/absolute/path/to/onnxruntime/lib + export ORT_SKIP_DOWNLOAD=1 + cargo clean -p ort-sys + cargo build --release -p linux-broadcast --features cuda + ``` + +5. Package the matching provider libraries. `ORT_PROVIDER_DIR` makes the + helper use the custom directory rather than its normal download cache: + + ```bash + cargo install cargo-deb + cargo deb --no-build -p linux-broadcast + ORT_PROVIDER_DIR="$ORT_LIB_PATH" packaging/build-cuda-addon.sh + ``` + + Install both generated packages together (replace the filenames when the + version changes): + + ```bash + sudo apt install --reinstall \ + ./target/debian/linux-broadcast_-1_amd64.deb \ + ./target/debian/linux-broadcast-cuda_-1_amd64.deb + ``` + +6. Verify the provider contains the requested image (requires the CUDA + toolkit's `cuobjdump`) and start the app with logs enabled: + + ```bash + cuobjdump --list-elf "$ORT_LIB_PATH/libonnxruntime_providers_cuda.so" | grep sm_120 + RUST_LOG=info linux-broadcast + ``` + + A successful run logs `ONNX Runtime: CUDA execution provider registered`, + retains the normal **GPU** footer state, and does not emit the CPU-fallback + warning. If it still falls back, preserve the complete ONNX Runtime error; + it usually identifies either a missing runtime dependency or an architecture + mismatch. + ### Option C — Build from source Use this when hacking on the code; the `.deb` is the right choice for everyday use. diff --git a/crates/app/src/ui/footer.rs b/crates/app/src/ui/footer.rs index 3a14307..ce599c9 100644 --- a/crates/app/src/ui/footer.rs +++ b/crates/app/src/ui/footer.rs @@ -24,6 +24,9 @@ impl App { if self.backend == Backend::Gpu { ui.add_space(space::SM); gpu_badge(ui); + } else if let Some(reason) = &self.gpu_fallback_reason { + ui.add_space(space::SM); + gpu_fallback_badge(ui, reason); } ui.add_space(space::MD); sep(ui); @@ -131,6 +134,27 @@ fn gpu_badge(ui: &mut egui::Ui) { }); } +/// Persistent, visible notice that CUDA started but failed on a real model +/// kernel and the pipeline recovered on CPU. The technical error stays in the +/// hover text so the compact footer remains readable. +fn gpu_fallback_badge(ui: &mut egui::Ui, reason: &str) { + egui::Frame::none() + .fill(color::DANGER_SOFT) + .stroke(Stroke::new(1.0, color::DANGER)) + .rounding(4.0) + .inner_margin(egui::Margin::symmetric(6.0, 1.0)) + .show(ui, |ui| { + ui.label( + egui::RichText::new("GPU unavailable · CPU") + .monospace() + .small() + .strong() + .color(color::DANGER), + ) + .on_hover_text(reason); + }); +} + fn sep(ui: &mut egui::Ui) { let (rect, _) = ui.allocate_exact_size(egui::vec2(1.0, 14.0), egui::Sense::hover()); ui.painter().vline( diff --git a/crates/app/src/ui/mod.rs b/crates/app/src/ui/mod.rs index f144f55..a743d43 100644 --- a/crates/app/src/ui/mod.rs +++ b/crates/app/src/ui/mod.rs @@ -223,6 +223,9 @@ pub(super) struct App { /// Execution backend the running pipeline's segmenter uses. Drives the /// footer's "GPU" badge; defaults to `Cpu` (no badge) when not running. backend: Backend, + /// Set only when a CUDA session accepted at startup later fails during + /// inference and the pipeline transparently continues on CPU. + gpu_fallback_reason: Option, error: Option, /// System tray handle. Held for the whole process lifetime. `None` /// only if the OS has no working tray host (logged at install time); @@ -320,6 +323,7 @@ impl App { last_preview_size: None, pipeline_state: PipelineState::default(), backend: Backend::default(), + gpu_fallback_reason: None, error: initial_error, tray, quit_requested: false, @@ -390,6 +394,7 @@ impl App { Ok(p) => { let cmd_tx = p.cmd_sender(); self.backend = p.backend(); + self.gpu_fallback_reason = p.gpu_fallback_reason(); // Push the saved toggle states to the freshly started // pipeline so they survive Stop+Start cycles (model // swaps, camera swaps). @@ -423,6 +428,7 @@ impl App { self.last_preview_size = None; self.pipeline_state = PipelineState::default(); self.backend = Backend::default(); + self.gpu_fallback_reason = None; // The next pipeline starts at gui_preview_active = false; clear // the cache so the next heartbeat re-sends. self.last_gui_preview_active = None; @@ -698,6 +704,8 @@ impl eframe::App for App { // ticks. if let Some(p) = &self.pipeline { self.pipeline_state = p.state(); + self.backend = p.backend(); + self.gpu_fallback_reason = p.gpu_fallback_reason(); } self.drain_preview(ctx); diff --git a/crates/pipeline/src/lazy.rs b/crates/pipeline/src/lazy.rs index 634accc..295bb7c 100644 --- a/crates/pipeline/src/lazy.rs +++ b/crates/pipeline/src/lazy.rs @@ -122,6 +122,14 @@ pub(crate) fn next_state(state: State, demand: Demand, now: Instant) -> State { } } +/// What `spawn_feeder` hands back: the feeder thread, the live execution +/// backend, and the reason for a GPU-to-CPU recovery once one has happened. +type FeederHandles = ( + std::thread::JoinHandle<()>, + Arc>, + Arc>>, +); + /// Spawn the feeder thread. Returns the join handle plus a stop-flag the /// caller (Pipeline) can flip to ask the thread to wind down promptly; /// the thread also exits naturally on `Command::Stop` and on cmd-channel @@ -137,16 +145,18 @@ pub(crate) fn spawn_feeder( state_pub: Arc>, stop_flag: Arc, source_builder: SourceBuilder, -) -> Result<(std::thread::JoinHandle<()>, Backend)> { - let segmenter = Segmenter::from_bytes( - cfg.model, - match cfg.model { - ModelKind::SelfieMulticlass => multiclass_onnx, - ModelKind::Rvm => rvm_onnx, - }, - ) - .context("load segmentation model")?; +) -> Result { + let model_onnx = match cfg.model { + ModelKind::SelfieMulticlass => multiclass_onnx, + ModelKind::Rvm => rvm_onnx, + }; + let segmenter = + Segmenter::from_bytes(cfg.model, model_onnx).context("load segmentation model")?; let backend = segmenter.backend(); + let backend_pub = Arc::new(Mutex::new(backend)); + let gpu_fallback_pub = Arc::new(Mutex::new(None)); + let backend_pub_for_thread = Arc::clone(&backend_pub); + let gpu_fallback_pub_for_thread = Arc::clone(&gpu_fallback_pub); let idle_loader = IdleLoader::new(cfg.width, cfg.height); @@ -169,6 +179,9 @@ pub(crate) fn spawn_feeder( let mut feeder = Feeder { cfg: cfg_for_thread, segmenter, + model_onnx, + backend_pub: backend_pub_for_thread, + gpu_fallback_pub: gpu_fallback_pub_for_thread, compositor: Compositor::new(), smoother: MaskSmoother::new(match cfg.model { ModelKind::Rvm => RVM_ALPHA, @@ -209,12 +222,21 @@ pub(crate) fn spawn_feeder( feeder.run(cmd_rx, watcher_rx, stop_flag); }) .context("spawn lb-feeder")?; - Ok((handle, backend)) + Ok((handle, backend_pub, gpu_fallback_pub)) } struct Feeder { cfg: PipelineConfig, segmenter: Segmenter, + /// Bytes of the selected model, retained so a delayed CUDA failure can + /// rebuild the same model with the CPU execution provider. + model_onnx: &'static [u8], + /// Live backend status published to the GUI. Unlike session creation, + /// CUDA compatibility is only proven after the first inference. + backend_pub: Arc>, + /// One-shot user-facing explanation emitted when GPU inference fails and + /// the feeder continues on CPU. + gpu_fallback_pub: Arc>>, compositor: Compositor, smoother: MaskSmoother, /// Smoothed virtual-PTZ crop driver. Held even when `framing_enabled` @@ -735,16 +757,60 @@ impl Feeder { if do_infer { let t_seg = Instant::now(); - match self - .segmenter - .segment(&frame_rgba, frame_w as usize, frame_h as usize) - { + let segment = + self.segmenter + .segment(&frame_rgba, frame_w as usize, frame_h as usize); + match segment { Ok(mut mask) => { self.smoother.smooth(&mut mask.data); segment_dur = Some(t_seg.elapsed()); self.cached_framing = self.compute_framing(&mask); self.cached_mask = Some(mask); } + Err(gpu_error) if self.segmenter.backend() == Backend::Gpu => { + let technical_reason = format!("{gpu_error:#}"); + let user_reason = format!( + "GPU acceleration was disabled after an inference error; LinuxBroadcast is now using CPU. {technical_reason}" + ); + log::warn!( + "{user_reason} To restore GPU acceleration, install an ONNX Runtime CUDA provider built for this GPU." + ); + + if let Err(cpu_error) = self.segmenter.fall_back_to_cpu(self.model_onnx) { + log::error!( + "GPU inference failed ({technical_reason}); CPU fallback could not start: {cpu_error:#}" + ); + return; + } + if let Ok(mut backend) = self.backend_pub.lock() { + *backend = Backend::Cpu; + } + if let Ok(mut reason) = self.gpu_fallback_pub.lock() { + *reason = Some(user_reason); + } + + // The GPU error left the recurrent state unusable, so + // run this very frame again against the clean CPU + // session. From here on every inference stays on CPU. + match self.segmenter.segment( + &frame_rgba, + frame_w as usize, + frame_h as usize, + ) { + Ok(mut mask) => { + self.smoother.smooth(&mut mask.data); + segment_dur = Some(t_seg.elapsed()); + self.cached_framing = self.compute_framing(&mask); + self.cached_mask = Some(mask); + } + Err(cpu_error) => { + log::error!( + "GPU inference failed ({technical_reason}); CPU retry also failed: {cpu_error:#}" + ); + return; + } + } + } Err(e) => { log::error!("segment: {e:#}"); return; diff --git a/crates/pipeline/src/pipeline.rs b/crates/pipeline/src/pipeline.rs index ae94594..7245df6 100644 --- a/crates/pipeline/src/pipeline.rs +++ b/crates/pipeline/src/pipeline.rs @@ -199,7 +199,8 @@ pub struct Pipeline { state: Arc>, feeder: Option>, stop_flag: Arc, - backend: Backend, + backend: Arc>, + gpu_fallback: Arc>>, } impl Pipeline { @@ -266,7 +267,7 @@ impl Pipeline { let watcher_rx = watcher.events().clone(); // 4. Feeder thread. - let (feeder, backend) = spawn_feeder( + let (feeder, backend, gpu_fallback) = spawn_feeder( cfg, multiclass_onnx, rvm_onnx, @@ -286,6 +287,7 @@ impl Pipeline { stop_flag, _watcher: watcher, backend, + gpu_fallback, }) } @@ -293,10 +295,22 @@ impl Pipeline { self.cmd_tx.clone() } - /// Execution backend the segmenter ended up using (CPU or GPU). - /// Fixed for the lifetime of the pipeline — model/EP are chosen at start. + /// Execution backend the segmenter is currently using (CPU or GPU). + /// A GPU session may fail only on its first kernel launch, in which case + /// the feeder switches it to CPU and this value changes accordingly. pub fn backend(&self) -> Backend { self.backend + .lock() + .map(|backend| *backend) + .unwrap_or_default() + } + + /// Explains a one-way GPU-to-CPU recovery, if one occurred. + pub fn gpu_fallback_reason(&self) -> Option { + self.gpu_fallback + .lock() + .ok() + .and_then(|reason| reason.clone()) } /// Snapshot the current public state. Cheap; backed by an diff --git a/crates/pipeline/src/segmenter.rs b/crates/pipeline/src/segmenter.rs index ab0866a..dd1fad9 100644 --- a/crates/pipeline/src/segmenter.rs +++ b/crates/pipeline/src/segmenter.rs @@ -109,11 +109,34 @@ impl Segmenter { Segmenter::Rvm(inner) => inner.backend, } } + + /// Replace a failed GPU session with a fresh CPU-only session. + /// + /// CUDA can accept a session at construction and only reject it when a + /// kernel runs (for example, when the provider binary has no kernel image + /// for a newer GPU architecture). Rebuilding here lets the caller recover + /// from that delayed failure without stopping the virtual camera. + pub fn fall_back_to_cpu(&mut self, onnx: &[u8]) -> Result<()> { + let replacement = match self { + Segmenter::Multiclass(_) => Segmenter::Multiclass(Box::new(MpInner::new_cpu(onnx)?)), + Segmenter::Rvm(_) => Segmenter::Rvm(Box::new(RvmInner::new_cpu(onnx)?)), + }; + *self = replacement; + Ok(()) + } } impl MpInner { fn new(onnx: &[u8]) -> Result { let (session, backend) = build_session(onnx)?; + Self::with_session(session, backend) + } + + fn new_cpu(onnx: &[u8]) -> Result { + Self::with_session(build_cpu_session(onnx)?, Backend::Cpu) + } + + fn with_session(session: Session, backend: Backend) -> Result { let input_name = session .inputs() .first() @@ -322,7 +345,15 @@ pub struct RvmInner { impl RvmInner { fn new(onnx: &[u8]) -> Result { let (session, backend) = build_session(onnx)?; - Ok(RvmInner { + Ok(Self::with_session(session, backend)) + } + + fn new_cpu(onnx: &[u8]) -> Result { + Ok(Self::with_session(build_cpu_session(onnx)?, Backend::Cpu)) + } + + fn with_session(session: Session, backend: Backend) -> Self { + RvmInner { session, backend, initial_states: [ @@ -334,7 +365,7 @@ impl RvmInner { prev_states: None, input_buf: Vec::new(), last_dims: (0, 0), - }) + } } fn reset(&mut self) { diff --git a/packaging/build-cuda-addon.sh b/packaging/build-cuda-addon.sh index 810ac8b..5feee53 100755 --- a/packaging/build-cuda-addon.sh +++ b/packaging/build-cuda-addon.sh @@ -10,18 +10,27 @@ VERSION=$(grep -m1 '^version' Cargo.toml | sed -E 's/.*"([^"]+)".*/\1/') ARCH=amd64 OUT="target/debian" -# 1. Ensure the CUDA 13 provider libs exist in the ort cache. -ORT_CUDA_VERSION=13 cargo build --release -p linux-broadcast --features cuda +# 1. Find the CUDA 13 provider libs. Set ORT_PROVIDER_DIR to package a custom +# ONNX Runtime build (for example, one compiled for a newer GPU architecture). +# Without it, build normally and take the newest CUDA 13 provider from ort's +# cache. The cache can contain cuda-12 and older cuda-13 builds, hence the +# ABI check below. +if [ -n "${ORT_PROVIDER_DIR:-}" ]; then + DIR="$ORT_PROVIDER_DIR" +else + ORT_CUDA_VERSION=13 cargo build --release -p linux-broadcast --features cuda + DIR=$(find "$HOME/.cache/ort.pyke.io" -name 'libonnxruntime_providers_cuda.so' \ + -exec sh -c 'readelf -d "$1" 2>/dev/null | grep -q libcublasLt.so.13 \ + && printf "%s %s\n" "$(stat -c %Y "$1")" "$(dirname "$1")"' _ {} \; \ + | sort -rn | head -1 | cut -d' ' -f2-) +fi -# 2. Locate the CUDA-13 provider libs. The cache can hold several build-hash -# dirs (cuda-12 and older cuda-13 builds), so filter to cuda-13 (NEEDED -# libcublasLt.so.13) and take the NEWEST by mtime — the build just produced -# above — to avoid shipping an ABI-mismatched .so. -DIR=$(find "$HOME/.cache/ort.pyke.io" -name 'libonnxruntime_providers_cuda.so' \ - -exec sh -c 'readelf -d "$1" 2>/dev/null | grep -q libcublasLt.so.13 \ - && printf "%s %s\n" "$(stat -c %Y "$1")" "$(dirname "$1")"' _ {} \; \ - | sort -rn | head -1 | cut -d' ' -f2-) -[ -n "$DIR" ] || { echo "error: CUDA 13 provider libs not found in ort cache" >&2; exit 1; } +[ -n "${DIR:-}" ] && [ -f "$DIR/libonnxruntime_providers_cuda.so" ] \ + && [ -f "$DIR/libonnxruntime_providers_shared.so" ] \ + || { echo "error: CUDA 13 provider libraries were not found" >&2; exit 1; } +readelf -d "$DIR/libonnxruntime_providers_cuda.so" 2>/dev/null \ + | grep -q 'libcublasLt.so.13' \ + || { echo "error: provider is not built for CUDA 13" >&2; exit 1; } # 3. Assemble the .deb tree. PKG="linux-broadcast-cuda"