Recover on CPU after CUDA inference failure

README.md

@@ -143,6 +143,107 @@ Verify it engaged: `RUST_LOG=info linux-broadcast` logs
 `ONNX Runtime: CUDA execution provider registered`. A working NVIDIA driver
 (`nvidia-smi`) supporting CUDA 13 is required.
 
+### New NVIDIA architectures: build a custom CUDA provider
+
+The prebuilt ONNX Runtime CUDA provider is compiled only for a fixed set of
+GPU architectures. If your GPU is newer than that set, session creation can
+succeed but the first inference can fail with
+`cudaErrorNoKernelImageForDevice`. LinuxBroadcast then shows **GPU unavailable
+· CPU** and continues on CPU. Updating only the NVIDIA driver or CUDA runtime
+cannot add a kernel image that was omitted from the provider.
+
+This happened, for example, with Blackwell consumer GPUs (RTX 50 series,
+compute capability / SM 120). The following procedure makes a local package
+for a newer architecture. It is intentionally an advanced build: the custom
+ONNX Runtime static library, its CUDA provider libraries, and the
+LinuxBroadcast binary must all come from the same ONNX Runtime version and
+build configuration.
+
+1. Identify the required CUDA architecture. Check the GPU name with
+   `nvidia-smi`, then look up its compute capability in NVIDIA's CUDA GPU table.
+   For an RTX 50 card use `120`. If one package must support several GPU
+   generations, provide a semicolon-separated list such as `89;120`; a build
+   for only `120` is not a universal replacement for older GPUs.
+
+2. Find the ONNX Runtime version used by this checkout. Build it once normally,
+   then inspect `ort-sys`'s build output:
+
+   ```bash
+   cargo build --release -p linux-broadcast --features cuda
+   find target -path '*/build/ort-sys-*/output' -type f \
+     -exec grep -H 'downloading from' {} \;
+   ```
+
+   The URL includes the ONNX Runtime version and CUDA major version. Use an
+   `ort-artifacts` revision that supports that exact ONNX Runtime release; do
+   not assume the newest artifact-builder revision is compatible with an older
+   release.
+
+3. Build ONNX Runtime from that release with the same CUDA major version and
+   cuDNN major version as the prebuilt provider, but add your architecture.
+   The `pykeio/ort-artifacts` static-build wrapper is a convenient way to
+   produce the layout required by the Rust `ort` crate. Its essential CMake
+   option is:
+
+   ```bash
+   -Donnxruntime_USE_CUDA=ON \
+   -DCMAKE_CUDA_ARCHITECTURES=120
+   ```
+
+   Start from the wrapper's normal CUDA build options for that release and
+   change only `CMAKE_CUDA_ARCHITECTURES`. Use an explicit value rather than
+   `native`, especially in a container: a build container usually cannot query
+   the host GPU. The resulting install directory must contain all three files:
+
+   ```text
+   libonnxruntime.a
+   libonnxruntime_providers_cuda.so
+   libonnxruntime_providers_shared.so
+   ```
+
+4. Point `ort-sys` at that directory and rebuild LinuxBroadcast. Cleaning
+   `ort-sys` matters because Cargo otherwise reuses the downloaded prebuilt
+   ONNX Runtime library:
+
+   ```bash
+   export ORT_LIB_PATH=/absolute/path/to/onnxruntime/lib
+   export ORT_SKIP_DOWNLOAD=1
+   cargo clean -p ort-sys
+   cargo build --release -p linux-broadcast --features cuda
+   ```
+
+5. Package the matching provider libraries. `ORT_PROVIDER_DIR` makes the
+   helper use the custom directory rather than its normal download cache:
+
+   ```bash
+   cargo install cargo-deb
+   cargo deb --no-build -p linux-broadcast
+   ORT_PROVIDER_DIR="$ORT_LIB_PATH" packaging/build-cuda-addon.sh
+   ```
+
+   Install both generated packages together (replace the filenames when the
+   version changes):
+
+   ```bash
+   sudo apt install --reinstall \
+     ./target/debian/linux-broadcast_<version>-1_amd64.deb \
+     ./target/debian/linux-broadcast-cuda_<version>-1_amd64.deb
+   ```
+
+6. Verify the provider contains the requested image (requires the CUDA
+   toolkit's `cuobjdump`) and start the app with logs enabled:
+
+   ```bash
+   cuobjdump --list-elf "$ORT_LIB_PATH/libonnxruntime_providers_cuda.so" | grep sm_120
+   RUST_LOG=info linux-broadcast
+   ```
+
+   A successful run logs `ONNX Runtime: CUDA execution provider registered`,
+   retains the normal **GPU** footer state, and does not emit the CPU-fallback
+   warning. If it still falls back, preserve the complete ONNX Runtime error;
+   it usually identifies either a missing runtime dependency or an architecture
+   mismatch.
+
 ### Option C — Build from source
 
 Use this when hacking on the code; the `.deb` is the right choice for everyday use.

crates/app/src/ui/footer.rs

@@ -24,6 +24,9 @@ impl App {
                     if self.backend == Backend::Gpu {
                         ui.add_space(space::SM);
                         gpu_badge(ui);
+                    } else if let Some(reason) = &self.gpu_fallback_reason {
+                        ui.add_space(space::SM);
+                        gpu_fallback_badge(ui, reason);
                     }
                     ui.add_space(space::MD);
                     sep(ui);
@@ -131,6 +134,27 @@ fn gpu_badge(ui: &mut egui::Ui) {
         });
 }
 
+/// Persistent, visible notice that CUDA started but failed on a real model
+/// kernel and the pipeline recovered on CPU. The technical error stays in the
+/// hover text so the compact footer remains readable.
+fn gpu_fallback_badge(ui: &mut egui::Ui, reason: &str) {
+    egui::Frame::none()
+        .fill(color::DANGER_SOFT)
+        .stroke(Stroke::new(1.0, color::DANGER))
+        .rounding(4.0)
+        .inner_margin(egui::Margin::symmetric(6.0, 1.0))
+        .show(ui, |ui| {
+            ui.label(
+                egui::RichText::new("GPU unavailable · CPU")
+                    .monospace()
+                    .small()
+                    .strong()
+                    .color(color::DANGER),
+            )
+            .on_hover_text(reason);
+        });
+}
+
 fn sep(ui: &mut egui::Ui) {
     let (rect, _) = ui.allocate_exact_size(egui::vec2(1.0, 14.0), egui::Sense::hover());
     ui.painter().vline(

crates/app/src/ui/mod.rs

@@ -223,6 +223,9 @@ pub(super) struct App {
     /// Execution backend the running pipeline's segmenter uses. Drives the
     /// footer's "GPU" badge; defaults to `Cpu` (no badge) when not running.
     backend: Backend,
+    /// Set only when a CUDA session accepted at startup later fails during
+    /// inference and the pipeline transparently continues on CPU.
+    gpu_fallback_reason: Option<String>,
     error: Option<String>,
     /// System tray handle. Held for the whole process lifetime. `None`
     /// only if the OS has no working tray host (logged at install time);
@@ -320,6 +323,7 @@ impl App {
             last_preview_size: None,
             pipeline_state: PipelineState::default(),
             backend: Backend::default(),
+            gpu_fallback_reason: None,
             error: initial_error,
             tray,
             quit_requested: false,
@@ -390,6 +394,7 @@ impl App {
             Ok(p) => {
                 let cmd_tx = p.cmd_sender();
                 self.backend = p.backend();
+                self.gpu_fallback_reason = p.gpu_fallback_reason();
                 // Push the saved toggle states to the freshly started
                 // pipeline so they survive Stop+Start cycles (model
                 // swaps, camera swaps).
@@ -423,6 +428,7 @@ impl App {
         self.last_preview_size = None;
         self.pipeline_state = PipelineState::default();
         self.backend = Backend::default();
+        self.gpu_fallback_reason = None;
         // The next pipeline starts at gui_preview_active = false; clear
         // the cache so the next heartbeat re-sends.
         self.last_gui_preview_active = None;
@@ -698,6 +704,8 @@ impl eframe::App for App {
         // ticks.
         if let Some(p) = &self.pipeline {
             self.pipeline_state = p.state();
+            self.backend = p.backend();
+            self.gpu_fallback_reason = p.gpu_fallback_reason();
         }
 
         self.drain_preview(ctx);

crates/pipeline/src/lazy.rs

@@ -122,6 +122,14 @@ pub(crate) fn next_state(state: State, demand: Demand, now: Instant) -> State {
     }
 }
 
+/// What `spawn_feeder` hands back: the feeder thread, the live execution
+/// backend, and the reason for a GPU-to-CPU recovery once one has happened.
+type FeederHandles = (
+    std::thread::JoinHandle<()>,
+    Arc<Mutex<Backend>>,
+    Arc<Mutex<Option<String>>>,
+);
+
 /// Spawn the feeder thread. Returns the join handle plus a stop-flag the
 /// caller (Pipeline) can flip to ask the thread to wind down promptly;
 /// the thread also exits naturally on `Command::Stop` and on cmd-channel
@@ -137,16 +145,18 @@ pub(crate) fn spawn_feeder(
     state_pub: Arc<Mutex<PipelineState>>,
     stop_flag: Arc<AtomicBool>,
     source_builder: SourceBuilder,
-) -> Result<(std::thread::JoinHandle<()>, Backend)> {
-    let segmenter = Segmenter::from_bytes(
-        cfg.model,
-        match cfg.model {
-            ModelKind::SelfieMulticlass => multiclass_onnx,
-            ModelKind::Rvm => rvm_onnx,
-        },
-    )
-    .context("load segmentation model")?;
+) -> Result<FeederHandles> {
+    let model_onnx = match cfg.model {
+        ModelKind::SelfieMulticlass => multiclass_onnx,
+        ModelKind::Rvm => rvm_onnx,
+    };
+    let segmenter =
+        Segmenter::from_bytes(cfg.model, model_onnx).context("load segmentation model")?;
     let backend = segmenter.backend();
+    let backend_pub = Arc::new(Mutex::new(backend));
+    let gpu_fallback_pub = Arc::new(Mutex::new(None));
+    let backend_pub_for_thread = Arc::clone(&backend_pub);
+    let gpu_fallback_pub_for_thread = Arc::clone(&gpu_fallback_pub);
 
     let idle_loader = IdleLoader::new(cfg.width, cfg.height);
 
@@ -169,6 +179,9 @@ pub(crate) fn spawn_feeder(
             let mut feeder = Feeder {
                 cfg: cfg_for_thread,
                 segmenter,
+                model_onnx,
+                backend_pub: backend_pub_for_thread,
+                gpu_fallback_pub: gpu_fallback_pub_for_thread,
                 compositor: Compositor::new(),
                 smoother: MaskSmoother::new(match cfg.model {
                     ModelKind::Rvm => RVM_ALPHA,
@@ -209,12 +222,21 @@ pub(crate) fn spawn_feeder(
             feeder.run(cmd_rx, watcher_rx, stop_flag);
         })
         .context("spawn lb-feeder")?;
-    Ok((handle, backend))
+    Ok((handle, backend_pub, gpu_fallback_pub))
 }
 
 struct Feeder {
     cfg: PipelineConfig,
     segmenter: Segmenter,
+    /// Bytes of the selected model, retained so a delayed CUDA failure can
+    /// rebuild the same model with the CPU execution provider.
+    model_onnx: &'static [u8],
+    /// Live backend status published to the GUI. Unlike session creation,
+    /// CUDA compatibility is only proven after the first inference.
+    backend_pub: Arc<Mutex<Backend>>,
+    /// One-shot user-facing explanation emitted when GPU inference fails and
+    /// the feeder continues on CPU.
+    gpu_fallback_pub: Arc<Mutex<Option<String>>>,
     compositor: Compositor,
     smoother: MaskSmoother,
     /// Smoothed virtual-PTZ crop driver. Held even when `framing_enabled`
@@ -735,16 +757,60 @@ impl Feeder {
 
             if do_infer {
                 let t_seg = Instant::now();
-                match self
-                    .segmenter
-                    .segment(&frame_rgba, frame_w as usize, frame_h as usize)
-                {
+                let segment =
+                    self.segmenter
+                        .segment(&frame_rgba, frame_w as usize, frame_h as usize);
+                match segment {
                     Ok(mut mask) => {
                         self.smoother.smooth(&mut mask.data);
                         segment_dur = Some(t_seg.elapsed());
                         self.cached_framing = self.compute_framing(&mask);
                         self.cached_mask = Some(mask);
                     }
+                    Err(gpu_error) if self.segmenter.backend() == Backend::Gpu => {
+                        let technical_reason = format!("{gpu_error:#}");
+                        let user_reason = format!(
+                            "GPU acceleration was disabled after an inference error; LinuxBroadcast is now using CPU. {technical_reason}"
+                        );
+                        log::warn!(
+                            "{user_reason} To restore GPU acceleration, install an ONNX Runtime CUDA provider built for this GPU."
+                        );
+
+                        if let Err(cpu_error) = self.segmenter.fall_back_to_cpu(self.model_onnx) {
+                            log::error!(
+                                "GPU inference failed ({technical_reason}); CPU fallback could not start: {cpu_error:#}"
+                            );
+                            return;
+                        }
+                        if let Ok(mut backend) = self.backend_pub.lock() {
+                            *backend = Backend::Cpu;
+                        }
+                        if let Ok(mut reason) = self.gpu_fallback_pub.lock() {
+                            *reason = Some(user_reason);
+                        }
+
+                        // The GPU error left the recurrent state unusable, so
+                        // run this very frame again against the clean CPU
+                        // session. From here on every inference stays on CPU.
+                        match self.segmenter.segment(
+                            &frame_rgba,
+                            frame_w as usize,
+                            frame_h as usize,
+                        ) {
+                            Ok(mut mask) => {
+                                self.smoother.smooth(&mut mask.data);
+                                segment_dur = Some(t_seg.elapsed());
+                                self.cached_framing = self.compute_framing(&mask);
+                                self.cached_mask = Some(mask);
+                            }
+                            Err(cpu_error) => {
+                                log::error!(
+                                    "GPU inference failed ({technical_reason}); CPU retry also failed: {cpu_error:#}"
+                                );
+                                return;
+                            }
+                        }
+                    }
                     Err(e) => {
                         log::error!("segment: {e:#}");
                         return;

crates/pipeline/src/pipeline.rs

@@ -199,7 +199,8 @@ pub struct Pipeline {
     state: Arc<Mutex<PipelineState>>,
     feeder: Option<std::thread::JoinHandle<()>>,
     stop_flag: Arc<AtomicBool>,
-    backend: Backend,
+    backend: Arc<Mutex<Backend>>,
+    gpu_fallback: Arc<Mutex<Option<String>>>,
 }
 
 impl Pipeline {
@@ -266,7 +267,7 @@ impl Pipeline {
         let watcher_rx = watcher.events().clone();
 
         // 4. Feeder thread.
-        let (feeder, backend) = spawn_feeder(
+        let (feeder, backend, gpu_fallback) = spawn_feeder(
             cfg,
             multiclass_onnx,
             rvm_onnx,
@@ -286,17 +287,30 @@ impl Pipeline {
             stop_flag,
             _watcher: watcher,
             backend,
+            gpu_fallback,
         })
     }
 
     pub fn cmd_sender(&self) -> Sender<Command> {
         self.cmd_tx.clone()
     }
 
-    /// Execution backend the segmenter ended up using (CPU or GPU).
-    /// Fixed for the lifetime of the pipeline — model/EP are chosen at start.
+    /// Execution backend the segmenter is currently using (CPU or GPU).
+    /// A GPU session may fail only on its first kernel launch, in which case
+    /// the feeder switches it to CPU and this value changes accordingly.
     pub fn backend(&self) -> Backend {
         self.backend
+            .lock()
+            .map(|backend| *backend)
+            .unwrap_or_default()
+    }
+
+    /// Explains a one-way GPU-to-CPU recovery, if one occurred.
+    pub fn gpu_fallback_reason(&self) -> Option<String> {
+        self.gpu_fallback
+            .lock()
+            .ok()
+            .and_then(|reason| reason.clone())
     }
 
     /// Snapshot the current public state. Cheap; backed by an