⚡ Bolt: Cache ML models in memory to prevent repeated disk loading

.Jules/bolt.md

@@ -1,3 +1,6 @@
 ## 2024-05-18 - Replacing moviepy with ffprobe/ffmpeg subprocess
 **Learning:** Instantiating `VideoFileClip` from `moviepy.editor` introduces significant performance overhead when used solely to check if a video has an audio stream, or to read the duration and size of the video, as it initializes the full subclip and PyGame dependencies. By using raw `subprocess` calls to `ffprobe` (reading from `-of json`) and `ffmpeg`, we can gather properties and perform extractions in <100ms compared to `moviepy`'s >300ms overhead.
 **Action:** When extracting audio, or querying video metadata like `duration` and `size`, favor `ffprobe -show_entries` rather than instantiating `moviepy` objects if `moviepy` is not otherwise needed. Be sure to handle cleanup of original code using `.close()` if replacing those initializations.
+## 2026-08-24 - Global caching of Large ML Models
+**Learning:** Large ML models (e.g., OpenAI Whisper and RVM) cause severe performance bottlenecks and potential Out-Of-Memory (OOM) issues if reloaded from disk repeatedly during processing.
+**Action:** Use `functools.lru_cache(maxsize=1)` to globally cache large models in memory and prevent repeated disk loading.

background_remover.py

@@ -7,6 +7,7 @@
 import cv2
 import numpy as np
 from moviepy.editor import VideoFileClip
+import functools
 
 # Configure logging
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
@@ -64,6 +65,7 @@ def reporthook(count, block_size, total_size):
     return model_path
 
 
+@functools.lru_cache(maxsize=1)
 def load_rvm_model(model_name: str = 'mobilenetv3', device: str = 'auto') -> torch.nn.Module:
     """
     Load RVM model.

processor.py

@@ -4,6 +4,7 @@
 
 import whisper
 from pydub import AudioSegment, silence
+import functools
 
 # Configure logging
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
@@ -82,14 +83,18 @@ def detect_filler_words(audio_path: str, model_size: str = "large-v3-turbo", fil
     """
     return detect_filler_words_whisper(audio_path, model_size, filler_words_list)
 
+@functools.lru_cache(maxsize=1)
+def get_whisper_model(model_size: str):
+    logging.info(f"Loading Whisper model ({model_size})...")
+    return whisper.load_model(model_size)
+
 def detect_filler_words_whisper(audio_path: str, model_size: str = "large-v3-turbo", filler_words_list: List[str] = None) -> List[Tuple[float, float]]:
     """
     Detects filler words using standard Whisper.
     Returns:
         Tuple of (List of (start, end) tuples, transcript string).
     """
-    logging.info(f"Loading Whisper model ({model_size})...")
-    model = whisper.load_model(model_size)
+    model = get_whisper_model(model_size)
     logging.info("Transcribing audio for filler word detection...")
     # We use a prompt to encourage transcribing filler words if possible, though Whisper is trained to remove them.
     # Sometimes standard transcription removes them. We can try to rely on word-level timestamps.