🦊 RotoFox — Smart Video Segmentation

Offline, AI-Powered Rotoscoping & Video Segmentation for Editors

The local alternative to After Effects Roto Brush & Runway — runs fully offline on your own GPU/CPU.

---

🖥️ Application Interface (Before / After)

Before: Overlay View Mode (Real-time Mask Track)	After: Isolated View Mode (Background Removed)

---

📖 What is RotoFox?

RotoFox is a desktop-grade, 100% offline rotoscoping tool. It lets video editors and filmmakers automate the process of creating precise video masks — separating subjects from backgrounds — using the power of Meta's Segment Anything Model 2 (SAM 2.1) and MatAnyone 2 for ultra-fine edge refinement (capturing fine details like hair strands, semi-transparent smoke, and motion blur).

Instead of frame-by-frame manual masking, RotoFox lets you click once (or draw a box) on a subject, and the AI automatically tracks and generates the mask across the entire video timeline. All computation runs locally on your own machine (supporting GPU CUDA acceleration as well as CPU fallbacks) — your footage never leaves your machine.

---

✨ Core Features

Feature	Description
🖱️ Interactive Point-and-Click	Left-click (Include / green dot) to define the foreground object. Right-click (Exclude / red dot) to exclude background leaks.
🟦 Box Selection Mode	Draw a bounding box around the subject for fast, precise localization prompts.
🤖 AI Propagation	Streams mask predictions frame-by-frame across the timeline using SAM 2's Video Predictor memory bank, forwards or backwards from the current frame.
💇 Hair-Level Matting	Integrates MatAnyone 2 to refine boundaries, producing high-quality alpha matte edges.
↩️ Undo / Redo History	Full state history with keyboard shortcuts (Ctrl+Z, Ctrl+Y, Ctrl+Shift+Z) synced dynamically to the AI backend.
🎞️ Multi-Object Timeline	Visual timeline controller supporting up to 7 concurrent, color-coded mask layers with real-time feedback.
⌨️ Editor-Friendly Shortcuts	Navigation designed for standard NLE workflows (Arrow keys, Spacebar, Page Up/Down, Home/End).
📤 Production-Ready Exports	Transparent ProRes 4444 (.mov), transparent VP9 WebM, Luma Matte video, PNG sequence (RGBA or matte), or a solid-colour composite. ProRes and transparent WebM require FFmpeg — see Export formats.
🔒 Secure & Offline	No telemetry and no cloud processing. The only network access is downloading model weights on first run; after that RotoFox works with no connection at all. The local API is protected by a per-session token.

---

📁 Detailed Directory Structure

Below is the layout of the project, explaining the roles of the main directories and files:

Smart Video Segmentation/
├── backend/
│   ├── main.py                  # FastAPI server entry point (restores WebSocket & API routes)
│   ├── requirements.txt         # Python dependencies (PyTorch, OpenCV, FastAPI, etc.)
│   ├── rotofox-backend.spec     # PyInstaller spec configuration for packaging the backend binary
│   ├── app/
│   │   ├── api/
│   │   │   ├── routes.py        # REST endpoints (handles video upload & setup wizards)
│   │   │   └── websockets.py    # WebSocket communication (processes click/box actions, tracking, exports)
│   │   ├── core/
│   │   │   └── engine_state.py  # Centralized tracking progress & cancellation controller
│   │   └── services/
│   │       ├── ai_engine.py     # Orchestrates SAM 2 & MatAnyone 2 models, exports, and math conversions
│   │       ├── cache_manager.py # Handles SSD storage directory caching for JPG frame extracts
│   │       ├── memory_manager.py# Manages system garbage collection and CUDA VRAM clearing
│   │       ├── model_manager.py # Profiles hardware, recommends models, and runs download hub
│   │       └── video_processor.py# Decodes video frames via OpenCV, handles FPS limits & resource closing
│   └── scripts/
│       ├── package_backend.py   # Script to package backend python scripts into a standalone sidecar
│       ├── setup_models.py      # Automated download utility for AI weights
│       └── setup_sam2.py        # Installs Segment Anything 2 module locally
├── frontend/
│   ├── package.json             # NPM dependencies & build scripts
│   ├── vite.config.js           # Vite server configuration
│   ├── src/
│   │   ├── App.jsx              # Main UI controller, states, dialog popups, and WebSocket listener
│   │   ├── index.css            # Styling core (dark-mode glassmorphic theme)
│   │   ├── hooks/
│   │   │   └── useAIEngine.js   # Custom React hook wrapper for the WebSocket server connection
│   │   └── components/
│   │       ├── canvas/
│   │       │   └── VideoCanvas.jsx  # Interactive canvas displaying video, prompts (dots/boxes), and masks
│   │       ├── layout/          # UI framework layouts
│   │       ├── setup/
│   │       │   └── SetupWizard.jsx  # Wizard helping download models on first run
│   │       ├── sidebar/
│   │       │   └── Toolbar.jsx      # toolbar for tools selection (Include, Exclude, Box, Export, Model Hub)
│   │       └── timeline/
│   │           └── TimelineController.jsx # Media timeline scrubbing, play/pause, frame tracks
│   └── src-tauri/
│       ├── tauri.conf.json      # Desktop application wrapper configurations
│       ├── Cargo.toml           # Rust package configuration
│       └── src/
│           ├── main.rs          # Tauri execution entry point
│           └── lib.rs           # Rust application setup (spawns PyInstaller sidecar binary automatically)
├── dist/                        # Holds portable distribution builds
├── docs/                        # Project documentation files (diagrams, walkthroughs)
├── run_all.bat                  # Developer shortcut: starts backend server and Vite frontend concurrently
├── run_backend.bat              # Dev shortcut: launches Python backend server
├── build_portable.bat           # Production builder: packages application into a portable zip folder
└── setup_and_build.bat          # Complete setup script: installs Node modules, Python venv, and builds target release

---

🏗️ System Architecture & Workflow

RotoFox uses a Local Hybrid Architecture where the Tauri-wrapped frontend communicates with a Python AI backend sidecar via local WebSockets and REST APIs.

1. Architectural Layout

graph TD
    User([User Editor]) -->|Interacts| Frontend[React + Vite Web App]
    
    subgraph Tauri Desktop Wrapper
        Frontend <-->|JSON Events| RustCore[Rust main.exe]
        RustCore -->|Spawns Sidecar Process| BackendSidecar[PyInstaller rotofox-backend.exe]
    end
    
    subgraph BackendSidecar [AI Backend: PyInstaller rotofox-backend.exe]
        API[FastAPI WebSocket & REST API Router]
        
        subgraph AIEngine [AI Orchestrator Engine]
            SAM2["SAM 2.1 Video Predictor<br>(Real-time Mask Segmentation & Tracking)"]
            MatAnyone["MatAnyone 2 Edge Refinement<br>(Fine Hair & Detail Matting on Export)"]
        end
        
        Cache[Cache Manager / SSD Frame Cache]
    end
    
    Frontend <-->|Local WebSockets & REST| API
    API <--> AIEngine
    API <--> Cache
    AIEngine -->|GPU/CPU Tensor Execution| PyTorch[PyTorch + CUDA/CPU]
    AIEngine -->|Frame Processing & Compiling| OpenCV[OpenCV / FFmpeg]

Loading

Core Backend AI Modules:

• Meta's SAM 2.1 (Segment Anything Model 2.1): Runs real-time video segmentation. When you place click prompts (include/exclude) or draw bounding boxes, SAM 2.1 predicts the subject's mask on the current frame and propagates (tracks) that mask forwards/backwards across the timeline frames using its memory attention bank.
• MatAnyone 2: Runs edge matte refinement during the project export phase. Coarse masks generated by SAM 2.1 are automatically sent to MatAnyone 2 to compute sub-pixel transparency (alpha mattes) for challenging edges like hair strands, smoke, transparency, and motion blur.

---

2. Execution Sequence Diagram

The diagram below details the sequence of network and local process events during a user's rotoscoping session:

sequenceDiagram
    autonumber
    actor User as User Editor
    participant FE as React Frontend
    participant Rust as Tauri Rust Wrapper
    participant BE as FastAPI Backend Sidecar
    participant SAM2 as SAM 2.1 Engine
    participant MA2 as MatAnyone 2 Engine

    Note over User, BE: Initializing Application
    Rust->>BE: Spawn sidecar process
    FE->>BE: Connect to WebSocket (ws://localhost:8000/ws/editor)
    
    Note over User, BE: Video Import & Setup
    User->>FE: Import Video File
    FE->>BE: POST /upload (video file)
    BE->>BE: Downsample to 30 FPS & extract frames to SSD cache
    BE->>SAM2: Initialize inference state with frame cache
    BE-->>FE: Return frames count & effective FPS
    
    Note over User, BE: Interactive Prompting
    User->>FE: Click/Draw Box on canvas
    FE->>BE: Send click/box coordinates {points, labels, frame_idx, obj_id}
    BE->>SAM2: add_new_points_or_box()
    SAM2-->>BE: Return prediction logits
    BE->>BE: Convert logits to base64 mask overlay
    BE-->>FE: Stream mask_update (base64 image)
    FE->>User: Render mask overlay on canvas
    
    Note over User, BE: Timeline Tracking (Propagation)
    User->>FE: Click "Track Forward"
    FE->>BE: Send tracking command over WS
    loop For each frame
        BE->>SAM2: propagate_in_video()
        SAM2-->>BE: Frame mask prediction
        BE->>BE: Save PNG mask to disk cache
        BE-->>FE: Stream tracking progress (frame, progress, mask_base64)
        FE->>User: Update timeline frame & overlay
    end
    BE-->>FE: Stream completed status
    
    Note over User, BE: Project Export
    User->>FE: Click "Export"
    FE->>BE: Send export settings {format, type, bg_color}
    alt If MatAnyone 2 is downloaded
        BE->>MA2: Load model & run edge matte refinement
        MA2-->>BE: Output refined sub-pixel alpha mattes
    else If MatAnyone 2 is not available
        BE->>BE: Fallback to SAM 2 raw masks
    end
    BE->>BE: Composite frames with masks (ProRes / Luma Matte)
    BE-->>FE: Stream export progress & export_completed (file_path)
    FE->>User: Show success message & export folder path

Loading

3. Processing Pipeline

1. Import & Downsample: When a video is uploaded, the backend downsamples high-frame-rate clips to a default of 30 FPS to prevent high RAM/VRAM usage. OpenCV extracts these frames into a cached JPEG directory (cache_workspace/video_<timestamp>/).
2. Interactive Prompting: The editor inputs point or box prompts on the canvas. The frontend sends the coordinates to the backend, which feeds them into SAM 2 and sends back a base64 encoded PNG mask overlay.
3. Timeline Propagation: When "Track Forward" is triggered, the SAM 2 Video Predictor runs propagation over the timeline, streaming progress packets back to the client.
4. MatAnyone 2 Edge Refinement: During export, SAM 2's coarse masks pass through the MatAnyone 2 neural net to refine fine details (like hair, transparency, or motion blur).
5. Compositing: OpenCV compiles the processed frames and masks to write the final output video format.

---

💻 Installation & Setup

🚀 Developer Quickstart (Local Run)

Double-click the run_all.bat script at the project root. This launcher automatically spins up the Python backend server and the Vite dev server in separate terminal windows.

Prerequisites

• OS: Windows 10/11. (The backend is cross-platform and the Tauri sidecar now resolves per-platform binaries, but only the Windows build is packaged and tested today.)
• Python: version 3.10 or higher (Python 3.11/3.12 recommended)
• Node.js: version 18 or 20 (installed with npm)
• GPU: NVIDIA GPU with 4 GB+ VRAM (CUDA 12.1) recommended. Runs on CPU, but 20-50x slower.
• FFmpeg (optional): needed for ProRes 4444 and transparent WebM export. pip install imageio-ffmpeg supplies one automatically, or put ffmpeg.exe next to RotoFox.exe.

---

Manual Step-by-Step Developer Setup

1. Backend Setup

# Navigate to the backend directory
cd backend

# Create a virtual environment
python -m venv .venv

# Activate the virtual environment
# Windows:
.venv\Scripts\activate
# Linux/macOS:
source .venv/bin/activate

# Install PyTorch with CUDA FIRST — a plain "pip install torch" resolves the
# CPU-only wheel on Windows and inference becomes 20-50x slower
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121

# Install the remaining dependencies
pip install -r requirements.txt

# Install the SAM 2 framework (clone + editable install, no weights)
python scripts/setup_sam2.py

# Download the model weights.
#   interactive:      python scripts/setup_models.py
#   non-interactive:  python scripts/setup_models.py --yes
#   specific variant: python scripts/setup_models.py --sam small
python scripts/setup_models.py

# Run the backend in development mode
#   --no-auth  lets a browser dev client connect without a session token
#   --reload   restarts on source changes
python main.py --no-auth --reload

Which models do I need? A SAM 2.1 checkpoint is required — without one, RotoFox cannot segment anything. MatAnyone 2 is optional: it refines hair and fine edges during export, and RotoFox falls back to the raw SAM 2 masks when it is absent.

2. Frontend Setup

# Navigate to the frontend directory
cd ../frontend

# Install node packages
npm install

# Option A: Run in browser
npm run dev

# Option B: Run as a desktop application
npm run tauri dev

---

📦 Building for Production

If you want to package RotoFox into a single, production-ready desktop installation wizard (.exe), follow these steps:

1. Automated Installation Setup & Build

Simply double-click setup_and_build.bat. It will automatically:

1. Create the backend Python virtual environment.
2. Install Python dependencies and configure SAM 2.
3. Package the Python backend into a standalone folder using PyInstaller.
4. Run npm install and compile the Tauri app into an installer executable.

2. Standalone Installer Path

Once built, you can find the single installation wizard at: frontend/src-tauri/target/release/bundle/nsis/RotoFox_1.0.0_x64-setup.exe

Double-clicking this file installs the desktop app. Running the installed shortcut automatically runs the React interface and launches the Python AI backend sidecar invisibly in the background.

---

---

🎬 Export formats

Format	Codec	Alpha	Requires FFmpeg	Best for
MOV	ProRes 4444	✅	Yes	Premiere / After Effects / Resolve
WebM	VP9 (yuva420p)	✅	Yes	Web delivery
PNG Sequence	PNG	✅	No	Maximum compatibility, lossless
MP4	H.264	❌	No*	Previews and review copies

_{*MP4 uses FFmpeg when available and falls back to OpenCV's mp4v encoder otherwise.}

Render types

• Transparent — real alpha channel; the background is genuinely empty.
• Luma Matte — pure white subject on pure black, for use as a track matte.
• Solid Color — subject composited over green, blue, black or white.
• Video Overlay — the mask tinted over the source plate, for checking coverage.

If FFmpeg is missing, MOV and transparent WebM are refused with an explanatory message rather than producing a broken file. PNG Sequence always works and needs no external tools.

---

🔧 Troubleshooting

"No tracking model installed" Open the Model Hub and download a SAM 2.1 checkpoint. Pick the one marked Best for your PC — it is chosen from your actual VRAM.

A model shows "Incomplete" The download was interrupted. Click Retry. Partial files are detected by size and never mistaken for a working checkpoint.

"Another RotoFox window is already connected" Only one editor may drive the AI engine at a time, because they would share one inference state. Close the other window.

Export says FFmpeg was not found Either pip install imageio-ffmpeg inside backend/.venv, install FFmpeg and put it on PATH, or drop ffmpeg.exe next to RotoFox.exe. PNG Sequence export works without it.

Where are the logs? %LOCALAPPDATA%\RotoFox\logs\backend.log for a packaged build, or backend/logs/backend.log when running from source.

Where is the cache? %LOCALAPPDATA%\RotoFox\cache_workspace for a packaged build, or backend/cache_workspace from source. The three most recent sessions are kept.

---

⌨️ Keyboard Shortcuts

Key Shortcut	Action
Spacebar	Play / Pause timeline playback
← / →	Seek backward / forward 1 frame
Page Up	Jump forward 10 frames
Page Down	Jump backward 10 frames
Home	Jump to first frame
End	Jump to last frame

All shortcuts are global — they work anywhere in the window except while typing in a text field. | Ctrl + Z | Undo last prompt (point or box) | | Ctrl + Y or Ctrl + Shift + Z | Redo last undone action |

---

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

1. Fork this repository.
2. Create a branch: git checkout -b feature/amazing-feature.
3. Commit your changes following Conventional Commits.
4. Push to the branch and open a Pull Request.

---

📄 License

This project is licensed under the MIT License. See the LICENSE file for more details.

---

Made with ❤️ for editors and creators who want premium local tools.
RotoFox — because every frame of your story matters. 🦊🎬