Skip to content
BranchesTags

Repository files navigation

ClarityForge

Local AI video upscaling, frame interpolation, and style transfer. Runs entirely on your machine — no cloud, no upload.

Stack: FastAPI · SQLite · asyncio · React + Mantine v8 · PyTorch 2.12 + CUDA 13.0

ClarityForge job configuration

Features

  • Upscaling — Real-ESRGAN, SPAN, HAT-L, Real-CUGAN, NVIDIA RTX Video Super Resolution (2× / 4×)
  • Frame interpolation — RIFE 4.26, IFRNet, GMFSS-Fortuna (2× / 4× / 8× fps)
  • AI Transform — Flux.2 Klein style transfer (BF16 / FP8 / GGUF / NVFP4)
  • Both at once — upscale then interpolate in a single job
  • Source trimming with an in-browser timeline scrubber
  • Real-time progress via WebSocket (fps, ETA, stage)
  • GPU monitoring (VRAM, utilisation, temperature, power, clock)
  • NVENC hardware encoding (HEVC / H.264)
  • NVDEC hardware decoding (optional — zero-copy GPU path)
  • File management page (list, delete, re-process uploads/outputs)
  • Real-time log viewer with Monaco Editor
  • Job queue recovery on server restart
  • Historical processing time estimation
  • Container auto-upgrade (MP4→MKV when source is MKV/WebM)
  • Subtitle and attachment stream preservation
  • Batch upscaling with configurable batch size
  • VRAM lifecycle management (automatic model load/unload)

Prerequisites

Requirement Notes
Python 3.13
Node.js 20+ for the frontend
CUDA Toolkit 13.0 RTX 30/40/50-series
FFmpeg (any recent) must be on PATH — https://ffmpeg.org/download.html
MKVToolNix must be on PATH — required for correct MKV seeking/audio/subtitle support
Git to clone vendor repos
Visual Studio 2022 Community optional — needed for torch.compile on Windows
NVIDIA Driver 550+ for NVDEC hardware decode

Install MKVToolNix:

winget install --id MoritzBunkus.MKVToolNix --exact

To enable torch.compile (2–3× speedup):

  1. Install VS 2022 with the "Desktop development with C++" workload
  2. Add cl.exe to your PATH: 
    C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\<ver>\bin\Hostx64\x64
  3. Set PYTHONUTF8=1 as a system environment variable (Win+R → sysdm.cpl → Advanced → Environment Variables → New). This must be set before Python starts so that locale.getpreferredencoding() returns UTF-8 — without it, torch's cl.exe subprocess output decoding fails with UnicodeDecodeError and the compile falls back to slow cudagraphs mode.

Backend Setup

cd backend

# 1. Install uv (fast package manager)
pip install uv

# 2. Create a virtualenv
uv venv   # creates backend/.venv/

# 3. Activate
#  PowerShell:   .venv\Scripts\Activate.ps1
#  CMD:          .venv\Scripts\activate.bat
#  Git Bash:     source .venv/Scripts/activate

# 4. Install PyTorch with CUDA 13.0 FIRST (required for RTX 50-series / Blackwell sm_120)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

# 5. Install the backend (uv reads pyproject.toml and respects the cu130 index)
uv pip install -e .

# 6. Clone vendor interpolator repos (not on PyPI)
git clone https://github.com/hzwer/Practical-RIFE vendor/Practical-RIFE
git clone https://github.com/98mxr/GMFSS_Fortuna  vendor/GMFSS_Fortuna

# 7. Start the server
uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload

Backend runs on http://localhost:8000. Model weights download automatically on first use.

Verify GPU

python -c "import torch; print(torch.cuda.get_device_name(0), torch.cuda.get_device_capability())"
# RTX 5090 → (12, 0)

Frontend Setup

cd frontend
npm install
npm run dev   # → http://localhost:5173

Frontend stack: React + Mantine v8 + Zustand + Framer Motion + Monaco Editor

Optional Performance Extras

NVDEC — Hardware GPU Video Decode

Eliminates CPU↔GPU transfers for input frames. Frames decode directly to CUDA tensors, freeing the CPU and cutting memory usage.

uv pip install -e ".[gpu-decode]"

Falls back to PyAV software decode automatically if not installed. Confirm in the backend log:

Using GPU decode (NVDEC via PyNvVideoCodec)

TensorRT — Fastest RIFE Inference (advanced)

Gives ~10–20% gain over torch.compile inductor. Caution: torch-tensorrt on PyPI pins its own torch version and will downgrade your cu130 build if installed naively.

Only install if the major.minor version matches your torch:

# Check your torch version first
python -c "import torch; print(torch.__version__)"
# e.g. 2.12.0+cu130 → install torch-tensorrt 2.12.x

uv pip install torch-tensorrt==2.12.0 --no-deps
# Then separately install any missing tensorrt libs as needed

Falls back to torch.compile inductor → eager automatically if absent. Confirm in the backend log:

Loaded RIFE from ... (torch_tensorrt backend)

Install NVDEC only

uv pip install -e ".[gpu-decode]"

Verify NVDEC

Run from backend/ with the venv active:

python -c "from app.processing.frame_extractor import nvdec_available; print(nvdec_available())"
# True if NVDEC available

Inference Backend Fallback Chain (RIFE)

The backend tries each tier in order at model-load time:

Tier Backend Requirement RTX 5090, 1080p 2× interp
1 torch_tensorrt pip install torch-tensorrt 130–170 fps
2 torch.compile inductor MSVC cl.exe on PATH 120–150 fps
3 Eager PyTorch nothing extra ~60 fps

The active backend is printed at model-load time.

VSCode Debugging

Open the clarityforge/ folder as your workspace root. Two launch configs are in .vscode/launch.json:

Config When to use
Backend: uvicorn (dev + reload) Normal dev — auto-reloads on file save
Backend: uvicorn (no reload) Breakpoints — --reload spawns a subprocess VSCode can't attach to

Press F5. Set your Python interpreter to backend/.venv/Scripts/python.exe (Ctrl+Shift+P → Python: Select Interpreter).

Supported Models

Upscalers

Model ID Scale Content VRAM Notes
realesrgan-x4plus 4× Live action ~2 GB RRDBNet 23 blocks
realesrgan-x4plus-anime6b 4× Anime ~1 GB RRDBNet 6 blocks
real-cugan-x2 2× Anime ~1 GB via spandrel
real-cugan-x4 4× Anime ~1 GB via spandrel
span-x4-nmks 4× Live action ~4 GB SPAN NMK-Superscale via spandrel
hat-l-x4 4× Live action ~8 GB via spandrel

Interpolators

Model ID FPS multiplier Content Notes
rife-4.26 2× / 4× / 8× General vendor/Practical-RIFE
ifrnet-vimeo 2× General IFRNet Vimeo90K

AI Transform (Style Transfer)

Applies full AI stylisation to a video using a text prompt. Processes every N keyframes through a FLUX diffusion model, then fills gaps with RIFE interpolation. Requires diffusers from git main (see setup below).

Model ID VRAM Notes
flux2-klein-4b ~13 GB FLUX.2-Klein 4B — fastest, HF snapshot download
flux2-klein-9b-fp8 ~17 GB FLUX.2-Klein 9B FP8 — quality/speed balance
flux2-nvfp4-4b ~6 GB FLUX.2-Klein 4B NVFP4 — RTX 5090 / Blackwell only, native FP4
flux2-nvfp4-9b ~10 GB FLUX.2-Klein 9B NVFP4 — RTX 5090 / Blackwell only, native FP4
flux2-gguf-q8 ~10 GB GGUF Q8_0 — any CUDA GPU ≥ sm_7.0, high quality
flux2-gguf-q4 ~6 GB GGUF Q4_K_M — any CUDA GPU ≥ sm_7.0, fast

Setting up AI Transform

# 1. Install diffusers from the git main branch (Flux2KleinPipeline not yet on PyPI)
uv pip install git+https://github.com/huggingface/diffusers.git

# 2. The gguf package is already in pyproject.toml — installed automatically by uv pip install -e .

# 3. (RTX 5090 / Blackwell only) NVFP4 native FP4 inference — already in pyproject.toml:
#    comfy-kitchen is installed automatically by uv pip install -e .
#    Hardware FP4 tensor-core acceleration requires cu130 PyTorch (step 4 above).
#    On older cu128 builds it still runs via eager fallback — correct results and
#    the same VRAM savings, but no throughput gain.

# 4. HF token — required for gated BFL models (flux2-klein-*, flux2-nvfp4-*).
#    Accept the license on the model card, then:
export HF_TOKEN=hf_your_token_here   # or add to backend/.env

Testing FLUX.2-Klein 9B FP8

  1. Start the backend: uvicorn app.main:app --reload
  2. Open the Models page → filter by AI Transform
  3. Click Download next to FLUX.2-Klein 9B FP8 — this does a snapshot_download of the full HuggingFace repo (~17 GB) to models_cache/style_transfer/flux2-klein-9b-fp8/. Ensure HF_TOKEN is set and you have accepted the Black Forest Labs license.
  4. Once status shows ready, upload a short clip (≤ 30 s recommended for first test)
  5. Select AI Transform as the operation, choose flux2-klein-9b-fp8, enter a prompt (e.g., convert to anime style, vibrant colors), adjust strength and keyframe interval, then submit.
  6. Watch progress: the job runs two phases — styling (diffusion, slow) then interpolating (RIFE, fast).

Note on GGUF models: flux2-gguf-q4 / flux2-gguf-q8 download only the GGUF weight file (~6/10 GB). On first run (not download), the base text encoder + VAE repo (black-forest-labs/FLUX.2-klein-9B) is auto-downloaded to models_cache/style_transfer/flux2-klein-base/ (~10 GB extra). This happens once and is cached.

Architecture

[Browser]  ←─WS─→  [FastAPI]  ←─asyncio.Queue─→  [GPU Worker Thread]
                                                           │
                                    ┌──────────────────────┤
                                    │   3-Thread Pipeline  │
                                    │   ─────────────────  │
                                    │   Decode  → Queue    │
                                    │   GPU     → Queue    │
                                    │   Encode (FFmpeg)    │
                                    └──────────────────────┘
  • Decode thread — PyAV software decode or NVDEC (if pynvvideocodec installed)
  • GPU thread — batch upscaling (spandrel) + interpolation (RIFE / GMFSS) in fp16
  • Encode thread — FFmpeg NVENC stdin pipe; final mux via mkvmerge (MKV) or FFmpeg (MP4)
  • ProgressBus — in-process pub/sub; worker writes, WebSocket fans out to all connected clients
  • VRAM lifecycle — automatic model load/unload; only one upscaler + one interpolator in VRAM
  • Job queue — SQLite-backed with recovery on restart
  • Model registry — SQLite tracking of downloads, VRAM requirements, last used

Environment Variables

Prefix: VE_. Place in backend/.env.

Variable Default Description
VE_DATA_DIR ./ Base directory for all data subdirectories
VE_UPLOADS_DIR ./uploads Uploaded input videos
VE_OUTPUTS_DIR ./outputs Finished video destination
VE_TEMP_DIR ./temp Scratch space for in-progress jobs
VE_MODELS_DIR ./models_cache Downloaded model weights
VE_THUMBNAILS_DIR ./thumbnails Job thumbnails
VE_JOBS_DB ./jobs.db SQLite job queue database
VE_TILE_SIZE 512 Upscaler tile size (pixels)
VE_TILE_OVERLAP 32 Upscaler tile overlap
VE_UPSCALE_BATCH_SIZE 1 Frames per upscaler forward pass
VE_USE_GPU_DECODE true Enable NVDEC if available
VE_MEMORY_BUDGET_GB 6 Max VRAM for frame extraction before spilling to disk

Output Filenames

Output files are named <original_filename>_<model_tag> with the container matching the source:

  • MKV/WebM source → .mkv output (muxed via mkvmerge for correct seeking)
  • MP4/other source → .mp4 output (muxed via FFmpeg)

Examples:

  • myvideo_rife.mkv — interpolation only (MKV source)
  • myvideo_realesrgan_rife.mkv — upscale + interpolation (MKV source)
  • myvideo_span.mp4 — upscale only (MP4 source)

If a file with that name already exists, a short job-ID suffix is appended.

WebSocket Events

Type Key Fields
job_progress current_stage, processed_frames, total_frames, throughput_fps, eta_seconds, progress_pct
job_status status — queued / running / done / failed / cancelled
gpu_stats gpu_util, mem_used_mb, mem_total_mb, temp_c, power_w, clock_mhz
model_download model_id, progress_pct, downloaded_mb, total_mb

API Endpoints

Jobs

Method Endpoint Description
POST /api/jobs Create new job
GET /api/jobs List all jobs
GET /api/jobs/{id} Get job details
DELETE /api/jobs/{id} Cancel/delete job
POST /api/jobs/{id}/restart Restart failed/cancelled job

Files

Method Endpoint Description
GET /api/files/uploads List uploaded files
GET /api/files/outputs List output files
DELETE /api/files/uploads/{filename} Delete upload
DELETE /api/files/outputs/{filename} Delete output
DELETE /api/files/uploads Clear all uploads
DELETE /api/files/outputs Clear all outputs
POST /api/files/uploads/{filename}/process Process existing upload

Models

Method Endpoint Description
GET /api/models List available models
POST /api/models/{model_id}/download Download model
POST /api/models/{model_id}/unload Unload model from VRAM

System

Method Endpoint Description
GET /api/system/logs?lines=N Get backend logs (default 500)
GET /api/system/disk-usage Get disk usage statistics
GET /api/system/capabilities Check system capabilities (NVDEC, etc)

Configuration

Server host/port live in config.toml at the repo root:

[server]
host = "127.0.0.1"
port = 8320

Everything else is environment-driven with the VE_ prefix. Copy .env.example to backend/.env and uncomment what you need — every setting is optional:

cp .env.example backend/.env

License

MIT — see LICENSE.

Third-party model weights and vendored repos (Practical-RIFE, GMFSS-Fortuna, Real-ESRGAN, Flux.2 Klein, …) carry their own licenses. Check each model card before redistributing or using output commercially.

About

Local AI video upscaling, frame interpolation, and style transfer. Runs entirely on your machine — no cloud, no upload.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages