video_example_clean.mp4

ComfyUI-FingerWindow-Local-SCAIL2

Two-hand finger-frame gesture → the quad your fingers form becomes a window into an AI character-swapped version of the same footage, generated entirely locally in ComfyUI. No API key, no cloud billing.

Author

Local port and everything past the tracking step (region cropping, head-drift auto-align, rim-refraction composite) by stark622.

Credit

This is a local, no-API-key ComfyUI port of the gesture-tracking pipeline from sophiamyang/finger-frame-effect-ai (and the wider finger-frame-effect family, including -lucy and the fal fork). The two-hand quad tracking — anatomical corner ordering, spread/area gates with hysteresis, teleport rejection, velocity-adaptive smoothing, dropout hold, presence fade — follows Sophia Yang's audited pipeline. Everything downstream of tracking (region cropping, head-drift auto-align, the screen-aligned rim-refraction composite) is this repo's own code, written for a local Wan2.1 SCAIL-2 backend instead of a hosted API.

App	Generation	Where it runs
finger-frame-effect	Canvas 2D filters	Browser, local, no AI
finger-frame-effect-ai	Gemini Omni Flash	Google API, your key, billed
finger-frame-effect-lucy	Decart Lucy 2.5	Decart API, your key, billed
finger-frame-effect-fal	Lucy 2.5 / FLUX.2 klein	fal API, your key, billed
ComfyUI-FingerWindow-Local-SCAIL2 (this repo)	Wan2.1 SCAIL-2 (local)	Your own GPU, no API key

Differences from finger-frame-effect-ai (the original)

• Local, no API key, no billing. The original sends your clip to Gemini Omni Flash per generation, billed per video, a few minutes each time. This runs on your own GPU, nothing leaves your machine.
• Character reference, not a style prompt. The original picks a style via text prompt (3D animated movie, anime, claymation, watercolor, or custom). This version does a full character-identity swap from HEAD.png + BODY.png reference photos instead — any character, photoreal or illustrated, not just a named style.
• Generates a cropped region, not the whole frame. The original's "true video model" regenerates the entire clip. This version's FingerWindowRegion generates only the union of everywhere the window travels across the clip plus the head — typically a fraction of the full frame (see the tested example below) — then pastes that back. Saves VRAM and generation time; the trade-off is it assumes a single continuous take per clip.
• Head-drift auto-align. Not present in the original. A local character-swap model places the swapped head where it wants, not pixel-matched to the real head, so FingerWindowPaste's auto_align measures and corrects that drift per frame.
• Rim-only refraction. The base app's glass-bulge effect displaces across the whole pane; this version's bevel only bends light in a border strip near the window edge, keeping the center pixel-aligned with the real footage underneath.

Tested on

RTX 4090 24GB / 64GB system RAM. A ~25 second / 618-frame 1280×720 driving clip: window visible on 585 of those frames, generation region ~41% of the full frame, full render (9 chunked SCAIL-2 passes, 6 steps each) in ~14 minutes.

A later 237-frame test, comparing standard chunked sampling against adding WanAnimatePlus SCAIL_2 TwoPhase Settings to the same graph: standard sampling ran the 4-chunk render in ~447s and ~441s across two runs; switching on two-phase sampling brought the same 4-chunk render down to ~366s (~18% faster), same LoRA stack, same region size. Not a rigorous benchmark — one clip, few runs — but consistent across chunks each time.

How it works

Same architecture as the rest of the family: the AI step does not generate content clipped to the finger quad. It generates a region of the frame — large enough to cover everywhere the window travels across the clip, plus the head — in one pass, and the tracked quad is used only as a screen-aligned reveal mask on top of that already-generated footage. Moving the window around does not re-trigger generation, it just moves the mask over material that already exists.

1. Track (FingerWindowTrack) — MediaPipe Hand Landmarker finds both hands per frame; the finger-frame quad is tracked with the family's pipeline (anatomical corner ordering, spread/area gates with hysteresis, teleport rejection, velocity-adaptive smoothing, dropout hold, presence fade).
2. Region (FingerWindowRegion) — computes the union of everywhere the tracked window travels, plus the head, and crops that down to a generation budget (target_pixels, default 832×480, snapped to multiples of 32). The output resolution follows the crop's own aspect ratio, not a fixed orientation — feed a landscape or a portrait driving video and the generated region comes out landscape or portrait to match, automatically. (Only landscape has been tested so far in practice.)
3. Generate — the cropped region goes through your Wan2.1 SCAIL-2 pipeline. Two reference images are used: HEAD.png (a head-only crop — identity) and BODY.png (a waist-up crop — outfit/pose reference), each run through background removal before SCAIL-2. Confirmed to work with both photoreal and illustrated/anime reference characters.
4. Paste (FingerWindowPaste) — pastes the generated region back to full-frame coordinates. Character-swap models place the generated head where they want, not where the real head was, so the composited head can drift from the real one. auto_align (new) measures that drift per frame from the eye-line of both faces and corrects it with a rigid move+scale of the paste rectangle — the generated pixels are never deformed, only repositioned.
5. Composite (FingerWindowComposite) — screen-aligned reveal through the tracked quad. The refraction was reworked to a rim-only bevel: the middle of the pane stays perfectly aligned with the real footage, and the lens bulge / motion drag / chroma aberration only affect a border strip near the window's edge — so nothing outside the window can leak in, and the content doesn't swim when the window is large.
6. Preview (FingerWindowPreview) — debug overlay: window quad in green, generation region in magenta, presence value.

Nodes

Node	Purpose
FingerWindowTrack	Tracks the two-hand quad across a batch of driving frames
FingerWindowRegion	Crops the union-of-motion + head region for generation, aspect-matched
FingerWindowPaste	Pastes a cropped region back to full-frame coordinates, with optional face-drift auto-align
FingerWindowComposite	Screen-aligned reveal with rim-only refraction
FingerWindowPreview	Debug overlay (quad + region + presence)
FingerWindowRefCrop	Face-aware framing of a reference head photo (utility, not part of the core pipeline)

Requirements

• ComfyUI, a GPU with enough VRAM for Wan2.1 14B i2v (tested on an RTX 4090 24GB / 64GB system RAM)
• Python packages: mediapipe, opencv-python (see requirements.txt)
• MediaPipe hand-landmark and face-detection models auto-download on first run; no manual download needed
• A Wan2.1 SCAIL-2 i2v pipeline (e.g. ComfyUI-WanVideoWrapper / ComfyUI-WanAnimatePlus), tested with:
  • Checkpoint: wan2.1_14B_SCAIL_2_fp8_scaled.safetensors
  • VAE: wan_2.1_vae.safetensors
  • CLIP vision: clip_vision_h.safetensors
  • LoRAs: wan2.1_SCAIL_2_relight_lora_bf16, lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16, wan2.1_SCAIL_2_DPO_lora_bf16
  • Reference segmentation: sam3.1_multiplex_fp16.safetensors

The FingerWindow* nodes themselves have no model-specific dependency — they only pass IMAGE batches in and out, so any i2v/v2v pipeline that takes a driving video and a reference image can sit in the generation step instead of SCAIL-2.

Install

The node code lives in the ComfyUI-FingerWindow-Local subfolder of this repo — clone the repo somewhere, then copy just that subfolder into custom_nodes (cloning the whole repo directly into custom_nodes creates a double-nested folder that ComfyUI won't pick up):

git clone https://github.com/stark622/ComfyUI-FingerWindow-Local-SCAIL2.git
cp -r ComfyUI-FingerWindow-Local-SCAIL2/ComfyUI-FingerWindow-Local ComfyUI/custom_nodes/
cd ComfyUI/custom_nodes/ComfyUI-FingerWindow-Local
pip install -r requirements.txt

Restart ComfyUI.

Workflow

workflows/finger_window_SCAIL2_public_v2.json — two-hand gesture tracked on a driving clip, region generation through Wan2.1 SCAIL-2, composited back with head-drift auto-align. v2 adds WanAnimatePlus SCAIL_2 TwoPhase Settings into the sampling graph — see "Tested on" above for the speed difference this made. Load your own driving video into the VHS_LoadVideo node, and your own HEAD.png / BODY.png reference images into the two LoadImage nodes.

examples/video_example_clean.mp4 — a rendered output sample.

tools/CONVERTER.bat (optional)

A drag-and-drop helper for prepping phone footage. Drop any video file onto it (anything ffmpeg can read — iPhone .mov, .mkv, etc.), pick a preset (1920×1080, 1280×720, or 720×1280 portrait), and it re-encodes to h264/yuv420p with the content letterboxed (not cropped) into that resolution — a safe, known-good input format for the workflow. Requires ffmpeg on PATH. Windows only (uses PowerShell for the preset picker).

Status / known limitations

• Tested with Wan2.1 SCAIL-2 specifically; other i2v backends untested but should work since the FingerWindow nodes are model-agnostic
• Region generation currently assumes a single continuous take (one gesture session per clip)
• Auto-resolution has only been exercised on landscape driving footage in practice, though the sizing logic itself is orientation-agnostic
• No install via ComfyUI-Manager registry yet — git clone only

License

MIT — see LICENSE. Copyright notice (and with it, authorship) must be kept in copies/forks; beyond that, do what you want with it, including selling.