Add end-to-end video input to the base Gemma 4 (gemma4) VLM

[fdagostino]

Summary

Add end-to-end video input to the base Gemma 4 VLM (model_type: "gemma4" — E2B/E4B). Gemma 4 has no separate video encoder: sampled frames run through the same vision tower as images, are trimmed to a smaller per-frame budget, and scatter onto the <video> soft-token positions. This wires both halves — the model forward and the Gemma4Processor — so a UserInput carrying video actually reaches the model (previously it was silently dropped).

What was missing

An earlier revision made the Gemma4 model accept videoPixelValues but nothing produced them: Gemma4Processor.prepare ignored input.videos, so LMInput.video was never populated. This PR completes the pipeline.

Changes (Libraries/MLXVLM/Models/Gemma4.swift)

Model side

• Gemma4Configuration decodes vision_soft_tokens_per_video_frame (default 70).
• getInputEmbeddings routes video through a new scatterVideoFeatures helper: each frame goes through the shared vision tower (producing visionSoftTokensPerImage pooled tokens), the first visionSoftTokensPerVideoFrame (70) are kept, and the resulting frames × 70 tokens scatter onto video_token_id. This mirrors the reference Swift port (VincentGourbin/gemma-4-swift-mlx, which pools to the image budget then truncates) and the pooling behaviour of Blaizzy/mlx-vlm.

Processor side

• Gemma4ProcessorConfiguration decodes video_token_id (258884), video_soft_tokens_per_frame (70), video_max_frames (32), and exposes a videoFixedSize (432×432 — a multiple of patch_size · pooling_kernel_size = 48, giving ~81 patch-pooled tokens that trim to 70 so the kept tokens cover the frame).
• Gemma4Processor.processVideos samples ≤32 frames at ~1 fps via MediaProcessing.asProcessedSequence (handles .frames / .url / .avAsset), preprocesses each frame to the video size (sRGB → bicubic resize → normalize), and stacks them into [totalFrames, C, H, W].
• Gemma4Processor.prepare now processes input.videos, sets LMInput.video, and expands each <video> placeholder into one BOI + video_token×70 + EOI block per sampled frame (per-video frame counts drive the expansion).

Tests (Tests/MLXLMTests/Gemma4VideoInputTests.swift)

1. Config decode — video_token_id / vision_soft_tokens_per_video_frame decode with the right defaults.
2. Image/video equivalence — with the budget ≥ the tower output, the same pixels as image vs. video give identical logits (video reuses the vision tower, scatters at video_token_id).
3. Multi-frame truncation — a 2-frame clip with budget 2 scatters exactly 4 tokens through the language model.
4. Processor frame extraction — processVideos on synthetic frames yields [F, C, 432, 432] with per-video counts summing to the frame axis.

Gemma4ChunkedPrefillTests and Gemma4UnifiedTests still pass. End-to-end decode from a real clip is exercised on-device by the downstream app (the same way image and audio were validated on iPad).

Scope / follow-ups

• Timestamps: the reference prepends a per-frame mm:ss marker inside each block; this PR emits the frame blocks without the timestamp text (temporal grounding refinement — easy follow-up).
• Gemma4Unified (12B) processor still lacks a video block. That path is encoder-free (per-frame patch tensors + position ids, a different mechanism than the E-model vision tower), so it's tracked separately; the base gemma4 E2B/E4B path (what most on-device use targets) is complete here.

Relation to #390 / #392

Independent of #390 (KV-shared load fix) but the gemma4 E4B model won't load without it. The audio PR #392 vendors the video model-level change and will need to pick up these processor changes on rebase.

---

🤖 Generated with Claude Code

[davidkoski]

OK, so the plan would be to take #390 (merged), then this, then #392?

[davidkoski]

fix merge conflict

[davidkoski]

Not a blocker, but this samples 1 frame per second from the video, up to the maxFrames (default is 32). If the video is longer, this will just take the prefix. mlx-vlm samples uniformly across the video.

I think the fix is just doing the arithmetic in targetFPS to match the frames desired vs the length. It looks like all the VLMs do something similar (fixed FPS), so I think we need something at the MediaProcessing level to assist with this and we can do all the models at once.

[davidkoski]

#429

[davidkoski]

Changes look good to me. Thank you!

[davidkoski]

Looks like this needs swift-format run.

[fdagostino]

Rebased on main and ran swift-format — the conflict was semantic (this PR predated the aspect-preserving image path from #405), so the resolution keeps #405's per-image slicing for stills untouched and routes video frames through the per-frame-budget scatter only. Also adapted the tiny-config tests to the new tower geometry (patch-count tokens) and dropped a helper #405 made redundant. 259 tests pass locally with the CI invocation.