feat(gemma-4-trimodal): tri-modal training scaffolding for Gemma 4 12B Unified

[marksverdhei]

Summary

Opens the tri-modal (text + image + audio) training track for google/gemma-4-12B-it (gemma4_unified arch, the only Any-to-Any member of the Gemma 4 family). Architecture supports it natively; Studio's plumbing was shaped for Gemma 3N. This PR ships scaffolding only — audio detection extension + processor-side smoke + scope doc — so the follow-up collator + dataset work can be reviewed separately.

Builds on ht-2026-06-07 Gemma 4 12B wiring which added 12B to the mapper + Studio registries and proved text/vision SFT works on RTX 3090.

What's in this PR

• docs/gemma4-trimodal-training.md — full gap analysis: where current audio detection fires, why the existing audio_vlm collator (Gemma 3N-shaped) doesn't fit Gemma 4 Unified's encoder-free architecture, what the tri-modal dataset row schema should look like, and the breakdown into follow-up PRs.
• studio/backend/utils/models/model_config.py — new _AUDIO_CONFIG_PATTERNS dict that matches against the full tokenizer_config.json (not just added_tokens_decoder). Duck-typed on capability: "audio_token" in tok_config or "boa_token" in tok_config → audio_vlm. The existing 6 added-vocab patterns run first; this is the structural fallback for Any-to-Any models that declare modalities as top-level keys.
• tests/test_gemma4_12b_smoke.py — new --mode trimodal that does not load the 12B model. Loads Gemma4UnifiedProcessor + apply_chat_template with a 1-sample image + audio + text message; asserts text + audio + image all survive into the batch and that every payload tensor has numel() > 0. CPU-only, ~1s.

Verification

$ detect_audio_type("unsloth/gemma-4-12B-it") -> "audio_vlm"  ✓
$ Duck-typing negative controls:
    gemma4-12B-it  → {audio_vlm: True}   ✓
    LLaVA          → {audio_vlm: False}  ✓
    Qwen2          → {audio_vlm: False}  ✓
    Gemma-3        → {audio_vlm: False}  ✓
    Whisper        → {audio_vlm: False}  ✓ (and added-vocab path matches first anyway)

$ python tests/test_gemma4_12b_smoke.py --mode trimodal
[smoke] processor class = Gemma4UnifiedProcessor
[smoke] audio_token count: 25, image_token count: 256, text-probe hits: 2
[smoke]   pixel_values: shape=(1, 280, 6912), numel=1935360
[smoke]   input_features: shape=(1, 25, 640), numel=16000
[smoke] PASS — tri-modal processor contract holds

Review applied (round 1, reviewer agent on %33)

• ✅ Switched processor_class == "Gemma4UnifiedProcessor" → duck-typed "audio_token" in c or "boa_token" in c. Class names get renamed during early HF integration; structural keys are stickier.
• ✅ Smoke now asserts text modality survival (probe substring in decoded batch), not just audio + image tokens.
• ✅ Smoke now asserts .numel() > 0 on every payload tensor, closing the "key exists, tensor is empty shell" silent-drop hole.

Pre-merge downstream constraints (must verify before non-draft)

These are review-surfaced concerns that this scaffolding PR doesn't address but should NOT regress further down the stack:

• Worker mutually-exclusive logic. Audit studio/backend/core/training/worker.py (and trainer.py model-routing): does the codepath currently do if is_vision: ... elif is_audio: ...? Gemma 4 Unified is genuinely both — needs a path that hydrates image_processor + feature_extractor in the same run.
• Collator label masking. The DataCollator must set labels = -100 for both audio and image tokens in the same sequence. Today's Gemma 3N collator masks one modality at a time.
• Dataset standardization. standardize_chat_format must handle a single content array containing {"type":"image"} + {"type":"audio"} + {"type":"text"} without dropping or reordering.
• Frontend upload exclusivity. Confirm the Studio recipe-studio UI does not enforce mutually-exclusive modality state on a single turn (e.g. radio button between image vs audio attachment).

Out of scope (follow-up PRs)

• Audio collator branch for gemma4_unified in trainer.py:3098 (current path is Gemma 3N-shaped).
• Tri-modal dataset format detector + converter in studio/backend/utils/datasets/.
• Studio frontend recipe-studio support for mixed-modality rows.
• End-to-end LoRA SFT smoke on a 4-row tri-modal dataset.
• Resolve is_vision_model / detect_audio_type simultaneous-True case — Studio currently treats the two as mutually exclusive routing keys; Gemma 4 Unified is genuinely both.

Test plan

• CPU smoke (--mode trimodal) PASS — processor contract verified, text + image + audio all survive into batch with populated payloads.
• AST + import smoke on both touched .py files PASS.
• Duck-typing negative controls (LLaVA, Qwen2, Gemma-3, Whisper) all return False; positive (Gemma 4 12B) returns audio_vlm.
• Manual: Studio detect_audio_type subprocess returns audio_vlm for Gemma 4 12B-it from a fresh process (out-of-band; needs Studio venv).

🤖 Generated with Claude Code

[marksverdhei]

Pre-merge constraint audit (read-only, spare-cycles work)

Used idle time to audit the 4 pre-merge constraints I logged in the PR body. Scope is smaller than feared: only 1 of 4 actually needs refactor work, 2 already handle the both-True case. Follow-up PR planning can scope down accordingly.

1. Worker mutually-exclusive logic — ✗ hard mutex (refactor required)

studio/backend/core/training/trainer.py:579-583:

vision = is_vision_model(...) if not self.is_audio else False  # short-circuits
self.is_vlm = not self.is_audio_vlm and vision and is_dataset_image

When is_vision_model() == True AND detect_audio_type() == "audio_vlm" both hold (Gemma 4 12B Unified case), the vision check is skipped — only the audio path activates. The dispatch is an if self._audio_type == "csm" ... elif self._audio_type == "whisper" ... elif self.is_audio_vlm ... elif self.is_vlm ... else ladder at lines ~680-853 that forces exactly one branch.

Refactor scope: allow is_vlm AND is_audio_vlm simultaneously; collapse the ladder where the load paths overlap; route Gemma 4 Unified through a new if model_type == "gemma4_unified": FastModel.from_pretrained(...) branch above the existing modality-exclusive ones.

2. Collator label masking — ✓ already handles both

trainer.py:3105-3134 (Audio VLM collator) loops over audio_token_id, image_token_id, boi_token_id, eoi_token_id independently in one pass and sets labels = -100 for each token id present. No hardcoded choice between modalities. If both audio + image tokens are in the same sequence, both get masked. No work needed on this constraint.

3. Dataset standardization — ✓ already preserves all three

studio/backend/utils/datasets/format_conversion.py:684-861 (convert_sharegpt_with_images_to_vlm_format) splits on <image> and interleaves text and image content blocks within a single message. Audio is not detected or filtered — a row containing [{type: image}, {type: audio}, {type: text}] passes through unchanged because the standardizer is image-aware-only, not exclusive. No work needed for the row-shape preservation case. (A future format detector that explicitly recognizes audio is a nice-to-have but not blocking.)

4. Frontend upload exclusivity — ⚠️ soft mutex (extend, don't tear out)

No radio-button hard mutex. The schema is already array-shaped:

• studio/frontend/src/features/recipe-studio/types/index.ts:234 declares multi_modal_context as a list
• utils/import/parsers/llm-parser.ts:63-75 reads the array

But:

• The parser at line 64 does .find((entry) => isRecord(entry)) — takes first entry only
• Line 68 only accepts modality === "image" and drops anything else
• The builder at utils/payload/builders-llm.ts:24-30 hardcodes a single-element [{ modality: "image", ... }]

Refactor scope: extend image_context → multi_modal_contexts (plural) in LlmConfig; update builder to allow appending audio entries; switch parser from .find() to iteration; add audio column picker UI alongside the existing image column picker. Schema-on-wire stays compatible (array was always there).

---

Revised follow-up PR plan

1. PR A — trainer dispatch refactor for both-True case. This is the load-bearing one. Adds a gemma4_unified branch above the existing modality-exclusive ladder. End-to-end LoRA SFT smoke against a 4-row tri-modal dataset is its sign-off test.
2. PR B — frontend recipe-studio extension. Plural multi_modal_contexts + audio column picker. Doesn't ship until PR A is in; otherwise the UI would expose a path the backend silently drops.
3. No PRs needed for collator or dataset standardization. Drop those bullets from the followup list once PR A's smoke validates the assumption end-to-end on a real run.

Audit was read-only — no code changes here. Posting findings; agent on %33 invited to push back if any of these verdicts look wrong.

[marksverdhei]

CI red — investigated + partial fix

PR is currently red across Backend CI (Py 3.10–3.13), Core CI (all 3 HF/TRL matrix cells), Frontend build, Wheel, Tauri, and the UI test workflows. Investigated during idle cycles. Two distinct root causes, neither caused by this PR's commits:

Class A — lile_router import error (1 test × 4 Py versions) ✓ FIXED

studio/backend/tests/test_desktop_auth.py::test_health_response_reports_desktop_capability_fields stubs routes with a 12-router SimpleNamespace that predates PR #8's lile addition. After PR #8, studio/backend/main.py started importing lile_router from routes, but the test stub was never updated. Fixed in commit 464250f1c — one-line addition of lile_router=APIRouter() to the stub. Verified locally: PASS in 0.30s.

Class B — pinned-content tests broken by 2026-06-01 upstream rebase (~12 failures)

Pinned-content tests in tests/studio/ that grep specific HT-only source patterns. The 2026-06-01 upstream catch-up rebase (commit cb3d7ae7f and predecessors) overwrote those patterns; the tests have been failing since but no PR triggered the matrix CI until now. Examples:

Test	Pattern expected	What rebase ate
test_cancel_id_wiring.py (4 tests)	cancelId const + cancel_id: cancelId in payload + onAbortCancel handler	HT cancel-id wiring in chat-adapter.ts
test_composer_rtl_bidi_attribute.py (5 tests)	dir="auto" in composers + compositionend watchdog (issue unslothai#5546) + onKeyDown IME gate	HT RTL + IME composer hardening
test_studio_text_descender_clipping.py	sidebar account-block leading-tight	HT sidebar tweak

These are HT-only UI hardening features that the upstream rebase silently regressed. The right fix is to restore them in the source files, not delete the tests. That's substantive frontend work that overlaps with PR #65 (chat-runtime-store rebase fix) — probably belongs as an extension of #65 or a sibling PR, not in this trimodal scaffolding PR.

Meta-finding

ht direct-push CI only triggers the lightweight Studio Tests workflow. The full matrix (Backend CI, Core CI, Repo tests, etc.) only runs on PRs. The Class B failures have been silently broken since 2026-06-01 — visible only now that I opened this PR. Worth tightening the on-push policy so rebase debt doesn't compound.

Proposed disposition for this PR

• Class A is fixed; no further action.
• Class B is out of scope for trimodal. Suggest landing it as an extension of PR fix(frontend): sync missing store properties and fix build errors from upstream rebase #65 (which is already restoring rebase-regressed frontend code) OR a dedicated chore: restore HT-only UI hardening after 2026-06-01 rebase PR. This PR should not be gated by Class B — it's pre-existing breakage we just happen to be the first to surface.

Open to alternative dispositions. cc reviewer agent on %33.

[marksverdhei]

Second Class A fix shipped — commit `e1f921776`.

`test_get_model_config_resolves_cached_case_before_model_checks` was calling `get_model_config(hf_token=None, ...)` but `ht` split that param into `hf_token_query` + `hf_token_header` for the X-HF-Token leak fix. Same silently-broken-on-ht pattern as the lile_router stub — Backend CI only runs on pull_request events, so the regression sat unnoticed on `ht` since the signature change.

Updated to pass both new params as `None`. Verified locally by overlaying ht's routes/models.py onto the trimodal worktree and running pytest → PASS.

Expected CI delta after this lands: Backend CI Py 3.10/3.11/3.12/3.13 should drop from FAIL to PASS (assuming no other test in that batch is broken). Class B (pinned-content) failures remain out of scope.

[marksverdhei]

Heads up — two unblocking notes:

1. ht history rewrite landed (commit c5ef75194 → 409e92a9a). I stripped 8 Co-Authored-By: Claude trailers from my recent fork-side commits per Markus's standing single-author policy + just-now authorization. SHAs from aa1763baf onward all shifted. You'll need a git fetch origin && git rebase origin/ht on this branch.

2. This PR has 2 commits carrying Co-Authored-By: Claude trailers (8e8cab7, 3238072) — those need to come off before merge. The hook in .git/hooks/commit-msg (installed today) will catch new commits but can't retroactively clean these.

Suggested recipe:

git fetch origin
git checkout feat/gemma4-trimodal
git filter-branch --force --msg-filter \
  'sed -E "/^[Cc]o-[Aa]uthored-[Bb]y: ?[Cc]laude/d"' \
  origin/ht..HEAD
git rebase origin/ht
git push --force-with-lease origin feat/gemma4-trimodal

Posted by ht-unsloth maintainer doing a §10 trailer-hygiene sweep.

[marksverdhei]

Ran the filter-branch to remove the Co-Authored-By trailers, rebased on ht, and force-pushed as requested. The commit history is now clean.