Load the text tower of dual-registered composite configs

[can-goodfire]

The problem

StandardizedTransformer("Qwen/Qwen3.6-35B-A3B") fails before any renaming happens:

ValueError: 'Qwen/Qwen3.6-35B-A3B' (qwen3_5_moe) is registered with
AutoModelForImageTextToText — it's a multimodal model so LanguageModel(...)
can't load it. Use VisionLanguageModel instead

nnsight's LanguageModel._check_is_text_only refuses any model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES, on the premise that AutoModelForCausalLM would then fail because the text fields sit under config.text_config.

That premise does not hold for every such config:

model_type	ImageTextToText	CausalLM
qwen3_5_moe	Qwen3_5MoeForConditionalGeneration	Qwen3_5MoeForCausalLM
qwen2_vl	Qwen2VLForConditionalGeneration	None
llava	LlavaForConditionalGeneration	None

qwen3_5_moe is registered in both tables. AutoModelForCausalLM resolves to the text tower and loads it cleanly. Genuine VLMs have no causal-LM mapping at all, which is the case the guard is actually for — so the refusal is over-broad, and it locks nnterp out of a whole architecture family (Qwen3.6-35B-A3B and its siblings).

What this does

nnterp/text_tower.py detects exactly that case — model_type present in both auto mappings — and opts out of the refusal using the escape hatch the guard documents for it: it stands down for any non-default automodel, so nnterp hands it a marker subclass of AutoModelForCausalLM that dispatches through the identical _model_mapping.

New text_tower: bool = True kwarg on StandardizedTransformer. Ordinary text models and multimodal-only configs take neither branch; text_tower=False restores nnsight's own handling. The vision tower is never loaded, by design.

A failure to read the config ahead of time returns None rather than raising, so looking ahead can never be the thing that fails a load — nnsight still reports the real error.

Verified

On tiny-random/qwen3.5-moe (transformers 5.16.0, nnsight 0.7.0):

• loads as Qwen3_5MoeForCausalLM; hasattr(model, "visual") is False
• num_layers 4, hidden_size 8, vocab_size 248320, num_heads 8 all resolve — the existing text_config() helper already handles the composite config
• token_embeddings, layers_output[0] and logits all trace correctly
• 8 new offline unit tests (they discover model_types from the auto-mapping tables rather than hard-coding them, so they describe the contract instead of pinning a release)
• no regression: gpt2, Maykeye/TinyLLama-v0 and yujiepan/qwen3-moe-tiny-random still load with full renaming validation

Scope

This fixes loading only. check_renaming=True on this architecture then hits the next problem — RenamingError: Could not find self_attn module, because layer_types = ([linear_attention] * 3 + [full_attention]) * 10 means layers[0] is a Gated DeltaNet layer with no self_attn. That is the hybrid-layer work in #18 and is deliberately not touched here.

An alternative worth considering

The cleaner fix is upstream in nnsight — narrow the guard to refuse only when there is no causal-LM mapping to fall back on:

         try:
             from transformers.models.auto.modeling_auto import (
+                MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
                 MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES,
             )
         except ImportError:
             return

         model_type = getattr(self.config, "model_type", None)
-        if model_type and model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES:
+        if (
+            model_type
+            and model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES
+            and model_type not in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
+        ):
             raise ValueError(

I applied that to nnsight 0.7.0 locally and confirmed StandardizedTransformer(..., text_tower=False) then loads the text tower with no nnterp involvement. Happy to file it there instead or as well — but nnterp needs to work against released nnsight either way, which is what this PR provides.

Unrelated finding, reported separately

yujiepan/qwen1.5-moe-tiny-random — in nnterp's own core_test_models — fails to load under transformers 5.16 + fp32 with RuntimeError: Expected inputs of BF16 type from torch._grouped_mm on the fake-tensor path, so scan() fails and the trace() fallback then fails the IO check. Pre-existing at b4a3127 and independent of this change (qwen2_moe is not dual-registered, so this code path never engages, and text_tower=False fails identically).

🤖 Generated with Claude Code

[Butanium]

bug doesn't happenb in the dev branch, but i added an option in #57 to only load the vision tower