[Feature] Port IntDim-E scoring and shrinking onto intdim, retire per-family scripts, and unify on Transformers 5.17 #25

Problem

#24 added less_is_moe.intdim with structural discovery (discover) and stock-load verification (verify), but nothing uses it yet. IntDim-E pruning still runs through eight per-family scripts, and the repository still needs three environment profiles with incompatible Transformers versions.

Per-family pruning code. src/less_is_moe/pruning/ holds eight scripts (4,126 lines) that differ mainly in how they locate expert weights:

They are coupled to other modules:

• Each neuron_drop_* imports calibration and model helpers from the matching expert_drop_*.
• neuron_structure_drop_{olmoe,qwen15_moe} import from their neuron_drop_*.
• Callers: the eight launchers in scripts/prune/, README.md, docs/SCRIPT_REFERENCE.md, and docs/MIGRATION.md.
• The structural scripts share 13 CLI flags; two also accept --calib_preset / --calib_preset_split.

Three Transformers versions. Each profile pins a different stack:

Transformers 4.x stores experts as per-expert nn.Linear lists and 5.x as fused 3D tensors, which is why the scripts carry both layouts. model_patches/hf/ also keeps 3,058 lines of Transformers 4.x-era modeling copies, including a 1,745-line modeling_qwen2_moe_old.py. The roadmap backbones (gpt-oss-120b, Qwen3.5-122B-A10B, gemma-4-26B-A4B) require Transformers 5.x.

Proposal

1. Port scoring and shrinking onto MoeLayerHandle. (@HectorHHZ Done in #26)

• score: freeze all parameters, enable gradients on the handle's expert tensors only, and accumulate mean(abs(gradient)) per intermediate unit from gate, up, and down through expert_weights. The criterion does not change.
• select: the existing IntDim-E (per-expert), IntDim-L (per-layer), and IntDim-G (global) scopes. Structural output stays IntDim-E only, as today.
• shrink: apply_units followed by updating intermediate_size_key, then verify on the saved checkpoint.
• One CLI, python -m less_is_moe.intdim.prune, exposing the union of today's flags, including --from_zeroed_model and --calib_preset.
• Move the calibration loaders shared by neuron_drop_* and expert_drop_* into one module, so the expert-drop baselines keep working after the neuron scripts are removed.

2. Prove equivalence tensor by tensor before deleting anything. For each of the four supported families, on a tiny model and on the real checkpoint with a fixed calibration set (@jayzou3773 Done in #27; #28):

• per-unit importance scores from the old and new code are identical (same dtype, same seed);
• the kept indices are identical for every scope;
• every tensor in the saved structural checkpoint is bitwise identical, including the config;
• a masked (neuron_drop_*) checkpoint is identical as well.

Keep the comparison as a regression test that pins a small fixture of old outputs, so it survives after the old scripts are deleted.

3. Retire the per-family scripts. Replace the eight launchers with thin wrappers around the new CLI, keeping their names and flags for one release, and update README.md, docs/SCRIPT_REFERENCE.md, and docs/MIGRATION.md @jayzou3773 @HectorHHZ .

4. Unify on Transformers 5.17. Replace the three profiles with one (@jayzou3773 Done in #27; #28):

This removes the Transformers metadata overrides in setup.sh, the per-expert nn.Linear code paths, and the 4.x modeling copies in model_patches/hf/. Whatever runtime patching remains for expert-drop baselines and router masks must be rechecked against vLLM 0.29.0.

Would this change the reported protocol or a reported number?

Steps 1–3: no. The criterion, scopes, and outputs stay the same, and step 2 proves it.

Step 4: possibly. The paper's numbers were produced on the three separate stacks above, and CONTRIBUTING treats environment isolation as part of the reported protocol. Before the old profiles are removed:

• rerun base and Fisher-IntDim-E evaluation for Qwen1.5-MoE, OLMoE, Qwen3-MoE, and Qwen3.5-35B-A3B on the unified stack;
• report any difference against the paper's tables, separating generation changes (vLLM, tokenizer, chat template) from pruning changes;
• keep the old profiles available, documented as the paper's reproduction stacks, until the differences are understood.

Current compatibility of intdim

tests/test_intdim.py on commit 8b7ed01 of #24, in two environments:

In the legacy run, Qwen1.5-MoE and OLMoE are discovered as per-expert ModuleList experts with the correct config keys (moe_intermediate_size and intermediate_size), and pass the save → verify → reload → re-discover round trip. The 13 skips are families whose config classes do not exist in Transformers 4.49 (Qwen3-MoE, Qwen3.5-MoE, gpt-oss, Gemma-4). The legacy environment omitted vLLM and flash-attn, which the tests do not use. The qwen3 profile (Transformers 4.53.1) has not been tested.

Done when

• python -m less_is_moe.intdim.prune produces bitwise-identical scores, kept indices, and checkpoints to the old scripts for all four families, enforced by a regression test.
• The eight neuron_drop_* / neuron_structure_drop_* modules are removed and their launchers call the new CLI.
• One environment profile on Transformers 5.17.0, vLLM 0.29.0, and PyTorch 2.13.0 runs pruning, verification, and evaluation for all supported families.
• Reported numbers are re-verified on that profile, with any difference explained.

Depends on #24. Part of #2