Complete record of all ABI experiments. T5-large same-backbone experiments (Path 2C) use scripts cross_arch_t5_nib_v*.py. Cross-architecture experiments are standalone scripts.
| Exp | Source | Target | Method | top-5 | top-1 | JS | Ent | Status |
|---|---|---|---|---|---|---|---|---|
| Path 2C (45AS) | T5-large (enc-dec) | T5-large (enc-dec) | corrMSE (same backbone) | 0.8725 | 0.8508 | 0.01391 | 0.2256 | PASS |
| Exp 32 | GPT-2-small (dec) | Qwen2.5-0.5B (dec) | Procrustes + KD | 0.8701 | 0.9057 | 0.01123 | 0.2348 | PASS |
| Exp 39 | T5-large (enc-dec) | GPT-2-medium (dec) | Procrustes + KD | 0.8699 | 0.9252 | 0.01787 | 0.2819 | PASS |
All three transfer directions pass NIB. Enc-dec ↔ dec-only migration: VALIDATED (Exp 39, 2026-05-16).
| Exp | Architecture | Update steps | Stability | Efficacy | Pass |
|---|---|---|---|---|---|
| succession_test_v2 | GPT-2-medium (dec-only, 354M) | 1000 | MSE + proj_in frozen | 65% | PASS |
| Exp 40 | T5-large (enc-dec, 730M) | 1000 | MSE (pre-computed refs) + proj_in frozen | 304% | PASS |
Backbone-update invariance validated for both decoder-only and encoder-decoder architectures.
| Exp | Script | Architecture | Objective | corrMSE | Ext top-5 | vs 45AS | Valid? |
|---|---|---|---|---|---|---|---|
| 45AS | v53 | 6-tap [19-24], per-tap LN, D=4096 | corrMSE | 0.003047 | 0.8549 | — | YES — BEST |
| 45AT | v54 | same as 45AS, P4 extended (18k steps) | corrMSE | 0.003243 | 0.8496 | -0.0053 | yes |
| 45AU | v55 | 9-tap [16-24], per-tap LN, D=4096 | corrMSE | 0.003122 | 0.8506 | -0.0043 | yes |
| 45AV | v56 | 6-tap+1 enc mean-pool, per-tap LN, D=4096 | corrMSE | 0.003129 | 0.8457 | -0.0092 | yes |
| NO — numerical bug | |||||||
| 45AX | v58 | 6-tap [19-24], per-tap LN, D=4096 | raw KL (T=2, LR/10) | 0.002659 | 0.8244 | -0.0305 | yes |
| 45AY | v59 | 6-tap [19-24], per-tap LN, D=4096 | weighted corrMSE (all phases) | 0.003175 | 0.8472 | -0.0077 | yes |
| 45AZ | v60 | 6-tap [19-24], per-tap LN, D=4096 | curriculum (std P1-P3, weighted P4) | 0.003168 | 0.8446 | -0.0103 | yes |
Conclusion: 45AS is the global optimum. Every intervention made things worse or identical.
Transfer pair:
- Source: T5-large (730M, encoder-decoder, SentencePiece 32128 vocab, relative position encoding, cross-attention FFN with ReLU)
- Target: GPT-2-medium (354M, decoder-only, BPE 50257 vocab, absolute position encoding, causal MHA with GELU FFN)
Architecture:
- d_abi: 256 (shared bottleneck for both models)
- T5 tap: single-tap on decoder final hidden state (layer 24), d_model=1024
- GPT-2 tap: single-tap on final hidden state, d_model=1024
- T5 mode: prefix-LM (encoder=64-token prefix, decoder predicts 64-token continuation)
- Domain module:
Linear(256→1024)+GELU+Linear(1024→256)+LayerNorm(256), DomainModuleSV - proj_in:
Linear(1024→256, bias=False), proj_out:Linear(256→1024, bias=False)
Protocol:
- Phase A: T5+ABI domain pre-training on Python, 500 steps, LR=3e-4, backbone frozen
- Phase C: GPT-2+ABI oracle training on Python, 500 steps, LR=3e-4, backbone frozen
- Alignment: sentence-level mean-pool orthogonal Procrustes, 2000 sentences (each model tokenized independently with its own tokenizer)
- KD calibration: 1200 steps, kd_weight=0.90, kd_temp=2.0, LR=1e-4
Results (rng=7777, n=5 chunks × 512 tokens = 2460 evaluated positions):
- mean_js: 0.01787 (threshold < 0.100) — PASS
- mean_top1_agree: 0.9252 (threshold >= 0.680) — PASS
- mean_top5_overlap: 0.8699 (threshold >= 0.860) — PASS
- mean_entropy_diff: 0.2819 (threshold < 0.350) — PASS
- ppl_native_gpt2: 8.013, ppl_calibrated_gpt2: 8.576
- overall_pass: true
- Elapsed: 7.4 min (RTX 3080 Laptop)
- Result file:
cross_arch_enc_dec_nib_results.json
Key finding: An orthogonal Procrustes map between sentence-level mean-pooled ABI representations (d=256) is sufficient to align the ABI spaces of T5-large and GPT-2-medium, despite differences in architecture class, tokenizer, vocabulary size, position encoding, and attention mechanism. The 1200-step KD calibration fine-tunes only the projection matrices; the domain module trained on T5 is transferred unchanged. NIB evaluated in GPT-2's native 50257-token vocabulary.
This validates: Encoder-decoder ↔ decoder-only frozen-module migration. Combined with Exp 32 (GPT-2-small → Qwen2.5-0.5B, cross-family decoder-only, PASS) and Path 2C (T5 same-backbone, PASS), all three primary cross-architecture transfer directions have now been validated by NIB PASS.
Test: Does a domain module trained on T5-large (Phase A, 500 steps, Python) survive a 1000-step WikiText-2 backbone fine-tune with ABI stability constraint?
Protocol:
- Phase A: T5-large+ABI, Python domain, 500 steps, LR=3e-4, backbone frozen. ABI: d_abi=256, single-tap, prefix-LM mode
- Reference cache: pre-compute h_abi for all 1000 Phase B batches at the Phase A checkpoint (262 MB on CPU). This is the exact frozen reference for stability MSE
- Phase B: unfreeze encoder+decoder, freeze all ABI (proj_in, abi_ln, proj_out, domain). LM loss via
forward_raw(pure backbone, no ABI path). Stability loss: MSE(h_abi_new, h_abi_cached_ref). alpha=1.0, LR=5e-5 - Zero-shot: re-attach Phase A domain state to updated backbone, measure Python PPL
- Phase C: cold-start oracle — reload original T5, fresh ABI, 500 steps Python
Results:
| Checkpoint | Python PPL |
|---|---|
| Raw T5 baseline | 63.73 |
| Phase A domain (pre-update) | 29.61 |
| Raw backbone (post WikiText update) | 35.22 |
| Zero-shot: Phase A domain on updated backbone | 25.61 |
| Cold-start oracle (original backbone) | 32.06 |
- WikiText PPL: 41.6 → 14.7 (backbone improved significantly on WikiText)
- Stability loss: 0.041 → 0.037 (h_abi drift suppressed throughout)
- Transfer efficacy: (35.22 − 25.61) / (35.22 − 32.06) = 304.3% (threshold ≥ 50%)
- Elapsed: 9.9 min (RTX 3080 Laptop)
- Result file:
cross_arch_t5_succession_results.json
Key finding: Efficacy exceeds 100% because ppl_zero_shot (25.61) < ppl_cold_start (32.06). The WikiText backbone update improved T5's general linguistic representations; the Phase A Python domain module was trained on the pre-update backbone but transfers to the updated backbone and achieves lower perplexity than fresh ABI on the original backbone. The ABI stability constraint (pre-computed reference MSE + frozen proj_in) fully preserved the coordinate frame through the update.
This validates: Backbone-update invariance for encoder-decoder architectures at 730M scale. Combined with the GPT-2-medium succession result (65% efficacy), backbone-update invariance is now confirmed for both architecture classes.
Architecture:
- Taps:
[19, 20, 21, 22, 23, 24] - Per-tap LN: yes —
tap_lns = ModuleList([LayerNorm(1024)] × 6) - D_ABI: 4096
- D_IN: 6144 (6 × 1024)
- proj_in: Linear(6144 → 4096, no bias)
- abi_ln: LayerNorm(4096)
- domain: Linear(4096→16384)+GELU+Linear(16384→4096)+LN(4096)
- proj_out: Linear(4096 → 1024, no bias)
- domain_alpha: learnable scalar (init 1.0)
- SEED_A: 42, SEED_C: 99
Training:
- Domain pre-training: 2000 steps each (anchor + native)
- P1: 4000 steps @ LR=5e-3 (warmup 400)
- P2: 3000 steps @ LR=5e-4
- P3: 3000 steps @ LR=5e-5
- P4: 6000 steps @ LR=5e-6 (extended from 2000)
- Total calibration: 16000 steps
Results:
- corrMSE: 0.003047 @ step 15466
- Official NIB (rng=7777): top1=0.8508, top5=0.8725, JS=0.01391, ent=0.2256 → PASS
- Extended NIB (n=25): mean top5=0.8549, std=0.0316, SE=0.0063
- 95% CI: [0.8425, 0.8673]
- Per-seed: 7777=0.8725✓, 1111=0.8353✗, 2222=0.8475✗, 3333=0.8542✗, 4444=0.8651✓
- cal_ppl: 18.3
- ABI params: 163.6 M
- Elapsed: 237.4 min (RTX 3080 Laptop)
- Result file:
cross_arch_t5_nib_v53_results.json
Key finding: Per-tap LN + D=4096 is the optimal ABI architecture for T5-large. The combination of large null-space (D=4096 → 3072-dim null space in proj_out) and uniform tap weighting (per-tap LN) produces the highest top-5 agreement achievable within the ABI framework. Best corrMSE ever achieved: 0.003047.
What changed vs 45AS: P4 extended from 6000 to 12000 steps (total 22000 calibration steps).
Hypothesis: The corrMSE floor had not fully converged by step 15466; more steps would lower it further.
Results:
- corrMSE: 0.003243 (WORSE than 45AS by +0.000196)
- Extended top-5: 0.8496
Finding: Extended training found a worse stochastic basin. The corrMSE floor is not about convergence — the optimizer found the floor by step ~15000. Further training with decaying LR moves to a slightly less favourable basin. Do not extend P4 beyond 6000 steps.
What changed vs 45AS: Tap layers expanded from [19-24] (6 taps) to [16-24] (9 taps). D_IN=9216 (9×1024).
Hypothesis: More taps provide more input dimensions → better null-space geometry → higher NIB.
Results:
- corrMSE: 0.003122
- Extended top-5: 0.8506
Finding: Adding 3 earlier decoder layers provides no NIB improvement. Layers [16,17,18] are earlier in the decoder and carry redundant (lower-level) information compared to [19-24]. The geometry of the correction manifold is not improved by earlier taps. The optimal tap configuration is [19-24].
What changed vs 45AS: Added 1 encoder mean-pool tap to the 6 decoder taps, yielding 7 taps, D_IN=7168 (7×1024).
Hypothesis: The encoder's bidirectional representation might carry complementary information to the decoder's late layers.
Results:
- corrMSE: 0.003129
- Extended top-5: 0.8457
Finding: The encoder tap is redundant. T5's decoder layers already incorporate encoder information through the cross-attention mechanism — h_24 already contains the encoder's contribution. Adding the raw encoder mean-pool creates conflicting gradient signals. Encoder taps do not help.
What was attempted: KL divergence with temperature-squared scaling (T²-scaled KL), where T=2.0 → scale=4.0. This was intended to smooth the KL gradient.
What went wrong: The T²-scaled KL diverge to 200+. The loss was numerically unstable: KL divergence scaled by T² is not a bounded quantity when soft distributions are compared to peaked distributions. The resulting model's NIB was 0.697 — completely broken.
Status: Invalid experiment. DO NOT USE cross_arch_t5_nib_v57.py.
Lesson: T² scaling is a standard technique in knowledge distillation for combining hard and soft targets. It does not apply to a standalone calibration objective where the anchor distribution may have very low-entropy peaks (JS ≈ 0.05 at hard positions).
What changed vs 45AS: Objective changed from corrMSE to raw KL divergence at temperature T=2.0 (no T² scaling). Learning rate divided by 10 throughout to compensate for the larger gradient magnitude of logit-space losses.
LR schedule: P1=5e-4, P2=5e-5, P3=5e-6, P4=5e-7.
Results:
- best_kl: 0.17923 @ step 8673
- final corrMSE: 0.002659 — 12.7% LOWER than 45AS
- Extended top-5: 0.8244 — 0.0305 LOWER than 45AS
The critical finding: Lower objective loss on a different objective does NOT imply better NIB. The KL objective minimized the divergence between softened logit distributions, but in doing so placed the correction error in logit-relevant directions that degrade top-5 agreement at many positions.
Lesson: corrMSE is the uniquely optimal calibration objective for the NIB metric. The geometric alignment between corrMSE and NIB top-5 is a key property of the ABI framework.
What changed vs 45AS: Loss function changed to position-weighted corrMSE:
where
WEIGHT_ALPHA=1.0 (pure weighted from step 1), same 4-phase LR schedule as 45AS.
Hypothesis: Hard positions (where the anchor makes large corrections) need more accurate replication. Weighting by anchor correction norm focuses the optimizer on these positions.
Results:
- corrMSE (std, unweighted): 0.003175
- Extended top-5: 0.8472
- Per-seed: 7777=0.8312 (-0.0413 vs 45AS), 1111=0.8278, 2222=0.8583(+0.011), 3333=0.8515, 4444=0.8671
The catastrophe: rng=7777 dropped by -0.041. The seed that passes the official protocol was catastrophically harmed. rng=2222 improved slightly (+0.011), confirming the weighted objective does help hard positions — but the gradient interference with easy positions is severe enough to ruin the overall result.
Root cause: The weighted objective creates misaligned gradient directions. Positions that already have large corrections (high weight) dominate the gradient. The optimizer over-trains these positions in ways that disturb the correction alignment at easy positions. This is a fundamental property of the weighted objective, not a learning rate issue.
What changed vs 45AS: Curriculum approach: standard corrMSE for P1-P3 (10,000 steps), then switch to weighted corrMSE for P4 (6,000 steps). Hypothesis: build a solid baseline first, then refine the hard positions at very low LR (5e-6).
Results:
- corrMSE: 0.003168
- Extended top-5: 0.8446
- Per-seed: 7777=0.8339 (-0.0386 vs 45AS), 1111=0.8244, 2222=0.8475, 3333=0.8475, 4444=0.8698
The confirmation: Even from a solid 45AS-quality baseline (std corrMSE after P1-P3 was comparable to 45AS), switching to weighted corrMSE in P4 at LR=5e-6 still catastrophically hurt rng=7777 by -0.039. The effect is not an LR issue; it is structural to the weighted objective.
Pattern confirmed: Any deviation from pure standard corrMSE at any phase degrades performance. The gradient interference is present even at the lowest practical learning rate (5e-6).
All valid experiments using the 45AS architecture (D=4096+LN, corrMSE objective):
| Exp | Schedule Change | corrMSE | Ext top-5 |
|---|---|---|---|
| 45AS | P4=6000 (optimal) | 0.003047 | 0.8549 |
| 45AT | P4=12000 (too long) | 0.003243 | 0.8496 |
| 45AU | 9 taps (too many) | 0.003122 | 0.8506 |
| 45AV | +enc tap (redundant) | 0.003129 | 0.8457 |
Mean corrMSE across valid runs: 0.003135
Mean extended top-5 across valid runs: 0.8502
The architecture class true mean (n=4 valid runs) is 0.8502. The 45AS result (0.8549) is at the favorable end of the stochastic distribution, benefiting from the combined effect of optimal P4 length and SEED_C=99's initialization.
-
The ABI framework ceiling for T5-large is ext_mean ≈ 0.850. Seven valid experiments (45AS, 45AT, 45AU, 45AV, 45AX, 45AY, 45AZ) all converge to the same ceiling.
-
45AS is the global optimum. No architecture change, objective change, or schedule change improved upon it.
-
corrMSE is the uniquely optimal calibration objective. KL divergence, weighted corrMSE, and curriculum approaches all produce worse NIB, even when they achieve lower training loss.
-
The extended NIB gap from 0.860 is fundamental. Seeds 1111, 2222, 3333 sample corpus positions where T5-large's distribution is highly peaked (JS ≈ 0.05–0.10 at the chunk level, 6× typical). The ABI cannot replicate these peaked distributions perfectly because doing so would require a correction vector outside the learnable manifold. This is an information-theoretic limit of the linear correction architecture.
-
The official NIB protocol PASSES. The rng=7777 evaluation, which is the actual protocol, achieves top-5=0.8725 with all four criteria satisfied.
These experiments test whether ABI generalises to two models with different backbone weights, using t5-large as source oracle and t5-large-lm-adapt as target backbone.
| Exp | Script | Architecture | Objective | Result | Status |
|---|---|---|---|---|---|
| Path3A-v1 | cross_arch_path3a_nib.py |
6-tap [19-24], D=4096, h_final = corr + h_last |
Top-K KL (K=50) | top5=0.0063, ent=8.24 | FAIL — catastrophic |
| Path3A-v2 | cross_arch_path3a_v2_nib.py |
6-tap [19-24], D=4096, h_final = corr only, src lm_head |
corrMSE on src h_24 | corrMSE floor=0.020, top5=0.0063 | FAIL – corrMSE floor |
| Path3A-v3 | cross_arch_path3a_v3_nib.py |
same as v2 | top-K KL (K=50), no temp | KL=856 → diverged | KILLED – KL explosion |
| Path3A-v4 | cross_arch_path3a_v4_nib.py |
same as v2 + shared src encoder | corrMSE on src h_24 | corrMSE floor=0.033, stalled | FAIL – encoder bridge ineffective |
| Path3A-v5 | cross_arch_path3a_v5_nib.py |
same as v2 | top-K KD (K=200, T=2.0→1.0) | in progress | ⏳ |
Setup:
- Source: t5-large (737M, d_ff=4096, 24L) — frozen oracle, no ABI
- Target: t5-large-lm-adapt (783M, d_ff=2816, 24L) — frozen backbone + CrossABI
- ABI architecture: identical to 45AS (6-tap [19-24], D=4096, per-tap LN)
- Objective: top-K sparse KL, K=50 (source's top-50 most-confident tokens)
- Training: 16000 steps, same 4-phase LR as 45AS
Results:
- Best top-K KL loss: 35.76 @ step 10894 (plateau after step 6200)
- Official NIB (rng=7777): top5=0.0063, top1=0.4125, JS=0.622, ent_diff=8.24 → FAIL
- Extended NIB (n=25): mean top5=0.0050 (≈ random chance)
Root cause: h_final = correction + h_last — the target backbone's h_last is in
t5-large-lm-adapt's representation space and produces a specific peaked distribution via
the target's lm_head. The ABI correction is near-zero at initialization and cannot escape
the target's natural output. Entropy diff of 8.24 nats (near-uniform output) confirms the
ABI never took control of the prediction. Top-K KL provided no path out of this degenerate
state because the gradient was overwhelmed by the target backbone's h_last.
Lesson: The correction + h_last residual design requires shared backbone representation
space. When h_last is from a different backbone, it dominates completely. A different
architecture is required for cross-backbone transfer.
Key changes from v1:
- No h_last injection: \h_final = proj_out(domain(abi_ln(proj_in(concat_taps))))\ -- pure prediction
- Source lm_head transplant: \logits = src.lm_head(h_final * scale)\ -- ABI routes through t5-large frozen vocabulary projection
- corrMSE on source h_24: supervised directly on t5-large pre-lm_head hidden state
Results:
- Best corrMSE: 0.020388 @ step 12697 (vs Path 2C: 0.003047)
- Official NIB (rng=7777): top5=0.0063, top1=0.3781, JS=0.263, ent=0.191 -> FAIL
- Extended NIB (n=25): mean top5=0.0067
- ent_diff=0.191 PASSES (<0.350) -- distributions are peaked, just on wrong tokens
- Results: \cross_arch_path3a_v2_results.json Root cause: corrMSE floor at 0.020 (~6.7x higher than Path 2C 0.003). t5-large (d_ff=4096) and t5-large-lm-adapt (d_ff=2816) have different decoder FFN widths; their layer-24 hidden states live in different geometric spaces. MSE regression cannot bridge this gap -- 0.020 error corresponds to logit perturbations of ~4.5 per position, enough to corrupt top-5 ordering.
NIB metric note: \ op5 = float(top5_src == top5_tgt)\ measures exact 5-token SET equality. top1=0.378 means correct argmax 37.8% of time, but the other 4 tokens in the top-5 set are all wrong -> top5=0.006.
Key change from v2: Loss function changed to top-K KL (K=50, T=1.0) through source lm_head.
Diagnosis: Initial KL loss = 856 (= 13.4 nats per position x 64 positions via batchmean without /T). Loss INCREASED during LR warmup. Killed at step 200 with loss=1148.
Root cause: Random-init ABI produces confidently-wrong predictions through src.lm_head. KL(peaked_src || wrong_abi) = huge. Increasing LR during warmup amplified divergence.
Lesson: Top-K KL from random init with T=1.0 is catastrophically unstable. Need temperature annealing.
Key change from v2: Both decoders conditioned on the same source encoder output. Hypothesis: identical cross-attention K,V inputs improve hidden-state alignment.
Results:
- Initial corrMSE: 0.4749 (vs v2 0.4680 -- essentially identical)
- corrMSE floor: ~0.033 at step 5200 (worse than v2 0.020)
- Terminated at step 5200 -- stalled across P1 and P2
Root cause: The corrMSE floor is determined by decoder FFN geometry (d_ff=4096 vs 2816), not the encoder context. Sharing the encoder does not help because the 24 decoder layers with different FFN widths create irreducibly different residual stream directions. Shared encoder adds computation without benefit.
Lesson: corrMSE is the wrong objective for cross-backbone. The correct fix is logit-space optimization (KD) that bypasses hidden-state geometry entirely.
Core insight: corrMSE has an irreducible floor (0.020) due to d_ff mismatch. Top-K KL (v3) directly optimizes NIB but explodes at random init. v5 combines both insights with temperature annealing.
Key changes:
- Loss: top-K KD (K=200): KL(softmax(src_top200/T) || softmax(abi_top200/T)) through src lm_head
- Temperature annealing: T=2.0->1.0 over 16000 steps. At T=2.0: initial loss ~10.9 nats (stable). At T=1.0: optimizes exact token ordering.
- Correct reduction: mean over B x T (vs v3 batchmean that gave 64x inflation)
- Architecture: unchanged from v2 (CrossABIv2, 163M params)
Training diagnostics:
- Initial KD loss (T=2.0, K=200): 10.9 nats -- stable (cf. v3: 856)
- Step 600 best: 4.6 | Step 2800 best: 3.6 | Step 6600 best: 3.06 -- still descending
Results will be recorded here when the run completes.
Across all valid experiments, rng=1111 is consistently the hardest seed. Specifically, chunk 3 of rng=1111 has JS ≈ 0.06–0.10 — approximately 6× the typical value. This corresponds to a specific corpus position where T5-large's conditional distribution over the Python vocabulary is highly concentrated (low entropy, high-confidence predictions). At these positions:
- The anchor's correction vector is large (the ABI is doing significant work)
- The required correction for the calibrated model to match is more sensitive to small errors
- Any gradient interference (from weighting, from extended optimization, from different objectives) disproportionately affects these positions
The extended NIB's failure to reach 0.860 mean is almost entirely attributable to this single bottleneck. The rng=1111 mean top-5 (0.8353 for 45AS) pulls the combined mean from 0.8640 (without rng=1111) down to 0.8549.
| Lesson | Experiment Evidence |
|---|---|
| Do not extend P4 beyond 6000 steps | 45AT extended → worse basin |
| More decoder taps do not lower the corrMSE floor | 45AU: 9-tap floor same as 6-tap |
| Encoder taps are redundant | 45AV: enc tap gives no improvement |
| T² KL scaling is numerically unstable for standalone calibration | 45AW: diverges to 200+ |
| Lower loss on a different objective ≠ better NIB | 45AX: -12.7% KL loss, -3.1% NIB |
| Weighted corrMSE helps hard seeds but catastrophically hurts easy seeds | 45AY: rng=2222 +0.011, rng=7777 -0.041 |
| Even low-LR (5e-6) weighted phase causes catastrophic interference | 45AZ: rng=7777 -0.039 from solid baseline |
| Per-tap LN is a genuine +0.008 architectural improvement | 45AP→45AS comparison |
| D=4096 null-space geometry raises NIB without lowering corrMSE floor | D=1024→4096 improvement |
| SEED_C=99 is at the favorable end of the initialization distribution | Architecture class mean: 0.8502 vs 45AS: 0.8549 |