Skip to content

Commit d8d1a85

Browse files
v3.46-trained: AMS_TRAINED_WEIGHTS shape-mismatch is warn+skip, not fatal
Root cause: 4.25 prefix_length_scaling_probe intentionally builds model_b with L_mem doubled (default 8 -> 16). The checkpoint was trained with L_mem=8, so L_mem-dependent tensors (e.g. mem_tokens[L_mem, d_LLM]) legitimately don't fit model_b — this is not a corrupt/incompatible ckpt, it's a deliberate Cfg scan. Old behavior: raise RuntimeError on any shape mismatch -> errored 4.25. New behavior: - Per-tensor shape mismatch is logged and skipped (first 5 detailed, rest summarized). - Hard failure only when the ckpt had non-backbone content (>10 tensors) AND zero tensors loaded — that is the §6 'wrong-SUT ckpt' pattern we must catch. Keeps the §6 protection against loading v344_trained.pt / v348_stacked.pt against a v3.46 SUT (they would mostly shape-mismatch and hit the loaded==0 guard), while letting L_mem-scaling probes proceed. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent 95a9ec1 commit d8d1a85

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

scheme_b_v344.py

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2541,14 +2541,23 @@ def _maybe_load_trained_weights(self):
25412541
else:
25422542
skipped += 1
25432543
prov = blob.get("provenance", "?") if isinstance(blob, dict) else "?"
2544+
total_nonbb_ckpt = sum(1 for k in sd if not k.startswith("backbone"))
25442545
print(f" [AMS_TRAINED_WEIGHTS] loaded={loaded} skipped={skipped} "
25452546
f"shape_errs={len(shape_errs)} path={path} provenance={prov}")
25462547
if shape_errs:
25472548
for n, s_model, s_ckpt in shape_errs[:5]:
2548-
print(f" ! shape mismatch {n}: model={s_model} ckpt={s_ckpt}")
2549+
print(f" ! shape mismatch (skipped) {n}: model={s_model} ckpt={s_ckpt}")
2550+
if len(shape_errs) > 5:
2551+
print(f" ... and {len(shape_errs) - 5} more shape mismatches, all skipped")
2552+
# Raise only if essentially nothing loaded AND the ckpt had content to offer:
2553+
# this catches the "loaded a v344/v348 ckpt against v3.46 shapes" mistake
2554+
# warned about in SPRINT_CLOSEOUT_v3.46.md \u00a76, without breaking probes
2555+
# like 4.25 that scale L_mem and legitimately have a few mismatching tensors.
2556+
if loaded == 0 and total_nonbb_ckpt > 10:
25492557
raise RuntimeError(
2550-
f"AMS_TRAINED_WEIGHTS shape mismatch on {len(shape_errs)} tensor(s); "
2551-
f"ckpt not compatible with current SUT shapes")
2558+
f"AMS_TRAINED_WEIGHTS loaded 0 non-backbone tensors "
2559+
f"(ckpt had {total_nonbb_ckpt}); shape_errs={len(shape_errs)}. "
2560+
f"ckpt appears incompatible with current SUT shapes")
25522561

25532562
def _compute_filler_centroid(self):
25542563
if self.content_classifier is None or self.backbone is None:

0 commit comments

Comments
 (0)