fix: derive checkpoint_format from resolved local path (LoRA base weights silently not loaded) - #161
Conversation
…ghts silently not loaded) Signed-off-by: Amirreza Rouhi <ar3755@drexel.edu>
|
Gentle bump on this one — the failure mode is worth flagging because it is silent: with a checkpoint-DB URI as The fix is a single conditional that derives Users appear to be hitting the same failure family in the predict2 repo as well (e.g. nvidia-cosmos/cosmos-predict2#176 — LoRA output identical to base plus checkpoint-loading warnings). The branch is conflict-free. Happy to add a regression test or adjust style if that helps review. Thanks! |
What
checkpoint_formatis computed from the configuredcheckpoint.load_pathstring, butget_checkpoint_path()may resolve a DCP-style URI to a consolidated.ptfile. The LoRA key-mapping branch is gated oncheckpoint_format == "pt", so PEFT-wrapped models (use_lora=True) load zero parameters under the non-strict load and train from random init — with no error raised. This derives the format from the resolved file instead.Fixes #160. Also matches the long-standing symptom reported in
nvidia-cosmos/cosmos-predict2#176(LoRA model produces identical/degenerate results vs. base; checkpoint-loading warnings).Change
One conditional in
cosmos_predict2/_src/predict2/utils/model_loader.py, immediately after theget_checkpoint_path(...)resolution:Verification
Cosmos-Predict2.5-2B LoRA post-training (rank 32, single H100), per the issue's reproduction:
_IncompatibleKeys(missing_keys=[<all base params>]), no LoRA-mapping lineMapped 689 LoRA keys from checkpoint to modelNo behavior change for runs whose
load_pathalready ends in.pt(the condition only promotesdcp→ptwhen the resolved file is a.pt).