Skip to content

fix: save and validate at final FSDP step boundary - #125

Open
number-eleven11 wants to merge 1 commit into
X-Square-Robot:mainfrom
number-eleven11:codex/fix-final-step-checkpoint
Open

number-eleven11 wants to merge 1 commit into
X-Square-Robot:mainfrom
number-eleven11:codex/fix-final-step-checkpoint

Conversation

@number-eleven11

Copy link
Copy Markdown

Summary

Run the existing checkpoint and validation interval predicates once more when train_loop reaches num_training_steps. This covers a final optimizer step that lands exactly on an interval boundary.

Fixes #124.

Why

Inside the loop, the save and validation checks run before global_step += 1. On the final iteration of a 100-step run, those checks still see step 99. The counter then becomes 100 and fit() exits before its epoch-end save and validation.

The added check runs only when the configured training limit has been reached, so normal intermediate interval handling is unchanged.

Reproduction / verification

  • Configure num_training_steps: 100, save_interval: 100, val_interval: 100, and ignore_until_interval: 0 in a working FSDP training setup.
  • Before the fix, the run exits without the step-100 checkpoint. With the fix, a 100-step training smoke run completed and wrote 0_100/model.safetensors; final interval validation also ran.
  • python3 -m py_compile wall_x/trainer/fsdp_trainer/fsdp_trainer.py
  • python3 -m pytest tests -q (1 passed, 1 skipped)
  • git diff --check

Only wall_x/trainer/fsdp_trainer/fsdp_trainer.py is changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FSDP final checkpoint and validation skipped when training ends on interval boundary

1 participant