feat: unified resume system with checkpoints.jsonl state tracking

[Hecate0821]

Summary

• Rebase the branch onto fix/pr168-merge-issues so the resume work sits on top of the shared RL utility fixes from PR feat: add FrozenLake multi-turn tool-call GRPO training example #168
• Preserve the ref_logp fix by making ref_logprobs optional throughout the RL pipeline so kl_beta=0 works without a reference model, including the training/utils/rl/grpo.py conflict resolution from the rebase
• Replace the dual-path resume logic (ResumeConfig + setup_resume regex parsing vs checkpoints.jsonl auto-resume) with a single source of truth: local_checkpoint_state.jsonl in a persistent log_path directory
• Add checkpoint_utils.py with ResumeState dataclass, resolve_resume(), load_dcp(), dataset fingerprint validation, training shape validation, checkpoint availability check, and DCP load timing
• Track dataset iteration progress (data_consumed) so resume skips already-trained data
• Add init_from_dcp config field for loading pretrained DCP weights on a fresh dataset
• Log perf/dcp_load_time to wandb on resume

Design

flowchart TD
    A[Training starts] --> B{local_checkpoint_state.jsonl exists?}
    B -->|Yes| C[Read last entry]
    C --> D["Validate: shape, dataset fingerprint"]
    D --> E["Load DCP, skip data_consumed rows"]
    B -->|No| F{init_from_dcp set?}
    F -->|Yes| G["Write initial entry: step=0, data_consumed=0"]
    G --> H[Load DCP weights, start from row 0]
    F -->|No| I[Fresh start, no DCP]

Loading

Changes

File	Change
training/utils/checkpoint_utils.py	New: ResumeState, resolve_resume(), load_dcp(), dataset fingerprint, shape validation
training/utils/resume.py	Deleted: replaced by checkpoint_utils
training/utils/config.py	Removed ResumeConfig dataclass
training/utils/__init__.py	Removed ResumeConfig, setup_resume exports
training/utils/validation.py	Removed resume parameter from validate_config
training/utils/client.py	Added list_checkpoints() method to ReconnectableClient
training/utils/rl/train.py	Added data_consumed counter to run_rl_loop
training/utils/rl/losses.py	PromptGroup.ref_logprobs now optional (None when no ref model); combine_prompt_groups handles None
training/utils/rl/grpo.py	Handle ref_logprobs=None with zero resp_ref; preserved through the PR #168 rebase conflict resolution
training/utils/rl/dapo.py	Same
training/utils/rl/gspo.py	Same
training/utils/rl/cispo.py	Same
training/recipes/rl_loop.py	Config: log_path + init_from_dcp. Resume via resolve_resume() + load_dcp(). ref_forward only called when use_reference=True. Save loop state alongside DCP
training/recipes/sft_loop.py	Same pattern with data_consumed and epoch/batch offset
training/recipes/dpo_loop.py	Migrated to new resume API
training/recipes/orpo_loop.py	Migrated to new resume API
training/tests/e2e/conftest.py	Added e2e_training_shape fixture
training/tests/e2e/test_grpo_resume_e2e.py	Updated for new resume API, training shapes, TOKYO region
training/tests/e2e/test_sft_resume_e2e.py	Updated for new resume API
training/tests/e2e/test_dpo_resume_e2e.py	Updated for new resume API

Test plan

• Run RL loop from scratch (no local_checkpoint_state.jsonl) -- verify fresh start
• Run RL loop with init_from_dcp -- verify DCP loads, training starts from row 0
• Kill and restart RL loop -- verify auto-resume from local_checkpoint_state.jsonl, rows skipped correctly
• Change dataset between runs -- verify fingerprint warning
• Change training shape between runs -- verify shape warning
• Verify perf/dcp_load_time and perf/dcp_save_time appear in wandb
• Run SFT loop with resume -- verify epoch/batch offset computed correctly
• Run with kl_beta=0 (no reference model) -- verify no IndexError, KL penalty is zero
• E2E GRPO resume test with ts-qwen3-30b-a3b-instruct-64k-rft-dev-cp8ep8-v1 in AP_TOKYO_2