【Task.31】feat(sft): add reward modeling and preference evaluation

docs/.vitepress/config.mts

@@ -244,6 +244,7 @@ export default defineConfig({
                 { text: 'Quick Start', link: '/en/guide/quick-start' },
                 { text: 'Customize Training', link: '/en/guide/customize-training' },
                 { text: 'SFT Training', link: '/en/guide/sft-training' },
+                { text: 'DPO Training', link: '/en/guide/dpo-training' },
                 { text: 'PPO Training', link: '/en/guide/ppo-training' },
                 { text: 'REINFORCE++', link: '/en/guide/reinforce-plus-plus' },
                 { text: 'REINFORCE++ Report', link: '/en/guide/reinforce-plus-plus-training-report' },
@@ -356,6 +357,7 @@ export default defineConfig({
                 { text: '快速上手', link: '/zh/guide/quick-start' },
                 { text: '自定义训练', link: '/zh/guide/customize-training' },
                 { text: 'SFT 训练', link: '/zh/guide/sft-training' },
+                { text: 'DPO 训练', link: '/zh/guide/dpo-training' },
                 { text: 'PPO 训练', link: '/zh/guide/ppo-training' },
                 { text: 'REINFORCE++', link: '/zh/guide/reinforce-plus-plus' },
                 { text: 'REINFORCE++ 训练与数值验证报告', link: '/zh/guide/reinforce-plus-plus-training-report' },

docs/en/guide/dpo-training.md

@@ -0,0 +1,72 @@
+# DPO Training
+
+Relax supports Direct Preference Optimization (DPO) through the offline SFT data path. The public Task 31 recipe is [`run-qwen3-0.6B-ultrafeedback-1xgpu.sh`](../../../scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh).
+
+## Prepare the preference subset
+
+Generate the deterministic UltraFeedback subset from its pinned dataset revision:
+
+```bash
+python scripts/data/prepare_ultrafeedback_preferences.py \
+  --output-dir /data/task31-ultrafeedback
+```
+
+The command creates train/eval JSONL and Parquet files plus `manifest.json`. For the published Task 31 subset, compare the generated manifest with the [reproducibility evidence bundle](https://github.com/user-attachments/files/31305744/task31-pr1-dpo-evidence-public-v2.tar.gz) before training. The manifest fixes the source revision, selected prompt IDs, rejection counts, and output SHA-256 values. Derived dataset files are intentionally not stored in Git.
+
+Each input row contains one complete preference pair:
+
+```json
+{
+  "prompt_id": "stable-id",
+  "chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
+  "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
+}
+```
+
+Chosen and rejected branches must have an identical prompt and different, non-empty assistant completions.
+
+## Launch standard DPO
+
+Download the pinned Qwen checkpoint, then set the model, data, and output locations expected by the standard entrypoint:
+
+```bash
+export MODEL_DIR=/models
+export MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca # full 40-character commit SHA
+export HF_CHECKPOINT="${MODEL_DIR}/Qwen3-0.6B-${MODEL_REVISION}"
+export PROMPT_DATA=/data/task31-ultrafeedback/ultrafeedback_train.parquet
+export SAVE_DIR=/checkpoints/task31-dpo
+
+hf download Qwen/Qwen3-0.6B --revision "${MODEL_REVISION}" --local-dir "${HF_CHECKPOINT}"
+bash scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh
+```
+
+The recipe defaults to 200 optimizer steps, 32 preference pairs per global batch, `beta=0.1`, and a 1,024-token branch limit. `GLOBAL_BATCH_SIZE`, `NUM_ROLLOUT`, `MAX_TOKENS_PER_GPU`, and `SAVE_INTERVAL` can be overridden explicitly.
+
+Standard DPO verifies the pinned repository revision against the local `HF_CHECKPOINT` directory, then reconstructs the frozen reference from that directory. Checkpoints include a reference-identity sidecar containing canonical parameter and fixed-probe digests. A missing or mismatched sidecar fails before the next forward pass.
+
+The probe digest is a byte-exact SHA-256 over frozen-reference log-probabilities, so resume assumes the same GPU model, driver, image, and kernel stack as the original run. Resuming on different hardware or software fails the probe check by design — treat it as an environment mismatch, not data corruption.
+
+Use `--dpo-reference-free` only when reference-free DPO is intended; do not combine it with the standard reference identity arguments.
+
+## Pair-aware batching
+
+One preference pair is one TransferQueue row. Its chosen and rejected branch lengths are combined into `custom_meta.total_lengths`; the pinned `SeqlenBalancedSampler` assigns complete rows and keeps equal pair counts across data-parallel ranks. Branches are expanded only after a rank receives its rows, so dynamic micro-batch reordering cannot split pair identity.
+
+## Metrics
+
+DPO emits the following training metrics under the `train/dpo/` namespace:
+
+- `loss`, `logps_chosen`, and `logps_rejected`;
+- `ref_logps_chosen` and `ref_logps_rejected` in standard mode;
+- `reward_chosen`, `reward_rejected`, and `reward_margin`;
+- `strict_accuracy`, `tie_rate`, and `tie_aware_accuracy`.
+
+For distributed parity claims, run DP=1 and DP=2 with the same image, model/data revisions, hyperparameters, and batch semantics, and retain the raw logs and reference digests.
+
+## Reward modeling and acceptance artifacts
+
+The companion recipe is `scripts/training/reward_modeling/run-qwen3-0.6B-ultrafeedback-1xgpu.sh`. It defaults to 200 optimizer steps and 32 pairs per global batch. Preference evaluation runs before the first optimizer step (step 0), periodically, and after the final completed step even when the interval does not divide the run length.
+
+Both DPO and reward modeling write acceptance data under `<SAVE_DIR>/<EXP_NAME>/preference_eval/`: the canonical probe contract and SHA-256, the DP/micro-batch plan and SHA-256, step-0/final per-pair JSONL, and a 10,000-replicate FP64 PCG64 paired-bootstrap summary. Final evaluation fails if probe preprocessing, pair order, or the batch plan differs from step 0. Retain this directory together with the expanded command, environment inventory, raw stdout/stderr, metrics, and curves.
+
+Reward-model Megatron checkpoints persist `sft_objective=reward_model`, `head_type=reward_model_terminal_v1`, and `checkpoint_role=actor`. Resume rejects missing or incompatible metadata, non-exact scalar-head keys/shapes, partial optimizer/RNG restoration, and PPO critic checkpoints.

docs/zh/guide/dpo-training.md

@@ -0,0 +1,72 @@
+# DPO 训练
+
+Relax 通过离线 SFT 数据链路支持 Direct Preference Optimization（DPO）。Task 31 的公开 recipe 是 [`run-qwen3-0.6B-ultrafeedback-1xgpu.sh`](../../../scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh)。
+
+## 准备偏好数据子集
+
+从固定的数据集 revision 生成确定性的 UltraFeedback 子集：
+
+```bash
+python scripts/data/prepare_ultrafeedback_preferences.py \
+  --output-dir /data/task31-ultrafeedback
+```
+
+命令会生成 train/eval JSONL、Parquet 以及 `manifest.json`。对于已发布的 Task 31 子集，训练前应将生成结果与[可复现性证据包](https://github.com/user-attachments/files/31305744/task31-pr1-dpo-evidence-public-v2.tar.gz)中的 manifest 对比。manifest 固定 source revision、选中的 prompt ID、拒绝原因计数和输出文件 SHA-256；派生数据文件本身不提交到 Git。
+
+每行输入承载一个完整 preference pair：
+
+```json
+{
+  "prompt_id": "stable-id",
+  "chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
+  "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
+}
+```
+
+chosen/rejected 必须共享完全相同的 prompt，并包含不同且非空的 assistant completion。
+
+## 启动标准 DPO
+
+下载固定版本的 Qwen checkpoint，然后设置标准入口所需的模型、数据和输出路径：
+
+```bash
+export MODEL_DIR=/models
+export MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca # 完整的 40 位 commit SHA
+export HF_CHECKPOINT="${MODEL_DIR}/Qwen3-0.6B-${MODEL_REVISION}"
+export PROMPT_DATA=/data/task31-ultrafeedback/ultrafeedback_train.parquet
+export SAVE_DIR=/checkpoints/task31-dpo
+
+hf download Qwen/Qwen3-0.6B --revision "${MODEL_REVISION}" --local-dir "${HF_CHECKPOINT}"
+bash scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh
+```
+
+recipe 默认运行 200 个 optimizer step，每个 global batch 为 32 个 preference pair，`beta=0.1`，单分支最大 1,024 token。可以显式覆盖 `GLOBAL_BATCH_SIZE`、`NUM_ROLLOUT`、`MAX_TOKENS_PER_GPU` 和 `SAVE_INTERVAL`。
+
+标准 DPO 会先在本地 `HF_CHECKPOINT` 目录中校验固定的 repository revision，再从该目录重建冻结 reference。checkpoint 带有 reference identity sidecar，其中保存 canonical parameter digest 和固定 probe digest；sidecar 缺失或不一致时，会在下一次 forward 前失败。
+
+probe digest 是对冻结 reference log-probability 的逐字节 SHA-256，因此 resume 假定 GPU 型号、驱动、镜像与内核栈与原运行完全一致。在不同硬件或软件环境上 resume 会按设计触发 probe 校验失败——这表示环境不匹配，而非数据损坏。
+
+只有明确需要 reference-free DPO 时才使用 `--dpo-reference-free`，不要同时传入标准 reference identity 参数。
+
+## Pair-aware batching
+
+一个 preference pair 对应一个 TransferQueue row。chosen/rejected 分支长度相加后写入 `custom_meta.total_lengths`；固定版本的 `SeqlenBalancedSampler` 分配完整 row，并保证各 data-parallel rank 的 pair 数相同。只有 rank 收到 pair row 后才展开两个分支，因此动态 micro-batch 重排不会破坏 pair identity。
+
+## 指标
+
+DPO 在 `train/dpo/` 命名空间下记录以下训练指标：
+
+- `loss`、`logps_chosen` 和 `logps_rejected`；
+- 标准模式下的 `ref_logps_chosen` 和 `ref_logps_rejected`；
+- `reward_chosen`、`reward_rejected` 和 `reward_margin`；
+- `strict_accuracy`、`tie_rate` 和 `tie_aware_accuracy`。
+
+如需声明分布式一致性，应在相同镜像、模型/数据 revision、超参数和 batch 语义下分别运行 DP=1、DP=2，并保留原始日志与 reference digest。
+
+## Reward Modeling 与验收产物
+
+配套 recipe 为 `scripts/training/reward_modeling/run-qwen3-0.6B-ultrafeedback-1xgpu.sh`，默认运行 200 个 optimizer step，global batch 为 32 pairs。偏好评测会在第一次 optimizer step 之前（step 0）、周期边界以及最终完成 step 后运行；最终评测不依赖 interval 恰好整除训练步数。
+
+DPO 与 Reward Modeling 都会在 `<SAVE_DIR>/<EXP_NAME>/preference_eval/` 下写出验收数据：canonical probe 合同及 SHA-256、DP/micro-batch plan 及 SHA-256、step-0/final 逐 pair JSONL，以及 10,000 次 FP64 PCG64 paired-bootstrap summary。若 final 与 step 0 的预处理、pair 顺序或 batch plan 不一致，评测会立即失败。提交证据时需将该目录与展开后的命令、环境清单、原始 stdout/stderr、metrics 和曲线一并保留。
+
+Reward Model 的 Megatron checkpoint 会持久化 `sft_objective=reward_model`、`head_type=reward_model_terminal_v1` 与 `checkpoint_role=actor`。resume 会拒绝缺失或不兼容的 metadata、非精确 scalar-head key/shape、只恢复部分 optimizer/RNG 状态，以及 PPO critic checkpoint。