【Task.31】feat(sft): add pair-aware DPO training

docs/.vitepress/config.mts

@@ -244,6 +244,7 @@ export default defineConfig({
                 { text: 'Quick Start', link: '/en/guide/quick-start' },
                 { text: 'Customize Training', link: '/en/guide/customize-training' },
                 { text: 'SFT Training', link: '/en/guide/sft-training' },
+                { text: 'DPO Training', link: '/en/guide/dpo-training' },
                 { text: 'PPO Training', link: '/en/guide/ppo-training' },
                 { text: 'REINFORCE++', link: '/en/guide/reinforce-plus-plus' },
                 { text: 'REINFORCE++ Report', link: '/en/guide/reinforce-plus-plus-training-report' },
@@ -356,6 +357,7 @@ export default defineConfig({
                 { text: '快速上手', link: '/zh/guide/quick-start' },
                 { text: '自定义训练', link: '/zh/guide/customize-training' },
                 { text: 'SFT 训练', link: '/zh/guide/sft-training' },
+                { text: 'DPO 训练', link: '/zh/guide/dpo-training' },
                 { text: 'PPO 训练', link: '/zh/guide/ppo-training' },
                 { text: 'REINFORCE++', link: '/zh/guide/reinforce-plus-plus' },
                 { text: 'REINFORCE++ 训练与数值验证报告', link: '/zh/guide/reinforce-plus-plus-training-report' },

docs/en/guide/dpo-training.md

@@ -0,0 +1,64 @@
+# DPO Training
+
+Relax supports Direct Preference Optimization (DPO) through the offline SFT data path. The public Task 31 recipe is [`run-qwen3-0.6B-ultrafeedback-1xgpu.sh`](../../../scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh).
+
+## Prepare the preference subset
+
+Generate the deterministic UltraFeedback subset from its pinned dataset revision:
+
+```bash
+python scripts/data/prepare_ultrafeedback_preferences.py \
+  --output-dir /data/task31-ultrafeedback
+```
+
+The command creates train/eval JSONL and Parquet files plus `manifest.json`. For the published Task 31 subset, compare the generated manifest with the [reproducibility evidence bundle](https://github.com/user-attachments/files/31305744/task31-pr1-dpo-evidence-public-v2.tar.gz) before training. The manifest fixes the source revision, selected prompt IDs, rejection counts, and output SHA-256 values. Derived dataset files are intentionally not stored in Git.
+
+Each input row contains one complete preference pair:
+
+```json
+{
+  "prompt_id": "stable-id",
+  "chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
+  "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
+}
+```
+
+Chosen and rejected branches must have an identical prompt and different, non-empty assistant completions.
+
+## Launch standard DPO
+
+Download the pinned Qwen checkpoint, then set the model, data, and output locations expected by the standard entrypoint:
+
+```bash
+export MODEL_DIR=/models
+export MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca # full 40-character commit SHA
+export HF_CHECKPOINT="${MODEL_DIR}/Qwen3-0.6B-${MODEL_REVISION}"
+export PROMPT_DATA=/data/task31-ultrafeedback/ultrafeedback_train.parquet
+export SAVE_DIR=/checkpoints/task31-dpo
+
+hf download Qwen/Qwen3-0.6B --revision "${MODEL_REVISION}" --local-dir "${HF_CHECKPOINT}"
+bash scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh
+```
+
+The recipe defaults to 200 optimizer steps, 32 preference pairs per global batch, `beta=0.1`, and a 1,024-token branch limit. `GLOBAL_BATCH_SIZE`, `NUM_ROLLOUT`, `MAX_TOKENS_PER_GPU`, and `SAVE_INTERVAL` can be overridden explicitly.
+
+Standard DPO verifies the pinned repository revision against the local `HF_CHECKPOINT` directory, then reconstructs the frozen reference from that directory. Checkpoints include a reference-identity sidecar containing canonical parameter and fixed-probe digests. A missing or mismatched sidecar fails before the next forward pass.
+
+The probe digest is a byte-exact SHA-256 over frozen-reference log-probabilities, so resume assumes the same GPU model, driver, image, and kernel stack as the original run. Resuming on different hardware or software fails the probe check by design — treat it as an environment mismatch, not data corruption.
+
+Use `--dpo-reference-free` only when reference-free DPO is intended; do not combine it with the standard reference identity arguments.
+
+## Pair-aware batching
+
+One preference pair is one TransferQueue row. Its chosen and rejected branch lengths are combined into `custom_meta.total_lengths`; the pinned `SeqlenBalancedSampler` assigns complete rows and keeps equal pair counts across data-parallel ranks. Branches are expanded only after a rank receives its rows, so dynamic micro-batch reordering cannot split pair identity.
+
+## Metrics
+
+DPO emits the following training metrics under the `train/dpo/` namespace:
+
+- `loss`, `logps_chosen`, and `logps_rejected`;
+- `ref_logps_chosen` and `ref_logps_rejected` in standard mode;
+- `reward_chosen`, `reward_rejected`, and `reward_margin`;
+- `strict_accuracy`, `tie_rate`, and `tie_aware_accuracy`.
+
+For distributed parity claims, run DP=1 and DP=2 with the same image, model/data revisions, hyperparameters, and batch semantics, and retain the raw logs and reference digests.

docs/zh/guide/dpo-training.md

@@ -0,0 +1,64 @@
+# DPO 训练
+
+Relax 通过离线 SFT 数据链路支持 Direct Preference Optimization（DPO）。Task 31 的公开 recipe 是 [`run-qwen3-0.6B-ultrafeedback-1xgpu.sh`](../../../scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh)。
+
+## 准备偏好数据子集
+
+从固定的数据集 revision 生成确定性的 UltraFeedback 子集：
+
+```bash
+python scripts/data/prepare_ultrafeedback_preferences.py \
+  --output-dir /data/task31-ultrafeedback
+```
+
+命令会生成 train/eval JSONL、Parquet 以及 `manifest.json`。对于已发布的 Task 31 子集，训练前应将生成结果与[可复现性证据包](https://github.com/user-attachments/files/31305744/task31-pr1-dpo-evidence-public-v2.tar.gz)中的 manifest 对比。manifest 固定 source revision、选中的 prompt ID、拒绝原因计数和输出文件 SHA-256；派生数据文件本身不提交到 Git。
+
+每行输入承载一个完整 preference pair：
+
+```json
+{
+  "prompt_id": "stable-id",
+  "chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
+  "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
+}
+```
+
+chosen/rejected 必须共享完全相同的 prompt，并包含不同且非空的 assistant completion。
+
+## 启动标准 DPO
+
+下载固定版本的 Qwen checkpoint，然后设置标准入口所需的模型、数据和输出路径：
+
+```bash
+export MODEL_DIR=/models
+export MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca # 完整的 40 位 commit SHA
+export HF_CHECKPOINT="${MODEL_DIR}/Qwen3-0.6B-${MODEL_REVISION}"
+export PROMPT_DATA=/data/task31-ultrafeedback/ultrafeedback_train.parquet
+export SAVE_DIR=/checkpoints/task31-dpo
+
+hf download Qwen/Qwen3-0.6B --revision "${MODEL_REVISION}" --local-dir "${HF_CHECKPOINT}"
+bash scripts/training/dpo/run-qwen3-0.6B-ultrafeedback-1xgpu.sh
+```
+
+recipe 默认运行 200 个 optimizer step，每个 global batch 为 32 个 preference pair，`beta=0.1`，单分支最大 1,024 token。可以显式覆盖 `GLOBAL_BATCH_SIZE`、`NUM_ROLLOUT`、`MAX_TOKENS_PER_GPU` 和 `SAVE_INTERVAL`。
+
+标准 DPO 会先在本地 `HF_CHECKPOINT` 目录中校验固定的 repository revision，再从该目录重建冻结 reference。checkpoint 带有 reference identity sidecar，其中保存 canonical parameter digest 和固定 probe digest；sidecar 缺失或不一致时，会在下一次 forward 前失败。
+
+probe digest 是对冻结 reference log-probability 的逐字节 SHA-256，因此 resume 假定 GPU 型号、驱动、镜像与内核栈与原运行完全一致。在不同硬件或软件环境上 resume 会按设计触发 probe 校验失败——这表示环境不匹配，而非数据损坏。
+
+只有明确需要 reference-free DPO 时才使用 `--dpo-reference-free`，不要同时传入标准 reference identity 参数。
+
+## Pair-aware batching
+
+一个 preference pair 对应一个 TransferQueue row。chosen/rejected 分支长度相加后写入 `custom_meta.total_lengths`；固定版本的 `SeqlenBalancedSampler` 分配完整 row，并保证各 data-parallel rank 的 pair 数相同。只有 rank 收到 pair row 后才展开两个分支，因此动态 micro-batch 重排不会破坏 pair identity。
+
+## 指标
+
+DPO 在 `train/dpo/` 命名空间下记录以下训练指标：
+
+- `loss`、`logps_chosen` 和 `logps_rejected`；
+- 标准模式下的 `ref_logps_chosen` 和 `ref_logps_rejected`；
+- `reward_chosen`、`reward_rejected` 和 `reward_margin`；
+- `strict_accuracy`、`tie_rate` 和 `tie_aware_accuracy`。
+
+如需声明分布式一致性，应在相同镜像、模型/数据 revision、超参数和 batch 语义下分别运行 DP=1、DP=2，并保留原始日志与 reference digest。