A math-reasoning LLM fine-tune — and the engineering story of shipping it in a 5-day sprint on rented consumer GPUs.
DeepSeek-R1-Distill-Qwen-7B → LoRA SFT (NuminaMath-CoT) → merge → LoRA DPO (math preference pairs), all with LLaMA-Factory. Built by a first-semester undergrad on a ¥1000 expiring cloud voucher.
🇨🇳 中文说明见文末 — Chinese version at the bottom.
The model weights are almost a side-effect. The real artifact is the process: taking a reasoning model from "fresh download" to "full SFT→DPO RLHF closure" in 48 hours of compute across two physically-isolated GPU clusters, while a compute voucher ticked down — and debugging every failure that consumer-grade, containerized, cross-province infrastructure could throw at it.
The full narrative — every error, its root cause, and the fix — is in docs/CASE_STUDY.md. A taste:
- A silent killer:
torchrun'sChildFailedErrorswallowed the real error behind 8-GPU distributed crashes. Dropping to a single GPU surfaced the true culprit — a one-line config drift indataset_info.json, not the hardware everyone suspected. - Shell line-continuation corruption via WebSSH paste mangled training commands ~5 times → abandoned shell scripts entirely for Python
subprocesslaunchers (immune to newline mangling). - The 32 GB trap: DPO needs policy + reference = two 7B BF16 models ≈ 30 GB. Solution: QLoRA (4-bit NF4 + double-quant) and merge-the-SFT-adapter-before-DPO to avoid adapter-stacking fragmentation.
- Cost engineering: two-tier compute (¥1.88/h download box vs ¥160/h cluster) and "release the container the second you stop using it" — finished ¥767 under the ¥1000 budget.
| Base model | deepseek-ai/DeepSeek-R1-Distill-Qwen-7B (MIT) |
| Method | LoRA (r=8, α=16, target=all, bf16) — SFT then DPO |
| Framework | LLaMA-Factory, chat template deepseekr1 |
| SFT data | NuminaMath-CoT, cleaned 100k → 90,217 samples |
| DPO data | 2,418 math preference pairs (prompt/chosen/rejected) |
| Final artifact | DPO LoRA adapter (77 MB) + SFT LoRA adapter (78 MB) |
| License | MIT (code & adapters) |
DeepSeek-R1-Distill-Qwen-7B ──SFT(LoRA, NuminaMath)──► DeepMath-SFT ──merge──► merged ──DPO(LoRA, prefs)──► DeepMath
| Stage | Config | Outcome |
|---|---|---|
| SFT | 8×5090, native DDP, LoRA, lr 5e-5, 1 epoch, 1,410 steps, 84 min | train loss 0.73 → 0.39 (clean convergence) |
| DPO | 1×5090, QLoRA NF4, lr 1e-6, 3 epochs, 456 steps, 39 min | train loss 0.6947 (≈ ln 2 cold-start baseline), reward accuracy ~0.5–0.55 |
Read this honestly: the SFT stage converged cleanly. The DPO stage ran to completion without mode collapse, but its gain is modest — reward accuracy hovers just above the 50/50 baseline and no held-out benchmark was run. This is a pipeline-delivery and debugging project, not a SOTA claim. The value demonstrated is shipping a complete, reproducible SFT→DPO pipeline under hard time/GPU/budget constraints — not benchmark numbers.
The release ships LoRA adapters (small, offline-friendly), not a merged model. Reconstruct the full model in one command (see ADR-0002 for why):
pip install -r requirements.txt
# 1. Get the adapters from the Hugging Face repo (see model card), then:
# 2. Merge base + SFT adapter + DPO adapter into a single model
python merge.py \
--base deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
--sft ./sft_adapter \
--dpo ./adapter \
--out ./DeepMath-merged
# 3. Run inference
python examples/inference.pyHugging Face model: SoFarSoGoodya/DeepMath-R1-Distill-Qwen-7B
| Doc | Contents |
|---|---|
| docs/CASE_STUDY.md | ★ The 5-day engineering narrative — every error → root cause → fix, the adversarial-review loop, hardware/memory/cost optimization. The resume centerpiece. |
| docs/TRAINING.md | Exact SFT + DPO hyperparameters and how to reproduce each stage. |
| docs/DATA.md | Datasets, cleaning pipeline, formats, and licenses. |
├── merge.py # one-command adapter → full-model merge
├── requirements.txt
├── src/ # the training & data pipeline (sanitized)
│ ├── download_data.py # fetch + sample NuminaMath-CoT
│ ├── process_data.py # clean (token-length filter) → train_data.json
│ ├── train_sft.py # SFT launcher (8-GPU native DDP)
│ ├── merge_sft.py # llamafactory-cli export (SFT adapter → base)
│ └── train_dpo.py # DPO launcher (single-GPU QLoRA)
├── configs/ # dataset_info.json + explored DeepSpeed configs
├── examples/inference.py
└── docs/ # case study, training, data, images
周柯衡 (SoFarSoGoodya) — built this at the end of his first semester as an undergraduate at Tsinghua University, Qiuzhen College (清华大学求真书院). 📧 zhoukh25@mails.tsinghua.edu.cn · GitHub: @SoFarSoGoodya
This was a solo learning project: from "weak Python, strong math theory" to a complete, reproducible SFT+DPO alignment pipeline in five days, by directing AI tooling effectively while owning every engineering decision.
Code and adapter weights: MIT (see LICENSE). Base model DeepSeek-R1-Distill-Qwen-7B is MIT; tokenizer derives from Qwen (Apache-2.0); SFT data NuminaMath-CoT is Apache-2.0. Please retain upstream attributions. See CITATION.cff.
DeepMath 是一个数学推理大模型的微调项目 —— 但真正的价值在于它是如何在 5 天内、在租来的消费级 GPU 上被"肝"出来的。
以 DeepSeek-R1-Distill-Qwen-7B 为基座,用 LLaMA-Factory 做 LoRA SFT(NuminaMath-CoT)→ 合并 → LoRA DPO(数学偏好对),完成了一条完整的 SFT→DPO(RLHF)对齐流水线。作者是一名大一上学期的本科生,预算是一张 ¥1000 的算力代金券。
这个项目最硬核的不是权重,而是过程:48 小时内横跨两个物理隔离的 GPU 集群,一边顶着代金券到期的倒计时,一边把消费级、容器化、跨省的基建能踩的坑全踩了一遍。完整复盘见 docs/CASE_STUDY.md,几个例子:
- 沉默的杀手:8 卡分布式崩溃时,
torchrun的ChildFailedError把真正的报错吞了。降到单卡才让真凶现形——不是大家怀疑的硬件,而是dataset_info.json里一处配置漂移。 - Shell 反斜杠续行被 WebSSH 粘贴破坏(~5 次复发)→ 彻底弃用 shell 脚本,改用 Python
subprocess启动器(对换行符免疫)。 - 32GB 显存陷阱:DPO 要同时装下策略+参考两个 7B 模型(≈30GB)→ 用 QLoRA(4-bit NF4 + 双重量化),并先把 SFT adapter 合并进基座再做 DPO,避免双 adapter 堆叠的显存碎片。
- 成本工程:两级算力策略(¥1.88/h 的下载机 vs ¥160/h 的训练集群)+ "用完立即释放容器" —— 最终 ¥1000 预算省下 ¥767。
- SFT:8×5090 原生 DDP,LoRA,1 个 epoch,1410 步,84 分钟,训练 loss 0.73 → 0.39(收敛干净)。
- DPO:单卡 5090,QLoRA NF4,3 个 epoch,456 步,39 分钟,训练 loss 0.6947(≈ ln 2 冷启动基线),奖励准确率 ~0.5–0.55。
坦白说:SFT 收敛良好;DPO 完整跑完、没有模式崩溃,但提升有限(奖励准确率仅略高于随机基线,且未跑独立的评测基准)。这是一个**"交付完整可复现流水线 + 工程排错"**的项目,而不是刷榜。它证明的是在严苛的时间/显存/预算约束下把一条 SFT→DPO 流水线跑通的能力。
发布的是 LoRA adapter(小、可离线),不是合并后的完整模型(原因见 ADR-0002)。一条命令重建完整模型:
pip install -r requirements.txt
python merge.py --base deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
--sft ./sft_adapter --dpo ./adapter --out ./DeepMath-merged
python examples/inference.pyHugging Face 模型: SoFarSoGoodya/DeepMath-R1-Distill-Qwen-7B
- docs/CASE_STUDY.md —— ★ 五日工程复盘(报错→定位→修复全记录),简历核心。
- docs/TRAINING.md —— SFT + DPO 精确超参与复现步骤。
- docs/DATA.md —— 数据集、清洗流程、格式与许可证。
代码与 adapter 权重采用 MIT。基座模型 DeepSeek-R1-Distill-Qwen-7B 为 MIT;tokenizer 源自 Qwen(Apache-2.0);SFT 数据 NuminaMath-CoT 为 Apache-2.0。请保留对上游的署名。