train/sft.py sets train_on_completions_only using ChatML markers:
instruction_template="<|im_start|>user",
response_template="<|im_start|>assistant\n",
That is correct for Qwen, and silently wrong for anything else. A Llama or Gemma
base would find no match, and the collator would mask nothing or everything —
either way the run completes and produces a model that is quietly worse, with no
error to point at.
Fix: derive the markers from the tokenizer's chat template, or keep a small
registry keyed by model family and fail loudly when the base model is not in
it. Silent misconfiguration that still trains is the worst failure mode here.
Reproducing it is cheap: point configs/smoke-1.5b.yaml at a non-Qwen base and
inspect the collator's labels for a batch.
train/sft.pysetstrain_on_completions_onlyusing ChatML markers:That is correct for Qwen, and silently wrong for anything else. A Llama or Gemma
base would find no match, and the collator would mask nothing or everything —
either way the run completes and produces a model that is quietly worse, with no
error to point at.
Fix: derive the markers from the tokenizer's chat template, or keep a small
registry keyed by model family and fail loudly when the base model is not in
it. Silent misconfiguration that still trains is the worst failure mode here.
Reproducing it is cheap: point
configs/smoke-1.5b.yamlat a non-Qwen base andinspect the collator's labels for a batch.