Distillation v1#18

Open

bl4ckse4m wants to merge 13 commits into

mainfrom

Commits on Apr 10, 2026

Fixed some issues with launching the project and added proper support for openrouter
bl4ckse4m
committed
Merge branch 'main' into distillation
bl4ckse4m
committed

Commits on Apr 17, 2026

Distillation pipeline on hotpot with smolLM2-1.7B-Instruct, plus workaround for trl reading wrong encoding on windows machine
bl4ckse4m
committed

Commits on May 1, 2026

Changed training for completion-only task, minor changes to triplet extraction function
bl4ckse4m
committed

Commits on May 3, 2026

Commits on May 17, 2026

Commits on May 20, 2026

SFTTrainer instead of basic Trainer
bl4ckse4m
committed

Commits on May 24, 2026

Changed lora_alpha from 32 to 64, added target modules k_proj, o_proj, trained longer for 5 epochs
bl4ckse4m
committed

Commits on May 25, 2026

Commits on May 31, 2026

Removed curriculum and hand-tuned weighting, replaced manual CE+weights with focal loss
bl4ckse4m
committed