Repository navigation
Reorganize train/eval configs by model directory and make total batch explicit - #48
Merged
iamtatsuki05 merged 13 commits intoAug 28, 2026
Merged
Conversation
Each config under config/constract_llm/train/{pt,ft} now lives in a
directory named after the run (stage included), matching the
eval/pt convention. Doc references updated accordingly.
…ames 1B/3B runs have size-scaled total batches (unlike the 0.5B reference at 4096), so their directory/run names now carry an explicit -bs<total> suffix (8-GPU effective total: 1B stage1/WSL 8192, 1B stage2/FT 1024, 3B stage1/WSL 16384, 3B stage2/FT 2048). All references, including eval configs and lowercase mteb variants, are renamed consistently. Config files inside each run directory are shortened to their kind (stage1/stage2/pt/ft).json since the directory carries the run name.
The bidirectional-converted Llama stage2 hub ids now carry the same -bs suffix as the runs they derive from (ModernBERT counterparts were already suffixed), and the README configuration-file lists reflect the per-run directory layout.
pt/<Model>/{stage1,stage2}[-bs<total>].json and
ft/<Model>/{pt,ft}[-bs<total>].json replace the per-run directories,
so each model's stages sit together and full paths stay short.
run_name and other in-config identifiers keep the full suffixed names.
- pt/<Model>/<task>.json: drop the redundant <Model>_ filename prefix - mteb_full_chunks/<Model>/<task>.json: split the model--task filenames - mteb, mteb_full, bench_sbert, isotropic: name files after the evaluated artifact (canonical case, full run name incl. -bs suffix) instead of lowercase / abbreviated JP aliases External baselines (sarashina2.2 base and its Bi conversion) keep their project labels since the same model is evaluated per language.
eval/pt/{glue,qa,csqa,isotropic} hold the pre-training model evals
(suite names follow scripts/constract_llm/eval/pt/run_*.py), and
eval/ft/{mteb,mteb_full,mteb_full_chunks,bench_sbert,isotropic} hold
the sentence-embedding evals. eval/analysis stays shared.
eval/ft/mteb/<Model>/{subset,full}.json and full_chunks/<task>.json
(subset = the 5-task quick run formerly under mteb/).
eval/pt/qa/<Model>/{squad|jsquad,csqa|jcsqa}.json replaces the flat
qa/ and csqa/ suites, mirroring glue/<Model>/<task>.json.
Unsuffixed 1B/3B train configs use the 0.5B effective totals on 8 GPUs (stage1/WSL 4096, stage2/FT 512), keeping per-device batch and adjusting gradient accumulation (3B FT halves per-device batch to fit 512). Matching eval configs (glue/qa/isotropic/mteb/bench_sbert) and Llama-Bi conversion configs are added; the size-scaled variants keep their -bs names (conversion configs renamed to *_bs<total>.json).
Within every eval suite the model directory is the base run name and the batch variant lives in the file name: <task>[-bs<total>].json for multi-config suites (glue, qa, mteb), default.json / bs<total>.json for single-config suites (isotropic, bench_sbert).
…_decay Use the majority style (e.g. 3e-4, 1e-5) instead of mixed 0.0003 / 3e-05 spellings; values are unchanged.
…path prefix Majority-style cleanup with values unchanged (relative paths lose the redundant ./ prefix only).
iamtatsuki05
force-pushed
the
feature/organize-train-config-directories
branch
from
August 28, 2026 11:38
55c9b0a to
a4dedb1
Compare
iamtatsuki05
marked this pull request as ready for review
August 28, 2026 11:40
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
The training configs under
config/constract_llm/train/{pt,ft}were flat files, and the eval configs mixed several naming schemes (canonical case vs lowercase, abbreviated JP aliases such assentence-llama-bi-1b,<Model>_<task>.jsonprefixes,model--taskfilenames), which made it hard to see which configs belong to the same model. In addition, the 1B/3B configs implicitly used size-scaled total batches (8192/16384 for pre-training, 1024/2048 for stage-2/fine-tuning at 8 GPUs) while the 0.5B recipe uses 4096/512, but nothing in the names said so, and the batch-matched 1B/3B recipe used by the current re-measurement campaigns had no canonical config in the repository at all.WHAT
Train config layout
train/pt/<Model>/{stage1,stage2}[-bs<total>].jsonandtrain/ft/<Model>/{pt,ft}[-bs<total>].json: one directory per model, stages as files (e.g.pt/Llama-EN-1B/stage1.json,ft/Sentence-Llama-Bi-EN-1B/ft-bs1024.json)-bs<total>suffix in file names,run_name,output_dir, hub ids and every eval reference (1B 8192/1024, 3B 16384/2048); 0.5B stays unsuffixedmodel/save_custom_model/llama_bi_*: unsuffixed configs for the batch-matched runs,*_bs<total>.jsonfor the size-scaled ones (Llama-Bi stage-2 hub ids are suffixed the same way as the ModernBERT ones)Eval config layout
eval/pt/{glue,qa,isotropic}for pre-training models (suite names followscripts/constract_llm/eval/pt/run_*.py;qagroups squad/csqa) andeval/ft/{mteb,bench_sbert,isotropic}for sentence models;mteb_fullandmteb_full_chunksare folded intomteb/<Model>/{subset,full}.json+full_chunks/<task>.jsonmodel_name_or_path, with the batch variant in the file name:<task>[-bs<total>].jsonfor multi-config suites,default.json/bs<total>.jsonfor single-config suitesFormatting
learning_rate/adam_epsilon/weight_decayuse the majority one-digit exponent notation (3e-4,1e-5) instead of mixed0.0003/3e-05./prefix on path valuesVERIFICATION
-bsrename (no hyperparameter changes), and every new unsuffixed 1B/3B config satisfiesper_device × 8 × grad_accum == 0.5B totalmodel_name_or_pathand its file-name batch suffix equals the model's (checked for alliamtatsuki05/*references); task file names matchtask_name.py/.sh/.md/.yaml/.json; every config path mentioned in the READMEs exists; no code, test, or CI job globs the config directories-bsartifacts need the old ids until they are re-uploaded or renamed; the campaign-side overrides (eval_steps400/800,save_total_limit80/unlimited,use_reentrant: falsefor ST) are intentionally not folded into these configs