Skip to content

Reorganize train/eval configs by model directory and make total batch explicit - #48

Merged
iamtatsuki05 merged 13 commits into
developfrom
feature/organize-train-config-directories
Aug 28, 2026
Merged

iamtatsuki05 merged 13 commits into
developfrom
feature/organize-train-config-directories

Conversation

@iamtatsuki05

@iamtatsuki05 iamtatsuki05 commented Aug 28, 2026 •

Copy link
Copy Markdown
Owner

WHY

The training configs under config/constract_llm/train/{pt,ft} were flat files, and the eval configs mixed several naming schemes (canonical case vs lowercase, abbreviated JP aliases such as sentence-llama-bi-1b, <Model>_<task>.json prefixes, model--task filenames), which made it hard to see which configs belong to the same model. In addition, the 1B/3B configs implicitly used size-scaled total batches (8192/16384 for pre-training, 1024/2048 for stage-2/fine-tuning at 8 GPUs) while the 0.5B recipe uses 4096/512, but nothing in the names said so, and the batch-matched 1B/3B recipe used by the current re-measurement campaigns had no canonical config in the repository at all.

WHAT

Train config layout

  • train/pt/<Model>/{stage1,stage2}[-bs<total>].json and train/ft/<Model>/{pt,ft}[-bs<total>].json: one directory per model, stages as files (e.g. pt/Llama-EN-1B/stage1.json, ft/Sentence-Llama-Bi-EN-1B/ft-bs1024.json)
  • Unsuffixed 1B/3B configs are new and match the 0.5B effective total batch on 8 GPUs (stage1 / WSL 4096, stage2 / FT 512); per-device batch is kept and gradient accumulation recomputed (3B FT halves per-device batch to fit 512)
  • The former size-scaled 1B/3B configs carry an explicit -bs<total> suffix in file names, run_name, output_dir, hub ids and every eval reference (1B 8192/1024, 3B 16384/2048); 0.5B stays unsuffixed
  • model/save_custom_model/llama_bi_*: unsuffixed configs for the batch-matched runs, *_bs<total>.json for the size-scaled ones (Llama-Bi stage-2 hub ids are suffixed the same way as the ModernBERT ones)

Eval config layout

  • eval/pt/{glue,qa,isotropic} for pre-training models (suite names follow scripts/constract_llm/eval/pt/run_*.py; qa groups squad/csqa) and eval/ft/{mteb,bench_sbert,isotropic} for sentence models; mteb_full and mteb_full_chunks are folded into mteb/<Model>/{subset,full}.json + full_chunks/<task>.json
  • Every suite uses one directory per model named after model_name_or_path, with the batch variant in the file name: <task>[-bs<total>].json for multi-config suites, default.json / bs<total>.json for single-config suites
  • External baselines (sarashina2.2 base and its Bi conversion) and same-model per-language variants keep their project labels
  • Matching eval configs are added for the new batch-matched 1B/3B models

Formatting

  • learning_rate / adam_epsilon / weight_decay use the majority one-digit exponent notation (3e-4, 1e-5) instead of mixed 0.0003 / 3e-05
  • 2-space indentation, trailing newline, and no redundant ./ prefix on path values
  • README configuration lists (EN/JA) updated to the new paths

VERIFICATION

  • All 591 config files parse; the pre-existing configs are byte-equivalent to the base after normalizing the -bs rename (no hyperparameter changes), and every new unsuffixed 1B/3B config satisfies per_device × 8 × grad_accum == 0.5B total
  • Every eval file's directory equals the base name of its model_name_or_path and its file-name batch suffix equals the model's (checked for all iamtatsuki05/* references); task file names match task_name
  • No remaining references to old paths in .py / .sh / .md / .yaml / .json; every config path mentioned in the READMEs exists; no code, test, or CI job globs the config directories
  • Two read-only reviews (no blocking findings)
  • Not covered here: the HF Hub repositories and local artifacts still use the pre-suffix names, so runs that consume existing -bs artifacts need the old ids until they are re-uploaded or renamed; the campaign-side overrides (eval_steps 400/800, save_total_limit 80/unlimited, use_reentrant: false for ST) are intentionally not folded into these configs

Each config under config/constract_llm/train/{pt,ft} now lives in a
directory named after the run (stage included), matching the
eval/pt convention. Doc references updated accordingly.
…ames

1B/3B runs have size-scaled total batches (unlike the 0.5B reference
at 4096), so their directory/run names now carry an explicit -bs<total>
suffix (8-GPU effective total: 1B stage1/WSL 8192, 1B stage2/FT 1024,
3B stage1/WSL 16384, 3B stage2/FT 2048). All references, including
eval configs and lowercase mteb variants, are renamed consistently.
Config files inside each run directory are shortened to their kind
(stage1/stage2/pt/ft).json since the directory carries the run name.
The bidirectional-converted Llama stage2 hub ids now carry the same
-bs suffix as the runs they derive from (ModernBERT counterparts were
already suffixed), and the README configuration-file lists reflect the
per-run directory layout.
pt/<Model>/{stage1,stage2}[-bs<total>].json and
ft/<Model>/{pt,ft}[-bs<total>].json replace the per-run directories,
so each model's stages sit together and full paths stay short.
run_name and other in-config identifiers keep the full suffixed names.
- pt/<Model>/<task>.json: drop the redundant <Model>_ filename prefix
- mteb_full_chunks/<Model>/<task>.json: split the model--task filenames
- mteb, mteb_full, bench_sbert, isotropic: name files after the
  evaluated artifact (canonical case, full run name incl. -bs suffix)
  instead of lowercase / abbreviated JP aliases
External baselines (sarashina2.2 base and its Bi conversion) keep their
project labels since the same model is evaluated per language.
eval/pt/{glue,qa,csqa,isotropic} hold the pre-training model evals
(suite names follow scripts/constract_llm/eval/pt/run_*.py), and
eval/ft/{mteb,mteb_full,mteb_full_chunks,bench_sbert,isotropic} hold
the sentence-embedding evals. eval/analysis stays shared.
eval/ft/mteb/<Model>/{subset,full}.json and full_chunks/<task>.json
(subset = the 5-task quick run formerly under mteb/).
eval/pt/qa/<Model>/{squad|jsquad,csqa|jcsqa}.json replaces the flat
qa/ and csqa/ suites, mirroring glue/<Model>/<task>.json.
@iamtatsuki05 iamtatsuki05 changed the title Organize train configs into per-run directories Reorganize train/eval configs by model directories with explicit total-batch suffixes Aug 28, 2026
Unsuffixed 1B/3B train configs use the 0.5B effective totals on 8 GPUs
(stage1/WSL 4096, stage2/FT 512), keeping per-device batch and adjusting
gradient accumulation (3B FT halves per-device batch to fit 512).
Matching eval configs (glue/qa/isotropic/mteb/bench_sbert) and Llama-Bi
conversion configs are added; the size-scaled variants keep their -bs
names (conversion configs renamed to *_bs<total>.json).
Within every eval suite the model directory is the base run name and the
batch variant lives in the file name: <task>[-bs<total>].json for
multi-config suites (glue, qa, mteb), default.json / bs<total>.json for
single-config suites (isotropic, bench_sbert).
…_decay

Use the majority style (e.g. 3e-4, 1e-5) instead of mixed 0.0003 /
3e-05 spellings; values are unchanged.
…path prefix

Majority-style cleanup with values unchanged (relative paths lose the
redundant ./ prefix only).
@iamtatsuki05
iamtatsuki05 force-pushed the feature/organize-train-config-directories branch from 55c9b0a to a4dedb1 Compare August 28, 2026 11:38
@iamtatsuki05
iamtatsuki05 marked this pull request as ready for review August 28, 2026 11:40
@iamtatsuki05 iamtatsuki05 changed the title Reorganize train/eval configs by model directories with explicit total-batch suffixes Reorganize train/eval configs by model directory and make total batch explicit Aug 28, 2026
@iamtatsuki05 iamtatsuki05 self-assigned this Aug 28, 2026
@iamtatsuki05
iamtatsuki05 merged commit 593a6ab into develop Aug 28, 2026
2 checks passed
@iamtatsuki05 iamtatsuki05 mentioned this pull request Aug 28, 2026
@iamtatsuki05
iamtatsuki05 deleted the feature/organize-train-config-directories branch September 6, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant