Repository navigation
Sync Develop to Main - #47
Merged
Merged
Conversation
…-resume Add opt-in DeepSpeed Universal Checkpoint resume for cross-world-size restarts
Each config under config/constract_llm/train/{pt,ft} now lives in a
directory named after the run (stage included), matching the
eval/pt convention. Doc references updated accordingly.
…ames 1B/3B runs have size-scaled total batches (unlike the 0.5B reference at 4096), so their directory/run names now carry an explicit -bs<total> suffix (8-GPU effective total: 1B stage1/WSL 8192, 1B stage2/FT 1024, 3B stage1/WSL 16384, 3B stage2/FT 2048). All references, including eval configs and lowercase mteb variants, are renamed consistently. Config files inside each run directory are shortened to their kind (stage1/stage2/pt/ft).json since the directory carries the run name.
The bidirectional-converted Llama stage2 hub ids now carry the same -bs suffix as the runs they derive from (ModernBERT counterparts were already suffixed), and the README configuration-file lists reflect the per-run directory layout.
pt/<Model>/{stage1,stage2}[-bs<total>].json and
ft/<Model>/{pt,ft}[-bs<total>].json replace the per-run directories,
so each model's stages sit together and full paths stay short.
run_name and other in-config identifiers keep the full suffixed names.
- pt/<Model>/<task>.json: drop the redundant <Model>_ filename prefix - mteb_full_chunks/<Model>/<task>.json: split the model--task filenames - mteb, mteb_full, bench_sbert, isotropic: name files after the evaluated artifact (canonical case, full run name incl. -bs suffix) instead of lowercase / abbreviated JP aliases External baselines (sarashina2.2 base and its Bi conversion) keep their project labels since the same model is evaluated per language.
eval/pt/{glue,qa,csqa,isotropic} hold the pre-training model evals
(suite names follow scripts/constract_llm/eval/pt/run_*.py), and
eval/ft/{mteb,mteb_full,mteb_full_chunks,bench_sbert,isotropic} hold
the sentence-embedding evals. eval/analysis stays shared.
eval/ft/mteb/<Model>/{subset,full}.json and full_chunks/<task>.json
(subset = the 5-task quick run formerly under mteb/).
eval/pt/qa/<Model>/{squad|jsquad,csqa|jcsqa}.json replaces the flat
qa/ and csqa/ suites, mirroring glue/<Model>/<task>.json.
Unsuffixed 1B/3B train configs use the 0.5B effective totals on 8 GPUs (stage1/WSL 4096, stage2/FT 512), keeping per-device batch and adjusting gradient accumulation (3B FT halves per-device batch to fit 512). Matching eval configs (glue/qa/isotropic/mteb/bench_sbert) and Llama-Bi conversion configs are added; the size-scaled variants keep their -bs names (conversion configs renamed to *_bs<total>.json).
Within every eval suite the model directory is the base run name and the batch variant lives in the file name: <task>[-bs<total>].json for multi-config suites (glue, qa, mteb), default.json / bs<total>.json for single-config suites (isotropic, bench_sbert).
…_decay Use the majority style (e.g. 3e-4, 1e-5) instead of mixed 0.0003 / 3e-05 spellings; values are unchanged.
…path prefix Majority-style cleanup with values unchanged (relative paths lose the redundant ./ prefix only).
…g-directories Reorganize train/eval configs by model directory and make total batch explicit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
WHAT