Repository navigation
Set 3B pre-training budget to 60B tokens (stage1 + stage2) per model - #52
Merged
Merged
Conversation
Align the 3B English pre-training budget with the 1B design (stage1 + stage2 ~= 20x non-embedding parameters, same epoch for Llama and ModernBERT within a language): - stage1 (FineWeb-Edu, seq 1024): 0.45 -> 0.65 epoch Llama-EN-3B 16,270 steps / 68.2B tokens, ModernBERT-EN-3B 13,892 steps / 58.3B tokens - stage2 (wiki40b, seq 8192): 3 -> 1 epoch (same as 1B) ~2.5B tokens each Totals: Llama-EN-3B 70.7B (25x of 2.83B), ModernBERT-EN-3B 60.8B (19.9x of 3.05B). JP configs are unchanged (0.8 / 6 epochs).
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The 3B pre-training budgets were epoch-based and uneven across models (EN 47B / 40B, JP 65B / 89B tokens). This PR fixes the budget to 60B tokens per model, stage1 + stage2 combined, mirroring how the 1B stage2 runs were started from a stage1 checkpoint that brought the total to ~20B.
Change
Llama-EN-3B/stage1.jsonLlama-JP-3B/stage1.jsonModernBERT-EN-3B/stage1.jsonModernBERT-JP-3B/stage1.jsonLlama-EN-3B/stage2.json,ModernBERT-EN-3B/stage2.jsonStep counts assume total batch 4096 × seq 1024 in stage1; per-step token actuals (4.19M CLM, 4.13–4.18M MLM with padding) come from
trainer_state.num_input_tokens_seen. Stage2 token counts are the 1B measurements (same data and total batch 512 × 8192).JP stage2 (6 epochs) and the
-bs16384/-bs2048legacy variants are untouched.Notes
mirei-1b3b-redo-20260725use explicitmax_steps13,700 / 13,800; GCPmben3-s1uses 0.65 epoch) are updated separately from this PR. ModernBERT-JP-3B stage1 already passed 13,800; its stage2 starts fromcheckpoint-13800.