[submit] 670004fca640 — val by hotkey 5GW4nyVgq36T… - #1590
Open
emphity1 wants to merge 1 commit into
Open
Conversation
This patch adds a value-pathway upgrade to the canonical model as a new module (model/_v8ve.py): (1) low-rank value embeddings — three learned vocab->512 tables shared U-net style across the outermost layer pairs, injected into attention V through per-layer zero-initialized up-projections; and (2) value-residual learning (Zhu et al. 2024, arXiv:2410.17897) — a per-layer zero-initialized scalar that mixes the first block's value stream into every later block. Both paths are exactly identity at initialization, so training starts from the canonical function and the levers engage progressively as the projections grow; all architecture choices live in dataclass defaults so the checkpoint reconstructs exactly from the base config fields (strict-load safe), and total parameters stay ~345M. The motivation is the token-limited regime of this track: value embeddings front-load token-identity signal into attention values (the highest-evidence single lever from the modded-nanoGPT speedrun lineage), while the value residual transports that enriched layer-0 signal to middle layers that carry no tables. In paired same-seed, same-data-order runs against the canonical recipe, the mid-training validation gap grows steadily through the schedule (train-loss gap roughly tripling between steps 1000 and 2000 as the zero-init paths charge up), consistent with the levers compounding through the decay phase. Training uses the canonical max-MFU WSD schedule with compile in max-autotune-no-cudagraphs mode for gradient-accumulation stability on torch 2.11; total_steps is sized to the measured step time so the normalized compute stays within the 4.75 H100-hour budget, and the decay phase is extended (stable/decay 0.50/0.50, min_lr 4e-6) after measuring the anneal endgame as the strongest gradient, and weight averaging is disabled after measuring that EMA lags rather than helps under this steep 1-sqrt anneal. --- ## Submission identifiers **bundle_hash:** `670004fca640cc13d874b80b4ba077e7baf8d704cc7a693828a125cfe8a66477` **miner_hotkey:** `5GW4nyVgq36TNAu9wJnvtC8wNvHciTzsTC87rYCx6aTFS4Te` **miner_github:** @emphity1 **signature:** `904a0310a74e3daec44a9f2d26640a8b…` Submitted via `scripts/miner_run.py`. The validator will compare this PR's diff against the bundle's `patch.diff` byte-for-byte.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch adds a value-pathway upgrade to the canonical model as a new module (model/_v8ve.py): (1) low-rank value embeddings — three learned vocab->512 tables shared U-net style across the outermost layer pairs, injected into attention V through per-layer zero-initialized up-projections; and (2) value-residual learning (Zhu et al. 2024, arXiv:2410.17897) — a per-layer zero-initialized scalar that mixes the first block's value stream into every later block. Both paths are exactly identity at initialization, so training starts from the canonical function and the levers engage progressively as the projections grow; all architecture choices live in dataclass defaults so the checkpoint reconstructs exactly from the base config fields (strict-load safe), and total parameters stay ~345M.
The motivation is the token-limited regime of this track: value embeddings front-load token-identity signal into attention values (the highest-evidence single lever from the modded-nanoGPT speedrun lineage), while the value residual transports that enriched layer-0 signal to middle layers that carry no tables. In paired same-seed, same-data-order runs against the canonical recipe, the mid-training validation gap grows steadily through the schedule (train-loss gap roughly tripling between steps 1000 and 2000 as the zero-init paths charge up), consistent with the levers compounding through the decay phase. Training uses the canonical max-MFU WSD schedule with compile in max-autotune-no-cudagraphs mode for gradient-accumulation stability on torch 2.11; total_steps is sized to the measured step time so the normalized compute stays within the 4.75 H100-hour budget, and the decay phase is extended (stable/decay 0.50/0.50, min_lr 4e-6) after measuring the anneal endgame as the strongest gradient, and weight averaging is disabled after measuring that EMA lags rather than helps under this steep 1-sqrt anneal.
Submission identifiers
bundle_hash:
670004fca640cc13d874b80b4ba077e7baf8d704cc7a693828a125cfe8a66477miner_hotkey:
5GW4nyVgq36TNAu9wJnvtC8wNvHciTzsTC87rYCx6aTFS4Teminer_github: @emphity1
signature:
904a0310a74e3daec44a9f2d26640a8b…Submitted via
scripts/miner_run.py. The validator will comparethis PR's diff against the bundle's
patch.diffbyte-for-byte.