Skip to content

Commit 5665873

Browse files
author
Sam Yoder
committed
Pushing model quality
1 parent 0f9039f commit 5665873

97 files changed

Lines changed: 7411 additions & 49 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎DOMINANCE_GATES.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,67 @@ Current local frontier:
5454

5555
The next rematch tier is full-corpus 15M/20M/25M using the same verifier discipline and
5656
requiring receiver-after-transfer certificates.
57+
58+
Full-corpus transition-head update:
59+
60+
- 14.32M transition-head LayerCake passes the 15M source/core and receiver-transfer gate:
61+
2.0382 BPB versus 2.0492 BPE, 122.5 s versus 131.5 s, 2.78x one-thread CPU
62+
no-repeat-4 generation, exact transfer into the 5.40M receiver, and transferred-domain
63+
BPB 1.4406 versus adapter BPB 2.1101.
64+
- 15.55M active-compute conv2 transition LayerCake reaches 2.0065 BPB versus the retained
65+
20.61M BPE at 2.0154, but trains in 134.9 s versus 113.5 s. This is a quality win and a
66+
training-time fail, so it is not promoted.
67+
68+
Verify the promoted transition-head 15M certificate:
69+
70+
```powershell
71+
python scripts/verify_scale15m_transition_frontier.py
72+
python scripts/verify_transformer_dominance_matrix.py
73+
python scripts/benchmark_cpu_deployment_resources.py
74+
python scripts/verify_game_ready_mobile_llm.py
75+
python scripts/verify_cross_backend_quality_scorecard.py
76+
python scripts/verify_many_domain_game_layers.py
77+
python scripts/verify_game_domain_training_workflow.py
78+
python scripts/verify_cross_domain_smoke_frontier.py
79+
python scripts/verify_cross_domain_adapter_frontier.py
80+
python scripts/verify_frontier_model_northstar.py
81+
```
82+
83+
The transition verifier requires explicit margins: at least 2% fewer parameters, at least
84+
0.5% better BPB, at least 1% faster training, at least 10% faster one-thread CPU
85+
generation, no extra training bytes, printable/alpha-space/diverse generation, no repeated
86+
4-gram or 8-gram in the checked sample, exact transfer PPL/logit/generation invariance,
87+
and at least 10% better transferred-domain BPB than the transformer adapter.
88+
89+
The game-ready verifier adds a deployment-oriented CPU/mobile-proxy gate: at least 2x
90+
one-thread CPU generation versus BPE, smaller/faster/better installable domain payloads
91+
than transformer adapters, exact receiver transfer, and an isolated local CPU resource
92+
certificate. The pruned LayerCake deployment artifact now beats the retained BPE on
93+
artifact size, parameter memory, peak RSS, and generation speed. The isolated CPU prefill
94+
microbench remains open, and real device latency, battery/thermal, game dialogue data,
95+
task-level game evaluation, and native int8 runtime are still not promoted.
96+
97+
The cross-backend scorecard separates CPU, GPU, latency, generation quality, training,
98+
and domain layers. Current CPU, latency, training, and domain gates pass; GPU generation
99+
quality passes but GPU generation speed remains OPEN, so no across-the-board CPU+GPU
100+
dominance claim is allowed.
101+
102+
The frontier north-star verifier is the master gate for the repository. It passes only
103+
when the currently promoted evidence is internally consistent, and separately reports
104+
open items that block the final claim: GPU generation speed, 20M training time, real
105+
mobile/device measurements, isolated CPU prefill, native int8 runtime, trained
106+
game-domain payloads, task-level NPC evaluation, and domain routing policy.
107+
108+
The game-domain workflow smoke proves custom game-style text can train a portable
109+
byte-GRU payload, quantize to int8, install into separate LayerCake runtimes, and migrate
110+
with exact PPL/logit/generation invariance. It is a workflow gate, not production
111+
dialogue-quality evidence.
112+
113+
The cross-domain smoke extends the same portable-payload workflow across dialogue, lore,
114+
quest/state, and technical prose. It must pass exact transfer and smoke quality gates for
115+
every domain. It still does not replace large external corpora, matched transformer
116+
adapter comparisons per domain, multi-seed runs, or task-level quality evaluation.
117+
118+
`verify_cross_domain_adapter_frontier.py` adds the matched-adapter smoke comparison:
119+
LayerCake must beat a BPE residual adapter on BPB, training seconds, and payload size for
120+
each domain while preserving exact source/receiver transfer.

‎NEXT_STEPS.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,3 +231,70 @@ selection now clear the 276k, 474k, 735k, 1.15M, 2.7M, 5.8M, 8.8M, 10.4M, 12.8M,
231231
19.4M, and 25.6M local probes. The next step is a controlled full-corpus
232232
15M/20M/25M rematch using the same dominance certificates and receiver-after-transfer
233233
checks.
234+
235+
The first full-corpus transition-head rematch is now promoted at 15M class:
236+
237+
- 14.32M transition-head LayerCake: 2.0382 BPB, 122.5 s, 9.42M train bytes;
238+
- 14.84M BPE comparator: 2.0492 BPB, 131.5 s, 10.32M estimated train bytes;
239+
- one-thread CPU no-repeat-4 generation: 2.78x BPE with stricter diversity gates;
240+
- transfer into the 5.40M receiver: PPL ratio 1.0, max logit diff 0, identical generation;
241+
- transferred-domain BPB: 1.4406 versus adapter BPB 2.1101.
242+
243+
The best current 20M-comparator candidate is the 15.55M active-compute conv2 transition
244+
model. It reaches 2.0065 BPB against the 20.61M BPE's 2.0154, but still misses the
245+
training-time gate at 134.9 s versus 113.5 s. The next architecture loop should target
246+
global/local fused training cost or faster convergence, because conv substitution closes
247+
part of the gap but does not yet win the 20M time gate.
248+
249+
Promotion now requires `python scripts/verify_transformer_dominance_matrix.py`. This
250+
matrix keeps local-methodology, full-corpus source, and receiver-after-transfer gates
251+
separate, and records the 20M result as OPEN rather than allowing a partial quality win to
252+
be marketed as full transformer dominance.
253+
254+
For the game-ready north star, promotion also requires
255+
`python scripts/verify_game_ready_mobile_llm.py`. It keeps the target concrete:
256+
CPU/mobile-proxy English generation, installable domain payloads, exact receiver transfer,
257+
and open requirements for real mobile hardware and task-level game evaluation. The next
258+
domain-specific step is to replace the current Python-domain payload with a small
259+
game-dialogue/game-state corpus and add NPC dialogue or game QA task gates.
260+
261+
CPU deployment resources are now measured with
262+
`python scripts/benchmark_cpu_deployment_resources.py`. The benchmark exports a pruned
263+
patch-runtime LayerCake artifact, runs LayerCake and BPE in separate fresh Python
264+
processes, and records artifact size, parameter memory, peak RSS, prefill, and generation.
265+
The current pruned LayerCake runtime is smaller, has lower peak RSS, and generates faster
266+
than the retained BPE comparator; the isolated prefill microbench remains open and should
267+
be treated as a local CPU optimization target.
268+
269+
Across-the-board backend claims require `python scripts/verify_cross_backend_quality_scorecard.py`.
270+
That scorecard currently passes training/quality/cost, CPU generation quality/speed,
271+
batch-1 prefill latency, and domain-layer gates, but keeps GPU generation speed OPEN.
272+
The no-repeat cached decoder now uses a tensorized mask instead of Python `tolist`
273+
candidate selection, improving the retained GPU no-repeat path while preserving quality.
274+
The next GPU-specific architecture task is a cached generation fast path that further
275+
reduces per-byte Python/kernel overhead or emits larger verified chunks without degrading
276+
the quality gates.
277+
278+
The master frontier gate is `python scripts/verify_frontier_model_northstar.py`. It now
279+
includes a many-domain proxy via `python scripts/verify_many_domain_game_layers.py`.
280+
That proxy proves three game-named payloads can be installed, migrated exactly, and kept
281+
isolated with zero selected-domain logit interference. The next non-proxy game task is to
282+
train real `game_dialogue`, `game_lore`, and `game_quest_state` payloads and add task-level
283+
NPC/game QA gates plus a routing-policy evaluator.
284+
285+
The custom game-domain training path is now smoke-tested:
286+
`train_portable_domain_decoder.py` accepts `--domain-file`, `eval_lossless_domain_decoder.py`
287+
accepts `--eval-file`, and `verify_game_domain_training_workflow.py` verifies a
288+
train/quantize/install/migrate loop on a game-dialogue fixture. This makes the repo ready
289+
to ingest a real game corpus, but the production claim still requires larger held-out
290+
game data and task-level dialogue/quest evaluation.
291+
292+
`verify_cross_domain_smoke_frontier.py` now broadens that smoke from one fixture to four
293+
text styles: game dialogue, game lore, game quest/state, and technical prose. The next
294+
promotion step is not more tiny fixtures; it is larger external corpora, matched
295+
transformer adapters per domain, multiple seeds, and task-level quality gates.
296+
297+
`verify_cross_domain_adapter_frontier.py` now supplies the first matched-adapter version
298+
of that check. The four smoke domains beat BPE residual adapters on BPB, training time,
299+
payload size, and exact transfer. The lore margin is very small, so the next promotion
300+
still needs external corpora and multi-seed confidence intervals.

‎README.md‎

Lines changed: 149 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,153 @@ Verify the combined core and migration certificate:
101101
python scripts/verify_northstar_mobile.py
102102
```
103103

104+
### New transition-head 15M frontier
105+
106+
The empirical byte-transition head and narrowed local decoder now produce a stronger
107+
15M-class source/core result while preserving exact receiver migration:
108+
109+
| Gate | LayerCake transition result | Comparator / threshold | Status |
110+
|---|---:|---:|---|
111+
| Parameters | **14.320M** | BPE: 14.844M | PASS |
112+
| General held-out BPB | **2.0382** | BPE: 2.0492 | PASS |
113+
| Training time, no profiling | **122.5 s** | BPE: 131.5 s | PASS |
114+
| Training bytes | **9.42M** | BPE: 10.32M estimated | PASS |
115+
| One-thread CPU no-repeat-4 generation | **2.78x BPE** | ratio > 1.10 plus diversity gates | PASS |
116+
| Lossless transfer to 5.40M receiver | PPL ratio 1.0; max logit diff 0; identical generation | exact | PASS |
117+
| Transferred-domain BPB | **1.4406** | adapter: 2.1101 | PASS |
118+
119+
Verify:
120+
121+
```powershell
122+
python scripts/verify_scale15m_transition_frontier.py
123+
python scripts/verify_transformer_dominance_matrix.py
124+
python scripts/verify_game_ready_mobile_llm.py
125+
python scripts/benchmark_cpu_deployment_resources.py
126+
python scripts/verify_cross_backend_quality_scorecard.py
127+
python scripts/verify_many_domain_game_layers.py
128+
python scripts/verify_game_domain_training_workflow.py
129+
python scripts/verify_cross_domain_smoke_frontier.py
130+
python scripts/verify_cross_domain_adapter_frontier.py
131+
python scripts/verify_frontier_model_northstar.py
132+
```
133+
134+
A 15.55M active-compute conv2 transition variant also produced a 20M-comparator quality
135+
win over the retained 20.61M BPE comparator, 2.0065 BPB versus 2.0154, but it trained in
136+
134.9 seconds versus the BPE comparator's 113.5 seconds. That is progress, not a 20M
137+
promotion.
138+
139+
### Game-ready CPU/mobile proxy gate
140+
141+
The current game-deployment thesis is now tracked separately from broad scale dominance:
142+
a small CPU-first English core plus installable domain payloads for game-specific data.
143+
144+
| Gate | Current evidence | Status |
145+
|---|---:|---|
146+
| Core smaller than BPE | 14.32M vs 14.84M params | PASS |
147+
| General English BPB | 2.0382 vs 2.0492 BPE | PASS |
148+
| Training time | 122.5 s vs 131.5 s BPE | PASS |
149+
| One-thread CPU generation | 2.78x BPE | PASS |
150+
| Domain payload size | 148,808 B vs 383,008 B adapter | PASS |
151+
| Domain training time | 51.3 s vs 183.1 s adapter | PASS |
152+
| Domain CPU throughput | 35.7K B/s vs 8.1K B/s adapter | PASS |
153+
| Lossless domain transfer | PPL ratio 1.0; max logit diff 0; identical generation | PASS |
154+
| Receiver after transfer | smaller, better BPB, faster training, faster CPU generation | PASS |
155+
| Pruned CPU deployment artifact | 0.96x BPE artifact size | PASS |
156+
| Isolated CPU peak RSS | 0.985x BPE peak RSS | PASS |
157+
| Isolated CPU generation | 2.13x BPE | PASS |
158+
| Isolated CPU prefill microbench | 0.86x BPE | OPEN |
159+
160+
Verify:
161+
162+
```powershell
163+
python scripts/benchmark_cpu_deployment_resources.py
164+
python scripts/verify_game_ready_mobile_llm.py
165+
```
166+
167+
This is still a desktop CPU/mobile-proxy certificate. Real game shipping still requires
168+
Android/iOS or target-console latency, battery/thermal, a game-dialogue/domain dataset,
169+
task-level NPC/game QA evaluation, and a native int8 runtime. Local isolated CPU peak RSS
170+
is now measured with separate fresh Python processes and passes against the retained BPE
171+
comparator; the separate isolated prefill microbench remains open.
172+
173+
### Cross-backend quality scorecard
174+
175+
LayerCake now tracks backend and quality dimensions separately so a CPU/mobile win cannot
176+
hide a GPU loss.
177+
178+
| Dimension | Current result | Status |
179+
|---|---:|---|
180+
| Training/quality/cost vs BPE | smaller, lower BPB, faster training, fewer bytes | PASS |
181+
| CPU generation quality/speed | quality gates pass; 317.1 B/s vs 146.8 B/s | PASS |
182+
| Batch-1 prefill latency | 2.96 ms vs 5.63 ms BPE | PASS |
183+
| Domain layers | smaller/faster/better than adapter; exact transfer | PASS |
184+
| GPU generation quality | quality gates pass | PASS |
185+
| GPU generation speed | 244.2 B/s vs 840.2 B/s BPE | OPEN |
186+
187+
Verify:
188+
189+
```powershell
190+
python scripts/verify_cross_backend_quality_scorecard.py
191+
```
192+
193+
An across-the-board CPU+GPU dominance claim is blocked until GPU generation speed also
194+
beats the transformer comparator.
195+
196+
### Frontier north-star gate
197+
198+
The master verifier aggregates the current promoted frontier evidence and explicitly
199+
keeps the larger north-star claim open until every remaining game/deployment gate exists.
200+
201+
```powershell
202+
python scripts/verify_frontier_model_northstar.py
203+
```
204+
205+
Current promoted gates:
206+
207+
- base 15M source/core frontier;
208+
- transformer dominance matrix promoted tiers;
209+
- cross-backend CPU/mobile-proxy scorecard;
210+
- game-ready CPU/mobile proxy;
211+
- receiver-after-transfer frontier;
212+
- many-domain install/migration/isolation mechanics.
213+
214+
Current open north-star items:
215+
216+
- GPU generation speed;
217+
- 20M full-corpus training-time dominance;
218+
- real mobile/device latency;
219+
- battery and thermal measurements;
220+
- isolated CPU prefill microbench;
221+
- native int8 runtime;
222+
- trained game-dialogue, lore, and quest-state payloads;
223+
- task-level NPC/game QA evaluation;
224+
- domain routing policy evaluation.
225+
226+
The many-domain proxy currently installs `game_dialogue`, `game_lore`, and
227+
`game_quest_state` payloads, verifies exact source/receiver migration for each, and
228+
checks that installing other domains does not change the selected domain's logits. It uses
229+
renamed copies of the current portable payload, so it proves install/migration/isolation
230+
mechanics, not game-domain quality.
231+
232+
The game-domain workflow smoke now trains a byte-GRU portable domain from
233+
`tests/fixtures/game_dialogue_smoke.txt`, quantizes it to int8, installs it into the
234+
15M source and 5.40M receiver, and verifies exact migration. Current smoke metrics:
235+
2.2185 BPB, 73.8% top-1 byte accuracy, PPL ratio 1.0, max logit diff 0.0, and identical
236+
generated bytes after transfer. This proves the train/quantize/install/migrate workflow
237+
for game-style text; it is not a production game-dialogue quality claim.
238+
239+
The cross-domain smoke extends that workflow to dialogue, lore, quest/state, and
240+
technical text. All four payloads train, quantize to int8, transfer exactly, and pass the
241+
smoke BPB/accuracy/printability gates. Current aggregate: mean BPB 2.2414, minimum top-1
242+
byte accuracy 71.97%, max transfer logit diff 0.0. This is broader workflow evidence, not
243+
an all-corpora dominance claim.
244+
245+
The cross-domain adapter frontier compares those four portable payloads against matched
246+
BPE residual adapters trained on the same fixture files. LayerCake wins all four smoke
247+
domains on domain BPB, training seconds, payload size, and exact source/receiver transfer.
248+
Worst BPB margin is narrow on lore, -0.0019 BPB, so this is a smoke win that needs larger
249+
external corpora and multi-seed replication before any broad domain-dominance claim.
250+
104251
## Strict same-PPL transfer
105252

106253
The original additive sparse brick does **not** preserve absolute PPL across independent
@@ -151,7 +298,8 @@ plus installable, domain-specific prediction payloads.
151298
It is not evidence that a mobile core has the same general intelligence as a larger core.
152299
PX transfers the domain capsule's behavior exactly because that capsule owns the selected
153300
domain prediction path. Routing, task-level code quality, native CPU/mobile kernels,
154-
memory, battery, and thermal behavior remain separate gates.
301+
battery, thermal behavior, and real-device memory/latency remain separate gates. Local
302+
desktop CPU peak RSS is measured separately in the deployment-resource certificate.
155303

156304
### Measured mobile domain-deployment win
157305

0 commit comments

Comments
 (0)