Phase 5c: Parts III + IV become data — every teaching chapter is now blocks

src/components/GuideBlocks.svelte

@@ -30,6 +30,7 @@
     figure,
     conceptFig,
     widget,
+    cards,
     demo,
     onLesson,
     onPreset
@@ -42,6 +43,8 @@
     conceptFig?: Snippet<[string]>;
     /** Renders an interactive island (widget blocks). */
     widget?: Snippet<[string]>;
+    /** Renders a family-tree chapter's optimizer-card slice (optcards blocks). */
+    cards?: Snippet<[string]>;
     /** Chapter-demo override when the CTA isn't the slug's chapterPreset. */
     demo?: { label: string; run: () => void };
     onLesson?: (slug: string) => void;
@@ -96,6 +99,13 @@
     </div>
   {:else if b.kind === 'widget'}
     {#if widget}{@render widget(b.id)}{/if}
+  {:else if b.kind === 'optcards'}
+    {#if cards}{@render cards(b.chapter)}{/if}
+  {:else if b.kind === 'frontier'}
+    <div class="opt-frontier">
+      <div class="opt-frontier-title">{b.title}</div>
+      <p>{@html richToHtml(b.text, dark)}</p>
+    </div>
   {:else if b.kind === 'concept'}
     <div class="concept">
       <div class="concept-text">

src/content/blocks.ts

@@ -48,6 +48,10 @@ export type Block =
   | { kind: 'hd'; text: Rich }
   /** A figure: the id names an app-side (later: computed) visual; the caption lives here. */
   | { kind: 'figure'; id: string; caption: Rich }
+  /** The optimizer story cards for one family-tree chapter (data in optimizerCards.ts). */
+  | { kind: 'optcards'; chapter: string }
+  /** The dashed closing box — the frontier beyond the playground. */
+  | { kind: 'frontier'; title: string; text: Rich }
   /**
    * An interactive island with no print analogue of its own (the schedule
    * gallery, later the race) — the shell supplies it; print will use a

src/content/chapters/ch-generalize.ts

@@ -0,0 +1,39 @@
+import type { Block } from '../blocks';
+
+/** Chapter 11 · Training loss isn’t the goal — generalization, the whole point. */
+export const chGeneralize: Block[] = [
+  {
+    kind: 'p',
+    text: 'Every chapter so far has worked to drive the *training* loss down. But that number is only a stand-in for what we actually want. We don’t care about fitting the data we already have — we care about predicting data we **haven’t seen**. Doing well on new data is **generalization**, and it is the whole point.'
+  },
+  {
+    kind: 'p',
+    text: 'The loss we minimize is the average error over the training set — the **empirical risk** — but the real target is the average error over *all* future data, the **true risk**. With limited or noisy data the two come apart. Push the training loss too low and the model starts memorizing the quirks and noise of *this* sample: training loss keeps falling while error on held-out data turns and climbs. That divergence is **overfitting**.'
+  },
+  {
+    kind: 'figure',
+    id: 'generalize-curves',
+    caption: 'Training loss (green) keeps falling; test loss (amber), measured on held-out data, bottoms out and then rises as the model begins fitting noise. The dip is where you’d want to stop.'
+  },
+  {
+    kind: 'p',
+    text: 'Two fixes follow directly. The first is to *measure* the gap: hold out part of the data as a **test** (or validation) set, and watch its loss alongside the training loss — that is the second curve in the **Loss History** panel. The second is **early stopping**: end training at the test-loss minimum rather than the training-loss minimum. It is the simplest regularizer there is, and — for a run started near zero — in the quadratic case it is provably close to an explicit weight penalty (Bishop, 1995; Goodfellow et al., 2016, §7.8).'
+  },
+  {
+    kind: 'p',
+    text: 'That penalty is **regularization**: instead of minimizing the loss alone, add a term that prefers smaller, simpler parameters,'
+  },
+  { kind: 'display', formula: 'regularizedLoss', center: true },
+  {
+    kind: 'p',
+    text: 'where $\\lambda$ sets how hard to pull toward zero. (An unrelated $\\lambda$, by the way — not the curvature $\\lambda_{\\max}$ from the learning-rate chapter. The alphabet is small and the field is greedy.) For plain SGD the gradient of that penalty is exactly **weight decay** — $\\boldsymbol{\\theta} \\leftarrow (1-\\gamma\\lambda)\\,\\boldsymbol{\\theta} - \\gamma\\nabla\\mathcal{L}$ — shrinking every weight a touch each step (Krogh & Hertz, 1991). Keep this $\\lambda$ in mind: you will meet it again on **AdamW** in the family tree, which decouples the decay from the adaptive scaling so it behaves like a true penalty again.'
+  },
+  {
+    kind: 'p',
+    text: 'Geometry has the last word, and it loops back to the noise chapter. Not all minima generalize equally: a *wide, flat* basin is forgiving — small shifts in the data barely move the loss — while a *sharp* one is brittle. Flat minima tend to generalize better (Hochreiter & Schmidhuber, 1997), the restless noise of small-batch SGD tends to settle into them, and very large batches tend to find sharper minima with a measurable generalization gap (Keskar et al., 2017). So the real target was never the exact bottom of the training bowl — it is a low, *wide* region that also sits low on data you will never see. Optimization gets you down; generalization decides whether down was worth reaching.'
+  },
+  {
+    kind: 'hd',
+    text: 'Two honest asterisks on this tidy story. First, “flat” is slippery: a network can be rescaled — same function, same predictions — while its measured sharpness changes arbitrarily, so naive flatness can’t be the whole answer (Dinh et al., 2017). Second, at scale good minima aren’t isolated dips like the ones drawn here: they connect into long low-loss valleys you can walk between without climbing (Garipov et al., 2018). The intuition survives — restless SGD prefers forgiving regions — but hold it as a compass, not a theorem.'
+  }
+];

src/content/chapters/ch-noise.ts

@@ -0,0 +1,42 @@
+import type { Block } from '../blocks';
+
+/** Chapter 10 · Mini-batches & the S in SGD — noisy arrows, √n, the noise ball. */
+export const chNoise: Block[] = [
+  {
+    kind: 'p',
+    text: 'Every gradient so far has been the **true** one — measured on all your data at once. That is **full-batch** descent: the **Batch size** dial set to *All*. It gives the cleanest possible arrow, and it is the most expensive thing you can do, because every single step has to read every single data point.'
+  },
+  {
+    kind: 'p',
+    text: 'Real datasets are far too large for that, so instead you *estimate* the gradient from a small random **batch** — a handful of points, freshly resampled each step. The arrow you get back is **noisy**: it jitters around the true downhill, because a different handful would have pulled in a slightly different direction. But it is cheap, and — this is the quiet miracle that makes modern training possible — it still points the right way *on average*. Averaging your way downhill through that noise is the **S** (stochastic) in **SGD**, stochastic gradient descent. (Two words of vocabulary while we are here: one batch update is an **iteration** or step; one full sweep through the whole dataset is an **epoch**.)'
+  },
+  {
+    kind: 'p',
+    text: 'Slide the **Batch size** down from *All* toward *1* and a faint **fan** of arrows opens at the marker: each ray is the gradient a different random batch would have handed you, so the *width of the fan is the noise itself.* The fewer points in the batch, the wider it spreads — and it spreads in a very specific way — the same law that steadies dice: average four rolls and the result wobbles about half as much as a single roll. The error of an average shrinks only with the *square root* of how many samples go into it, so a batch of 4 is roughly twice as steady as a batch of 1, and you need 16 to halve the noise again. That is the law of diminishing returns behind every batch-size choice: a batch of 32 already looks almost as calm as the full dataset, for a fraction of the cost.'
+  },
+  {
+    kind: 'hd',
+    text: 'At scale this √n law becomes an economic one. The useful ratio is noise to signal: below a problem-specific *critical batch size*, doubling the batch lets you (roughly) double $\\gamma$ for the same trajectory — the linear-scaling rule behind giant training runs; above it, extra data per step buys calm the run no longer needs (Goyal et al., 2017; McCandlish et al., 2018). Bigger is not better — bigger is *quieter*, and quiet has a price and a ceiling.'
+  },
+  {
+    kind: 'p',
+    text: 'And the noise is not pure cost. A little jitter is genuinely **useful**: a noisy step can rattle the marker out of a shallow dip or a flat saddle that a perfectly smooth step would have settled into and never left, and the constant restlessness tends to steer a run toward *wide, gentle* basins — the forgiving kind that generalize to new data — rather than narrow, brittle cracks. This is why a touch of stochasticity is often kept on purpose, even when the full gradient is affordable.'
+  },
+  {
+    kind: 'hd',
+    text: 'The fan tells the truth in 2-D, but up there it would look strange: two random directions among a million axes are almost always nearly *perpendicular* — there are countless ways to be orthogonal and only one way to agree. So gradient noise mostly pushes *sideways*, at right angles to the true downhill, rather than backwards against it. A noisy run drifts and wanders far more than it backtracks — one reason SGD keeps making progress even when individual arrows look hopeless.'
+  },
+  {
+    kind: 'p',
+    text: 'The bill comes due at the *end*. Because the gradient never goes quiet, SGD never fully stops: near the bottom it stops descending and starts **orbiting**, buzzing around the minimum inside a small **noise ball** whose radius grows with both the step size $\\gamma$ and the width of the fan. On the loss curve it shows up as a fuzzy *band* rather than a clean line that flatlines — the run has arrived, but it can’t hold still. This is where the **schedule** from the last chapter earns its keep: a $\\gamma$ bled toward zero draws that ball in tight, turning the restless buzz into a soft landing. Under noise, decay isn’t a luxury — it is *how a stochastic run converges at all.*'
+  },
+  {
+    kind: 'figure',
+    id: 'noise-ball',
+    caption: 'Under noisy gradients the run never quite stops — it orbits the minimum in a cloud whose radius grows with $\\gamma$ (left). Bleed $\\gamma$ toward zero and the cloud draws in to a point (right): the schedule, doing its quiet job.'
+  },
+  {
+    kind: 'look',
+    text: 'Watch it: set a small **Batch size** so the loss settles into a fuzzy band on **Const**, then switch the schedule to **Cosine** and see the band pinch shut over the final steps.'
+  }
+];

src/content/chapters/index.ts

@@ -14,6 +14,16 @@ import { chStep } from './ch-step';
 import { chGamma } from './ch-gamma';
 import { chCurvature } from './ch-curvature';
 import { chSchedule } from './ch-schedule';
+import { chNoise } from './ch-noise';
+import { chGeneralize } from './ch-generalize';
+import {
+  chRavine,
+  chMomentum,
+  chAdaptive,
+  chAdam,
+  chSecondOrder,
+  chSelfTuning
+} from './part4';
 
 export const chapterBlocks: Record<string, Block[]> = {
   'ch-bowl': chBowl,
@@ -24,5 +34,13 @@ export const chapterBlocks: Record<string, Block[]> = {
   'ch-step': chStep,
   'ch-gamma': chGamma,
   'ch-curvature': chCurvature,
-  'ch-schedule': chSchedule
+  'ch-schedule': chSchedule,
+  'ch-noise': chNoise,
+  'ch-generalize': chGeneralize,
+  'ch-ravine': chRavine,
+  'ch-momentum': chMomentum,
+  'ch-adaptive': chAdaptive,
+  'ch-adam': chAdam,
+  'ch-second-order': chSecondOrder,
+  'ch-self-tuning': chSelfTuning
 };

src/content/chapters/part4.ts

@@ -0,0 +1,62 @@
+import type { Block } from '../blocks';
+
+/**
+ * Part IV · The optimizer family tree. The prose of these chapters lives in
+ * the story cards (optimizerCards.ts); the chapters themselves are thin —
+ * the ravine opener carries the conditioning story and the race, the rest
+ * are their card slices, and the last closes with the frontier box.
+ */
+
+/** Chapter 12 · The ravine, and the race — the part opener. */
+export const chRavine: Block[] = [
+  {
+    kind: 'p',
+    text: 'Plain gradient descent has one recurring nemesis: the **ravine** — a valley far steeper across than along. The $\\gamma$ that’s safe on the steep walls is hopeless along the gentle floor, so the marker rattles wall to wall. Every optimizer in the picker is a patch for that pain (or the new pain the last patch created) — 170 years of *fix what just broke*: a single trunk of fixes that, once it reaches Adam, finally splits into the branches still being explored today. The picker is grouped to match.'
+  },
+  {
+    kind: 'p',
+    text: 'That ravine has a precise name: **ill-conditioning**. A smooth bowl curves at two rates — gently along its floor ($\\lambda_{\\min}$) and steeply across it ($\\lambda_{\\max}$) — and their ratio is the **condition number** $\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$. A round bowl has $\\kappa = 1$ and one good step reaches the bottom; a long, thin ravine has a huge $\\kappa$, and that one number sets how slowly you converge. Even with the best fixed step, $\\gamma = 2/(\\lambda_{\\min}+\\lambda_{\\max})$, each move closes the gap to the minimum by only a factor $(\\kappa-1)/(\\kappa+1)$ — which creeps toward 1 as $\\kappa$ grows, so a stretched valley crawls no matter how you tune $\\gamma$. Momentum sharpens that to roughly $(\\sqrt{\\kappa}-1)/(\\sqrt{\\kappa}+1)$, a $\\sqrt{\\kappa}$ speed-up — the first hint of why the whole family below exists.'
+  },
+  {
+    kind: 'figure',
+    id: 'ravine-heat',
+    caption: 'The ravine: a valley far steeper across than along. One safe step size makes plain GD (white) rattle wall to wall while it crawls along the floor; momentum (violet) builds speed down the valley and glides to the minimum.'
+  },
+  {
+    kind: 'p',
+    text: 'Every fix that follows is a leaf on one tree. Here is the whole lineage at a glance — 170 years from Cauchy’s root to today’s canopy, each branch running parent → child:'
+  },
+  {
+    kind: 'figure',
+    id: 'family-tree',
+    caption: 'Every leaf is an optimizer in the picker; branches run parent → child, and the dashed violet strand marks where momentum and the adaptive line merge into Adam. Colours match the race below, and new methods join the canopy as the field grows.'
+  },
+  {
+    kind: 'p',
+    text: 'Here they are racing on the same ravine from the same start — every one running its real update rule, the dots arriving in their true step counts. Click a name to add or remove it; hover one to pick it out of the pack:'
+  },
+  { kind: 'widget', id: 'ravine-race' },
+  { kind: 'optcards', chapter: 'ch-ravine' }
+];
+
+/** Chapter 13 · Momentum & Nesterov. */
+export const chMomentum: Block[] = [{ kind: 'optcards', chapter: 'ch-momentum' }];
+
+/** Chapter 14 · A learning rate per parameter. */
+export const chAdaptive: Block[] = [{ kind: 'optcards', chapter: 'ch-adaptive' }];
+
+/** Chapter 15 · Adam — and the fork. */
+export const chAdam: Block[] = [{ kind: 'optcards', chapter: 'ch-adam' }];
+
+/** Chapter 16 · Second order: Newton & Sophia. */
+export const chSecondOrder: Block[] = [{ kind: 'optcards', chapter: 'ch-second-order' }];
+
+/** Chapter 17 · The last knob — Prodigy, and the frontier past the playground. */
+export const chSelfTuning: Block[] = [
+  { kind: 'optcards', chapter: 'ch-self-tuning' },
+  {
+    kind: 'frontier',
+    title: 'The frontier — and why it isn’t in the picker',
+    text: 'The optimizers winning 2025’s biggest training runs — **Muon** (used to train Kimi K2 and GLM), **Shampoo**, and **SOAP** — share a trick this playground can’t show. They treat a layer’s weights as a *matrix* and precondition *across* it: Muon (*momentum orthogonalized by Newton–Schulz*) straightens the momentum matrix, Shampoo and SOAP whiten it. With only two independent numbers, $\\alpha$ and $\\beta$, there is no matrix to exploit — strip the structure away and they collapse to methods already in the list. That matrix structure is exactly why they scale to billions of parameters, and exactly why a two-parameter sandbox is the wrong stage for them. To meet them you have to leave the playground — which is a fair note to end the tree on.'
+  }
+];

src/content/formulas.ts

@@ -11,6 +11,7 @@ export const formulas = {
   gradientDefinition: String.raw`\nabla \mathcal{L} = \begin{bmatrix} \partial \mathcal{L}/\partial \alpha \\[2pt] \partial \mathcal{L}/\partial \beta \end{bmatrix}`,
   stepRule: String.raw`\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma\, \nabla \mathcal{L}`,
   stability: String.raw`\gamma < \frac{2}{\lambda_{\max}}`,
+  regularizedLoss: String.raw`\min_{\boldsymbol{\theta}}\;\; \mathcal{L}(\boldsymbol{\theta}) \;+\; \tfrac{\lambda}{2}\,\lVert \boldsymbol{\theta}\rVert^2`,
   // The derivative chapter: nudge-and-divide, then let the nudge shrink.
   derivativeLimit: String.raw`\frac{d\mathcal{L}}{d\alpha} \;=\; \lim_{h \to 0}\; \frac{\mathcal{L}(\alpha + h) - \mathcal{L}(\alpha)}{h}`,
   partialDef: String.raw`\frac{\partial \mathcal{L}}{\partial \alpha} \;=\; \lim_{h \to 0}\; \frac{\mathcal{L}(\alpha + h,\; \beta) - \mathcal{L}(\alpha,\; \beta)}{h}`,