feat(track-w): EWC hardening — honest rehearsal vs EWC - #46
Merged
Conversation
- zero_grad() after Fisher accumulation loop (residual grads hygiene) - NOTE comment on nll_loss(reduction='mean') squared-mean convention - n_classes assumption documented for HardSplitTask dense batches
Fix estimate_fisher to infer n_classes from all batches, preventing IndexError at seeds where max class absent from batch 0.
Add run_w4_ewc_sweep() helper (lams grid, n=5 seeds). Fix test_ewc_beats_none_baseline: judge on multi-seed mean at best-lam instead of fragile seed=0 / lam=1.0. Best lam=1000, ewc=0.532 < none=0.767 — gate passes. Trace: docs/superpowers/research/2026-05-30-w4-ewc-comparison.json
Add EWC-1..EWC-4 to run_audit() using real T5 JSON keys. 34 OK, 0 DIVERGENT under --ci.
E702 semicolons split across 3 files, E741 ambiguous lambda var renamed lam_val, E501 factcheck line wrapped, N812 noqa added in ewc/rehearsal/pilot, I001+F401 auto-fixed. Spec mx.array -> Tensor.
There was a problem hiding this comment.
Pull request overview
This PR hardens Track-W’s continual-learning evaluation by introducing a genuinely hard shared-head sequential task (HardSplitTask) and adding an EWC implementation, then providing a unified runner + integration tests to compare none / rehearsal / ewc with traceability via research JSON + factcheck audit hooks.
Changes:
- Add
HardSplitTask(two sequentialHardFlowProxyTasksubtasks sharing a 12-class head) to induce catastrophic forgetting without mitigation. - Add continual-learning utilities: diagonal-Fisher EWC (
estimate_fisher,penalty) and a reusableRehearsalBuffer. - Add
run_w4_compare()+run_w4_ewc_sweep()plus unit/integration tests and research/factcheck documentation artifacts.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| track_w/tasks/hard_split.py | New hard shared-head sequential task wrapper used for W4 comparisons. |
| track_w/continual/init.py | Introduces the Track-W continual-learning subpackage. |
| track_w/continual/ewc.py | Adds EWC Fisher estimation + quadratic penalty for W4 comparisons. |
| track_w/continual/rehearsal.py | Extracts rehearsal batch-mixing into a reusable buffer helper. |
| scripts/track_w_pilot.py | Adds unified run_w4_compare() scaffold and run_w4_ewc_sweep(). |
| scripts/factcheck_audit.py | Adds audit hooks for the new EWC/HardSplitTask claims. |
| tests/unit/track_w/test_hard_split_task.py | Unit tests validating HardSplitTask shapes/label space/non-identity. |
| tests/unit/track_w/test_ewc.py | Unit tests validating Fisher keys/non-negativity and penalty behavior. |
| tests/unit/track_w/test_run_w4_compare_smoke.py | Smoke tests for run_w4_compare() result shape/labels. |
| tests/integration/track_w/test_w4_hard_split_baseline.py | Integration baseline asserting catastrophic forgetting without mitigation. |
| tests/integration/track_w/test_w4_method_comparison.py | Multi-seed slow integration comparing methods on mean forgetting. |
| tests/integration/track_w/test_gate_w4_ewc.py | Integration gate sweeping λ and asserting EWC beats none at best λ (or skips). |
| docs/superpowers/specs/2026-05-30-track-w-hardening-ewc-design.md | Spec for the hardening design and evaluation protocol. |
| docs/superpowers/plans/2026-05-30-track-w-hardening-ewc.md | Implementation plan and checklist for the hardening work. |
| docs/superpowers/research/2026-05-30-w4-ewc-comparison.json | Captures reported multi-seed results and sweep summary for traceability. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ------- | ||
| Scalar tensor, differentiable w.r.t. wml.parameters(). | ||
| """ | ||
| pen = torch.tensor(0.0) |
Comment on lines
+52
to
+71
| wml.eval() | ||
| n_batches = len(data_loader) | ||
| for x, y in data_loader: | ||
| logits = wml.emit_head_pi(wml.core(x))[:, : int(n_classes)] | ||
| log_probs = F.log_softmax(logits, dim=-1) | ||
| # Use true label log-likelihood for a tighter diagonal estimate. | ||
| # NOTE: nll_loss(reduction='mean') gives a batch-averaged gradient, so this is | ||
| # the squared mean, not the mean of squares — the standard empirical-Fisher (EWC) | ||
| # convention. lam absorbs the scale factor. | ||
| nll = F.nll_loss(log_probs, y) | ||
| wml.zero_grad() | ||
| nll.backward() | ||
| for name, p in wml.named_parameters(): | ||
| if p.grad is not None: | ||
| fisher[name] += p.grad.detach() ** 2 | ||
| for name in fisher: | ||
| fisher[name] /= n_batches | ||
| wml.zero_grad() # clean residual grads from the last backward pass | ||
| wml.train() | ||
| return fisher |
Comment on lines
+63
to
+65
| loss_new = _loss(task1, n_new) | ||
| loss_old = _loss(self._task0, n_old) | ||
| return (loss_new * n_new + loss_old * n_old) / self.total_batch |
Comment on lines
+9
to
+13
| """Prove the task is genuinely hard: forgetting >= 0.50 without mitigation. | ||
|
|
||
| Threshold 0.50 traces to docs/superpowers/research/2026-05-30-w4-ewc-comparison.json | ||
| key "baseline_none_forgetting_threshold". | ||
| """ |
Comment on lines
+15
to
+19
| """n=5 seeds: rehearsal and ewc both strictly beat none on mean forgetting. | ||
|
|
||
| Results traced to docs/superpowers/research/2026-05-30-w4-ewc-comparison.json | ||
| keys "multi_seed_none_mean", "multi_seed_rehearsal_mean", "multi_seed_ewc_mean". | ||
| """ |
Comment on lines
+52
to
+71
| wml.eval() | ||
| n_batches = len(data_loader) | ||
| for x, y in data_loader: | ||
| logits = wml.emit_head_pi(wml.core(x))[:, : int(n_classes)] | ||
| log_probs = F.log_softmax(logits, dim=-1) | ||
| # Use true label log-likelihood for a tighter diagonal estimate. | ||
| # NOTE: nll_loss(reduction='mean') gives a batch-averaged gradient, so this is | ||
| # the squared mean, not the mean of squares — the standard empirical-Fisher (EWC) | ||
| # convention. lam absorbs the scale factor. | ||
| nll = F.nll_loss(log_probs, y) | ||
| wml.zero_grad() | ||
| nll.backward() | ||
| for name, p in wml.named_parameters(): | ||
| if p.grad is not None: | ||
| fisher[name] += p.grad.detach() ** 2 | ||
| for name in fisher: | ||
| fisher[name] /= n_batches | ||
| wml.zero_grad() # clean residual grads from the last backward pass | ||
| wml.train() | ||
| return fisher |
Comment on lines
+63
to
+65
| loss_new = _loss(task1, n_new) | ||
| loss_old = _loss(self._task0, n_old) | ||
| return (loss_new * n_new + loss_old * n_old) / self.total_batch |
Comment on lines
+9
to
+13
| """Prove the task is genuinely hard: forgetting >= 0.50 without mitigation. | ||
|
|
||
| Threshold 0.50 traces to docs/superpowers/research/2026-05-30-w4-ewc-comparison.json | ||
| key "baseline_none_forgetting_threshold". | ||
| """ |
Comment on lines
+15
to
+19
| """n=5 seeds: rehearsal and ewc both strictly beat none on mean forgetting. | ||
|
|
||
| Results traced to docs/superpowers/research/2026-05-30-w4-ewc-comparison.json | ||
| keys "multi_seed_none_mean", "multi_seed_rehearsal_mean", "multi_seed_ewc_mean". | ||
| """ |
| ------- | ||
| Scalar tensor, differentiable w.r.t. wml.parameters(). | ||
| """ | ||
| pen = torch.tensor(0.0) |
electron-rare
added a commit
that referenced
this pull request
May 30, 2026
CLAUDE.md: track_w/ line updated — 6 substrates implemented + continual/ sub-package (ewc.py + rehearsal.py, PR #46). README.md: new Track-W hardening section — rehearsal 0.034 vs EWC@λ=1000 0.532 vs none 0.767 on HardSplitTask, honest framing, trace to w4-ewc-comparison.json, API summary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
But
Durcir gate-W (déjà passée) en passant d'une seule mitigation continual-learning (rehearsal, sur tâche triviale) à une comparaison honnête de deux méthodes sur une tâche dure à tête de sortie partagée.
Contenu
track_w/tasks/hard_split.py—HardSplitTask: 2 sous-tâches XOR-dures à tête 12-class partagée (oubli catastrophique sans mitigation, ≥0.50).track_w/continual/ewc.py— EWC (Fisher diagonale + pénalité quadratique), respecte W-1/W-2 (codebook inclus).track_w/continual/rehearsal.py—RehearsalBufferréutilisable (refactor,run_w4_rehearsalbit-stable).scripts/track_w_pilot.py—run_w4_compare(method)+run_w4_ewc_sweep(λ).Finding scientifique (honnête, n=5)
rehearsal ≫ EWC ≫ none. EWC bat none de façon robuste (Δ+0.235) mais reste loin de rehearsal sur cette géométrie XOR — rapporté tel quel, aucun gate truqué.
Vérification
gate-w-passedOK (52 passed).Spec :
docs/superpowers/specs/2026-05-30-track-w-hardening-ewc-design.md· Plan :docs/superpowers/plans/2026-05-30-track-w-hardening-ewc.md🤖 Generated with Claude Code