This repository contains a small set of controlled experiments studying event-driven parameter anchoring as a deployment-time stability mechanism for neural networks.
The goal is not to improve training or replace EMA.
The goal is to understand when resisting adaptation is preferable to continuous smoothing under uncertain or delayed feedback.
In production, many ML systems operate in a post-convergence regime:
- Models are already trained
- Updates are small and corrective
- Feedback may be delayed, partial, or noisy
- False updates can be more damaging than slow adaptation
Standard stabilization methods such as Exponential Moving Average (EMA) continuously smooth parameters, regardless of whether incoming gradients reflect real concept change or noise.
This work explores an alternative:
Anchor parameters by default, and relax the anchor only when instability is detected.
All experiments compare three regimes under identical compute and models:
-
Plain SGD
Fast adaptation, unstable under noise -
EMA (Polyak averaging)
Continuous smoothing, robust but lagging -
Event-Driven Anchoring (Event-GMO)
- Fixed anchor by default
- Anchor relaxes only when loss variance exceeds a threshold
- Event-triggered, not continuously smoothed
Event-Driven Anchoring is not EMA and does not update unless an instability event is observed.
ema_vs_gmo_post_convergence.py
Small corrective updates after convergence.
Result:
Event-Driven Anchoring reduces variance relative to Plain and is comparable to EMA.
ema_vs_gmo_concept_shift.py
Decision boundary genuinely changes.
Expected and observed behavior:
- EMA adapts better
- Event-Driven Anchoring adapts slower
- Fixed anchoring fails
This failure is intentional and necessary.
A stabilizer that never fails is not trustworthy.
ema_vs_gmo_event_anchor.py
Anchor relaxes only after sustained instability.
Result:
Event-Driven Anchoring:
- Adapts faster than EMA
- Is more stable than Plain SGD
- Does not collapse into EMA behavior
Stability should be event-triggered, not continuously smoothed, when feedback is uncertain.
Event-Driven Anchoring is useful only in narrow deployment regimes:
- delayed feedback
- noisy updates
- high cost of false adaptation
It is not a training algorithm and not a universal optimizer.
- Not intended to replace EMA
- Not suitable for rapid or known concept shifts
- No claims about generalization or optimality
This is an exploratory systems note with documented success and failure modes.
MIT