docs(monitor): the firing-rate channel cannot be calibrated, and here is why - #119
Merged
Merged
Conversation
… is why
"An arrival-process test would give it a stated false-alarm rate" sat at the top of
docs/edge_monitor.md's open list from the day the monitor shipped, resting on the
claim that trade arrivals are approximately Poisson. Building it showed the claim is
false. It is removed rather than softened, and the measurements are recorded so
nobody builds it twice.
The prize was real and is worth stating: a count-based detector spots a halved
firing rate in 0.6 years against 5.1 for the expectancy CUSUM, about eight times
faster, on the failure the other two channels cannot see.
Arrivals are overdispersed, index 3.43 against Poisson's 1.0. Roughly 40% is a
secular trend (+0.49 trades/yr per year, r=+0.63); the rest is sub-annual
clustering, with lag-1 autocorrelation of detrended annual counts at +0.04.
Three detectors, each designed for a 10-year budget, measured against the book's
own arrivals:
exponential CUSUM on gaps, full span 3.6 yr 0.36x
... 7-year window 3.5 yr 0.35x
... 5-year window 1.8 yr 0.18x
Poisson CUSUM on monthly counts 3.2 yr 0.32x
All fire 3x to 5x more often than stated, and in the dangerous direction: the
expectancy CUSUM's 1.39x skew inflation buys margin, this spends it.
Two hypotheses of mine failed along the way and both are recorded rather than
quietly dropped:
* "A recent window restores the Poisson model." Annual-count dispersion does fall
to 0.68 at 7 years, but a CUSUM runs on individual gaps, not annual counts, and
those stay non-exponential at every window. Coarse and fine scale came apart,
and the 5-year window is the worst of the four.
* "Then calibrate h empirically." That is the honest fallback and it fixes the
centre: h moves 4.26 -> 6.22. But bootstrapping the calibration, h ranges
4.15..8.18 and the DELIVERED budget ranges 5.3..77.9 years. A stated 10-year
rate that is truly somewhere in 5..78 is not a stated rate.
The root cause is data, not model. 34 trades/yr gives 82 monthly periods in a
7-year window, and run length depends on a tail 82 samples cannot pin down. The gap
detector is worse: crossing its boundary needs two or three of the largest gaps back
to back, so it is a rare-combination test over the tail of a 233-gap sample, and its
ARL curve is visibly stepped.
So the channel keeps its ratio rule and its SLIPPING cap. An uncalibrated rule that
says so beats a calibrated-looking one that is wrong by 3x, and manufacturing a
stated rate to satisfy "only calibrated detectors may escalate" would defeat the
rule it was meant to satisfy.
tests/test_frequency_calibration.py pins the general claims on synthetic data (80
periods leave an 8.6x spread in the delivered budget, 2,000 leave 1.3x) so the limit
is re-derivable without the private book, and asserts the DEGRADED path still runs
only through the calibrated CUSUM.
390 tests pass, ruff clean, mkdocs --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
"An arrival-process test would give it a stated false-alarm rate" sat at the top of
docs/edge_monitor.md's open list from the day the monitor shipped, resting on the claim that trade arrivals are approximately Poisson. Building it showed the claim is false. It is removed rather than softened, and the measurements are recorded so nobody builds it twice.The prize was real, and worth stating: a count-based detector spots a halved firing rate in 0.6 years against 5.1 for the expectancy CUSUM — about eight times faster, on the failure the other two channels cannot see.
Arrivals are not Poisson
Dispersion index 3.43 against Poisson's 1.0. Roughly 40% is a secular trend (+0.49 trades/yr per year, r=+0.63); the rest is sub-annual clustering, with lag-1 autocorrelation of detrended annual counts at +0.04.
Three detectors, each designed for a 10-year budget, measured against the book's own arrivals:
All fire 3x to 5x more often than stated, and in the dangerous direction: the expectancy CUSUM's 1.39x skew inflation buys margin, this spends it.
Two of my hypotheses failed, and both are recorded
hempirically." The honest fallback, and it fixes the centre:hmoves 4.26 → 6.22. But bootstrapping the calibration,hranges 4.15–8.18 and the delivered budget ranges 5.3 to 77.9 years. A stated 10-year rate that is truly somewhere in 5–78 is not a stated rate.Root cause is data, not model
34 trades/yr gives 82 monthly periods in a 7-year window, and run length depends on a tail that 82 samples cannot pin down. The gap detector is worse: crossing its boundary needs two or three of the largest gaps back to back, so it is a rare-combination test over the tail of a 233-gap sample, and its ARL curve is visibly stepped as a result.
So the channel keeps its ratio rule and its
SLIPPINGcap. An uncalibrated rule that says so beats a calibrated-looking one wrong by 3x, and manufacturing a stated rate to satisfy "only calibrated detectors may escalate" would defeat the rule it was meant to satisfy.Reproducer
tests/test_frequency_calibration.pypins the general claims synthetically — 80 periods leave an 8.6x spread in the delivered budget where 2,000 leave 1.3x — so the limit is re-derivable without the private book. It also asserts theDEGRADEDpath still runs only through the calibrated CUSUM, and points a future reader at the measurements if they change it.390 tests pass, ruff clean,
mkdocs build --strictclean.🤖 Generated with Claude Code