Skip to content

docs(monitor): the firing-rate channel cannot be calibrated, and here is why - #119

Merged
mspinola merged 1 commit into
mainfrom
claude/frequency-calibration-finding
Aug 3, 2026
Merged

docs(monitor): the firing-rate channel cannot be calibrated, and here is why#119
mspinola merged 1 commit into
mainfrom
claude/frequency-calibration-finding

Conversation

@mspinola

@mspinola mspinola commented Aug 3, 2026

Copy link
Copy Markdown
Owner

"An arrival-process test would give it a stated false-alarm rate" sat at the top of docs/edge_monitor.md's open list from the day the monitor shipped, resting on the claim that trade arrivals are approximately Poisson. Building it showed the claim is false. It is removed rather than softened, and the measurements are recorded so nobody builds it twice.

The prize was real, and worth stating: a count-based detector spots a halved firing rate in 0.6 years against 5.1 for the expectancy CUSUM — about eight times faster, on the failure the other two channels cannot see.

Arrivals are not Poisson

Dispersion index 3.43 against Poisson's 1.0. Roughly 40% is a secular trend (+0.49 trades/yr per year, r=+0.63); the rest is sub-annual clustering, with lag-1 autocorrelation of detrended annual counts at +0.04.

Three detectors, each designed for a 10-year budget, measured against the book's own arrivals:

Detector Delivered vs stated
exponential CUSUM on gaps, full span 3.6 yr 0.36x
...7-year window 3.5 yr 0.35x
...5-year window 1.8 yr 0.18x
Poisson CUSUM on monthly counts 3.2 yr 0.32x

All fire 3x to 5x more often than stated, and in the dangerous direction: the expectancy CUSUM's 1.39x skew inflation buys margin, this spends it.

Two of my hypotheses failed, and both are recorded

  • "A recent window restores the Poisson model." Annual-count dispersion does fall to 0.68 at 7 years, but a CUSUM runs on individual gaps, not annual counts, and those stay non-exponential at every window. Coarse and fine scale came apart, and the 5-year window is the worst of the four.
  • "Then calibrate h empirically." The honest fallback, and it fixes the centre: h moves 4.26 → 6.22. But bootstrapping the calibration, h ranges 4.15–8.18 and the delivered budget ranges 5.3 to 77.9 years. A stated 10-year rate that is truly somewhere in 5–78 is not a stated rate.

Root cause is data, not model

34 trades/yr gives 82 monthly periods in a 7-year window, and run length depends on a tail that 82 samples cannot pin down. The gap detector is worse: crossing its boundary needs two or three of the largest gaps back to back, so it is a rare-combination test over the tail of a 233-gap sample, and its ARL curve is visibly stepped as a result.

So the channel keeps its ratio rule and its SLIPPING cap. An uncalibrated rule that says so beats a calibrated-looking one wrong by 3x, and manufacturing a stated rate to satisfy "only calibrated detectors may escalate" would defeat the rule it was meant to satisfy.

Reproducer

tests/test_frequency_calibration.py pins the general claims synthetically — 80 periods leave an 8.6x spread in the delivered budget where 2,000 leave 1.3x — so the limit is re-derivable without the private book. It also asserts the DEGRADED path still runs only through the calibrated CUSUM, and points a future reader at the measurements if they change it.

390 tests pass, ruff clean, mkdocs build --strict clean.

🤖 Generated with Claude Code

… is why

"An arrival-process test would give it a stated false-alarm rate" sat at the top of
docs/edge_monitor.md's open list from the day the monitor shipped, resting on the
claim that trade arrivals are approximately Poisson. Building it showed the claim is
false. It is removed rather than softened, and the measurements are recorded so
nobody builds it twice.

The prize was real and is worth stating: a count-based detector spots a halved
firing rate in 0.6 years against 5.1 for the expectancy CUSUM, about eight times
faster, on the failure the other two channels cannot see.

Arrivals are overdispersed, index 3.43 against Poisson's 1.0. Roughly 40% is a
secular trend (+0.49 trades/yr per year, r=+0.63); the rest is sub-annual
clustering, with lag-1 autocorrelation of detrended annual counts at +0.04.

Three detectors, each designed for a 10-year budget, measured against the book's
own arrivals:

  exponential CUSUM on gaps, full span   3.6 yr   0.36x
  ... 7-year window                      3.5 yr   0.35x
  ... 5-year window                      1.8 yr   0.18x
  Poisson CUSUM on monthly counts        3.2 yr   0.32x

All fire 3x to 5x more often than stated, and in the dangerous direction: the
expectancy CUSUM's 1.39x skew inflation buys margin, this spends it.

Two hypotheses of mine failed along the way and both are recorded rather than
quietly dropped:

  * "A recent window restores the Poisson model." Annual-count dispersion does fall
    to 0.68 at 7 years, but a CUSUM runs on individual gaps, not annual counts, and
    those stay non-exponential at every window. Coarse and fine scale came apart,
    and the 5-year window is the worst of the four.
  * "Then calibrate h empirically." That is the honest fallback and it fixes the
    centre: h moves 4.26 -> 6.22. But bootstrapping the calibration, h ranges
    4.15..8.18 and the DELIVERED budget ranges 5.3..77.9 years. A stated 10-year
    rate that is truly somewhere in 5..78 is not a stated rate.

The root cause is data, not model. 34 trades/yr gives 82 monthly periods in a
7-year window, and run length depends on a tail 82 samples cannot pin down. The gap
detector is worse: crossing its boundary needs two or three of the largest gaps back
to back, so it is a rare-combination test over the tail of a 233-gap sample, and its
ARL curve is visibly stepped.

So the channel keeps its ratio rule and its SLIPPING cap. An uncalibrated rule that
says so beats a calibrated-looking one that is wrong by 3x, and manufacturing a
stated rate to satisfy "only calibrated detectors may escalate" would defeat the
rule it was meant to satisfy.

tests/test_frequency_calibration.py pins the general claims on synthetic data (80
periods leave an 8.6x spread in the delivered budget, 2,000 leave 1.3x) so the limit
is re-derivable without the private book, and asserts the DEGRADED path still runs
only through the calibrated CUSUM.

390 tests pass, ruff clean, mkdocs --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mspinola
mspinola merged commit 5f443df into main Aug 3, 2026
3 checks passed
@mspinola
mspinola deleted the claude/frequency-calibration-finding branch August 3, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant