feat(validation): deflated_expectancy, the number the monitor should anchor to - #116
Conversation
…anchor to
docs/edge_monitor.md carried this at the top of its open list since the monitor
shipped: EdgeBaseline took a deflated_expectancy float and nothing in the package
produced one. deflated_sharpe corrects a Sharpe and returns a PROBABILITY, which is
the right output for a gate and useless to a monitor. A monitor needs a number in R,
and without one the only anchor available was the sample mean, which is the number
the parameters were optimized on.
The conversion rides on the bar deflated_sharpe already uses. SR0 is the expected
maximum per-trade Sharpe of N noise trials; carry it back into R by the winner's own
sigma and subtract:
deflated = mu - sigma * SR0
Both functions now call one _expected_max_sharpe, so two corrections for one search
cannot disagree about how big the search was.
Three choices worth recording:
* It takes trial LOGS, not trial Sharpes, unlike deflated_sharpe. The Sharpes are
computed inside so their clock cannot be got wrong: multiplying a per-month
Sharpe by a per-trade sigma gives a haircut in no units at all, silently, which
is the v0.4.0 units bug in a new costume.
* It is a bias correction, NOT a significance test, and the docstring, the __str__
and the property name all say so. It removes the selection bias a search of this
size is EXPECTED to produce, so a pure-noise winner still clears zero roughly
half the time: measured 56% / 47% / 44% for N = 5 / 20 / 100, against
deflated_sharpe correctly calling 0% of the same draws significant. I named the
property `survives` first, which reads as a verdict it does not deliver; it is
`is_positive`, and the measurement that caught this is pinned as its reproducer.
* EdgeBaseline.from_log accepts the result OBJECT as well as a float, because
handing over the wrong field of a result you already computed anchors the monitor
to the pre-correction number while reporting deflated=True.
Two findings from wiring the example to the real function rather than a stand-in:
* The worked example faked this step with `raw_mean * 0.8` under a comment starting
"Pretend". With a real 64-config search the haircut is 26% of the raw edge rather
than 20%, and the naive baseline flatters by 36% rather than 25%. Every figure in
tutorial §14 moved and is re-pinned.
* tests/test_edge_monitor_example.py REBUILT the baseline it exists to pin instead
of importing it, so all five assertions passed unchanged while the example they
guard printed entirely different numbers. It now imports promoted_book(). A guard
that reconstructs what it guards is not a guard.
The payoff is now a test rather than a claim: run one untouched, fully healthy live
book against both baselines and the naive one returns DEGRADED where the deflated one
returns HOLDING. An inflated reference does not merely mis-scale the ratios, it
manufactures alarms on books that never decayed.
373 tests pass, ruff clean, mkdocs --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Rebased onto #115, which landed while this was open. The two overlap more than the titles suggest and both changes are kept: #115 moved the ARL budget to calendar years, so every figure in the worked example and tutorial §14 moved again on top of the ones this PR already moved. Recomputed with both applied and re-pinned: #115 also fixed the dead firing-rate channel from crucible's end, by falling back to The headline result is unchanged by the rebase: run one untouched, fully healthy live book against both baselines and the naive one still returns DEGRADED where the deflated one returns HOLDING. 378 tests pass, ruff clean, |
15413f3 to
19814cb
Compare
Bumps pyproject and crucible.__version__ to 0.6.0 and moves [Unreleased] into a dated [0.6.0] section. The second release today, deliberate rather than sloppy. 0.5.0 shipped the edge monitor; running it against a real 47-market book rather than the synthetic fixtures it was built on immediately exposed three calibration problems and confirmed the one gap already known. Leaving those in a published version for weeks is worse than two tags in one day. In order of how much it matters to anyone already on 0.5.0: 1. deflated_expectancy exists (#116). 0.5.0's EdgeBaseline accepted a deflated number and nothing in the package produced one, so every real baseline was the raw in-sample mean and the monitor's central argument was a docstring. 2. The false-alarm budget is denominated in years, not trades (#115). Any design built under 0.5.0 against a baseline that knows its firing rate recalibrates here, and for a slow book substantially: the old default gave a 23-trades/yr book a nominal detection latency of 24 years. 3. The opportunity-set channel works for logs without entry_date (#115), where 0.5.0 silently switched it off for an ordinary class of books. Minor rather than patch on the merits: 2 changes behaviour and 1 adds public API. 378 tests pass, ruff clean, mkdocs --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/edge_monitor.mdcarried this at the top of its open list since the monitor shipped:EdgeBaselinetook adeflated_expectancyfloat and nothing in the package produced one.deflated_sharpecorrects a Sharpe and returns a probability, which is the right output for a gate and useless to a monitor. A monitor needs a number in R, and without one the only anchor available was the sample mean, which is the number the parameters were optimized on.The conversion
It rides on the bar
deflated_sharpealready uses.SR0is the expected maximum per-trade Sharpe of N noise trials; carry it back into R by the winner's own sigma and subtract:Both functions now call one
_expected_max_sharpe, so two corrections for one search cannot disagree about how big the search was.Three choices worth recording
deflated_sharpe. The Sharpes are computed inside so their clock cannot be got wrong: multiplying a per-month Sharpe by a per-trade sigma gives a haircut in no units at all, silently, which is the v0.4.0 units bug in a new costume.__str__and the property name all say so. It removes the selection bias a search of this size is expected to produce, so a pure-noise winner still clears zero roughly half the time: measured 56% / 47% / 44% for N = 5 / 20 / 100, againstdeflated_sharpecorrectly calling 0% of the same draws significant. I named the propertysurvivesfirst, which reads as a verdict it does not deliver; it isis_positive, and the measurement that caught this is pinned as its reproducer.EdgeBaseline.from_logaccepts the result object as well as a float, because handing over the wrong field of a result you already computed anchors the monitor to the pre-correction number while reportingdeflated=True.Two findings from wiring the example to the real function
raw_mean * 0.8under a comment beginning "Pretend". With a real 64-config search the haircut is 26% of the raw edge rather than 20%, and the naive baseline flatters by 36% rather than 25%. Every figure in tutorial §14 moved and is re-pinned.tests/test_edge_monitor_example.pyrebuilt the baseline it exists to pin instead of importing it, so all five assertions passed unchanged while the example they guard printed entirely different numbers. It now importspromoted_book(). A guard that reconstructs what it guards is not a guard.The payoff, as a test rather than a claim
Run one untouched, fully healthy live book against both baselines: the naive one returns DEGRADED where the deflated one returns HOLDING. An inflated reference does not merely mis-scale the ratios, it manufactures alarms on books that never decayed.
373 tests pass, ruff clean,
mkdocs build --strictclean.🤖 Generated with Claude Code