Skip to content

feat(validation): deflated_expectancy, the number the monitor should anchor to #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 1 commit into from
Aug 3, 2026
Merged

feat(validation): deflated_expectancy, the number the monitor should anchor to #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 49 additions & 9 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,43 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Fixed
- **The opportunity-set channel switched itself off for any log without `entry_date`.**
`_trades_per_year` read `entry_date` only, so a log built from a return column and an
exit date, which is an ordinary shape, produced `trades_per_year=None` on the baseline
and `frequency_ratio=None` in every verdict. It said so in `reasons` and nothing was
wrong-but-silent, yet the effect was that the one channel covering a signal that stops
firing was dead by default for a whole class of books. It now falls back to `exit_date`
(a rate is `n / span`, and either column dates the same trades closely enough), with
`entry_date` still preferred when both are present.
### Added
- **`validation.deflated_expectancy`**, closing the gap that stood at the top of
`docs/edge_monitor.md`'s open list since the monitor shipped. `EdgeBaseline` took a
`deflated_expectancy` float and nothing in the package produced one: `deflated_sharpe`
corrects a Sharpe and returns a **probability**, which is the right output for a gate
and useless to a monitor. A monitor needs a number in R to anchor to, and without one
the only available anchor was the sample mean, which is the number the parameters were
optimized on.

The conversion rides on the bar `deflated_sharpe` already uses: `SR0`, the expected
maximum per-trade Sharpe of N noise trials, carried back into R by the winner's own
sigma and subtracted, `deflated = mu - sigma * SR0`. Both now call one
`_expected_max_sharpe`, so two corrections for one search cannot disagree about how big
the search was. `EdgeBaseline.from_log` accepts the `DeflatedExpectancy` object as well
as a float, because handing over the wrong field of a result you already computed
anchors the monitor to the pre-correction number while reporting `deflated=True`.

**It takes trial LOGS, not trial Sharpes**, unlike `deflated_sharpe`. The Sharpes are
computed inside so their clock cannot be got wrong: multiplying a per-month Sharpe by a
per-trade sigma yields a haircut in no units at all, silently, which is the v0.4.0 units
bug in a new costume.

**It is a bias correction, not a significance test**, and the docstring, the `__str__`
and the property name all say so. It removes the selection bias a search of this size is
*expected* to produce, so a pure-noise winner still clears zero roughly half the time
(measured at 56% / 47% / 44% for N = 5 / 20 / 100, against `deflated_sharpe` correctly
calling 0% of the same draws significant). The result's property is `is_positive` rather
than `survives` for exactly that reason. A correction that leaves nothing raises when it
reaches `EdgeBaseline`, whose existing refusal now names deflation as a cause.

### Changed
- **`examples/edge_monitor.py` now runs a real 64-config search** and deflates the winner,
where it previously applied a hardcoded `raw_mean * 0.8` under a comment beginning
"Pretend". Every figure in tutorial §14 moved as a result (the haircut is 26% of the raw
edge, not 20%, and the naive baseline flatters by 36%, not 25%). The example now exposes
`promoted_book()` so the tutorial's numbers have one source.

- **The CUSUM's false-alarm budget is now stated in CALENDAR TIME.** New
`Thresholds.monitor_arl0_years` (default 25), converted using the baseline's own firing
rate; `monitor_arl0_trades` stays as the fallback when the rate is unknown.
Expand Down Expand Up @@ -46,6 +72,20 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
docstring and [docs/edge_monitor.md](docs/edge_monitor.md) now carry the graded table,
and two tests pin it so the claim cannot quietly drift back.

### Fixed
- **`tests/test_edge_monitor_example.py` rebuilt the baseline it was supposed to pin**,
instead of importing it. So the "reproducibility guard" for tutorial §14 passed
unchanged while the example it guards printed entirely different numbers. It now imports
`promoted_book()`. A guard that reconstructs what it guards is not a guard.
- **The opportunity-set channel switched itself off for any log without `entry_date`.**
`_trades_per_year` read `entry_date` only, so a log built from a return column and an
exit date, which is an ordinary shape, produced `trades_per_year=None` on the baseline
and `frequency_ratio=None` in every verdict. It said so in `reasons` and nothing was
wrong-but-silent, yet the effect was that the one channel covering a signal that stops
firing was dead by default for a whole class of books. It now falls back to `exit_date`
(a rate is `n / span`, and either column dates the same trades closely enough), with
`entry_date` still preferred when both are present.

## [0.5.0] - 2026-08-02

The edge-monitor release. The gauntlet asks "is this edge real?" once, over a fixed log;
Expand Down
3 changes: 2 additions & 1 deletion docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ flowchart TD
| Module | Package? | Purpose | Public entry points |
|---|---|---|---|
| **`edge`** | `edge/` (trade_log, simulator, metrics, stats) | Produce and describe the `TradeLog`; the honesty layer (CI + p-value). | `TradeLog`, `barrier_trades`, `edge_report`, `reality_check`, `bootstrap_ci`, `block_bootstrap_pvalue`, `random_entry_null` |
| **`validation`** | `validation/` (holdout, walk_forward, permutation, pbo, search_space, gate, gauntlet, thresholds, diagnostics, monitor) | Out-of-sample survival, data-mining corrections, the audited gauntlet, and the post-promotion decay monitor. | `holdout`, `walk_forward`, `sign_permutation_pvalue`, `sidak_correction`, `spa_test`, `pbo_cscv`, `deflated_sharpe`, `SearchSpaceLog`, `run_gauntlet`, `Thresholds`, `edge_monitor`, `EdgeBaseline` |
| **`validation`** | `validation/` (holdout, walk_forward, permutation, pbo, search_space, gate, gauntlet, thresholds, diagnostics, monitor) | Out-of-sample survival, data-mining corrections, the audited gauntlet, and the post-promotion decay monitor. | `holdout`, `walk_forward`, `sign_permutation_pvalue`, `sidak_correction`, `spa_test`, `pbo_cscv`, `deflated_sharpe`, `deflated_expectancy`, `SearchSpaceLog`, `run_gauntlet`, `Thresholds`, `edge_monitor`, `EdgeBaseline` |
| **`breadth`** | `breadth.py` (single file) | How many *independent* bets a correlated set of return streams holds. | `effective_n`, `participation_ratio`, `Breadth` |
| **`ml`** | `ml/` (ic, decay, redundancy, pit) | The same honesty aimed at a model's scores — a predictions frame, not a `TradeLog`. | `information_coefficient`, `alpha_gate`, `quantile_decay`, `fold_ic`, `redundancy_droplist`, `asof_window` |
| **`report`** | `report/` (tearsheet, scorecards) | Self-contained HTML tearsheets. Plotly, behind the `[report]` extra; **not** re-exported at top level. | `tearsheet`, `gauntlet_report`, `fullrange_scorecard`, `monitor_panel` |
Expand Down Expand Up @@ -144,6 +144,7 @@ simulator), this is the output you must return.
| `EdgeReport` | `edge/metrics.py:130` | dataclass | `n, win_rate, expectancy, profit_factor, payoff_ratio, sqn` (+ excursion optionals) |
| `PBOResult` | `validation/pbo.py:68` | dataclass | `pbo, logits, oos_below_zero, degradation_slope/_r2, n_configs, n_splits, n_blocks`; `.label` ROBUST/GUARDED/OVERFIT |
| `DeflatedSharpe` | `validation/pbo.py:194` | dataclass | `observed_sharpe, deflated_sharpe, sr0_threshold, n_trials, n_obs, skew, kurtosis`; `.label` SIGNIFICANT/MARGINAL/NOT SIGNIFICANT |
| `DeflatedExpectancy` | `validation/pbo.py:324` | dataclass | `observed_expectancy, deflated_expectancy, haircut, sigma, sr0_threshold, n_trials, n_trades, n_scored`; `.is_positive`, `.retained`. A bias correction in R, not a test |
| `Breadth` | `breadth.py:22` | frozen | `n_eff, n_assets, eigenvalues, loadings, corr`; `.redundancy` = n_assets/n_eff |
| `Fold` / `WalkForwardResult` | `validation/walk_forward.py:62,75` | dataclass | fold detail; `folds, stitched, param_grid` |
| `HoldoutResult` | `validation/holdout.py:51` | dataclass | early/late `Verdict`s |
Expand Down
98 changes: 76 additions & 22 deletions docs/edge_monitor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ already produces the corrected version of exactly that figure: `deflated_sharpe`
is a **deflated** expectancy is measuring decay from a number that was defensible in
the first place.

`validation.deflated_expectancy` now does that conversion; see
[Deflating the baseline](#deflating-the-baseline) below.

Worth flagging in the reference numbers: OOS expectancy (`+0.158%`) is *above* IS
(`+0.134%`). That is backwards from the usual optimization bias. Either the search was
narrow (small honest N), the in-sample window was hostile, or the out-of-sample period
Expand Down Expand Up @@ -160,10 +163,13 @@ crucible emits HOLDING / SLIPPING / DEGRADED. It does not emit "cut to half size
### What shipped

```python
from crucible.validation import EdgeBaseline, cusum_design, edge_monitor, empirical_arl
from crucible.validation import (
EdgeBaseline, cusum_design, deflated_expectancy, edge_monitor, empirical_arl,
)

# ONCE, at promotion. Freeze the result.
base = EdgeBaseline.from_log(validated_log, deflated_expectancy=0.08, n_variants=64)
corrected = deflated_expectancy(validated_log.r, [t.r for t in trials], n_trials=64)
base = EdgeBaseline.from_log(validated_log, deflated_expectancy=corrected, n_variants=64)

design = cusum_design(base) # k and h derived from Thresholds, not typed in
verdict = edge_monitor(live_log, base)
Expand Down Expand Up @@ -246,12 +252,71 @@ so is not.
The rolling ratio and the firing-rate ratio cap out at `SLIPPING`. This is the part
worth keeping if nothing else here survives review, and
[`examples/edge_monitor.py`](https://github.com/mspinola/crucible/blob/main/examples/edge_monitor.py)
shows why. On a book whose true edge is fully intact and a quarter **above** baseline,
the 200-trade trailing read ranges from -31% to 240% of baseline on noise alone and dips
under the 50% line in 9% of windows, while the CUSUM peaks at 53% of its threshold and
shows why. On a book whose true edge is fully intact and **above** baseline,
the 200-trade trailing read ranges from -25% to 197% of baseline on noise alone and dips
under the 50% line in 11% of windows, while the CUSUM peaks at 82% of its threshold and
never fires. A "cut at 50% of baseline" rule would have cut a healthy book on whichever
window you happened to read.

## Deflating the baseline

Defect 1 was the largest gap on this page for as long as it stood: the argument for
building the monitor here rather than copying the reference implementation rested on
anchoring to a search-corrected number, and nothing in the package produced one.
`deflated_sharpe` corrects a Sharpe and returns a **probability**, which is the right
output for a gate and useless to a monitor. A monitor needs a number in R.

`validation.deflated_expectancy` writes the conversion, on the bar `deflated_sharpe`
already uses:

```
SR0 = expected MAXIMUM per-trade Sharpe of N noise trials
(Bailey/López de Prado, scaled by the spread of the trial Sharpes)
deflated = mu - sigma * SR0
```

The bar lives in Sharpe units, so it is carried back into R by the winner's own sigma
before being subtracted. Both functions now call one `_expected_max_sharpe`, so the two
corrections for one search cannot disagree about how big the search was.

```python
from crucible.validation import deflated_expectancy, EdgeBaseline

d = deflated_expectancy(winner.r, [t.r for t in every_variant_tried], n_trials=log)
base = EdgeBaseline.from_log(winner, deflated_expectancy=d, n_variants=log.n_variants)
```

Three decisions worth recording.

**It takes trial LOGS, not trial Sharpes**, unlike `deflated_sharpe`. The Sharpes are
computed inside, so their clock cannot be got wrong. A per-month Sharpe and a per-trade
Sharpe are different numbers on different scales, and multiplying the wrong one by a
per-trade sigma produces a haircut in no units at all, silently. That is the v0.4.0 units
bug in a new costume, and the fix is to not accept the ambiguous input.

**It is a bias correction, not a significance test, and the docstring says so in those
words.** It removes the selection bias a search of this size is *expected* to produce.
The realized maximum sits above its own mean about half the time, so a pure-noise winner
still clears zero here roughly as often as not: measured at 56% / 47% / 44% for N = 5 /
20 / 100, while `deflated_sharpe` correctly calls 0% of the same draws significant
(reproducer:
`tests/test_deflated_expectancy.py::test_the_haircut_is_a_bias_correction_not_a_test`).
The result object's property is therefore named `is_positive` rather than `survives`,
because the first draft called it `survives` and that reads as a verdict it does not
deliver. Establish the edge is real with the gauntlet; use this to decide what to anchor
to afterwards.

**It over-corrects a genuine edge, deliberately.** A winner chosen partly for real signal
carries less selection bias than the pure-luck maximum being subtracted, so the deflated
number sits below the truth. For a monitor baseline that is the safer direction: too low
a bar makes the monitor slow to call decay, too high a bar makes it cry wolf, and a
spurious alarm forces a re-optimization that taxes the honest N of the next verdict.

A correction that leaves nothing raises rather than returning a smaller baseline.
`EdgeBaseline` already refused a non-positive expectancy; the message now names deflation
as a cause, because a book whose edge does not survive its own search is not a monitoring
problem.

## Settled

These were the open questions this page carried before #109. Merging answered them, so
Expand All @@ -268,25 +333,14 @@ they are recorded here with what decided them rather than left looking live.

Ordered by how much each one undercuts the argument for the module.

1. **Nothing deflates an expectancy.** `deflated_expectancy` is still a number the
caller supplies. `deflated_sharpe` corrects a Sharpe ratio, not a per-trade mean, so
the conversion is unwritten. This is the largest gap: anchoring to a search-corrected
baseline was the whole reason to build this here rather than copy the reference
implementation, and until it exists that advantage is a docstring rather than a
feature.
2. **Nothing calls it.** The monitor needs a caller to freeze an `EdgeBaseline` at the
moment of promotion and hold it. In this stack that is `crucible_stack.orchestrate`
(which owns the `DeploymentLedger` and the promotion event) or `livebook`. Neither
knows the module exists, so today it runs only in its own tests. Sibling-repo work,
not crucible's.
3. **The firing-rate channel is uncalibrated.** It compares a ratio to a threshold, so
1. **The firing-rate channel is uncalibrated.** It compares a ratio to a threshold, so
it can only ever say `SLIPPING`. Trade arrivals are approximately Poisson, so an
arrival-process test would give it a stated false-alarm rate and let it stand beside
the CUSUM.
4. **It has never met real decay.** Only synthetic decay, generated to test it. The ARL
2. **It has never met real decay.** Only synthetic decay, generated to test it. The ARL
figures are design targets, not field results. The first honest test is the first
promoted book that genuinely degrades.
5. **The README still does not mention the module,** though it enumerates every other
3. **The README still does not mention the module,** though it enumerates every other
subpackage's API with a worked block. The worked example, tutorial §14 and the
`report.monitor_panel` block all exist now; the README is the last documentation gap.

Expand All @@ -301,11 +355,11 @@ amount of code.
The improvement over the reference version is not the detector. It is what the detector
is anchored to: a search-corrected expectancy instead of the optimized in-sample one,
denominated in R instead of percent of account, watched alongside the opportunity set
rather than in isolation. Two of those three are built. The first is not, which is why
it sits at the top of [Still open](#still-open).
rather than in isolation. All three are now built, the first of them last and only after
this page had carried it as the top open item for several revisions.

The 2020 dip in the reference output is the uncalibrated alarm firing while the
calibrated one stayed silent. That is not evidence the monitor works, and the same
pattern reproduces here: in §14 of the tutorial, a book whose edge never decayed shows a
trailing read swinging between -31% and 240% of baseline. Hence the rule that only a
trailing read swinging between -25% and 197% of baseline. Hence the rule that only a
detector with a stated false-alarm rate may escalate to `DEGRADED`.
Loading
Loading