Skip to content

Fix/dual noise units and compose cholesky - #16

Open
0xgny wants to merge 2 commits into
hongjin-he:mainfrom
0xgny:fix/dual-noise-units-and-compose-cholesky
Open

Fix/dual noise units and compose cholesky#16
0xgny wants to merge 2 commits into
hongjin-he:mainfrom
0xgny:fix/dual-noise-units-and-compose-cholesky

Conversation

@0xgny

@0xgny 0xgny commented Aug 7, 2026

Copy link
Copy Markdown

fix for issue:#15

What problem this addresses

Two dimensional-analysis bugs in the pure-numpy core. Both were invisible to
the test suite: the only assertion on λ_η was >= 0, which a dimensionally
inverted quantity satisfies.

1. state/noise.py — λ_η was a count, not a rate.
calibrate() computed lambda_eta = N_jumps * dt, the reciprocal of an
intensity. σ_τ had the matching problem: sqrt(BPV/n) is per-BAR volatility.

2. events/operators.pycompose() squared the Cholesky factor wrongly.
Sigma_w is documented as a Cholesky factor and read as S @ S.T everywhere
else, but compose() used S @ S. For systemic_crisis_operator — the one
constructor with off-diagonal noise — S @ S is not PSD, so np.linalg.cholesky
raised LinAlgError. A systemic crisis could not be composed with any other
event at all.

Connection to the theoretical framework

Theorem III.1 ADDS σ_τ² and λ_η·m₂^η, so both must share a time unit. They
didn't, which made the bound depend on your bar size rather than on the market.
Now: T = n·dt, σ_τ² = BPV/T, λ_η = N_jumps/T.

Fix 2 restores the composition law in §IV: Cov = L₁L₁ᵀ + A₁(L₂L₂ᵀ)A₁ᵀ, which
is what makes the monoid/groupoid claim in the module docstring hold.

Results

Ground truth: 3 jumps/day, 2% daily vol, 10 days, two sampling rates.

true before (5-min) before (1-min) after (5-min) after (1-min)
λ_η (jumps/day) 3.0 0.38 0.077 3.00 3.00
σ_τ (daily vol) 0.020 0.0022 0.0010 0.0198 0.0199
Cramér-Rao bound 9.5e-4 2.0e-4 8.4e-3 8.3e-3

compose(rate_change, systemic_crisis): LinAlgError → works.

Replication: python -m pytest tests/test_noise.py tests/test_events.py -v

Testing

Tests 133 → 144. Each new test was verified to FAIL on the parent commit:
5/6 new noise tests, 4/5 new event tests. The ones that pass on old code are
deliberate — each suite keeps a diagonal-noise case guarding against
over-correction.

Tests pin semantics, not magic numbers: λ is a rate (held jump count fixed
while varying elapsed time — the comparison a count-shaped estimator can't
fake), and covariance stays associative under composition.

Known limitation, not addressed here

λ_η units are now correct, but the Lee–Mykland detector is underpowered on
daily bars (needs ~3.5σ moves). demo/run_egamec.py reports 0.001 jumps/day
against a synthetic truth of 0.01–0.10. Separate problem; flagging it rather
than papering over it.

One downstream effect

notebooks/day03_dual_noise.ipynb prints σ_τ against a hardcoded "true"
string. The estimate is now correct (0.190 vs true annual vol 0.20 — that
notebook's dt is in years), but the hardcoded comparison is a per-bar number
mislabeled "/day", so the line now visibly disagrees. Two-line notebook fix;
happy to include it here or split it out — reviewer's call.

Questions for reviewers

  1. Should calibrate() take an explicit unit (per="day") rather than
    inferring from dt? The current contract is "whatever unit dt is in,"
    which is what the notebook tripped on.
  2. Is the 1e-9 Cholesky jitter in compose() the right regularizer, or
    should near-singular Σ raise instead of being silently nudged?

PS: More details in changes.md

0xgny and others added 2 commits August 7, 2026 12:25
…sition

Two dimensional-analysis bugs in the pure-numpy core, each with regression
tests that fail on the parent commit.

1. state/noise.py — λ_η was a count, not a rate

   calibrate() computed `lambda_eta = N_jumps * dt`, the reciprocal of an
   intensity. Three jumps planted in one day of 5-min bars reported 0.038
   instead of ~3, and the value did not change when the sample got longer.
   σ_τ had the matching problem: `sqrt(BPV/n)` is per-BAR volatility, so
   Theorem III.1 was adding a per-bar variance to a per-day jump term.

   Both are now per unit time on the clock dt is expressed in:
     T = n·dt,  σ_τ² = BPV/T,  λ_η = N_jumps/T
   which makes cramer_rao_bound(h) invariant to sampling frequency, as a
   statement about a process should be.

   The only existing assertion on lambda_eta was `>= 0`, which a
   dimensionally inverted quantity satisfies, so the suite stayed green.

   demo/run_egamec.py passed dt=1/252 for DAILY bars (declaring each bar to
   be 1/252 of a day); corrected to dt=1.0.

2. events/operators.py — compose() squared the Cholesky factor wrongly

   Sigma_w is documented as a Cholesky factor and read as `S @ S.T`
   everywhere else in the codebase, but compose() used `S @ S`. These agree
   only for diagonal S.

   systemic_crisis_operator is the one constructor with off-diagonal noise,
   and nothing composed it. For that operator `S @ S` is not PSD, so the
   np.linalg.cholesky in compose() raised LinAlgError: a systemic crisis
   could not be composed with any other event at all.

Tests: 133 -> 144. New tests pin semantics (λ is a rate; covariance obeys
L1 L1ᵀ + A1 L2 L2ᵀ A1ᵀ and stays associative) rather than magic numbers, and
each suite keeps a case covering the previously-correct diagonal path to
guard against over-correction.

Not addressed here, filed separately: demo/data/memory2026/*.csv are
referenced by SOURCES.md and hindcast_memory_2026.py but never committed, so
that demo raises FileNotFoundError on a fresh clone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant