Fix NNS.dep/NNS.copula independence anchor (discordant partial moment) - #117
Merged
Conversation
…ents Mirrors OVVO-Financial/NNS#49. The discordant partial moment (DPM_nD) treats a point as concordant when it is all-below OR all-above the target -- both fully-aligned orthants -- so under independence P(discordant) is 1 - 2*0.5**n, which is 0.5 for the bivariate copula, not 1 - 0.5**n = 0.75. The old 0.75 anchor left a fixed 1/3 residual in each of the two discordant terms, so sqrt(mean(0, 0, 1/3, 1/3)) = sqrt(1/6) ~ 0.41: a dependence floor that never vanished for independent data regardless of sample size. Correcting the anchor to 1 - 2*0.5**n makes the measure consistent -- independent data now decays toward 0 as n grows -- while leaving perfectly dependent cases at 1. Changed the bivariate kernels (_copula_signed, _copula_degree0_unsigned in both the Python fallback and the native _nnscore bindings; anchor 0.5) and the n-dimensional nns_copula (copula.py and native; 1 - 2*0.5**n). The concordance anchor was already correct at 0.5 and is unchanged. Adds a consistency invariant that only holds under the corrected anchor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019F6ZjcfXSxZWGMuSmQGmLN
regenerate_r_cache.py installs R NNS from the vendored tools/NNS/ source, so it must carry the same indep_D fix (0.75 -> 0.5 in NNS_dep.cpp, and 1-(0.5^n) -> 1-2*(0.5^n) in Copula.R) for the regenerated parity cache to reflect the corrected R behavior. Mirrors OVVO-Financial/NNS#49. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019F6ZjcfXSxZWGMuSmQGmLN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python port of OVVO-Financial/NNS#49 (merged).
Problem
nns_depreported a large, non-vanishing dependence for independent data (≈0.60at n=100, still~0.44asn→∞). A consistent dependence measure must decay to 0 under independence.Root cause
The copula blends four normalized deviations from an independence anchor. Measured at n=200k (sampling noise → 0):
DPM_nDcounts a point concordant when it is all-below OR all-above the target — both fully-aligned orthants — so under independenceP(discordant) = 1 − 2·0.5ⁿ(0.5 bivariate), not1 − 0.5ⁿ = 0.75. The mismatch leaves ⅓ in each discordant term, and the finalsqrtturns it into a permanent floor:sqrt(1/6) ≈ 0.41.Fix
indep_D = 1 − 2·0.5ⁿ(→0.5bivariate), in all four implementations:src/nns/dependence.py—_copula_signed,_copula_degree0_unsignedsrc/nns/copula.py— n-dimensionalnns_copulasrc/nns/_nnscore_bindings.cpp— native bivariate (×3) and nD copulaThe concordance anchor was already correct at 0.5 and is unchanged.
Effect (native-validated)
Independent dependence now decays toward 0 instead of plateauing:
Discrimination preserved: identical/linear = 1.00, quadratic = 0.99. Adds
test_nns_dep_independence_null_is_consistent, which only passes under the corrected anchor.This intentionally changes every
nns_dep/nns_copulavalue, so the committedtests/_r_cache.json(old-anchor R references) is stale and thetests/parity/test_dependence.py/test_copula.pysuites will fail until it is regenerated against the fixed R NNS (#49):Cross-check already in hand: when #49's CI ran R CMD check, R produced copula fixtures
B=0.2554, C=0.2000, D=0.2470, E=0.2000— matching this fixed port exactly on the same inputs, confirming R↔Python parity under the new anchor.🤖 Generated with Claude Code
Generated by Claude Code