From 1fb89718e2e0b350c6ac6a9a7a146bd5e45e61d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 17 Jun 2026 17:28:33 +0000 Subject: [PATCH] Remove point-in-time provenance/audit docs Delete three snapshot documents that captured one-off audit state at the time of specific PRs and were not referenced by any code, CI, or other docs: - native_original_src_coverage.md (C++ binding audit table) - original_tests_adoption.md (original R test adoption inventory) - r13_cache_regeneration.md (one-time cache regeneration provenance) The authoritative, living state for these concerns lives in the test suite, the committed cache, and docs/parity.md. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DZHM4Lz9Z7bfJunct9mJBt --- docs/native_original_src_coverage.md | 159 ------------------------- docs/original_tests_adoption.md | 47 -------- docs/r13_cache_regeneration.md | 170 --------------------------- 3 files changed, 376 deletions(-) delete mode 100644 docs/native_original_src_coverage.md delete mode 100644 docs/original_tests_adoption.md delete mode 100644 docs/r13_cache_regeneration.md diff --git a/docs/native_original_src_coverage.md b/docs/native_original_src_coverage.md deleted file mode 100644 index abdc11d0..00000000 --- a/docs/native_original_src_coverage.md +++ /dev/null @@ -1,159 +0,0 @@ -# Native original C++ source coverage audit - -This document audits the vendored C++ core under `extern/NNS-core/include/nns` and -`extern/NNS-core/src`. The goal is targeted native coverage for original C++ core -source files, not binding the whole Python package and not changing PyPI packaging. - -Status values used below: - -- `bound-public`: exposed through `_nnscore` and routed from an existing public Python API. -- `bound-private`: exposed through private `_nnscore` bindings for backend support/smoke tests. -- `cxx-exists-unbound`: present in C++ but intentionally not bound in this PR. -- `python-only`: Python implementation exists without a direct C++ binding in this PR. -- `no-python-wrapper`: no existing public Python wrapper was found. -- `internal-helper`: helper intentionally treated as private backend support. -- `unclear`: semantics/shape mapping need more audit before binding. - -## Audited C++ files - -- `extern/NNS-core/src/partial_moments.cpp` -- `extern/NNS-core/src/central_tendencies.cpp` -- `extern/NNS-core/src/fast_lm.cpp` -- `extern/NNS-core/src/internal_functions.cpp` -- `extern/NNS-core/src/dependence.cpp` -- `extern/NNS-core/src/distance.cpp` -- `extern/NNS-core/src/partition.cpp` -- `extern/NNS-core/src/seasonality.cpp` -- `extern/NNS-core/src/stochastic_dominance.cpp` - -## Coverage table - -| C++ header | C++ function or type | C++ source file | Existing Python public function | Existing Python module | Currently bound in `_nnscore` | Should be public Python API | Should be private backend helper only | Binding priority | Notes | -|---|---|---|---|---|---|---|---|---|---| -| `partial_moments.hpp` | `PMMatrixResult` | `partial_moments.cpp` | `pm_matrix` result dict | `nns.pm_matrix` | bound-public | yes | no | Phase 1 | Bound as dict preserving Python `cov.matrix` key. | -| `partial_moments.hpp` | `lpm` | `partial_moments.cpp` | `lpm` | `nns.core` | bound-public | yes | no | Phase 1 | Existing binding confirmed and routed. | -| `partial_moments.hpp` | `upm` | `partial_moments.cpp` | `upm` | `nns.core` | bound-public | yes | no | Phase 1 | Existing binding confirmed and routed. | -| `partial_moments.hpp` | `lpm_v` | `partial_moments.cpp` | `lpm` vector target path | `nns.core` | bound-public | yes | no | Phase 1 | Also exposed as private explicit `_nnscore.lpm_v`. | -| `partial_moments.hpp` | `upm_v` | `partial_moments.cpp` | `upm` vector target path | `nns.core` | bound-public | yes | no | Phase 1 | Also exposed as private explicit `_nnscore.upm_v`. | -| `partial_moments.hpp` | `lpm_ratio_v` | `partial_moments.cpp` | `lpm_ratio` | `nns.core` | bound-public | yes | no | Phase 1 | Routed through native when available. | -| `partial_moments.hpp` | `upm_ratio_v` | `partial_moments.cpp` | `upm_ratio` | `nns.core` | bound-public | yes | no | Phase 1 | Routed through native when available. | -| `partial_moments.hpp` | `co_lpm` | `partial_moments.cpp` | `co_lpm` | `nns.co_moments` | bound-public | yes | no | Phase 1 | Scalar smoke binding plus vector route. | -| `partial_moments.hpp` | `co_upm` | `partial_moments.cpp` | `co_upm` | `nns.co_moments` | bound-public | yes | no | Phase 1 | Scalar smoke binding plus vector route. | -| `partial_moments.hpp` | `d_lpm` | `partial_moments.cpp` | `d_lpm` | `nns.co_moments` | bound-public | yes | no | Phase 1 | Scalar smoke binding plus vector route. | -| `partial_moments.hpp` | `d_upm` | `partial_moments.cpp` | `d_upm` | `nns.co_moments` | bound-public | yes | no | Phase 1 | Scalar smoke binding plus vector route. | -| `partial_moments.hpp` | `co_lpm_v` | `partial_moments.cpp` | `co_lpm` vector target path | `nns.co_moments` | bound-public | yes | no | Phase 1 | Preserves recycled target vector behavior. | -| `partial_moments.hpp` | `co_upm_v` | `partial_moments.cpp` | `co_upm` vector target path | `nns.co_moments` | bound-public | yes | no | Phase 1 | Preserves recycled target vector behavior. | -| `partial_moments.hpp` | `d_lpm_v` | `partial_moments.cpp` | `d_lpm` vector target path | `nns.co_moments` | bound-public | yes | no | Phase 1 | Preserves recycled target vector behavior. | -| `partial_moments.hpp` | `d_upm_v` | `partial_moments.cpp` | `d_upm` vector target path | `nns.co_moments` | bound-public | yes | no | Phase 1 | Preserves recycled target vector behavior. | -| `partial_moments.hpp` | `clpm_nd` | `partial_moments.cpp` | `co_lpm_nd` | `nns.dependence` | bound-private | yes | no | Phase 1 | Bound for native coverage; public routing deferred because current Python shape semantics need separate parity work. | -| `partial_moments.hpp` | `cupm_nd` | `partial_moments.cpp` | `co_upm_nd` | `nns.dependence` | bound-private | yes | no | Phase 1 | Bound for native coverage; public routing deferred. | -| `partial_moments.hpp` | `dpm_nd` | `partial_moments.cpp` | `dpm_nd` | `nns.dependence` | bound-private | yes | no | Phase 1 | Bound for native coverage; public routing deferred. | -| `partial_moments.hpp` | `clpm_nd_batch` | `partial_moments.cpp` | none | none | bound-private | no | yes | Phase 1 | Backend vectorized helper only. | -| `partial_moments.hpp` | `pm_matrix` | `partial_moments.cpp` | `pm_matrix` | `nns.pm_matrix` | bound-public | yes | no | Phase 1 | Routed through native with column-major flattening. | -| `central_tendencies.hpp` | `gravity` | `central_tendencies.cpp` | `nns_gravity` | `nns.central_tendencies` | bound-private | yes | no | Phase 5 | Already bound before this PR; public routing was already present/available through module behavior. | -| `central_tendencies.hpp` | `rescale` | `central_tendencies.cpp` | `nns_rescale` | `nns.central_tendencies` | cxx-exists-unbound | yes | no | Phase 5 | Left unbound to avoid changing risk-neutral/min-max edge behavior without parity tests. | -| `central_tendencies.hpp` | `mode` | `central_tendencies.cpp` | `nns_mode` | `nns.central_tendencies` | bound-private | yes | no | Phase 5 | Already bound before this PR. | -| `fast_lm.hpp` | `FastLmResult` | `fast_lm.cpp` | `_fast_lm` result dict | `nns.multivariate_regression` | bound-private | no | yes | Phase 2 | Existing `fast_lm` binding confirmed. | -| `fast_lm.hpp` | `FastLmMultResult` | `fast_lm.cpp` | none found | none | bound-private | no | yes | Phase 2 | Added native binding; no public route because no existing public wrapper uses it directly. | -| `fast_lm.hpp` | `fast_lm` | `fast_lm.cpp` | `_fast_lm` helper | `nns.multivariate_regression` | bound-private | no | yes | Phase 2 | Existing binding confirmed; remains backend-only. | -| `fast_lm.hpp` | `fast_lm_mult` | `fast_lm.cpp` | none found | none | bound-private | no | yes | Phase 2 | Added smoke-tested backend binding. | -| `internal_functions.hpp` | `ValueKind` | `internal_functions.cpp` | none | none | cxx-exists-unbound | no | yes | Phase 3 | Enum is only useful if `is_fcl` is exposed; Python has no direct type mapping need. | -| `internal_functions.hpp` | `is_fcl` | `internal_functions.cpp` | `_is_fcl` internal equivalent | `nns.regression` | cxx-exists-unbound | no | yes | Phase 3 | Not bound; Python object dtype/factor detection is richer than the C++ enum boundary. | -| `internal_functions.hpp` | `Factor` | `internal_functions.cpp` | factor helpers | `nns.categorical` | bound-private | no | yes | Phase 3 | Mapped to `(codes, levels)` arguments, not exposed as a C++ class. | -| `internal_functions.hpp` | `DummyMatrix` | `internal_functions.cpp` | factor helpers | `nns.categorical` | bound-private | no | yes | Phase 3 | Returned as dict with flat column-major data, names, nrow, ncol. | -| `internal_functions.hpp` | `factor_2_dummy` | `internal_functions.cpp` | `factor_2_dummy` | `nns.categorical` | bound-private | yes | yes | Phase 3 | Bound only as private backend helper; public routing deferred. | -| `internal_functions.hpp` | `factor_2_dummy_fr` | `internal_functions.cpp` | `factor_2_dummy_fr` | `nns.categorical` | bound-private | yes | yes | Phase 3 | Bound only as private backend helper; public routing deferred. | -| `internal_functions.hpp` | `vec_sd` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Safe numeric helper bound for backend use. | -| `internal_functions.hpp` | `col_sd` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Safe numeric helper bound for backend use with explicit dimensions. | -| `internal_functions.hpp` | `is_discrete` | `internal_functions.cpp` | internal checks | multiple | bound-private | no | yes | Phase 3 | Safe numeric helper bound for backend use. | -| `internal_functions.hpp` | `TimeSeriesVectors` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Dict result for private backend support. | -| `internal_functions.hpp` | `ForecastVectors` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Dict result for private backend support. | -| `internal_functions.hpp` | `generate_vectors` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Safe explicit vector/list conversion. | -| `internal_functions.hpp` | `generate_lin_vectors` | `internal_functions.cpp` | none public | none | bound-private | no | yes | Phase 3 | Safe explicit vector/list conversion. | -| `internal_functions.hpp` | `ARMAWeights` | `internal_functions.cpp` | ARMA internals | `nns.arma` | cxx-exists-unbound | no | yes | Phase 4 | Left unbound; structured ARMA weighting semantics need parity tests. | -| `internal_functions.hpp` | `arma_seas_weighting` | `internal_functions.cpp` | ARMA internals | `nns.arma` | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because period/covariance frame semantics need separate validation. | -| `internal_functions.hpp` | `meboot_part` | `internal_functions.cpp` | `nns_meboot` internals | `nns.meboot` | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because it has random seed and boundary semantics requiring dedicated parity tests. | -| `internal_functions.hpp` | `meboot_expand_sd` | `internal_functions.cpp` | `nns_meboot` internals | `nns.meboot` | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because it mutates column-major ensemble buffers in place. | -| `internal_functions.hpp` | `force_clt` | `internal_functions.cpp` | `nns_meboot` internals | `nns.meboot` | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because it mutates buffers and affects stochastic bootstrap distributions. | -| `internal_functions.hpp` | `SampleResult` | `internal_functions.cpp` | sampling internals | none | cxx-exists-unbound | no | yes | Phase 4 | Structured output; no current public API route. | -| `internal_functions.hpp` | `up_sample` | `internal_functions.cpp` | none public | none | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because class balancing and seed semantics need a public contract first. | -| `internal_functions.hpp` | `down_sample` | `internal_functions.cpp` | none public | none | cxx-exists-unbound | no | yes | Phase 4 | Left unbound because class balancing and seed semantics need a public contract first. | -| `dependence.hpp` | `DepResult` | `dependence.cpp` | `nns_dep`/`nns_cor` result pieces | `nns.dependence` | cxx-exists-unbound | yes | no | Phase 5 | Requires pre-hashed partition labels for `dep_pair`; leave unbound pending wrapper design. | -| `dependence.hpp` | `DepMatrixResult` | `dependence.cpp` | matrix results | `nns.dependence` | cxx-exists-unbound | yes | no | Phase 5 | Structured matrix result; leave until parity for matrix orientation is added. | -| `dependence.hpp` | `dep_pair` | `dependence.cpp` | `nns_dep`, `nns_cor` | `nns.dependence` | cxx-exists-unbound | yes | no | Phase 5 | Needs partition hash inputs not exposed by current Python public API. | -| `dependence.hpp` | `dep_matrix` | `dependence.cpp` | dependence matrix APIs | `nns.dependence` | cxx-exists-unbound | yes | no | Phase 5 | Safe candidate later; not bound in this PR to avoid output shape changes. | -| `distance.hpp` | `distance` | `distance.cpp` | `nns_distance` | `nns.distance` | cxx-exists-unbound | yes | no | Phase 5 | Left unbound; current Python code includes rescaling/weighting behavior requiring parity comparison. | -| `distance.hpp` | `distance_path` | `distance.cpp` | distance path behavior | `nns.distance` | cxx-exists-unbound | yes | no | Phase 5 | Left unbound pending k/path output contract tests. | -| `distance.hpp` | `distance_bulk` | `distance.cpp` | `nns_distance_bulk` | `nns.distance` | cxx-exists-unbound | yes | no | Phase 5 | Left unbound pending row/column-major parity tests. | -| `distance.hpp` | `distance_path_parallel` | `distance.cpp` | none direct | none | cxx-exists-unbound | no | yes | Phase 5 | Parallel helper; no public wrapper. | -| `distance.hpp` | `distance_path_single_parallel` | `distance.cpp` | none direct | none | cxx-exists-unbound | no | yes | Phase 5 | Parallel helper; no public wrapper. | -| `partition.hpp` | `PartitionRow` | `partition.cpp` | partition result rows | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Structured object mapping deferred. | -| `partition.hpp` | `RegressionPoint` | `partition.cpp` | regression points | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Structured object mapping deferred. | -| `partition.hpp` | `SegmentH` | `partition.cpp` | `segments_h` | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Structured object mapping deferred. | -| `partition.hpp` | `SegmentV` | `partition.cpp` | `segments_v` | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Structured object mapping deferred. | -| `partition.hpp` | `PartitionResult` | `partition.cpp` | `nns_part` result dict | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Complex R-compatible payload; not changed in this PR. | -| `partition.hpp` | `partition` | `partition.cpp` | `nns_part` | `nns.part` | cxx-exists-unbound | yes | no | Phase 5 | Safe candidate later, but output shape/labels must remain exact. | -| `seasonality.hpp` | `SeasonalityResult` | `seasonality.cpp` | `nns_seas` result pieces | `nns.seasonality` | cxx-exists-unbound | yes | no | Phase 5 | Structured result left unbound pending parity tests. | -| `seasonality.hpp` | `seasonality` | `seasonality.cpp` | `nns_seas` | `nns.seasonality` | cxx-exists-unbound | yes | no | Phase 5 | Left unbound because modulo and result-shape semantics need public parity tests. | -| `stochastic_dominance.hpp` | `fsd_uni` | `stochastic_dominance.cpp` | `fsd_uni` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Candidate for future; not required by current native routing tests. | -| `stochastic_dominance.hpp` | `ssd_uni` | `stochastic_dominance.cpp` | `ssd_uni` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Candidate for future. | -| `stochastic_dominance.hpp` | `tsd_uni` | `stochastic_dominance.cpp` | `tsd_uni` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Candidate for future. | -| `stochastic_dominance.hpp` | `fsd` | `stochastic_dominance.cpp` | `fsd` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Matrix orientation and index base must be validated before routing. | -| `stochastic_dominance.hpp` | `ssd` | `stochastic_dominance.cpp` | `ssd` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Matrix orientation and index base must be validated before routing. | -| `stochastic_dominance.hpp` | `tsd` | `stochastic_dominance.cpp` | `tsd` | `nns.stochastic_dominance` | cxx-exists-unbound | yes | no | Phase 5 | Matrix orientation and index base must be validated before routing. | -| `stochastic_dominance.hpp` | `StochSupResult` | `stochastic_dominance.cpp` | `nns_ss` result dict | `nns.stochastic_superiority` | bound-private | yes | no | Existing | Already bound before this PR. | -| `stochastic_dominance.hpp` | `stochastic_superiority` | `stochastic_dominance.cpp` | `nns_ss` | `nns.stochastic_superiority` | bound-private | yes | no | Existing | Already bound before this PR. | - -## Python APIs routed through native in this PR - -- `nns.core.lpm` -- `nns.core.upm` -- `nns.core.lpm_ratio` -- `nns.core.upm_ratio` -- `nns.co_moments.co_lpm` -- `nns.co_moments.co_upm` -- `nns.co_moments.d_lpm` -- `nns.co_moments.d_upm` -- `nns.pm_matrix.pm_matrix` - -## Functions newly bound in `_nnscore` - -- Partial moment vector and ratio helpers: `lpm_v`, `upm_v`, `lpm_ratio_v`, `upm_ratio_v`. -- Co-partial moment helpers: `co_lpm`, `co_upm`, `d_lpm`, `d_upm`, `co_lpm_v`, `co_upm_v`, `d_lpm_v`, `d_upm_v`. -- N-dimensional/backend helpers: `clpm_nd`, `cupm_nd`, `dpm_nd`, `clpm_nd_batch`, `pm_matrix`. -- Fast linear model helper: `fast_lm_mult` (`fast_lm` was already bound). -- Private internal helpers: `is_discrete`, `vec_sd`, `col_sd`, `factor_2_dummy`, `factor_2_dummy_fr`, `generate_vectors`, `generate_lin_vectors`. - -## Functions already bound before this PR - -- `lpm` -- `upm` -- `gravity` -- `mode` -- `fast_lm` -- `stochastic_superiority` - -## Intentionally left unbound or Python-only - -- `central_tendencies::rescale`: Python remains authoritative until min-max/risk-neutral edge cases have direct parity tests. -- `dependence::{dep_pair, dep_matrix}` and result types: `dep_pair` needs pre-hashed partition labels, and matrix orientation/routing needs a dedicated test suite. -- `distance::*`: existing Python wrappers include public rescaling, class, weighting, and k-path behavior. They remain Python-only until shape and parity tests are added. -- `partition::*`: complex R-compatible result payload is left Python-only to avoid changing dictionary/list shapes. -- `seasonality::*`: structured result and modulo behavior need separate parity coverage. -- `stochastic_dominance::{fsd_uni, ssd_uni, tsd_uni, fsd, ssd, tsd}`: public Python implementations remain in place; matrix output index conventions need explicit tests before native routing. -- `internal_functions::{is_fcl, arma_seas_weighting, meboot_part, meboot_expand_sd, force_clt, up_sample, down_sample}`: intentionally not bound in this PR. The ARMA and meboot helpers involve structured outputs, mutation, random seeds, or statistical distribution semantics. Sampling helpers need a public class-balancing contract before exposure. - -## Additional notes - -- Regression is not treated as a direct C++ binding unless a C++ equivalent exists. The `fast_lm` and `fast_lm_mult` helpers are private backend utilities, not replacements for the Python NNS regression API. -- `internal_functions.cpp` is treated mostly as private backend support. Its bindings are not public top-level Python exports. -- Public APIs call `from nns._native import nnscore`; if `nnscore()` returns a module they use native C++, and if it returns `None` they fall back to the existing Python implementation. -- Windows local MinGW builds may fail to load `_nnscore`; official Windows wheels should be built with MSVC. - -## Non-source-support headers in `extern/NNS-core/include/nns` - -| C++ header | C++ function or type | C++ source file | Existing Python public function | Existing Python module | Currently bound in `_nnscore` | Should be public Python API | Should be private backend helper only | Binding priority | Notes | -|---|---|---|---|---|---|---|---|---|---| -| `nns.hpp` | umbrella header includes component modules | none | none | none | internal-helper | no | yes | none | Include-only aggregator; no functions or result types to bind. | -| `parallel.hpp` | parallel execution helpers | header/internal support | none | none | internal-helper | no | yes | none | Build/runtime support for C++ core parallel loops; no public Python API. | -| `version.hpp` | `NNS_CORE_VERSION_MAJOR`, `NNS_CORE_VERSION_MINOR`, `NNS_CORE_VERSION_PATCH`, `NNS_CORE_VERSION` | none | none | none | cxx-exists-unbound | no | yes | none | Compile-time version macros; not bound in this PR. | diff --git a/docs/original_tests_adoption.md b/docs/original_tests_adoption.md deleted file mode 100644 index 16a4c8e1..00000000 --- a/docs/original_tests_adoption.md +++ /dev/null @@ -1,47 +0,0 @@ -# Original R Tests Adoption - -`original_tests/` is the authoritative source for parity coverage added from the original R NNS test suite. The inventory below records every file currently present under `original_tests/`, including non-test artifacts. - -## Inventory and adoption matrix - -| Original file path | R function or behavior tested | Python equivalent | Current Python module | Fixture or R cache needed | Pytest file created | Adoption status | Notes | -|---|---|---|---|---|---|---|---| -| `original_tests/testthat.R` | R `testthat` package harness (`library(testthat)`, `library(NNS)`, `test_check("NNS")`) | No runtime pytest equivalent; repository pytest invocation is the harness | n/a | none | n/a | no-python-equivalent | Harness file is inventoried but not converted because Python uses pytest directly. | -| `original_tests/testthat/Rplots.pdf` | Plot artifact generated by R tests | No Python API behavior | n/a | none | n/a | no-python-equivalent | Binary PDF artifact is inventoried and intentionally not used or compared by CI. See `docs/plot_parity_policy.md`. | -| `original_tests/testthat/test_ANOVA.R` | `NNS::NNS.ANOVA(cbind(x,y,z))` certainty and `pairwise=TRUE` matrix | `nns.nns_anova` | `src/nns/anova.py` | `tests/fixtures/original_tests_expected.json` stores R certainty and pairwise matrix | `tests/parity/test_original_anova.py` | adopted | Uses the original `x`, `y`, and `z` vectors parsed from the R file; tolerance follows the R test (`1e-4`). | -| `original_tests/testthat/test_Copula.R` | `NNS.copula` for bivariate continuous/discrete and 3-column continuous/discrete | `nns.nns_copula` (bivariate continuous and discrete; multivariate continuous and discrete) | `src/nns/copula.py` | `tests/fixtures/original_tests_expected.json` stores all four R expected values | `tests/parity/test_original_dependence.py` | adopted | All four original cases are adopted: bivariate continuous `nns_copula(x, y)`, bivariate discrete `nns_copula(x, y, continuous=False)`, three-column continuous `nns_copula(Z)`, and three-column discrete `nns_copula(Z, continuous=False)`. `Z` is an `(observations, variables)` matrix (rows = observations, columns = variables), matching R's `data.frame(x, y, z)`. Each matches its R fixture to `1e-5`. | -| `original_tests/testthat/test_FSD_SSD_TSD.R` | `NNS.FSD`, `NNS.SSD`, and `NNS.TSD` dominance labels for original vectors and squared-vector dominance cases | `nns.fsd`, `nns.ssd`, `nns.tsd` | `src/nns/stochastic_dominance.py` | `tests/fixtures/original_tests_expected.json` stores R labels | `tests/parity/test_original_stochastic.py` | adopted | Plot flags in the R source are intentionally not represented because Python parity tests compare return values and CI must not create plot devices. Python bidirectional `fsd` currently implements the discrete path. | -| `original_tests/testthat/test_Partial_Moments.R` | `LPM`, `UPM`, `Co.UPM`, `Co.LPM`, `D.LPM`, `D.UPM`, `LPM.ratio`, `UPM.ratio`, `PM.matrix`, normalized covariance identity, and survival `NNS.CDF` | `nns.lpm`, `nns.upm`, `nns.co_upm`, `nns.co_lpm`, `nns.d_lpm`, `nns.d_upm`, `nns.lpm_ratio`, `nns.upm_ratio`, `nns.pm_matrix`, `nns.nns_cdf` | `src/nns/core.py`, `src/nns/co_moments.py`, `src/nns/pm_matrix.py`, `src/nns/cdf.py` | `tests/fixtures/original_tests_expected.json` stores R scalar outputs, covariance matrices, and survival CDF table | `tests/parity/test_original_partial_moments.py` | partially-adopted | Scalar partial moments, PM matrix covariance outputs, and survival CDF are adopted. The normalized covariance identity is duplicate-existing-coverage-style behavior and is not reasserted in the original parity file. R data-frame dimname behavior is now exposed as an optional `pm_matrix(..., names=[...])` parameter that echoes column labels under a `"names"` key without altering the numeric NumPy arrays; `test_pm_matrix_optional_names_match_r_dataframe_without_changing_numbers` proves names match R while numeric parity is unaffected. | -| `original_tests/testthat/test_Partition_Map.R` | `NNS.part(x,y, Voronoi=FALSE, min.obs.stop=TRUE)` order, full row-wise partition table, and regression points | `nns.nns_part` | `src/nns/part.py` | `tests/fixtures/original_tests_expected.json` stores R order and regression points; quadrant and prior quadrant vectors are parsed from the original R file | `tests/parity/test_original_partition.py` | adopted | Preserves row order, quadrant labels, prior quadrant labels, and regression point order. | -| `original_tests/testthat/test_SD_efficient_Set.R` | `NNS.SD.efficient.set` for degrees 1-3 and FSD discrete/continuous type | `nns.sd_efficient_set` | `src/nns/stochastic_dominance.py` | `tests/fixtures/original_tests_expected.json` stores the R efficient-set name order | `tests/parity/test_original_stochastic.py` | adopted | Converts Python column indices back to the original R names (`x`, `y`, `z`, `xx`, `yy`, `zz`) to preserve name and order parity. | -| `original_tests/testthat/test_Uni_SD_Routines.R` | `NNS.FSD.uni`, `NNS.SSD.uni`, and `NNS.TSD.uni` unidirectional dominance flags | `nns.fsd_uni`, `nns.ssd_uni`, `nns.tsd_uni` | `src/nns/stochastic_dominance.py` | `tests/fixtures/original_tests_expected.json` stores R integer outputs | `tests/parity/test_original_stochastic.py` | adopted | Uses original vectors and squared-vector cases. FSD discrete and continuous paths from R are both represented. | - -## Fixture policy - -- CI parity tests do **not** require `Rscript`; adopted original tests compare Python outputs against committed expected values in `tests/fixtures/original_tests_expected.json` and/or literal vectors parsed from `original_tests/testthat/*.R`. -- No Python-generated expected values are used. Expected values in `tests/fixtures/original_tests_expected.json` are copied from the R test expectations in `original_tests/`. -- No stochastic original test required seed preservation in this inventory. The original vectors appear committed as deterministic numeric fixtures from the R files. - -## Resolved former gaps - -- `NNS.copula(..., continuous=FALSE)` (discrete) and three-column `NNS.copula` - (multivariate continuous and discrete) are now implemented and adopted with - direct fixture-backed parity tests. The Python `nns_copula` accepts either two - 1-D vectors (bivariate) or a single 2-D `(observations, variables)` matrix - (multivariate, any column count `>= 2`), plus a `continuous` flag. -- R data-frame naming behavior in `PM.matrix` is addressed by the optional - `pm_matrix(..., names=[...])` parameter (NumPy-first; labels echoed under a - `"names"` key). A parity test proves names match R and numeric matrices are - unchanged. - -## Intentional, permanent divergences (not blockers) - -- R plot flags and the `Rplots.pdf` artifact are not adopted into pytest because - CI parity compares returned values and never graphics-device artifacts. See - `docs/plot_parity_policy.md`. -- `PM.matrix` matrices remain NumPy-first arrays without R-style dimnames; - labels are available only via the optional `names` echo described above. - -## Out of scope - -The NNS-python migration remains out of scope. The `nns` package name is unchanged. diff --git a/docs/r13_cache_regeneration.md b/docs/r13_cache_regeneration.md deleted file mode 100644 index 229f2b64..00000000 --- a/docs/r13_cache_regeneration.md +++ /dev/null @@ -1,170 +0,0 @@ -# R NNS 13.0 parity cache regeneration provenance - -This document records the fresh, from-empty regeneration of -`tests/_r_cache.json` against live vendored R NNS 13.0. - -## Environment - -| Item | Value | -| --- | --- | -| Date of fresh regeneration | 2026-06-13 (UTC) | -| OS | Ubuntu 24.04.4 LTS (Linux 6.18.5 x86_64) | -| Python | 3.11.15 | -| R | R version 4.3.3 (2024-02-29) "Angel Food Cake" | -| Rscript | Rscript (R) version 4.3.3 (2024-02-29) | -| Vendored NNS source | `tools/NNS` (extracted package directory, preferred over `tools/NNS_13.0.tar.gz`) | -| `packageVersion("NNS")` | `13.0` | - -R NNS itself was installed exclusively from the vendored repository source via -`scripts/install_local_r_nns.py` (`R CMD INSTALL tools/NNS`), never from CRAN. -R package dependencies required to load vendored NNS (`data.table`, -`doParallel`, `foreach`, `Rcpp`, `RcppParallel`, `rgl`, `xts`, `zoo`, -`jsonlite`) were installed as Ubuntu binary packages; `Rfast` (plus its `zigg` -dependency) was built from the upstream GitHub release source -`RfastOfficial/Rfast` tag `v2.1.5.1-apollo` because CRAN was unreachable in -the regeneration environment. Only dependencies came from external archives; -NNS came from `tools/NNS`. - -## Commands run - -```bash -python scripts/install_local_r_nns.py -Rscript -e "suppressPackageStartupMessages(library(NNS)); cat(as.character(packageVersion('NNS')))" -# printed: 13.0 - -# Fresh regeneration: moves tests/_r_cache.json to tests/_r_cache.json.bak, -# starts from an EMPTY cache, and repopulates every entry with a live R call. -python scripts/regenerate_r_cache.py --fresh -- -n 0 tests/parity - -# Replay checks after regeneration -PYNNS_R_CACHE_ONLY=1 python -m pytest -q -n 0 tests/parity -python -m pytest -q tests/parity/test_r13_smoke.py -python -m pytest -q tests/invariants -ruff check . -mypy -python -m build -``` - -All cache-only/offline toggles (`PYNNS_R_CACHE_ONLY`, `NNS_R_CACHE_ONLY`, -`PYNNS_OFFLINE`, `NNS_OFFLINE`, `CI`) were unset for the regeneration run. -The `--fresh` mode added to `scripts/regenerate_r_cache.py` in this change -refuses to run in CI, verifies a live local R NNS 13.0 install before touching -anything, then moves the existing cache aside so no existing entry can be -reused during regeneration. - -The fresh regeneration was run twice: once to surface the true Python parity -gaps against live R 13.0 (216 test failures), and a second time from an empty -cache after the Python fixes below, confirming every parity test passes -against entries produced exclusively by live R calls. - -## Results - -| Item | Value | -| --- | --- | -| `nns_version` | `13.0` | -| `schema_version` | `1` | -| Final entry count | 2385 | -| Prior entry count | 2406 | - -The entry count dropped from 2406 to 2385 because 21 entries in the old -committed cache are stale: an instrumented cache-only replay confirmed that no -current parity test computes those 21 keys, so a from-empty regeneration never -recreates them. All 2385 keys requested by the current test suite were -regenerated from live R. - -### Deterministic cache entries that changed - -456 of the 2384 shared keys changed value relative to the previously committed -cache, summarized by NNS function (mapped by instrumenting the cache-key -computation during a full replay): - -| Function | Changed entries | -| --- | --- | -| `PM.matrix` | 216 | -| `NNS.reg` | 147 | -| `NNS.boost` (numeric harness) | 25 | -| `LPM.VaR` | 12 | -| `UPM.VaR` | 12 | -| `NNS.ANOVA` (custom harness) | 12 | -| `NNS.M.reg` | 10 | -| `NNS.stack` (numeric harness) | 6 | -| `NNS.dep` | 3 | -| `NNS.caus` | 3 | -| `NNS.moments` | 2 | -| `dy.d` (scalar harnesses) | 4 | -| factor-predictor harnesses (boost/stack/reg) | 4 | - -The fresh live R 13.0 values were treated as authoritative in every case. - -### Python parity fixes required - -Fresh regeneration surfaced 216 failing parity tests. All of them traced to -four deterministic divergences between NNS Python and vendored R NNS 13.0, and -Python was fixed to match R in each case: - -1. **`LPM.VaR` / `UPM.VaR` integer degrees 1–4** (`src/nns/var.py`): - R 13.0 replaced the `optimize()` search with an exact polynomial - root-finding inversion on the located order-statistic interval - (`.NNS_LPM_VaR_integer`). Python now ports that algorithm (prefix power - sums, break-ratio interval location, Brent root with - `tol = .Machine$double.eps^0.5`, and the same boundary/fallback handling). - This also fixed the `NNS.reg`/`NNS.M.reg` confidence intervals and - stack/boost prediction intervals built on these helpers. -2. **Distance-kernel lognormal rank weight** (`src/nns/distance.py`): - R 13.0's bulk path kernels (`NNS_distance_path_single_parallel_cpp`, used - for fitted values when `n.best > 1` and for multi-point estimates) use the - population sd of ranks `sqrt((k^2 - 1) / 12)` in the lognormal weight, - while the single-point `NNS_distance_cpp` kernel keeps the sample sd. - Python now provides `nns_distance_path_single_bulk` mirroring the bulk - kernel and keeps `nns_distance` on the single-point formulation. -3. **`NNS.M.reg` out-of-hull multi-point extrapolation** - (`src/nns/multivariate_regression.py`): R 13.0 vectorized the multi-point - outsider path (bulk kernel estimates, `pmax(distance, 1e-10)` gradient - guards) and removed the old dims-dropping behavior for a single outsider - row that Python previously emulated. Python now mirrors the new path. -4. **`NNS.ARMA` numeric multi-lag seasonal weighting** (`src/nns/arma.py`): - R weights each numeric seasonal factor by reversing the series with the - factor's *position* in the `seasonal.factor` vector - (`variable[seq(length(variable), 1, -i)]`), not its lag value. Python now - matches, which resolved the two previously `xfail`-ed Sunspots ARMA and - macro-like VAR practical examples; both now pass against live R and the - `xfail` markers were removed. - -Structural test updates justified by fresh R output: R 13.0's `NNS.boost` -returns only `results`, `pred.int`, `feature.weights`, and -`feature.frequency` (no `n.best`), so the Python return dictionary and the -test assertions that expected `n.best` were updated, and the NaN handling of -final boost estimates now matches R (`stack` falls back to `reg` only when -absent; NaNs are filled with the gravity of the remaining estimates). - -The only remaining `xfail` is the balanced Iris `NNS.boost` diagnostic, which -is documented as a stochastic sampling gap (R RNG-driven CV-index, feature -subset, and up/down-sampling draws cannot be reproduced bit-for-bit with -NumPy's RNG). No deterministic parity gap is excluded from the suite. - -### Manual comparison harness - -No `scripts/compare_nns.py` / `scripts/compare_nns_r13.py` diagnostic scripts -exist in this repository, so there was no `$RPM`-based univariate `NNS.reg` -extraction to fix. The univariate regression-point diagnostic lives in -`tests/parity/test_r13_smoke.py` and already extracts regression points via -`NNS.reg(..., multivariate.call = TRUE)$y`, matching the supported R-side -extraction. - -### ARMA nonseasonal nonlinear reconciliation - -The suspected live mismatch (Python `[125.25, 107.75, 158.75, 213.66]` vs R -`[128.50, 113.50, 155.50, 213.66]`) was re-run against live vendored R NNS -13.0 before regeneration: - -- Live R `NNS.ARMA(series, h = 4, seasonal.factor = FALSE, method = "nonlin")` - on the 24-point AirPassengers-style series returned - `128.5, 113.5, 155.5, 213.6666666667`. -- Python `nns_arma(series, h=4, seasonal_factor=False, method="nonlin")` - returned `128.5, 113.5, 155.5, 213.66666667`. - -Python matches live R 13.0 exactly; the previously reported divergent Python -values could not be reproduced with the current implementation. The hardcoded -smoke expectations in `tests/parity/test_r13_smoke.py` (LPM/UPM, copulas, -regression points, seasonal ARMA, nonseasonal nonlinear ARMA, seeded stack) -were each verified against this fresh live R 13.0 run and required no changes.