Bit-stable parity-cache inputs; shard the R cache per function; drop the numpy pin - #79
Closed
gitRasheed wants to merge 1 commit into
Closed
Bit-stable parity-cache inputs; shard the R cache per function; drop the numpy pin#79gitRasheed wants to merge 1 commit into
gitRasheed wants to merge 1 commit into
Conversation
Root cause chain from the numpy 2.5.1 CI incident, fixed at the source:
- test_co_moments generated correlated inputs with rng.multivariate_normal,
which routes through LAPACK SVD - outside NumPy's stream-stability
guarantee, so cache keys hashed from those inputs depended on the BLAS
build and CPU. Inputs now come from two standard_normal draws correlated
by the closed-form 2x2 Cholesky factor: exactly rounded IEEE arithmetic,
bit-identical everywhere. The numpy<2.5 CI pin is removed; this PR's CI
passing on unpinned numpy across the runner fleet is the proof.
- tests/_r_cache.json (22 MB single blob) is now tests/_r_cache/ with one
shard per R function (59 files). Cache keys gain a function-name prefix
('NNS.reg:<sha256>'; digest part unchanged) which routes entries to
shards inside tests/_r.py - no test files changed. Regens now show up
in git as per-function diffs instead of a whole-file rewrite, and
unchanged shards are byte-identical on rewrite.
- Cache contents: canonical values preserved for every pre-existing key.
A full live regen against the vendored R NNS 13.0 reproduced 1929 of
2385 entries exactly; the 456 that differed are known borderline
partition values that vary by R build environment (the committed canon
came from the June 13 cloud regen; this machine's build produces a
different deterministic variant - same phenomenon that June commit
itself recorded vs its predecessor). Canonical values were restored for
all 456; only the 720 new Cholesky-input keys carry fresh live-R values,
and those functions verified environment-stable (live values match
Python exactly).
- scripts/regenerate_r_cache.py, run_live_r_parity_for_changed_api.py,
sync manifests, docs, and .gitignore updated for the directory layout.
Full suite cache-only: 2203 passed (the 2 practical-example failures are
the known local-build items, green in CI).
Collaborator
Author
|
Parking this: keeping the single-file cache and the numpy<2.5 pin for now; the deterministic-inputs + canonical-environment work will be folded into the NNS 13.1 cache regeneration, where the key churn is free anyway. The patch stays recoverable from this PR's commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out the numpy 2.5.1 incident at the root instead of the freeze. Three changes, one story:
1. Deterministic test inputs (the cure)
test_co_momentsgenerated correlated inputs withrng.multivariate_normal— LAPACK SVD, outside NumPy's stream-stability guarantee, so the sha256 cache keys hashed from those inputs depended on the BLAS build and runner CPU. Inputs are now twostandard_normaldraws correlated with the closed-form 2×2 Cholesky factor (y = ρx + √(1−ρ²)z): exactly-rounded IEEE arithmetic, bit-identical on every platform. A repo-wide audit found no other LAPACK-dependent test inputs.The numpy
<2.5pin is removed from all four workflows — this PR's CI passing on unpinned numpy across the runner fleet is the empirical proof the fix works.2. Per-function cache shards
tests/_r_cache.json(22 MB blob) →tests/_r_cache/with 59 shards, one per R function (NNS.reg.json,Co.LPM.json, …). Keys gain a function prefix (NNS.reg:<sha256>, digest unchanged) that routes entries to shards insidetests/_r.py— zero test files changed. Future regens appear in git as reviewable per-function diffs; unchanged shards rewrite byte-identically (no churn). Cache-miss errors now name the function and shard instead of a bare hash.3. Cache provenance finding (documented for the 13.1 regen)
A full live regen against the vendored R NNS 13.0 on this machine reproduced 1,929/2,385 entries exactly. The 456 that differed are borderline partition values that vary by R build environment: the committed canon came from the June 13 cloud-session regen, and that June commit itself recorded the same 456-key flip vs its predecessor. R is fully deterministic per-environment (verified across runs and thread counts) — the variance is compiler/toolchain-level, the same fragility class as the numpy/OpenBLAS incident but in the Rcpp layer.
Canonical values are preserved for every pre-existing key; only the 720 new Cholesky-input keys carry fresh live-R values, and those functions verified environment-stable (live R matched Python exactly). Recommendation before the 13.1 regen: pin the canonical regen environment with a container (e.g.
rocker/r-ver+ date-pinned CRAN snapshot) so any machine reproduces the canon byte-for-byte — and this is worth surfacing as an upstream observation about NNS's partition boundaries.Verification
mypy --strict,ruffclean; regen script validates all 59 shards (2,385 entries, NNS 13.0).