Skip to content

Bit-stable parity-cache inputs; shard the R cache per function; drop the numpy pin - #79

Closed
gitRasheed wants to merge 1 commit into
mainfrom
deterministic-r-cache
Closed

Bit-stable parity-cache inputs; shard the R cache per function; drop the numpy pin#79
gitRasheed wants to merge 1 commit into
mainfrom
deterministic-r-cache

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Closes out the numpy 2.5.1 incident at the root instead of the freeze. Three changes, one story:

1. Deterministic test inputs (the cure)

test_co_moments generated correlated inputs with rng.multivariate_normal — LAPACK SVD, outside NumPy's stream-stability guarantee, so the sha256 cache keys hashed from those inputs depended on the BLAS build and runner CPU. Inputs are now two standard_normal draws correlated with the closed-form 2×2 Cholesky factor (y = ρx + √(1−ρ²)z): exactly-rounded IEEE arithmetic, bit-identical on every platform. A repo-wide audit found no other LAPACK-dependent test inputs.

The numpy <2.5 pin is removed from all four workflows — this PR's CI passing on unpinned numpy across the runner fleet is the empirical proof the fix works.

2. Per-function cache shards

tests/_r_cache.json (22 MB blob) → tests/_r_cache/ with 59 shards, one per R function (NNS.reg.json, Co.LPM.json, …). Keys gain a function prefix (NNS.reg:<sha256>, digest unchanged) that routes entries to shards inside tests/_r.pyzero test files changed. Future regens appear in git as reviewable per-function diffs; unchanged shards rewrite byte-identically (no churn). Cache-miss errors now name the function and shard instead of a bare hash.

3. Cache provenance finding (documented for the 13.1 regen)

A full live regen against the vendored R NNS 13.0 on this machine reproduced 1,929/2,385 entries exactly. The 456 that differed are borderline partition values that vary by R build environment: the committed canon came from the June 13 cloud-session regen, and that June commit itself recorded the same 456-key flip vs its predecessor. R is fully deterministic per-environment (verified across runs and thread counts) — the variance is compiler/toolchain-level, the same fragility class as the numpy/OpenBLAS incident but in the Rcpp layer.

Canonical values are preserved for every pre-existing key; only the 720 new Cholesky-input keys carry fresh live-R values, and those functions verified environment-stable (live R matched Python exactly). Recommendation before the 13.1 regen: pin the canonical regen environment with a container (e.g. rocker/r-ver + date-pinned CRAN snapshot) so any machine reproduces the canon byte-for-byte — and this is worth surfacing as an upstream observation about NNS's partition boundaries.

Verification

  • Full suite cache-only: 2,203 passed (2 practical-example failures are the known local-build items, green in CI — one of the previous three healed).
  • mypy --strict, ruff clean; regen script validates all 59 shards (2,385 entries, NNS 13.0).

Root cause chain from the numpy 2.5.1 CI incident, fixed at the source:

- test_co_moments generated correlated inputs with rng.multivariate_normal,
  which routes through LAPACK SVD - outside NumPy's stream-stability
  guarantee, so cache keys hashed from those inputs depended on the BLAS
  build and CPU. Inputs now come from two standard_normal draws correlated
  by the closed-form 2x2 Cholesky factor: exactly rounded IEEE arithmetic,
  bit-identical everywhere. The numpy<2.5 CI pin is removed; this PR's CI
  passing on unpinned numpy across the runner fleet is the proof.

- tests/_r_cache.json (22 MB single blob) is now tests/_r_cache/ with one
  shard per R function (59 files). Cache keys gain a function-name prefix
  ('NNS.reg:<sha256>'; digest part unchanged) which routes entries to
  shards inside tests/_r.py - no test files changed. Regens now show up
  in git as per-function diffs instead of a whole-file rewrite, and
  unchanged shards are byte-identical on rewrite.

- Cache contents: canonical values preserved for every pre-existing key.
  A full live regen against the vendored R NNS 13.0 reproduced 1929 of
  2385 entries exactly; the 456 that differed are known borderline
  partition values that vary by R build environment (the committed canon
  came from the June 13 cloud regen; this machine's build produces a
  different deterministic variant - same phenomenon that June commit
  itself recorded vs its predecessor). Canonical values were restored for
  all 456; only the 720 new Cholesky-input keys carry fresh live-R values,
  and those functions verified environment-stable (live values match
  Python exactly).

- scripts/regenerate_r_cache.py, run_live_r_parity_for_changed_api.py,
  sync manifests, docs, and .gitignore updated for the directory layout.

Full suite cache-only: 2203 passed (the 2 practical-example failures are
the known local-build items, green in CI).
@gitRasheed

Copy link
Copy Markdown
Collaborator Author

Parking this: keeping the single-file cache and the numpy<2.5 pin for now; the deterministic-inputs + canonical-environment work will be folded into the NNS 13.1 cache regeneration, where the key churn is free anyway. The patch stays recoverable from this PR's commits.

@gitRasheed gitRasheed closed this Jul 4, 2026
@gitRasheed
gitRasheed deleted the deterministic-r-cache branch July 4, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant