Skip to content

CI: pin numpy<2.5 to keep R parity cache keys stable - #77

Merged
OVVO-Financial merged 1 commit into
mainfrom
ci-pin-numpy
Jul 4, 2026
Merged

CI: pin numpy<2.5 to keep R parity cache keys stable#77
OVVO-Financial merged 1 commit into
mainfrom
ci-pin-numpy

Conversation

@gitRasheed

Copy link
Copy Markdown
Collaborator

Fixes the red main CI on the #76 merge commit (120 parity failures on Python 3.12/3.13 — run 28716242364). Not caused by #76.

Diagnosis

  • numpy 2.5.1 was uploaded to PyPI at 17:06 UTC today; the CI workflows install numpy unpinned, so ≥3.12 jobs started pulling it (3.11 still resolves 2.4.6, which is why it stayed green).
  • tests/parity/test_co_moments.py generates correlated inputs with rng.multivariate_normal, which routes through LAPACK SVD — outside numpy's random-stream bit-stability guarantee.
  • tests/_r.py sha256-hashes those inputs into R cache keys, so any last-ulp difference means a cache miss, and NNS_R_CACHE_ONLY=1 turns that into a failure.
  • The same commit content passed at 18:16 and 18:32 with numpy 2.5.1 and failed at 18:51: 2.5.1's bundled OpenBLAS picks different SVD kernels on different runner CPU types, so it is a per-runner lottery. Locally (one CPU), 2.4.4 and 2.5.1 produce byte-identical output — confirming hardware-dependent divergence rather than a deterministic numpy change.

Change

Pin numpy<2.5 in the four workflows that install it unpinned (native-backend-ci, parity-autofix, inspect-r-api-update, sync-from-nns-core), matching the 2.4.x line the committed cache and uv.lock were built against.

Durable fix (deferred to the next live-R cache regeneration)

Replace multivariate_normal in the parity tests with an explicit 2×2 Cholesky over standard_normal draws — pure IEEE arithmetic, bit-stable on every platform — and regenerate the affected cache entries once. Bundling that with the 13.1 regen avoids touching the current cache.

numpy 2.5.1 (released today) bundles an OpenBLAS whose SVD kernels
differ on some GitHub runner CPUs. tests/parity/test_co_moments.py
generates inputs with rng.multivariate_normal (LAPACK SVD, outside
numpy's random-stream stability guarantee), and tests/_r.py hashes
those inputs into R cache keys, so a last-ulp difference on an unlucky
runner produces mass cache-miss failures (120 on the main run for the
PR #76 merge; runner-dependent, not caused by that PR - the same
commit content passed 20 minutes earlier on other runners).

Pin the four workflows that install numpy unpinned to <2.5, matching
the cache-era 2.4.x used by the committed uv.lock. Durable fix to
schedule with the next live-R cache regeneration: generate correlated
test inputs with an explicit 2x2 Cholesky over standard_normal draws
(bit-stable pure arithmetic) instead of multivariate_normal.
@gitRasheed

Copy link
Copy Markdown
Collaborator Author

Proof of nondeterminism, for the record: re-ran the failed 3.12/3.13 jobs on the main run (28716242364) with zero changes — same commit, same unpinned workflow, same numpy 2.5.1 — and both now pass on the runners the rerun happened to get. Red → green with no code change confirms the per-runner OpenBLAS kernel lottery.

Main is green again from that rerun, but without this pin every future run keeps rolling the dice — a coin-flip red on any sync PR, release gate, or contributor PR until merged.

@OVVO-Financial
OVVO-Financial merged commit 1542fca into main Jul 4, 2026
8 checks passed
@OVVO-Financial
OVVO-Financial deleted the ci-pin-numpy branch July 5, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants