CI: pin numpy<2.5 to keep R parity cache keys stable - #77
Merged
Conversation
numpy 2.5.1 (released today) bundles an OpenBLAS whose SVD kernels differ on some GitHub runner CPUs. tests/parity/test_co_moments.py generates inputs with rng.multivariate_normal (LAPACK SVD, outside numpy's random-stream stability guarantee), and tests/_r.py hashes those inputs into R cache keys, so a last-ulp difference on an unlucky runner produces mass cache-miss failures (120 on the main run for the PR #76 merge; runner-dependent, not caused by that PR - the same commit content passed 20 minutes earlier on other runners). Pin the four workflows that install numpy unpinned to <2.5, matching the cache-era 2.4.x used by the committed uv.lock. Durable fix to schedule with the next live-R cache regeneration: generate correlated test inputs with an explicit 2x2 Cholesky over standard_normal draws (bit-stable pure arithmetic) instead of multivariate_normal.
Collaborator
Author
|
Proof of nondeterminism, for the record: re-ran the failed 3.12/3.13 jobs on the main run (28716242364) with zero changes — same commit, same unpinned workflow, same numpy 2.5.1 — and both now pass on the runners the rerun happened to get. Red → green with no code change confirms the per-runner OpenBLAS kernel lottery. Main is green again from that rerun, but without this pin every future run keeps rolling the dice — a coin-flip red on any sync PR, release gate, or contributor PR until merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the red main CI on the #76 merge commit (120 parity failures on Python 3.12/3.13 — run 28716242364). Not caused by #76.
Diagnosis
tests/parity/test_co_moments.pygenerates correlated inputs withrng.multivariate_normal, which routes through LAPACK SVD — outside numpy's random-stream bit-stability guarantee.tests/_r.pysha256-hashes those inputs into R cache keys, so any last-ulp difference means a cache miss, andNNS_R_CACHE_ONLY=1turns that into a failure.Change
Pin
numpy<2.5in the four workflows that install it unpinned (native-backend-ci, parity-autofix, inspect-r-api-update, sync-from-nns-core), matching the 2.4.x line the committed cache anduv.lockwere built against.Durable fix (deferred to the next live-R cache regeneration)
Replace
multivariate_normalin the parity tests with an explicit 2×2 Cholesky overstandard_normaldraws — pure IEEE arithmetic, bit-stable on every platform — and regenerate the affected cache entries once. Bundling that with the 13.1 regen avoids touching the current cache.