Replace hardcoded Lanczos eigenvalue goldens with property-based checks#3086
Open
says1117 wants to merge 2 commits into
Open
Replace hardcoded Lanczos eigenvalue goldens with property-based checks#3086says1117 wants to merge 2 commits into
says1117 wants to merge 2 commits into
Conversation
Golden-value comparisons via devArrMatch/CompareApprox were flaky across CUDA versions/architectures because FP reduction order isn't deterministic. Replace with residual, orthogonality, unit-norm, and ordering checks that verify the defining properties of an eigenpair instead of exact values. Fixes NVIDIA#2519
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughChangesLanczos validation
Estimated code review effort: 4 (Complex) | ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp/tests/sparse/solver/lanczos.cu`:
- Around line 79-87: Synchronize the RAFT stream before every host read of
results produced by host-scalar `raft::linalg::dot` in `compute_frobenius_norm`
and the affected Lanczos calculations involving `rayleigh`, `residual_sq`,
`v_norm_sq`, and `dot_ij`. Use the RAFT stream synchronization helper, and
replace the existing raw `cudaStreamSynchronize` with that helper.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 74ae28a5-be53-450a-8676-c9ef4fdfbe94
📒 Files selected for processing (1)
cpp/tests/sparse/solver/lanczos.cu
Author
|
Test files aren't in Doxygen scope per cpp/REVIEW_GUIDELINES.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2519.
The problem
The Lanczos gtests checked computed eigenvalues against hardcoded arrays (
devArrMatch+CompareApprox(1e-5)). That's fragile: floating-point addition isn't associative, so cuSPARSE/cuBLAS can sum things in a different order depending on CUDA version and GPU architecture, and the last few digits of an eigenvalue shift. #3021 hit this on A100/ARM64 —0.01367824vs0.01369178, a difference just outside the tolerance, even though the solve was correct.The fix
Instead of comparing against a fixed answer, check that what came back is actually a valid eigenpair:
||A*v - λv||should be tiny. This is the same thing the solver's owntoleranceconfig is defined to converge on, so it's not an arbitrary check - it's the solver's own success criterion.λ ≈ (vᵀAv)/(vᵀv). Added this because the residual alone gets loose on matrices with a large norm, and can miss a small eigenvalue being slightly wrong. The Rayleigh quotient pins it down directly.rmat_lanczos_testsalready had the matrix-vector-multiply helper needed for the residual check sitting there unused; I just started using it.lanczos_testsneeded it added. Eigenvectors were already being computed by every test, just never checked - no solver changes needed.Left the reproducibility check alone (it runs the solve twice with the same seed and diffs the results exactly - that's a determinism check, not a golden-value comparison, so it isn't affected by the cross-platform issue).
Picking the tolerance
I didn't want to just guess a number, so I instrumented the checks, ran all 11 test fixtures, and looked at what the residuals actually came out to:
Two things stood out. First, the residual scales with
‖A‖ · machine epsilon, not with the matrix sizen- the small fixtures (n=100) and the big RMAT one (n=4096) land in the same range. That mattered a lot (see below). Second, orthogonality and unit-norm error are both just small multiples of epsilon.So the tolerances ended up being:
```
matrix_tol = 500 × max(‖A‖, 1) × ε — residual, Rayleigh, ordering
unit_tol = 1000 × ε — unit norm, orthogonality
```
Every measured value sits at 0.04%–3.8% of its tolerance - comfortable margin for cross-platform noise, without being so loose the check stops meaning anything.
Making sure I didn't just weaken the test
I proved these checks can actually fail, by corrupting a solver's output on purpose and confirming the test catches it:
Worth being honest about: my first attempt at this got it wrong. I'd included an
nfactor in the residual tolerance, and it turned outRmatLanczosTestFstill passed even with every eigenvalue scaled by 1% - a real regression from the old test, which would've caught that instantly. Digging into the data above is what showed the residual doesn't actually scale withn, and once I fixed that (and added the Rayleigh check), the same corruption test failed cleanly across the board, including on the hardest case - the smallest eigenvalue in the RMAT set, where a 1% error is only 0.08 in absolute terms.Verification
Built and ran on an RTX 2060 SUPER (sm_75), CUDA 13.3, WSL2:
```
$ cmake --build cpp/build -j2 --target SOLVERS_TEST
[2/2] Linking CXX executable gtests/SOLVERS_TEST
$ ./cpp/build/gtests/SOLVERS_TEST --gtest_filter='Lanczos'
[==========] 11 tests from 11 test suites ran. (162255 ms total)
[ PASSED ] 11 tests.
```
pre-commitpasses clean.For reviewers
Refs #2519, #3021.