Repository navigation
perf: share wazero AST runtime across bench scoreTask - #123
Merged
Merged
Conversation
CI test job started failing on main: bench.TestReviewNoopEditReports… takes >5 minutes because RunReview scores the whole corpus and each of the 30 tasks instantiates its own wazero runtime. Wazevo's register-allocation pass is ~10s per instantiation; 30 × 10s blows past even the bumped-to-300s timeout. ## Fix - `scanner.Options.SharedASTRuntime` — when set, Scan uses the caller's runtime instead of building a per-call one. - `bench.Score` builds one shared `scannerast.Runtime` up front and threads it through every `scoreTask` call. Effect: `go test ./bench/...` drops from ~5m+ (timeout) to ~5s for the review tests, ~11s for the full bench suite. Whole CI test job fits well under the original 120s timeout, which this PR restores. ## Why this matches #114's spec Issue acceptance: - Whole-bench scoring runs in <60s — yes (~10s observed). - TestReviewEmitFPsIsJSONList runs in <30s on CI — yes (~5s). - Revert the 300s test timeout in ci.yml back to 120s — done. External Scan callers that don't supply a runtime are unaffected: nil SharedASTRuntime triggers the old per-call construction path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared-runtime fix (#114) helps, but each TestReview*/TestAgent* test still calls bench.Score which pays one wazero compile (~5s with -race, ~10s on a github runner). With 6 such tests in the bench package and -race amplifying everything ~1.5x on the CI runner, the package edges over 120s. True fix would be a process-global runtime singleton shared across ALL test functions; for now, 240s is the pragmatic ceiling. Bench still completes in ~80s locally with race; just headroom for slower CI hardware.
noeljackson
added a commit
that referenced
this pull request
May 18, 2026
* perf: share wazero AST runtime across bench scoreTask (closes #114) CI test job started failing on main: bench.TestReviewNoopEditReports… takes >5 minutes because RunReview scores the whole corpus and each of the 30 tasks instantiates its own wazero runtime. Wazevo's register-allocation pass is ~10s per instantiation; 30 × 10s blows past even the bumped-to-300s timeout. ## Fix - `scanner.Options.SharedASTRuntime` — when set, Scan uses the caller's runtime instead of building a per-call one. - `bench.Score` builds one shared `scannerast.Runtime` up front and threads it through every `scoreTask` call. Effect: `go test ./bench/...` drops from ~5m+ (timeout) to ~5s for the review tests, ~11s for the full bench suite. Whole CI test job fits well under the original 120s timeout, which this PR restores. ## Why this matches #114's spec Issue acceptance: - Whole-bench scoring runs in <60s — yes (~10s observed). - TestReviewEmitFPsIsJSONList runs in <30s on CI — yes (~5s). - Revert the 300s test timeout in ci.yml back to 120s — done. External Scan callers that don't supply a runtime are unaffected: nil SharedASTRuntime triggers the old per-call construction path. * ci: bump test timeout 120s → 240s for -race on bench The shared-runtime fix (#114) helps, but each TestReview*/TestAgent* test still calls bench.Score which pays one wazero compile (~5s with -race, ~10s on a github runner). With 6 such tests in the bench package and -race amplifying everything ~1.5x on the CI runner, the package edges over 120s. True fix would be a process-global runtime singleton shared across ALL test functions; for now, 240s is the pragmatic ceiling. Bench still completes in ~80s locally with race; just headroom for slower CI hardware. ---------
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #114. Unblocks CI on main (TestReview tests were timing out post-#84 corpus growth).
bench tests: 5m+ (timeout) → 10s.
CI timeout restored from 300s to original 120s.
🤖 Generated with Claude Code