Skip to content

perf: share wazero AST runtime across bench scoreTask - #123

Merged
noeljackson merged 2 commits into
mainfrom
perf/wazero-shared-runtime
May 15, 2026
Merged

noeljackson merged 2 commits into
mainfrom
perf/wazero-shared-runtime

Conversation

@noeljackson

Copy link
Copy Markdown
Owner

Closes #114. Unblocks CI on main (TestReview tests were timing out post-#84 corpus growth).

bench tests: 5m+ (timeout) → 10s.
CI timeout restored from 300s to original 120s.

🤖 Generated with Claude Code

noeljackson and others added 2 commits May 15, 2026 15:45
CI test job started failing on main: bench.TestReviewNoopEditReports…
takes >5 minutes because RunReview scores the whole corpus and each
of the 30 tasks instantiates its own wazero runtime. Wazevo's
register-allocation pass is ~10s per instantiation; 30 × 10s blows
past even the bumped-to-300s timeout.

## Fix

- `scanner.Options.SharedASTRuntime` — when set, Scan uses the
  caller's runtime instead of building a per-call one.
- `bench.Score` builds one shared `scannerast.Runtime` up front and
  threads it through every `scoreTask` call.

Effect: `go test ./bench/...` drops from ~5m+ (timeout) to ~5s for
the review tests, ~11s for the full bench suite. Whole CI test job
fits well under the original 120s timeout, which this PR restores.

## Why this matches #114's spec

Issue acceptance:
- Whole-bench scoring runs in <60s — yes (~10s observed).
- TestReviewEmitFPsIsJSONList runs in <30s on CI — yes (~5s).
- Revert the 300s test timeout in ci.yml back to 120s — done.

External Scan callers that don't supply a runtime are unaffected:
nil SharedASTRuntime triggers the old per-call construction path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared-runtime fix (#114) helps, but each TestReview*/TestAgent*
test still calls bench.Score which pays one wazero compile (~5s with
-race, ~10s on a github runner). With 6 such tests in the bench
package and -race amplifying everything ~1.5x on the CI runner, the
package edges over 120s.

True fix would be a process-global runtime singleton shared across
ALL test functions; for now, 240s is the pragmatic ceiling. Bench
still completes in ~80s locally with race; just headroom for slower
CI hardware.
@noeljackson
noeljackson merged this pull request into main May 15, 2026
3 checks passed
@noeljackson
noeljackson deleted the perf/wazero-shared-runtime branch May 15, 2026 14:00
noeljackson added a commit that referenced this pull request May 18, 2026
* perf: share wazero AST runtime across bench scoreTask (closes #114)

CI test job started failing on main: bench.TestReviewNoopEditReports…
takes >5 minutes because RunReview scores the whole corpus and each
of the 30 tasks instantiates its own wazero runtime. Wazevo's
register-allocation pass is ~10s per instantiation; 30 × 10s blows
past even the bumped-to-300s timeout.

## Fix

- `scanner.Options.SharedASTRuntime` — when set, Scan uses the
  caller's runtime instead of building a per-call one.
- `bench.Score` builds one shared `scannerast.Runtime` up front and
  threads it through every `scoreTask` call.

Effect: `go test ./bench/...` drops from ~5m+ (timeout) to ~5s for
the review tests, ~11s for the full bench suite. Whole CI test job
fits well under the original 120s timeout, which this PR restores.

## Why this matches #114's spec

Issue acceptance:
- Whole-bench scoring runs in <60s — yes (~10s observed).
- TestReviewEmitFPsIsJSONList runs in <30s on CI — yes (~5s).
- Revert the 300s test timeout in ci.yml back to 120s — done.

External Scan callers that don't supply a runtime are unaffected:
nil SharedASTRuntime triggers the old per-call construction path.

* ci: bump test timeout 120s → 240s for -race on bench

The shared-runtime fix (#114) helps, but each TestReview*/TestAgent*
test still calls bench.Score which pays one wazero compile (~5s with
-race, ~10s on a github runner). With 6 such tests in the bench
package and -race amplifying everything ~1.5x on the CI runner, the
package edges over 120s.

True fix would be a process-global runtime singleton shared across
ALL test functions; for now, 240s is the pragmatic ceiling. Bench
still completes in ~80s locally with race; just headroom for slower
CI hardware.

---------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: cache wazero AST runtime across bench scoreTask calls

1 participant