Repository navigation
feat: Add Vitest bench suite + CPU profile script - #79
Merged
Merged
Conversation
Iteration-time performance tooling — the second of the three concerns from the perf-architecture discussion. Answers "is this change actually faster?" via Vitest's built-in `bench()` mode (Tinybench under the hood), and provides a one-command CPU profiler wrapper for finding hotspots. ## What's new **Four `.bench.ts` files** in `src/__benchmarks__/`, one per fixture schema. Each measures ops/sec for `mock(schema)` via `bench()`. Baseline numbers at v1.5.4 (Node 26, AMD Ryzen 5): | Schema | ops/sec | RME | | -------------------------------------------- | ------: | -----: | | `mock(messageLikeSchema)` recursive-lazy | 3,188 | ±3.68% | | `mock(applicationLikeSchema)` wide-flat | 13,147 | ±2.28% | | `mock(channelLikeSchema)` intersect+variant | 27,419 | ±2.47% | | `mock(interactionLikeSchema)` multi-lazy | 25,933 | ±2.39% | These are within-process ratios — absolute numbers vary by hardware, but ratios between two runs in the same session are stable for comparing baseline vs feature-branch perf. **Vitest discovery configured** via a new `test.benchmark` block in `vite.config.ts` so `.bench.ts` files are picked up by `vp test bench` without polluting the default `vp test` run. The bench suite is explicitly opt-in — it doesn't gate CI. **`scripts/profile.mjs` convenience wrapper** runs the bench suite under `node --cpu-prof` and drops the resulting `.cpuprofile` in `./profiles/` (gitignored). Prints instructions for opening it in Chrome DevTools. Invoke via `vp run profile`. **`src/__benchmarks__/README.md`** documents how to use the suite, how to interpret ops/sec and RME, and how to add new benchmarks. The "5% threshold" guidance there is the practical rule for distinguishing real regressions from noise. ## What's NOT in this PR This is the iteration-time perf tooling. The catastrophe-detection canary (asserting call-count ceilings on the same fixtures) shipped in #77 — they share the fixture library deliberately. CPU profile output viewing is left as a manual step (open in Chrome DevTools). A separate flame-graph generator could be wired in later via the npm script if anyone wants the on-CLI variant; for now the manual workflow is what the original perf report used. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Review or Edit in CodeSandboxOpen the branch in Web Editor • VS Code • Insiders |
Contributor
|
The changes in this PR will be included in the next version bump.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Closes concerns #2 and #3 from the perf-architecture discussion — iteration-time tooling. PR #77 shipped the regression canary (concern #1: catastrophe detection). This PR ships the contributor-facing tools for evaluating perf changes and finding hotspots.
__tests__/perfRegression.spec.ts__benchmarks__/*.bench.tsvp test bench, opt-inscripts/profile.mjsvp run profile, opt-inAll three reuse the fixture library from #77, so adding a new realistic-shape pattern lights up in all three places at once.
What's new
Vitest
bench()suite undersrc/__benchmarks__/Four
.bench.tsfiles, one per fixture schema. Each measuresops/secformock(schema)using Vitest's built-in bench mode (Tinybench under the hood). Baseline numbers at v1.5.4 on the dev machine:mock(messageLikeSchema)recursive-lazymock(applicationLikeSchema)wide-flatmock(channelLikeSchema)intersect+variantmock(interactionLikeSchema)multi-lazy unionAbsolute numbers vary by hardware; the value is the within-process ratio when comparing a baseline branch to a feature branch. RME stays under 4% on all four cases, comfortably below the README's 5% "real regression vs. noise" threshold.
Vitest discovery
vite.config.tsgains atest.benchmarkblock pointing atsrc/__benchmarks__/**/*.bench.{ts,tsx}. Bench files are picked up byvp test benchonly — they don't pollute the defaultvp testrun, so the suite stays opt-in and doesn't gate CI.scripts/profile.mjsCPU profile wrapperRuns the bench suite under
node --cpu-prof, drops the resulting.cpuprofilein./profiles/(gitignored), and prints instructions for opening it in Chrome DevTools. Invoke viayarn profileorvp run profile. Optional QoL — the bench output alone is usually enough; profiling is for "find the next hotspot" investigations like the one that produced the original perf report.Documentation
src/__benchmarks__/README.mdcovers:vp test bench, file-filter syntax)ops/secandRMEPackage scripts
Both are convenience aliases —
vp test benchandnode scripts/profile.mjsstill work directly.What's NOT in this PR
No CI integration to post bench results as PR comments. Researched several options (
benchmark-action/github-action-benchmark, CodSpeed, openpgpjs fork, DIY). Vitest's bench mode is still officially experimental (the runner emits a warning on every invocation) and doesn't have a stable JSON reporter, so any CI integration would require an adapter script that may need a rewrite when Vitest 5 lands its API redesign. Decided to defer — the bench suite is fully useful as a local-iteration tool; PR-comment automation is the nice-to-have on top.The catastrophe canary in #77 already covers the only regressions that truly need to block at PR time. Anything subtler than catastrophic is well-served by local bench runs during a perf PR's iteration.
Verification
vp check --fixclean across 72 filesBump
patch— pure tooling, no public-API or runtime changes. Bumpy file included.🤖 Generated with Claude Code