Skip to content

feat: Add Vitest bench suite + CPU profile script - #79

Merged
Saeris merged 1 commit into
mainfrom
perf/vitest-bench-suite
Jun 3, 2026
Merged

Saeris merged 1 commit into
mainfrom
perf/vitest-bench-suite

Conversation

@Saeris

@Saeris Saeris commented Jun 3, 2026

Copy link
Copy Markdown
Owner

Summary

Closes concerns #2 and #3 from the perf-architecture discussion — iteration-time tooling. PR #77 shipped the regression canary (concern #1: catastrophe detection). This PR ships the contributor-facing tools for evaluating perf changes and finding hotspots.

Concern Question it answers Where it lives When it runs
#1 (shipped in #77) Did anything catastrophic break? __tests__/perfRegression.spec.ts Every CI run
#2 (this PR) Is this change actually faster? __benchmarks__/*.bench.ts vp test bench, opt-in
#3 (this PR) Where should I look to make this faster? scripts/profile.mjs vp run profile, opt-in

All three reuse the fixture library from #77, so adding a new realistic-shape pattern lights up in all three places at once.

What's new

Vitest bench() suite under src/__benchmarks__/

Four .bench.ts files, one per fixture schema. Each measures ops/sec for mock(schema) using Vitest's built-in bench mode (Tinybench under the hood). Baseline numbers at v1.5.4 on the dev machine:

Schema ops/sec RME
mock(messageLikeSchema) recursive-lazy 3,188 ±3.68%
mock(applicationLikeSchema) wide-flat 13,147 ±2.28%
mock(channelLikeSchema) intersect+variant 27,419 ±2.47%
mock(interactionLikeSchema) multi-lazy union 25,933 ±2.39%

Absolute numbers vary by hardware; the value is the within-process ratio when comparing a baseline branch to a feature branch. RME stays under 4% on all four cases, comfortably below the README's 5% "real regression vs. noise" threshold.

Vitest discovery

vite.config.ts gains a test.benchmark block pointing at src/__benchmarks__/**/*.bench.{ts,tsx}. Bench files are picked up by vp test bench only — they don't pollute the default vp test run, so the suite stays opt-in and doesn't gate CI.

scripts/profile.mjs CPU profile wrapper

Runs the bench suite under node --cpu-prof, drops the resulting .cpuprofile in ./profiles/ (gitignored), and prints instructions for opening it in Chrome DevTools. Invoke via yarn profile or vp run profile. Optional QoL — the bench output alone is usually enough; profiling is for "find the next hotspot" investigations like the one that produced the original perf report.

Documentation

src/__benchmarks__/README.md covers:

  • How to run (vp test bench, file-filter syntax)
  • How to interpret ops/sec and RME
  • The "5% threshold" rule for distinguishing real regressions from noise
  • How to add a new benchmark
  • How to drop into the CPU profiler for hotspot discovery

Package scripts

"bench": "vp test bench --run",
"profile": "node scripts/profile.mjs"

Both are convenience aliases — vp test bench and node scripts/profile.mjs still work directly.

What's NOT in this PR

No CI integration to post bench results as PR comments. Researched several options (benchmark-action/github-action-benchmark, CodSpeed, openpgpjs fork, DIY). Vitest's bench mode is still officially experimental (the runner emits a warning on every invocation) and doesn't have a stable JSON reporter, so any CI integration would require an adapter script that may need a rewrite when Vitest 5 lands its API redesign. Decided to defer — the bench suite is fully useful as a local-iteration tool; PR-comment automation is the nice-to-have on top.

The catastrophe canary in #77 already covers the only regressions that truly need to block at PR time. Anything subtler than catastrophic is well-served by local bench runs during a perf PR's iteration.

Verification

  • ✅ vp check --fix clean across 72 files
  • ✅ Bench suite executes end-to-end with all 4 files reporting stable results
  • ✅ Existing tests unaffected (bench discovery is isolated to its own glob)
  • ⏳ CI on this branch

Bump

patch — pure tooling, no public-API or runtime changes. Bumpy file included.

🤖 Generated with Claude Code

Iteration-time performance tooling — the second of the three concerns
from the perf-architecture discussion. Answers "is this change actually
faster?" via Vitest's built-in `bench()` mode (Tinybench under the
hood), and provides a one-command CPU profiler wrapper for finding
hotspots.

## What's new

**Four `.bench.ts` files** in `src/__benchmarks__/`, one per fixture
schema. Each measures ops/sec for `mock(schema)` via `bench()`. Baseline
numbers at v1.5.4 (Node 26, AMD Ryzen 5):

| Schema                                       | ops/sec | RME    |
| -------------------------------------------- | ------: | -----: |
| `mock(messageLikeSchema)` recursive-lazy     |   3,188 | ±3.68% |
| `mock(applicationLikeSchema)` wide-flat      |  13,147 | ±2.28% |
| `mock(channelLikeSchema)` intersect+variant  |  27,419 | ±2.47% |
| `mock(interactionLikeSchema)` multi-lazy     |  25,933 | ±2.39% |

These are within-process ratios — absolute numbers vary by hardware,
but ratios between two runs in the same session are stable for
comparing baseline vs feature-branch perf.

**Vitest discovery configured** via a new `test.benchmark` block in
`vite.config.ts` so `.bench.ts` files are picked up by `vp test bench`
without polluting the default `vp test` run. The bench suite is
explicitly opt-in — it doesn't gate CI.

**`scripts/profile.mjs` convenience wrapper** runs the bench suite
under `node --cpu-prof` and drops the resulting `.cpuprofile` in
`./profiles/` (gitignored). Prints instructions for opening it in
Chrome DevTools. Invoke via `vp run profile`.

**`src/__benchmarks__/README.md`** documents how to use the suite,
how to interpret ops/sec and RME, and how to add new benchmarks. The
"5% threshold" guidance there is the practical rule for distinguishing
real regressions from noise.

## What's NOT in this PR

This is the iteration-time perf tooling. The catastrophe-detection
canary (asserting call-count ceilings on the same fixtures) shipped
in #77 — they share the fixture library deliberately.

CPU profile output viewing is left as a manual step (open in Chrome
DevTools). A separate flame-graph generator could be wired in later
via the npm script if anyone wants the on-CLI variant; for now the
manual workflow is what the original perf report used.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@codesandbox

codesandbox Bot commented Jun 3, 2026

Copy link
Copy Markdown

Review or Edit in CodeSandbox

Open the branch in Web Editor • VS Code • Insiders

Open Preview

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

bumpy-frog

The changes in this PR will be included in the next version bump.

patch Patch releases

  • valimock 1.5.4 → 1.5.5

Bump files in this PR

Click here if you want to add another bump file to this PR


This comment is maintained by bumpy.

@Saeris
Saeris merged commit 23a85be into main Jun 3, 2026
4 checks passed
@Saeris
Saeris deleted the perf/vitest-bench-suite branch June 3, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant