Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .bumpy/perf-bench-suite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
valimock: patch
---

Add Vitest `bench()` files for iteration-time performance evaluation, reusing the fixture library from the regression canary. Includes a `scripts/profile.mjs` convenience wrapper for CPU profiling. Pure tooling — no public-API or runtime changes.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,7 @@ cypress/e2e/build

# Claude Code local agent state
.claude/

# CPU profiles emitted by scripts/profile.mjs
profiles/
*.cpuprofile
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@
"check": "vp check",
"format": "vp check --fix",
"test": "vp test --fileParallelism",
"typecheck": "vp typecheck"
"typecheck": "vp typecheck",
"bench": "vp test bench --run",
"profile": "node scripts/profile.mjs"
},
"dependencies": {
"randexp": "^0.5.3"
Expand Down
70 changes: 70 additions & 0 deletions scripts/profile.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#!/usr/bin/env node
/**
* Run the Valimock benchmark suite under node's V8 CPU profiler and drop a
* `.cpuprofile` file in `./profiles/` for analysis in Chrome DevTools.
*
* Usage:
* vp run profile # all benchmarks
* vp run profile -- messageLikeSchema # filter to one bench file
*
* Open the resulting file in Chrome:
* 1. devtools → ⋮ menu → More tools → Performance insights
* 2. drag the .cpuprofile file into the panel
*
* The Self Time column surfaces hotspots; ranked self-time is what surfaced
* `findFakerForKeyName` as the top regression cause in the original perf
* report.
*/
import { spawnSync } from "node:child_process";
import { existsSync, mkdirSync, readdirSync, statSync } from "node:fs";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";

const repoRoot = resolve(fileURLToPath(import.meta.url), `../..`);
const profilesDir = join(repoRoot, `profiles`);

if (!existsSync(profilesDir)) mkdirSync(profilesDir);

console.log(`>> Running benchmarks under --cpu-prof, output dir: ./profiles/`);
console.log(`>> Bench filter: ${process.argv.slice(2).join(` `) || `(all)`}`);

// Use Vitest's CLI binary directly with --cpu-prof. We can't use `vp test bench`
// here because the --cpu-prof flag has to be passed to the actual node process,
// not the vp wrapper.
const vitestBin = join(repoRoot, `node_modules/vitest/dist/cli.js`);
if (!existsSync(vitestBin)) {
console.error(`error: vitest CLI not found at ${vitestBin}. Run \`vp install\` first.`);
process.exit(1);
}

const result = spawnSync(
process.execPath,
[`--cpu-prof`, `--cpu-prof-dir=${profilesDir}`, vitestBin, `bench`, `--run`, ...process.argv.slice(2)],
{ stdio: `inherit`, cwd: repoRoot }
);

if (result.status !== 0) {
console.error(`>> Bench run exited with status ${result.status}`);
process.exit(result.status ?? 1);
}

// Find the most recently-written .cpuprofile in profiles/ and surface its path.
const cpuProfiles = readdirSync(profilesDir)
.filter((f) => f.endsWith(`.cpuprofile`))
.map((f) => ({ name: f, mtime: statSync(join(profilesDir, f)).mtimeMs }))
.sort((a, b) => b.mtime - a.mtime);

if (cpuProfiles.length === 0) {
console.warn(`>> No .cpuprofile files written — check that vitest actually ran.`);
process.exit(1);
}

const latest = cpuProfiles[0];
console.log(``);
console.log(`>> Wrote: ./profiles/${latest.name}`);
console.log(`>> Open in Chrome DevTools:`);
console.log(` devtools → ⋮ menu → More tools → Performance insights`);
console.log(` drag ./profiles/${latest.name} into the panel`);
console.log(``);
console.log(`>> Or inspect via the CLI:`);
console.log(` npx -y v8-cpu-prof-flamegraph ./profiles/${latest.name}`);
66 changes: 66 additions & 0 deletions src/__benchmarks__/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Performance benchmarks

This directory holds Vitest `bench()` files for iteration-time performance evaluation. They are **not run by default** — they're tools for contributors evaluating a perf change, not catastrophe-detection canaries.

For the regression canary that runs in normal CI, see [`src/__tests__/perfRegression.spec.ts`](../__tests__/perfRegression.spec.ts).

## Running

```bash
# Run all benchmarks
vp test bench

# Run a single bench file
vp test bench messageLikeSchema

# Run with a custom number of iterations (default: tinybench-driven)
vp test bench --benchmark.iterations=200
```

Output looks like:

```
BENCH Summary
mock(messageLikeSchema) > default options
3,420 ops/sec ±0.42% (456 samples)
```

`ops/sec` is the headline number. Higher is faster. The `±0.42%` is the relative margin of error — anything under ~1% is stable enough to compare runs.

## How to evaluate a perf change

1. Run the bench on `main` (or whatever baseline you're comparing against) and save the output.
2. Switch to your branch, run the bench again.
3. Compare ops/sec across runs **on the same machine, in the same session**. Absolute numbers vary by hardware and load — ratios within a session are stable.

A change that drops ops/sec by >5% (well above the RME) is a regression. A change that improves it by >5% is a win. Anything in the middle is noise and shouldn't drive decisions.

## What's here

| File | Fixture | Pattern |
| -------------------------------- | ----------------------------------- | --------------------------------------- |
| `messageLikeSchema.bench.ts` | recursive lazy through nullish | the v1.5.0 wrapper-recursion shape |
| `channelLikeSchema.bench.ts` | `intersect([common, variant(...)])` | discriminated unions with shared fields |
| `applicationLikeSchema.bench.ts` | wide-flat object | string-pipeline-dominated, no recursion |
| `interactionLikeSchema.bench.ts` | multi-`v.lazy()` union | cross-schema references |

The fixture schemas themselves live in [`fixtures/`](./fixtures/) and are shared with the regression canary spec.

## Adding a new benchmark

A new bench file should:

1. Live in this directory with a `.bench.ts` suffix (Vitest's discovery glob).
2. Import only from `fixtures/` and Valimock's public exports. No discordkit imports, no network, no filesystem.
3. Wrap each measurement in a `describe()` block named after the schema being mocked. Multiple `bench()` calls inside the same `describe` get side-by-side comparison in the reporter — use that to compare configurations of the same workload.
4. Construct the `Valimock` instance _outside_ the `bench()` callback when possible. Constructor cost is real (Map + Set initialization) but is a one-time hit, not a per-mock cost — including it in the timing measurement would obscure the actual mock cost.

## Profiling for hotspot discovery

The bench suite tells you _whether_ a change is faster. To find out _where_ the time is going, use the CPU profiler:

```bash
node --cpu-prof --cpu-prof-dir=./profiles node_modules/vitest/dist/cli.js bench --run
```

Open the resulting `.cpuprofile` in Chrome DevTools → Performance → "Load Profile". Self-time columns surface the top hotspots; that's how the original perf report identified `findFakerForKeyName` as 21.4% of CPU time.
19 changes: 19 additions & 0 deletions src/__benchmarks__/applicationLikeSchema.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import { bench, describe } from "vite-plus/test";
import { Valimock } from "../Valimock.js";
import { applicationLikeSchema } from "./fixtures/index.js";

/**
* Benchmark: wide-flat object, no recursion.
*
* Dominated by string-pipeline cost (keyNameGenerators, format generators,
* the findFakerForKeyName cache, etc.). Useful as the canonical
* "no-recursion" comparison case — if THIS bench regresses, the cause
* is in the string pipeline or the dispatch loop, not a wrapper handler.
*/
describe(`mock(applicationLikeSchema)`, () => {
const m = new Valimock({ onWarn: () => {} });

bench(`default options`, () => {
m.mock(applicationLikeSchema);
});
});
18 changes: 18 additions & 0 deletions src/__benchmarks__/channelLikeSchema.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { bench, describe } from "vite-plus/test";
import { Valimock } from "../Valimock.js";
import { channelLikeSchema } from "./fixtures/index.js";

/**
* Benchmark: `intersect([common, variant(...)])` discriminated unions.
*
* Exercises the intersect deepMerge path (the v1.5.1 variant-discriminator
* regression) and variant resolution. No recursion — useful as a control
* for the difference between shape-driven and recursion-driven cost.
*/
describe(`mock(channelLikeSchema)`, () => {
const m = new Valimock({ onWarn: () => {} });

bench(`default options`, () => {
m.mock(channelLikeSchema);
});
});
18 changes: 18 additions & 0 deletions src/__benchmarks__/interactionLikeSchema.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { bench, describe } from "vite-plus/test";
import { Valimock } from "../Valimock.js";
import { interactionLikeSchema } from "./fixtures/index.js";

/**
* Benchmark: multi-`v.lazy()` union with cross-schema references.
*
* Exercises lazy resolution interleaved with discriminated union picking
* and nested nullish wrappers. Closest match to discordkit's Interaction
* schema, which mixes all three patterns.
*/
describe(`mock(interactionLikeSchema)`, () => {
const m = new Valimock({ onWarn: () => {} });

bench(`default options`, () => {
m.mock(interactionLikeSchema);
});
});
21 changes: 21 additions & 0 deletions src/__benchmarks__/messageLikeSchema.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { bench, describe } from "vite-plus/test";
import { Valimock } from "../Valimock.js";
import { messageLikeSchema } from "./fixtures/index.js";

/**
* Benchmark: the recursive-lazy-through-nullish pattern.
*
* messageLikeSchema mirrors discordkit's Message schema — the shape that
* drove the v1.5.0 regression. Healthy ops/sec is in the low thousands;
* the v1.5.2 eager-recursion bug dropped it to ~38 ops/sec.
*
* Run all benches: vp test bench
* Run only this file: vp test bench messageLikeSchema
*/
describe(`mock(messageLikeSchema)`, () => {
const m = new Valimock({ onWarn: () => {} });

bench(`default options`, () => {
m.mock(messageLikeSchema);
});
});
11 changes: 10 additions & 1 deletion vite.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -643,6 +643,15 @@ export default defineConfig({
include: ["**/*.{test,spec}.{ts,tsx}"],
exclude: ["**/node_modules/**", "**/dist/**"],
environment: "node",
passWithNoTests: true
passWithNoTests: true,
// Benchmark suite (run via `vp test bench`). Files live under
// src/__benchmarks__/ and reuse the fixture library there. Kept out of
// the default `vp test` run so the iteration-time bench suite doesn't
// gate CI — it answers "is this change actually faster?" not "did
// anything break".
benchmark: {
include: ["src/__benchmarks__/**/*.bench.{ts,tsx}"],
exclude: ["**/node_modules/**", "**/dist/**"]
}
}
});