chore(benchmark): methodology improvements — cache, runner, statistics, corpus pinning

[w-martin]

Context

This issue tracks methodology improvements to benchmarks/benchmark_checkers.py that are independent of the corpus-targeting bug (see #1) and cache-path bug (see #2). These changes improve statistical validity and reproducibility.

Issues

1. RUNS=10 insufficient for high-variance tools

pyrefly on the small corpus produced 3.78s ± 7.53s (coefficient of variation > 100%) — statistically meaningless. 10 runs is too few to characterise tools with startup variance. Increase to at least 20 timed runs with outlier rejection (e.g. discard runs outside 2 standard deviations and report interquartile range alongside mean ± std).

2. ty numbers likely stale

ty is a pre-release (0.0.x) with rapid iteration. Each minor release has seen significant performance changes. The README table should be re-run whenever ty has a major version bump, and the table caption should record the tool version and date.

3. Great Expectations corpus not pinned

The script clones GE at --depth 1 from HEAD of main. A fresh machine will get a different corpus than the machine that last updated the README table. Pin to a specific tag or commit hash:

GE_REPO_URL = "https://github.com/great-expectations/great_expectations.git"
GE_COMMIT = "some-stable-tag-or-sha"  # e.g. "1.4.3"

And pass --branch <GE_COMMIT> to the clone command.

4. No machine spec or date recorded in README table

Benchmark results are only meaningful relative to the machine that produced them. The README table should record:

• Date benchmarks were run
• Machine: CPU model, core count, RAM, OS version
• Whether other applications were running (ideally: isolated machine, Spotlight/Time Machine disabled)

Add a table caption or footnote with this information when running --update-readme.

5. Warmup is a no-op for cache-clearing tools

For tools with needs_cache_clear=True, the cache is cleared before each warmup run as well as each timed run. This means warmup runs are all cold-cache — there is no warm state to warm into, and the warmup label is misleading. Either:

• Remove warmup entirely for cache-clearing tools (it does nothing), or
• Only clear cache before the first warmup run (so the warmup warms the OS page cache, and timed runs start from OS-warm state)

Document which cold-state is being measured: OS-cold or tool-warm.

6. Consider adopting hyperfine for CLI tools

hyperfine is the standard Rust-ecosystem CLI benchmarking tool. It handles warmup, shell overhead subtraction, outlier detection, and JSON output. For ruff, ty, pyrefly, and the typedframes binary (all Rust CLIs), hyperfine would be more rigorous than the current subprocess.run() + time.perf_counter_ns() approach:

hyperfine --warmup 3 --runs 20 \
  "ruff check --no-cache {target}" \
  "typedframes check {target}"

This is a larger refactor but would make the benchmark script more defensible.