bench: pin the guard tier, lower the alert threshold, close two scenario gaps - #421
Merged
Conversation
…rio gaps The guard tier could not have caught a full revert of the work landed today. Its alert threshold was 150%, and reverting the arity-specialised creator call reads as 149.3% -- under the threshold, in a job that cannot fail the build. The threshold was calibrated when individual changes were worth 5-10%; they are now worth 20-33%. Lowering it required pinning first, which benchmarks/README.md already said: unpinned, the short scenarios calibrate to iterations=1 and their medians quantize to one perf_counter tick, ~41 ns, which was 23% of G2. The 11 scenarios under ~2 us are now pinned to fixed rounds x iterations so a round spans 50-150 us and the timer pair is under 0.2%. Scenarios at or above ~10 us keep calibration -- the tick is already under 0.5% there -- as do the ones needing per-round setup, which cannot raise iterations without timing a warm repeat instead of the cold case they exist for. Measured after pinning, four consecutive full-tier runs: every scenario within 3.3% except G2, which produced one run 17.9% fast. Threshold set to 120%, which catches a full revert of three of today's four wins and misses the context fold at ~106%, which nothing that survives runner noise would catch. Still non-gating, so a false positive is a comment. Pinning changes the reported statistic to a median of per-round means, so pre-pinning numbers are not comparable: the stored baseline cache key gains a -v2- prefix, orphaning the old entries. Expect one-off apparent gains across that boundary -- G16 moved 250 -> 168 ns purely by coming off the grid. Two scenarios added. G8b is G8 with cache=True, giving the cached cold-miss builders their own signal (27.6 us against G8's 21.5 us) -- previously only covered incidentally inside G15, which batches 50 misses into one timed call. G18 times a hop through an Alias to a cached source, ~227 ns against G2's ~157 ns; the existing frame-count test catches the hop being deleted, not the hop getting slower. Retires planning/deferred/2026-07-28-cold-miss-guard-benchmark.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The guard tier could not have caught a full revert of the work merged today.
alert-thresholdwas"150%", and reverting the arity-specialised creator call reads as 149.3% — under the threshold, in a job that already cannot fail the build.The threshold was calibrated when individual changes were worth 5-10%. They are now worth 20-33%.
Design
benchmarks/README.mdalready stated the ordering this has to follow — "revisit only if the alert threshold is ever lowered near the percentages above — then pinning has to come first" — because unpinned, the short scenarios calibrate toiterations=1and their medians quantize to oneperf_countertick (~41 ns, 23% of G2). So:rounds x iterations(benchmarks/_pinned.py), so a round spans 50-150 µs and the timer pair is under 0.2%. Scenarios at or above ~10 µs keep calibration — the tick is already under 0.5% there — as do the ones needing per-round setup, which cannot raiseiterationswithout timing a warm repeat instead of the cold case they exist for.-v2-prefix, orphaning the old entries. Expect one-off apparent gains across the boundary: G16 moved 250 → 168 ns purely by coming off the grid.Verification
just test-ci: 502 passed, 100% coverage.just lint-ci: clean. Full guard tier: 25 passed.Measured headroom after pinning — four consecutive full-tier runs, quiet machine: every scenario within 3.3%, except G2 (the smallest at ~156 ns) which produced one run at 135 ns, a 17.9% spread. That outlier read faster, so it would not trip a regression alert, but it is the scenario to distrust first. Before pinning, one tick alone was 23% of G2.
I also checked
g3_deep_chainspecifically after seeing a 15.9% swing between two early runs: isolated it is stable at 1.8%, and across four full-tier runs it is 2.9%. The early swing was a single fast outlier, not instability.New scenarios:
cache=True, so_compile_cached_factory's cold-miss builders get their own signal: 27.6 µs against G8's 21.5 µs. Previously covered only incidentally inside G15, which batches 50 misses into one timed call and dilutes a single builder ~50x. Retiresplanning/deferred/2026-07-28-cold-miss-guard-benchmark.md.Aliasto a cached source, ~227 ns against G2's ~157 ns.test_alias_hop_costs_exactly_one_resolver_framecatches the hop being deleted; nothing caught it getting slower.Non-goals
fail-on-alertstays false. A 120% threshold is only workable because a false positive costs a comment rather than a red build.ubuntu-latestrunners are noisier, and it should be revisited once there is CI history to measure. The README says so.Before merging
benchmarks/README.mdrewritten, including the reversal of its own "deliberately not done" paragraph on pinning.just lint-ciandjust test-cipass.