Skip to content

bench: pin the guard tier, lower the alert threshold, close two scenario gaps - #421

Merged
lesnik512 merged 1 commit into
mainfrom
bench/pin-guard-tier
Aug 3, 2026
Merged

bench: pin the guard tier, lower the alert threshold, close two scenario gaps#421
lesnik512 merged 1 commit into
mainfrom
bench/pin-guard-tier

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Why

The guard tier could not have caught a full revert of the work merged today. alert-threshold was "150%", and reverting the arity-specialised creator call reads as 149.3% — under the threshold, in a job that already cannot fail the build.

change merged today full revert reads as
arity ladder (#420) 149.3%
alias hop (#413) 127.8%
by-type inline (#419) 124.5%
context fold (#418) ~106%

The threshold was calibrated when individual changes were worth 5-10%. They are now worth 20-33%.

Design

benchmarks/README.md already stated the ordering this has to follow — "revisit only if the alert threshold is ever lowered near the percentages above — then pinning has to come first" — because unpinned, the short scenarios calibrate to iterations=1 and their medians quantize to one perf_counter tick (~41 ns, 23% of G2). So:

  1. Pin. The 11 scenarios under ~2 µs get fixed rounds x iterations (benchmarks/_pinned.py), so a round spans 50-150 µs and the timer pair is under 0.2%. Scenarios at or above ~10 µs keep calibration — the tick is already under 0.5% there — as do the ones needing per-round setup, which cannot raise iterations without timing a warm repeat instead of the cold case they exist for.
  2. Reset the baseline. Pinning changes the reported statistic to a median of per-round means, so pre- and post-pinning numbers are not comparable. The cache key gains a -v2- prefix, orphaning the old entries. Expect one-off apparent gains across the boundary: G16 moved 250 → 168 ns purely by coming off the grid.
  3. Lower to 120%. Catches a full revert of three of the four wins above; misses the context fold at ~106%, which nothing surviving runner noise would catch. Stays non-gating.
  4. Close both scenario gaps.

Verification

just test-ci: 502 passed, 100% coverage. just lint-ci: clean. Full guard tier: 25 passed.

Measured headroom after pinning — four consecutive full-tier runs, quiet machine: every scenario within 3.3%, except G2 (the smallest at ~156 ns) which produced one run at 135 ns, a 17.9% spread. That outlier read faster, so it would not trip a regression alert, but it is the scenario to distrust first. Before pinning, one tick alone was 23% of G2.

I also checked g3_deep_chain specifically after seeing a 15.9% swing between two early runs: isolated it is stable at 1.8%, and across four full-tier runs it is 2.9%. The early swing was a single fast outlier, not instability.

New scenarios:

  • G8b — G8 with cache=True, so _compile_cached_factory's cold-miss builders get their own signal: 27.6 µs against G8's 21.5 µs. Previously covered only incidentally inside G15, which batches 50 misses into one timed call and dilutes a single builder ~50x. Retires planning/deferred/2026-07-28-cold-miss-guard-benchmark.md.
  • G18 — a hop through an Alias to a cached source, ~227 ns against G2's ~157 ns. test_alias_hop_costs_exactly_one_resolver_frame catches the hop being deleted; nothing caught it getting slower.

Non-goals

  • Not made gating. fail-on-alert stays false. A 120% threshold is only workable because a false positive costs a comment rather than a red build.
  • The 120% figure is provisional. It was chosen against locally-measured noise; shared ubuntu-latest runners are noisier, and it should be revisited once there is CI history to measure. The README says so.
  • The comparative tier is untouched — it was already pinned.

Before merging

  • Behaviour changed? No library code in this PR; benchmarks/README.md rewritten, including the reversal of its own "deliberately not done" paragraph on pinning.
  • Rejected an alternative? Lowering the threshold without pinning — rejected inline, and it is what the README already warned against.
  • Found real work you are not doing now? Re-tuning 120% against real CI variance; stated in the README rather than filed, since it needs history this PR creates.
  • just lint-ci and just test-ci pass.

…rio gaps

The guard tier could not have caught a full revert of the work landed today. Its
alert threshold was 150%, and reverting the arity-specialised creator call reads
as 149.3% -- under the threshold, in a job that cannot fail the build. The
threshold was calibrated when individual changes were worth 5-10%; they are now
worth 20-33%.

Lowering it required pinning first, which benchmarks/README.md already said:
unpinned, the short scenarios calibrate to iterations=1 and their medians
quantize to one perf_counter tick, ~41 ns, which was 23% of G2. The 11 scenarios
under ~2 us are now pinned to fixed rounds x iterations so a round spans
50-150 us and the timer pair is under 0.2%. Scenarios at or above ~10 us keep
calibration -- the tick is already under 0.5% there -- as do the ones needing
per-round setup, which cannot raise iterations without timing a warm repeat
instead of the cold case they exist for.

Measured after pinning, four consecutive full-tier runs: every scenario within
3.3% except G2, which produced one run 17.9% fast. Threshold set to 120%, which
catches a full revert of three of today's four wins and misses the context fold
at ~106%, which nothing that survives runner noise would catch. Still non-gating,
so a false positive is a comment.

Pinning changes the reported statistic to a median of per-round means, so
pre-pinning numbers are not comparable: the stored baseline cache key gains a
-v2- prefix, orphaning the old entries. Expect one-off apparent gains across that
boundary -- G16 moved 250 -> 168 ns purely by coming off the grid.

Two scenarios added. G8b is G8 with cache=True, giving the cached cold-miss
builders their own signal (27.6 us against G8's 21.5 us) -- previously only
covered incidentally inside G15, which batches 50 misses into one timed call.
G18 times a hop through an Alias to a cached source, ~227 ns against G2's
~157 ns; the existing frame-count test catches the hop being deleted, not the
hop getting slower.

Retires planning/deferred/2026-07-28-cold-miss-guard-benchmark.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lesnik512
lesnik512 merged commit 336f983 into main Aug 3, 2026
9 checks passed
@lesnik512
lesnik512 deleted the bench/pin-guard-tier branch August 3, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant