bench: pin the guard tier, lower the alert threshold, close two scenario gaps

.github/workflows/benchmarks.yml

@@ -27,9 +27,9 @@ jobs:
         uses: actions/cache/restore@v4
         with:
           path: ./cache
-          key: benchmark-baseline-${{ runner.os }}-${{ github.run_id }}
+          key: benchmark-baseline-v2-${{ runner.os }}-${{ github.run_id }}
           restore-keys: |
-            benchmark-baseline-${{ runner.os }}-
+            benchmark-baseline-v2-${{ runner.os }}-
       - name: Compare against main and comment (non-gating)
         uses: benchmark-action/github-action-benchmark@v1
         continue-on-error: true
@@ -40,10 +40,10 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           comment-always: ${{ github.event_name == 'pull_request' }}
           fail-on-alert: false
-          alert-threshold: "150%"
+          alert-threshold: "120%"
       - name: Save benchmark baseline (main only)
         if: github.ref == 'refs/heads/main'
         uses: actions/cache/save@v4
         with:
           path: ./cache
-          key: benchmark-baseline-${{ runner.os }}-${{ github.run_id }}
+          key: benchmark-baseline-v2-${{ runner.os }}-${{ github.run_id }}

benchmarks/README.md

@@ -19,6 +19,7 @@ cost. Runs in CI (informational, non-gating) and locally via `just bench`.
 | G7 | Full lifecycle batch: K=100 x (build REQUEST -> sync-init cached resolve -> `await close_async()`) | real per-request cost incl. async teardown |
 | G7c | Control: K=100 empty awaits in one loop entry | residual event-loop floor inside G7 |
 | G8 | Cold first-resolve: build root container + compile + resolve, depth 6 | construction + first-compile cost |
+| G8b | G8 with every provider `cache=True` | `_compile_cached_factory`'s cold-miss builders, read against G8 |
 | G9 | Context resolve: request value by type + APP dep, warm child | non-pure context-folding path |
 | G10 | `validate()` on a depth-6 chain (isolated via `pedantic`) | graph-validation traversal, deep |
 | G11 | `validate()` on a wide 10-sibling graph (isolated via `pedantic`) | graph-validation traversal, fan-out |
@@ -28,6 +29,7 @@ cost. Runs in CI (informational, non-gating) and locally via `just bench`.
 | G15 | Concurrent first-resolve, N threads (double-checked creation lock) | free-threaded creation-lock contention |
 | G16 | Warm by-type `resolve(SomeType)`, small graph | `find_provider` lookup on the integration/`@inject` path |
 | G17 | Warm by-type `resolve(SomeType)`, 200-provider registry | lookup cost at realistic registry scale |
+| G18 | Warm resolve through an `Alias` to a cached source | the alias hop, read against G2 |
 
 **Rules.** Containers are built/warmed in setup, never inside the timed call —
 **except G8**, which builds the root container *inside* the timed call on
@@ -45,54 +47,45 @@ iteration entered the loop separately. Divide the G7 number by 100 for per-reque
 read G7c — the same batch shape with an empty body — as the residual floor still inside it
 (~15% at K=100).
 
-### Guard-tier medians sit on the platform timer's grid
-
-Unlike the comparative tier, guard scenarios are **not** pinned to a fixed
-`rounds × iterations`; pytest-benchmark calibrates each one, and the short ones land
-on `iterations=1`. Their medians are therefore quantized to one tick of
-`time.perf_counter` — **41 ns** on an Apple M4, and a comparable figure on any platform
-(`time.get_clock_info("perf_counter").resolution`). As a share of the value that is large
-for the sub-microsecond scenarios:
-
-| Scenario | median | one tick |
-|---|---|---|
-| G2 cached resolve | ~180 ns | **23%** |
-| G16/G17 by-type resolve | ~220-250 ns | 16-18% |
-| G1 transient resolve | ~330 ns | 12% |
-| G5 cross-scope | ~420 ns | 10% |
-| G6/G6b child build | ~560-630 ns | 7% |
-| G3 deep chain | ~830 ns | 5% |
-| G4 and everything slower | ≥1.4 µs | ≤3% |
-
-**A one-tick move is resolution, not signal.** Two runs of an unchanged G6 will happily
-report 541 ns and 584 ns — exactly one tick apart, which reads as an 8% regression and is
-nothing at all. This has already caused one false reading of a real change.
-
-That is fine for what CI does with these numbers: the benchmarks workflow is non-gating
-(`fail-on-alert: false`) and alerts at `150%`, i.e. a 50% regression, which no amount of
-tick noise reaches. It is **not** fine for judging a small change locally. For that, measure
-the specific call directly with enough iterations to escape the grid, rather than reading a
-guard median:
-
-```python
-import statistics, timeit
-
-n = 200_000
-print(
-    statistics.median(
-        timeit.timeit(lambda: container.build_child_container(scope=Scope.REQUEST), number=n) / n * 1e9
-        for _ in range(9)
-    ),
-    "ns",
-)
-```
-
-Pinning the guard tier the way the comparative tier is pinned would remove the grid, but it
-is deliberately not done: it would break the stored CI baseline every scenario is compared
-against, it changes the reported statistic to a median-of-means, and each scenario would
-need its own hand-tuned pair (G14 at ~1.4 ms per call cannot take the settings G2 wants).
-Revisit it only if the alert threshold is ever lowered near the percentages above — then
-pinning has to come first.
+### The sub-2 microsecond guard scenarios are pinned
+
+Scenarios costing under ~2 us are pinned to a fixed `rounds x iterations`
+(`benchmarks/_pinned.py`), so one round spans 50-150 us and the `time.perf_counter` pair is
+under 0.2% of the value. Everything at or above ~10 us keeps pytest-benchmark's calibration --
+the timer is already under 0.5% there -- as do the scenarios needing per-round setup, which
+cannot raise `iterations` without timing a warm repeat instead of the cold case they exist for.
+
+**This was not always so, and the reason it changed is worth keeping.** Unpinned, the short
+scenarios calibrate to `iterations=1` and their medians quantize to one tick -- ~41 ns on an
+Apple M4, which was 23% of G2 and 16-18% of the by-type scenarios. Two runs of an unchanged G6
+would report 541 ns and 584 ns, exactly one tick apart, reading as an 8% regression that is
+nothing at all. That was tolerable while the alert threshold sat at 150%, far above any tick
+noise. It stopped being tolerable when individual changes started being worth 20-33% each: a
+*complete revert* of the arity-specialised creator call reads as 149.3%, which the old threshold
+would not have caught.
+
+**Pinning changes the reported statistic** from a median of single calls to a median of
+per-round means -- the same statistic the comparative tier reports. Pre-pinning numbers are
+therefore not comparable to post-pinning ones, which is why the stored CI baseline was reset
+(the cache key carries a `-v2-` prefix; the old entries are orphaned rather than deleted).
+Expect apparent one-off "improvements" across that boundary: G16 moved 250 -> 168 ns purely by
+coming off the grid.
+
+**The alert threshold is 120% and the job stays non-gating.** 120% catches a full revert of
+three of the four optimizations landed on 2026-08-03 (arity ladder 149.3%, alias hop 127.8%,
+by-type inline 124.5%) and misses the fourth (the context fold, ~106%), which no threshold that
+survives shared-runner variance would catch.
+
+Measured headroom, four consecutive full-tier runs on a quiet machine after pinning: every
+scenario within **3.3%**, except G2 -- the smallest at ~156 ns -- which produced one run at
+135 ns, a 17.9% spread. That outlier read *faster*, so it would not trip a regression alert, but
+it is the reason G2 is the scenario to distrust first. Before pinning, a single tick alone was
+23% of G2.
+
+The number is still provisional: shared `ubuntu-latest` runners are noisier than this, and it
+should be revisited once there is CI history to measure. Because `fail-on-alert` is false, a
+false positive costs a comment rather than a red build -- which is the trade that makes a
+threshold this low workable at all.
 
 ### Concurrency (G14/G15)
 

benchmarks/_pinned.py

@@ -0,0 +1,24 @@
+"""Fixed round/iteration pairs for the sub-2-microsecond guard scenarios.
+
+pytest-benchmark calibrates each scenario, and the short ones land on ``iterations=1``. Their
+medians are then quantized to one ``time.perf_counter`` tick — ~41 ns on an Apple M4, which is
+23% of the smallest scenario's value. A 10-30 ns move is unreadable at that resolution, and the
+optimizations these scenarios exist to protect are now worth 20-33% each.
+
+Pinning puts every round above ~80 us, so the timer pair is under 0.1% of the measured value.
+Scenarios at or above ~10 us keep calibration — the tick is already under 0.5% of those — as do
+the ones needing per-round setup, which cannot take ``iterations`` above 1 without measuring a
+warm repeat instead of the cold case they exist for.
+
+The reported statistic changes with pinning: a median of per-round *means* rather than of single
+calls. That is the same statistic the comparative tier reports, and it is why the stored CI
+baseline is reset when these land.
+"""
+
+#: Rounds per scenario, matching the comparative tier.
+ROUNDS = 200
+
+#: Iterations, chosen so one round spans ~80-150 us at each scenario's measured per-call cost.
+ITER_UNDER_300NS = 500
+ITER_UNDER_1US = 200
+ITER_UNDER_2US = 100

benchmarks/test_guard_by_type.py

@@ -10,6 +10,7 @@
 
 import dataclasses
 
+from benchmarks._pinned import ITER_UNDER_300NS, ROUNDS
 from modern_di import Container, Group, Scope, providers
 
 
@@ -33,7 +34,7 @@ def test_g16_resolve_by_type(benchmark):
     container = Container(scope=Scope.APP, groups=[ByTypeGroup])
     container.open()
     container.resolve(Service)  # warm the cache and the compiled resolver
-    result = benchmark(container.resolve, Service)
+    result = benchmark.pedantic(container.resolve, args=(Service,), rounds=ROUNDS, iterations=ITER_UNDER_300NS)
     assert isinstance(result, Service)
     assert isinstance(result.dep, Dep)
 
@@ -60,6 +61,6 @@ def test_g17_resolve_by_type_large_registry(benchmark):
     container = Container(scope=Scope.APP, groups=[_WIDE_REGISTRY_GROUP])
     container.open()
     container.resolve(Service)  # warm
-    result = benchmark(container.resolve, Service)
+    result = benchmark.pedantic(container.resolve, args=(Service,), rounds=ROUNDS, iterations=ITER_UNDER_300NS)
     assert isinstance(result, Service)
     assert isinstance(result.dep, Dep)

benchmarks/test_guard_cold.py

@@ -66,3 +66,30 @@ def test_g8_cold_first_resolve(benchmark):
     result = benchmark(_cold_build_and_resolve)
     assert isinstance(result, C0)
     assert isinstance(result.c1.c2.c3.c4.c5, C5)
+
+
+# --- G8b: the same shape with caching on, so the cached cold-miss builders are timed ---
+class CachedChainGroup(Group):
+    c5 = providers.Factory(creator=C5, scope=Scope.APP, cache=True)
+    c4 = providers.Factory(creator=C4, scope=Scope.APP, cache=True)
+    c3 = providers.Factory(creator=C3, scope=Scope.APP, cache=True)
+    c2 = providers.Factory(creator=C2, scope=Scope.APP, cache=True)
+    c1 = providers.Factory(creator=C1, scope=Scope.APP, cache=True)
+    c0 = providers.Factory(creator=C0, scope=Scope.APP, cache=True)
+
+
+def _cold_build_and_resolve_cached() -> C0:
+    container = Container(scope=Scope.APP, groups=[CachedChainGroup])
+    container.open()
+    return container.resolve_provider(CachedChainGroup.c0)
+
+
+def test_g8b_cold_first_resolve_cached(benchmark):
+    # G8's `cache=True` sibling. G8 is all-transient, so it never reaches
+    # `_compile_cached_factory`'s cold-miss builders (`build_cold` / `create_cold`); the only
+    # other coverage is incidental inside G15, which batches 50 misses into one timed call and
+    # dilutes a single builder ~50x. This times six of them against G8 as the control, so a
+    # regression confined to the cached cold path is readable as the G8b/G8 difference.
+    result = benchmark(_cold_build_and_resolve_cached)
+    assert isinstance(result, C0)
+    assert isinstance(result.c1.c2.c3.c4.c5, C5)

benchmarks/test_guard_lifecycle.py

@@ -13,6 +13,7 @@
 import asyncio
 import dataclasses
 
+from benchmarks._pinned import ITER_UNDER_1US, ROUNDS
 from modern_di import Container, Group, Scope, providers
 
 
@@ -31,7 +32,9 @@ class BuildGroup(Group):
 def test_g6_build_child_container(benchmark):
     app = Container(scope=Scope.APP, groups=[BuildGroup])
     app.open()
-    result = benchmark(app.build_child_container, scope=Scope.REQUEST)
+    result = benchmark.pedantic(
+        app.build_child_container, kwargs={"scope": Scope.REQUEST}, rounds=ROUNDS, iterations=ITER_UNDER_1US
+    )
     assert result.scope is Scope.REQUEST
 
 
@@ -40,7 +43,7 @@ def test_g6b_build_child_container_auto_scope(benchmark):
     # scope and never exercises it; this guards the memoized auto-increment step against regressing.
     app = Container(scope=Scope.APP, groups=[BuildGroup])
     app.open()
-    result = benchmark(app.build_child_container)
+    result = benchmark.pedantic(app.build_child_container, rounds=ROUNDS, iterations=ITER_UNDER_1US)
     assert result.scope is Scope.SESSION
 
 

benchmarks/test_guard_resolve.py

@@ -8,6 +8,7 @@
 
 import dataclasses
 
+from benchmarks._pinned import ITER_UNDER_1US, ITER_UNDER_2US, ITER_UNDER_300NS, ROUNDS
 from modern_di import Container, Group, Scope, providers
 
 
@@ -35,7 +36,9 @@ class SingletonGroup(Group):
 def test_g1_transient_resolve(benchmark):
     container = Container(scope=Scope.APP, groups=[TransientGroup])
     container.open()
-    result = benchmark(container.resolve_provider, TransientGroup.svc)
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(TransientGroup.svc,), rounds=ROUNDS, iterations=ITER_UNDER_1US
+    )
     assert isinstance(result, Service)
     assert isinstance(result.dep, Dep)
 
@@ -44,7 +47,9 @@ def test_g2_cached_resolve(benchmark):
     container = Container(scope=Scope.APP, groups=[SingletonGroup])
     container.open()
     container.resolve_provider(SingletonGroup.svc)  # warm the cache
-    result = benchmark(container.resolve_provider, SingletonGroup.svc)
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(SingletonGroup.svc,), rounds=ROUNDS, iterations=ITER_UNDER_300NS
+    )
     assert isinstance(result, Service)
 
 
@@ -91,7 +96,9 @@ class ChainGroup(Group):
 def test_g3_deep_chain(benchmark):
     container = Container(scope=Scope.APP, groups=[ChainGroup])
     container.open()
-    result = benchmark(container.resolve_provider, ChainGroup.c0)
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(ChainGroup.c0,), rounds=ROUNDS, iterations=ITER_UNDER_1US
+    )
     assert isinstance(result, C0)
     assert isinstance(result.c1.c2.c3.c4.c5, C5)
 
@@ -178,7 +185,9 @@ class WideGroup(Group):
 def test_g4_wide_resolve(benchmark):
     container = Container(scope=Scope.APP, groups=[WideGroup])
     container.open()
-    result = benchmark(container.resolve_provider, WideGroup.wide)
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(WideGroup.wide,), rounds=ROUNDS, iterations=ITER_UNDER_2US
+    )
     assert isinstance(result, Wide)
     assert isinstance(result.l9, L9)
 
@@ -204,7 +213,9 @@ def test_g5_cross_scope(benchmark):
     app.open()
     req = app.build_child_container(scope=Scope.REQUEST)
     req.open()
-    result = benchmark(req.resolve_provider, CrossScopeGroup.req_svc)
+    result = benchmark.pedantic(
+        req.resolve_provider, args=(CrossScopeGroup.req_svc,), rounds=ROUNDS, iterations=ITER_UNDER_1US
+    )
     assert isinstance(result, RequestService)
     assert isinstance(result.app, AppService)
 
@@ -239,7 +250,9 @@ def test_g9_context_resolve(benchmark):
     app.open()
     req = app.build_child_container(scope=Scope.REQUEST, context={RequestObj: RequestObj()})
     req.open()
-    result = benchmark(req.resolve_provider, ContextGroup.handler)
+    result = benchmark.pedantic(
+        req.resolve_provider, args=(ContextGroup.handler,), rounds=ROUNDS, iterations=ITER_UNDER_1US
+    )
     assert isinstance(result, Handler)
     assert isinstance(result.req, RequestObj)
     assert isinstance(result.dep, AppDep)
@@ -267,5 +280,31 @@ def test_g12_override_active_resolve(benchmark):
     container.open()
     container.override(OverrideChainGroup.sentinel, Sentinel())
     container.resolve_provider(OverrideChainGroup.c0)  # warm
-    result = benchmark(container.resolve_provider, OverrideChainGroup.c0)
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(OverrideChainGroup.c0,), rounds=ROUNDS, iterations=ITER_UNDER_2US
+    )
     assert isinstance(result, C0)
+
+
+# --- G18: alias hop, against G2 as the control ---
+class AliasIface: ...
+
+
+class AliasGroup(Group):
+    dep = providers.Factory(creator=Dep, scope=Scope.APP)
+    source = providers.Factory(creator=Service, scope=Scope.APP, cache=True)
+    alias = providers.Alias(source_type=Service, bound_type=AliasIface)
+
+
+def test_g18_alias_hop(benchmark):
+    # An alias forwards to its source's compiled resolver. Its cost is this scenario minus G2
+    # (the same cached provider resolved directly), which is the only way the hop is visible --
+    # `test_alias_hop_costs_exactly_one_resolver_frame` catches the hop being deleted, not the
+    # hop getting slower.
+    container = Container(scope=Scope.APP, groups=[AliasGroup])
+    container.open()
+    container.resolve_provider(AliasGroup.alias)  # warm: compile + fill the source's cache
+    result = benchmark.pedantic(
+        container.resolve_provider, args=(AliasGroup.alias,), rounds=ROUNDS, iterations=ITER_UNDER_300NS
+    )
+    assert isinstance(result, Service)