From 20a4ae91495b6c2fa1ec71b47c6d1f16ef2d6979 Mon Sep 17 00:00:00 2001 From: Maarten Marsman Date: Wed, 21 Oct 2026 16:16:56 +0200 Subject: [PATCH] ci(fast-checks): raise the every-run timeout to 45 min, state the budget as measured The ubuntu-latest runner slowed every test file by 1.4-2x between the August and October 2026 images (fast-checks runs 31265365814 and 39602904192). The every-run tier now takes 25 min of tests and the job timed out at 30 min on every push. The budget statements in the workflow, the tier helper and MAINTAINERS quote the job's measured test phase instead of a fixed number. --- .github/workflows/fast-checks.yaml | 22 +++++++-------------- .github/workflows/nightly-validation.yaml | 2 +- .github/workflows/weekly-certification.yaml | 2 +- dev/review-2026-08/MAINTAINERS.md | 10 ++++++---- tests/testthat/helper-tiers.R | 12 ++++------- 5 files changed, 19 insertions(+), 29 deletions(-) diff --git a/.github/workflows/fast-checks.yaml b/.github/workflows/fast-checks.yaml index 714529a9..e92812ea 100644 --- a/.github/workflows/fast-checks.yaml +++ b/.github/workflows/fast-checks.yaml @@ -10,19 +10,11 @@ # Every push to develop and every pull request. No gate, so the run is # exactly what a maintainer sees locally: contracts, validation, wiring, # the product surface, and the tight numerical unit guards that catch a -# shifted constant in one run. Budget: <= 15 min of tests in THIS job, -# which keeps the whole job near half its 30-minute timeout. -# -# Read the budget off this job's own test phase. The tiering audit's -# "~90 s" is a different measurement -- an OPTIMIZED build on a developer -# machine -- and nothing this job prints can be compared against it. -# devtools::test() compiles a debug build (pkgbuild::compiler_flags(debug -# = TRUE) gives "-UNDEBUG -Wall -pedantic -g -O0"), and this job keeps it -# that way on purpose (see the extra-packages note below). Measured on -# identical content that build costs 2.38x, and this 2-core runner a -# further ~2.31x over one developer machine. So quote CI minutes here, or -# say which build a local number came from; a bare seconds figure is not -# comparable across the two and let this tier drift unnoticed once. +# shifted constant in one run. The budget is this job's own test phase: +# 25 min on the October 2026 ubuntu-latest runner (run 39602904192), +# against timeout-minutes: 45. A local devtools::test() wall time is +# not comparable: it compiles a debug build (-O0) on a different +# machine. Measure from a run of this job that finished. # # T1 nightly .github/workflows/nightly-validation.yaml. Daily 03:00 UTC # on develop, BGMS_RUN_SLOW_TESTS=true, <= 60 min, timeout 90. @@ -48,9 +40,9 @@ on: jobs: fast-tests: - name: T0 every-run tier (~90 s of tests) + name: T0 every-run tier runs-on: ubuntu-latest - timeout-minutes: 30 + timeout-minutes: 45 env: GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} R_KEEP_PKG_SOURCE: yes diff --git a/.github/workflows/nightly-validation.yaml b/.github/workflows/nightly-validation.yaml index 065d0c8c..88d3fa88 100644 --- a/.github/workflows/nightly-validation.yaml +++ b/.github/workflows/nightly-validation.yaml @@ -8,7 +8,7 @@ # # T0 every run local devtools::test() + .github/workflows/fast-checks.yaml # Contracts, validation, wiring, the product surface, the tight numerical -# unit guards. No gate. Budget ~90 s. +# unit guards. No gate; budget stated in fast-checks.yaml. # # T1 nightly THIS FILE. Daily 03:00 UTC on develop. # BGMS_RUN_SLOW_TESTS=true. diff --git a/.github/workflows/weekly-certification.yaml b/.github/workflows/weekly-certification.yaml index 52736b4b..f96d4d0c 100644 --- a/.github/workflows/weekly-certification.yaml +++ b/.github/workflows/weekly-certification.yaml @@ -8,7 +8,7 @@ # # T0 every run local devtools::test() + .github/workflows/fast-checks.yaml # Contracts, validation, wiring, the product surface, the tight numerical -# unit guards. No gate. Budget ~90 s. +# unit guards. No gate; budget stated in fast-checks.yaml. # # T1 nightly .github/workflows/nightly-validation.yaml. Daily 03:00 UTC. # BGMS_RUN_SLOW_TESTS=true. diff --git a/dev/review-2026-08/MAINTAINERS.md b/dev/review-2026-08/MAINTAINERS.md index 299dae5e..6e3ad876 100644 --- a/dev/review-2026-08/MAINTAINERS.md +++ b/dev/review-2026-08/MAINTAINERS.md @@ -218,7 +218,7 @@ be T1. Budget is the tie-breaker, never the criterion. | Tier | Cadence | Content, by bug class | Budget | Gate | |---|---|---|---|---| -| **T0** every run | local `devtools::test()`; every push to develop and every PR (`fast-checks.yaml`) | contracts, validation, wiring, unit guards, product surface | **≤ 15 min of tests** in the `fast-checks` job (`timeout-minutes: 30`) | none | +| **T0** every run | local `devtools::test()`; every push to develop and every PR (`fast-checks.yaml`) | contracts, validation, wiring, unit guards, product surface | the `fast-checks` job's own test phase: 25 min on the October 2026 runner (`timeout-minutes: 45`) | none | | **T1** nightly heartbeat | daily 03:00 UTC on **develop** (`nightly-validation.yaml`) | curated calibration subset — graph-law and prior-chain identities, gauge detector, single surface-vs-gold cells, RB saturation, concordance smokes. "Does the settled math still hold tonight" | **≤ 60 min** on the 2-core runner; `timeout-minutes: 90` | `BGMS_RUN_SLOW_TESTS=true` | | **T2** weekly certification | Sunday 03:00 UTC on **develop** (`weekly-certification.yaml`) | the heavy Monte-Carlo machinery — SBC suites, full parameter-recovery sweeps, full NUTS-vs-MH condition grids, n = 2e6 MC channels, refit cross-validations (incl. the F-049 gate) | `timeout-minutes: 360` | `BGMS_RUN_CERTIFICATION=true` (the T2 workflow sets **both** vars, so a weekly run also carries T1) | @@ -233,9 +233,11 @@ machine and landed 89.7 s; `devtools::test()` and every CI tier compile a *debug* build (`pkgbuild::compiler_flags(debug = TRUE)` is `-UNDEBUG -Wall -pedantic -g -O0`). Measured on byte-identical content the build is worth **2.38x**, and the 2-core runner a further **~2.31x** over one developer -machine (`dev/review-2026-08/reports/40-t0-retier.md`). So the audit's 89.7 s -and anything CI reports were never comparable, and T0 drifted for five weeks -without the drift being visible to either number. The reference instrument is +machine (`dev/review-2026-08/reports/40-t0-retier.md`). The audit's 89.7 s +and anything CI reports are not comparable. The runner itself also moves: +between the August and October 2026 images of `ubuntu-latest` every test file +ran 1.4-2x slower (fast-checks runs 31265365814 and 39602904192), which took +the every-run tier from 14 to 25 min of tests. The reference instrument is the `fast-checks` job's own test phase. When re-tiering, measure from a run that FINISHED, in that job, and quote minutes. diff --git a/tests/testthat/helper-tiers.R b/tests/testthat/helper-tiers.R index c3ee89a6..983e89b6 100644 --- a/tests/testthat/helper-tiers.R +++ b/tests/testthat/helper-tiers.R @@ -7,14 +7,10 @@ # T0 every run (local devtools::test(), and every push/PR) # Contracts, validation, wiring, the product surface, and the tight # numerical unit guards that catch a shifted constant in one run. -# No gate. Budget: <= 15 min of tests in the fast-checks job. -# -# State which build any local number came from. devtools::test() -# compiles a debug build (-O0), the tiering audit measured an optimized -# one, and the two differ by 2.38x on identical content -- so the -# audit's "~90 s" cannot be compared with a devtools::test() wall time -# or with what CI prints. The fast-checks job is the reference -# instrument; see .github/workflows/fast-checks.yaml. +# No gate. Budget: the fast-checks job's own test phase (25 min on the +# October 2026 runner, timeout 45 min); see +# .github/workflows/fast-checks.yaml. A local devtools::test() wall +# time is a debug build on a different machine and is not comparable. # # CRAN runs a SUBSET of T0: the heavy internal numerical guard files # are excluded there via skip_heavy_guard_on_cran() below (maintainer