diff --git a/.claude/agents/ci-config-reviewer.md b/.claude/agents/ci-config-reviewer.md index 07da7d95817..4762ff89884 100644 --- a/.claude/agents/ci-config-reviewer.md +++ b/.claude/agents/ci-config-reviewer.md @@ -21,7 +21,8 @@ items in that doc are where the real bugs hide. Review the changed CI files in the diff: `.circleci/config.yml`, `.circleci/continue/*`, `.circleci/scripts/*`, `.github/workflows/*`, plus any script a CI command runs (e.g. `ops/scripts/`, `packages/*/scripts/ops/`). Also review any -justfile recipe, script or tool config in the diff that a CI job runs, for item 11. +justfile recipe, script or tool config in the diff that a CI job runs, for item 11, +and any test-code skip in the diff that changes what a CI job runs, for item 12. Findings about untouched config are out of scope unless a change in the diff breaks them. @@ -30,7 +31,7 @@ them. 1. Read the guide in full, including "How CI is wired here". 2. Determine which CI files changed, and which changed recipes, scripts or tool config a CI job runs. -3. For each change, walk the repo-specific checklist (items 1–11) first, then the +3. For each change, walk the repo-specific checklist (items 1–12) first, then the relevant general CircleCI / GitHub Actions section. Map every finding to the specific rule it violates. 4. Confirm the author validated locally where the guide requires it (merged-config diff --git a/AGENTS.md b/AGENTS.md index 343b3a0272c..8e55cbbcc10 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,7 +97,7 @@ Before writing or refactoring code, read [docs/ai/dev-workflow.md](docs/ai/dev-w and the relevant language guide below. More detailed guidance for AI agents can be found in: - [docs/ai/ci-ops.md](docs/ai/ci-ops.md) - CI/CD operations -- [docs/ai/ci-config-review.md](docs/ai/ci-config-review.md) - Reviewing changes to CI config (`.circleci/`, `.github/workflows/`), or to anything a CI job runs: gate coverage, required checks, path filtering, caching, jobs that never run on a PR, plus general CircleCI/GHA best practices +- [docs/ai/ci-config-review.md](docs/ai/ci-config-review.md) - Reviewing changes to CI config (`.circleci/`, `.github/workflows/`), or to anything a CI job runs: gate coverage, required checks, path filtering, caching, jobs that never run on a PR, cost cuts that must not regress coverage, maintainability or security, plus general CircleCI/GHA best practices - [docs/ai/docker.md](docs/ai/docker.md) - Docker image builds: making every external fetch (apt/apk/curl/wget) retry so registry/CDN blips don't flake CI - [docs/ai/contract-dev.md](docs/ai/contract-dev.md) - Smart contract development - [docs/ai/standard-validator-review.md](docs/ai/standard-validator-review.md) - Reviewing `StandardValidator` for assertions it should make but doesn't: cross-game symmetry, diff-driven coverage, read-versus-assert, plus the false-positive traps (pass-through getters, implementation-pinned immutables) that make naive gap-hunting noisy. Pairs with the `standard-validator-reviewer` agent diff --git a/docs/ai/ci-config-review.md b/docs/ai/ci-config-review.md index 2c50f251ec0..15e379cfa9c 100644 --- a/docs/ai/ci-config-review.md +++ b/docs/ai/ci-config-review.md @@ -274,6 +274,43 @@ here. Treat items 1–4 as **blocking**. warnings, which turns forge's compile cache off, and `warm-script-cache.sh` failed. #24103 merged green, and #24162 fixed the nightly build. +12. **Cost cuts must not regress coverage, maintainability or security.** A diff + that saves CI time or credits (skipped tests, fewer shards, a narrower matrix, a + job moved off the PR path) must not buy the saving with a loss on any of these. + Each small trade looks harmless, and together they wear CI down. This covers + skips in test code (`t.Skip`, build tags, test filters) as well as CI config. + - **Coverage.** Name what stops running and show which job still runs it, on + the same trigger (PR, merge queue), with the same binaries, config and inputs, + or show that the difference doesn't reach what the tests exercise. A run that + looks duplicated can differ in image, env or client kind, so check rather than + assume. Moving a job to develop-only or scheduled keeps coverage but loses + pre-merge signal: judge it by "Choosing where a new job runs" and item 11. + - **Security.** A cut must not drop a security check (`cargo deny`, digest or + signature verification) or move it off the PR path, and must not let PR code + write a cache that privileged jobs restore. + - **Drift.** If the saving depends on contributors remembering something (a + skip call in each new constructor, a list kept in sync, a doc rule), require + a check that fails when they forget. Without one, the duplication or the + coverage gap creeps back with no signal. Better still, derive the decision + from data the code already has, so there is nothing to remember. + - **Skip expiry.** A skip must end when its reason does. Check that something + notices when the skipped variant gains the support it lacked; otherwise the + skip keeps dropping coverage after it is no longer needed. + - **Numbers.** Item 8 applies: cite before/after wall time and cost from a real + run. + + Grade a coverage loss that no other run covers like item 11, by what then fails + after merge. Grade drift and skip-expiry gaps below that. + + Example: the acceptance job that sets `DEVSTACK_L2CL_KIND=kona-node` can skip + tests whose preset starts only op-node, but only with a check that fails when a + new op-node-only preset doesn't skip. A hand-written list of the presets that + skip is not such a check: a new preset is on no list, so nothing fails and the + job silently runs the op-node tests a second time. Checks that catch it: a test + that requires every exported preset to declare which CL kinds it starts, or a + guard that fails a test in that job that never started a CL of the kind + `DEVSTACK_L2CL_KIND` selects. + ## General best practices Repo is CircleCI-primary; GitHub Actions footprint is small but these apply there.