Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .claude/agents/ci-config-reviewer.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ items in that doc are where the real bugs hide.
Review the changed CI files in the diff: `.circleci/config.yml`,
`.circleci/continue/*`, `.circleci/scripts/*`, `.github/workflows/*`, plus any script
a CI command runs (e.g. `ops/scripts/`, `packages/*/scripts/ops/`). Also review any
justfile recipe, script or tool config in the diff that a CI job runs, for item 11.
justfile recipe, script or tool config in the diff that a CI job runs, for item 11,
and any test-code skip in the diff that changes what a CI job runs, for item 12.
Findings about untouched config are out of scope unless a change in the diff breaks
them.

Expand All @@ -30,7 +31,7 @@ them.
1. Read the guide in full, including "How CI is wired here".
2. Determine which CI files changed, and which changed recipes, scripts or tool config a
CI job runs.
3. For each change, walk the repo-specific checklist (items 1–11) first, then the
3. For each change, walk the repo-specific checklist (items 1–12) first, then the
relevant general CircleCI / GitHub Actions section. Map every finding to the
specific rule it violates.
4. Confirm the author validated locally where the guide requires it (merged-config
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Before writing or refactoring code, read [docs/ai/dev-workflow.md](docs/ai/dev-w
and the relevant language guide below. More detailed guidance for AI agents can be found in:

- [docs/ai/ci-ops.md](docs/ai/ci-ops.md) - CI/CD operations
- [docs/ai/ci-config-review.md](docs/ai/ci-config-review.md) - Reviewing changes to CI config (`.circleci/`, `.github/workflows/`), or to anything a CI job runs: gate coverage, required checks, path filtering, caching, jobs that never run on a PR, plus general CircleCI/GHA best practices
- [docs/ai/ci-config-review.md](docs/ai/ci-config-review.md) - Reviewing changes to CI config (`.circleci/`, `.github/workflows/`), or to anything a CI job runs: gate coverage, required checks, path filtering, caching, jobs that never run on a PR, cost cuts that must not regress coverage, maintainability or security, plus general CircleCI/GHA best practices
- [docs/ai/docker.md](docs/ai/docker.md) - Docker image builds: making every external fetch (apt/apk/curl/wget) retry so registry/CDN blips don't flake CI
- [docs/ai/contract-dev.md](docs/ai/contract-dev.md) - Smart contract development
- [docs/ai/standard-validator-review.md](docs/ai/standard-validator-review.md) - Reviewing `StandardValidator` for assertions it should make but doesn't: cross-game symmetry, diff-driven coverage, read-versus-assert, plus the false-positive traps (pass-through getters, implementation-pinned immutables) that make naive gap-hunting noisy. Pairs with the `standard-validator-reviewer` agent
Expand Down
37 changes: 37 additions & 0 deletions docs/ai/ci-config-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,43 @@ here. Treat items 1–4 as **blocking**.
warnings, which turns forge's compile cache off, and `warm-script-cache.sh`
failed. #24103 merged green, and #24162 fixed the nightly build.

12. **Cost cuts must not regress coverage, maintainability or security.** A diff
that saves CI time or credits (skipped tests, fewer shards, a narrower matrix, a
job moved off the PR path) must not buy the saving with a loss on any of these.
Each small trade looks harmless, and together they wear CI down. This covers
skips in test code (`t.Skip`, build tags, test filters) as well as CI config.
- **Coverage.** Name what stops running and show which job still runs it, on
the same trigger (PR, merge queue), with the same binaries, config and inputs,
or show that the difference doesn't reach what the tests exercise. A run that
looks duplicated can differ in image, env or client kind, so check rather than
assume. Moving a job to develop-only or scheduled keeps coverage but loses
pre-merge signal: judge it by "Choosing where a new job runs" and item 11.
- **Security.** A cut must not drop a security check (`cargo deny`, digest or
signature verification) or move it off the PR path, and must not let PR code
write a cache that privileged jobs restore.
- **Drift.** If the saving depends on contributors remembering something (a
skip call in each new constructor, a list kept in sync, a doc rule), require
a check that fails when they forget. Without one, the duplication or the
coverage gap creeps back with no signal. Better still, derive the decision
from data the code already has, so there is nothing to remember.
- **Skip expiry.** A skip must end when its reason does. Check that something
notices when the skipped variant gains the support it lacked; otherwise the
skip keeps dropping coverage after it is no longer needed.
- **Numbers.** Item 8 applies: cite before/after wall time and cost from a real
run.

Grade a coverage loss that no other run covers like item 11, by what then fails
after merge. Grade drift and skip-expiry gaps below that.

Example: the acceptance job that sets `DEVSTACK_L2CL_KIND=kona-node` can skip
tests whose preset starts only op-node, but only with a check that fails when a
new op-node-only preset doesn't skip. A hand-written list of the presets that
skip is not such a check: a new preset is on no list, so nothing fails and the
job silently runs the op-node tests a second time. Checks that catch it: a test
that requires every exported preset to declare which CL kinds it starts, or a
guard that fails a test in that job that never started a CL of the kind
`DEVSTACK_L2CL_KIND` selects.

## General best practices

Repo is CircleCI-primary; GitHub Actions footprint is small but these apply there.
Expand Down
Loading