This document provides guidance for AI agents working with CI/CD operational tasks in the Optimism monorepo.
For Docker image build failures — especially flaky apt/apk/curl downloads from package registries and CDNs — see docker.md.
Watch every push to a terminal state — AGENTS.md requires it. Most jobs run on
CircleCI and report as commit statuses, so gh sees them alongside the GitHub Actions
and Wiz checks:
gh pr checks <pr> --watch --fail-fast # blocks until done, exits on the first failure
gh pr checks <pr> --required # only the merge-gating checks
gh pr checks <pr> --json name,bucket,link --jq '.[]|select(.bucket=="fail")'--watch blocks until every reported check settles — the main workflow alone runs
~25 minutes, past most agent command timeouts — so background it or re-invoke it
instead of running it as a blocking call.
Notes that matter in practice:
- Wait for the gates the ruleset requires, not the individual jobs: the four
CircleCI fan-in gates (
ci-gate,required-contracts-ci,required-rust-ci,required-rust-e2e) plus thedependency-reviewGitHub Actions check; on aproposal/**branch,ci-gate-proposalsandrequired-contracts-ci. A gate reports last, so it can still be pending while every job you were watching is green. On skip paths the gate is produced by analways-succeedcompanion, so a gate that never reports at all is a config bug (see ci-config-review.md item 2), not something to wait out. - A first push is not the only push to watch. Rebases, review fixups, and merge-queue rebases each start a new pipeline against a different merge base.
- Triage before rerunning. Rule out an inherited failure (next section), then check
whether the test is a known flake. The
generate-flaky-tests-reportjob publishes aflaky-test-reportsartifact, but it coversop-acceptance-testsonly, is scoped to the pipeline's own branch (use adeveloppipeline's copy, not your PR's) and does not run on fast paths; for every other suite, look for an open flake issue instead. A rerun that hides a real regression costs more than the minutes it saved, and a confirmed flake needs an issue, not a silent retry. Reruns through the CircleCI v2 API need a personal API token inCIRCLE_TOKEN— not theCIRCLECI_CLI_TOKENthe CLI reads for ci-config-review.md'scircleci config validate --org-slug, and not theCIRCLE_API_TOKENin.circleci/, which is the in-job context token. For flakes inop-acceptance-tests//op-devstack/, flake-prevention.md catalogues the recurring causes.
Before assuming a red check is caused by your change, rule out a failure the branch
inherited — especially a long-lived branch that has drifted from develop:
- Check the diff scope:
git diff origin/develop...HEAD --name-only(three dots). If the failing test exercises code your branch never touches, it almost certainly isn't your regression. - Check
develop: the failure may be a known flake, or a real bug a laterdevelopcommit already fixed — your branch just predates the fix. Look for an open flake issue or a recent fix PR touching the failing area. - Rebase onto latest
developbefore deeper debugging — stale branches miss upstream fixes, and a rebase often clears failures that were never about your change.
Worked example (#21356): go-tests-short failed on op-deployer integration tests on a
branch that only added a new op-core/types package. The diff touched nothing in
op-deployer; the failures were a data-dependent flake already fixed on develop by
#21396. Rebasing made CI green with no change to the branch's own work.
A successful "Restore Go module cache" step does not prove that Go reads the restored
modules. Machine images default GOPATH to ~/.go_workspace, so the shared
restore-go-mod-cache helper sets GOMODCACHE=$HOME/go/pkg/mod through BASH_ENV. If a
job downloads modules after a cache hit, compare go env GOMODCACHE with the cached path.
The key includes go.mod, so a toolchain-directive bump saves a new snapshot that holds the
downloaded toolchain. .circleci/scripts/test-go-mod-cache.py checks both properties.
A CI-only compile error like:
op-core/superchain/bundle/bundle.go:NN: pattern superchain-configs.zip: no matching files found
means the job builds Go that transitively links op-core/superchain/bundle without the bundle
it //go:embeds. The zip is gitignored and built only by the prep-superchain job
(or just build-superchain-go locally), so every job that compiles an
op-core/superchain/bundle linker must provision it. The linker set is large — op-node and
several other binaries, but also op-e2e, op-acceptance-tests, op-deployer, and the
kona/op-reth Go tests (TestBundleReachability in op-core/superchain/bundle/deps_test.go
pins the full set) — so a Go-only change can red-wash unrelated-looking jobs
(rust-e2e) all at compile time. It passes locally and in review because the
developer already has the zip on disk (see go-dev.md).
Provision the bundle in the failing job:
main.ymljobs (theprep-superchainjob lives in that workflow): addprep-superchainto the job'srequiresand attach its workspace (attach_workspace) — the same patterngo-testsuses.- Other workflows (
rust-ci.yml,rust-e2e.yml— noprep-superchainjob there): add an in-jobjust build-superchain-gostep (verify mode: regenerates from the superchain-registry submodule and asserts the committed.sha256). justrecipes that compile such Go: have the recipe runjust build-superchain-gofirst.
When adding a new op-core/superchain/bundle consumer, audit every CI job that compiles it.
The repo runs a scheduled CircleCI job every 4 hours that validates TODO comments don't reference closed GitHub issues. When this job fails, issues need to be reopened.
- Find the failed TODO checker job in CircleCI (scheduled workflow named
scheduled-todo-issues) - Identify which issues were closed but still have active TODOs in the codebase
- For each issue:
- Determine who closed it (using GitHub timeline API)
- Read the actual TODO comment from the code
- Reopen with proper attribution and context
- Include file location and CircleCI job link
For complete step-by-step instructions with all commands and error handling, see: .claude/skills/fix-todo/SKILL.md
The skill includes:
- Detailed commands for querying CircleCI and GitHub APIs
- How to find who closed an issue
- Comment template for reopening
- Error handling for edge cases
- Output format and requirements