Skip to content

ci: isolate deploy gates and serialize deploys per service - #615

Merged
creatornader merged 1 commit into
mainfrom
fix/isolate-log-smoke-gate
Jul 29, 2026
Merged

ci: isolate deploy gates and serialize deploys per service#615
creatornader merged 1 commit into
mainfrom
fix/isolate-log-smoke-gate

Conversation

@creatornader

Copy link
Copy Markdown
Owner

Items #2 and #3 from the graph-node incident follow-ups. Written fresh — see the provenance note at the bottom.

Both fixes are the same idea: a job's result should mean what it says.

1. The log smoke check moves out of deploy-log-node

Other jobs gate on deploy-log-node.result. While the smoke check lived inside that job, any post-deploy failure turned the whole job red and took them with it.

That cost a real canary run. In run 30437471595: deploy-graph-node succeeded, deploy-archive-node succeeded, deploy-directory-node succeeded, deploy-log-node failed on a favicon-parity mismatch after its deploy had succeeded, and graph-canary was skipped — for a graph-node release that shipped fine. Same reasoning that moved the canary out of that job in #601.

This isolation matters more than it looks, because that check cannot currently go green on demand. Two independent reasons, both verified:

  • It compares live bytes across two independently deployed repos (atrib.dev and the explorer), so an icon change has an unavoidable red window between the two deploys. No merge order avoids it.
  • check-log-smoke.mjs:277 requireImmutableCache throws unless the policy includes immutable, and /static/* ships cache-control: public, max-age=86400, immutable. I measured age: 53617 on /static/favicon.ico while the check was failing — ~15h of cached bytes. A CDN can serve stale bytes for up to a day after a correct deploy.

Forbidding revalidation cannot coexist with byte-equality on a stable path. Reconciling that is its own change and I haven't made it here — changing cache policy or the assertion is a product decision spanning two repos. Until then the gate has to be able to fail alone, which is what this PR delivers.

2. Per-service deploy concurrency

The workflow-level group is deploy-services-${{ head_sha }} — keyed per commit, so every commit gets its own group and nothing serializes. Two runs from different commits could both reach flyctl deploy against the same app, and a slower deploy of an older commit could land after a newer one and quietly roll production back. Four merges landed inside twenty minutes on 2026-07-29, so it's reachable, not theoretical.

The group is per job, not workflow-wide, and that distinction is the whole point. A single wide group cancels the previously pending run when a new one queues. Since change detection is per-commit (git diff HEAD^ HEAD), cancelling commit A's run would drop A's graph-node deploy entirely whenever commit B touched only log-node. Per job, only runs deploying the same service contend, and the survivor is always the newest — whose tree already contains the older change. cancel-in-progress: false so a deploy already applying finishes rather than being killed part-way.

Behaviour change

scenario before after
log+graph deploy ok, favicon check fails canary skipped canary runs, log-smoke red
log-node deploy itself fails canary skipped canary skipped, log-smoke skipped
only graph-node deployed canary runs canary runs, log-smoke skipped
two commits both deploying graph-node may overlap serialized, newest wins
commit A graph-node + commit B log-node both deploy both deploy (per-job groups)

Verification

  • actionlint exit 0 on this file and on the whole .github/workflows directory — validates needs references and if expressions, not just YAML shape.
  • YAML parses; all four deploy jobs carry their own group; log-smoke and graph-canary correctly carry none (they mutate nothing).
  • deploy-log-node loses setup-node, which existed only for the smoke script.

Provenance note

An uncommitted edit implementing roughly this appeared in my worktree mid-session, alongside a bare fix/deploy-gate-isolation branch. I did not write it, and the session I suspected proved it wasn't theirs either (no .github/ changes in either of their worktrees; that branch has zero commits and is checked out nowhere). Origin unexplained.

I reverted it and wrote this from scratch rather than commit code of unknown provenance. The patch is preserved outside the repo if anyone wants to compare. Worth noting the version that appeared used a workflow-level concurrency group — the variant that drops deploys, per the analysis above.

Two fixes to deploy-services, both about a job's result meaning what it says.

The log smoke check moves out of deploy-log-node into its own job. Other jobs
gate on deploy-log-node's result, so while the check lived inside it any
post-deploy failure turned the whole job red and took them along. On 2026-07-29
a favicon-parity mismatch between the marketing site and the explorer's static
assets, nothing to do with log-node, failed after a successful deploy and
suppressed the graph canary for a graph-node release that had shipped fine
(run 30437471595). Same reasoning that moved the canary out of that job.

Isolation matters more here than it looks, because part of that check cannot go
green on demand. It compares live bytes across two independently deployed repos,
so an icon change has an unavoidable red window between the two deploys, and it
requires an immutable cache policy while /static/* ships max-age=86400,
immutable, so a CDN can serve stale bytes for up to a day after a correct
deploy (observed age: 53617 on /static/favicon.ico while the check was failing).
Forbidding revalidation cannot coexist with byte-equality on a stable path.
Reconciling that is its own change; until then the gate has to fail alone.

Each deploy job also gains its own concurrency group. The workflow-level group
is keyed per head_sha, which puts every commit in its own group and serializes
nothing: two runs from different commits could both reach flyctl deploy for the
same app, and a slower deploy of an older commit could land after a newer one
and roll production back. Four merges landed inside twenty minutes that day, so
it is reachable rather than theoretical.

The group is per job rather than workflow-wide on purpose. A single wide group
would cancel the previously pending run, and since change detection is
per-commit, cancelling commit A's run would drop A's graph-node deploy whenever
commit B touched only log-node. Per job, only runs deploying the same service
contend, and the survivor is the newest, whose tree already contains the older
change. cancel-in-progress stays false so a deploy already applying finishes.
@creatornader
creatornader merged commit 9f90fc8 into main Jul 29, 2026
14 checks passed
@creatornader
creatornader deleted the fix/isolate-log-smoke-gate branch July 29, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant