ci: isolate deploy gates and serialize deploys per service - #615
Merged
Conversation
Two fixes to deploy-services, both about a job's result meaning what it says. The log smoke check moves out of deploy-log-node into its own job. Other jobs gate on deploy-log-node's result, so while the check lived inside it any post-deploy failure turned the whole job red and took them along. On 2026-07-29 a favicon-parity mismatch between the marketing site and the explorer's static assets, nothing to do with log-node, failed after a successful deploy and suppressed the graph canary for a graph-node release that had shipped fine (run 30437471595). Same reasoning that moved the canary out of that job. Isolation matters more here than it looks, because part of that check cannot go green on demand. It compares live bytes across two independently deployed repos, so an icon change has an unavoidable red window between the two deploys, and it requires an immutable cache policy while /static/* ships max-age=86400, immutable, so a CDN can serve stale bytes for up to a day after a correct deploy (observed age: 53617 on /static/favicon.ico while the check was failing). Forbidding revalidation cannot coexist with byte-equality on a stable path. Reconciling that is its own change; until then the gate has to fail alone. Each deploy job also gains its own concurrency group. The workflow-level group is keyed per head_sha, which puts every commit in its own group and serializes nothing: two runs from different commits could both reach flyctl deploy for the same app, and a slower deploy of an older commit could land after a newer one and roll production back. Four merges landed inside twenty minutes that day, so it is reachable rather than theoretical. The group is per job rather than workflow-wide on purpose. A single wide group would cancel the previously pending run, and since change detection is per-commit, cancelling commit A's run would drop A's graph-node deploy whenever commit B touched only log-node. Per job, only runs deploying the same service contend, and the survivor is the newest, whose tree already contains the older change. cancel-in-progress stays false so a deploy already applying finishes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Items #2 and #3 from the graph-node incident follow-ups. Written fresh — see the provenance note at the bottom.
Both fixes are the same idea: a job's result should mean what it says.
1. The log smoke check moves out of
deploy-log-nodeOther jobs gate on
deploy-log-node.result. While the smoke check lived inside that job, any post-deploy failure turned the whole job red and took them with it.That cost a real canary run. In run 30437471595:
deploy-graph-nodesucceeded,deploy-archive-nodesucceeded,deploy-directory-nodesucceeded,deploy-log-nodefailed on a favicon-parity mismatch after its deploy had succeeded, andgraph-canarywas skipped — for a graph-node release that shipped fine. Same reasoning that moved the canary out of that job in #601.This isolation matters more than it looks, because that check cannot currently go green on demand. Two independent reasons, both verified:
atrib.devand the explorer), so an icon change has an unavoidable red window between the two deploys. No merge order avoids it.check-log-smoke.mjs:277requireImmutableCachethrows unless the policy includesimmutable, and/static/*shipscache-control: public, max-age=86400, immutable. I measuredage: 53617on/static/favicon.icowhile the check was failing — ~15h of cached bytes. A CDN can serve stale bytes for up to a day after a correct deploy.Forbidding revalidation cannot coexist with byte-equality on a stable path. Reconciling that is its own change and I haven't made it here — changing cache policy or the assertion is a product decision spanning two repos. Until then the gate has to be able to fail alone, which is what this PR delivers.
2. Per-service deploy concurrency
The workflow-level group is
deploy-services-${{ head_sha }}— keyed per commit, so every commit gets its own group and nothing serializes. Two runs from different commits could both reachflyctl deployagainst the same app, and a slower deploy of an older commit could land after a newer one and quietly roll production back. Four merges landed inside twenty minutes on 2026-07-29, so it's reachable, not theoretical.The group is per job, not workflow-wide, and that distinction is the whole point. A single wide group cancels the previously pending run when a new one queues. Since change detection is per-commit (
git diff HEAD^ HEAD), cancelling commit A's run would drop A's graph-node deploy entirely whenever commit B touched only log-node. Per job, only runs deploying the same service contend, and the survivor is always the newest — whose tree already contains the older change.cancel-in-progress: falseso a deploy already applying finishes rather than being killed part-way.Behaviour change
Verification
actionlintexit 0 on this file and on the whole.github/workflowsdirectory — validatesneedsreferences andifexpressions, not just YAML shape.log-smokeandgraph-canarycorrectly carry none (they mutate nothing).deploy-log-nodelosessetup-node, which existed only for the smoke script.Provenance note
An uncommitted edit implementing roughly this appeared in my worktree mid-session, alongside a bare
fix/deploy-gate-isolationbranch. I did not write it, and the session I suspected proved it wasn't theirs either (no.github/changes in either of their worktrees; that branch has zero commits and is checked out nowhere). Origin unexplained.I reverted it and wrote this from scratch rather than commit code of unknown provenance. The patch is preserved outside the repo if anyone wants to compare. Worth noting the version that appeared used a workflow-level concurrency group — the variant that drops deploys, per the analysis above.