docs: build-once rollout runbook (owner item 18) - #210
Merged
Merged
Conversation
This was referenced Sep 11, 2026
Owner item 18. Operator runbook for rolling the canary head (b0cadcd, PR #209) onto the live Dokploy instance and enabling the build-once policy per unit. Covers: current state and default-off confirmation with file refs; pre-flight backups (pg_dump, volume tars, rollback image tar) with verification steps; the digest-pinned service update and its verification checklist; per-unit candidacy for all 89 application and 42 compose units with a batch order; the required-checks mapping per repo; the known gaps carried out of the four review rounds; and the measured compute the change is expected to recover. Docs only. Nothing here was executed against the live instance.
The live instance has 102 applications and 80 compose units; the audit export the per-unit tables are built from has 89 and 42. Roughly 13 applications and 38 compose units are therefore unclassified, and the org-wide switch would enforce them anyway. Adds that to section 1.1 with the two rules to classify them by.
…ost figure Recounted from applications.csv and composes.csv: 80 of 89 application units and 39 of 42 compose units have no explicit watchPaths, not 83 of 89. Names the nine applications that do. Also attributes the 139 auto-deploy figure to spec section 1 rather than leaving it approximate.
introduced Two additions, both found while opening this PR. Section 2.1 gains a third trap: dokploy.yml triggers on every push to canary with no paths filter, so even a docs-only merge republishes the image and moves the canary, latest and version tags. The digest capture this runbook opens with is only valid if canary is frozen for the rollout window, and this PR is itself such a merge. Section 5 gains G6: pull-request.yml job pr-check (test) fails on every PR targeting canary since #209. Five tests, all in application.real.test.ts, all from db.query.buildPolicySettings being undefined in that file's hand-written db mock. Evidenced against three runs of the same workflow: green before #209 (34048304936), the same five failures at the #209 head (34543739811) and at this docs-only branch (34611157554). Inherited, not caused here, and not a pre-existing fork baseline. The four review rounds could not see it because the same file already failed on the reviewer's Windows host for an unrelated reason.
…ming Three corrections after #211 and #212 landed on canary. The republish point in section 2.1 was overstated. Step 1 reads the rollback digest off the running service on the host, not off a tag, and a republish adds a GHCR version rather than rewriting the old manifest, so a merge to canary cannot invalidate it. Rewritten to say what actually follows: pin the digest you are rolling to, and keep the docker save, because each republish leaves the previous build untagged and that is what retention sweeps collect. Also records that the fork is in neither Dokploy CSV, so a merge to canary redeploys nothing. Section 3.1 and 3.2 now account for canary having moved twice since b0cadcd. Both follow-ups are runtime-neutral - a CI trigger and a file under __test__ - and either build is a valid rollout target. Section 3.2 now shows how to resolve the current head's digest, and keeps the b0cadcd digest for anyone who would rather roll the reviewed commit and nothing else. Section 5 G6 and section 7 step 0 record that the test regression is fixed: #212 (e92e4ad), run 34615190789, 214 files passed, 2314 passed / 1 skipped / 0 failed.
AminDhouib
force-pushed
the
docs/build-once-rollout-runbook
branch
from
September 11, 2026 15:41
3020df2 to
5c76728
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto
canary(e92e4ad) so this lands on top of #211 and #212.docs/**is now covered by #211'spaths-ignore, so merging this does notrebuild or retag the image.
Docs only. Adds
docs/build-once-rollout-runbook.md, the operator runbook forowner item 18: rolling this branch's merged build-once policy (PR #209, merge
b0cadcd) onto the live Dokploy instance, and enabling it per unit.Nothing in this PR touches code, tests, schema or CI. Nothing in the runbook was
executed against the live instance; the only live calls made while writing it
were three read-only settings/overview reads, used for the version and unit
count.
The failing test: diagnosed here, fixed in #212
pr-check (test)fails on this PR, and the failure is inherited from #209.It is not caused by this branch and it is not a pre-existing fork baseline.
b0161304(port/upr-5182, 2026-09-06)6d27c886, the #209 head merged asb0cadcd3b7e233, this branchThis branch's diff against
canaryis one markdown file, 1123 insertions, zerodeletions. It cannot change a test result, and the failure set matches the #209
head's exactly.
All five failures are in
apps/dokploy/__test__/deploy/application.real.test.ts:Root cause: that file mocks
@dokploy/server/dbwith a hand-writtenquerynamespace (
application.real.test.ts:15-53) listing onlyapplications,deployHook,domains,patchandmember. #209 reachesdb.query.buildPolicySettings, which isundefinedthere. The fix is one linein the mock.
Why four review rounds missed it. The reviewers ran the suite locally on
Windows and compared failing-test sets against the merge base, which matched
exactly in both directions.
S/track2/w6-review-2.md:78listsdeploy/application.real.test.tsas item 5 of the files that already fail onthat host, for an unrelated reason:
Command failed: mkdir -p C:\…. The filewas red before and after, so a set comparison could not show that #209 changed
why it is red. On Linux CI the Windows path problem does not exist and the
file fails for the new build-policy reason instead.
Production risk is low - the real
dbis built from the full schema barrel,which now exports the build-policy tables. But the fork's "never crashes a
deploy" guard covers a missing organization relation only, not a partial
db,and the second failing assertion shows the consequence: the deployment log
carried
[build-policy] cannot read properties of undefinedwhere the deploy'sown error belonged. Worth fixing before the policy is enabled, because a red
pull-request.ymlmasks the next real regression, and #209's integration testis the designated upstream-merge tripwire.
Fixed in #212 (
e92e4ad, now oncanary), which added the missingnamespace to that mock. Run
34615190789:
Test Files 214 passed (214),Tests 2314 passed | 1 skipped (2315), zerofailures, with
application.real.test.tspassing. Recorded in the runbook as§5 G6, and §7 step 0 now reads as done. This branch is rebased onto that fix, so
its own checks run against a green base.
What the runbook contains
v0.30.5-community.1, 247 units on theinstance, 131 in the audit export, 0 of them building on a build server and
0 with a registry configured. Fork canary head
b0cadcd. What feat(build-policy): enforced remote builds, registry push, deploy-by-digest, queue coalescing, required checks #209 adds, anda default-off confirmation with file and line references
(
schema/build-policy.ts:58,services/build-policy/policy.ts:65-68,settings.ts:9-22and:44-58, plus the two test files that pin it).pg_dumpof thedokploydatabase out of thedokploy-postgrestask, tar backups of the three named volumes and the/etc/dokploybind mount, and adocker saveof the currently runningimage. Each with a verification step that lists contents rather than checking
a file size. Expected downtime, and the full rollback sequence.
docker service update, a five-partverification checklist, first-hour regression signals, and five rollback
triggers decided in advance.
each marked candidate or not with the reason, in a batch order that starts
with landing and docs units. The exact settings to flip, and a
required-checks mapping per repo taken from the jobs that actually ran in the
last 30 days.
Four findings the runbook records
apps/dokploy/package.jsonon
canaryis alreadyv0.30.5-community.1, which is what the live instancereports today. Version is useless as a rollout marker; the runbook pins and
verifies by digest throughout.
canary,latestandv0.30.5-community.1on GHCR all now point at theb0cadcdbuild. Thepre-feat(build-policy): enforced remote builds, registry push, deploy-by-digest, queue coalescing, required checks #209 image is untagged, so a retention sweep can reach it. The runbook
makes capturing the running digest and saving the image to a tar the first
pre-flight step, before any backup.
canarymoves those tags again, which is the bannerat the top of this PR.
to exclude the later batches first, then remove exclusions a batch at a time.
80 of 89 application units and 39 of 42 compose units have no explicit
watchPaths, so the derived default would start filtering pushes fleet-widethe moment it is flipped.
Checks on the file itself
LF throughout (
git ls-files --eolreportsi/lf w/lf). Byte-level scan forGitHub, AWS, Slack and OpenAI token shapes, PEM headers and credential-bearing
URLs: zero matches. No secret value appears in the file; the four Notifly units
whose git URLs embed an OAuth token are referred to by unit name only.