Skip to content

docs: build-once rollout runbook (owner item 18) - #210

Merged
AminDhouib merged 5 commits into
canaryfrom
docs/build-once-rollout-runbook
Sep 11, 2026
Merged

AminDhouib merged 5 commits into
canaryfrom
docs/build-once-rollout-runbook

Conversation

@AminDhouib

@AminDhouib AminDhouib commented Sep 11, 2026

Copy link
Copy Markdown
Member

Rebased onto canary (e92e4ad) so this lands on top of #211 and #212.
docs/** is now covered by #211's paths-ignore, so merging this does not
rebuild or retag the image.

Docs only. Adds docs/build-once-rollout-runbook.md, the operator runbook for
owner item 18: rolling this branch's merged build-once policy (PR #209, merge
b0cadcd) onto the live Dokploy instance, and enabling it per unit.

Nothing in this PR touches code, tests, schema or CI. Nothing in the runbook was
executed against the live instance; the only live calls made while writing it
were three read-only settings/overview reads, used for the version and unit
count.

The failing test: diagnosed here, fixed in #212

pr-check (test) fails on this PR, and the failure is inherited from #209.
It is not caused by this branch and it is not a pre-existing fork baseline.

Run Head Contains #209 Test files Tests Failures
34048304936 b0161304 (port/upr-5182, 2026-09-06) no 190 passed (190) 1912 passed, 1 skipped (1913) 0
34543739811 6d27c886, the #209 head merged as b0cadcd yes 1 failed, 213 passed (214) 5 failed, 2309 passed, 1 skipped (2315) 5
34611157554 3b7e233, this branch yes 1 failed, 213 passed (214) 5 failed, 2309 passed, 1 skipped (2315) the same 5

This branch's diff against canary is one markdown file, 1123 insertions, zero
deletions. It cannot change a test result, and the failure set matches the #209
head's exactly.

All five failures are in apps/dokploy/__test__/deploy/application.real.test.ts:

TypeError: Cannot read properties of undefined (reading 'findFirst')
 ❯ findBuildPolicySettings packages/server/src/services/build-policy/settings.ts:18
 ❯ previewBuildPolicyDecision packages/server/src/services/build-policy/resolve.ts:57
 ❯ resolveBuildPolicy         packages/server/src/services/build-policy/resolve.ts:107
 ❯ planApplicationBuild       packages/server/src/services/build-policy/apply.ts:243
 ❯ deployApplication          packages/server/src/services/application.ts:211

Root cause: that file mocks @dokploy/server/db with a hand-written query
namespace (application.real.test.ts:15-53) listing only applications,
deployHook, domains, patch and member. #209 reaches
db.query.buildPolicySettings, which is undefined there. The fix is one line
in the mock.

Why four review rounds missed it. The reviewers ran the suite locally on
Windows and compared failing-test sets against the merge base, which matched
exactly in both directions. S/track2/w6-review-2.md:78 lists
deploy/application.real.test.ts as item 5 of the files that already fail on
that host, for an unrelated reason: Command failed: mkdir -p C:\…. The file
was red before and after, so a set comparison could not show that #209 changed
why it is red. On Linux CI the Windows path problem does not exist and the
file fails for the new build-policy reason instead.

Production risk is low - the real db is built from the full schema barrel,
which now exports the build-policy tables. But the fork's "never crashes a
deploy" guard covers a missing organization relation only, not a partial db,
and the second failing assertion shows the consequence: the deployment log
carried [build-policy] cannot read properties of undefined where the deploy's
own error belonged. Worth fixing before the policy is enabled, because a red
pull-request.yml masks the next real regression, and #209's integration test
is the designated upstream-merge tripwire.

Fixed in #212 (e92e4ad, now on canary), which added the missing
namespace to that mock. Run
34615190789:
Test Files 214 passed (214), Tests 2314 passed | 1 skipped (2315), zero
failures, with application.real.test.ts passing. Recorded in the runbook as
§5 G6, and §7 step 0 now reads as done. This branch is rebased onto that fix, so
its own checks run against a green base.

What the runbook contains

  1. Current state. Live version v0.30.5-community.1, 247 units on the
    instance, 131 in the audit export, 0 of them building on a build server and
    0 with a registry configured. Fork canary head b0cadcd. What feat(build-policy): enforced remote builds, registry push, deploy-by-digest, queue coalescing, required checks #209 adds, and
    a default-off confirmation with file and line references
    (schema/build-policy.ts:58, services/build-policy/policy.ts:65-68,
    settings.ts:9-22 and :44-58, plus the two test files that pin it).
  2. Pre-flight. pg_dump of the dokploy database out of the
    dokploy-postgres task, tar backups of the three named volumes and the
    /etc/dokploy bind mount, and a docker save of the currently running
    image. Each with a verification step that lists contents rather than checking
    a file size. Expected downtime, and the full rollback sequence.
  3. Rollout. A digest-pinned docker service update, a five-part
    verification checklist, first-hour regression signals, and five rollback
    triggers decided in advance.
  4. Per-unit enablement. All 89 application units and all 42 compose units,
    each marked candidate or not with the reason, in a batch order that starts
    with landing and docs units. The exact settings to flip, and a
    required-checks mapping per repo taken from the jobs that actually ran in the
    last 30 days.
  5. Known gaps carried out of the four review rounds, plus G6 above.
  6. Estimated compute saved, from the measured aggregate.

Four findings the runbook records

  • The version string will not change on rollout. apps/dokploy/package.json
    on canary is already v0.30.5-community.1, which is what the live instance
    reports today. Version is useless as a rollout marker; the runbook pins and
    verifies by digest throughout.
  • There is no tag to roll back to. canary, latest and
    v0.30.5-community.1 on GHCR all now point at the b0cadcd build. The
    pre-feat(build-policy): enforced remote builds, registry push, deploy-by-digest, queue coalescing, required checks #209 image is untagged, so a retention sweep can reach it. The runbook
    makes capturing the running digest and saving the image to a tar the first
    pre-flight step, before any backup.
  • Merging anything to canary moves those tags again, which is the banner
    at the top of this PR.
  • Enabling is org-wide, not per-unit. The only way to roll out in batches is
    to exclude the later batches first, then remove exclusions a batch at a time.
    80 of 89 application units and 39 of 42 compose units have no explicit
    watchPaths, so the derived default would start filtering pushes fleet-wide
    the moment it is flipped.

Checks on the file itself

LF throughout (git ls-files --eol reports i/lf w/lf). Byte-level scan for
GitHub, AWS, Slack and OpenAI token shapes, PEM headers and credential-bearing
URLs: zero matches. No secret value appears in the file; the four Notifly units
whose git URLs embed an OAuth token are referred to by unit name only.

@AminDhouib AminDhouib changed the title docs: build-once rollout runbook (owner item 18) docs: build-once rollout runbook (owner item 18) - DO NOT MERGE outside the rollout window Sep 11, 2026
Owner item 18. Operator runbook for rolling the canary head (b0cadcd, PR #209)
onto the live Dokploy instance and enabling the build-once policy per unit.

Covers: current state and default-off confirmation with file refs; pre-flight
backups (pg_dump, volume tars, rollback image tar) with verification steps;
the digest-pinned service update and its verification checklist; per-unit
candidacy for all 89 application and 42 compose units with a batch order;
the required-checks mapping per repo; the known gaps carried out of the four
review rounds; and the measured compute the change is expected to recover.

Docs only. Nothing here was executed against the live instance.
The live instance has 102 applications and 80 compose units; the audit export
the per-unit tables are built from has 89 and 42. Roughly 13 applications and
38 compose units are therefore unclassified, and the org-wide switch would
enforce them anyway. Adds that to section 1.1 with the two rules to classify
them by.
…ost figure

Recounted from applications.csv and composes.csv: 80 of 89 application units and
39 of 42 compose units have no explicit watchPaths, not 83 of 89. Names the nine
applications that do. Also attributes the 139 auto-deploy figure to spec section
1 rather than leaving it approximate.
 introduced

Two additions, both found while opening this PR.

Section 2.1 gains a third trap: dokploy.yml triggers on every push to canary
with no paths filter, so even a docs-only merge republishes the image and moves
the canary, latest and version tags. The digest capture this runbook opens with
is only valid if canary is frozen for the rollout window, and this PR is itself
such a merge.

Section 5 gains G6: pull-request.yml job pr-check (test) fails on every PR
targeting canary since #209. Five tests, all in application.real.test.ts, all
from db.query.buildPolicySettings being undefined in that file's hand-written
db mock. Evidenced against three runs of the same workflow: green before #209
(34048304936), the same five failures at the #209 head (34543739811) and at
this docs-only branch (34611157554). Inherited, not caused here, and not a
pre-existing fork baseline. The four review rounds could not see it because the
same file already failed on the reviewer's Windows host for an unrelated reason.
…ming

Three corrections after #211 and #212 landed on canary.

The republish point in section 2.1 was overstated. Step 1 reads the rollback
digest off the running service on the host, not off a tag, and a republish adds
a GHCR version rather than rewriting the old manifest, so a merge to canary
cannot invalidate it. Rewritten to say what actually follows: pin the digest you
are rolling to, and keep the docker save, because each republish leaves the
previous build untagged and that is what retention sweeps collect. Also records
that the fork is in neither Dokploy CSV, so a merge to canary redeploys nothing.

Section 3.1 and 3.2 now account for canary having moved twice since b0cadcd.
Both follow-ups are runtime-neutral - a CI trigger and a file under __test__ -
and either build is a valid rollout target. Section 3.2 now shows how to resolve
the current head's digest, and keeps the b0cadcd digest for anyone who would
rather roll the reviewed commit and nothing else.

Section 5 G6 and section 7 step 0 record that the test regression is fixed:
#212 (e92e4ad), run 34615190789, 214 files passed, 2314 passed / 1 skipped /
0 failed.
@AminDhouib
AminDhouib force-pushed the docs/build-once-rollout-runbook branch from 3020df2 to 5c76728 Compare September 11, 2026 15:41
@AminDhouib AminDhouib changed the title docs: build-once rollout runbook (owner item 18) - DO NOT MERGE outside the rollout window docs: build-once rollout runbook (owner item 18) Sep 11, 2026
@AminDhouib
AminDhouib merged commit 295df49 into canary Sep 11, 2026
3 checks passed
@AminDhouib
AminDhouib deleted the docs/build-once-rollout-runbook branch September 11, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant