You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Seed the dedicated wedding backup credential and repoint the ExternalSecret and ObjectStore #3252
The mechanism for seeding an externally-sourced R2 credential already exists and is declarative —
measured on main:
k8s/bases/infrastructure/vault-seed/push-secret-seed-r2-credentials.yaml pushes r2_access_key_id / r2_secret_access_key from the per-environment variables-cluster Secret
into OpenBao at infrastructure/backup/r2, on a 1h refreshInterval (deliberately not
push-once, so it re-seeds after OpenBao data loss).
k8s/bases/apps/wedding-app/external-secret-db-backup.yaml then reads that shared path.
k8s/bases/apps/wedding-app/object-store.yaml writes to s3://${r2_bucket}/cnpg/wedding-db.
So isolating the tenant is an additive change to the same pattern: one more seeded path, and two
references repointed.
Expected outcome
The wedding database backup reads a dedicated OpenBao path seeded from the dedicated credential,
and writes to the dedicated bucket. Retirement of shared recovery access, including observed
denial of the shared destination, remains governed by #3253 and the security parent #2867.
Smallest useful change, in order
Add the dedicated credential to the per-environment encrypted bootstrap Secret and add a matching data entry to push-secret-seed-r2-credentials.yaml, seeding a dedicated path
(e.g. apps/wedding-app/backup/r2).
Repoint external-secret-db-backup.yaml's two remoteRef.keys at that path.
Repoint object-store.yaml's destinationPath at the dedicated bucket.
Acceptance criteria
The dedicated credential is seeded declaratively and survives a from-zero rebuild (same
mechanism and refreshInterval semantics as the shared one).
The wedding ExternalSecret references the dedicated path, not infrastructure/backup/r2.
The ObjectStore references the dedicated destination.
A maintained local S3-compatible fixture exercises the same manifest and secret-delivery
path using disposable credentials. The original MinIO implementation was replaced by
SeaweedFS in the reviewed fixture maintenance, preserving these acceptance semantics.
CI proves both references point at the dedicated path/destination, so a regression to the
shared credential fails the build.
Delivery: #4309 merged on 2026-10-01. The dedicated archive is active, a fresh backup completed, and catalogue catch-up proved that backup and continuous WAL. #4316 and protected main run 36856764796 proved a dedicated-only restore and cleanup. #4320/#4328 and protected main run 36889607396 completed bootstrap acceptance. After #4346's fixture maintenance, protected main run 36955408481 again completed empty-store seeding, periodic reseeding, controller-projected storage access and cleanup. Fresh scoped reads on 2026-10-02 confirm the dedicated projection is Ready and the database selects the dedicated archive. Shared recovery access remains active pending the separate denial/retirement gates in #3253; closing this delivery does not close #2867.
#2735 - fix(velero): isolate kopia repository password [merged] #3086 - security(minio): read the root credential from a file instead of the environment [merged] #3109 - fix(deploy): avoid redundant GHCR node reboots [merged] #3203 - security(vault-backup): disposition the last two CKV_K8S_40 findings as one deferred exception [merged]
📝 Issue Planner
Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.
Create Plan
🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your .coderabbit.yaml:
issue_enrichment:
auto_enrich:
enabled: false
💬 Have feedback or questions? Drop into our discord!
Unblocked — the dependency shipped. This issue's blocker read #3251 | last-verified 2026-09-03: **not provisioned**, and #3251 is now closed as completed: the
dedicated bucket and bucket-scoped identity exist, denial against the shared bucket was observed
directly, and the encrypted per-environment handoff merged in #3690 and is verified present and
registered on main. The credential this issue needs to seed therefore exists.
Removed the blocked label and the stale blocker line, so oldest-first selection can pick this up
rather than skipping it. ⚠️ Worth noting why this needed a human-shaped check: a blocker line citing
an issue that has since closed still reads as a live blocker to every lane, because the skip test
reads the label and the record, not the referenced issue's state. Nothing expires a label. This record
had also been sitting on a last-verified date of 2026-09-03 — six days stale — and carried no <blocker-kind> token, so it was reported as conforming-but-legacy rather than as needing a re-check.
What remains here is engineering work, not waiting: seed the credential through the vault-seed PushSecret, repoint the ExternalSecret and ObjectStore at the dedicated destination, exercise the
same path against local MinIO without production credentials, and add the CI assertion that a
regression to the shared credential fails the build. The staging manifests #3690 landed
(external-secret-db-backup-dedicated.yaml, push-secret-seed-wedding-db-backup-r2.yaml) are the
starting point.
⚠️ Until this lands, wedding-db still uses the shared backup identity — the isolation is
available, not in force. That is the whole point of the remaining cutover.
The production verifier proves the shared store is still active and the dedicated one is correctly wired. Steps 1 and 2 of "Smallest useful change" are therefore staged, not pending.
Acceptance criterion conflict, decided: "Local MinIO exercises the same path" contradicts fix(backup): stage Wedding dedicated R2 store #3727. That PR deliberately removed the local dedicated credential rather than present an untested local bucket as proof. I'm treating the CI wiring tests, which already fail on an early cutover, a wrong destination or credential reuse, as the regression guard. The local criterion is superseded, unless someone adds a real local bucket that the staging tests exercise.
Timing: the wedding database finished a data-loss recovery a few hours ago. This cutover needs a watched deploy window, and each step must be checked before the next one runs. It should not merge unattended.
Remaining cutover split into three ordered sub-issues
The 2026-09-13 status above lists what is left. Each step changes production backup data and needs its own review and its own production run, so I have split them into separate issues, in the order they must happen:
Delivered and verified. #4320 and #4328 are merged. Protected main run 36889607396, attempt 1 at 35ebd80429cc25b6ce958ad9080aef4406626045, completed successfully.
The operator receipt explicitly reports all four acceptance outcomes: empty-store seeding, periodic reseeding after loss, storage write/read using only the controller-projected credential, and cleanup. The repair stage also checks the source credential was not rewritten to trigger recovery. A separate cleanup step succeeded, and independent read-only API checks confirm both the disposable namespace and its owned controller policy are absent.
The first failed, cleaned run remains useful RED evidence; its pullability blocker was fixed at the root cause rather than bypassed. This completes this issue’s bootstrap acceptance. Shared recovery-access retirement remains governed separately by #3253.
Completed acceptance was revalidated on 2026-10-02. The active database selects the dedicated archive and the dedicated secret projection is Ready. The protected main fixture acceptance run is successful at the merged fixture-maintenance revision; the earlier bootstrap acceptance run and dedicated restore proof remain recorded.
The checklist now reflects the reviewed S3-compatible fixture implementation and its completed outcomes. This closes the bootstrap delivery only. Shared-access denial and retirement remain open in #3253, and the security parent #2867 remains open until that work is verified.
Activity
coderabbitai commentedon Aug 19, 2026
🔗 Related PRs
#2735 - fix(velero): isolate kopia repository password [merged]
#3086 - security(minio): read the root credential from a file instead of the environment [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]
#3203 - security(vault-backup): disposition the last two CKV_K8S_40 findings as one deferred exception [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
chore: agent-claim #3252 nonce=e6b383bb3c2ad5be6e27e415ef0ea091
devantler commentedon Sep 9, 2026
Unblocked — the dependency shipped. This issue's blocker read
#3251 | last-verified 2026-09-03: **not provisioned**, and #3251 is now closed as completed: thededicated bucket and bucket-scoped identity exist, denial against the shared bucket was observed
directly, and the encrypted per-environment handoff merged in #3690 and is verified present and
registered on
main. The credential this issue needs to seed therefore exists.Removed the⚠️ Worth noting why this needed a human-shaped check: a blocker line citing
blockedlabel and the stale blocker line, so oldest-first selection can pick this uprather than skipping it.
an issue that has since closed still reads as a live blocker to every lane, because the skip test
reads the label and the record, not the referenced issue's state. Nothing expires a label. This record
had also been sitting on a
last-verifieddate of 2026-09-03 — six days stale — and carried no<blocker-kind>token, so it was reported as conforming-but-legacy rather than as needing a re-check.What remains here is engineering work, not waiting: seed the credential through the vault-seed
PushSecret, repoint theExternalSecretandObjectStoreat the dedicated destination, exercise thesame path against local MinIO without production credentials, and add the CI assertion that a
regression to the shared credential fails the build. The staging manifests #3690 landed
(
external-secret-db-backup-dedicated.yaml,push-secret-seed-wedding-db-backup-r2.yaml) are thestarting point.
wedding-dbstill uses the shared backup identity — the isolation isavailable, not in force. That is the whole point of the remaining cutover.
chore: agent-claim #3252 nonce=9b54460900a566ceea1e6f618d4e89cf
devantler commentedon Sep 13, 2026
Where this actually stands (2026-09-13)
The body above predates two merges. Anyone picking this up should start here:
Already done:
seed-wedding-db-backup-r2pushes toapps/wedding-app/backup/r2, andwedding-db-backup-r2-dedicatedreads it.wedding-db-dedicatedObjectStore on the dedicated bucket.What remains is one controlled cutover, in this order (per fix(backup): stage Wedding dedicated R2 store #3727 and
docs/dr/runbook.md):wedding-db-dedicated.The shared credential stays active until Verify the dedicated wedding backup identity, retire shared access, and update the DR runbooks #3253 verifies denial.
Acceptance criterion conflict, decided: "Local MinIO exercises the same path" contradicts fix(backup): stage Wedding dedicated R2 store #3727. That PR deliberately removed the local dedicated credential rather than present an untested local bucket as proof. I'm treating the CI wiring tests, which already fail on an early cutover, a wrong destination or credential reuse, as the regression guard. The local criterion is superseded, unless someone adds a real local bucket that the staging tests exercise.
Timing: the wedding database finished a data-loss recovery a few hours ago. This cutover needs a watched deploy window, and each step must be checked before the next one runs. It should not merge unattended.
chore: agent-claim #3252 nonce=43f21b0c57c56ae9d75d814a10ecbaeb
devantler commentedon Sep 13, 2026
Remaining cutover split into three ordered sub-issues
The 2026-09-13 status above lists what is left. Each step changes production backup data and needs its own review and its own production run, so I have split them into separate issues, in the order they must happen:
wedding-db-dedicatedand prove new backups and WAL land there. Blocked by Mirror the shared Wedding backup catalogue into the dedicated bucket and prove parity #3767.#3253 (verify the dedicated identity and retire the shared credential) stays last. Once #3769 has a recorded proof, it can start.
Two constraints I found while scoping #3767 that the earlier plan did not state:
The recovery workflow from #3738 finished successfully at 2026-09-13T02:03Z, so nothing from the incident blocks this sequence.
devantler commentedon Oct 1, 2026
Delivered and verified. #4320 and #4328 are merged. Protected main run 36889607396, attempt 1 at
35ebd80429cc25b6ce958ad9080aef4406626045, completed successfully.The operator receipt explicitly reports all four acceptance outcomes: empty-store seeding, periodic reseeding after loss, storage write/read using only the controller-projected credential, and cleanup. The repair stage also checks the source credential was not rewritten to trigger recovery. A separate cleanup step succeeded, and independent read-only API checks confirm both the disposable namespace and its owned controller policy are absent.
The first failed, cleaned run remains useful RED evidence; its pullability blocker was fixed at the root cause rather than bypassed. This completes this issue’s bootstrap acceptance. Shared recovery-access retirement remains governed separately by #3253.
chore: agent-claim #3252 nonce=a29c6f84b1562a6340bee1f790cce770
devantler commentedon Oct 2, 2026
Completed acceptance was revalidated on 2026-10-02. The active database selects the dedicated archive and the dedicated secret projection is Ready. The protected main fixture acceptance run is successful at the merged fixture-maintenance revision; the earlier bootstrap acceptance run and dedicated restore proof remain recorded.
The checklist now reflects the reviewed S3-compatible fixture implementation and its completed outcomes. This closes the bootstrap delivery only. Shared-access denial and retirement remain open in #3253, and the security parent #2867 remains open until that work is verified.
chore: agent-claim #3252 nonce=1093d43258f824e1ecd8370d6591760a