diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index b9c6ffc..bccf586 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -1207,8 +1207,7 @@ a new deployment platform or automatic schema mutation. Status -Accepted for repository implementation. Production acceptance remains pending -A-owned live validation. +Accepted. A-owned P7.2 production validation passed on 2026-10-05. --- @@ -1256,9 +1255,48 @@ new application table, public endpoint, provider lock-in, or multi-region HA. Status -Accepted for repository implementation. Production scheduling, destination, -restore exercise, and milestone acceptance remain pending A-owned validation. +Accepted for repository implementation. A-owned local/off-site backup and +isolated restore validation passed on 2026-10-05, but scheduled systemd backup +failed because Docker Compose attempted to read `/srv/wctm/.env` as `wctm`. +P7.3 production acceptance remains pending a successful live scheduler proof. --- -Next decision number: D-033. +## D-033 + +Date + +2026-10-05 + +Decision + +Proposed after P7.3 closure: P7.2 is production-complete, while P7.3 still +requires a successful scheduled-backup systemd proof on authoritative +`waltpack`. Once P7.3 passes, Phase 7 would remain incomplete and pause before +P7.4. P7.4–P7.8 would remain unstarted and available for later resumption when +merchant/pilot evidence justifies them; historical provenance review would +remain assigned to P7.8. Phase 6 remains deferred and unstarted. + +Private Pilot Readiness is proposed as the next active initiative after P7.3 +closure. Its bounded sequence is +privacy/data-flow audit and merchant-facing disclosures; licensing/legal +baseline; WordPress connector admin UI polish; `wctm.walterbyte.com` onboarding +and product-surface polish; focused security/quality QA; and a private merchant +pilot with real users. Engineering is frozen except for pilot-blocking defects. +No new feature expansion or unrestricted public launch is approved. Each pilot +task requires separate authorization; this decision implements none of them. + +Reason + +The deployment and recovery evidence can support a controlled private pilot +after the scheduled-backup path passes live validation. +Real merchant usage and payment evidence should guide further infrastructure +and product investment before resuming the remaining Phase 7 or Phase 6 work. + +Status + +Proposed; not accepted until P7.3 scheduled-backup production proof passes. + +--- + +Next decision number: D-034. diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index e5dd8ac..f16a080 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -2,7 +2,7 @@ **Generated:** 2026-07-19 -**Updated:** 2026-10-04 +**Updated:** 2026-10-05 **Reason:** Maintaining implementation continuity @@ -43,7 +43,7 @@ n8n is **NOT** part of the production architecture (D-008, prototype only). --- -## 3. Architectural Decisions (D-001–D-032) +## 3. Architectural Decisions (D-001–D-033) | ID | Decision | Status | | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | @@ -79,8 +79,9 @@ n8n is **NOT** part of the production architecture (D-008, prototype only). | D-030 | P7.1 uses a temporary decrypt-only previous key, current-key-only writes, and a trusted-shell row-transactional rotation command for the exact five encrypted database fields | Accepted | | D-031 | P7.2 separates restricted backend runtime from an ephemeral explicit Prisma migration identity and defines revision-locked fail-fast deployment plus volume-preserving immutable data-service reconciliation | Accepted | | D-032 | P7.3 establishes atomic verified PostgreSQL backups, explicit retention, verified provider-neutral off-host copy, daily systemd scheduling, isolated guarded restore, DR dependencies, and target-only RPO/RTO | Accepted | +| D-033 | Proposed Phase 7 pause after P7.3 scheduled-backup proof, with Private Pilot Readiness next and Phase 6/P7.4–P7.8 deferred | Proposed | -Next decision number: **D-033**, if a future task produces a genuine +Next decision number: **D-034**, if a future task produces a genuine architectural or product decision. --- @@ -1031,7 +1032,7 @@ Final closure evidence: --- -### Phase 7 — Production Readiness (current) +### Phase 7 — Production Readiness (P7.3 scheduler proof pending) #### P7.1 — Production Security Baseline (complete; production validated) @@ -1181,7 +1182,7 @@ Dependency and automated evidence: missing or excessive privileges across all 20 application tables, while the existing privileged owner remains available for Prisma migration execution. -#### P7.2 — Production Migration & Deployment Path (repository complete; awaiting validation) +#### P7.2 — Production Migration & Deployment Path (production complete) - Implementation commit: `91e16fd5933ffa4e6386d649f21f9e2592833ed5 chore(ops): establish production @@ -1202,10 +1203,14 @@ migration and deployment path`. continuity and exact image configuration, and leaves apps stopped for cutover. - Failure guidance distinguishes safe application-image restoration from irreversible/forward-only schema effects and hands database recovery to P7.3. - Repository implementation is ready for B review; production acceptance is - pending A's combined runbook. +- A-owned production validation on `waltpack` passed at reviewed revision + `034b2f565fd6b2fc50d6942ff1a40ca16598703c`: fresh backend/bot/migration + builds, distinct migration identity, 16 applied migrations with safe repeat, + removed credential/container, restricted `wctm_runtime`, preserved immutable + PostgreSQL/Redis and named volumes, `deployment path: PASS`, local/public + health/readiness, and bounded functional smoke. P7.2 is complete. -#### P7.3 — Backup, Restore & Disaster Recovery (repository complete; awaiting validation) +#### P7.3 — Backup, Restore & Disaster Recovery (scheduled proof pending) - Implementation commit: `ddde3c705b249a29a131a3bfc96763f17b6e9cbc chore(ops): add production backup @@ -1234,9 +1239,22 @@ and recovery path`. app deployment; bounded smoke; post-deployment backup; isolated restore; and final permission/migration/backup/secret/health scans. -This repository cycle performs no VPS, production, secret, live database, -deployment, schema, migration, or product mutation. P7.2 and P7.3 are not -production-complete. P7.4+, Phase 6, and product work remain unstarted. +- A-owned backup/off-site/restore validation passed: verified custom dump + `wctm-postgres-20261005T133652Z-034b2f565fd6.dump` (2,392,191 bytes, + 16 migrations), OneDrive three-artifact copy with streamed SHA-256 PASS, + isolated network-none restore with 16 migrations/21 public tables and + critical counts `tenants=7`, `users=6`, `stores=2`, `orders=54`, cleanup, + and clean final health/role/permission/secret scans. The live systemd service + then failed with `/srv/wctm/.env` permission denied as `wctm`; the timer was + not enabled. P7.3 remains open until the corrected scheduler passes live. + +This closure branch performs no VPS, production, secret, live database, +deployment, schema, migration, or product mutation. Phase 7 remains incomplete; +P7.4–P7.8 and Phase 6 remain deferred/unstarted. D-033 and Private Pilot +Readiness are proposed pending P7.3 closure. Scheduled backup now selects the +sole PostgreSQL container via Docker Compose labels without parsing the full +app `.env`; `RCLONE_CONFIG` remains in protected external configuration. This +repository correction has not run on production here. ## 5. Current Repository Structure @@ -1245,15 +1263,15 @@ NestJS API, `telegram-bot/` for the grammY process, and `wp-content/plugins/` fo the lightweight connector. The larger `apps/`, `packages/`, and `infrastructure/` layout remains a planned target rather than current structure. -Current work branch: `fix/p7.2-production-build-baseline` from main -`75913223c214214829452d5ac7740b3a9576a218`. +Current closure branch: `docs/p7.2-p7.3-production-closure` from main +`034b2f565fd6b2fc50d6942ff1a40ca16598703c`. --- ## 6. Current Blockers -No open Phase-5 or P7.1 blocker remains. Public launch remains gated by -P7.2/P7.3 production acceptance and unstarted P7.4–P7.8. Historical provenance +P7.3 scheduled-backup production proof remains open. Public launch also remains +gated by unstarted P7.4–P7.8 and a separate A decision. Historical provenance remains unresolved and assigned to P7.8. Monitoring remains P7.4, DATE-001 P7.5, network/runtime resilience P7.6, and AuditLog structural immutability P7.7. @@ -1262,12 +1280,13 @@ P7.7. ## 7. Current Task -P7.1 is complete. Production migrated successfully from `hetz` to authoritative -`waltpack` at `/srv/wctm`; the old `hetz` WCTM deployment was removed and is not -a rollback host. This is operational evidence, not final P7.2/P7.3 acceptance. -Both still await their supported A-owned production-validation runbook on -`waltpack` and remain open. P7.4 is next only after both acceptances and remains -unstarted. Do not start P7.4+, Phase 6, or product expansion. +P7.1 and P7.2 are complete. A-owned P7.3 backup/off-site/restore checks passed +on authoritative `waltpack` at `/srv/wctm`, but the scheduled service failed +on `.env` access as `wctm` and its timer was not enabled. The earlier migration +from `hetz` succeeded; its WCTM deployment was removed and is not a rollback +host. P7.3 requires live scheduler proof. Phase 7 remains incomplete; +P7.4–P7.8 and Phase 6 are deferred/unstarted. D-033 proposes Private Pilot +Readiness next after P7.3 closure; no pilot implementation begins here. ### Last completed product milestone: M22 diff --git a/docs/MASTER-ROADMAP.md b/docs/MASTER-ROADMAP.md index d141878..c2b934e 100644 --- a/docs/MASTER-ROADMAP.md +++ b/docs/MASTER-ROADMAP.md @@ -284,7 +284,7 @@ readiness work, or authorize unrestricted public launch. ## Phase 7 — Production Readiness Status: -🔵 Current phase — next after Phase 5, by explicit A decision D-029 +🔵 In progress through P7.3; Phase 7 remains incomplete Phase 7 executes before Phase 6. It is production hardening and readiness only; it does not include pricing, billing, subscriptions, plan differentiation, @@ -294,30 +294,37 @@ commercial/product expansion. Milestones, in order: - P7.1 — Production Security Baseline — ✅ Complete -- P7.2 — Production Migration & Deployment Path — 🟡 Repository implemented; - awaiting B review and A-owned production validation -- P7.3 — Backup, Restore & Disaster Recovery — 🟡 Repository implemented; - awaiting B review and A-owned production validation -- P7.4 — Monitoring & Alerting -- P7.5 — Data & Time Correctness -- P7.6 — Network & Runtime Reliability -- P7.7 — Audit & Operational Integrity -- P7.8 — Final Launch Readiness Gate - -P7.1 is complete following accepted production validation. P7.2 and P7.3 now -have repository-side implementation for explicit least-privilege migration, -deterministic deployment, immutable database/cache image reconciliation, -verified backup/off-host retention/scheduling, isolated restore, and disaster -recovery. Both remain open pending B review and A-owned live validation; neither -is production-complete. P7.4 is next only after both are accepted. P7.4–P7.8 -and Phase 6 remain unstarted; Phase 7 is not complete. +- P7.2 — Production Migration & Deployment Path — ✅ Complete / production validated +- P7.3 — Backup, Restore & Disaster Recovery — 🟡 Backup/off-site/restore + validated; scheduled systemd backup proof pending +- P7.4 — Monitoring & Alerting — ⬜ Deferred / unstarted +- P7.5 — Data & Time Correctness — ⬜ Deferred / unstarted +- P7.6 — Network & Runtime Reliability — ⬜ Deferred / unstarted +- P7.7 — Audit & Operational Integrity — ⬜ Deferred / unstarted +- P7.8 — Final Launch Readiness Gate — ⬜ Deferred / unstarted + +P7.1 and P7.2 are complete. P7.3 local/off-site backup and isolated restore +passed, but the scheduled service failed when Docker Compose attempted to read +`/srv/wctm/.env` as `wctm`. The corrected scheduler awaits live systemd proof; +the timer remains disabled. D-033 proposes pausing Phase 7 before P7.4 and +making Private Pilot Readiness next only after P7.3 closure. P7.4–P7.8 and +Phase 6 remain deferred/unstarted; historical provenance review stays assigned +to P7.8. No feature expansion or unrestricted public launch is authorized. + +## Private Pilot Readiness — Proposed Next Initiative + +Status: ⬜ Proposed under D-033, pending P7.3 scheduled-backup proof and a +separate acceptance. Its proposed scope remains privacy/data-flow and merchant +disclosures, legal/licensing baseline, connector admin UI and +`wctm.walterbyte.com` polish, focused QA, then a real-user private merchant +pilot. No pilot implementation begins in this branch. --- ## Phase 6 — SaaS Platform Status: -⬜ Deferred / unstarted until Phase 7 completes and A separately authorizes it +⬜ Deferred / unstarted; requires a separate A decision after pilot evidence Deliverables diff --git a/docs/PHASE_BREAKDOWN.md b/docs/PHASE_BREAKDOWN.md index e3e725a..a75077b 100644 --- a/docs/PHASE_BREAKDOWN.md +++ b/docs/PHASE_BREAKDOWN.md @@ -715,10 +715,11 @@ M22 final decision: PASS. Phase 5 final decision: COMPLETE. M17–M22 and all approved MVP Telegram product features are complete, Persian/English manager UX and backend-authoritative entitlement enforcement are operational, and no open Phase-5 feature blocker remains. Phase 6 commercial SaaS work has not started; -Phase 7 production-readiness is now current by D-029, and unrestricted public -launch is not approved. +Phase 7 production-readiness was started by D-029. P7.3 remains open pending +scheduled-backup proof, so the D-033 pause is proposed, not accepted; +unrestricted public launch is not approved. -## Phase 7 — Production Readiness 🔵 Current +## Phase 7 — Production Readiness 🔵 In progress / incomplete ### P7.1 — Production Security Baseline ✅ Complete / Production Validated @@ -760,7 +761,7 @@ production identity was replaced for backend runtime by the validated restricted role. Historical release-provenance findings remain without history rewrite and are still assigned to P7.8. -### P7.2 — Production Migration & Deployment Path 🟡 Repository Complete / Awaiting Validation +### P7.2 — Production Migration & Deployment Path ✅ Complete / Production Validated - D-031 establishes an ephemeral, non-root Compose migration job containing the exact checked-out Prisma migration state. A distinct protected migration URL @@ -772,10 +773,13 @@ rewrite and are still assigned to P7.8. - Guarded PostgreSQL/Redis reconciliation requires a verified backup and exact named-volume confirmation, preserves volumes, recreates only those services on the reviewed immutable references, and verifies health/data continuity. -- Repository implementation is ready for B review. Production validation and - A acceptance remain required; P7.2 is not production-complete. +- A-owned production validation on `waltpack` passed: fresh images, distinct + migration identity, 16 applied migrations with an idempotent repeat, removed + ephemeral credential/container, restricted `wctm_runtime`, preserved named + volumes/data, corrected deployment PASS, local/public health/readiness, and + bounded product smoke. P7.2 is production-complete. -### P7.3 — Backup, Restore & Disaster Recovery 🟡 Repository Complete / Awaiting Validation +### P7.3 — Backup, Restore & Disaster Recovery 🟡 Scheduled Proof Pending - D-032 establishes atomic PostgreSQL custom-format backups, SHA-256 and readability verification, non-secret metadata, explicit 14-valid-set default @@ -787,12 +791,17 @@ rewrite and are still assigned to P7.8. - The DR runbook covers application/container, deployment, database, and total VPS loss; distinguishes APP-key decryption dependency from other service credentials; and sets target-only 24-hour RPO/eight-hour RTO objectives. -- Repository implementation is ready for B review. A must choose the off-host - destination, install/enable scheduling, and execute the combined live backup, - deployment and isolated-restore runbook. P7.3 is not production-complete. - -P7.4 remains next only after P7.2/P7.3 production acceptance. P7.4 through -P7.8 are unstarted. Phase 7 is not complete. +- A-owned production validation passed: custom dump and local SHA-256, OneDrive + three-artifact transfer with streamed remote SHA-256, isolated network-none + restore with 16 migrations/21 public tables and checked critical counts, + cleanup, and healthy final service/security scans. Scheduled systemd backup + then failed because Docker Compose attempted to read `/srv/wctm/.env` as + `wctm`; the timer was not enabled. P7.3 remains open until the corrected + scheduler passes live proof. Scheduled off-site execution retains the + explicit protected `RCLONE_CONFIG` path. + +P7.4–P7.8 remain unstarted/deferred. Phase 7 is incomplete; D-033 proposes a +pause after P7.3 closure. Historical provenance review remains assigned to P7.8. ### Approved Phase 7 order @@ -805,11 +814,23 @@ P7.8 are unstarted. Phase 7 is not complete. 7. P7.7 — Audit & Operational Integrity 8. P7.8 — Final Launch Readiness Gate +## Private Pilot Readiness ⬜ Proposed next initiative + +- Privacy/data-flow audit and merchant-facing disclosures +- Licensing/legal baseline +- WordPress connector admin UI polish +- `wctm.walterbyte.com` onboarding/product-surface polish +- Focused security/quality QA +- Private merchant pilot with real users and usage/payment evidence + +D-033 proposes the engineering freeze except for pilot-blocking defects after +P7.3 closes. No pilot implementation is authorized in this task. + ## Phase 6 — SaaS Platform ⬜ Deferred / unstarted - Subscriptions, plans, billing, dashboard, tenant administration, and usage - limits remain deferred until Phase 7 completes and A separately authorizes - Phase 6. + limits remain deferred until merchant/pilot evidence supports a separate A + decision to authorize Phase 6. ## Phase 8 — Public Launch ⬜ Planned diff --git a/docs/PROJECT_STATE.md b/docs/PROJECT_STATE.md index 0db6f4e..f63a716 100644 --- a/docs/PROJECT_STATE.md +++ b/docs/PROJECT_STATE.md @@ -11,29 +11,28 @@ Phase 5 Closure is fully implemented, reviewed, merged, migrated, deployed, and operationally validated under accepted D-028. M17–M22 and all approved MVP Telegram product features are complete; Persian/English manager UX and backend- authoritative basic entitlement enforcement are operational. Phase 4 and -M1–M16 remain complete and unchanged. By explicit A decision D-029, the current -phase is Phase 7 — Production Readiness, which executes before Phase 6. P7.1 — -Production Security Baseline is complete after successful A-owned production -validation. Phase 7 is not complete. P7.2 — Production Migration & Deployment -Path and P7.3 — Backup, Restore & Disaster Recovery are repository-implemented -but remain open pending their supported A-owned production-validation runbook -on `waltpack`. -P7.4 is next only after both are accepted. Phase 6 commercial SaaS work remains -deferred and unstarted until Phase 7 completes and A separately authorizes it. +M1–M16 remain complete and unchanged. D-029 advanced the project into Phase 7 — +Production Readiness before Phase 6. P7.1 and P7.2 — Production Migration & +Deployment Path are complete. P7.3 — Backup, Restore & Disaster Recovery has +passed live backup/off-site/restore validation but remains open after the +scheduled systemd backup failed to read `/srv/wctm/.env` through Docker Compose. +The timer was not enabled. D-033 and Private Pilot Readiness remain proposed +until the corrected scheduler passes live proof. Phase 7 is incomplete; +P7.4–P7.8 and Phase 6 remain deferred/unstarted. Unrestricted public launch is not approved. --- Current Task -P7.1 — Production Security Baseline is complete. P7.2 and P7.3 repository work -is implemented. Production migrated successfully from `hetz` to the authoritative -`waltpack` host at `/srv/wctm`; the old `hetz` WCTM deployment was removed and -is not a rollback host. The migration is operational evidence, not final P7.2/ -P7.3 acceptance. Both await their supported production-validation runbook on -`waltpack`. Neither is production-complete. P7.4 remains unstarted and is -next only after both acceptances. No product feature work is active. Do not -begin P7.4+, Phase 6, or product expansion without separate approval. +P7.2 production validation passed on authoritative `waltpack` at reviewed +revision `034b2f565fd6b2fc50d6942ff1a40ca16598703c`. P7.3 backup, +OneDrive off-site SHA-256, and isolated restore passed, but the first live +systemd scheduler start failed with `.env` permission denied as `wctm`. The +timer was not enabled. This task removes the scheduled backup's unintended +Compose `.env` dependency in the repository; A-owned live scheduler proof is +still required before P7.3 or D-033 closes. The earlier migration from `hetz` +succeeded; its WCTM deployment was removed and is not a rollback host. --- @@ -45,8 +44,8 @@ Project Version Repository -Current work branch: `fix/p7.2-production-build-baseline` from main -`75913223c214214829452d5ac7740b3a9576a218`. +Current closure branch: `docs/p7.2-p7.3-production-closure` from main +`034b2f565fd6b2fc50d6942ff1a40ca16598703c`. M22 implementation commit: `35f9e72335c8ed0c6a039497d92bed763dc68fb5 feat(entitlements): add MVP tenant @@ -786,7 +785,7 @@ WooCommerce Webhooks Current Branch -chore/p7.2-p7.3-production-operations +docs/p7.2-p7.3-production-closure --- @@ -804,7 +803,7 @@ updatable timestamp. A future approved decision must define enforcement. The normal production backend image intentionally omits the Prisma CLI. P7.2 adds a separate ephemeral operations image with exact Prisma tooling and -migration state; live validation of that path remains pending A. +migration state; A-owned live validation of that path passed on `waltpack`. --- @@ -816,8 +815,8 @@ AuditLog immutability enforcement is deferred to a future approved task. Current Blockers -No open Phase-5 or P7.1 blocker remains. Public launch remains gated by pending -P7.2/P7.3 production acceptance and unstarted P7.4–P7.8. Four historical +P7.3 scheduled-backup production proof remains open. Public launch also remains +gated by unstarted P7.4–P7.8 and a separate A decision. Four historical implementation-provenance findings remain unresolved for A's separate history- remediation decision and the P7.8 final release audit; current tracked content is clean and Git history was not rewritten. @@ -826,15 +825,15 @@ is clean and Git history was not rewritten. Next Milestone -P7.2 and P7.3 require B review and one A-owned combined live validation session. -After both are accepted, P7.4 — Monitoring & Alerting is next but unstarted. +The next gate is A-owned P7.3 systemd scheduler proof. D-033 proposes Private +Pilot Readiness afterward; P7.4–P7.8 remain deferred/unstarted. --- Last Completed -P7.1 — Production Security Baseline is the last operationally closed milestone. -P7.2/P7.3 are repository-complete only. Phase 7 remains in progress. +P7.2 — Production Migration & Deployment Path is production-complete. P7.3 and +Phase 7 remain incomplete. --- @@ -846,7 +845,7 @@ Excellent Last Updated -2026-09-16 +2026-10-05 --- @@ -886,9 +885,8 @@ P7.1 Closure State - A has formally accepted D-030. Its staged architecture and history remain unchanged. A completed the protected production execution and validation; P7.1 is closed. -- D-029 is Accepted. Phase 7 runs P7.1 through P7.8 before deferred Phase 6. - P7.2/P7.3 now await review and live validation; P7.4 and later remain - unstarted. +- D-029 remains Accepted. D-033 proposes a Phase 7 pause and Private Pilot + Readiness after P7.3 scheduler proof; P7.4 and later remain unstarted. - Production configuration rejects committed development and test secret placeholders, short bot service credentials, pilot mode, debug/verbose log levels, and unrelated cross-boundary secret reuse without reporting values. @@ -974,21 +972,22 @@ P7.1 Closure State projection and exactly one correct Telegram notification/callback, and final health/readiness. Recent permission and secret-pattern/log-leak scans were clean. One isolated ordinary HTTP 404 was non-blocking. -- P7.2/P7.3 repository implementation is complete and ready for B review, but - neither milestone is production-complete. D-031 provides explicit ephemeral +- P7.2 repository implementation and A-owned production validation are + complete. D-031 provides explicit ephemeral privileged migration plus deterministic deployment and immutable PostgreSQL/ Redis reconciliation while retaining `wctm_runtime` for backend runtime. D-032 provides verified custom-format backup, retention, off-host transfer, - daily scheduling templates, isolated restore, and DR procedures. A must run - the combined production-validation gates before acceptance. -- P7.4–P7.8, Phase 6, and product expansion remain unstarted. Monitoring stays + daily scheduling templates, isolated restore, and DR procedures. Its live + backup/off-site/restore gates passed, but scheduled systemd execution failed + on `.env` access and P7.3 remains pending. +- P7.4–P7.8, Phase 6, and product expansion remain deferred/unstarted. Monitoring stays P7.4, DATE-001 P7.5, network/runtime resilience P7.6, AuditLog structural immutability P7.7, and historical provenance/final launch gate P7.8. The provenance findings remain unresolved. --- -P7.2/P7.3 Repository Implementation State +P7.2 Closed / P7.3 Scheduled Proof Pending - P7.2 implementation commit: `91e16fd5933ffa4e6386d649f21f9e2592833ed5 chore(ops): establish production @@ -996,8 +995,9 @@ migration and deployment path`. - P7.3 implementation commit: `ddde3c705b249a29a131a3bfc96763f17b6e9cbc chore(ops): add production backup and recovery path`. -- D-031 and D-032 are Accepted for repository implementation; D-029 and D-030 - remain Accepted and unchanged. +- D-031/P7.2 are production-validated. D-032 remains accepted for repository + implementation while P7.3 awaits scheduler proof. D-029/D-030 remain + accepted; D-033 is proposed, not accepted. - `migrate` is an operations-profile Compose service built from a dedicated backend Dockerfile target. It runs non-root/read-only, contains the exact checked-out Prisma CLI/migrations, accepts a distinct credential only for one @@ -1027,10 +1027,28 @@ and recovery path`. only for a historical controlled rotation-window backup, any necessary previous key. JWT/callback/internal/bot/database secrets restore their own trust boundaries and do not decrypt stored ciphertext. -- Repository implementation is ready for B review. Neither P7.2 nor P7.3 is - production-complete until A performs and accepts live validation. No VPS, - production, secret, database, deployment, schema, migration, or product - mutation occurred during repository implementation. +- A-owned production validation on `waltpack` at + `034b2f565fd6b2fc50d6942ff1a40ca16598703c` passed. P7.2: fresh + backend/bot/migration images, separate migration identity, 16 applied Prisma + migrations with safe second execution, removed privileged container and + credential, restricted `wctm_runtime` verification, all-PASS config audit, + immutable PostgreSQL/Redis references and preserved named volumes/data, + `deployment path: PASS`, local/public health/readiness, and bounded product + smoke. P7.3 partial evidence: verified 2,392,191-byte custom dump + `wctm-postgres-20261005T133652Z-034b2f565fd6.dump`, OneDrive three-artifact + copy with streamed remote SHA-256 PASS, isolated network-none restore with + 16 migrations/21 tables and checked critical counts, cleanup, and clean final + service/security scans. The live systemd service start then failed when + Docker Compose attempted to read `/srv/wctm/.env` as `wctm`; the timer was not + enabled. Exact evidence is in the combined operations runbook. +- Scheduled backup now discovers the sole running PostgreSQL container through + reviewed Compose labels with Docker directly, then passes its ID to the + existing backup script. Protected external configuration supplies only the + non-secret project label, backup path/retention, off-site destination, and + `RCLONE_CONFIG` path. This repository correction was not run on production by C. +- P7.2 is complete; P7.3 remains pending live scheduler proof. Phase 7 remains + incomplete, P7.4–P7.8 and Phase 6 remain deferred, and D-033/Private Pilot + Readiness remain proposed. --- diff --git a/docs/SETUP.md b/docs/SETUP.md index aae0245..bf9688f 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -1088,10 +1088,23 @@ scripts/ops/backup-retention.sh --directory /var/backups/wctm --keep 14 scripts/ops/backup-retention.sh --directory /var/backups/wctm --keep 14 --apply ``` -Daily systemd templates are under `ops/systemd/`; they are not installed or -enabled by repository implementation. A must review the service user, checkout -and backup paths, Docker access, and provider-neutral rclone destination before -following the installation commands in the operations runbook. +Daily systemd templates are under `ops/systemd/`; repository implementation +does not install or enable them. For off-site scheduling, set the real absolute +`RCLONE_CONFIG` path in protected `/etc/wctm/backup.conf`. The `wctm` service +also needs the reviewed non-secret `WCTM_COMPOSE_PROJECT` label. Scheduled +backup selects one running PostgreSQL container through Docker labels and +passes its ID to the existing backup script, so it does not parse Compose or +read `/srv/wctm/.env`. Do not relax `.env` permissions or copy it into the +backup configuration. The `wctm` service account must read the separate +protected rclone config. OneDrive/OAuth remotes also +need `wctm` write access to the file and its parent directory for token +refreshes; the service allows `/var/lib/wctm/rclone` for a `0600` config under +a `0700` directory. `ProtectHome=true` blocks the interactive user's +home-directory config. The scheduled script rejects a +missing/unreadable path before backup, and rclone inherits the configured path. +Keep credentials outside Git and unit files. A must review the service user, +checkout and backup paths, Docker access, and destination before following the +installation commands in the operations runbook. Restore testing always uses a generated, non-networked PostgreSQL 16.15 container and volume and exposes no production-target option: diff --git a/docs/operations/P7_2_P7_3_RUNBOOK.md b/docs/operations/P7_2_P7_3_RUNBOOK.md index 0beb7e3..cc1497c 100644 --- a/docs/operations/P7_2_P7_3_RUNBOOK.md +++ b/docs/operations/P7_2_P7_3_RUNBOOK.md @@ -1,7 +1,47 @@ # P7.2/P7.3 Production Operations Runbook -Status: repository implementation ready for review and A-owned production -validation. Nothing in this document marks P7.2 or P7.3 production-complete. +Status: P7.2 production validation PASS on authoritative `waltpack` at reviewed +revision `034b2f565fd6b2fc50d6942ff1a40ca16598703c` (2026-10-05). P7.3 +backup/off-site/restore checks passed, but its scheduled systemd service failed; +P7.3 remains open pending a successful live scheduler proof. D-033 is proposed, +not accepted. Phase 7 remains incomplete. + +## Production validation record + +P7.2 PASS: `waltpack` built fresh backend, bot, and migration images; the +privileged migration used a distinct non-runtime identity, found all 16 Prisma +migrations applied with none pending, and remained idempotent on a second run. +The ephemeral migration container and protected credential file were removed. +Backend identity remained exactly `wctm_runtime`: `elevatedFlags=false`, +`schemaUsage=true`, `schemaCreate=false`, `databaseTemp=false`, +`migrationTableDml=false`, `ownedObjects=0`, `passed=true`. Production config +audit was all PASS. PostgreSQL and Redis were already on the reviewed immutable +references, retained `wctm_postgres_data` and `wctm_redis_data`, reported +PostgreSQL ready/Redis PONG, and passed data-continuity checks. The corrected +deployment returned `deployment path: PASS`; local and public health/readiness +passed. Bounded smoke passed web login, Tenant/account read, Telegram `/status`, +`/orders` plus detail, `/search`, `/report`, `/stock`, and Store/connector health. + +P7.3 completed checks: custom-format backup +`wctm-postgres-20261005T133652Z-034b2f565fd6.dump` was 2,392,191 bytes, +recorded 16 migrations, and passed local checksum verification. OneDrive held +the `.dump`, `.dump.json`, and `.dump.sha256` artifacts; remote content SHA-256 +passed by `streamed-sha256`. An isolated PostgreSQL restore with `network=none` +passed with 16 migrations, 21 public tables, and critical rows `tenants=7`, +`users=6`, `stores=2`, `orders=54`; generated resources were removed. Final +health/readiness, restricted role, backend/bot, PostgreSQL/Redis, and synchronized +`origin/main` checks passed; permission/database/migration error and secret- +pattern log scans were clean. The first live `systemctl start +wctm-backup.service` failed with `open /srv/wctm/.env: permission denied`: +Docker Compose tried to parse the entire application environment as service +user `wctm`. That account could read/write its rclone config, write the backup +directory, and access Docker. The timer was not enabled. The repository +scheduler correction below still requires live proof; P7.3 is not complete. + +The earlier real `hetz` to `waltpack` server migration also showed successful +custom-format PostgreSQL restore, Redis state transfer, separate recreation of +cluster-level `wctm_runtime`, and public readiness/Telegram `/status` after +cutover. The old `hetz` WCTM deployment was removed and is not a rollback host. ## Safety and identity boundary @@ -178,12 +218,51 @@ the newest valid backup are never deleted. Nothing schedules deletion unless the operator explicitly installs the scheduled wrapper or runs `--apply`. The repository templates provide a daily Ubuntu systemd timer. Review paths, -service account, Docker access, and off-host destination before installation: +service account, Docker access, and off-host destination before installation. +Set the non-secret `WCTM_COMPOSE_PROJECT` to the exact reviewed +`com.docker.compose.project` label of the running PostgreSQL container. The +scheduled script uses `docker ps` with both that project label and +`com.docker.compose.service=postgres`, requires exactly one running full +container ID, and passes it to the existing backup script. It does not invoke +Docker Compose or read `/srv/wctm/.env`, and needs no PostgreSQL username, +password, or application environment in `/etc/wctm/backup.conf`. Preserve the +production `.env` permissions and do not duplicate it. Set +`WCTM_OFFSITE_DESTINATION` and replace the example `RCLONE_CONFIG` placeholder +with the real absolute path to a readable regular rclone config in the external +`/etc/wctm/backup.conf`. The service runs as `wctm` with `ProtectHome=true`, so +Walter's interactive `/home/walter/.config/rclone/rclone.conf` is not accessible +inside the unit. Provision a separate service config from protected storage at +`/var/lib/wctm/rclone/rclone.conf`: directory owned by `wctm:wctm` at `0700`, +file owned by `wctm:wctm` at `0600`. Set `RCLONE_CONFIG` to that path. OAuth +remotes such as OneDrive must be able to save refreshed tokens, so both the +file and its parent directory must be writable by `wctm`; the unit grants write +access only to this protected directory and the backup directory. A different +config location requires a matching reviewed `ReadWritePaths` allowance. Keep +OAuth tokens in the external config, never in the unit, backup environment +file, or Git. Check access as the service account before starting the timer. A +missing or unreadable configured path makes scheduled off-site backup fail +non-zero before creating a new backup, without printing config content or +credentials. + +Docker socket access already grants the `wctm` account broad control over +containers and can expose container secrets. This correction avoids widening +direct filesystem access to `.env`; it does not make Docker access a narrow +capability. Keep the account dedicated to this operation and review its Docker +access separately. No new filesystem secret grant is required here. + +Install the example, provision the protected rclone config, replace its path +placeholder in the external environment file, then start the units: ```bash sudo install -d -o root -g wctm -m 0750 /etc/wctm +sudo install -d -o wctm -g wctm -m 0700 /var/lib/wctm/rclone sudo install -o root -g wctm -m 0640 \ ops/systemd/backup.conf.example /etc/wctm/backup.conf +# Provision /var/lib/wctm/rclone/rclone.conf from protected storage at 0600. +sudoedit /etc/wctm/backup.conf +sudo -u wctm test -r /var/lib/wctm/rclone/rclone.conf +sudo -u wctm test -w /var/lib/wctm/rclone/rclone.conf +sudo -u wctm test -w /var/lib/wctm/rclone sudo install -o root -g root -m 0644 \ ops/systemd/wctm-backup.service /etc/systemd/system/wctm-backup.service sudo install -o root -g root -m 0644 \ @@ -191,6 +270,13 @@ sudo install -o root -g root -m 0644 \ sudo systemctl daemon-reload sudo systemctl start wctm-backup.service sudo systemctl status wctm-backup.service +``` + +Require the one-shot service to finish successfully, including off-site +content SHA-256 verification, before enabling the timer. Stop on any failure. +Then: + +```bash sudo systemctl enable --now wctm-backup.timer systemctl list-timers wctm-backup.timer journalctl -u wctm-backup.service @@ -198,7 +284,8 @@ journalctl -u wctm-backup.service These are instructions for A; C does not install or enable them. The unit contains no secret. `/etc/wctm/backup.conf` holds paths and the optional remote -name; provider credentials use the operator's protected rclone configuration. +name; `RCLONE_CONFIG` is a path only. Provider credentials remain in the +operator's protected rclone configuration. The service's non-zero exit and journal entry make backup/copy/retention failures observable. Confirm the `wctm` account can access Docker without broadening unrelated host privileges before enabling the timer. diff --git a/ops/systemd/backup.conf.example b/ops/systemd/backup.conf.example index eabc379..ae98a1c 100644 --- a/ops/systemd/backup.conf.example +++ b/ops/systemd/backup.conf.example @@ -1,5 +1,13 @@ # Install as /etc/wctm/backup.conf with root:wctm ownership and mode 0640. WCTM_BACKUP_DIRECTORY=/var/backups/wctm WCTM_BACKUP_RETENTION_COUNT=14 -# Optional provider-neutral rclone remote. Configure credentials outside Git. +# Replace with the reviewed com.docker.compose.project label on the running +# PostgreSQL container (normally wctm on /srv/wctm). No database secret is needed. +WCTM_COMPOSE_PROJECT=SET_REVIEWED_COMPOSE_PROJECT +# Replace with the real absolute path of a protected config readable by wctm. +# For OAuth remotes, wctm must also write the file and its parent directory to +# persist token refreshes. The service allows /var/lib/wctm/rclone for this. +# Keep credentials in that separate file, not this env file or the unit. +RCLONE_CONFIG=SET_ABSOLUTE_READABLE_RCLONE_CONFIG_PATH +# Optional provider-neutral rclone remote. Requires the path above when enabled. # WCTM_OFFSITE_DESTINATION=remote-name:wctm-production diff --git a/ops/systemd/wctm-backup.service b/ops/systemd/wctm-backup.service index dfe3b8b..d410cb6 100644 --- a/ops/systemd/wctm-backup.service +++ b/ops/systemd/wctm-backup.service @@ -15,7 +15,7 @@ NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true -ReadWritePaths=/var/backups/wctm +ReadWritePaths=/var/backups/wctm /var/lib/wctm/rclone [Install] WantedBy=multi-user.target diff --git a/scripts/ops/scheduled-backup.sh b/scripts/ops/scheduled-backup.sh index 98139d0..250ad2f 100755 --- a/scripts/ops/scheduled-backup.sh +++ b/scripts/ops/scheduled-backup.sh @@ -4,6 +4,31 @@ umask 077 : "${WCTM_BACKUP_DIRECTORY:?WCTM_BACKUP_DIRECTORY is required}" retention_count=${WCTM_BACKUP_RETENTION_COUNT:-14} +compose_project=${WCTM_COMPOSE_PROJECT:-} +[[ "$compose_project" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || { + echo >&2 'scheduled backup refused: WCTM_COMPOSE_PROJECT must name the reviewed Compose project' + exit 64 +} +if [[ -n "${WCTM_OFFSITE_DESTINATION:-}" ]]; then + rclone_config=${RCLONE_CONFIG:-} + [[ "$rclone_config" = /* && -f "$rclone_config" && -r "$rclone_config" ]] || { + echo >&2 'scheduled backup refused: RCLONE_CONFIG must be an absolute path to a readable regular file for off-site copy' + exit 66 + } + export RCLONE_CONFIG=$rclone_config +fi +container_id=$(docker ps --no-trunc \ + --filter "label=com.docker.compose.project=$compose_project" \ + --filter 'label=com.docker.compose.service=postgres' \ + --format '{{.ID}}') || { + echo >&2 'scheduled backup failed: PostgreSQL container discovery failed' + exit 1 +} +[[ "$container_id" =~ ^[0-9a-f]{64}$ ]] || { + echo >&2 'scheduled backup refused: expected exactly one running PostgreSQL container for the reviewed Compose project' + exit 1 +} +export WCTM_POSTGRES_CONTAINER=$container_id result_file=$(mktemp) trap 'rm -f -- "$result_file"' EXIT HUP INT TERM diff --git a/scripts/ops/test-scheduled-backup.sh b/scripts/ops/test-scheduled-backup.sh new file mode 100755 index 0000000..4e41d02 --- /dev/null +++ b/scripts/ops/test-scheduled-backup.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +set -euo pipefail + +fixture=$(realpath scripts/ops/test-fixtures/rclone) +workspace=$(mktemp -d) +trap 'chmod 0600 "$workspace/rclone.conf" "$workspace/.env" 2>/dev/null || true; rm -rf -- "$workspace"' EXIT +mkdir -p "$workspace/scripts/ops" "$workspace/bin" "$workspace/backups" +cp scripts/ops/scheduled-backup.sh scripts/ops/backup-postgres.sh \ + scripts/ops/offsite-rclone.sh "$workspace/scripts/ops/" + +cat >"$workspace/scripts/ops/backup-retention.sh" <<'SH' +#!/usr/bin/env bash +printf 'retention\n' >>"$TEST_WORKSPACE/operations.log" +SH +cat >"$workspace/bin/docker" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >>"$TEST_WORKSPACE/docker.log" +case "${1:-}" in + ps) + [[ "$*" = "ps --no-trunc --filter label=com.docker.compose.project=wctm --filter label=com.docker.compose.service=postgres --format {{.ID}}" ]] || exit 99 + case "${TEST_CONTAINER_MATCHES:-one}" in + one) printf '%s\n' "$TEST_EXPECTED_CONTAINER" ;; + none) ;; + multiple) printf '%s\n%s\n' "$TEST_EXPECTED_CONTAINER" "$TEST_SECOND_CONTAINER" ;; + *) exit 99 ;; + esac + ;; + exec) + shift + if [[ "${1:-}" = -i ]]; then + shift + [[ "${1:-}" = "$TEST_EXPECTED_CONTAINER" && "${2:-}" = pg_restore && "${3:-}" = --list ]] || exit 99 + cat >/dev/null + exit 0 + fi + [[ "${1:-}" = "$TEST_EXPECTED_CONTAINER" ]] || exit 99 + shift + if [[ "${1:-}" = pg_dump && "${2:-}" = --version ]]; then + printf 'pg_dump (PostgreSQL) 16.15\n' + exit 0 + fi + [[ "${1:-}" = sh && "${2:-}" = -c ]] || exit 99 + case "${3:-}" in + *'current_database()'*) printf 'wctm_test\n' ;; + *'_prisma_migrations'*) printf '16\n' ;; + *'show server_version'*) printf '16.15\n' ;; + *'pg_dump -U'*) printf 'PGDMP synthetic local test data\n' ;; + *) exit 99 ;; + esac + ;; + compose) + echo >&2 'test failed: scheduled backup invoked Docker Compose' + exit 99 + ;; + *) exit 99 ;; +esac +SH +cat >"$workspace/bin/git" <<'SH' +#!/usr/bin/env bash +[[ "$*" = 'rev-parse HEAD' ]] || exit 99 +printf '%s\n' '123456789abcdef0123456789abcdef012345678' +SH +cat >"$workspace/bin/rclone" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +[[ ${RCLONE_CONFIG:-} = "$TEST_EXPECTED_CONFIG" ]] || { + echo >&2 'test failed: rclone did not receive configured path' + exit 1 +} +exec "$TEST_RCLONE_FIXTURE" "$@" +SH +chmod +x "$workspace"/scripts/ops/*.sh "$workspace"/bin/* + +export TEST_WORKSPACE=$workspace TEST_RCLONE_FIXTURE=$fixture +export TEST_EXPECTED_CONFIG="$workspace/rclone.conf" +export TEST_EXPECTED_CONTAINER=$(printf 'a%.0s' {1..64}) +export TEST_SECOND_CONTAINER=$(printf 'b%.0s' {1..64}) +export WCTM_BACKUP_DIRECTORY="$workspace/backups" +export WCTM_COMPOSE_PROJECT=wctm +export PATH="$workspace/bin:$PATH" +printf '[testremote]\ntype = local\n' >"$TEST_EXPECTED_CONFIG" +chmod 0600 "$TEST_EXPECTED_CONFIG" +export RCLONE_CONFIG=$TEST_EXPECTED_CONFIG +printf 'unreadable production env sentinel\n' >"$workspace/.env" +chmod 000 "$workspace/.env" +[[ ! -r "$workspace/.env" ]] || { + echo >&2 'test setup failed: .env is still readable' + exit 1 +} + +run_scheduled() { + (cd "$workspace" && scripts/ops/scheduled-backup.sh) +} + +export WCTM_OFFSITE_DESTINATION="testremote:$workspace/native" +run_scheduled >"$workspace/native.log" 2>&1 +grep -q 'content_sha256=verified method=native-sha256' "$workspace/native.log" +dump=$(find "$workspace/backups" -maxdepth 1 -name 'wctm-postgres-*.dump' -print) +[[ -n "$dump" && -f "$dump.json" && -f "$dump.sha256" ]] +(cd "$(dirname "$dump")" && sha256sum -c "$(basename "$dump").sha256") >/dev/null +grep -q '"format": "postgresql-custom"' "$dump.json" +grep -q '"completedMigrationCount": 16' "$dump.json" +[[ -f "$workspace/native/$(basename "$dump")" && -f "$workspace/native/$(basename "$dump").json" && -f "$workspace/native/$(basename "$dump").sha256" ]] +if grep -q '^compose ' "$workspace/docker.log"; then + echo >&2 'test failed: backup required Docker Compose and its .env' + exit 1 +fi + +rm -f "$workspace/backups"/wctm-postgres-*.dump* +export WCTM_OFFSITE_DESTINATION="testremote:$workspace/streamed" +WCTM_TEST_RCLONE_NATIVE_HASH=unavailable run_scheduled >"$workspace/streamed.log" 2>&1 +grep -q 'content_sha256=verified method=streamed-sha256' "$workspace/streamed.log" + +rm -f "$workspace/backups"/wctm-postgres-*.dump* +export WCTM_OFFSITE_DESTINATION="testremote:$workspace/corrupt" +if WCTM_TEST_RCLONE_NATIVE_HASH=unavailable WCTM_TEST_RCLONE_CORRUPT_DUMP=true \ + run_scheduled >"$workspace/corrupt.log" 2>&1; then + echo >&2 'test failed: same-size remote corruption was accepted' + exit 1 +fi +dump=$(find "$workspace/backups" -maxdepth 1 -name 'wctm-postgres-*.dump' -print) +[[ $(wc -c <"$workspace/corrupt/$(basename "$dump")" | tr -d ' ') = $(wc -c <"$dump" | tr -d ' ') ]] +grep -q 'remote dump SHA-256 mismatch' "$workspace/corrupt.log" + +docker_calls=$(wc -l <"$workspace/docker.log" | tr -d ' ') +WCTM_BACKUP_DIRECTORY='' run_scheduled >"$workspace/missing-backup-directory.log" 2>&1 && { + echo >&2 'test failed: missing backup directory was accepted' + exit 1 +} +grep -q 'WCTM_BACKUP_DIRECTORY is required' "$workspace/missing-backup-directory.log" +WCTM_COMPOSE_PROJECT='' run_scheduled >"$workspace/missing-project.log" 2>&1 && { + echo >&2 'test failed: missing Compose project was accepted' + exit 1 +} +grep -q 'WCTM_COMPOSE_PROJECT must name the reviewed Compose project' "$workspace/missing-project.log" +[[ $(wc -l <"$workspace/docker.log" | tr -d ' ') = "$docker_calls" ]] + +TEST_CONTAINER_MATCHES=none run_scheduled >"$workspace/no-container.log" 2>&1 && { + echo >&2 'test failed: missing PostgreSQL container was accepted' + exit 1 +} +grep -q 'expected exactly one running PostgreSQL container' "$workspace/no-container.log" +TEST_CONTAINER_MATCHES=multiple run_scheduled >"$workspace/multiple-containers.log" 2>&1 && { + echo >&2 'test failed: ambiguous PostgreSQL container was accepted' + exit 1 +} +grep -q 'expected exactly one running PostgreSQL container' "$workspace/multiple-containers.log" + +RCLONE_CONFIG='' run_scheduled >"$workspace/missing-rclone.log" 2>&1 && { + echo >&2 'test failed: missing RCLONE_CONFIG was accepted' + exit 1 +} +grep -q 'RCLONE_CONFIG must be an absolute path to a readable regular file' "$workspace/missing-rclone.log" +chmod 000 "$TEST_EXPECTED_CONFIG" +[[ ! -r "$TEST_EXPECTED_CONFIG" ]] || { + echo >&2 'test setup failed: rclone config is still readable' + exit 1 +} +if run_scheduled >"$workspace/unreadable-rclone.log" 2>&1; then + echo >&2 'test failed: unreadable RCLONE_CONFIG was accepted' + exit 1 +fi +grep -q 'RCLONE_CONFIG must be an absolute path to a readable regular file' "$workspace/unreadable-rclone.log" + +if grep -q 'unreadable production env sentinel' "$workspace"/*.log; then + echo >&2 'test failed: production env content reached a log' + exit 1 +fi +if grep -q 'scheduled backup: PASS' "$workspace/corrupt.log" "$workspace/missing-backup-directory.log" "$workspace/missing-project.log" "$workspace/no-container.log" "$workspace/multiple-containers.log" "$workspace/missing-rclone.log" "$workspace/unreadable-rclone.log"; then + echo >&2 'test failed: unsuccessful backup reported PASS' + exit 1 +fi +echo 'scheduled backup: PASS no-compose-env exact-container missing-config rclone-path custom-format SHA-256 offsite-integrity secret-safe-logs'