Validate provider-target authority product-by-product

[cbusillo]

Intent

Collect live evidence that provider-target identity authority works across generic-web, VeriReel, SYO, and Odoo before retiring compatibility.

Current Status

Active after #1104 merged and deployed.

Cutover evidence:

• PR Cut over provider-target authority reads #1113 merged at 3cfd9ddaa6826a56a6edec28fb2093f0d06eaf91.
• Main CI run 26831888615 succeeded.
• Main Security run 26831888503 succeeded.
• Main CodeQL run 26831888738 succeeded.
• Deploy Launchplane run 26832039279 succeeded.
• Post-deploy health returned status: ok, storage_backend: postgres.

Next validation action: run Provider Target Operations audit for phase-two-initial, then inspect product-by-product evidence before any #1106 retirement work.

Finish Line

Each product family has clean audit, deploy/promotion/read evidence, and health evidence showing no compatibility fallback was needed.

Acceptance Criteria

• generic-web lower-risk lane validates resolver shape first.
• VeriReel testing validates app target identity, health URLs, promotion/rollback surface, and stable environment reads before prod.
• SYO testing/prod validate product profile/lane behavior and context-cutover assumptions.
• Odoo non-prod stable validates compose target identity, post-deploy, runtime identity, overrides, backup gates, and target replacement planning before prod.
• Prod lanes run only after lower-risk evidence is clean.
• Evidence is recorded in the parent or owning issue current status without secrets.

Quality Gates

Audit zero mismatches, dry-run deploy/promotion identity parity, service route deploy or promotion evidence, health/readiness checks, and CI/deploy health after merges.

[cbusillo]

Validation Progress — 2026-06-02

Post-cutover validation has started after #1113 and the #1114 authz-grant unblock.

Provider-target authority audit:

• Provider Target Operations audit run 26832757071 on main / 3cfd9ddaa6826a56a6edec28fb2093f0d06eaf91 succeeded.
• All Phase Two initial lanes returned status=ok, blocked=0, warnings=0, counts={"complete":1}:
  • discord-blue/prod
  • verireel/testing
  • verireel/prod
  • cm/testing
  • cm/prod
  • opw/testing
  • opw/prod

Validation workflow authz unblock:

• Initial Live Target Runtime dry-runs failed with authorization_denied because live-target-runtime.yml only had managed grants for SYO.
• PR Grant runtime validation workflow for Phase Two lanes #1114 merged at 8f8b43da8180e4ed50450788f2805282a9b92a8c to add managed live-target-runtime plan/apply grants for Phase Two validation scopes.
• Main CI 26833348276, Security 26833349131, CodeQL 26833347836, and Deploy Launchplane 26833503372 succeeded. Post-deploy health returned status=ok, storage_backend=postgres.

Live Target Runtime dry-run evidence after #1114 deploy:

• verireel/testing: run 26833691272 succeeded. Resolved target ver-testing-app (application), deploy not triggered, env not updated. Dry-run reports changed runtime keys and runtime_key_safety.status=skipped; this needs product-specific review before any apply/deploy.
• cm/testing: run 26833692919 succeeded. Resolved target cm-testing (compose), changed keys [], runtime key safety pass, env not updated, deploy not triggered.
• opw/testing: run 26833694777 succeeded. Resolved target opw-testing (compose), changed keys [], runtime key safety pass, env not updated, deploy not triggered.
• discord-blue/prod: run 26833696151 failed after authz succeeded with runtime_key_safety_failed / binding_disabled, trace launchplane_req_8a5867d2f9d141f3a44a9f03be0105d3.

Current blocker:

• Discord Blue live-target runtime validation is blocked on runtime key-safety binding being disabled. This is product/runtime configuration, not a provider-target authority mismatch.
• VeriReel testing has provider-target identity evidence but changed-key/key-safety-skipped dry-run output needs review before using runtime sync as stronger validation.

Next safe step: inspect/repair the Discord Blue runtime key-safety binding path or use a narrower read-only product environment validation surface for Discord Blue before proceeding to prod-lane validation and #1106 retirement.

[cbusillo]

Update while continuing Phase Two validation:

• discord-blue/prod live-target-runtime dry-run exposed a legitimate managed-secret blocker: runtime key-safety failed with binding_disabled for DISCORD_TOKEN.
• Root cause: product onboarding had created the expected DISCORD_TOKEN binding as a disabled placeholder, but Launchplane did not yet have a configured managed runtime secret binding for discord-blue/prod.
• Opened fix PR: Retire managed secret placeholders on config apply #1115

PR #1115 makes the product-config/onboarding path safe for the actual secret configuration step:

• Product-config retires same-route disabled runtime secret placeholders after writing the configured managed secret.
• Product-config now fails closed before writing if an existing configured duplicate would leave runtime key-safety ambiguous.
• Product onboarding seed re-imports preserve existing configured bindings instead of recreating disabled placeholders.
• Existing binding created_at is preserved on onboarding re-import.

Local validation for #1115:

• changed-file ruff check: pass
• changed-file ruff format --check: pass
• mypy control_plane tests: pass
• markdownlint-cli2 docs/secrets.md: pass
• focused onboarding/runtime/product-config tests: 89 tests, pass
• full uv run python -m unittest: 1,971 tests, pass

After #1115 is merged and deployed, the remaining operational unblock is to configure the real DISCORD_TOKEN through product-config/operator UI or a private local-operator source, then rerun discord-blue/prod live-target-runtime dry-run/apply.

[cbusillo]

PR #1115 is merged and deployed.

Evidence:

• PR: Retire managed secret placeholders on config apply #1115
• Merge commit: 1329f771eef3a60bd807843bcb5b66a13739e295
• Main CI: https://github.com/cbusillo/launchplane/actions/runs/26836707240 — pass
• Main Security: https://github.com/cbusillo/launchplane/actions/runs/26836707119 — pass
• Main CodeQL: https://github.com/cbusillo/launchplane/actions/runs/26836707150 — pass
• Deploy Launchplane: https://github.com/cbusillo/launchplane/actions/runs/26836873214 — pass
• Health check: status=ok, storage_backend=postgres, trace launchplane_req_f1fa23e603104daf87d13dd0537459c0

What changed:

• Product-config now retires same-route disabled runtime secret placeholders after writing the configured managed secret.
• Product-config fails closed before writing if an existing configured duplicate would leave runtime key-safety ambiguous.
• Product onboarding seed re-imports preserve existing configured bindings instead of recreating disabled placeholders.
• Existing secret binding created_at survives onboarding re-import.

Remaining blocker:

• Configure the real DISCORD_TOKEN managed secret for discord-blue/prod through product-config/operator UI or a private local-operator source. Do not pass the plaintext token through issues, PRs, chat, CLI args, or workflow logs.
• Then rerun discord-blue/prod live-target-runtime dry-run/apply.

[cbusillo]

Discord Blue runtime validation unblocked — 2026-06-02

The Discord Blue managed-secret/runtime sync blocker is resolved and validated.

Fixes merged/deployed:

• PR Scope live target runtime sync to product expectations #1116: Scope live target runtime sync to product expectations #1116
  • Merge commit: 93723122d390049015098a2facc1a5438d8019a6
  • CI: https://github.com/cbusillo/launchplane/actions/runs/26839751423 — pass
  • Security: https://github.com/cbusillo/launchplane/actions/runs/26839749796 — pass
  • CodeQL: https://github.com/cbusillo/launchplane/actions/runs/26839750904 — pass
  • Deploy Launchplane: https://github.com/cbusillo/launchplane/actions/runs/26839933620 — pass
  • Health: status=ok, storage_backend=postgres, trace launchplane_req_524b9c576ad04c68bf8bec57ce9088ea
• PR Declare Discord Blue expected runtime config #1117: Declare Discord Blue expected runtime config #1117
  • Merge commit: 7dd6ccee7991f3057316dd780c103b59eb97389a
  • CI: https://github.com/cbusillo/launchplane/actions/runs/26840506550 — pass
  • Security: https://github.com/cbusillo/launchplane/actions/runs/26840505947 — pass
  • CodeQL: https://github.com/cbusillo/launchplane/actions/runs/26840507473 — pass
  • Deploy Launchplane: https://github.com/cbusillo/launchplane/actions/runs/26840649694 — pass
  • Health: status=ok, storage_backend=postgres, trace launchplane_req_17eb5c1d16dd40c1bee4a891d404dffd

What changed:

• Live target runtime sync now filters desired env to the selected product profile lane's expected runtime environment keys and runtime managed-secret binding keys.
• The legacy CLI path now requires --product, closing the unscoped mutation bypass.
• Runtime key-safety evaluates matching global/context/instance runtime secret bindings for product-declared runtime secrets.
• Discord Blue seed onboarding now declares the expected runtime contract for discord-blue/prod:
  • runtime env key: DISCORD_BLUE_STATE_DIR
  • managed runtime secret binding: DISCORD_TOKEN

Operational evidence:

• Product-config dry-run configured DISCORD_TOKEN from the existing private LXC config source without printing plaintext: trace launchplane_req_632faedbe3d84db888af522c7ec8ff7e
• Product-config apply created the managed runtime secret binding: trace launchplane_req_36c8ceadfcda4411a37f0674a523a5fb
• Pre-scope Discord Blue dry-run exposed unsafe global-key blast radius: run 26838004882; it was not applied.
• After PR Scope live target runtime sync to product expectations #1116, Discord Blue dry-run failed closed with runtime_environment_empty because the product profile had not yet declared expected runtime keys: run 26840123776, trace launchplane_req_46f4a65112934f75a0b7d03598f669f7.
• Seed import apply updated only discord-blue-product-onboarding: run 26843047661, trace launchplane_req_b1b47966c20e4b30931e3ae19f7bd2b2.
• Discord Blue dry-run after seed import passed with only DISCORD_TOKEN changed and runtime key-safety pass: run 26843096579, trace launchplane_req_04991176f3604ebaa43cf41cfb89aeae.
• Discord Blue apply+deploy succeeded: run 26843136307, trace launchplane_req_1b1008e0a3434e7aa36ebd1ac3790cf8.
  • Applied: true
  • Env updated: true
  • Verification: pass
  • Verified key count: 2
  • Runtime key-safety: pass for DISCORD_TOKEN
  • Deploy result: deployment=0kbYiCdw1VAti87H0bXdb status=done
• Final Discord Blue steady-state dry-run passed: run 26843170548, trace launchplane_req_0828284a361a4c23b9a73148dc6e1399.
  • changed_keys=[]
  • missing_keys=[]
  • desired key count 2
  • unchanged key count 2
  • runtime key-safety pass

Result: discord-blue/prod is now Launchplane-managed for its declared runtime contract and no unrelated Odoo/GitHub/global keys are part of the live sync delta. Continue #1105 with the remaining product-by-product validation lanes; #1106 should still wait until the rest of the live paths are proven clean.

[cbusillo]

VeriReel testing runtime validation — 2026-06-02

verireel/testing live-target-runtime validation is now clean after the product-scoped sync and runtime key-safety policy updates.

Fixes and policy evidence:

• PR Classify VeriReel runtime secret bindings #1118: Classify VeriReel runtime secret bindings #1118
  • Merge commit: 7383269c6624bb395d43807f6c490c7191018ea9
  • CI: https://github.com/cbusillo/launchplane/actions/runs/26844184054 — pass
  • Security: https://github.com/cbusillo/launchplane/actions/runs/26844184892 — pass
  • CodeQL: https://github.com/cbusillo/launchplane/actions/runs/26844184254 — pass
  • Deploy Launchplane: https://github.com/cbusillo/launchplane/actions/runs/26844340502 — pass
  • Health: status=ok, storage_backend=postgres, trace launchplane_req_4174cacaa01f45bba853df491ed06f94
• PR Allow VeriReel runtime bindings in testing #1119: Allow VeriReel runtime bindings in testing #1119
  • Merge commit: 9dc9860f726199b1bbdd7c43ed6d969f610a32b8
  • CI: https://github.com/cbusillo/launchplane/actions/runs/26844788344 — pass
  • Security: https://github.com/cbusillo/launchplane/actions/runs/26844788402 — pass
  • CodeQL: https://github.com/cbusillo/launchplane/actions/runs/26844788341 — pass
  • Deploy Launchplane: https://github.com/cbusillo/launchplane/actions/runs/26844956395 — pass
  • Health: status=ok, storage_backend=postgres, trace launchplane_req_edcf72b13b834980b52ea668d2d088f3
• Runtime key-safety seed import after Allow VeriReel runtime bindings in testing #1119: run 26845083051, trace launchplane_req_57d93b617ed34383bbd535196d6d0a03, active policy runtime-key-safety-policy-2026-06-02t20-08-51-148570z-b2c0c9481064, rule count 11.

What changed:

• Added runtime key-safety classifications for VeriReel runtime secret bindings:
  • BETTER_AUTH_SECRET
  • VERIREEL_SECRETS_MASTER_KEY
  • VERIREEL_CRON_SECRET
  • POSTGRES_PASSWORD
• Final classification is shared_safe, restricted to context verireel and instances testing/prod, because the runtime key-safety model has one rule per binding key and testing targets cannot use prod_only.

Operational evidence:

• Initial verireel/testing dry-run after product-scoped sync failed closed with runtime_environment_empty: run 26843745422, trace launchplane_req_c3386112916445a5912cef0effe9ed96.
• Product-config dry-run before policy correction failed key-safety: trace launchplane_req_7693d28676014a84a29821418036c7d8.
• Product-config dry-run after corrected policy passed and planned four managed runtime secrets: trace launchplane_req_7b13c9b6b36c44f19674911a5d9c2f94.
• Product-config apply created four managed runtime secrets for verireel/testing: trace launchplane_req_91210d9062a842a3bad988b53c0fdc99.
• Live-target-runtime dry-run passed with only POSTGRES_PASSWORD missing from the live target and runtime key-safety pass: run 26845149852, trace launchplane_req_780bad99b6fb4eaab6c2703b1975273e.
• Live-target-runtime apply+deploy succeeded: run 26845196543, trace launchplane_req_4079d42e6cc6453582990bd7bf902d3c.
  • Applied: true
  • Env updated: true
  • Verification: pass
  • Verified key count: 4
  • Runtime key-safety: pass for all four VeriReel bindings
  • Deploy result: deployment=uHaasrsH2yfrYF8eKSri_ status=done
• Final steady-state dry-run passed: run 26845234396, trace launchplane_req_4f97824176dd46dfbd38b6deee5275fb.
  • changed_keys=[]
  • missing_keys=[]
  • desired key count 4
  • unchanged key count 4
  • runtime key-safety pass

Result: verireel/testing now has provider-target identity evidence and clean Launchplane-managed live runtime sync evidence. Next validation can move to another non-prod lane (cm/testing or opw/testing) before any prod lane expansion.

[cbusillo]

Current blocker after VeriReel testing — 2026-06-02

After discord-blue/prod and verireel/testing were validated, I started the next non-prod lane: cm/testing.

Result:

• Live Target Runtime dry-run for odoo-tenant-cm / cm/testing failed closed: run 26845298087.
• Error: runtime_environment_empty
• Message: Product 'odoo-tenant-cm' has no expected runtime keys for cm/testing.
• Trace: launchplane_req_df358f02437d4fe39e78cb4fbd363fd0

Interpretation:

• This is not a provider-target authority mismatch.
• This is a product-profile contract gap exposed by the new product-scoped live-target-runtime behavior: the Odoo CM profile does not yet declare expected runtime environment keys or runtime managed-secret bindings for cm/testing, so Launchplane refuses to sync any resolved runtime payload to the live target.

Why I stopped here:

• Unlike Discord Blue, where the contract was obvious from the seed runtime record (DISCORD_BLUE_STATE_DIR) and expected secret (DISCORD_TOKEN), the Odoo stable runtime contract is broader and should be declared deliberately from the Odoo lane/runtime policy rather than inferred from live env.
• Next safe work is to define the expected Odoo testing-lane runtime contract in product onboarding seed material, validate with dry-run, and only then apply live-target-runtime for Odoo lanes.

Current clean validation completed in #1105:

• discord-blue/prod: clean steady-state live runtime sync and deploy evidence.
• verireel/testing: clean steady-state live runtime sync and deploy evidence.

Remaining non-prod blocker:

• Odoo CM/OPW testing lanes need explicit expected runtime config declarations before live-target-runtime can validate under the product-scoped sync model.