Skip to content

Verify independent remote state and lock recovery #55

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Summary

Finish the prepared Cloudflare state backend and prove independent recovery before routine declarative changes are enabled.

Motivation

The guarded first apply has passed, with preserved state lineage and an independently authenticated recovery generation. The proposed state bucket now exists. Creating that bucket does not establish a working remote backend, real lock contention or recovery from a clean environment.

Proposed solution

Use a confirmed, first-attempt main workflow on independent hosted execution. Transfer only the authenticated current state and paired desired configuration, preserving its lineage and refusing an existing or uncertain destination. Use the existing approved provider identity to obtain bounded storage access for the operation, without introducing another manually managed provider key. Exercise the native OpenTofu backend and retain recoverable ciphertext before and after the operation.

Acceptance criteria

  • Disabled or invalid invocations load no credential and contact no provider; valid execution is bound to reviewed main, the approved operator, first attempt and exact authenticated recovery generation.
  • The transfer cannot overwrite an existing state or create a second authoritative lineage. Partial or uncertain operations retain independently recoverable evidence and require diagnosis before re-entry.
  • Native backend initialization, source-bound state readback and a no-change plan pass using the pinned tool and provider.
  • A second native operator fails to acquire the actual remote lock while the first holds it, then succeeds after normal release; no forced unlock or disabled locking is used.
  • A clean working environment recovers using only the repository, approved private custody and the independent backend, with another no-change plan.
  • Credentials and plaintext state stay out of public output. General provider applies and DNS adoption stay disabled until the native receipts pass.
  • Complete current-head validation, substantive review, protected delivery, native execution and independent recovery verification.

Part of #21. Continues the first migration tracked by devantler-tech/monorepo#3274 and unblocks the storage prerequisites for devantler-tech/platform#5328. The earlier first-apply proof is run 41466246062; it is not remote-backend acceptance.

Activity

  1. added theissue type on Nov 2, 2026
  2. coderabbitai commented on Nov 2, 2026

    @coderabbitai
    🔗 Related PRs

    #56 - feat(recovery): verify isolated state locking and recovery [merged]
    #59 - feat(recovery): guard production state migration and independent recovery [closed]
    #60 - chore(recovery): retain independently recovered backend state [closed]
    #58 - feat(recovery): verify native migration before state cutover [closed]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  3. moved this to 📥 Backlog in 🌊 Project Boardon Nov 2, 2026
  4. added 4 commits that reference this issue on Nov 2, 2026
  5. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The first synthetic backend trial 41482968414 stopped at stage=configuration, before any provider request. It did not initialize the remote backend or touch the authoritative state.

    The handoff defect is reproduced: the installed GitHub CLI strips final line endings from stdin secrets (its exact released implementation), while the authenticated native capture includes its final LF. The byte-digest check correctly refuses that changed input. Restoring only the missing capture LF reproduces the exact authenticated bytes and passes; changed content and extra whitespace still refuse.

    A small source correction is being taken through ordinary validation and review. The failed run will not be rerun. A new confirmed first attempt follows only after protected delivery and successful main validation.

  6. added a commit that references this issue on Nov 2, 2026
  7. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    After #56 and the diagnosed transport fix #57, confirmed first-attempt run 41485829209 passed at reviewed main b4a18ccddac0af5f304156a9e9d47000c0b20a83: CLOUDFLARE_STATE_BACKEND=VERIFIED scope=synthetic-state locking=native recovery=fresh-directory provider_resources=0. Its main validation also passed.

    Independent download and authenticated decryption verified the exact encrypted evidence inventory. Native OpenTofu configured the actual S3 backend; the contender received HTTP 412 while the native lock was held, followed by normal release. A fresh directory recovered byte-equivalent state contents with the same lineage and serial and produced a zero-change plan. The existing credential was sufficient; no additional long-lived credential was created.

    This accepts the isolated prerequisite only. The real imported infrastructure state has not been migrated, its clean-environment provider recovery remains unverified, and general applies stay disabled. The earlier failed preflight remains recorded with its diagnosed cause rather than being counted as a passed trial.

  8. added 2 commits that reference this issue on Nov 2, 2026
  9. self-assigned this
    on Nov 2, 2026
  10. added a commit that references this issue on Nov 2, 2026
  11. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The enhanced native migration trial failed its preservation guard. Independent authentication of the encrypted evidence confirms that the copy completed and all logical fields matched, but its lineage changed. Only the isolated synthetic prefix was involved; the real infrastructure state remains in its retained original generation.

    The pinned remote state manager refreshes an absent destination during first persistence and resets the imported metadata (upstream implementation). The latest patch retains that implementation. A native CLI test against a disposable S3 protocol fixture reproduces the lineage failure. Initializing the empty destination through a saved, provider-free zero-change plan preserves lineage locally; occupied-state and competing-writer controls are being added before review and a new real R2 trial.

    PR #59 stays draft on this measured blocker. The lineage requirement remains enforced, and no production migration or general apply is accepted by this result.

  12. added a commit that references this issue on Nov 2, 2026
  13. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    PR #59 merged as 1628f233a82f2200e9cd07581b5ee8c755bac062; its source tree exactly matches the reviewed head. The new first-attempt provider rehearsal passed at that main revision. I independently authenticated the encrypted evidence and verified native migration preserving identity and contents, nondecreasing serial, lock contention refusal with normal release, and exact recovery from a fresh directory.

    Proceeding with the separately confirmed state-only migration and independent runner recovery under the maintainer's approval. This does not enable general resource changes. Permanent authenticated recovery custody remains required before accepting the migration.

  14. added 2 commits that reference this issue on Nov 2, 2026
  15. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Acceptance complete for this slice. The confirmed first-attempt migration and independent recovery passed in two separate hosted jobs. Independent authentication verified preserved state identity and contents, native lock contention refusal and normal release, and zero resource-change plans in both jobs.

    PR #60 retained the independently recovered paired ciphertext through protected merge as adfb1b5c6f57fee0a6d8e379112df538356eb873. I fetched main and authenticated those exact files again; the previous generation remains in history. The one-use state handoff is removed, and the independent recovery key remains outside Actions.

    The broader Cloudflare work continues in #21 and the alerting storage work in devantler-tech/platform#5328. General resource changes remain separately gated.

  16. moved this from 📥 Backlog to ✅ Done in 🌊 Project Boardon Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

Priority

None yet

Start date

None yet

Target date

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions