Skip to content

Guard host recovery against changing Node resource versions #6364

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Description

Guarded storage-host recovery can leave application scheduling paused when an unrelated Node status update occurs during its safety checks. The release PATCH currently uses the resource version captured before those checks. Recovery also reports the same generic command failure for several distinct stages, making a safe resume harder to diagnose.

Steps to Reproduce

  1. Start the existing host-recovery fixture in its saved returning phase.
  2. Advance only the Node resource version during the native or peer safety observations.
  3. Require the release PATCH to use the current resource version while retaining its identity preconditions.

Expected Behavior

Recovery uses a freshly verified version of the same owned Node immediately before releasing scheduling and before removing its completion marker. An uncertain write is accepted only through exact identity-bound readback; no second reboot or broad write retry is allowed. Full storage and database recovery remains required for completion.

Actual Behavior

Both Node mutations can use an older resource version. A rejected PATCH holds the cordon and the generic error does not identify the failed recovery stage. This code window is confirmed; its connection to the recent production hold is not yet proven.

Environment

The reviewed Platform host-memory recovery workflow.

Acceptance Criteria

  • Demonstrate the resource-version race with RED/GREEN fixture evidence.
  • Preserve native host, disk, boot, ownership, current-main and peer-copy checks.
  • Confirm ambiguous journal or scheduling writes with exact readback and one write only.
  • Keep private command output out of diagnostics.
  • Verify a real resumed recovery and application readiness separately from local tests.

Part of #5791

Activity

  1. self-assigned this
    on Nov 15, 2026
  2. added theissue type on Nov 15, 2026
  3. coderabbitai commented on Nov 15, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #5806 - feat(storage): prepare hosts and temporary raw storage for V2 [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  4. moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 15, 2026
  5. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Production deployments and dependent sign-in delivery are blocked by unfinished storage-host recovery. This is a scoped rung-0 WIP exception under the consumer's immediate live-breakage rule; the fresh board census is already over its implementation ceiling. Work is limited to the recovery defect and its safety regression coverage, with ordinary review and production gates retained.

    Priority rationale: Urgent, because the recovery hold blocks production delivery across applications. Service class: Expedite. End-to-end effort: Medium, including adversarial fixtures, CI, independent review, protected deployment and native resumed-recovery/application readback. First implementation start: 2026-11-15. The existing reviewed recovery is still queued; the proposed correction grants no direct cluster mutation, queue priority change or repeat reboot.

  6. updated Priority Urgent and Effort Medium on Nov 15, 2026
  7. added 4 commits that reference this issue on Nov 15, 2026
  8. moved this from 🏃🏻‍♂️ In Progress to ✅ Done in 🌊 Project Boardon Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

Urgent

Effort

Medium

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions