Skip to content

Use autoscaled local disks for V2 with graceful scale-down #6351

Description

@devantler

🤖 Generated by the Agentic Engineer

Problem and outcome

The platform currently excludes elastic workers' local disks from replica placement. The maintainer wants Longhorn V2 to use that existing capacity while preventing data loss and workload interruption during scale-down.

The hypothesis is that qualified raw storage on elastic workers, combined with verified replica evacuation and graceful workload handling, can use already provisioned resources without sacrificing durability or making empty nodes unreclaimable. Native Longhorn drain and autoscaler support should be evaluated before adding another mechanism.

Acceptance criteria

  • Eligible existing and future elastic workers have independently qualified raw-device geometry, encryption, V2 runtime and capacity. Ordinary compute/storage configurations are exercised in both feature states before rollout.
  • A removal preserves three healthy copies on distinct remaining nodes. Destination capacity, replica completion, runtime ownership and workload health are checked; missing evidence, insufficient space, an unhealthy copy or an overlapping maintenance operation blocks scale-down.
  • Trace the complete autoscaler-to-provider deletion path, including empty and unready-node paths. A Kubernetes Node disappearing is not proof that cloud deletion was safely admitted or completed.
  • Prefer verified native replica eviction. Establish how the V2 instance manager, autoscaler cooperation, drain policy and eviction timeouts interact; implement any necessary bounded workaround in our repositories rather than waiting for upstream changes.
  • Respect workload disruption budgets and block removal of workloads that cannot move without interruption. Successful evacuation lets the emptied node actually scale down; cancellation and retries preserve ownership and do not cause endless rebuilds.
  • Prove a real controlled scale-down with V2 data and continuous workload checks across preflight, evacuation, provider deletion and post-removal stability. Observe zero failed requests or data mismatches, three healthy remaining copies, and native provider absence. Negative capacity, health and incomplete-observation cases must refuse deletion.
  • Update declarative provisioning, placement and operating documentation, including safe disablement/rollback. Re-evaluate durable capacity and cost using the qualified elastic design before committing to additional permanent disks.

The initial baseline is that ordinary elastic disks are excluded. The success signal is observed use of eligible local disks and a successful, uninterrupted controlled scale-down with the durability conditions above. The measurement window is the complete controlled scale-down and its post-removal readback, not a merge or a Ready Pod. Rough size: large, likely several independently reviewable changes in Platform and, if required by provisioning or deletion hooks, KSail.

This is part of the existing migration: every application and retained source must still be accounted for on V2, and iSCSI must be removed from existing and future nodes. Tracking this capability does not establish migration completion or authorize additional permanent spend.

Part of #5582

Activity

  1. moved this to 📥 Backlog in 🌊 Project Boardon Nov 15, 2026
  2. self-assigned this
    on Nov 15, 2026
  3. added a commit that references this issue on Nov 15, 2026
  4. updated Effort High and Priority High on Nov 15, 2026
  5. moved this from 📥 Backlog to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 15, 2026
  6. added a commit that references this issue on Nov 16, 2026
  7. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The maintainer has now approved the minimum additional disk capacity necessary to proceed with the migration. The selection must first use qualified existing capacity and justify the smallest safe addition with three independent replicas, evacuation capacity and the existing storage reserves. This updates the earlier unresolved capacity permission; it does not choose disk quantities or sizes.

    Keep this work on the existing migration track. Refresh the complete volume footprint and native disk geometry, then bind the smallest viable addition to its provider quote and availability through reviewed delivery. Production recovery #6433 and the failed footprint observation remain unresolved, so no disk has been provisioned or storage migration accepted.

  8. added a commit that references this issue on Nov 16, 2026
  9. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The maintainer has extended this migration’s task-specific capacity authorization to nodes as well as disks when capacity prevents the requested requirements from fitting. Use qualified existing resources first and select the minimum suitable increment. This supersedes the earlier unresolved permanent-capacity permission; another generic ceiling question is unnecessary.

    The grant does not select a node count or disk size. Exact live capacity, provider pricing and availability, ownership, review and production/data-safety gates still apply. The recovery dependency remains active; no node or disk was added and no migration result is accepted by this authorization record.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

Priority

High

Start date

None yet

Target date

None yet

Effort

High

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions