Skip to content

Longhorn has no schedulable capacity left, so nightly backups and new databases cannot get volumes #6320

Open

Description

@devantler

🤖 Generated by the Agentic Engineer

Expected Behavior

A new Longhorn volume can be created: nightly backups can stage their temporary data-mover volumes, and a newly deployed database gets its storage.

Actual Behavior

Longhorn has no schedulable capacity left on any worker. Every new volume is refused with precheck new replica failed: insufficient storage, so:

  • Nightly full backups lose volumes. Since 12 November every daily full backup has ended PartiallyFailed with data uploads failing timeout on preparing data upload (6 errors on 14 November). Tonight's backup has 3 data uploads stuck Accepted because their temporary volumes cannot be scheduled. This is a different cause from Live ClickHouse file churn cancels its volume and fails every daily full backup #4869 (ClickHouse file churn).
  • New stateful workloads stall. The customer-identity database's volume (platform#6313, in the merge queue now) cannot attach, so its HelmRelease install will time out.

Read-only evidence (15 November, ~04:15Z):

  • Every worker's scheduled storage is within ~2-4% of its schedulable limit (over-provisioning 100%, minus reserved space). Every volume asks for 3 replicas, so a 5 GiB volume needs 5 GiB free on each of three workers.
  • Actual disk use is also high: each worker is roughly 10 percentage points above the 25% minimum-available floor, so simply raising over-provisioning buys little.
  • Reclaimable scheduled storage found on the cluster:
  • Velero's temporary data-mover volumes are full-size, 3-replica copies of each source volume, although they live only for the length of one upload.

Nothing alerts on this (#3099): it surfaced through the live-health probe.

Steps to Reproduce

  1. List Longhorn volumes in any non-attached state during the 02:17Z backup window, or create any new Longhorn PVC.
  2. Read the volume's Scheduled condition: ReplicaSchedulingFailure ... insufficient storage.

Proposed direction

  1. Reclaim the waste first, through reviewed changes: retire the orphaned CNPG volume, the finished recovery-drill copies and the detached duplicate once each is confirmed unneeded (retain-policy and backup checks first).
  2. Give Velero's temporary backup volumes a 1-replica storage class through the node-agent backupPVC configuration, so a nightly backup no longer needs 3 x the size of every volume it is copying.
  3. Keep the capacity headroom work (Verify provider headroom for Longhorn V2 migration capacity #5731, Preserve existing storage while preparing the third V2 worker #6000) as the lasting fix, and add the missing alert from Nothing alerts on a Longhorn volume stuck unattachable, so a storage fault surfaced as a merge-queue outage #3099.

Acceptance criteria

  • A new 5 GiB Longhorn volume schedules on prod.
  • The next daily full backup has no timeout on preparing data upload errors.
  • The customer-identity database volume attaches.

Activity

  1. added theissue type on Nov 15, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 15, 2026
  3. coderabbitai commented on Nov 15, 2026

    @coderabbitai
    Contributor
    ⚠️ Possible Duplicate Issue(s)


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  4. self-assigned this
    on Nov 15, 2026
  5. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Related in-flight work: #6282 (Codex lane, draft #6289) restores replica headroom for one declared V1 claim. It works the same logical replica budget this issue found exhausted cluster-wide, so a reclaim here and that repair should be sequenced, not run in parallel. This issue covers the cluster-wide symptom (no new volume schedules: nightly backup data movers and new databases), which #6282 does not.

  6. added 4 commits that reference this issue on Nov 15, 2026
  7. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Started the temporary-backup-storage repair in draft #6321. It reduces intermediate-copy demand while retaining source redundancy; it does not close this incident or reclaim existing source volumes. Local validation passed, and CI, review and runtime backup/restore proof remain outstanding.

    Service class: Expedite, with a WIP exception for confirmed live storage scheduling failures affecting backups and a newly deployed stateful service. This is incident repair, not an additional unrelated Ready pull. First started: 2026-11-15, evidenced by the implementation and signed commit in #6321. Existing recovery-copy cleanup also needs a safe continuation after its partial deletion; the current observation is recorded on #6041.

  8. added a commit that references this issue on Nov 15, 2026
  9. moved this from 📥 Backlog to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 15, 2026
  10. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Read-only acceptance check, 16 November at 12:20 UTC:

    • Today's daily full backup remains PartiallyFailed (four errors, one warning). Its three CSI DataUploads all failed with timeout on preparing data upload. Those records establish that the backup acceptance criterion is still unmet; they do not attribute the fourth error.
    • The customer identity database has recovered: both instances are Ready, continuous archiving is working, and its latest database backup succeeded. The database attachment symptom is no longer the outstanding part of this incident.
    • Native disk accounting at the current 100% over-provisioning limit still leaves one eligible disk below the room needed for a new 5 GiB replica. No test volume was created, and this observation is not a successful scheduling trial.
    • fix(backups): prepare smaller temporary copies safely #6321 merged as preparation. The live setting remains longhorn, so the smaller temporary-copy path is still off. Its complete upload and representative restore have not been demonstrated.

    Keep this issue open. The next delivery is the documented, separately reviewed staging validation rollout, followed by an actual complete backup and representative restore before changing the default. Preserve source redundancy and retention, cover the source storage classes actually present, and coordinate capacity changes with the existing migration owner. This check changed no cluster resources.

  11. added 4 commits that reference this issue on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions