Skip to content

Filesystem backups fail when a stateful pod moves during a backup #6431

Open

Description

@devantler

🤖 Generated by the Agentic Engineer

Description

A filesystem backup can fail when a stateful workload is replaced or moves to another node while the backup is waiting. The transfer retains its original node but resolves the workload by name again, so it can search that node for the replacement pod's mount. The daily recovery point is then incomplete even if storage capacity is sufficient.

Read-only investigation of the 16 November backup identified one such failed PodVolumeBackup separately from the three CSI preparation timeouts tracked in #6320. The recorded pod identity and node differ from the replacement used in the mount-path error. Detailed runtime identifiers remain in the private investigation record.

Steps to Reproduce

  1. Use disposable data in a stateful workload and start a filesystem backup.
  2. Hold the transfer before volume exposure, replace the pod with the same name, and schedule the replacement elsewhere.
  3. Release the transfer and verify whether the backup completes against the intended data and pod identity.

The production record exhibits this identity change; a controlled reproduction has not yet run.

Expected Behavior

A backup must either retain a valid source identity or safely restart its transfer against a verified replacement. It must never report a successful recovery point with missing volume data.

Actual Behavior

The volume-exposure operation fails to locate its mount and the scheduled backup is partially failed. This is a separate mechanism from insufficient replica capacity and ClickHouse file churn.

Source investigation

Velero 1.18.4 selects the PVB's recorded node, then passes only the pod namespace, name and volume to exposure. The exposer reads the current same-name pod before finding its host path. This supports the replacement-race diagnosis; it does not prove that every mount-path error has this cause.

Acceptance Criteria

  • Reproduce the replacement race with disposable data, retaining an unchanged-pod positive control.
  • Deliver a reviewed prevention or recovery path that preserves volume coverage, source identity, encryption, retention and existing deadlines. Check whether portfolio rollout coordination can solve it before considering an upstream contribution.
  • Prove the replacement scenario produces a complete backup and restores representative data, rather than merely suppressing the error.
  • Verify a complete scheduled backup after deployment.

Scope: one backup identity/lifecycle failure. Keep the independent capacity incident and existing recovery work open.

Part of #4980

Activity

  1. added theissue type on Nov 16, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 16, 2026
  3. self-assigned this
    on Nov 16, 2026
  4. added 2 commits that reference this issue on Nov 16, 2026
  5. added a commit that references this issue on Nov 16, 2026
  6. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The upstream release list currently has stable v1.18.4 and the newer v1.19.0-rc.1 prerelease. In the release candidate, the controller still selects the PVB's recorded node, passes namespace, pod name and volume to exposure, and the exposer fetches the current same-name pod. That interface does not bind exposure to the original pod UID. This source comparison supports the investigation; it does not establish a runtime reproduction or a repaired version.

    The next engineering route is to assess whether portfolio rollout/backup coordination can preserve the source pod until its filesystem transfer is terminal. A schedule offset alone cannot establish that guarantee. Any proposed coordination must have a finite deadline, preserve emergency recovery, and retain volume coverage, encryption and retention. If that cannot satisfy the replacement scenario, record why before considering an upstream change.

    The native dependency on #6433 records the prerequisite for cluster-dependent acceptance. After recovery is verified, run the existing issue's disposable-data experiment with these independent observations:

    1. Record the source pod UID, node, volume identity and a seeded file's checksum; complete an unchanged-pod backup and restore as the positive control.
    2. Hold the comparison transfer before exposure, replace the same-name pod on another node, then release the transfer and record which source identity is used.
    3. Require complete volume inventory and the expected checksum after restore. A terminal transfer, a suppressed error or a successful backup object alone does not establish data recovery.

    The controlled experiment and prevention remain unverified. No cluster changes, prerelease adoption or upstream writes were performed in this investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions