Skip to content

Qualify peer artifact rollback under ENOSPC - #74

Merged
abhishek-anand merged 6 commits into
mainfrom
feature/enospc-rollback
Sep 1, 2026
Merged

Qualify peer artifact rollback under ENOSPC#74
abhishek-anand merged 6 commits into
mainfrom
feature/enospc-rollback

Conversation

@abhishek-anand

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in test failpoint after peer artifact disk reservation
  • drive an isolated node filesystem below the required localization size using observed free space
  • require a private 503 response with no partial artifact, snapshot, branch, replica, or staging state
  • verify the source VM remains healthy and a clean retry completes cross-node restore
  • detach the test loop device with the supported losetup --detach invocation

Validation

  • cargo test --locked -p taritd --features test-failpoints (383 passed)
  • strict clippy, formatting, shell syntax, and shellcheck
  • exact c8i tree c7a12de7ed677968e151d6b5df2de015739c4d68
  • Ubuntu 24.04 OCI E2E on Linux 6.6.155 and Linux 5.10.230
  • real KVM, TAP networking, authenticated peer transfer, live fork, lazy restore, ENOSPC rollback, scale-to-zero, HTTP wake, durable egress, shared-volume recovery, replica repair, physical cleanup, and terminal teardown

@abhishek-anand
abhishek-anand marked this pull request as ready for review August 31, 2026 23:20
Copilot AI lite review requested due to automatic review settings August 31, 2026 23:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the peer artifact replication E2E gate to deterministically exercise (and verify) rollback behavior when node-B runs near ENOSPC during peer artifact localization, using an opt-in failpoint that pauses immediately after disk reservation.

Changes:

  • Add a test-failpoints-gated taritd failpoint to pause after artifact localization disk reservation.
  • Enhance the E2E script to mount node-B onto a loop-backed btrfs filesystem, exhaust free space below the required localization size, and assert a private 503 with no partial replica/snapshot/branch/staging state.
  • Improve teardown by using the supported losetup --detach loop device cleanup path.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
orch/tests/e2e_peer_artifact_replication.sh Adds near-ENOSPC fixture (loop+btrfs), drives storage pressure during localization pause, validates lossless rollback + clean retry, and updates cleanup behavior.
orch/crates/taritd/src/ops.rs Introduces an opt-in localization pause failpoint immediately after disk reservation (gated behind test-failpoints).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread orch/tests/e2e_peer_artifact_replication.sh
@abhishek-anand
abhishek-anand merged commit ad756dd into main Sep 1, 2026
12 checks passed
@abhishek-anand
abhishek-anand deleted the feature/enospc-rollback branch September 1, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants