Skip to content

ci: add early disk cleanup to Test and Fuzz jobs - #18

Merged
secsovereign merged 1 commit into
mainfrom
cursor/early-disk-cleanup-test-fuzz-a0e1
Oct 6, 2026
Merged

secsovereign merged 1 commit into
mainfrom
cursor/early-disk-cleanup-test-fuzz-a0e1

Conversation

@secsovereign

Copy link
Copy Markdown
Member

Summary

Add early disk cleanup steps to the Test and Fuzz CI jobs to prevent ENOSPC failures on self-hosted runners before heavy cargo builds begin.

Problem

Run 37505615506 failed on zeus-workstation with No space left on device (os error 28) while unpacking crates into CARGO_HOME=/mnt/data/github-runner/zeus-workstation/_work/_temp/cargo.

Key observation: The host root filesystem was only 48% full (/dev/mapper/ArchinstallVg-root 148G 67G 74G 48% /), so this is a volume-level issue on /mnt/data, not a host disk expansion issue. The existing failure dump only ran df -h /, missing the actual full volume.

Solution

Add an "Early disk cleanup" step at the start of Test and Fuzz jobs (before cache restoration / cargo test / fuzz build) that frees:

  • Stale cargo registry/src and git/checkouts under RUNNER_TEMP/cargo (the isolated CARGO_HOME used by this workflow)
  • Sibling run directories under BLVM_RUNNER_DATA_ROOT (same cleanup pattern as Setup job and late Test cleanup)
  • Bloated runner-cache/target directories (when >10GB, prune entries >60min old)
  • Stale fuzz/target artifacts from prior runs (Fuzz job only)

Also hardens the "Dump disk usage on failure" step in Test to report:

  • /mnt/data volume (where the ENOSPC actually occurred)
  • RUNNER_TEMP, CARGO_HOME, BLVM_RUNNER_DATA_ROOT paths and sizes
  • Better diagnostic info for future disk pressure debugging

Notes

  • Mirrors cleanup patterns already present in this workflow (Setup's "Emergency pre-flight disk cleanup", Test's late "Clean test binaries and stale run dirs")
  • Runs before heavy work starts, so the job can begin with free space
  • Does not touch PR fix(ci): use REPO_ACCESS_TOKEN for release tags when workflows change #17 (release-tag fix) or #46 (soft parks)
  • No Rust source changes
  • Josh is not needed for host disk expansion — this is a per-volume cleanup issue
Open in Web Open in Cursor 

Add 'Early disk cleanup (free cargo caches and sibling run dirs)' step
at the start of both Test and Fuzz jobs, before cache restoration and
heavy cargo builds. This addresses ENOSPC failures on self-hosted runners
(e.g. run 37505615506 on zeus-workstation) where disk pressure on the
/mnt/data volume caused failures unpacking crates into CARGO_HOME.

The cleanup targets:
- Stale cargo registry/src and git/checkouts under RUNNER_TEMP/cargo
- Sibling run directories under BLVM_RUNNER_DATA_ROOT
- Bloated runner-cache/target directories (>10GB)
- Stale fuzz/target artifacts (Fuzz job only)

Also improves 'Dump disk usage on failure' in Test job to report:
- /mnt/data volume (where ENOSPC occurred)
- RUNNER_TEMP, CARGO_HOME, BLVM_RUNNER_DATA_ROOT paths and sizes
- Better diagnostic output for future disk pressure issues

The host root (/) was only 48% full in the failed run, so this is a
volume-level issue not requiring host disk expansion.

Co-authored-by: Secure Sovereign <secsovereign@users.noreply.github.com>

@secsovereign secsovereign left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK once marked ready + CI green on this branch (and after merge, confirm main Test/Fuzz no longer ENOSPC on /mnt/data).

Looks right: early cleanup before heavy cargo in Test and Fuzz; frees RUNNER_TEMP/cargo registry/src + git/checkouts (the ENOSPC path), sibling run dirs under BLVM_RUNNER_DATA_ROOT, and bloated runner-cache targets; failure dump now includes /mnt/data + key env paths. Fuzz fuzz/target cleanup moved earlier (duplicate late step removed). Soft parks / #46 held; #17 rebase still after main is green.

@secsovereign
secsovereign marked this pull request as ready for review October 6, 2026 18:22
@secsovereign
secsovereign merged commit 0b6231c into main Oct 6, 2026
4 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants