Repository navigation
ci: add early disk cleanup to Test and Fuzz jobs - #18
Merged
Merged
Conversation
Add 'Early disk cleanup (free cargo caches and sibling run dirs)' step at the start of both Test and Fuzz jobs, before cache restoration and heavy cargo builds. This addresses ENOSPC failures on self-hosted runners (e.g. run 37505615506 on zeus-workstation) where disk pressure on the /mnt/data volume caused failures unpacking crates into CARGO_HOME. The cleanup targets: - Stale cargo registry/src and git/checkouts under RUNNER_TEMP/cargo - Sibling run directories under BLVM_RUNNER_DATA_ROOT - Bloated runner-cache/target directories (>10GB) - Stale fuzz/target artifacts (Fuzz job only) Also improves 'Dump disk usage on failure' in Test job to report: - /mnt/data volume (where ENOSPC occurred) - RUNNER_TEMP, CARGO_HOME, BLVM_RUNNER_DATA_ROOT paths and sizes - Better diagnostic output for future disk pressure issues The host root (/) was only 48% full in the failed run, so this is a volume-level issue not requiring host disk expansion. Co-authored-by: Secure Sovereign <secsovereign@users.noreply.github.com>
secsovereign
commented
Oct 6, 2026
secsovereign
left a comment
Member
Author
There was a problem hiding this comment.
OK once marked ready + CI green on this branch (and after merge, confirm main Test/Fuzz no longer ENOSPC on /mnt/data).
Looks right: early cleanup before heavy cargo in Test and Fuzz; frees RUNNER_TEMP/cargo registry/src + git/checkouts (the ENOSPC path), sibling run dirs under BLVM_RUNNER_DATA_ROOT, and bloated runner-cache targets; failure dump now includes /mnt/data + key env paths. Fuzz fuzz/target cleanup moved earlier (duplicate late step removed). Soft parks / #46 held; #17 rebase still after main is green.
secsovereign
marked this pull request as ready for review
October 6, 2026 18:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add early disk cleanup steps to the Test and Fuzz CI jobs to prevent ENOSPC failures on self-hosted runners before heavy cargo builds begin.
Problem
Run 37505615506 failed on
zeus-workstationwithNo space left on device (os error 28)while unpacking crates intoCARGO_HOME=/mnt/data/github-runner/zeus-workstation/_work/_temp/cargo.Key observation: The host root filesystem was only 48% full (
/dev/mapper/ArchinstallVg-root 148G 67G 74G 48% /), so this is a volume-level issue on/mnt/data, not a host disk expansion issue. The existing failure dump only randf -h /, missing the actual full volume.Solution
Add an "Early disk cleanup" step at the start of Test and Fuzz jobs (before cache restoration / cargo test / fuzz build) that frees:
RUNNER_TEMP/cargo(the isolated CARGO_HOME used by this workflow)BLVM_RUNNER_DATA_ROOT(same cleanup pattern as Setup job and late Test cleanup)Also hardens the "Dump disk usage on failure" step in Test to report:
/mnt/datavolume (where the ENOSPC actually occurred)RUNNER_TEMP,CARGO_HOME,BLVM_RUNNER_DATA_ROOTpaths and sizesNotes