Skip to content

Cap build retention with branch-conditional properties() - #13

Open
jjpersch wants to merge 1 commit into
masterfrom
jjpersch/jenkins-build-retention
Open

Cap build retention with branch-conditional properties()#13
jjpersch wants to merge 1 commit into
masterfrom
jjpersch/jenkins-build-retention

Conversation

@jjpersch

Copy link
Copy Markdown
Contributor

Summary

Add a branch-conditional buildDiscarder so builds stop being retained forever. Mainline keeps 60 days/100 builds; PR and topic branches keep 30 days/5.

Background

The 2026-08-09 dev-build-cm disk-full outage was caused by unbounded Jenkins build retention: the controller's 250G /var/lib/jenkins volume hit 99% and the built-in node went offline, blocking all builds and the multibranch scans that prune closed-PR jobs. A fleet audit after the incident found this repo's Jenkinsfile sets no buildDiscarder at all, so its jobs retain every build indefinitely. The volume here is far lower than the repos that actually filled the disk, so this is preventive rather than urgent — but it is the same unbounded shape, and the existing top-level properties() call makes the fix a small one.

What changed

  • The top-level branch-conditional properties() call now owns build retention for every branch: mainline gets logRotator(daysToKeepStr: '60', numToKeepStr: '100'), PR/topic branches get logRotator(daysToKeepStr: '30', numToKeepStr: '5') alongside the existing disableConcurrentBuilds(abortPrevious: true).
  • Both day and count caps are set deliberately: a day-only rotator keeps every build produced inside the window, which is what left the controller unbounded.

Verification

  • The identical pattern built green on monetate-server (branch build and PR-30702 build 2, both SUCCESS), and the resulting job config was verified via the Jenkins API to carry the properties()-set rotator values.
  • Left to CI: this PR's own build exercises the edited Jenkinsfile end to end.

Deployment plan and risks

Merging is the deployment; jobs pick the new retention up on their next run. No declarative options{} block in this repo owns a discarder, so properties() is the sole owner and there is no transitional run without one. Build history beyond the caps is permanently discarded on the next run's post-build cleanup; if older build logs from this repo are still wanted, export them before merging.

The job set no buildDiscarder at all, so every build was retained
forever. Mainline now keeps 60 days/100 builds; PR and topic branches
keep 30 days/5. Both caps are required: a day-only rotator keeps every
build a busy job produces inside the window, which is unbounded disk
on the controller -- the pattern behind the 2026-08-09 dev-build-cm
disk-full outage. Same treatment as monetate-server PR 30702
(monetate/monetate-server#30702).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@jjpersch
jjpersch requested review from Riccardo-Maio and a lite review from Copilot August 10, 2026 01:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds branch-conditional Jenkins job properties to enforce bounded build retention, preventing unbounded /var/lib/jenkins growth and future disk-full outages.

Changes:

  • Apply buildDiscarder(logRotator(...)) via top-level properties() for all branches.
  • Use stricter retention on non-mainline branches (30 days / 5 builds) while keeping mainline longer (60 days / 100 builds).
  • Preserve existing non-mainline disableConcurrentBuilds(abortPrevious: true) behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants