Survive concurrent SonarQube analyses (fixes spurious red builds) - #70
Merged
Conversation
Main build 20 went red with no quality problem: 452 frontend and 417 backend tests passed and the gate read OK, but the build failed anyway. The compute-engine task had been rejected server-side: Date of analysis cannot be older than the date of the last known analysis on this project. Value: "...T22:54:38+0000". Latest analysis: "...T22:54:38+0000". Two analyses submitted in the same second -- main's build and a branch build -- and SonarQube failed the loser. The Jenkins log shows only "SonarQube task '<id>' status is 'FAILED'", none of the usual quality gate wording, and the stored gate still reads OK because the winner's result is what got saved. Confirmed via api/ce/activity: two tasks, one SUCCESS one FAILED, identical submittedAt. The root cause is that every build analyses into the one `pantrie` project. Isolating branches with sonar.branch.name is NOT available: verified against a throwaway project that the scanner rejects it with "Developer Edition or above is required" -- this server is Community Build 26.7. lock() would be the next-best fix, but the Lockable Resources plugin is not installed on this controller. So the analysis and gate check merge into one stage and retry together. Only the collision path retries: waitForQualityGate throws when the CE task fails, while abortPipeline:false makes a genuine ERROR gate return normally, so real failures still fail fast instead of paying for a second full scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the failure that red-ed main build 20, and re-greens main.
What actually failed
Build 20 passed everything that matters — 452 frontend tests, 417 backend tests, and the quality gate read OK (
new_violations 0, coverage 99.7%). It failed anyway, because SonarQube rejected the analysis server-side:Two analyses submitted in the same second — main's build and a branch build — and SonarQube failed the loser. Confirmed via
api/ce/activity:This is nasty to diagnose: the Jenkins log shows only
SonarQube task '<id>' status is 'FAILED'— none of the usual "Pipeline aborted due to quality gate failure" wording — and querying the gate afterwards returns OK, because the winner's result is what got stored. It looks like a passing gate on a failing build.Why not the obvious fixes
Every build (main, branch, PR) analyses into the single
pantrieproject, which is what allows the collision.sonar.branch.namewould isolate them, but it is not available here. Verified empirically against a throwaway project rather than assumed — the scanner rejects it outright:This server is Community Build 26.7. Adding that property would have broken every build.
lock('sonar-pantrie')would be the cleanest fix, but the Lockable Resources plugin is not installed on this controller (checked all 376 installed plugins — no lock/throttle/concurrency plugin).lock()would fail at runtime.What this does instead
Analysis and the gate check merge into one stage and retry together, so a collision simply re-scans with a fresh timestamp.
Crucially, only the collision path retries.
waitForQualityGatethrows when the compute-engine task fails — that's what we want to re-run — whileabortPipeline: falsemakes a genuineERRORgate return normally and fall through to an explicit check. Real quality failures still fail fast instead of paying for a second full scan.Worst case if
waitForQualityGatereturns null rather than throwing, the explicit check errors out — i.e. today's behaviour, not worse.The Jenkinsfile comment records both rejected options so the next person doesn't retry them, including the note to prefer
lock()if the plugin is ever installed.Note
lastStableBuildis currently 19. Merging this produces a fresh main build that should go green and supersede build 20.🤖 Generated with Claude Code