Repository navigation
fix(ci): retry the Dokany MSI download instead of failing on one blip - #26
Merged
Merged
Conversation
The smoke-windows job went red because `Invoke-WebRequest` died fetching the Dokany MSI. It was not a hang -- the request failed in about 21 seconds -- so the whole Windows smoke run reported a failure over a network blip rather than anything about the code, on a PR that only deleted a dead CI job. Note what is NOT the problem: this job already carries `timeout-minutes: 25`. The six-hour apt hang fixed in fleet-ci was a different failure with a different cause, and the ceiling here was already in place. `-MaximumRetryCount` would not have covered it either: PowerShell retries HTTP 429 and 5xx, not a dropped connection, which is what happened. Hence an explicit loop that catches any terminating error -- three attempts, 15s then 30s backoff, `-TimeoutSec 120` per request, and the partial file removed between attempts so a half-written MSI cannot be picked up by the next one. Exhausting the retries throws with the cause named as the GitHub releases endpoint rather than the code, so the next person reading a red smoke run is not sent looking in the wrong place. Also rejects a truncated download (`< 1MB`). A partial MSI installs as a corrupt one and fails later with a msiexec exit code that says nothing about why. PROVEN ON A REAL RUNNER, not asserted. No arm64 PowerShell image exists, so this could not be exercised locally the way the fleet-ci apt retry was. Instead the negative control ran on windows-latest with the URL pointed at a 404: 04:11:29 attempt 1/3 -> 404 04:11:45 attempt 2/3 -> 404 (15s backoff) 04:12:15 attempt 3/3 -> 404 (30s backoff) 04:12:15 exit 1 Three attempts, the backoff intervals as designed, the real error text surfaced on each, and a fast failure at 46 seconds rather than a pass or a hang. The scratch branch carrying that control has been deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
smoke-windowsjob went red becauseInvoke-WebRequestdied fetching the Dokany MSI. It was not a hang — the request failed in about 21 seconds — so the whole Windows smoke run reported a failure over a network blip, on a PR that only deleted a dead CI job.What is not the problem
This job already carries
timeout-minutes: 25. The six-hour apt hang fixed in fleet-ci was a different failure with a different cause; the ceiling here was already in place. Worth stating, because the two got conflated when they were first reported together.-MaximumRetryCountwould not have covered it either: PowerShell retries HTTP 429 and 5xx, not a dropped connection, which is what happened.The fix
An explicit loop that catches any terminating error — three attempts, 15s then 30s backoff,
-TimeoutSec 120per request, and the partial file removed between attempts so a half-written MSI cannot be picked up by the next one. Exhausting the retries throws with the cause named as the GitHub releases endpoint, not the code, so the next person reading a red smoke run is not sent looking in the wrong place.It also rejects a truncated download (
< 1MB). A partial MSI installs as a corrupt one and fails later with a msiexec exit code that says nothing about why.Proven on a real runner, not asserted
No arm64 PowerShell image exists, so this could not be exercised locally the way the fleet-ci apt retry was. Instead the negative control ran on
windows-latestwith the URL pointed at a 404:Three attempts, the backoff intervals as designed, the real error text surfaced on each, and a fast failure at 46 seconds rather than a pass or a hang. The scratch branch carrying that control has been deleted.