Skip to content

fix(ci): retry the Dokany MSI download instead of failing on one blip - #26

Merged
h4x0r merged 1 commit into
mainfrom
fix/retry-dokany-download
Aug 20, 2026
Merged

h4x0r merged 1 commit into
mainfrom
fix/retry-dokany-download

Conversation

@h4x0r

@h4x0r h4x0r commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The smoke-windows job went red because Invoke-WebRequest died fetching the Dokany MSI. It was not a hang — the request failed in about 21 seconds — so the whole Windows smoke run reported a failure over a network blip, on a PR that only deleted a dead CI job.

What is not the problem

This job already carries timeout-minutes: 25. The six-hour apt hang fixed in fleet-ci was a different failure with a different cause; the ceiling here was already in place. Worth stating, because the two got conflated when they were first reported together.

-MaximumRetryCount would not have covered it either: PowerShell retries HTTP 429 and 5xx, not a dropped connection, which is what happened.

The fix

An explicit loop that catches any terminating error — three attempts, 15s then 30s backoff, -TimeoutSec 120 per request, and the partial file removed between attempts so a half-written MSI cannot be picked up by the next one. Exhausting the retries throws with the cause named as the GitHub releases endpoint, not the code, so the next person reading a red smoke run is not sent looking in the wrong place.

It also rejects a truncated download (< 1MB). A partial MSI installs as a corrupt one and fails later with a msiexec exit code that says nothing about why.

Proven on a real runner, not asserted

No arm64 PowerShell image exists, so this could not be exercised locally the way the fleet-ci apt retry was. Instead the negative control ran on windows-latest with the URL pointed at a 404:

04:11:29  attempt 1/3 -> 404
04:11:45  attempt 2/3 -> 404      (15s backoff)
04:12:15  attempt 3/3 -> 404      (30s backoff)
04:12:15  exit 1

Three attempts, the backoff intervals as designed, the real error text surfaced on each, and a fast failure at 46 seconds rather than a pass or a hang. The scratch branch carrying that control has been deleted.

The smoke-windows job went red because `Invoke-WebRequest` died fetching the
Dokany MSI. It was not a hang -- the request failed in about 21 seconds -- so
the whole Windows smoke run reported a failure over a network blip rather than
anything about the code, on a PR that only deleted a dead CI job.

Note what is NOT the problem: this job already carries `timeout-minutes: 25`.
The six-hour apt hang fixed in fleet-ci was a different failure with a different
cause, and the ceiling here was already in place.

`-MaximumRetryCount` would not have covered it either: PowerShell retries HTTP
429 and 5xx, not a dropped connection, which is what happened. Hence an explicit
loop that catches any terminating error -- three attempts, 15s then 30s backoff,
`-TimeoutSec 120` per request, and the partial file removed between attempts so
a half-written MSI cannot be picked up by the next one.

Exhausting the retries throws with the cause named as the GitHub releases
endpoint rather than the code, so the next person reading a red smoke run is not
sent looking in the wrong place.

Also rejects a truncated download (`< 1MB`). A partial MSI installs as a corrupt
one and fails later with a msiexec exit code that says nothing about why.

PROVEN ON A REAL RUNNER, not asserted. No arm64 PowerShell image exists, so this
could not be exercised locally the way the fleet-ci apt retry was. Instead the
negative control ran on windows-latest with the URL pointed at a 404:

  04:11:29  attempt 1/3 -> 404
  04:11:45  attempt 2/3 -> 404      (15s backoff)
  04:12:15  attempt 3/3 -> 404      (30s backoff)
  04:12:15  exit 1

Three attempts, the backoff intervals as designed, the real error text surfaced
on each, and a fast failure at 46 seconds rather than a pass or a hang. The
scratch branch carrying that control has been deleted.
@h4x0r
h4x0r merged commit d387d81 into main Aug 20, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant