west sdk install fails repo-wide: sdk-ng's releases-list endpoint returns [] and west misreports it as "Unavailable SDK version: 1.0.1"

[hkngln]

Problem

getting-started.yml's step 7, install the Zephyr SDK (west sdk install, the printed remedy), fails on every PR that reaches it. The message it prints blames the pinned version:

FATAL ERROR: Unavailable SDK version: 1.0.1.Please select from the list below:

##[error]west sdk install failed 3 times -- this is a real failure, not a transient fetch.

The pin is fine. GET /repos/zephyrproject-rtos/sdk-ng/releases is returning an empty array, west sdk install resolves --version against that list, and it reports an empty list the same way it reports a genuinely absent version. Note the blank line after Please select from the list below: in the log — the list it offers is empty. That blank line is the whole diagnosis.

This is upstream/GitHub-side, not a tan-cli defect and not the pin's fault. Filed because the failure is currently indistinguishable from "our pin is wrong", and because the same command is the remedy tan doctor prints to customers.

Evidence

Observed on run 32039026116 (PR #839, branch fix/813-fold-752-and-gate-fragments, 2026-08-17). The retry loop at getting-started.yml:536-549 burned all three attempts in 50 seconds, each one fetching the list in ~1.3 s and failing immediately:

14:30:05.9961750Z FATAL ERROR: Unavailable SDK version: 1.0.1.Please select from the list below:
14:30:06.0258117Z ##[warning]west sdk install attempt 1/3 failed; retrying in 15s
14:30:21.3205938Z Fetching Zephyr SDK list...
14:30:22.6511747Z FATAL ERROR: Unavailable SDK version: 1.0.1.Please select from the list below:
14:30:22.6807951Z ##[warning]west sdk install attempt 2/3 failed; retrying in 30s
14:30:52.9685789Z Fetching Zephyr SDK list...
14:30:56.6586055Z FATAL ERROR: Unavailable SDK version: 1.0.1.Please select from the list below:
14:30:56.6879199Z ##[error]west sdk install failed 3 times -- this is a real failure, not a transient fetch.

Fast and empty, three times. Not a slow or partial fetch.

Not a rate limit, which is what this step's own comment predicts. getting-started.yml:490-501 documents the fetch_releases API rate limit exceeded failure mode and passes GH_TOKEN (line 505) to lift it. Measured from a workstation with a fully authenticated, unthrottled token:

$ gh api rate_limit --jq '.resources.core'
{"limit":5000,"remaining":4985,"reset":1786978845,"used":15}

The release, the tag and the assets are all alive. Only the list endpoint is empty. Same token, same minute:

$ gh api "repos/zephyrproject-rtos/sdk-ng/releases?per_page=100" --jq 'length'
0
$ gh api "repos/zephyrproject-rtos/sdk-ng/releases?per_page=100"
[]

$ gh api repos/zephyrproject-rtos/sdk-ng/releases/latest --jq '.tag_name, .published_at'
v1.0.1
2026-03-25T02:04:03Z

$ gh api "repos/zephyrproject-rtos/sdk-ng/tags?per_page=10" --jq '.[].name'
v1.0.1
v1.0.0
v1.0.0-rc1
v1.0.0-beta1
v0.18.0-alpha4
v0.18.0-alpha3
v0.18.0-alpha2
v0.18.0-alpha1
v0.17.4
v0.17.4-rc1

The release assets download by direct URL, HTTP 200, while the list is empty:

$ curl -sIL -o /dev/null -w "%{http_code}\n" \
    "https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v1.0.1/zephyr-sdk-1.0.1_linux-x86_64_minimal.tar.xz"
200
$ curl -sIL -o /dev/null -w "%{http_code}\n" \
    "https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v1.0.1/toolchain_gnu_linux-x86_64_arm-zephyr-eabi.tar.xz"
200

Both redirect to a signed release-assets.githubusercontent.com URL and serve.

The version is not hardcoded here. getting-started.yml:511 reads it out of the pinned alp-sdk:

version="$(python3 -c 'import json; print(json.load(open("alp-sdk/metadata/toolchains.json"))["zephyrSdk"]["version"])')"

PINNED_SDK_TAG is 88318e759958529fbbd8fe9d481373681c0fa78d (read from parity.yml, per getting-started.yml:305-316), and at that ref:

$ gh api "repos/alplabai/alp-sdk/contents/metadata/toolchains.json?ref=88318e759958529fbbd8fe9d481373681c0fa78d" \
    --jq '.content' | base64 -d | python3 -c 'import json,sys; d=json.load(sys.stdin)["zephyrSdk"]; print(d["version"]); print(d["baseUrl"])'
1.0.1
https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v1.0.1/

versionDerivation in that file states the pin is re-derived, not hand-copied: "Equal to the SDK_VERSION file at the Zephyr revision metadata/bootstrap.json's zephyr.version pins (currently v4.4.1) -- verified with git show <zephyr.version>:SDK_VERSION against the Zephyr checkout, not copied by hand. scripts/check_toolchain_lock.py re-derives and asserts this at every run." So there is no wrong-pin hypothesis left to chase.

Time-bounded, which rules out "always broken". Same pin, same workflow, same day:

12:29  PR #838  fix/802-aen-gate-table                 getting-started  success
14:26  PR #839  fix/813-fold-752-and-gate-fragments    getting-started  failure

The last 12 getting-started runs are all success except the 14:26 one. Nothing in #839's diff reaches this step — it touches .github/workflows/release.yml, CHANGELOG.md, changelog.d/752.fixed.md and changelog.d/813.fixed.md only.

Impact

CI, now: every PR that reaches step 7 goes red until the upstream list repopulates. getting-started is not a required context (the five are seam1 -- plan-shape parity, the three python -- pytest across python/ legs, and zizmor · workflow security), so this does not hard-block merges — it makes them UNSTABLE and adds one red check that every reviewer now has to re-diagnose from scratch, because the message names our pin.

Customers, worse: west sdk install is not just a CI step, it is the remedy tan doctor prints. getting-started.yml:24 says so:

#   west sdk install         <- the remedy tan's own build-readiness check PRINTS

While the list endpoint is empty, that printed remedy fails for every user on every network — a home IP with its own unauthenticated quota included, since this is not a quota problem — and the error tells them their SDK version is unavailable. The predictable response is to start editing a pin that is correct.

The retry loop cannot help and says the opposite. Its own text asserts this is a real failure, not a transient fetch. That is accurate and still misleading: real, not transient, and not ours. Three attempts over 50 s cannot outlast an upstream list outage.

Suggested fix

Two separable pieces. The first is worth doing regardless of when upstream recovers, because it is about making the red readable:

1. Distinguish "list empty" from "version absent" before calling west sdk install. One gh api probe in the step: if /releases is empty while /releases/latest resolves, the list is broken, not the pin — emit ::warning:: naming the upstream outage and skip or soft-fail rather than printing a message that accuses the pin. The house rule this repo already applied to sdk list --online in clean-host.yml covers exactly this case: "A gate that goes red for a reason unrelated to its subject stops being read."

2. A list-independent acquisition path. The pinned alp-sdk already carries everything needed — baseUrl, per-host filename, sizeBytes and sha256 for all six artifacts (linux-x86_64, windows-x86_64, macos-aarch64 x minimal-sdk, arm-zephyr-eabi-toolchain) — and the direct URLs serve HTTP 200 right now. bootstrap acquires the arm-zephyr-eabi cross toolchain (ADR 0021 Lane 1 P1) #474 (bootstrap acquires the arm-zephyr-eabi cross toolchain, ADR 0021 Lane 1 P1) is the existing home for this; this outage is a concrete argument for it, since a metadata-driven download with a sha256 check does not depend on the release-list API at all.

Deliberately not proposed: bumping or pinning a different zephyrSdk.version. Nothing is wrong with 1.0.1, every other version would resolve against the same empty list, and the value is re-derived upstream by scripts/check_toolchain_lock.py.

Verifying recovery

$ gh api "repos/zephyrproject-rtos/sdk-ng/releases?per_page=100" --jq 'length'

Non-zero means upstream is back and this failure mode disappears on its own. That is the reason to keep item 1 above: the next occurrence should be legible without repeating this whole investigation.

[hkngln]

Follow-up measurement that makes suggested fix 2 concrete: the list-independent path already exists in this repo, in parity.yml, and it stayed green through this outage.

Run 32039025829 is the sibling of the failing run — same PR, same push, same PINNED_SDK_TAG. getting-started's step 7 died on the empty release list while seam2 -- materialise + real build + Renode smoke finished in 3 minutes, Zephyr SDK and all.

The difference is how each acquires the SDK. parity.yml:1245-1265 derives the artifact straight from the pinned alp-sdk metadata:

d = json.load(open("alp-sdk/metadata/toolchains.json"))
sdk = d["zephyrSdk"]
art = next(a for a in sdk["artifacts"]
           if a["host"] == "linux-x86_64" and a["component"] == "minimal-sdk")
version = sdk["version"]
url = sdk["baseUrl"] + art["filename"]
sha256 = art["sha256"]

then parity.yml:1270-1278:

curl -fsSL -o /tmp/zephyr-sdk.tar.xz "${ZEPHYR_SDK_URL}"
ACTUAL="$(sha256sum /tmp/zephyr-sdk.tar.xz | awk '{print $1}')"
if [ "${ACTUAL}" != "${ZEPHYR_SDK_SHA256}" ]; then
  echo "::error::Zephyr SDK sha256 mismatch: expected ${ZEPHYR_SDK_SHA256}, got ${ACTUAL}"
...
tar -xJf /tmp/zephyr-sdk.tar.xz -C "$HOME/zephyr-sdk" --strip-components=1
"$HOME/zephyr-sdk/setup.sh" -t arm-zephyr-eabi -c

No GET /releases anywhere in it — which is exactly why it is unaffected, and it matches the HTTP 200 direct-asset measurements in the body above.

Two consequences for the issue:

1. Impact narrows. Only getting-started.yml is red from this. seam2 is not, and it is the job that does a real toolchain-backed build. Nothing about the product's ability to build is broken by the outage — only the path west sdk install takes.
2. Suggested fix 2 is a port, not a design. The metadata read, the URL construction, the sha256 verification and the setup.sh -t arm-zephyr-eabi call are all already written and running here. bootstrap acquires the arm-zephyr-eabi cross toolchain (ADR 0021 Lane 1 P1) #474 remains the right home for the customer-facing half, but nothing has to be invented for the CI half.

The customer-facing problem is unchanged and is still the larger one: tan doctor prints west sdk install, that command goes through the empty list, and its error blames a pin that is correct.

[hkngln]

Upstream has recovered. Correcting the body's "now" framing before it goes stale.

The recovery check this issue ends with:

$ gh api "repos/zephyrproject-rtos/sdk-ng/releases?per_page=100" --jq 'length'
100

It was 0 when this was filed. getting-started is green again on the next PR to run it — PR #841, 16:14, first install -- install.sh -> bootstrap -> init -> build: SUCCESS.

The outage window, from the workflow's own run history:

12:29  fix/802-aen-gate-table                 success
14:26  fix/813-fold-752-and-gate-fragments    FAILURE     <- the run this issue is about
16:14  fix/812-parity-concurrency-timeouts    success

Roughly 12:29-16:14, a single failed run inside it. So:

• The "every PR that reaches step 7 goes red" impact has expired. Nothing is currently red from this, and nothing needs to be re-run except PR ci(release): refuse to publish with unfolded changelog fragments, and fold #752 where it shipped (#813) #839's stale check.
• The customer-facing impact has expired too, for now, and it will come back the next time the list endpoint hiccups. It is not something this repo controls.
• Suggested fix 1 is the surviving reason to keep this open, and recovery is exactly why: the next occurrence will print Unavailable SDK version: <pin> again, name a pin that is correct again, and cost someone the same investigation again. A one-gh api probe that distinguishes "list empty" from "version absent" turns a repeat of today into a one-line ::warning::.

Suggested fix 2 (the metadata-driven download) is unchanged and is still a port rather than a design — see the previous comment for the parity.yml:1245-1278 implementation that carried seam2 through the outage untouched.

Retitling territory: this is no longer "fails repo-wide", it is "an upstream list outage is indistinguishable from a bad pin, and will recur". Leaving the title as filed rather than rewriting history mid-thread; say the word if you would rather it moved.