Skip to content

fix(sdl): fetch oci modules from public registries through the anonymous token flow - #1480

Open
doublewhy wants to merge 5 commits into
devfrom
1479-oci-anonymous-token
Open

doublewhy wants to merge 5 commits into
devfrom
1479-oci-anonymous-token

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

oci: imports failed against public registries: ghcr.io, Docker Hub and ECR Public answer an anonymous /v2/ request with 401 and a Bearer challenge, which the transport turned into Failed to fetch OCI metadata from <url>. Found by the #1344 audit (finding NS-2; record in #1461).

Such a 401 now gets the distribution specification's anonymous token flow: one token request to the HTTPS realm, one retry. The token is attached with Request.add_unredirected_header, so blob 307 redirects reach storage without it.

Requirement UIDs

  • None. Transport fix with no requirement scope.

Related Issues

Closes #1479

Merge order: before #1484, which rebases onto it (both edit module_registry/__init__.py).

ADR Impact

Changes

  • New module_registry/_registry_auth.py: fail-closed Bearer challenge parsing, an https-only realm, a no-redirect token request capped at 8 MiB, a per-import token cache and _registry_get. _json_request, _bytes_request and resolution.py:_resolve_oci_import use it.
Verification Plan bullet (verbatim) Change Test that fails without it CI lane Evidence
A new test module, implementations/python/tests/test_issue_<this issue>_oci_anonymous_token.py, runs hermetic loopback servers: - test_issue_1479_oci_anonymous_token.py test-shard CI
An end-to-end parse_sdl_file import must succeed with one token request, one anonymous attempt and one retry per registry URL, and no Authorization at the storage origin. _registry_auth.py:_registry_get, resolution.py:_resolve_oci_import the two end-to-end import cases test-shard CI
Further cases: a registry that rejects the token (exactly one retry), an http realm, a redirecting token endpoint, a 401 without a Bearer challenge, two Bearer challenges, an access_token-only response, malformed and oversized token responses, a 401 from a redirect target, and the challenge grammar. The module fails on dev. _registry_auth.py:_parse_bearer_challenge, _is_https_url, _anonymous_token the matching cases (54 of 57 fail on dev) test-shard CI
Show that the storage-origin test fails when the token is added as an ordinary header. _registry_auth.py:_registry_get (add_unredirected_header) end-to-end and redirect tests fail with add_header test-shard local swap
Run test_sdl_module_registry.py with default markers and with -m integration. - - test-shard, integration 328 passed; 5 passed
Rerun the reproduction above against ghcr.io, Docker Hub and ECR Public through the production path. - - manual live check
Run nox -s verify-fast-feedback -- --base-rev origin/dev, nox -s lint, make policy, both research evidence checkers and the three evidence test modules. - - fast-feedback, verify all passed

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge
  • No coverage regression

uv run --project implementations/python --frozen --all-extras python -m pytest <modules> -q -p no:cacheprovider: new module 57 passed. With dev's package code, 54 fail (the 401 or AttributeError); the other 3 pin unchanged behavior. With add_header instead of add_unredirected_header, storage receives the token and answers 403.

Neighbors (test_sdl_module_registry.py, three more, new module): 328 passed; 5 with -m integration. _registry_auth.py and __init__.py have full branch coverage.

Live check (2026-10-10; ghcr.io, Docker Hub, ECR Public): 401, one token request, a 200 retry; each blob 307 reached storage without Authorization and matched its digest. parse_sdl_file now stops at module validation; ghcr.io's OCI-index tag needs index Accept support (#1540).

Follow-ups: a real RAES module imported from a public registry, and token requests through an HTTP proxy (#1544); token refresh within one resolution (#1542).

Evidence: collected with blitz-tools/collect_research_evidence.py on this branch: 6 pinned artifacts via _execute_artifact (38 of 42 stages passed, 4 preregistered non-execution), 20 of 20 formal observations as expected, 14 participant functions (18 passed), checkout_state clean. research_capture record wrote capture-aae28f85... (specification) and capture-72bb493c... (formal) from dev's c9ca186a and 3bad3945; drift outside provenance, IDs and authored limitations: the implementation digest, formal commands[0] (retained-case-replay, what ran) and Python 3.12.3. Both checkers pass at the head and on the git merge-tree result with origin/dev; nox -s contracts passed.

CI on head deaed7d (base dev 348c490): run 38473234393 succeeded; 28 checks pass.

Ground Control Checks

  • Repository policy command passes
  • The new tests fail without the fix; the redirect tests also fail with add_header

Traceability

  • IMPLEMENTS: implementations/python/packages/raes/module_registry/_registry_auth.py, implementations/python/packages/raes/module_registry/__init__.py, implementations/python/packages/raes/module_registry/resolution.py
  • TESTS: implementations/python/tests/test_issue_1479_oci_anonymous_token.py

Checklist

  • Code follows the project coding standards
  • FM: not applicable, no semantic change (HTTP transport only).
  • No model or published contract schema changes.
  • PR title is a Conventional Commit; fix cuts a patch release.
  • Architectural docs: no change needed; see Documentation.

Documentation

  • Verified unchanged: docs/public/guides/cli.md, docs/explain/sdl/runtime-architecture.md, docs/explain/sdl/limitations.md.

@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch 2 times, most recently from d013e1b to 7dd887f Compare October 10, 2026 17:55
@doublewhy
doublewhy marked this pull request as ready for review October 10, 2026 18:20
doublewhy added a commit that referenced this pull request Oct 12, 2026
…ter entry

Each open entry in the divergence register now names its issue: #1525
to #1557 for the production changes, and #1521 for the smaller
corrections. The entries that open pull requests now address move to
"Addressed by pull requests" with their issues: GH-7 (#1491, #1490),
NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2
(#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560
and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and
"What remains" names #1522 for the unverified and unexamined
boundaries.
@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch from 7dd887f to 7f1db16 Compare October 13, 2026 10:08
@doublewhy
doublewhy marked this pull request as draft October 13, 2026 19:39
…ous token flow

ghcr.io, Docker Hub and ECR Public answer an anonymous registry request
with 401 and a WWW-Authenticate Bearer challenge, and the module
registry reported every such import as a fetch failure.

A 401 from the requested URL that carries exactly one Bearer challenge
is now answered with one anonymous token request to the challenge's
HTTPS realm (no redirects followed) and one retry. The challenge is
parsed strictly, tokens are cached per realm, service and scope for one
import resolution, and the token is attached as an unredirected header,
so urllib never sends it on the 307 to a registry's blob storage.
Timeouts, size limits, digests and the urlopen seam are unchanged.
…quested url

A realm whose host is percent-encoded or not IDNA-encodable passed the
realm check and then escaped parse_sdl_file as http.client.InvalidURL
or UnicodeEncodeError. The realm check now refuses a "%" in the
authority and any host the IDNA codec rejects, so such a challenge
fails with an SDLParseError before any token request.

The token response's oversize message named the token URL built from
the challenge's realm, service and scope. It now names the requested
URL, like every other message in the flow.

A second parse_sdl_file import against the same registry now pins the
one-cache-per-import token lifetime.
…nder the line cap

Dev's #1426 grew resolution.py to 492 lines, so the token threading put it at
501, over the ADR-015 500-line cap. Formatting only.
Collected with blitz-tools/collect_research_evidence.py on this branch's source (6 pinned artifacts through the production pipeline, 10 retained cases via replay_case, 4 production processor cases, 14 participant functions), then recorded with python -m tools.research_capture record.
@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch from 7f1db16 to deaed7d Compare October 13, 2026 21:47
@doublewhy
doublewhy marked this pull request as ready for review October 14, 2026 03:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant