fix(sdl): fetch oci modules from public registries through the anonymous token flow #1479

Gap Claim

oci: imports cannot be resolved from public registries that use the distribution token flow. ghcr.io, Docker Hub (registry-1.docker.io) and ECR Public (public.ecr.aws) all use it, even for anonymous pulls.

Every registry request in raes.module_registry goes through _json_request or _bytes_request (implementations/python/packages/raes/module_registry/__init__.py:156-164, :179-186). Each opens the URL once with urlopen and turns every URLError, HTTPError included, into Failed to fetch OCI metadata from <url> or Failed to fetch OCI blob from <url>. Nothing reads WWW-Authenticate, requests a token or sends Authorization. The resolver makes up to four requests per import (implementations/python/packages/raes/module_registry/resolution.py): tags/list unless the import is locked (:284), the manifest (:290-293), the config blob (:330-332) and the bundle blob (:342-345).

The distribution token specification (docs/spec/auth/token.md in distribution/distribution) describes the flow:

1. The registry answers 401 with WWW-Authenticate: Bearer realm="...",service="...",scope="...".
2. The client requests a token from realm, with service and each space-delimited scope entry as query parameters. Anonymous requests are allowed.
3. The JSON response carries token, or access_token for OAuth 2.0 compatibility.
4. The client repeats the request with Authorization: Bearer <token>.

Reproduction on dev (35122105) through the production path. The script below calls parse_sdl_file on a root that imports one public artifact, next to a raes-trust.yaml that allows the registry with require_signatures: false. It ran with the project environment's Python 3.14.4 on 2026-10-10 at 15:20Z:

== parse_sdl_file(imports oci:ghcr.io/oras-project/oras)
SDLParseError: Failed to fetch OCI metadata from https://ghcr.io/v2/oras-project/oras/tags/list
  cause: HTTPError 401 for https://ghcr.io/v2/oras-project/oras/tags/list
  WWW-Authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:oras-project/oras:pull"
== parse_sdl_file(imports oci:registry-1.docker.io/library/hello-world)
SDLParseError: Failed to fetch OCI metadata from https://registry-1.docker.io/v2/library/hello-world/tags/list
  cause: HTTPError 401 for https://registry-1.docker.io/v2/library/hello-world/tags/list
  WWW-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/hello-world:pull"
== parse_sdl_file(imports oci:public.ecr.aws/docker/library/hello-world)
SDLParseError: Failed to fetch OCI metadata from https://public.ecr.aws/v2/docker/library/hello-world/tags/list
  cause: HTTPError 401 for https://public.ecr.aws/v2/docker/library/hello-world/tags/list
  WWW-Authenticate: Bearer realm="https://public.ecr.aws/token/",service="public.ecr.aws",scope="aws"

The hermetic tests never see a challenge: _OCIHandler and _OCIRegistry in implementations/python/tests/test_sdl_module_registry.py:96-170 serve every endpoint anonymously and never redirect. The #1344 fixture-provenance audit found the gap (finding NS-2 in the record #1461 proposes).

Two facts constrain the fix:

• All three registries answer a blob request with 307 to a storage host on another origin, with a signed query (Evidence). urllib.request.HTTPRedirectHandler.redirect_request copies every header in Request.headers into the redirected request, whatever its origin. A fix that adds Authorization through Request(url, headers=...) would therefore send the registry token to the storage host. Headers added with Request.add_unredirected_header are not copied. CPython's own Basic and Digest auth handlers attach Authorization that way.
• ECR Public's challenge scope is aws, not a repository:<name>:pull entry. The token request must pass the scope through exactly as the registry sent it.

Evidence

Raw 401 responses, captured with curl 8.7.1 on 2026-10-10 at 15:19Z:

$ curl -sS -D - https://ghcr.io/v2/oras-project/oras/tags/list
HTTP/2 401
content-type: application/json
www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:oras-project/oras:pull"
date: Sat, 10 Oct 2026 15:19:06 GMT
content-length: 73
x-github-request-id: C221:34DA4A:22BD60:42D47E:6ACA576A
x-github-edge-region: westus3

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

$ curl -sS -D - https://registry-1.docker.io/v2/library/hello-world/tags/list
HTTP/2 401
date: Sat, 10 Oct 2026 15:19:11 GMT
content-type: application/json
content-length: 162
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/hello-world:pull"
strict-transport-security: max-age=31536000

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"library/hello-world","Action":"pull"}]}]}

$ curl -sS -D - https://public.ecr.aws/v2/docker/library/hello-world/tags/list
HTTP/2 401
date: Sat, 10 Oct 2026 15:19:17 GMT
content-type: application/json; charset=utf-8
content-length: 58
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://public.ecr.aws/token/",service="public.ecr.aws",scope="aws"

{"errors":[{"code":"DENIED","message":"Not Authorized"}]}

The rest of the flow, made by hand with Python's http.client on 2026-10-10 at 15:19Z. The script printed no token value and no signed query.

• All three tokens match RFC 6750's b64token syntax (56, 2,705 and 1,592 characters).
• The tag manifests are OCI image indexes, and the amd64 child manifests are OCI image manifests. Both answered 200 with the token.
• No token endpoint redirects.
• registry.gitlab.com (realm https://gitlab.com/jwt/auth) and nvcr.io (realm https://nvcr.io/proxy_auth) challenge the same way, and their realms also answer 200 directly.
• gcr.io, quay.io and mcr.microsoft.com answer the anonymous request with 200, a path this issue leaves unchanged.

Existing Surface Audit

• Transport. _json_request and _bytes_request are the module registry's only HTTP calls; publishing.py writes local OCI layouts. Both pass the explicit _OCI_LIMITS.timeout_seconds (30 s) and read through _read_capped (__init__.py:137-153), which pre-checks an advisory Content-Length and then reads max_bytes + 1. The limits are 8 MiB for metadata and 128 MiB for the bundle (_OCIResourceLimits, _extraction.py:26-47).
• Integrity. The manifest digest is checked against the lockfile when the import is locked (resolution.py:296-299). The config and bundle bytes are checked against the manifest's digests (:333-334, :346-347). Signatures, root-file binding and the extraction cache follow. None of these checks depends on how the bytes were fetched.
• Registry selection. An oci: registry must be listed in raes-trust.yaml (resolution.py:396-398). _registry_base_url (:68-79) uses https unless the registry is loopback or sets allow_insecure_http. RegistryTrustPolicy (models.py:28-31) has no credential field.
• Scope records. The OCI import fetches are unbounded and allow memory/disk exhaustion #12 preflight (docs/decisions/issue-12-oci-resource-limits-preflight.md:101-107) requires every urlopen call to pass an explicit timeout and feed the bounded reader. It also forbids credentials in source strings, argv, logs and messages. The [BUG] Harden OCI module publication, caching, parsing, and image trust #1096 record keeps registry authentication out of scope (docs/decisions/issue-1096-oci-module-registry-reliability-remediation.md:104). An anonymous token is not a user credential: no secret is read, stored or sent.
• Seams. The improve-modularity: split aces_sdl/module_registry.py (681 lines) #48 package-split record (docs/decisions/issue-48-module-registry-package-split-preflight.md:50-67) keeps raes.module_registry.urlopen, _OCI_LIMITS, _json_request and _bytes_request patchable. test_sdl_module_registry.py patches all four.
• Redirects. urlopen's default opener follows 301, 302, 303, 307 and 308 for GET, up to 10 hops. It copies Request.headers into each new request, but not unredirected headers.
• Tests. No test answers 401 or redirects a blob.
• Size. __init__.py has 488 lines and resolution.py 486, against the 500-line cap.

Alternatives

1. Add Authorization to the retried request's headers: rejected. urllib copies it into the 307 request to the storage origin.
2. Write a redirect handler that keeps Authorization on same-origin redirects and drops it on cross-origin ones: rejected. It replaces urllib's redirect policy for a case none of the surveyed registries uses, since each of their blob redirects leaves the registry origin.
3. Switch to requests or httpx, whose redirect handling drops Authorization when the host changes: rejected. It adds a runtime dependency to raes and replaces the transport that the OCI import fetches are unbounded and allow memory/disk exhaustion #12 limits and the improve-modularity: split aces_sdl/module_registry.py (681 lines) #48 seams are built on.
4. Fetch a token before the first request, as the Docker client does with a /v2/ ping: rejected. It adds requests for registries that allow anonymous access, and only the challenge supplies the scope (ECR Public's is aws).
5. Read credentials from a Docker config, a credential helper or the environment: rejected. Registry authentication stays out of scope ([BUG] Harden OCI module publication, caching, parsing, and image trust #1096), and public pulls need no credential.
6. Answer a 401 that carries one Bearer challenge with one anonymous token request and one retry, and send the token as an unredirected header: selected.

Chosen Architecture

A new private module, raes/module_registry/_registry_auth.py, holds the flow:

• Challenge. The parser is strict. The response's WWW-Authenticate fields must hold exactly one Bearer challenge, in RFC 7235 auth-param syntax: token or quoted-string values, case-insensitive names, no duplicates, nothing after the last parameter. realm is required. Other parameters are ignored, as RFC 6750 allows. A malformed challenge fails closed with an SDLParseError.
• Realm. Only an absolute https URL with a host and no userinfo or fragment is accepted.
• Token request. It is anonymous: no Authorization header. It goes through an opener that has an HTTPS handler and no HTTP or redirect handler. It uses the existing timeout and _read_capped at the metadata limit, and decodes through _decode_json_object. It takes token, else access_token, and requires RFC 6750 b64token syntax, so a token cannot inject a header.
• Cache. Tokens are kept by realm, service and scope in an object that lives for one import resolution.
• Retry. The original request is repeated exactly once. The token is added with Request.add_unredirected_header, so no redirected request carries it, same-origin or cross-origin.
• Origin of the challenge. Only a 401 from the requested URL is answered, not one from a redirect target.

Around it:

• _json_request and _bytes_request keep their names, messages, limits and challenge-free path, and gain an optional tokens argument. Their request code moves into the new module, which still calls urlopen through the package facade, to keep __init__.py under the cap.
• _resolve_oci_import creates the token cache and passes it to the manifest, config and bundle fetches.
• Messages name only the requested URL. No token, challenge or response text appears in them.
• A 401 without a Bearer challenge fails with today's message. Trust policy, allow_insecure_http, digests, size limits, timeouts, signatures, the cache and the lockfile are unchanged.
• The change edits package code bound by the research evidence, so both captures are republished at the next free releases.

Verification Plan

• A new test module, implementations/python/tests/test_issue_<this issue>_oci_anonymous_token.py, runs hermetic loopback servers:
  • a registry that answers 401 with a challenge until it gets the expected token, and redirects every blob with 307 to a second origin;
  • an HTTPS token endpoint signed by a throwaway CA, trusted through SSL_CERT_FILE;
  • a storage origin that records and refuses any request carrying Authorization.
• An end-to-end parse_sdl_file import must succeed with one token request, one anonymous attempt and one retry per registry URL, and no Authorization at the storage origin.
• Further cases: a registry that rejects the token (exactly one retry), an http realm, a redirecting token endpoint, a 401 without a Bearer challenge, two Bearer challenges, an access_token-only response, malformed and oversized token responses, a 401 from a redirect target, and the challenge grammar. The module fails on dev.
• Show that the storage-origin test fails when the token is added as an ordinary header.
• Run test_sdl_module_registry.py with default markers and with -m integration.
• Rerun the reproduction above against ghcr.io, Docker Hub and ECR Public through the production path.
• Run nox -s verify-fast-feedback -- --base-rev origin/dev, nox -s lint, make policy, both research evidence checkers and the three evidence test modules.

import sys
import textwrap
from pathlib import Path
from urllib.error import HTTPError

from raes._errors import SDLParseError
from raes.parser import parse_sdl_file

SOURCES = (
    ("ghcr.io", "oras-project/oras"),
    ("registry-1.docker.io", "library/hello-world"),
    ("public.ecr.aws", "docker/library/hello-world"),
)


def _write(path: Path, text: str) -> Path:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(textwrap.dedent(text).lstrip(), encoding="utf-8")
    return path


def main(scratch: Path) -> None:
    for registry, repository in SOURCES:
        case = scratch / registry
        _write(
            case / "raes-trust.yaml",
            f"""
            schema_version: raes-trust/v1
            registries:
              {registry}:
                require_signatures: false
            """,
        )
        root = _write(
            case / "root.yaml",
            f"""
            name: root
            imports:
              - source: oci:{registry}/{repository}
                namespace: remote
            """,
        )
        print(f"== parse_sdl_file(imports oci:{registry}/{repository})")
        try:
            parse_sdl_file(root)
        except SDLParseError as exc:
            print(f"SDLParseError: {exc}")
            cause = exc.__cause__
            if isinstance(cause, HTTPError):
                print(f"  cause: HTTPError {cause.code} for {cause.url}")
                for value in cause.headers.get_all("WWW-Authenticate") or []:
                    print(f"  WWW-Authenticate: {value}")
                cause.close()
            elif cause is not None:
                print(f"  cause: {type(cause).__name__}: {cause}")
        else:
            print("parsed without error")


if __name__ == "__main__":
    main(Path(sys.argv[1]))