Skip to content

Allowlist nvcr.io for docker update jobs - #302

Merged
v-robaiken merged 1 commit into
dependabot:mainfrom
rajathagasthya:rajathagasthya/allowlist-nvcr-io
Oct 12, 2026
Merged

v-robaiken merged 1 commit into
dependabot:mainfrom
rajathagasthya:rajathagasthya/allowlist-nvcr-io

Conversation

@rajathagasthya

@rajathagasthya rajathagasthya commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What are you trying to accomplish?

Hosted Dependabot docker version updates for images on nvcr.io (NVIDIA's NGC container registry) fail since proxy-egress-enforce was enabled. The proxy returns 403 before the request reaches the registry, and the updater reports it as private_source_authentication_failure:

proxy | GET https://nvcr.io:443/v2/nvidia/cuda/tags/list
proxy | * egress not allowlisted nvcr.io
      | 403 https://nvcr.io:443/v2/nvidia/cuda/tags/list
updater | Handled error whilst updating nvidia/cuda: private_source_authentication_failure {source: "nvcr.io"}

Public NGC images need no sign-in, so there is no registries: credential a user could add for them. The NGC Catalog User Guide (section 2.4, "Accessing NGC Software") states: "Pulling public software from the NGC Catalog does not require sign-in."

This adds nvcr.io and layers.nvcr.io as exact hosts to the docker: &docker_registries anchor, so docker_compose and devcontainers inherit them.

Anything you want to highlight for special attention from reviewers?

The request chain (checked anonymously):

  • /v2/ answers 401 with www-authenticate: Bearer realm="https://nvcr.io/proxy_auth",service="registry". The token service is on the same host.
  • proxy_auth returns a guest token without credentials (iss: authn.nvidia.com, sub: sf:guestclient). The token grants pull only, even when push,pull is requested. For a repository that is not public, the token's access list is empty.
  • tags/list and manifests/<tag> return 200 with that token.
  • blobs/<digest> returns 307 to layers.nvcr.io with a CloudFront signed URL (Expires, Key-Pair-Id, Signature). layers.nvcr.io returns 403 for unsigned requests.

Why layers.nvcr.io is included. The observe-mode logs only show nvcr.io (proxy_auth and tags/list), because the docker updater reads tags and manifests. I added the blob host to follow the rule in TestEgressAllowlist_PublicVendorOCIRegistriesAllowed (registry, token service, and blob redirect target), the same as the Elastic and Chainguard entries. It only serves URLs that nvcr.io signs, so it is not a general download host. I can remove it if you prefer to add only observed hosts.

Why the exact form is safe. Tenancy is path-based (nvcr.io/<org>/<team>/<image>). Private NGC org images on the same host still need an NGC API key, which users configure under registries:. This is the same model as Docker Hub, ghcr.io, and .quay.io, which are already on the list.

Ownership evidence. TLS certificate for both hosts: subject=CN=nvcr.io, SAN nvcr.io, *.nvcr.io, issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M04. nvcr.io resolves to public AWS addresses, and layers.nvcr.io resolves to CloudFront addresses. The token issuer is authn.nvidia.com. The NGC Catalog User Guide (section 2.11.2.1) names nvcr.io as the NGC container registry.

How will you know you've accomplished your goal?

Reproduction: a real job in NVIDIA/mig-parted (run), where the proxy logs * egress not allowlisted nvcr.io for nvidia/cuda, nvidia/distroless/go, and nvidia/cloud-native/k8s-mig-manager. The same job passed on 2026-09-29 under proxy-egress-observe (run), and has failed every day since 2026-09-30.

Regression coverage in internal/handlers/egress_allowlist_test.go, TestEgressAllowlist_PublicVendorOCIRegistriesAllowed:

  • Positive: the tags/list and manifests paths from the failing job, the same-host token endpoint, and a layers.nvcr.io blob path.
  • Negative child: https://evil.nvcr.io/v2/ and https://evil.layers.nvcr.io/registry/. These start to pass if either entry is widened to a leading dot.

Tests were not run locally. This PR is a draft until CI passes.

Checklist

  • I have run the complete test suite to ensure all tests and linters pass.
  • I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
  • I have written clear and descriptive commit messages.
  • I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
  • I have ensured that the code is well-documented and easy to understand.

@rajathagasthya
rajathagasthya marked this pull request as ready for review October 7, 2026 21:06
@rajathagasthya
rajathagasthya requested a review from a team as a code owner October 7, 2026 21:06
Hosted Dependabot docker version updates for images on nvcr.io (the
NVIDIA NGC container registry) fail under proxy-egress-enforce: the
allowlist 403s nvcr.io before the request reaches the registry, and the
updater reports it as `private_source_authentication_failure`.

Public NGC images need no sign-in, so there is no `registries:`
credential a user could configure for them.

Add nvcr.io and layers.nvcr.io as exact hosts on the `docker` anchor,
so docker_compose and devcontainers inherit them:

  * /v2/ advertises `Bearer realm="https://nvcr.io/proxy_auth"`. The
    token service is on the same host and issues a guest token with
    pull access to public repositories only.
  * Blob requests 307-redirect to layers.nvcr.io, NVIDIA's CloudFront
    distribution. It serves only URLs signed by nvcr.io and returns 403
    for unsigned requests.

Exact only: tenancy is path-based (nvcr.io/<org>/<team>/<image>), and
private org images still need an NGC API key under `registries:`.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
@rajathagasthya
rajathagasthya force-pushed the rajathagasthya/allowlist-nvcr-io branch from e1c9b9e to dd5f4b0 Compare October 9, 2026 15:33
@v-robaiken v-robaiken added the allowlist Changes the default egress allowlist label Oct 12, 2026
@v-robaiken
v-robaiken merged commit fc47d47 into dependabot:main Oct 12, 2026
107 of 108 checks passed
@rajathagasthya
rajathagasthya deleted the rajathagasthya/allowlist-nvcr-io branch October 12, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

allowlist Changes the default egress allowlist

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants