Allow Gemfury storage backend for jobs using a Gemfury registry

[sbresin]

Alternative to #286. Only one of the two should be merged. #286 adds the Gemfury bucket to the static defaults (1 line). This PR derives it per job instead, so only jobs that use Gemfury can reach it. I opened both so you can pick the approach that fits your allowlist policy, and I'll close the other one.

What are you trying to accomplish?

Since the egress allowlist started enforcing, every Dependabot update that downloads a package from a Gemfury registry fails. The authenticated registry request succeeds, but the download is blocked:

proxy | GET https://pypi.fury.io:443/<account>/-/ver_xxxxx/<pkg>-16.0.1-py3-none-any.whl
proxy | * authenticating python index request (host: pypi.fury.io)
proxy | 302 https://pypi.fury.io:443/<account>/-/ver_xxxxx/<pkg>-16.0.1-py3-none-any.whl
proxy | GET https://gemfury.s3-accelerate.dualstack.amazonaws.com:443/gems/<id>/<file>?X-Amz-...
proxy | * egress not allowlisted gemfury.s3-accelerate.dualstack.amazonaws.com
proxy | 403 https://gemfury.s3-accelerate.dualstack.amazonaws.com:443/gems/<id>/<file>?X-Amz-...
updater | Handled error whilst updating pyjwt: ... {source: "https://gemfury.s3-accelerate.dualstack.amazonaws.com"}

Every Gemfury registry endpoint 302-redirects package downloads to a short-lived pre-signed URL on one Gemfury-owned S3 bucket, gemfury.s3-accelerate.dualstack.amazonaws.com. That host appears in no credential field, so credentialHosts never adds it. I checked this directly against both pypi.fury.io (wheel) and npm.fury.io (tarball): both redirect to that exact host. Older files are sometimes served inline with a 200, which is why the failure shows up per package rather than per registry.

This PR adds the bucket to registryRedirectHosts, next to the ECR starport bucket. It's added only for jobs that have a credential for a *.fury.io host.

Anything you want to highlight for special attention from reviewers?

Why not the static defaults: the bucket is shared by all Gemfury accounts, with the account in the URL path. add-egress-allowlist-domain rules out shared multi-tenant hosts like that for the static defaults, because allowing the host would expose every tenant's content to every job. Deriving it per job keeps it closed for everyone who doesn't already use Gemfury. This is the same reasoning as the existing ECR case.

Why the derivation is safe: the derived host is a constant. Nothing from the credential is interpolated into it, so a crafted credential can't widen it. It's also matched exactly, like all dynamic hosts. The trigger is strings.HasSuffix(host, ".fury.io"). That covers all Gemfury endpoints (pypi., npm., npm-proxy., gem., ...) without keeping a list that would go stale. The worst a lookalike credential could do is open this one fixed Gemfury bucket for its own job.

Ownership evidence:

• pypi.fury.io and npm.fury.io themselves issue the redirect to this host.
• The TLS certificate is Amazon's (CN=*.s3-accelerate.amazonaws.com, issuer Amazon RSA 2048 M01).
• The bucket name is gemfury. S3 bucket names are globally unique.

Compared with #286: #286 is the smaller change and follows the JFrog shared S3 buckets precedent. This PR follows the rule in add-egress-allowlist-domain for multi-tenant hosts more strictly, at the cost of a little code in registryRedirectHosts.

How will you know you've accomplished your goal?

New tests in internal/handlers/egress_dynamic_hosts_test.go:

• TestRegistryRedirectHosts_GemfuryStorageDerived:
  • With a pypi.fury.io credential, both the registry and the bucket are allowed.
  • Still blocked:
    • a child host (evil.gemfury.s3-accelerate...)
    • a lookalike bucket (gemfuryx.s3-accelerate...)
    • the same bucket on a different endpoint (gemfury.s3.amazonaws.com)
    • the shared parent (attacker.s3-accelerate.dualstack.amazonaws.com)
• TestRegistryRedirectHosts_OnlyGemfuryHosts:
  • The bucket is derived from pypi., npm., npm-proxy. and gem.fury.io.
  • Nothing is derived from fury.io, pypi.fury.io.evil.com, pypifury.io or evil.com.
  • Several Gemfury credentials yield a single entry.
• TestRegistryRedirectHosts_NotAddedWithoutGemfuryCredential: a job without a Gemfury credential can't reach the bucket.

I checked that the tests catch a regression: loosening the matcher to strings.Contains(h, "fury.io") makes the lookalike cases fail.

script/test (-race -shuffle=on -count=2) passes, and so do go vet and gofmt -l.

Checklist

• I have run the complete test suite to ensure all tests and linters pass.
• I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
• I have written clear and descriptive commit messages.
• I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
• I have ensured that the code is well-documented and easy to understand.

[Copilot]

Copilot review overview

🟢 Approval recommended

The narrowly scoped implementation is consistent with existing dynamic-host handling and has comprehensive regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds per-job access to Gemfury’s shared download bucket without widening the global egress allowlist.

Changes:

• Derives the exact Gemfury S3 host from *.fury.io credentials.
• Adds positive, isolation, lookalike, and deduplication tests.

File	Description
internal/​handlers/​egress_dynamic_hosts.go	Adds conditional Gemfury redirect-host derivation.
internal/​handlers/​egress_dynamic_hosts_test.go	Verifies exact matching and credential scoping.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sbresin]

superseded by #288

[v-abhishekbhaskar]

Hey @sbresin, thanks for the PR! I've covered this in #288, along with another redirected host derivation as well. Those changes should be deployed now, so your domains should be allowlisted now. Apologies for overriding your changes in this PR!