[FEATURE] Fetch upstream with the client's credential instead of a shared token #68

Is your feature request related to a problem?

The proxy reaches upstream with one static credential (--upstream-auth-header). For it to mirror many repos, that credential must read all of them. On GitLab that usually means a group-level access token, so a leak of that one secret (from the secret store, the pod env, or the deployment) exposes every repo in the group.

#56 (for #40) validates each client's own credential per repo, which bounds what a client can read through the proxy. It explicitly keeps the shared upstream credential for clone/fetch, so the blast radius of that token is unchanged.

CI jobs already hold a short-lived, repo-scoped credential (GitLab CI_JOB_TOKEN, GitHub GITHUB_TOKEN). When a request needs upstream, the proxy could use that credential and need no long-lived upstream secret.

Describe the solution you'd like

An opt-in upstream credential mode in which the proxy clones and fetches with the credential of the request that triggered the upstream work. It builds on #56 and requires a provider mode (gitlab / github).

Why this fits the current design:

• Every upstream call is already request-triggered, with no background refresh. info_refs and upload_pack call GitCache::ensure_fresh (clone, fetch, want-by-SHA, all through fetch_cmd), and the LFS batch and object handlers call post_batch. A live client credential is therefore always available when upstream is contacted.
• The credential already reaches git through env-based config (git_config_env -> http.extraHeader in GIT_CONFIG_*). It never appears in argv or is persisted in the mirror's config. Only the value's source changes.
• feat: implement provider for token based auth #56 already requires TLS on --upstream in provider mode (probe_base), so the fetch carries the client credential over the same checked URL as the probe.

Configuration sketch:

--upstream-credential static|client      (env GITCACHEPROXY_UPSTREAM_CREDENTIAL, default static)

• static: the current behavior, using --upstream-auth-header.
• client: forward the requesting client's credential upstream. Startup rejects it unless --auth-provider is gitlab or github (custom has no upstream-shaped credential), and rejects it together with --upstream-auth-header, following feat: implement provider for token based auth #56's forbid pattern for conflicting flags. There is no static fallback: in provider mode feat: implement provider for token based auth #56 already denies a request without a credential before any handler runs, so a fallback would never be reached. Public repos still need the client to present a credential, which the git hosts accept. The startup warning "no upstream auth header set; contacting upstream anonymously" does not apply in this mode and is replaced by a line naming it.

Request flow in client mode:

1. feat: implement provider for token based auth #56 validation runs first, unchanged. The credential is checked against the exact repo being served. Validation stays mandatory and is not replaced by the fetch, because a cache hit (fetch_ttl not expired, or a coalesced waiter) never touches upstream, and the mirror is shared across clients.

2. The proxy takes the upstream-shaped credential from feat: implement provider for token based auth #56's Auth::forwarded_credential(&headers). It returns Basic as sent, which preserves the gitlab-ci-token username, and turns Bearer into Basic oauth2:<token> (GitLab) or x-access-token:<token> (GitHub). The value is marked sensitive, and the result is None outside github/gitlab mode. The validation probe already builds its header through the same Credential::forwarded, which a test covers, so the check and the fetch cannot disagree. It is computed from the request headers, so it also works when the verdict came from the TTL cache.

   Nothing consumes it yet: Verdict carries no payload, and gate only inserts the Route extension. This proposal adds the wiring. After an Allow, gate inserts the forwarded credential as an UpstreamCredential request extension next to Route. The handlers pass it per call: ensure_fresh(repo, want_fetch, body, cred) -> fetch_cmd(cred), and likewise post_batch. In static mode the extension is absent and GitConfig.upstream_auth_header is used as today.

3. Coalescing keeps its current shape. The RepoSlot.fetch_lock holder does the clone or fetch with its credential, and waiters, already validated for the same repo, are served from the result. If the holder's fetch fails (for example, the token expired mid-fetch), last is not stamped, so the next waiter fetches with its own credential. That already falls out of the existing lock.

4. Upstream auth failure maps to 401, not 502. Today every ensure_fresh error is a 502 upstream fetch failed. In client mode the credential is the client's, so a fetch that fails with an auth diagnostic becomes a 401 with the Basic challenge, as if validation had denied it, and the (credential, repo) entry is dropped from feat: implement provider for token based auth #56's verdict cache so the next request re-probes. This is the --auth-ttl-seconds window: a token revoked after a cached Allow. transient_upstream_error already classifies "authentication failed" and "returned error: 401/403/404" as permanent, so the fetch error only needs to carry that classification out. static mode keeps the 502.

LFS:

• The batch request carries the client credential, so post_batch forwards it like git does.
• An object GET does not carry it; it carries the grant feat: implement provider for token based auth #56 issues in the download action. On a cache miss ensure_object re-batches upstream, today with the static credential. Rather than keeping per-object state in the proxy, the grant carries the forwarded credential: the grant body becomes an AEAD ciphertext whose plaintext is the credential (empty in static mode) and whose associated data is repo, oid, size and the expiry. The AEAD tag replaces the HMAC, so a grant for one object still cannot fetch another's, same as feat: implement provider for token based auth #56's check today. The key is derived from --auth-signing-key (already at least 32 bytes) with a distinct label, so sealing and any other use of the key never share material.
• gate unseals the grant on the object GET and inserts the same UpstreamCredential extension as for git routes, so ensure_object's existing re-batch path simply uses it. The wiring is identical for all four routes.
• Why not a map in the proxy: the batch and the object GET can land on different replicas, and feat: implement provider for token based auth #56 already assumes replicas behind one address share the signing key. A sealed grant is stateless, so it works across replicas and needs no bound or expiry bookkeeping. The expires_in feat: implement provider for token based auth #56 advertises (15 minutes) stays as is; if the sealed credential dies first, the re-batch fails upstream and is mapped to 401 per step 4 above.
• Size: the grant rides in the download action's Authorization header, not the URL, so it is not written to access logs. A Basic credential adds roughly 150 bytes after sealing and base64.

Credential hygiene:

• Never log the credential or include it in error contexts. Upstream stderr can echo URLs but not extraHeader values.
• The credential lives in the git subprocess env, readable only by the proxy's own uid, which matches the static token today.
• Threat model, for the README security section: the proxy process and its git children now handle many clients' tokens in memory, and sealed copies ride in LFS grants. They are short-lived and repo-scoped, but a compromised proxy, or a leaked signing key, can use them for their remaining lifetime. Against that, no long-lived upstream secret exists anywhere in the deployment.

Out of scope:

• Any change to the read-only, pull-only invariant. Credentials are only used for clone, fetch and LFS download.
• Scheduled or background refresh. It would need a stored credential, which is what this mode removes.

Platform behavior to document:

• A GitLab CI_JOB_TOKEN can read its own project, plus projects whose job token allowlist includes the caller. Cross-project and submodule clones through the proxy follow the same rules as direct clones.
• With no TTL, a cold repo pays one validation plus one upstream clone. Warm repos behave as in feat: implement provider for token based auth #56.

Deliverables beyond src/:

• README: flag table row, the Auth model section, and the security section (threat model above, plus the 401 mapping).
• Chart: an upstreamCredential value; the upstream secret becomes optional in client mode; chart/README.md.
• Optional: a credential label (static / client) on the upstream fetch counter, so an operator can confirm which mode a fleet is on.

Tests:

• Unit: git_config_env with a per-call header. feat: implement provider for token based auth #56 already covers the credential conversion and its parity with the probe.
• Unit: config rejects client without a github/gitlab provider, with custom, and with --upstream-auth-header.
• Unit (grant): seal/unseal round-trip; a tampered grant, or one presented for another repo/oid/size, is refused; an expired grant is refused, on the paused tokio clock; a static mode grant carries no credential.
• Unit: the auth diagnostics transient_upstream_error already treats as permanent are classified as auth failures.
• Integration, using a new fixture. The e2e and retry suites use a file:// upstream, which ignores headers, and feat: implement provider for token based auth #56's tests/helpers mock only answers info/refs. The new fixture is git http-backend behind a small local CGI shim that requires and records Authorization, and it can also serve feat: implement provider for token based auth #56's validation probe. Cases:
  • A fetch uses the requesting client's header: Basic forwarded verbatim, Bearer as oauth2: / x-access-token:.
  • When the holder's fetch fails, a waiter retries with its own credential.
  • An upstream 401 on fetch returns 401 with the challenge, and the next request re-probes instead of hitting the verdict cache.
  • An LFS object miss re-batches upstream with the credential unsealed from the grant.
  • Two proxies sharing a signing key: a grant issued by one is honored by the other.

Describe alternatives you've considered

• Narrower static tokens (project tokens, or a GitLab service account added only to selected projects) or one proxy per subgroup. These shrink the blast radius but still keep a long-lived secret, and cache sharing is split across deployments.
• A static map of path prefix -> token. It is simpler but still needs long-lived secrets, just more of them.
• feat: implement provider for token based auth #56 alone. It secures client access, but the upstream secret stays as broad as the fleet.
• For LFS, an in-memory map of upstream download actions keyed by (repo, oid), consulted on the object GET. It needs a bound and expiry, breaks across replicas, and requires capping the advertised expires_in by upstream's. The sealed grant above avoids all three.

Scope check

• Keeps the proxy strictly read-only and pull-only (it must never push or proactively replicate to upstream).
• Not already tracked in the README "Status / scope" (LRU eviction, per-repo histograms, scheduled refresh) or listed there as a non-goal (git-free image).

Additional context

Depends on #56 (provider validation, and Auth::forwarded_credential added in c50e2eb). The auth.rs, server.rs and lfs.rs names above refer to #56 as of a4b5f7d and may move before it merges. Related: #40.

I am happy to implement this once the approach is agreed, as a follow-up to #56. If forwarded_credential is dropped from #56 because nothing uses it yet, the follow-up PR adds it back.