Configurable JWKS URI and in-cluster backendRef for internal JWT auth

[tylerpotts]

Problem

On private-CA / BYO-cert deployments (the mode platform.tls.secretName exists for), the internal endpoint rejects every JWT with:

Jwks remote fetch is failed        (response to the client, HTTP 401)
jwt_authn_access_denied{Jwks_remote_fetch_is_failed}   (Envoy access log)

with the proxy stuck in a ~1/sec async JWKS fetch retry loop. Root cause, confirmed on a live cluster (Envoy Gateway v1.6.2): the SecurityPolicy's remoteJWKS.uri is <issuerURL>/protocol/openid-connect/certs, so Envoy resolves the external Keycloak hostname (hairpinning through its own gateway), is served the private-CA certificate, and fails TLS verification because the JWKS fetcher only trusts public roots. An in-cluster curl of the same URL fails without the CA and succeeds with -k, isolating the failure to trust, not reachability. Policy status is Accepted throughout; the failure is purely runtime.

Fix

Split key fetching from issuer validation, mirroring the key-manager's existing external-issuer / in-cluster-URL split (keyManager.keycloak.url vs issuerURL):

• Operator: new optional config LLM_OIDC_JWKS_URI (overrides the JWKS URL; default unchanged) and LLM_OIDC_JWKS_BACKEND_SERVICE / _NAMESPACE / _PORT, which render a remoteJWKS.backendRefs Service entry on the JWT SecurityPolicies. With an http:// in-cluster Keycloak URL plus the backendRef, the key fetch needs no trust material at all, while issuer continues to validate the token's external iss claim. Applies to both LLMModel and PassthroughModel policies via the shared buildJWTSecurityPolicy.
• Chart: auth.oidc.jwks.{uri,backendRef.{name,namespace,port}} plumb the env vars, and a ReferenceGrant (from: SecurityPolicy in the release namespace, to: the Service) is rendered in the backend Service's namespace when it differs from the release namespace, which cross-namespace backendRefs require. The operator's ClusterRole already covers everything involved; no RBAC change.
• Drive-by fix: LLM_OIDC_AUDIENCE was never rendered by the operator Deployment template even though auth.oidc.audience exists in values and config.go reads it.

Default rendering is byte-identical to before; remoteJWKS.backendRefs requires Envoy Gateway >= v1.2 only when the backendRef is actually configured.

Example values for a private-CA cluster:

auth:
  oidc:
    issuerURL: https://keycloak.nebari.example.com/realms/nebari
    jwks:
      uri: http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080/realms/nebari/protocol/openid-connect/certs
      backendRef:
        name: keycloak-keycloakx-http
        namespace: keycloak
        port: 8080

Relation to #61

This does not implement OIDC discovery (the JWKS path default is still Keycloak-shaped), but auth.oidc.jwks.uri now also unblocks non-Keycloak providers by letting deployments state the JWKS URL explicitly. Docs updated accordingly (architecture callouts + configuration table).

Testing

• make test passes (envtest suite included; reconcilers at 95.5% coverage).
• New table-driven cases in auth_test.go: default URI derivation with no backendRefs, URI override without backend, backendRefs rendering with and without namespace.
• helm lint passes; helm template verified for (a) defaults, byte-identical env/no ReferenceGrant, and (b) full JWKS values, correct env vars plus the ReferenceGrant in the Service's namespace.

[github-actions]

📄 Docs preview for fix/jwks-in-cluster-backendref:
https://fix-jwks-in-cluster-backendr.llm-serving-pack.pages.dev