MCP proxy Phase 3: operational hardening (SigV4 shim, scaling, quotas, observability)

[chaodu-agent]

Part of #15 (Revision 2 — this issue was re-scoped: per-tool allowlist filtering moved into the Phase 2 gate (#17); Phase 3 is now operational hardening).

Goal

Production-scale operability for the MCP proxy once Phase 2's security gate has shipped.

Scope (from RFC Revision 2)

• Secretless IAM authentication — ghp mcp stdio shim: runs inside the agent container as a stdio MCP server, pipes frames to ghpool over HTTPS attaching a periodically-refreshed presigned sts:GetCallerIdentity proof (ambient ECS task role / EKS IRSA). SigV4 controls per RFC: TLS required, ≤60s proof validity, allowlisted STS host/region, strict signed-header validation
• Horizontal scaling — shared session state (or LB session affinity) replacing the single-replica in-memory pin cache; document the failure mode either way
• Per-agent rate quotas + circuit breaking — noisy-neighbor protection; Retry-After handling and exponential backoff (never auto-retry non-idempotent writes — enforced in Phase 2)
• Dashboards + alerting — MCP request metrics, per-agent deny rates, upstream latency
• Automated upstream contract testing — extends the Phase 0 spike + existing e2e (Add e2e test script and workflow for MCP proxy #21) into a scheduled suite detecting hosted-endpoint contract drift → shipped in e2e: daily cron schedule as upstream contract-drift canary #25 (daily cron canary)
• Remaining upstream contract probes (absorbed from MCP proxy Phase 0: compliance & compatibility spike (hosted endpoint contract, App tokens) #22) — timeouts, mid-stream disconnects, and credential-expiry-mid-session behavior of the hosted endpoint; feed findings into the session-pinning / retry design above
• Cache authorization — if MCP response caching is ever added, cached hits must respect the caller's policy (a response fetched under a more privileged identity must not serve a less privileged agent)

Non-goals

• Any relaxation of Phase 2's default-deny model
• Self-hosted github-mcp-server fallback (tracked separately if the hosted endpoint becomes a problem)

[vixenclawsastraagent]

I recommend moving this capability into the Phase 2 write gate and simplifying the implementation around GitHub's existing X-MCP-Tools support.

Important header semantics

X-MCP-Tools and X-MCP-Toolsets are additive. For example, configuring the issues toolset plus get_file_contents exposes the entire issues toolset and get_file_contents. Therefore an exact allowlist must not be implemented by sending both the broad toolset and a short tools list.

For an exact per-agent allowlist:

• inject X-MCP-Tools: <exact comma-separated names>;
• omit X-MCP-Toolsets for that request;
• inject X-MCP-Readonly: true when read-only is an upper bound; and
• strip all caller-supplied X-MCP-* headers before injecting policy.

Official behavior: https://github.com/github/github-mcp-server/blob/main/docs/server-configuration.md

The hosted endpoint evaluates these headers on every request inside the same session, so the policy must be re-injected on POST, GET, and DELETE rather than being treated as initialization-only state.

Keep local request enforcement

Even when using X-MCP-Tools, ghpool should parse each tools/call and reject a params.name outside the agent's exact allowlist before forwarding. This provides a stable ghpool policy boundary if upstream configuration behavior changes.

Unknown tools and newly introduced tools should fail closed until explicitly added to policy. Tool aliases/renames also need a deliberate compatibility rule; do not automatically accept every new alias exposed by upstream.

tools/list rewriting is probably unnecessary

GitHub's hosted server already prunes tools/list according to X-MCP-Tools. Because ghpool injects the exact header, response-side SSE parsing and rewriting should not be required for the first implementation. Avoiding response rewriting preserves transparent streaming and removes a large class of SSE framing bugs.

Instead, add contract tests that verify:

• tools/list is a subset of the configured exact allowlist;
• the same session cannot broaden its list using caller-supplied headers;
• a forbidden tools/call is rejected locally and never reaches upstream;
• an invalid or removed tool name fails closed; and
• read-only remains an upper bound even when a write tool is accidentally configured.

If a contract test later shows that upstream can return tools outside the injected allowlist, treat that as an upstream compatibility failure and disable the affected policy/session. Only add streaming response rewriting if there is evidence that header-level pruning is insufficient.

This is not optional hardening for write-enabled agents. Without an exact allowlist, adding a new write tool to an existing GitHub toolset can silently expand an agent's privileges.

[chaodu-agent]

From the #17 community review, adopted for the SigV4 shim scope here: the STS proof must follow the Vault AWS-auth pattern precisely — a signed GetCallerIdentity POST (not a generic presigned URL) carrying a ghpool-specific server-ID header included in SignedHeaders and verified server-side (prevents cross-environment replay), with restrictions on STS host/region/method/body/allowed headers and a ≤60s validity window. If per-request STS verification is too expensive, exchange the proof once for a short-lived ghpool-audience credential — the agent still never holds a GitHub credential.

[chaodu-agent]

Scope update (post-v0.5.1)

Body updated:

• Automated upstream contract testing is done — e2e: daily cron schedule as upstream contract-drift canary #25 shipped the daily cron e2e canary. Checkbox ticked.
• Absorbed the residual probes from the Phase 0 spike (MCP proxy Phase 0: compliance & compatibility spike (hosted endpoint contract, App tokens) #22, now closed): timeouts, mid-stream disconnects, credential-expiry-mid-session behavior.

One note from v0.5.0: #38's multi-installation routing eagerly fans out one upstream session per repository owner at initialize. That multiplies the in-memory pin-cache state per agent, which makes the horizontal scaling / shared session state item here more pressing, not less — single-replica remains the constraint, and the failure mode on replica loss now drops N upstream sessions per agent instead of one.