test(e2e-prod): response-schema validation — suite records, gate validates

[jiashuoz]

What

Closes the audited conformance gap: nothing validated live response bodies against api/openapi.yaml — harness/client.ts checks status codes, suites assert hand-picked fields, and harness/coverage.ts records 2xx pairs only. A dropped required field, a wrong type, or an error path returning a bare string instead of the envelope all shipped invisibly.

The suite is deliberately zero-dependency (the ops release pipeline runs it with no npm install), so validation cannot happen in-process. The split: the suite records, the gate validates.

• harness/responses.ts (new) — records every ApiClient response as a {method, path, status, contentType, kind, body} sample in reports/response-samples/ (per-pid shards, same exit-flush + pretest-clean pattern as coverage.ts). Records all statuses, not just 2xx: every op documents default → ErrorEnvelope, so each 401/404/422 the suites already provoke is a free error-contract check. The coverage channel's 2xx-only filter is untouched — coverage answers "did the op run?", this answers "was the body well-shaped?".
• response_schema_gate.py (new) — maps each sample to its operationId with coverage_gate.py's matcher, picks the response schema for the exact status (falling back to default), resolves $refs against components, validates under Draft 2020-12 (the spec is OpenAPI 3.1). Needs jsonschema alongside the existing pyyaml. Extra fields are not violations (responses are deliberately additionalProperties: true, and format stays annotation-only); missing required fields, wrong types, empty/non-JSON bodies where JSON is documented, and undocumented statuses are. House gate contract: exit 2 on no-shards/inconclusive (never 0), 1 on violations, 0 clean; empty ALLOWLIST with the stale-entry check.
• harness/client.ts — one recordResponse(...) call after body parse.
• package.json — coverage:gate:responses script; pretest clears the new shard dir.
• test_gates.py — 6 new subprocess tests (skip cleanly without jsonschema), including explicit proof the gate can FAIL: missing required field, wrong type, and non-JSON body each exit 1; extra fields exit 0; no shards exit 2.

Measured against live staging (partial run, 2026-08-02)

Suites 01–13 against api-staging.e2a.dev (run cut short deliberately; a Free-plan account capped agent-heavy paths). 379 samples, 13 shards, 28/72 operations sampled (274× 2xx + 105× 4xx), 360 valid, 1 violating sample:

VIOLATIONS (1 group(s), 1 sample(s)):
  - getAgent 404 (1x, e.g. /v1/agents/..%2F..%2Fetc%2Fpasswd)
      non-JSON body where spec documents application/json (default): 'Not Found'

Root cause (verified by curl + config): Caddy URI-normalizes ..%2F.. so the path escapes /v1/* and hits the api-host allowlist catch-all (respond "Not Found" 404 — ops Caddyfile.staging/Caddyfile), so the request never reaches the app. The server's own routeNotFound correctly returns the JSON envelope for real /v1 paths (control: bogus agent → JSON not_found + x-request-id). The attribution to getAgent is a recorder artifact (the client-side pathname keeps %2F encoded, so it maps as one segment); the finding itself — the hosted edge answers some /v1-looking requests with a non-envelope plain-text 404 — is real, and is an edge-config question for e2a-ops, not a server bug.

Every app-generated response sampled — including all 105 error responses — conformed. Not sampled (suites 14+ not reached): templates, webhooks, reviews (mostly), api-keys, contacts, suppressions, trash, attachments, starter templates.

Not in this PR

• Not wired into the ops release pipeline, not blocking. Wiring is a later e2a-ops PR (pip install jsonschema next to the existing pyyaml step + running coverage:gate:responses). The measured violation rate (1 infra-layer finding, 0 app-layer, across 379 samples) suggests it can block with at most one justified allowlist entry once the remaining 44 ops have been measured on a full staging-gate run.

Verification

• npm run typecheck clean; node --test harness/*.test.ts 18/18.
• python3 -m unittest test_gates -v 12/12 (venv with pyyaml + jsonschema).
• Synthetic-violation proof the gate can fail:

VIOLATIONS (2 group(s), 2 sample(s)):
  - getAccount 401 (1x, e.g. /v1/account)
      [default schema] $.error.code: 401 is not of type 'string'
  - getAccount 401 (1x, e.g. /v1/account)
      [default schema] $.error: 'request_id' is a required property

GATE: FAIL — the above response(s) do not conform to api/openapi.yaml. Fix the server or the spec, or add an ALLOWLIST entry with a justification.

• Missing/empty shard dir → exit 2 with "no response-sample shards … did the suite run?" (never a vacuous pass).

🤖 Generated with Claude Code