api: abort a response mid-serialization once it exceeds a byte budget

[ohohoreilly]

The api can accept a single GraphQL request it cannot survive. On 2026-08-19 one client sent ~19 requests/min of a query that allocates ~1.5 GB each against a 2 GiB pod limit, killing 5 pods before it was tracked down and fixed in the client's repo. That resolution depended on having write access to another team's code, which doesn't generalize. The API should refuse what it can't serve.

What the measurements rule out

Canary runs (isolated pod, app: api-canary, one request each, cgroup sampled every 200 ms):

root first × nested relations	response	duration	peak RSS delta
50 × 1000	3.05 MB	6.6 s	+152 Mi
100 × 100	5.89 MB	5.8 s	+260 Mi
500 × 100	29.0 MB	18.1 s	+724 Mi
1000 × 1000	62.9 MB	33.3 s	+1,569 Mi

Not a static budget over declared limits. 50 × 1000 and 500 × 100 are the same 50k product but cost 152 Mi vs 724 Mi, because declared limits are upper bounds and only realized rows are paid for. A walker over the query document over-rejects cheap queries and still misses expensive ones.

Not the cost ceiling. GRAPHQL_COST_REJECT_THRESHOLD is armed at 500; the killer scored 274, and the 08-06 incident scored 228. A ceiling low enough to catch 274 also catches ~2–3% of ordinary traffic (≤260 is 96.8% of requests). Cost counts nodes; the OOM is driven by bytes hydrated per node.

Not concurrency. See #897/#900 — the cap moved 10→6→10; 6 doubled 503s to 1414/h with no OOM reduction, because this is a single-request kill.

Not a memory-limit bump. 4 of 8 pods share one node with 6.5 Gi allocatable; 3 Gi limits would risk node-level eviction, which is a worse failure mode.

So the only axis the data supports is realized bytes/rows, counted at runtime, with an abort.

Proposed shape

Count as the response materializes and abort past a ceiling, returning a structured error instead of dying. A ~10–15 MB ceiling would have fired on ~78 requests/18 h (~4/h) while bounding peak RSS to roughly 300–750 Mi.

Open design question — where to count:

1. Per-connection realized rows, accumulated per request, checked as each connection resolves. Aborts before the whole response exists. Note paginationCapPlugin's docstring warns makeWrapResolversPlugin only reliably influences SQL at root level — but counting rows doesn't need to influence SQL, so wrapping should be viable here.
2. During serialization, which is where the 25–50× amplification actually happens. Catches everything, but by then the object graph is already in memory, so it bounds the string not the graph.

(1) prevents more of the allocation; (2) is closer to the real cost. Possibly both, with (1) as the guard and (2) as the backstop.

Follow the existing Phase 1 / Phase 2 convention from costLoggerPlugin: land the accounting observational with an env-gated ceiling (0 = off), confirm against real traffic what a ceiling would have rejected, then enforce.

Related

• perf(api): stop double-serializing the responses that OOM pods #901 removes an amplifier on this path — the response-size metric was calling JSON.stringify a second time on exactly these responses. Its estimateJsonBytes walker is a possible basis for (2), though as written it runs post-hoc in onExecuteDone.
• The deeper fix, separate and larger, is cutting the 25–50× factor itself: streaming/chunked serialization rather than building one giant JSON string.
• geobrowser/geo-skills#5 fixes the docs that taught clients to write this shape.
• Root cause writeup: geo-explorers/postgres_to_geo#36, and their follow-up #37.

[ohohoreilly]

Phase 1 landed and deployed — #905, merged and live on v2 as d7aa24e1.

Enforcement is off (GRAPHQL_RESPONSE_BYTE_CEILING=0), as designed. What shipped:

• measureJsonBytes(value, limit) — the existing walker with an early exit, so an oversized response is abandoned rather than walked.
• useResponseBudget() — measures every response, warns over GRAPHQL_RESPONSE_BYTE_LOG_THRESHOLD (10 MB) with fingerprint + caller identity, and exposes gaia_api_graphql_response_bytes (unbiased over all responses, unlike _response_size_bytes which samples only ≥1s queries) plus _budget_exceeded_total / _budget_refused_total.
• Refusal path returns BAD_USER_INPUT / HTTP 400 and drops data. BAD_USER_INPUT specifically — errorMasking unmasks only that and SERVICE_UNAVAILABLE, and isClientError uses the same set to keep refusals out of Sentry. A bespoke code yields a 500 with the message stripped; there's an e2e test pinning it.
• Two amplifiers on this path removed: instrumentation now reuses the shared measurement instead of walking twice, and valkeyCache.set tests size with the walker before JSON.stringify (its docstring previously conceded "the stringify still happens").

On the open design question — I implemented (2) and deliberately not (1). PostGraphile v4 issues one SQL query per root field with nested collections as lateral joins, so by the time a nested connection's resolver runs its whole subtree is already hydrated; counting there observes an allocation rather than preventing it. (1) only prevents work on multi-root documents, and the 08-19 killer was single-root. Worth revisiting only if Phase 1 logs show over-budget requests are multi-root.

Phase 2 is not ready to arm. First data from one pod: 3,268 responses, p100 under 2 MB, mean 27 KB, budget_exceeded_total = 0. A 10–15 MB ceiling would reject nothing today. But that's ~30 min on 1 of 8 pods, and the client that caused the incident was already fixed in geo-explorers/postgres_to_geo#36 — so the pathological traffic is simply gone and this sample can't tell us where a ceiling belongs. Needs days of Prometheus data across pods before setting a number.

That same data retires a risk I flagged pre-deploy: the walk now runs unconditionally on the hot path, which I worried about given the 08-06 event-loop-starvation incident. Measured cost is ~0.6–0.7× a JSON.stringify on large payloads, but at an observed p100 of 2 MB it's microseconds, not the ~20 ms I extrapolated from a 5.6 MB payload. 8/8 api pods, 0 restarts, no OOMKills post-deploy.

Keeping this open for Phase 2. The remaining deeper item — streaming/chunked serialization to cut the 25–50× factor itself — is still unaddressed and still the larger fix.

[ohohoreilly]

Status check (2026-09-17) — Phase 1 shipped, Phase 2 did not, so this is not done.

api/src/kg/responseBudgetPlugin.ts exists and is wired into postgraphile.ts via useResponseBudget(). It measures every response, warns at GRAPHQL_RESPONSE_BYTE_LOG_THRESHOLD (10 MB default), bounds its own measuring walk, and exports gaia_api_graphql_response_budget_exceeded_total and _refused_total.

But enforcement is off. RESPONSE_BYTE_CEILING defaults to 0, which the code documents as disabling refusal entirely — the deliberate shipped default. And GRAPHQL_RESPONSE_BYTE_CEILING is set neither in api/k8s/v2/api.yaml nor on the live api deployment.

So today the api still cannot refuse what it cannot serve; it can only tell you afterwards that it served something enormous. The original scenario — one client's query allocating ~1.5 GB against a 2 GiB limit — would still kill pods.

What remains is arming it: pick a ceiling from the _exceeded_total counter and the warn logs now that there is real data, set it in the v2 manifest, and confirm _refused_total moves. Worth checking the counters before choosing a number rather than guessing one.