fix(runtime): stop dispatching control-plane requests abandoned before their body completes

[doublewhy]

Summary

#1091 asks for complete acceptance cases and documentation for the control-plane HTTP request-size boundary, RequestSizeLimitMiddleware. The new cases drive the middleware, and the composed P2 adapter, through the public ASGI interface with a scripted server channel. They found one defect: a request whose client disconnected before its body was complete still reached the app. On dev, a POST /workflows/reconcile-timeouts that declared Content-Length: 10 and then disconnected ran reconcile_workflow_timeouts, a GET /snapshot ran get_snapshot, and body routes answered the departed client with 400. The middleware also stored the partial bytes in scope["state"]["raw_body"].

The middleware now treats a disconnect before the final body frame as an abandoned request. No route runs, nothing is sent, and no partial body is kept. Accepted, oversized and invalid-length requests behave as before. A new "Bound request bodies" section in the public control-plane guide covers the bounded routes, the limit setting, the counting point, buffering and replay, disconnects, rejections, and the ASGI server and proxy duties. It promises no streaming pass-through. The guard change moves the bound implementation digest, so both research evidence captures are republished.

Existing cases. Done-when 1 starts with an inventory. These size-guard tests were already on dev:

• test_issue_1093_request_rejection_offload.py: limit and pending-bound validation, non-HTTP pass-through, coalesce then delegate, 100 empty frames, disconnect before the body (renamed here, see Changes), streamed overflow without a header, a 1 MiB chunk refused before it is copied, the exact stream boundary, seven ambiguous Content-Length values, audit offload and saturation, and an OSError audit failure for a 400 and a declared 413.
• test_runtime_control_plane_api.py: declared overflow (:1601), invalid Content-Length (:1621), chunked overflow (:1648), and no reads after the crossing chunk (:1804).
• test_issue_1186_control_plane_recovery_operations.py: the audit-failure log omits the error and the request path (:946), and an oversized method is still audited (:975).
• test_issue_1359_runtime_api_trust_boundary.py: a 413 carries no-store (:593), and max_request_bytes must be an int (:837).

The new module adds what those lack: misleading Content-Length values, empty and one-byte frames at the middleware and at the composed adapter, disconnects, seven route kinds with and without a bearer token, intact delivery to an endpoint, and three audit-failure error types. A few matrix cells repeat the shape of an old case, such as one over-limit chunk with no header, so that the grid has no holes. The old tests are kept on purpose. Most check something the new module does not, such as the raw_body state, the copy guard, the parser variants, a failing audit store, a run through TestClient, read counts at the composed adapter, or the request-path redaction. Two are close overlaps: test_request_size_middleware_rejects_stream_that_exceeds_limit_without_header (the over-limit-absent cell) and test_request_size_middleware_does_not_retain_empty_transport_messages (100 empty frames, where the matrix uses up to 3). The matrix repeats their shapes with more assertions. They stay, because removing #1093's tests is outside #1091's scope.

#1444 (the missing profile nonclaims) is stacked on this branch at 11854670, so it carries this branch's first three commits. This branch's fourth commit, 03709bfa, changes only the new test module, which #1444 does not touch. #1443 (the API-404 clause-to-profile map) is based on dev and does not depend on this branch.

Requirement UIDs

• API-404

Related Issues

Closes #1091

ADR Impact

• ADR-104: the change stays inside API-404-C3's bounded P2 admission. No ADR text changes.

Changes

• raes_runtime/control_plane_api_guards.py: RequestSizeLimitMiddleware reads the body through _receive_bounded_body(), which returns the complete body, OVERSIZED, or ABANDONED. An http.disconnect before the final http.request frame returns ABANDONED, so the middleware sets no raw_body, calls no app and sends no response. Before, it replayed http.disconnect to the app with the partial bytes in raw_body. The 400/413 paths, the single replayed message and the delegation of later receive() calls are unchanged.
• tests/test_issue_1091_request_limit_boundary.py (new, 138 cases, ASGI messages only, no framework-private body state). The module docstring names the existing size-guard modules. No two cases of one test share an input, and every case that expects a response requires exactly one http.response.start:
  • Middleware matrix (33 cases): 8 bodies (empty, under-limit, exact-limit, over-limit, 16 and 17 one-byte frames, exact and over-limit bodies split by empty frames) × 5 Content-Length variants: absent, truthful, and the fixed values 0, 16 (the limit) and 17. The ids name a fixed value by what it declares (declares-zero, declares-limit, declares-past-limit), because one value can understate one body and overstate another. The 7 pairs where a fixed value equals the true length are left out, because they repeat the truthful case. An admitted body is replayed once with its exact bytes, and later reads reach the server. A refused one gets the stable 413 after exactly 0 reads for a declared overflow, or at the crossing frame otherwise. Refusal never calls the app and records one http-request-rejected audit.
  • Disconnects: 4 abandonment points × 3 declared lengths (absent, declares-zero, declares-limit) at the middleware, and 7 routes × 2 points on the composed adapter. No endpoint runs, nothing is sent, and nothing is audited.
  • Composed-adapter overflow: an over-limit declaration, a streamed overflow with no length, and an understated length. Each runs on 7 routes (the API description GET /openapi.json, public probe, administrative read, body and bodyless mutation, operator resolution, unrouted path), with and without a bearer token. Each gets one 413 {"detail":"request too large"} with Cache-Control: no-store and one rejection audit, and no endpoint runs.
  • Accepted delivery (27 cases): empty, under-limit and exact-limit cancellation bodies × 3 chunkings × 4 Content-Length variants (absent, truthful, declares-zero, declares-limit), less the 9 that repeat another case (one-byte frames of the empty body, and a fixed value equal to the true length). Each reaches cancel_workflow with the exact reason.
  • Audit failure: OSError, RuntimeError and sqlite3.OperationalError, each carrying a sentinel, for the invalid-length, declared-overflow and streamed-overflow rejections. The status, body and headers stay stable and no endpoint runs. The sentinel appears in neither the response nor the captured logs. The guard's logger records only the fixed failure message, and no record carries a traceback.
• tests/test_issue_1093_request_rejection_offload.py: test_request_size_middleware_accepts_disconnect_before_body pinned the old dispatch. It is renamed ..._abandons_disconnect_before_body and now asserts that no route runs and no raw_body is set.
• docs/public/guides/control-plane.md: new "Bound request bodies" section. The check covers every method and path, including probes, API description routes and unrouted paths. It documents ControlPlaneSecurityConfig.max_request_bytes (default 1,000,000), declared-length and counted-byte refusal, whole-body buffering with no streaming to routes, rejection bodies, no-store and audit-failure behavior, and server and proxy duties. On disconnects, it says a route runs only once the adapter has read the last body chunk. It also says the server can report a disconnect even after the client has sent the whole body, so each mutation should carry an Idempotency-Key that is reused on retry. The locked uvicorn 0.42.0 does this: receive() returns http.disconnect once the client has gone, even with body bytes buffered (uvicorn/protocols/http/h11_impl.py:541-542, httptools_impl.py:573-574).
• docs/requirements/API-404/requirement.md: IMPLEMENTS, DOCUMENTS and TESTS lines for the guard, the guide and the new module, in the existing Bind bearer authorization to target scope: delivered #1090/Complete HTTP request-limit boundary tests and documentation #1091/fix(runtime): offload backend execution from the event loop #1093 block.
• Specification-coverage release 69.0.0 (execution-snapshot-v69.json, analysis-v69.json, issue-1091 bundle) replays the retained matrix against this branch's source.
• Formal-validation release 70.0.0 (execution-snapshot-v70.json, analysis-v70.json, retest-v70.json) replays the retained formal cases with baseline 69.0.0 and no deviation.
• Revision pins advanced in tools/check_specification_coverage.py, tools/formal_semantic_validation/, and the three evidence test modules. Both research indexes record the new releases.

Test Plan

• Unit tests pass
• Integration tests pass if applicable
• Full completion suite required in CI before merge
• No coverage regression

All results below are for head 03709bfa.

pytest test_issue_1091_request_limit_boundary.py test_issue_1093_request_rejection_offload.py passed (159 passed). With -m integration both modules select nothing (159 deselected, exit 5), because neither has integration cases.

With only control_plane_api_guards.py restored to origin/dev, 26 of the 138 new cases fail: the 12 middleware and 14 composed-adapter disconnect cases. The renamed #1093 test fails as well. The composed-adapter failures show get_snapshot, reconcile_workflow_timeouts and control_plane_readiness calls, 400 There was an error parsing the body responses on the body routes, a 404 on the unrouted path, and a 200 carrying the API description on GET /openapi.json.

Collected ids: 146 at 3bd15898, 130 at 11854670 (16 repeats dropped, none added or renamed), and 138 at 03709bfa. 03709bfa renames 36 ids, adds 8 and drops none. In the two matrices, understated, overstated-to-limit and overstated-past-limit become declares-zero, declares-limit and declares-past-limit (17 middleware ids, 11 delivery ids). The middleware disconnect ids understated and at-limit become declares-zero and declares-limit (8 ids). The 8 new ids are the API description route's 6 overflow and 2 disconnect cases. A script over the module's own parameters finds no two cases with the same frames and declared length in either matrix, and no two composed routes with the same method and path.

Two temporary guard mutations, reverted afterwards, show that the tighter assertions catch defects that the module at 3bd15898 missed. The first adds await send({"type": "http.response.start", "status": 413, "headers": []}) for status 413 just before the JSONResponse in _reject (control_plane_api_guards.py:245), so every 413 carries a second response start. It fails 65 cases: 17 middleware refusals, 42 composed-adapter refusals, and the 6 audit-failure cases that expect a 413. The module at 11854670 fails 59, because it had no API description cases. The module at 3bd15898 passes all 146 of its cases. The second mutation repeats the _LOGGER.error call in _reject's audit-failure branch. It fails the 9 audit-failure cases, and the module at 3bd15898 passes all 146.

The other suites that build the size guard or the HTTP adapter (test_issue_1186_control_plane_recovery_operations.py, test_issue_1359_runtime_api_trust_boundary.py, test_runtime_control_plane_api.py) passed (302 passed). Their -m integration run selects nothing (302 deselected). The API-404 drift tests (test_issue_1185_api_404_profile_alignment.py, test_issue_1189_control_plane_profile_declarations.py) passed (20 passed), including the check that every local traceability artifact exists.

Coverage of raes_runtime.control_plane_api_guards from the two boundary modules reaches every line of the changed _handle_http, _receive_bounded_body and _replay paths. The file's remaining misses are the seal and no-store middleware branches and an unreachable int() fallback. All of these predate this change.

Evidence: tools/check_specification_coverage.py and tools/check_formal_semantic_validation.py both pass. Neither 11854670 nor 03709bfa changes an input the captures bind (no *.py under implementations/python/packages, pyproject.toml or uv.lock), so releases 69.0.0 and 70.0.0 still hold. The three evidence modules pass in the default lane (209 passed, 19 deselected) and with -m integration (19 passed, 209 deselected).

nox -s verify-fast-feedback -- --base-rev origin/dev passed: hygiene, policy, lint and the 5 changed pytest modules (368 passed, 19 deselected). nox -s lint passed. make policy passed; requirement governance is skipped there because the branch name and scope files select no UID, as on the earlier API-404 issue branches. Forcing it with tools/check_requirement_governance.py --base-rev origin/dev --requirement-uid API-404 also exits 0. nox -s docs-local passed (Vale 0 errors in 22 files, Sphinx -W HTML, public output inventory).

Ground Control Checks

• Repository policy command passes
• Regression tests fail on the pre-fix code and pass after
• Pre-push code review and test-quality review: not run for this lane

Traceability

• IMPLEMENTS: API-404 ← implementations/python/packages/raes_runtime/control_plane_api_guards.py
• TESTS: API-404 ← implementations/python/tests/test_issue_1091_request_limit_boundary.py, API-404 ← implementations/python/tests/test_issue_1093_request_rejection_offload.py, API-404 ← implementations/python/tests/test_specification_coverage.py, API-404 ← implementations/python/tests/test_formal_semantic_validation.py, API-404 ← implementations/python/tests/test_issue_989_versioned_evidence.py
• DOCUMENTS: API-404 ← docs/public/guides/control-plane.md

Checklist

• Code follows the project coding standards
• FM1: a fail-closed transport admission rule. Invariants: only a complete body within the limit is dispatched; a frame that would cross the limit is refused before it is buffered; a disconnect before the final frame dispatches and sends nothing. Table-driven tests pin each one. No contract, schema, profile or capability declaration changes, and no executable or runtime adoption is claimed beyond this adapter admission change.
• No published contract schema or model changed, so nothing is regenerated
• PR title is a Conventional Commit; fix(runtime) because abandoned requests no longer reach routes
• Architectural docs updated if applicable: the public guide is updated; ADR-104 is unchanged

Documentation

Updated: docs/public/guides/control-plane.md ("Bound request bodies").