Skip to content

fix: reject truncated and non-success API responses - #60

Merged
steipete merged 1 commit into
mainfrom
fix/http-response-boundaries
Sep 15, 2026
Merged

steipete merged 1 commit into
mainfrom
fix/http-response-boundaries

Conversation

@steipete

Copy link
Copy Markdown
Collaborator

The shared HTTP client silently discarded bytes after its 1 MiB read limit. A valid JSON prefix followed by whitespace and additional data could therefore produce a successful result from a malformed response. It also decoded final HTTP 300/302 bodies as successful place results and reported HTTP 304 as an empty response, losing the upstream status.

Read one extra byte to detect overflow, reject oversized successful responses, and decode results only for final 2xx statuses. Non-2xx responses retain their status in APIError; oversized error bodies use a concise size diagnostic. This applies to every Places and Routes workflow through the shared request path. The existing payload limit and redirect policy remain in place. Documentation and the Unreleased changelog describe the corrected behavior.

Validation:

  • Regression tests failed before the fix for oversized bodies, trailing data past the limit, HTTP 300/302/304, and a custom redirect policy returning http.ErrUseLastResponse; all pass afterward.
  • Built CLI against a local synthetic HTTP server: oversized/trailing-data/302 cases changed from exit 0 with a place result to exit 1 without output. A valid body exactly 1 MiB still succeeds; 304 and oversized 502 retain their status.
  • make lint test coverage passed (93.1% total coverage); go test -race ./..., go vet ./..., and go build ./... passed.
  • Documentation metadata tests and regeneration passed without drift; pinned gosec and govulncheck found no issues.
  • Independent Codex review passed with no actionable P0–P2 findings. No authenticated Google API calls were needed; proof uses synthetic fixtures.

@clawsweeper

clawsweeper Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 15, 2026
@clawsweeper

clawsweeper Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge. Reviewed September 15, 2026, 3:15 AM ET / 07:15 UTC.

ClawSweeper review

What this changes

The shared Places and Routes HTTP client rejects oversized responses and final non-2xx statuses, with regression tests and documentation of the resulting errors.

Merge readiness

⛔ Blocked before merge - 1 item remains

This remains a useful, focused fix: current main and v0.4.11 retain both defects, and no replacement implementation was identified. No actionable patch defect was found.

Priority: P2
Reviewed head: ca5895d233b3d2fcfec1a92c01fae5c652ca4693

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused repair with appropriate regression coverage and reported CLI validation; no blocking correctness finding was identified.
Proof confidence 🌊 off-meta tidepool Not applicable: The author is a collaborator, so the external-contributor proof gate does not apply; no authority boundary changes. The body reports relevant built-CLI before/after results through a local HTTP server exercising the shared response reader, but no raw transcript was independently inspected.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The author is a collaborator, so the external-contributor proof gate does not apply; no authority boundary changes. The body reports relevant built-CLI before/after results through a local HTTP server exercising the shared response reader, but no raw transcript was independently inspected.
Evidence reviewed 7 items Current main still contains the defect: The shared request path reads only 1 MiB without detecting overflow and treats only statuses of 400 or higher as API errors. GitHub's main branch endpoint still resolves to the inspected base.
Latest release also retains the defect: The v0.4.11 source contains the same truncating read and status threshold; the requested behavior is not already shipped in that release.
Bounded shared-path repair: The introduced code reads one extra byte, rejects oversized successes, and preserves non-2xx status codes while replacing oversized error bodies with a fixed diagnostic. Existing redirect restrictions and key redaction remain intact.
Findings None None.
Security None None.

How this fits together

The shared HTTP client receives API responses for the Go library and CLI, then passes successful payloads to workflow-specific JSON decoders. Its errors prevent result output and produce CLI exit code 1.

flowchart TD
  A[Library or CLI request] --> B[Shared HTTP client]
  B --> C[API response]
  C --> D[Check size and HTTP status]
  D -->|Valid response| E[Decode workflow result]
  D -->|Rejected response| F[Return diagnostic error]
  E --> G[Library result or CLI output]
  F --> H[Library error or CLI exit 1]
Loading

Before merge

  • Resolve merge risk (P1) - Custom endpoints that previously returned usable JSON under final 3xx statuses, or valid JSON followed by padding beyond 1 MiB, will now fail instead of producing results. This intentional compatibility tightening is documented and exercised in the reported before/after CLI validation.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test delta Production +17/-4; tests +95/-1; documentation +3 The production growth is confined to shared response validation and is supported by focused boundary coverage.

Merge-risk options

Maintainer options:

  1. Accept the documented response tightening (recommended)
    Retain the collaborator-proposed rejection of oversized and final non-success responses, accepting that custom endpoints relying on their previous accidental success will need correction.

Technical review

Best possible solution:

Keep response validation centralized, preserve valid responses through exactly 1 MiB, and retain the documented upgrade behavior and before/after CLI coverage.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: serve valid JSON padded to the old 1 MiB boundary followed by more data, or return JSON with final HTTP 302. Current main can decode these as success; this read-only review did not execute the scenarios.

Is this the best way to solve the issue?

Yes. Reading one extra byte and validating the final status in the existing shared request path is a narrow repair that avoids duplicating checks across workflows.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against f408ca16e16a.

Labels

Label changes:

  • add P2: This repairs incorrect handling of oversized or non-success upstream responses without evidence of an urgent widespread outage.
  • add merge-risk: 🚨 compatibility: Custom endpoints relying on successful decoding of final 3xx or oversized JSON-prefix responses will now return errors.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a collaborator, so the external-contributor proof gate does not apply; no authority boundary changes. The body reports relevant built-CLI before/after results through a local HTTP server exercising the shared response reader, but no raw transcript was independently inspected.

Label justifications:

  • P2: This repairs incorrect handling of oversized or non-success upstream responses without evidence of an urgent widespread outage.
  • merge-risk: 🚨 compatibility: Custom endpoints relying on successful decoding of final 3xx or oversized JSON-prefix responses will now return errors.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a collaborator, so the external-contributor proof gate does not apply; no authority boundary changes. The body reports relevant built-CLI before/after results through a local HTTP server exercising the shared response reader, but no raw transcript was independently inspected.

Evidence

What I checked:

  • Current main still contains the defect: The shared request path reads only 1 MiB without detecting overflow and treats only statuses of 400 or higher as API errors. GitHub's main branch endpoint still resolves to the inspected base. (internal/places/client.go:114, f408ca16e16a)
  • Latest release also retains the defect: The v0.4.11 source contains the same truncating read and status threshold; the requested behavior is not already shipped in that release. (internal/places/client.go:114, c0cdace7197b)
  • Bounded shared-path repair: The introduced code reads one extra byte, rejects oversized successes, and preserves non-2xx status codes while replacing oversized error bodies with a fixed diagnostic. Existing redirect restrictions and key redaction remain intact. (internal/places/client.go:119, ca5895d233b3)
  • Boundary and error regression coverage: Read the complete tests covering below/exactly/above 1 MiB, unknown content length, trailing data beyond the limit, final 300/302/304 statuses, oversized 502 bodies, and custom redirect cancellation. Tests were inspected, not executed. (internal/places/client_response_test.go:12, ca5895d233b3)
  • Reported production-path validation: The supplied complete PR body reports a built CLI against a local HTTP server: oversized, trailing-data, and 302 cases changed from successful result output to exit 1 without output; exactly 1 MiB remained successful, and 304/oversized 502 retained their statuses. This is an author-reported result, not an independently observed transcript. (ca5895d233b3)
  • Prior shared-client ownership: Recent path history identifies Peter Steinberger on the shared-client refactor and redirect/error hardening. GitHub commit metadata verifies steipete as the author of the latter change. Older local follow-history and blame traversal encountered unavailable objects; no original-feature introduction claim is made. (internal/places/client.go:110, 84fff29e4fd4)

Likely related people:

  • unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete

Copy link
Copy Markdown
Collaborator Author

Reviewed the compatibility note and accepted the documented response validation fix. Decoding final non-2xx statuses as successful Places/Routes results, or silently discarding bytes beyond the existing size cap, was accidental behavior. The shared client now reports these failures consistently, while valid responses through exactly 1 MiB and followed same-origin redirects retain their behavior. Custom endpoints relying on the accidental success behavior should return a valid 2xx response within the existing limit.

The before/after built CLI fixtures independently reproduced both defects, the new regression tests pass, and independent review found no actionable P0–P2 defect. CI run https://github.com/openclaw/goplaces/actions/runs/34940613504 passed on ca5895d233b3d2fcfec1a92c01fae5c652ca4693, including Linux, Windows, macOS audit, six-target reproducibility, and release contracts; CodeQL is also green. The bounded route-input fuzz pass completed 959,671 executions without a failure.

@steipete
steipete merged commit 415ff61 into main Sep 15, 2026
12 checks passed
@steipete
steipete deleted the fix/http-response-boundaries branch September 15, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant