Gap cloud-27 · behavior-difference · confidence: verified (the no-retry-on-truncated-range part and the backoff); suspected (raw HttpIOException escaping) · tracked in #242
Baseline: Midge 7d39f86 (0.1.1) vs Pants 2507c66. Midge paths are relative to the midge repo, Pants paths to this repo. Midge behavior is used as the proxy for the not-yet-extracted lsm-spec behavioral contract.
Prior: none. #62 (related: transport indeterminacy) and #127 (related: flush backoff) are both closed.
Midge: src/storage/cloud/executor.rs:12-13, 441-521. There are 3 retries (4 attempts) with exponential backoff of
50ms × 2^n capped at 2^4. Transient statuses are exactly 408/425/429/500/502/503/504. Transport, body, and decode errors on reads count as
transient and are retried, including a partial body disconnect (should_retry_read_given_partial_response_body_disconnect, :1151).
Each attempt's timeout is clamped to the remaining deadline (:488-501).
Pants: S3ObjectStore.cs:325-369, AzureBlobObjectStore.cs:~340-396, GcsObjectStore.cs:~230-284. There are 3 attempts
total (MaximumAttempts = 3) with no backoff: only await Task.Yield() between attempts (S3:367, Azure:394, GCS:282).
IsRetryable treats 408, 429, and any status >= 500 as retryable (S3:617, Azure:398, GCS:858), so 425 is not retried while 501 and 505 are.
Only HttpRequestException is retried. For ranged GETs the body is read in CloudHttpResponseReader.BufferRangeAsync
(Protocol/CloudHttpResponseReader.cs:94-110), which converts only EndOfStreamException to PantsIOException. That exception
is not caught by the retry loop, so a truncated or disconnected range body fails the read on the first attempt
(CloudProviderRangeContractTests.ShouldRejectTruncatedRangeBodiesAndDisposeThem asserts failure with no retry). A mid-body
socket reset throws HttpIOException (an IOException, not EndOfStream), which likely escapes unwrapped and unretried (suspected).
Why it matters: cold SST point reads and scans issue many small ranged GETs (disk-resident reads). A single transient body reset
fails the user read, and with 429 throttling the three immediate retries are spent in microseconds, which makes the throttling worse.
Acceptance criteria:
- Reads retry transient statuses (408/425/429/500/502/503/504) with bounded exponential backoff inside the operation deadline. Each backoff sleep is also bounded by that deadline.
- Decide whether other 5xx statuses (501/505/…) are permanent and document the decision.
- A read whose headers are valid but whose body is truncated or reset (full or ranged) is retried, then succeeds on a clean second response. The test asserts two requests were made and the correct bytes returned.
- An
HttpIOException from body reads never escapes unwrapped. It surfaces as PantsIOException or PantsTimeoutException.
- Mutations remain non-retried (the existing
ShouldNotRetryConditional* tests stay green).
Prior: none. #62 (related: transport indeterminacy) and #127 (related: flush backoff) are both closed.
Midge:
src/storage/cloud/executor.rs:12-13, 441-521. There are 3 retries (4 attempts) with exponential backoff of50ms × 2^n capped at 2^4. Transient statuses are exactly 408/425/429/500/502/503/504. Transport, body, and decode errors on reads count as
transient and are retried, including a partial body disconnect (
should_retry_read_given_partial_response_body_disconnect, :1151).Each attempt's timeout is clamped to the remaining deadline (:488-501).
Pants:
S3ObjectStore.cs:325-369,AzureBlobObjectStore.cs:~340-396,GcsObjectStore.cs:~230-284. There are 3 attemptstotal (
MaximumAttempts = 3) with no backoff: onlyawait Task.Yield()between attempts (S3:367, Azure:394, GCS:282).IsRetryabletreats 408, 429, and any status >= 500 as retryable (S3:617, Azure:398, GCS:858), so 425 is not retried while 501 and 505 are.Only
HttpRequestExceptionis retried. For ranged GETs the body is read inCloudHttpResponseReader.BufferRangeAsync(
Protocol/CloudHttpResponseReader.cs:94-110), which converts onlyEndOfStreamExceptionto PantsIOException. That exceptionis not caught by the retry loop, so a truncated or disconnected range body fails the read on the first attempt
(
CloudProviderRangeContractTests.ShouldRejectTruncatedRangeBodiesAndDisposeThemasserts failure with no retry). A mid-bodysocket reset throws
HttpIOException(an IOException, not EndOfStream), which likely escapes unwrapped and unretried (suspected).Why it matters: cold SST point reads and scans issue many small ranged GETs (disk-resident reads). A single transient body reset
fails the user read, and with 429 throttling the three immediate retries are spent in microseconds, which makes the throttling worse.
Acceptance criteria:
HttpIOExceptionfrom body reads never escapes unwrapped. It surfaces as PantsIOException or PantsTimeoutException.ShouldNotRetryConditional*tests stay green).