fix(ship): adapt batches after HTTP 413

[ronheichman]

Why

numbat ship targets 4 MiB batches, but complete-record reads can exceed that target by one record.
Receivers may reject those requests with HTTP 413.
Release v0.2.0 treats every non-2xx response as a terminal batch failure.
The durable checkpoint then remains behind the entire rejected batch.

Deployment

Deployment status is INFERRED.
A downstream image configuration pins public release v0.2.0.
No deployed binary was inspected.

What changed

• The HTTP sink exposes response status through wrapped errors without changing diagnostic text.
• ship bisects only rejected 413 batches at complete NDJSON boundaries.
• Delivery remains ordered, and each successful prefix is checkpointed before its suffix begins.
• A 413 on one record leaves that record durable and reports its offset and size.
• Non-413 responses and ambiguous transport errors retain the previous at-least-once behavior.

Direct HTTP output outside ship keeps its existing buffering behavior.
No wire, schema, configuration, or successful-delivery behavior changes.

Public references include the v0.2.0 shipping path.
The new adaptive implementation handles 413 responses.
The synthetic tests cover ordering and durability.

Actual red and green

All behavior evidence below uses local synthetic HTTP servers and temporary NDJSON files.

RED before implementation:

$ go test ./cmd/numbat -run '^TestShipSplitsRejectedBatchAtRecordBoundaries$' -count=1 -v
=== RUN   TestShipSplitsRejectedBatchAtRecordBoundaries
    ship_test.go:202: drain: deliver: http sink: server returned 413 Request Entity Too Large
--- FAIL: TestShipSplitsRejectedBatchAtRecordBoundaries (0.08s)
FAIL

GREEN after implementation:

$ go test ./cmd/numbat -run '^(TestShipSplitsRejectedBatchAtRecordBoundaries|TestShipPersistsAcceptedPrefixBeforeFailedSuffix|TestShipLeavesSingleRejectedRecordUnacknowledged|TestShipAmbiguousTransportFailureKeepsCheckpoint|TestShipRejectedFourMiBTargetOvershootKeepsCheckpoint|TestShipZeroLossAcrossOutage)$' -count=1 -v
--- PASS: TestShipZeroLossAcrossOutage (0.01s)
--- PASS: TestShipSplitsRejectedBatchAtRecordBoundaries (0.01s)
--- PASS: TestShipPersistsAcceptedPrefixBeforeFailedSuffix (0.01s)
--- PASS: TestShipLeavesSingleRejectedRecordUnacknowledged (0.00s)
--- PASS: TestShipAmbiguousTransportFailureKeepsCheckpoint (0.00s)
--- PASS: TestShipRejectedFourMiBTargetOvershootKeepsCheckpoint (0.04s)
PASS

Verification

go mod tidy
git diff --exit-code -- go.mod go.sum
go generate ./rules
git diff --exit-code -- rules/internal/checked
test -z "$(git ls-files --others --exclude-standard -- rules/internal/checked)"
test -z "$(gofmt -l .)"
git diff --check
go vet ./...
go test -race ./...
go build -buildvcs=false ./cmd/numbat
go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run
go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 fmt --diff
go run golang.org/x/vuln/cmd/govulncheck@v1.6.0 ./...
go test -coverprofile=/tmp/numbat-coverage.out ./...
GOOS=windows GOARCH=amd64 go test -exec=true ./...
GOOS=darwin GOARCH=amd64 go test -exec=true ./...

Every fuzz target listed in .github/workflows/ci.yml passed for 30 seconds.
Coverage was 84.6%.

GitHub reports no hosted check runs or status contexts for the final commit.
Native Windows tests and native macOS race tests remain unverified.
The Windows and macOS commands above use -exec=true for cross-compilation only.
They do not execute tests on those platforms.

Built with Codex