Skip to content

cloud: charge provider upload/readback payloads to the resource budget and avoid per-attempt payload copies #344

Description

@smiggleworth

Gap cloud-30 · missing-capability · confidence: suspected (may overlap with the admission/storage domain) · tracked in #242
Baseline: Midge 7d39f86 (0.1.1) vs Pants 2507c66. Midge paths are relative to the midge repo, Pants paths to this repo. Midge behavior is used as the proxy for the not-yet-extracted lsm-spec behavioral contract.

Prior: none

Midge: upload bodies carry a ResourceReservation that stays charged until the transport releases its body slices, and one admitted buffer is
shared across retries without being copied (executor.rs:55-70; tests should_keep_upload_memory_charged_until_transport_body_slices_are_released
:673, should_share_admitted_payload_across_retries_without_copying_or_releasing_it :694;
cloud/retained_tests.rs:36 should_keep_upload_charged_when_callback_adapter_times_out_before_backend_completion).
Range reads also take reservations (admitted.rs read_range_admitted) and a response limit equal to the range length (s3.rs:1178-1240).

Pants: S3ObjectStore.cs:382, GcsObjectStore.cs:304, and AzureBlobObjectStore.cs:252 copy the payload per request
(new ByteArrayContent(data.ToArray())). SST/WAL publication reads whole files (ProviderCloudPersistence.cs:625, 652;
ProviderCloudCompactionPublisher.cs:47, 88) and then fetches the full object back with GetAsync for verification (:118, :680). None of these
buffers is charged to Storage/Internal/ResourceBudget.cs. The payload stays referenced by the in-flight request after a deadline
cancellation, with no accounting.

Why it matters: one large compaction output can briefly hold about 3× its size unaccounted (the file, the copy, and the readback). This is
an unbounded-resource risk under memory budgets that Midge enforces.

Acceptance criteria:

  • Upload payloads are reserved against the shared budget before submission and released only after the HTTP content is disposed, including after a deadline or cancellation.
  • No per-attempt copy of the payload is made (use ReadOnlyMemoryContent or a shared content).
  • Readback verification is either bounded (ranged or streamed hash) or charged.
  • A test shows the budget stays charged while the request is in flight after the caller's deadline fires, and is released after the transport completes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:cloudReal cloud provider backendsarea:perfPerformance-critical data structure / algorithm parityarea:storageStorage engine internals (flush, hybrid persistence, I/O, recovery)parity-gapFeature not yet at parity with Midgepriority:P2Normal: bounded hardening, coverage, maintainability, or non-critical behavior gap

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions