Skip to content

feat(artifacts): cap the size of a download from a URI - #251

Merged
bergkvist merged 5 commits into
mainfrom
tobias/cap-artifact-download-size
Nov 20, 2026
Merged

bergkvist merged 5 commits into
mainfrom
tobias/cap-artifact-download-size

Conversation

@bergkvist

@bergkvist bergkvist commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Artifact::download_raw_from_uri and Artifact::download_raw_from_uri_par take a new max_bytes argument. A larger artifact fails with the new ArtifactTooLarge error before the body is read.
  • The S3 path checks the size from the HEAD request and from the GET response. The HTTPS path checks Content-Length, and it counts the bytes while it reads when the server sends no length.
  • The parallel HTTPS path checks the size from HEAD before it sends a GET. Each range response must have status 206 and exactly the requested number of bytes, and the read stops at that number.
  • The parallel S3 path sends the ETag from HEAD as If-Match on each range GET. The download fails if the object changes after HEAD.
  • The other download functions pass u64::MAX, so they have no limit, as before.

Motivation

The cluster fulfiller and executor use these two functions to copy the program and the stdin of a request into cluster storage. They read the full artifact into memory. A requester controls the size of both artifacts, so one very large upload can use all the memory of the service. The service then restarts and reads the same artifact again.

With this argument, the fulfiller can refuse a program above 512 MiB (succinctlabs/sp1-cluster#217) before it reads the body.

Validation

  • A unit test serves a 10-byte body from a local socket, with and without Content-Length. A limit of 10 bytes passes and a limit of 9 bytes gives ArtifactTooLarge.
  • Four tests run the parallel paths against a local socket:
    • An oversized size from HEAD gives ArtifactTooLarge, and no GET is sent.
    • A server that honors ranges gives the full object.
    • A server that returns the full object for a range, with status 200 or 206, gives an error. Before the fix, this case panicked.
    • An S3 object that changes after HEAD gives an error. Before the fix, the download returned the first bytes of the new object.
  • No test covers the S3 size checks.

Caveats

  • Both functions have a new argument, so each caller must change when it updates this crate. This repository has no caller. sp1-cluster calls download_raw_from_uri_par, and sp1-cluster-private calls download_raw_from_uri in three places.
  • Only the parallel S3 path fails when the object changes during the download. The parallel HTTPS path sends no If-Match, because that header never matches a weak ETag.

@bergkvist bergkvist changed the title feat(artifacts): cap the size of a parallel download feat(artifacts): cap the size of a download from a URI Nov 17, 2026
Comment thread crates/network/artifacts/src/lib.rs Resolved
Comment thread crates/network/artifacts/src/lib.rs Resolved
Comment thread crates/network/artifacts/src/lib.rs Resolved

@fakedev9999 fakedev9999 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@bergkvist
bergkvist merged commit b28438b into main Nov 20, 2026
5 checks passed
@bergkvist
bergkvist deleted the tobias/cap-artifact-download-size branch November 20, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants