Skip to content

fix(fulfiller): cap program download at 512 MiB - #217

Open
bergkvist wants to merge 1 commit into
tobias/cap-program-decode-sizefrom
tobias/cap-fulfiller-download-size
Open

bergkvist wants to merge 1 commit into
tobias/cap-program-decode-sizefrom
tobias/cap-fulfiller-download-size

Conversation

@bergkvist

Copy link
Copy Markdown
Contributor

Summary

  • The fulfiller refuses a program above 512 MiB before it reads the body, when it copies the program into cluster storage.
  • It then fails the request on the network with PROGRAM_TOO_LARGE. The request does not stay assigned until its deadline.
  • The spn-* crates point at the branch of feat(artifacts): cap the size of a download from a URI network#251, which adds the size limit to the download function.

Motivation

This answers a review comment on #214. The fulfiller reads the full program into memory before a worker checks its size. A requester controls that size, so one very large upload can use all the memory of the fulfiller. The fulfiller then restarts and reads the same program again.

The limit is the same max_bytes value that the artifact clients use, so the fulfiller and the workers refuse the same programs.

Validation

  • A unit test covers the mapping from the refused download to PROGRAM_TOO_LARGE.
  • cargo test for the fulfillment, fulfiller and artifact crates passes. Clippy with the CI flags, cargo fmt --check and cargo deny check pass.
  • No test here downloads a program above the limit. feat(artifacts): cap the size of a download from a URI network#251 tests the limit in the download function.

Caveats

Comment thread Cargo.toml
spn-network-types = { git = "https://github.com/succinctlabs/network", branch = "main", features = ["network"] }
spn-pricing = { git = "https://github.com/succinctlabs/network", branch = "main" }
spn-utils = { git = "https://github.com/succinctlabs/network", branch = "main" }
spn-artifacts = { git = "https://github.com/succinctlabs/network", branch = "tobias/cap-artifact-download-size" }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a reminder to get it back to the "main" branch before merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants