Skip to content

fix: cap program size at 512 MiB and report program errors - #214

Open
bergkvist wants to merge 12 commits into
mainfrom
tobias/cap-program-decode-size
Open

bergkvist wants to merge 12 commits into
mainfrom
tobias/cap-program-decode-size

Conversation

@bergkvist

@bergkvist bergkvist commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • The S3 and Redis artifact clients refuse a program above 512 MiB before download and stop the zstd decode at the same cap. Other artifact types keep no limit.
  • A refused program fails the proof at once, and the worker records the network's ProofRequestError code in the failure payload: PROGRAM_TOO_LARGE for the cap, INVALID_PROGRAM for a zstd frame or bincode payload that does not decode. The fulfiller passes that code to the network.
  • The SP1 family moves to sp1 main at 749b0c6, which adds the two codes to the SDK's proto types, and zstd moves to 0.14 with it. The pinned succinctlabs/network crates move to fcb2b52 for the same codes on the fulfiller side.

Motivation

The fulfiller and the gateway store program bytes from customers as they arrive. A 3 MB zstd bomb can unpack to 100 GB on a worker and kill the node. The coordinator treats a dead worker as an infrastructure event and requeues its tasks without a retry count, so one bomb can take node after node until the request deadline. With the cap, the worker returns a fatal task error and the proof fails on the first attempt.

The 512 MiB limit matches the proxy cap in network-services#1288.

Validation

  • Unit tests cover the decode cap at 512 MiB and one byte over, a decode buffer that never grows past the cap, corrupt zstd input, invalid bincode, the worker's error classification, the fulfiller's pass-through, and old payloads without a code.
  • Redis-backed tests ran against a local Redis 7 container: 12 passed. One stores a program one byte over the cap in chunks, and the chunk pre-check rejects it before any chunk is fetched. One stores 17 one-byte chunks, and the pre-check accepts them. One stores an inline program one byte over the cap, and the STRLEN pre-check rejects it before the GET.
  • Workspace cargo test: 231 passed, 0 failed, 19 ignored (the ignored tests need Redis).
  • Clippy with the CI flags, cargo fmt --check and cargo deny check pass. The advisories job passes again after the merge of main.
  • No test covers the S3 download. The small-object path now reads with a Range header.

Caveats

@bergkvist bergkvist changed the title fix: cap decoded program size at 512 MiB fix: cap program size at 512 MiB and report program errors Nov 5, 2026
Comment thread crates/artifact/src/lib.rs Resolved
Comment thread crates/artifact/src/redis.rs Resolved
Comment thread crates/artifact/src/s3.rs
}
})
.await?;
crate::check_size(size as u64, artifact_type)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we enforce the cap while reading the S3 response? An object can change between HEAD and GET.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 40068e8. The small-object path now reads with a Range that ends at the size that passed the check, so S3 returns no more than that when the object changes after the HEAD. The chunked path already read by ranges inside that size. An empty object now returns early, because S3 rejects a range on it.

Comment thread crates/artifact/src/redis.rs Outdated
.query_async(&mut conn)
.await
.map_err(transient)?;
let stored_bytes = (total_chunks - 1) * *CHUNK_SIZE + last_chunk_bytes;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the actual stored chunk sizes here? If the uploader uses 1 MiB chunks but the reader uses the default 32 MiB, a 17 MiB file gets counted as 513 MiB and rejected even though it’s under the limit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in be828d9. The pre-check now sums HSTRLEN over all chunks in one pipeline, so it no longer uses the chunk size of the reader. A Redis-backed test stores 17 one-byte chunks for a program and expects the download to pass. The old check counted them as 512 MiB + 1 byte and refused the program.

@0xernesto

Copy link
Copy Markdown
Contributor

Should we also cap the fulfiller’s program download at 512 MiB? It looks like the fulfiller loads the entire compressed program into memory when it copies it to cluster storage before the worker checks the size.

@bergkvist

Copy link
Copy Markdown
Contributor Author

@0xernesto Yes. I checked the path: copy_artifact reads the full program into memory through download_raw_from_uri_par from spn-artifacts, which has no size limit. A very large upload can use all the memory of the fulfiller before a worker checks the size.

The download function is in the network repository, so the fix is two PRs:

I kept it out of this PR so that this one does not wait for the network change.

.query_async(&mut conn)
.await
.map_err(transient)?;
crate::check_size(chunk_bytes.iter().sum(), artifact_type)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check covers chunked programs. The inline GET returns earlier, so it can fetch an oversized program if the writer’s chunk threshold and Redis bulk limit are raised. Could we enforce the same cap before fetching inline program bytes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d8b50cd. The inline path now reads STRLEN and checks the cap before the GET. A Redis-backed test raises proto-max-bulk-len, stores an inline program of 512 MiB + 1 byte and expects TooLarge. Before the change the download returned all of it.

The check costs one more round trip for each inline download, for every artifact type. I can limit it to the capped types if that cost matters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants