Repository navigation
Conversation
| } | ||
| }) | ||
| .await?; | ||
| crate::check_size(size as u64, artifact_type)?; |
There was a problem hiding this comment.
Could we enforce the cap while reading the S3 response? An object can change between HEAD and GET.
There was a problem hiding this comment.
Fixed in 40068e8. The small-object path now reads with a Range that ends at the size that passed the check, so S3 returns no more than that when the object changes after the HEAD. The chunked path already read by ranges inside that size. An empty object now returns early, because S3 rejects a range on it.
| .query_async(&mut conn) | ||
| .await | ||
| .map_err(transient)?; | ||
| let stored_bytes = (total_chunks - 1) * *CHUNK_SIZE + last_chunk_bytes; |
There was a problem hiding this comment.
Could we use the actual stored chunk sizes here? If the uploader uses 1 MiB chunks but the reader uses the default 32 MiB, a 17 MiB file gets counted as 513 MiB and rejected even though it’s under the limit.
There was a problem hiding this comment.
Fixed in be828d9. The pre-check now sums HSTRLEN over all chunks in one pipeline, so it no longer uses the chunk size of the reader. A Redis-backed test stores 17 one-byte chunks for a program and expects the download to pass. The old check counted them as 512 MiB + 1 byte and refused the program.
|
Should we also cap the fulfiller’s program download at 512 MiB? It looks like the fulfiller loads the entire compressed program into memory when it copies it to cluster storage before the worker checks the size. |
|
@0xernesto Yes. I checked the path: The download function is in the
I kept it out of this PR so that this one does not wait for the |
| .query_async(&mut conn) | ||
| .await | ||
| .map_err(transient)?; | ||
| crate::check_size(chunk_bytes.iter().sum(), artifact_type) |
There was a problem hiding this comment.
This check covers chunked programs. The inline GET returns earlier, so it can fetch an oversized program if the writer’s chunk threshold and Redis bulk limit are raised. Could we enforce the same cap before fetching inline program bytes?
There was a problem hiding this comment.
Fixed in d8b50cd. The inline path now reads STRLEN and checks the cap before the GET. A Redis-backed test raises proto-max-bulk-len, stores an inline program of 512 MiB + 1 byte and expects TooLarge. Before the change the download returned all of it.
The check costs one more round trip for each inline download, for every artifact type. I can limit it to the capped types if that cost matters.
Summary
ProofRequestErrorcode in the failure payload:PROGRAM_TOO_LARGEfor the cap,INVALID_PROGRAMfor a zstd frame or bincode payload that does not decode. The fulfiller passes that code to the network.mainat 749b0c6, which adds the two codes to the SDK's proto types, andzstdmoves to 0.14 with it. The pinnedsuccinctlabs/networkcrates move to fcb2b52 for the same codes on the fulfiller side.Motivation
The fulfiller and the gateway store program bytes from customers as they arrive. A 3 MB zstd bomb can unpack to 100 GB on a worker and kill the node. The coordinator treats a dead worker as an infrastructure event and requeues its tasks without a retry count, so one bomb can take node after node until the request deadline. With the cap, the worker returns a fatal task error and the proof fails on the first attempt.
The 512 MiB limit matches the proxy cap in network-services#1288.
Validation
STRLENpre-check rejects it before theGET.cargo test: 231 passed, 0 failed, 19 ignored (the ignored tests need Redis).cargo fmt --checkandcargo deny checkpass. The advisories job passes again after the merge ofmain.Rangeheader.Caveats
STRLENround trip to each inline Redis download, for every artifact type.b38b612#185.