You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bound decoded-byte growth before reserving a claimed chunk size #2403
On main at e0cf82a8, zarr::Array::decode_bytes reserves 64 MiB from a chunk-size claim before discovering that the decoded payload is only eight bytes.
The input is a valid 17-byte Zstandard frame containing eight zero bytes. With a uint8 Zarr array/chunk shape of eight, it decodes successfully. With the same frame and a shape of 67,108,864, it is correctly refused (a chunk holds 8 bytes, not 67108864), but only after unzstd reserves want.min(64 << 20).
In separate native probe processes, /proc/self/status VmPeak stayed at 5,972 KiB for the valid eight-byte case and rose to 71,512 KiB for the mismatched metadata. This measures peak virtual address space, not resident memory. The direct reservation is also visible in raster/zarr.rs::unzstd.
Scope I plan to take
Use one internal bounded decoded-byte reader for Zarr's Zstandard chunks (including its blosc Zstandard streams) and GeoTIFF LERC's Zstandard wrapper. Both already use Read decoders and output-byte limits; they currently accumulate their output separately.
The shared helper would:
grow from decoded bytes received, rather than reserve an untrusted expected length;
check each growth before allocation, use fallible reservation, and keep capacity within the caller's existing byte limit;
detect an extra decoded byte and return an error instead of accepting a truncated result;
propagate read failures and Cancel, with checks between bounded reads.
No memory ceiling is raised or redefined. Format-specific validation stays where it belongs: dimension products, codec headers, LZ4 expansion bounds and chunk layout checks are not replaced by a generic byte cap. No dependency, serialized field, command parameter or UI change is planned. This stays clear of #2298's time-window work, #2389's spreadsheet work and #2032's coordinate migration.
Tests will cover the small-frame/large-claim regression, valid decoded output, exact and exceeded limits, read failure and cancellation. Existing Zarr/blosc and LERC tests must keep passing, followed by the normal full and wasm gates. This is part of making allocation limits shared checks, rather than adding another independent reader-specific cap.
Activity