Skip to content

Bound decoded-byte growth before reserving a claimed chunk size #2403

Open

Description

@Mavrick-F

Reproduced allocation before rejection

On main at e0cf82a8, zarr::Array::decode_bytes reserves 64 MiB from a chunk-size claim before discovering that the decoded payload is only eight bytes.

The input is a valid 17-byte Zstandard frame containing eight zero bytes. With a uint8 Zarr array/chunk shape of eight, it decodes successfully. With the same frame and a shape of 67,108,864, it is correctly refused (a chunk holds 8 bytes, not 67108864), but only after unzstd reserves want.min(64 << 20).

In separate native probe processes, /proc/self/status VmPeak stayed at 5,972 KiB for the valid eight-byte case and rose to 71,512 KiB for the mismatched metadata. This measures peak virtual address space, not resident memory. The direct reservation is also visible in raster/zarr.rs::unzstd.

Scope I plan to take

Use one internal bounded decoded-byte reader for Zarr's Zstandard chunks (including its blosc Zstandard streams) and GeoTIFF LERC's Zstandard wrapper. Both already use Read decoders and output-byte limits; they currently accumulate their output separately.

The shared helper would:

  • grow from decoded bytes received, rather than reserve an untrusted expected length;
  • check each growth before allocation, use fallible reservation, and keep capacity within the caller's existing byte limit;
  • detect an extra decoded byte and return an error instead of accepting a truncated result;
  • propagate read failures and Cancel, with checks between bounded reads.

No memory ceiling is raised or redefined. Format-specific validation stays where it belongs: dimension products, codec headers, LZ4 expansion bounds and chunk layout checks are not replaced by a generic byte cap. No dependency, serialized field, command parameter or UI change is planned. This stays clear of #2298's time-window work, #2389's spreadsheet work and #2032's coordinate migration.

Tests will cover the small-frame/large-claim regression, valid decoded output, exact and exceeded limits, read failure and cancellation. Existing Zarr/blosc and LERC tests must keep passing, followed by the normal full and wasm gates. This is part of making allocation limits shared checks, rather than adding another independent reader-specific cap.

Activity

  1. self-assigned this
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions