diff --git a/.github/RELEASE_ANNOUNCEMENT.md b/.github/RELEASE_ANNOUNCEMENT.md index e1e852c..925b23d 100644 --- a/.github/RELEASE_ANNOUNCEMENT.md +++ b/.github/RELEASE_ANNOUNCEMENT.md @@ -2,7 +2,9 @@ We're pleased to announce **ExZarr v1.2.0**: Zarr 3.1 conformance, standard `sharding_indexed` interoperability with zarr-python, optional byte-range cloud -I/O, AzureSDK migration, and **precompiled Zig codec NIFs** for Hex/Livebook. +I/O, AzureSDK migration, and compression codecs from +[ExCodecs](https://hex.pm/packages/ex_codecs): precompiled pure-Rust NIFs, so +ExZarr has no native code and needs no Zig, compiler or system libraries. ## Highlights @@ -15,16 +17,21 @@ I/O, AzureSDK migration, and **precompiled Zig codec NIFs** for Hex/Livebook. ### Standard sharding + range reads -- Spec-correct `sharding_indexed` v1.0 (dense uint64 index, empty sentinel) +- Spec-correct `sharding_indexed` v1.0 (dense uint64 index, empty sentinel), + readable and writable by zarr-python - Optional `chunk_info` / `read_chunk_range` backend capabilities - Range-aware inner-chunk reads with full-shard fallback - S3, GCS, Azure, filesystem, and memory backends -### Precompiled Zig codec NIFs +### Codecs via ExCodecs -- Hex/Livebook installs download platform NIFs (no Zig toolchain required) -- Source builds via `EX_ZARR_BUILD=1` or when checksums are missing -- See [docs/PRECOMPILATION.md](../docs/PRECOMPILATION.md) +- zstd, lz4, snappy, blosc, bzip2 and crc32c from ExCodecs, precompiled for + macOS, Linux (glibc and musl) and Windows; zlib/gzip via Erlang `:zlib` +- numcodecs-compatible byte formats: zarr-python reads ExZarr's lz4, bzip2 + and blosc data, and ExZarr reads zarr-python's +- New `compressor_config:` option (e.g. `[level: 9]`, or Blosc `cname`, + `shuffle`, `typesize`), stored in `.zarray` / `zarr.json` +- `config :ex_zarr, default_zarr_version: 3` now applies to new arrays ### AzureSDK migration @@ -58,10 +65,16 @@ end ## Upgrade notes +- OTP 26 or later is required +- Drop any Zig / system compression-library setup; it is no longer used - Azure users: add `azure_sdk`, remove `azurex` - Rewrite arrays written with ExZarr’s pre-1.2 private shard index - Cloud v3 arrays use `zarr.json` and `c/...` keys - GCS/Azure HTTP client: `{:req, "~> 0.6.1"}` (CVE-2026-49755) +- New lz4/bzip2 data uses the numcodecs formats; 1.2 reads 1.1 data, but 1.1 + cannot read lz4/bzip2 data written by 1.2 +- `ExZarr.create/1` returns errors for unknown compressors, invalid + `compressor_config` and unsupported Zarr versions ## Known limitations @@ -79,4 +92,4 @@ end ## Full changelog -See [CHANGELOG.md](../CHANGELOG.md#120---2026-10-03). +See [CHANGELOG.md](../CHANGELOG.md#120---2026-10-04). diff --git a/CHANGELOG.md b/CHANGELOG.md index 7939d01..49d3b09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,65 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added -- `compressor_config:` option for `ExZarr.create/1` (e.g. `[level: 9]`; Blosc - also `cname`, `shuffle`, `typesize`), validated per compressor by the new - `ExZarr.Codecs.CompressorConfig`. v2 arrays store the settings in `.zarray` - in numcodecs form and reuse them after reopening (including arrays created - by zarr-python); v3 arrays store them in the codec configuration. The - option was documented before but never implemented. -- `compressor: :gzip` for Zarr v3 arrays. - -### Fixed -- `config :ex_zarr, default_zarr_version` is now applied by `ExZarr.create/1` - and `ExZarr.Nx`. Unset, new arrays remain v2; `ExZarr.Version` previously - documented (and returned) 3 although nothing used it. -- `ExZarr.create/1` rejects unknown compressors, invalid compressor settings - and Zarr versions other than 2 or 3 with an error tuple. Previously an - unknown compressor failed on the first write and an invalid version raised. -- Zarr v3 arrays created with a compressor that has no v3 codec (`:snappy`, - custom codecs without `codecs:`) were silently stored uncompressed; they now - return `{:error, {:unsupported_codec_for_v3, compressor}}`. -- `:zlib` honours a compression `level`. -- v2 `.zarray` compressor levels now record the level ExZarr actually uses - (zlib 6, zstd 3) instead of a fixed 5. - -### Changed -- Compression codecs (zstd, lz4, snappy, blosc, bzip2, crc32c) now come from - [ExCodecs](https://hex.pm/packages/ex_codecs) `~> 0.2.4`: pure-Rust NIFs - with precompiled binaries for macOS, Linux (glibc and musl) and Windows. - ExZarr no longer contains native code; Zig, zigler, zigler_precompiled and - system compression libraries are no longer needed. -- `:lz4` writes the numcodecs `LZ4` format (4-byte little-endian size prefix) - and `:bzip2` writes plain bzip2 streams (numcodecs `BZ2`), so zarr-python can - read them. Data written by earlier ExZarr versions (8-byte size prefix) is - still read. -- v2 `.zarray` compressor metadata uses numcodecs configurations: `lz4` - (`acceleration`), `blosc` (`cname`, `clevel`, `shuffle`, `blocksize`) and - `bz2` (previously `bzip2`). The old `bzip2` id is still read. zarr-python can - now open ExZarr v2 arrays compressed with lz4, blosc and bzip2. -- The v3 `blosc` codec honours `cname`, `clevel`, `shuffle` and `typesize` from - its configuration; without them ExZarr keeps BloscLZ, byte shuffle, - typesize 1. -- Decompressed chunks are capped at 256 MiB (`config :ex_zarr, - max_chunk_bytes: ...`) to guard against decompression bombs. -- Minimum OTP is 26 (ExCodecs NIFs target NIF 2.17). -- `ExZarr.Codecs.compress/3` accepts options as a map as well as a keyword list. - -### Removed -- `ExZarr.Codecs.ZigCodecs`, `ExZarr.Codecs.CompressionConfig`, the - `fix_nif_rpaths` Mix task, the precompile workflow and - `docs/PRECOMPILATION.md` / `docs/COMPRESSION_SETUP.md`. -- `COMPRESSION_LIB_DIRS`, `HOMEBREW_PREFIX`, `BZIP2_STATIC_LIB`, `ZIG_PATH` and - `EX_ZARR_BUILD` no longer have any effect. - -## [1.2.0] - 2026-10-03 +## [1.2.0] - 2026-10-04 ### Zarr 3.1 Interoperability & Range-Aware Cloud I/O ### Added -- Precompiled Zig NIFs for `ExZarr.Codecs.ZigCodecs` via ZiglerPrecompiled - (Linux/macOS x86_64 + aarch64); see `docs/PRECOMPILATION.md` - Optional storage backend callbacks: `chunk_info/2`, `read_chunk_range/4`, conditional `read_chunk_range/5` (`if_match:`), `capabilities/1`, `put_layout/2` - `ExZarr.Storage.ObjectKeys` for centralized v2/v3 object naming, including @@ -82,6 +28,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 matrix (zarr-python 2.18.3 / 3.2.1 / 3.3.0 / 3.4.0) - Azurite-oriented Azure integration test (tagged `:azure`) - Showcase: `examples/range_aware_sharded_nx.exs` (run with `mix run`) +- `compressor_config:` option for `ExZarr.create/1` (e.g. `[level: 9]`; Blosc + also `cname`, `shuffle`, `typesize`), validated per compressor by the new + `ExZarr.Codecs.CompressorConfig`. v2 arrays store the settings in `.zarray` + in numcodecs form and reuse them after reopening (including arrays created + by zarr-python); v3 arrays store them in the codec configuration. The + option was documented before but never implemented. +- `compressor: :gzip` for Zarr v3 arrays. ### Fixed - `sharding_indexed` follows the v1.0 spec geometry (chunk grid = shard shape, @@ -104,6 +57,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 files in the store - Scalar (`shape: {}`) and zero-length dimension support in MetadataV3 / Array - Dimension-name over-validation (spec strings/nil; uniqueness not required at metadata) +- `config :ex_zarr, default_zarr_version` is now applied by `ExZarr.create/1` + and `ExZarr.Nx`. Unset, new arrays remain v2; `ExZarr.Version` previously + documented (and returned) 3 although nothing used it. +- `ExZarr.create/1` rejects unknown compressors, invalid compressor settings + and Zarr versions other than 2 or 3 with an error tuple. Previously an + unknown compressor failed on the first write and an invalid version raised. +- Zarr v3 arrays created with a compressor that has no v3 codec (`:snappy`, + custom codecs without `codecs:`) were silently stored uncompressed; they now + return `{:error, {:unsupported_codec_for_v3, compressor}}`. +- `:zlib` honours a compression `level`. +- v2 `.zarray` compressor levels now record the level ExZarr actually uses + (zlib 6, zstd 3) instead of a fixed 5. ### Changed - `create/1` with `shard_shape:` writes spec metadata: `shard_shape` is the @@ -116,8 +81,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 in backend state - `Storage.chunk_info/2` returns `{:error, :unsupported}` for backends without `chunk_info/2` instead of downloading the object -- Optional HTTP client is now `req ~> 0.6.1` (with `override: true` until - `azure_sdk` widens its `req ~> 0.5` constraint); `ex_aws` updated to 2.7.x +- Optional HTTP client is now `req ~> 0.6.1` (`azure_sdk`'s `req ~> 0.5` already + allows 0.6); `ex_aws` updated to 2.7.x +- Compression codecs (zstd, lz4, snappy, blosc, bzip2, crc32c) now come from + [ExCodecs](https://hex.pm/packages/ex_codecs) `~> 0.2.4`: pure-Rust NIFs + with precompiled binaries for macOS, Linux (glibc and musl) and Windows. + ExZarr no longer contains native code; Zig, zigler, zigler_precompiled and + system compression libraries are no longer needed. +- `:lz4` writes the numcodecs `LZ4` format (4-byte little-endian size prefix) + and `:bzip2` writes plain bzip2 streams (numcodecs `BZ2`), so zarr-python can + read them. Data written by earlier ExZarr versions (8-byte size prefix) is + still read. +- v2 `.zarray` compressor metadata uses numcodecs configurations: `lz4` + (`acceleration`), `blosc` (`cname`, `clevel`, `shuffle`, `blocksize`) and + `bz2` (previously `bzip2`). The old `bzip2` id is still read. zarr-python can + now open ExZarr v2 arrays compressed with lz4, blosc and bzip2. +- The v3 `blosc` codec honours `cname`, `clevel`, `shuffle` and `typesize` from + its configuration; without them ExZarr keeps BloscLZ, byte shuffle, + typesize 1. +- Decompressed chunks are capped at 256 MiB (`config :ex_zarr, + max_chunk_bytes: ...`) to guard against decompression bombs. +- Minimum OTP is 26 (ExCodecs NIFs target NIF 2.17). +- `ExZarr.Codecs.compress/3` accepts options as a map as well as a keyword list. ### Security - Bump `req` past CVE-2026-49755 (decompression-bomb DoS); GCS binary object @@ -127,6 +112,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `azurex` dependency and Azurex-based adapter code - Unused optional `google_api_storage` (GCS uses Goth + Req directly; this also unblocked `mime` 2.x required by patched Req) +- `ExZarr.Codecs.ZigCodecs`, `ExZarr.Codecs.CompressionConfig`, the + `fix_nif_rpaths` Mix task, the precompile workflow and + `docs/PRECOMPILATION.md` / `docs/COMPRESSION_SETUP.md`. +- `COMPRESSION_LIB_DIRS`, `HOMEBREW_PREFIX`, `BZIP2_STATIC_LIB`, `ZIG_PATH` and + `EX_ZARR_BUILD` no longer have any effect. ### Known limitations - Storage transformers are not executed (metadata only) diff --git a/README.md b/README.md index d5aba69..8118cb6 100644 --- a/README.md +++ b/README.md @@ -872,9 +872,10 @@ Key modules: See [ROADMAP.md](docs/ROADMAP.md) for the full release plan. -**v1.2.0 (released 2026-10-03)** - Zarr 3.1 interoperability & range-aware cloud I/O: +**v1.2.0 (released 2026-10-04)** - Zarr 3.1 interoperability & range-aware cloud I/O: spec `sharding_indexed`, optional byte-range reads, version-aware cloud keys, -AzureSDK migration, Python fixture/CI matrix. +AzureSDK migration, codecs via ExCodecs (no native toolchain), `compressor_config`, +Python fixture/CI matrix. **Upcoming** (high level): diff --git a/docs/INTEROPERABILITY.md b/docs/INTEROPERABILITY.md index 1df0cdb..bfc2c51 100644 --- a/docs/INTEROPERABILITY.md +++ b/docs/INTEROPERABILITY.md @@ -392,7 +392,7 @@ then the same `mix test --include python …` commands. ### Notes - The `python` executable used by Mix tests must be on `PATH` (tests shell out to it). -- Elixir/Zig/system libs are the same as a normal ExZarr test run (`mix deps.get`, Zig for NIFs). +- Elixir setup is the same as a normal ExZarr test run (`mix deps.get`); no native toolchain is needed. - Without `--include python` / `--only python_fixtures`, these tests are skipped. - Fixture generator details: `test/support/python_fixtures/README.md`. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7ed04a8..6354eb9 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -2,7 +2,7 @@ ## v1.2.0 — Zarr 3.1 Interoperability & Range-Aware Cloud I/O -Released 2026-10-03. Backward compatible with v1.1 (rewrite pre-1.2 private shards). +Released 2026-10-04. Backward compatible with v1.1 (rewrite pre-1.2 private shards; OTP 26+). - [x] Zarr 3.1 metadata: scalars, zero-length dims, dimension names, extensions, `storage_transformers` - [x] Spec-correct `sharding_indexed` v1.0 @@ -10,6 +10,8 @@ Released 2026-10-03. Backward compatible with v1.1 (rewrite pre-1.2 private shar - [x] Range-aware sharded reads with full-shard fallback - [x] Version-aware cloud object keys (S3 / GCS / Azure) - [x] Azure Blob backend on `azure_sdk ~> 0.4.1` (azurex removed) +- [x] Codecs via ExCodecs (precompiled pure-Rust NIFs; no Zig or system libraries) +- [x] `compressor_config:` and `default_zarr_version` applied - [x] Python fixture generator + CI matrix job for zarr 2.x / 3.2 / 3.3 / 3.4 --- diff --git a/docs/release_notes_v1_2_0.md b/docs/release_notes_v1_2_0.md index 0c6ef89..0e302b8 100644 --- a/docs/release_notes_v1_2_0.md +++ b/docs/release_notes_v1_2_0.md @@ -4,8 +4,10 @@ ExZarr v1.2.0 focuses on Zarr 3.1 conformance, standard `sharding_indexed` interoperability, optional byte-range cloud I/O, migrating Azure Blob -storage to `azure_sdk`, and shipping **precompiled Zig codec NIFs** for -Hex/Livebook installs (see `PRECOMPILATION.md`). +storage to `azure_sdk`, and moving the compression codecs to +[ExCodecs](https://hex.pm/packages/ex_codecs): precompiled pure-Rust NIFs, so +ExZarr itself has no native code and needs no Zig, compiler or system +libraries. ## Highlights @@ -18,16 +20,36 @@ Hex/Livebook installs (see `PRECOMPILATION.md`). ### Standard sharding + range reads -- `sharding_indexed` v1.0 dense uint64 index with empty sentinel +- `sharding_indexed` v1.0 dense uint64 index with empty sentinel, readable and + writable by zarr-python - Optional `chunk_info` / `read_chunk_range` backend capabilities - Range-aware inner-chunk reads with full-shard fallback - S3, GCS, Azure, filesystem, and memory backends -### Precompiled Zig codec NIFs - -- Hex/Livebook installs download platform NIFs (no Zig toolchain required) -- Source builds via `EX_ZARR_BUILD=1` or missing checksum file -- CI workflow: `.github/workflows/precompile.yml` +### Codecs via ExCodecs + +- zstd, lz4, snappy, blosc, bzip2 and crc32c come from ExCodecs `~> 0.2.4`, + precompiled for macOS, Linux (glibc and musl) and Windows. zlib/gzip use + Erlang's `:zlib`. +- Byte formats match numcodecs, so zarr-python reads ExZarr's lz4, bzip2 and + blosc data (v2 and v3), and ExZarr reads zarr-python's. +- New `compressor_config:` option, for example: + + ```elixir + ExZarr.create( + shape: {1000, 1000}, + chunks: {100, 100}, + compressor: :blosc, + compressor_config: [cname: :zstd, level: 5, shuffle: :bit, typesize: 8] + ) + ``` + + v2 arrays keep the settings in `.zarray` (numcodecs form); v3 arrays keep + them in the codec configuration. +- `config :ex_zarr, default_zarr_version: 3` now applies to new arrays + (unset: v2, as before). +- Decompressed chunks are capped at 256 MiB by default + (`config :ex_zarr, max_chunk_bytes: ...`). ### AzureSDK migration @@ -54,8 +76,8 @@ ExZarr.open( ### Python interoperability matrix -Fixture generator under `test/support/python_fixtures/` and a CI job covering -zarr-python 2.x / 3.2 / 3.3 / 3.4. +Fixture generator under `test/support/python_fixtures/` and a blocking CI job +covering zarr-python 2.18.3 / 3.2.1 / 3.3.0 / 3.4.0, in both directions. ### Security @@ -64,10 +86,19 @@ zarr-python 2.x / 3.2 / 3.3 / 3.4. ## Upgrade -- Azure users: add `{:azure_sdk, "~> 0.4.1", optional: true}` and remove `azurex` -- Arrays written with ExZarr’s pre-1.2 private shard index must be rewritten -- Cloud v3 arrays now use `zarr.json` and `c/...` keys -- GCS/Azure users: use `{:req, "~> 0.6.1"}` (ExZarr overrides older azure_sdk caps) +- **OTP 26 or later** is required (ExCodecs NIFs target NIF 2.17). +- Remove any Zig, `COMPRESSION_LIB_DIRS`, `EX_ZARR_BUILD` or system + compression-library setup; it is no longer used. +- Azure users: add `{:azure_sdk, "~> 0.4.1", optional: true}` and remove `azurex`. +- GCS/Azure users: use `{:req, "~> 0.6.1"}`. +- Arrays written with ExZarr's pre-1.2 private shard index must be rewritten. +- Cloud v3 arrays now use `zarr.json` and `c/...` keys. +- New lz4 and bzip2 data uses the numcodecs formats. ExZarr 1.2 still reads + data written by 1.1, but ExZarr 1.1 cannot read lz4/bzip2 data written by 1.2. +- `ExZarr.create/1` now returns an error for unknown compressors, invalid + `compressor_config` and Zarr versions other than 2 or 3, and for v3 + compressors without a v3 codec (such as `:snappy`), which were previously + stored uncompressed. ## Known limitations @@ -75,4 +106,4 @@ zarr-python 2.x / 3.2 / 3.3 / 3.4. - No adaptive range coalescing - No partial writes inside cloud shards -See [CHANGELOG.md](../CHANGELOG.md) and [ROADMAP.md](ROADMAP.md). +See [CHANGELOG.md](https://github.com/AnimaLogica/ExZarr/blob/main/CHANGELOG.md#120---2026-10-04) and [ROADMAP.md](https://github.com/AnimaLogica/ExZarr/blob/main/docs/ROADMAP.md).