[thanos]

ExZarr's codecs now run on ex_codecs 0.2.4, and ExZarr has no native code of its own. Everything is uncommitted on windows_build_fixes_-codec_switch.

Verification:

• Full suite: 146 doctests, 70 properties, 1649 tests, 0 failures. Compile with warnings as errors, format, credo, dialyzer, mix docs --warnings-as-errors and sobelow all pass, and the example runs.
• Data from older ExZarr: I built the pre-switch commit (with its Zig NIFs) in a temporary worktree and saved one encoded payload per codec as fixtures in test/fixtures/legacy_codecs/. All seven decode with the new code.
• zarr-python:
  • Byte level: the five codecs zarr-python has decode in numcodecs; snappy has no numcodecs codec.
  • Arrays: v2 arrays round-trip both ways with zarr-python 2.18 for zlib, zstd, lz4, blosc and bzip2.
  • Suites: the v3 suite passes 19/19 on zarr-python 3.4.0 and the v2 suite 14/14 on 2.18.3; the fixture test passes on 2.18.3, 3.2.1 and 3.4.0.

What changed:

• Dependencies: added {:ex_codecs, "~> 0.2.4"}; zigler, zigler_precompiled and their 16 transitive packages are gone. ex_doc is now dev-only, because the reason it was needed in prod (a zigler dependency) is gone.
• Codecs (ExZarr.Codecs): all built-ins go through ExCodecs, except zlib, which stays on Erlang's :zlib.
  • Behaviour unchanged: crc32c error atoms are kept, and blosc keeps its old defaults (BloscLZ, byte shuffle, typesize 1).
  • LZ4 and bzip2 now write the numcodecs formats, which zarr-python couldn't read before. Data written by ExZarr 1.1/1.2 (8-byte size prefix) still decodes.
  • New option: compress/3 also accepts options as a map.
  • New limit: decoded chunks are capped at 256 MiB, configurable with config :ex_zarr, max_chunk_bytes: ..., as protection against decompression bombs.
• v2 metadata (storage.ex): zarr-python couldn't open ExZarr v2 arrays compressed with lz4, blosc or bzip2, because every compressor was written as {"id": ..., "level": 5}. New arrays get numcodecs configurations, and bzip2 is written as "bz2". Old "bzip2" is still open.
• v3 blosc codec: now applies cname, clevel, shuffle and typesize from zarr.json.
• CI: removed the Zig and system-library steps (12 steps in total), and deleted precompile.yml, including the uncommitted Windows fix in it, which is no longer needed.
• Removed: the Zig sources, ZigCodecs, CompressionConfig, the fix_nif_rpaths task and its tests, the two obsolete docs, the leftover NIF binaries in priv/lib, and an autogenerated .zig file that had been committed by mistake.
• Docs:
  • Rewritten: README, CONFIGURATION.md, the installation, troubleshooting, overview and codec guides.
  • Updated: interop, glossary, contributing, core concepts, v3 status, release checklist, CHANGELOG (under [Unreleased]).
  • Corrected: minimum OTP is now 26, which ex_codecs' NIFs require, and Windows is listed as supported.
• Tests added: codec_formats_test.exs (legacy and numcodecs fixtures, output formats, v3 blosc config, v2 metadata, size cap), plus four numcodecs compressors in the v2 fixture generator.

Decisions for you:

1. Version: mix.exs still says 1.2.0. The v1.2.0 tag exists but Hex only has 1.1.0, so you can ship this as part of 1.2.0 or bump it.
2. Historical docs: I left docs/release_notes_v1_2_0.md and .github/RELEASE_ANNOUNCEMENT.md unchanged, since they describe 1.2.0 as tagged with Zig NIFs. If this goes into 1.2.0, they need updating.

Problems found but not fixed:

• config :ex_zarr, default_zarr_version is documented but never applied; ExZarr.create always defaults to v2.
• The codec guide's compressor_config: option doesn't exist anywhere in the code.

Power cut: zig_codecs.ex had come back as an empty file, so I deleted it again. A clean recompile and a full test run after the outage both passed.

Verification:

• Full suite: 155 doctests, 70 properties, 1663 tests, 0 failures. Format, credo, dialyzer and mix docs --warnings-as-errors all pass.
• zarr-python: the v2 suite (2.18) passes 14/14 and the v3 suite (3.4) 19/19. Arrays created with compressor_config open in zarr-python with exactly the configured settings, e.g. Zstd(level=19) and Blosc(cname='lz4', clevel=7, shuffle=BITSHUFFLE).

1. default_zarr_version

• ExZarr.create/1 and ExZarr.Nx now use the setting whenever zarr_version: isn't given.
• Unset, the default stays v2. Every call site already defaulted to v2 "for backward compatibility", so switching to v3 would have been a breaking change. ExZarr.Version.default_version/0 (whose docs and fallback said v3, though nothing called it) now returns 2.
• An invalid version, passed or configured, now returns {:error, {:invalid_zarr_version, v}}. Before, it crashed with a FunctionClauseError.

2. compressor_config:

• New: ExZarr.Codecs.CompressorConfig validates the settings for each compressor:

Compressor	Settings
zlib, gzip	level 0–9
zstd	level 1–22
bzip2	level 1–9
blosc	cname, level/clevel, shuffle, typesize
lz4, snappy, crc32c, none	none
custom codecs	passed through unchanged

• v2 arrays: settings apply on every write, are saved in .zarray in numcodecs form, and are read back on open. So later writes, including to arrays created by zarr-python, keep the same settings.
• v3 arrays: settings go into the compressor codec's configuration in zarr.json.
• zlib: now honours level. It used to ignore it.
• Docs: ExZarr.create/1, the codec guide (LZ4 has no levels; gzip is v3-only), CONFIGURATION.md and the CHANGELOG are updated. test/ex_zarr/compressor_config_test.exs covers all of this.

Behaviour changes worth knowing:

• v3 arrays could silently skip compression. A v3 array created with a compressor that has no v3 codec (:gzip, :snappy, or a custom codec without codecs:) was stored uncompressed. A test named "works with compression codecs" passed that way.
  • :gzip is now a real v3 compressor.
  • The others now return {:error, {:unsupported_codec_for_v3, c}} instead of silently not compressing.
• Unknown compressors fail at create. compressor: :gzip on v2, for example, now returns {:error, {:unsupported_codec, …}} from create, instead of create succeeding and every write failing.
• v2 metadata records the real default level: zlib 6 and zstd 3, instead of a hardcoded 5.
• Blosc typesize isn't persisted for v2 arrays: numcodecs' v2 Blosc config has no field for it, so it only applies until the array is reopened. This is documented.

Dialyzer: its cached PLT in priv/plts was stale (built against Elixir 1.20.2; this shell runs 1.19.5). I deleted and rebuilt it; it's a local, untracked cache.

• closed wire ex_codecs into ExZarr behind a config switch #290
• closed Dependencies: add {:ex_codecs, "~> 0.2.4"}; remove zigler, zigler_precompiled and their 16 transitive packages. #291
• closed Use Codecs ExCodecs, except zlib, which stays on Erlang's :zlib. #292
• closed LZ4 and bzip2 must write the numcodecs formats #293
• closed compress/3 should accepts options as a map. #294
• closed add New limit: decode chunks should be capped at 256 MiB, configurable with config :ex_zarr, max_chunk_bytes: #295
• closed arr-python can't open ExZarr v2 arrays compressed with lz4, blosc or bzip2, #296
• closed v3 blosc codec: should apply cname, clevel, shuffle and typesize from zarr.json. #297
• closed CI: remove the Zig and system-library steps #298
• closed Remove: the Zig sources, ZigCodecs, CompressionConfig, the fix_nif_rpaths task and its tests, the two obsolete docs, the leftover NIF binaries in priv/lib, and an autogenerated .zig file that had been committed by mistake. #299
• closed Rewrite: README, CONFIGURATION.md, the installation, troubleshooting, overview and codec guides. #300
• closed Update: interop, glossary, contributing, core concepts, v3 status, release checklist, CHANGELOG (under [Unreleased]). #301
• closed Correct: minimum OTP is now 26, which ex_codecs' NIFs require, and Windows is listed as supported. #302
• closed Add test for codec_formats_test.exs (legacy and numcodecs fixtures, output formats, v3 blosc config, v2 metadata, size cap), plus four numcodecs compressors in the v2 fixture generator. #303
• closed fix: config :ex_zarr, default_zarr_version is documented but never applied; ExZarr.create always defaults to v2. #304
• closed fix - The codec guide's compressor_config: option doesn't exist anywhere in the code. #305

Description

Related Issues

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code refactoring
• Performance improvement
• Test improvement

Changes Made

Testing

• All existing tests pass
• Added new tests for new functionality
• Tested manually with:

  # Example usage

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

Breaking Changes

None / N/A

Additional Notes