From b86c41bfa3076250c9d035257317bb36fc57afa7 Mon Sep 17 00:00:00 2001 From: thanos Date: Sun, 4 Oct 2026 15:08:05 -0400 Subject: [PATCH] ExZarr's codecs now run on ex_codecs 0.2.4, and ExZarr has no native code of its own. Everything is uncommitted on windows_build_fixes_-codec_switch. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verification: - Full suite: 146 doctests, 70 properties, 1649 tests, 0 failures. Compile with warnings as errors, format, credo, dialyzer, mix docs --warnings-as-errors and sobelow all pass, and the example runs. - Data from older ExZarr: I built the pre-switch commit (with its Zig NIFs) in a temporary worktree and saved one encoded payload per codec as fixtures in test/fixtures/legacy_codecs/. All seven decode with the new code. - zarr-python: - Byte level: the five codecs zarr-python has decode in numcodecs; snappy has no numcodecs codec. - Arrays: v2 arrays round-trip both ways with zarr-python 2.18 for zlib, zstd, lz4, blosc and bzip2. - Suites: the v3 suite passes 19/19 on zarr-python 3.4.0 and the v2 suite 14/14 on 2.18.3; the fixture test passes on 2.18.3, 3.2.1 and 3.4.0. What changed: - Dependencies: added {:ex_codecs, "~> 0.2.4"}; zigler, zigler_precompiled and their 16 transitive packages are gone. ex_doc is now dev-only, because the reason it was needed in prod (a zigler dependency) is gone. - Codecs (ExZarr.Codecs): all built-ins go through ExCodecs, except zlib, which stays on Erlang's :zlib. - Behaviour unchanged: crc32c error atoms are kept, and blosc keeps its old defaults (BloscLZ, byte shuffle, typesize 1). - LZ4 and bzip2 now write the numcodecs formats, which zarr-python couldn't read before. Data written by ExZarr 1.1/1.2 (8-byte size prefix) still decodes. - New option: compress/3 also accepts options as a map. - New limit: decoded chunks are capped at 256 MiB, configurable with config :ex_zarr, max_chunk_bytes: ..., as protection against decompression bombs. - v2 metadata (storage.ex): zarr-python couldn't open ExZarr v2 arrays compressed with lz4, blosc or bzip2, because every compressor was written as {"id": ..., "level": 5}. New arrays get numcodecs configurations, and bzip2 is written as "bz2". Old "bzip2" is still open. - v3 blosc codec: now applies cname, clevel, shuffle and typesize from zarr.json. - CI: removed the Zig and system-library steps (12 steps in total), and deleted precompile.yml, including the uncommitted Windows fix in it, which is no longer needed. - Removed: the Zig sources, ZigCodecs, CompressionConfig, the fix_nif_rpaths task and its tests, the two obsolete docs, the leftover NIF binaries in priv/lib, and an autogenerated .zig file that had been committed by mistake. - Docs: - Rewritten: README, CONFIGURATION.md, the installation, troubleshooting, overview and codec guides. - Updated: interop, glossary, contributing, core concepts, v3 status, release checklist, CHANGELOG (under [Unreleased]). - Corrected: minimum OTP is now 26, which ex_codecs' NIFs require, and Windows is listed as supported. - Tests added: codec_formats_test.exs (legacy and numcodecs fixtures, output formats, v3 blosc config, v2 metadata, size cap), plus four numcodecs compressors in the v2 fixture generator. Decisions for you: 1. Version: mix.exs still says 1.2.0. The v1.2.0 tag exists but Hex only has 1.1.0, so you can ship this as part of 1.2.0 or bump it. 2. Historical docs: I left docs/release_notes_v1_2_0.md and .github/RELEASE_ANNOUNCEMENT.md unchanged, since they describe 1.2.0 as tagged with Zig NIFs. If this goes into 1.2.0, they need updating. Problems found but not fixed: - config :ex_zarr, default_zarr_version is documented but never applied; ExZarr.create always defaults to v2. - The codec guide's compressor_config: option doesn't exist anywhere in the code. Power cut: zig_codecs.ex had come back as an empty file, so I deleted it again. A clean recompile and a full test run after the outage both passed. Verification: - Full suite: 155 doctests, 70 properties, 1663 tests, 0 failures. Format, credo, dialyzer and mix docs --warnings-as-errors all pass. - zarr-python: the v2 suite (2.18) passes 14/14 and the v3 suite (3.4) 19/19. Arrays created with compressor_config open in zarr-python with exactly the configured settings, e.g. Zstd(level=19) and Blosc(cname='lz4', clevel=7, shuffle=BITSHUFFLE). 1. default_zarr_version - ExZarr.create/1 and ExZarr.Nx now use the setting whenever zarr_version: isn't given. - Unset, the default stays v2. Every call site already defaulted to v2 "for backward compatibility", so switching to v3 would have been a breaking change. ExZarr.Version.default_version/0 (whose docs and fallback said v3, though nothing called it) now returns 2. - An invalid version, passed or configured, now returns {:error, {:invalid_zarr_version, v}}. Before, it crashed with a FunctionClauseError. 2. compressor_config: - New: ExZarr.Codecs.CompressorConfig validates the settings for each compressor: | Compressor | Settings | |---------------------------|----------------------------------------| | zlib, gzip | level 0–9 | | zstd | level 1–22 | | bzip2 | level 1–9 | | blosc | cname, level/clevel, shuffle, typesize | | lz4, snappy, crc32c, none | none | | custom codecs | passed through unchanged | - v2 arrays: settings apply on every write, are saved in .zarray in numcodecs form, and are read back on open. So later writes, including to arrays created by zarr-python, keep the same settings. - v3 arrays: settings go into the compressor codec's configuration in zarr.json. - zlib: now honours level. It used to ignore it. - Docs: ExZarr.create/1, the codec guide (LZ4 has no levels; gzip is v3-only), CONFIGURATION.md and the CHANGELOG are updated. test/ex_zarr/compressor_config_test.exs covers all of this. Behaviour changes worth knowing: - v3 arrays could silently skip compression. A v3 array created with a compressor that has no v3 codec (:gzip, :snappy, or a custom codec without codecs:) was stored uncompressed. A test named "works with compression codecs" passed that way. - :gzip is now a real v3 compressor. - The others now return {:error, {:unsupported_codec_for_v3, c}} instead of silently not compressing. - Unknown compressors fail at create. compressor: :gzip on v2, for example, now returns {:error, {:unsupported_codec, …}} from create, instead of create succeeding and every write failing. - v2 metadata records the real default level: zlib 6 and zstd 3, instead of a hardcoded 5. - Blosc typesize isn't persisted for v2 arrays: numcodecs' v2 Blosc config has no field for it, so it only applies until the array is reopened. This is documented. Dialyzer: its cached PLT in priv/plts was stale (built against Elixir 1.20.2; this shell runs 1.19.5). I deleted and rebuilt it; it's a local, untracked cache. - closed #290 - closed #291 - closed #292 - closed #293 - closed #294 - closed #295 - closed #296 - closed #297 - closed #298 - closed #299 - closed #300 - closed #301 - closed #302 - closed #303 - closed #304 - closed #305 --- .dialyzer_ignore.exs | 4 - .github/RELEASE_CHECKLIST.md | 23 +- .github/workflows/ci.yml | 55 --- .github/workflows/precompile.yml | 172 -------- CHANGELOG.md | 55 +++ README.md | 44 +- config/config.exs | 19 - config/dev.exs | 3 - config/test.exs | 3 - docs/COMPRESSION_SETUP.md | 313 -------------- docs/CONFIGURATION.md | 305 +++----------- docs/PRECOMPILATION.md | 65 --- docs/ZARR_V3_STATUS.md | 2 +- docs/guides/compression_codecs.md | 231 ++++------- docs/guides/contributing.md | 16 +- docs/guides/core_concepts.md | 16 +- docs/guides/glossary.md | 20 +- docs/guides/installation.md | 384 +++-------------- docs/guides/overview.md | 74 ++-- docs/guides/python_interop.md | 14 +- docs/guides/troubleshooting.md | 210 +++------- lib/ex_zarr.ex | 19 +- lib/ex_zarr/array.ex | 55 ++- lib/ex_zarr/codecs.ex | 391 +++++++----------- .../.Elixir.ExZarr.Codecs.ZigCodecs.zig | 366 ---------------- .../codecs/.Elixir.ExZarr.Codecs.ZigNifs.zig | 94 ----- lib/ex_zarr/codecs/compression_config.ex | 235 ----------- lib/ex_zarr/codecs/compressor_config.ex | 284 +++++++++++++ lib/ex_zarr/codecs/pipeline_v3.ex | 61 ++- lib/ex_zarr/codecs/registry.ex | 42 +- lib/ex_zarr/codecs/zig_codecs.ex | 132 ------ lib/ex_zarr/codecs/zig_codecs.zig | 366 ---------------- lib/ex_zarr/metadata.ex | 5 + lib/ex_zarr/metadata_v3.ex | 9 +- lib/ex_zarr/nx.ex | 5 +- lib/ex_zarr/storage.ex | 22 +- lib/ex_zarr/version.ex | 14 +- lib/mix/tasks/fix_nif_rpaths.ex | 192 --------- mix.exs | 20 +- mix.lock | 19 +- native/codecs/compression.zig | 79 ---- test/ex_zarr/codecs/codec_formats_test.exs | 190 +++++++++ .../codecs/compression_config_test.exs | 379 ----------------- test/ex_zarr/compressor_config_test.exs | 205 +++++++++ test/ex_zarr_codecs_test.exs | 22 - test/fixtures/legacy_codecs/blosc.bin | Bin 0 -> 961 bytes test/fixtures/legacy_codecs/bzip2.bin | Bin 0 -> 897 bytes test/fixtures/legacy_codecs/crc32c.bin | Bin 0 -> 40004 bytes test/fixtures/legacy_codecs/lz4.bin | Bin 0 -> 927 bytes test/fixtures/legacy_codecs/snappy.bin | Bin 0 -> 2601 bytes test/fixtures/legacy_codecs/zlib.bin | Bin 0 -> 1068 bytes test/fixtures/legacy_codecs/zstd.bin | Bin 0 -> 774 bytes .../numcodecs/blosc_blosclz_shuffle.bin | Bin 0 -> 961 bytes .../numcodecs/blosc_lz4_bitshuffle.bin | Bin 0 -> 4607 bytes .../numcodecs/blosc_zstd_noshuffle.bin | Bin 0 -> 798 bytes test/fixtures/numcodecs/bz2.bin | Bin 0 -> 889 bytes test/fixtures/numcodecs/generate.py | 29 ++ test/fixtures/numcodecs/lz4.bin | Bin 0 -> 923 bytes test/fixtures/numcodecs/zstd.bin | Bin 0 -> 774 bytes test/mix/tasks/fix_nif_rpaths_test.exs | 175 -------- .../python_fixtures/generate_fixtures.py | 22 +- 61 files changed, 1423 insertions(+), 4037 deletions(-) delete mode 100644 .github/workflows/precompile.yml delete mode 100644 docs/COMPRESSION_SETUP.md delete mode 100644 docs/PRECOMPILATION.md delete mode 100644 lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigCodecs.zig delete mode 100644 lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigNifs.zig delete mode 100644 lib/ex_zarr/codecs/compression_config.ex create mode 100644 lib/ex_zarr/codecs/compressor_config.ex delete mode 100644 lib/ex_zarr/codecs/zig_codecs.ex delete mode 100644 lib/ex_zarr/codecs/zig_codecs.zig delete mode 100644 lib/mix/tasks/fix_nif_rpaths.ex delete mode 100644 native/codecs/compression.zig create mode 100644 test/ex_zarr/codecs/codec_formats_test.exs delete mode 100644 test/ex_zarr/codecs/compression_config_test.exs create mode 100644 test/ex_zarr/compressor_config_test.exs create mode 100644 test/fixtures/legacy_codecs/blosc.bin create mode 100644 test/fixtures/legacy_codecs/bzip2.bin create mode 100644 test/fixtures/legacy_codecs/crc32c.bin create mode 100644 test/fixtures/legacy_codecs/lz4.bin create mode 100644 test/fixtures/legacy_codecs/snappy.bin create mode 100644 test/fixtures/legacy_codecs/zlib.bin create mode 100644 test/fixtures/legacy_codecs/zstd.bin create mode 100644 test/fixtures/numcodecs/blosc_blosclz_shuffle.bin create mode 100644 test/fixtures/numcodecs/blosc_lz4_bitshuffle.bin create mode 100644 test/fixtures/numcodecs/blosc_zstd_noshuffle.bin create mode 100644 test/fixtures/numcodecs/bz2.bin create mode 100644 test/fixtures/numcodecs/generate.py create mode 100644 test/fixtures/numcodecs/lz4.bin create mode 100644 test/fixtures/numcodecs/zstd.bin delete mode 100644 test/mix/tasks/fix_nif_rpaths_test.exs diff --git a/.dialyzer_ignore.exs b/.dialyzer_ignore.exs index 675839a..0faee2e 100644 --- a/.dialyzer_ignore.exs +++ b/.dialyzer_ignore.exs @@ -7,10 +7,6 @@ {"lib/ex_zarr/telemetry.ex", :contract_supertype}, # Pattern match and coverage warnings - false positives from conditional compilation - {"lib/ex_zarr/codecs.ex", :pattern_match}, - {"lib/ex_zarr/codecs.ex", :pattern_match_cov}, - {"lib/ex_zarr/codecs/compression_config.ex", :pattern_match_cov}, - {"lib/ex_zarr/codecs/registry.ex", :pattern_match_cov}, {"lib/ex_zarr/storage/backend/mongo_gridfs.ex", :pattern_match}, # Callback arg type mismatches - behavior signatures are intentionally flexible diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md index edda5d4..bfed4cb 100644 --- a/.github/RELEASE_CHECKLIST.md +++ b/.github/RELEASE_CHECKLIST.md @@ -17,19 +17,18 @@ Use this checklist when preparing and publishing the release. - [x] `docs/release_notes_v1_2_0.md` current - [x] `docs/ROADMAP.md` marks v1.2.0 released; next is v1.3.0 - [x] `docs/INTEROPERABILITY.md` Python interop testing section current -- [x] GitHub org links use **AnimaLogica/ExZarr** (Hex `source_url`, badges, Zig NIF `base_url`) +- [x] GitHub org links use **AnimaLogica/ExZarr** (Hex `source_url`, badges) - [ ] Examples / showcase reviewed (`examples/range_aware_sharded_nx.exs`) ### Version Numbers - [x] `mix.exs` → `1.2.0` - [x] Guides / livebooks / notebooks install snippets → `~> 1.2` -- [x] ZiglerPrecompiled `base_url` → `https://github.com/AnimaLogica/ExZarr/releases/download/v#{version}` +- [ ] `ex_codecs` dependency points at a published release with a complete checksum file - [x] CHANGELOG comparison links include 1.2.0 / 1.1.0 / 1.0.0 ### Testing - [ ] `mix test` - [ ] Python interop (see docs/INTEROPERABILITY.md): fixtures + `--include python` for v2 and v3 -- [ ] Precompile dry-run locally if changing NIFs (`docs/PRECOMPILATION.md`) ## Git and GitHub @@ -39,30 +38,16 @@ Use this checklist when preparing and publishing the release. - [ ] Push: `git push origin main && git push origin v1.2.0` - [ ] Confirm tag on https://github.com/AnimaLogica/ExZarr -### Precompiled NIFs -- [ ] Wait for `.github/workflows/precompile.yml` to attach `.tar.gz` assets to the GitHub Release - (targets: linux x86_64/arm64, macOS arm64, Windows x86_64) -- [ ] If a tag already exists and the workflow was fixed after tagging: push the fix, then - `gh workflow run "Build precompiled NIFs" -f release_tag=v1.2.0` -- [ ] Generate checksums: - ```bash - EX_ZARR_BUILD=1 mix deps.get - mix zigler_precompiled.download ExZarr.Codecs.ZigCodecs --all --print - ``` -- [ ] Commit `checksum-ExZarr.Codecs.ZigCodecs.exs` -- [ ] (Optional) mark `zigler` `optional: true` again once checksums ship — see `docs/PRECOMPILATION.md` - ### GitHub Release - [ ] Draft release for tag `v1.2.0` - [ ] Title: `ExZarr v1.2.0 - Zarr 3.1 Interoperability & Range-Aware Cloud I/O` - [ ] Body from `.github/RELEASE_ANNOUNCEMENT.md` / `docs/release_notes_v1_2_0.md` -- [ ] Ensure NIF assets from precompile workflow are attached - [ ] Publish as latest release ## Hex.pm Publication ### Prepare -- [ ] Review `mix.exs` `package/0` (files include `checksum-*.exs`, CHANGELOG, LICENSE) +- [ ] Review `mix.exs` `package/0` (files include lib, CHANGELOG, LICENSE) - [ ] `mix docs` - [ ] `mix hex.build` and inspect `ex_zarr-1.2.0.tar` @@ -85,7 +70,7 @@ Use this checklist when preparing and publishing the release. ## Rollback Plan (If Needed) 1. **Hex**: `mix hex.retire ex_zarr 1.2.0 --reason security` (or other reason), then publish a patch -2. **GitHub**: mark release as pre-release / yank NIF assets if corrupted; ship hotfix tag +2. **GitHub**: mark release as pre-release; ship hotfix tag 3. **Comms**: update announcement channels and docs with workaround ## Release Date diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 58ed9d0..3b81ea0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,19 +42,6 @@ jobs: # otp: '25.3' steps: - - name: Install system libraries - run: | - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - - name: Install Zig - uses: goto-bus-stop/setup-zig@v2 - with: - version: 0.16.0 - - - name: Verify Zig toolchain - run: | - echo "ZIG_EXECUTABLE_PATH=$(which zig)" >> "$GITHUB_ENV" - zig version - name: Checkout code uses: actions/checkout@v4 @@ -111,21 +98,6 @@ jobs: runs-on: ubuntu-latest steps: - - name: Install system libraries - run: | - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - - - name: Install Zig - uses: goto-bus-stop/setup-zig@v2 - with: - version: 0.16.0 - - - name: Verify Zig toolchain - run: | - echo "ZIG_EXECUTABLE_PATH=$(which zig)" >> "$GITHUB_ENV" - zig version - - name: Checkout code uses: actions/checkout@v4 @@ -177,21 +149,6 @@ jobs: MIX_ENV: dev steps: - - name: Install system libraries - run: | - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - - - name: Install Zig - uses: goto-bus-stop/setup-zig@v2 - with: - version: 0.16.0 - - - name: Verify Zig toolchain - run: | - echo "ZIG_EXECUTABLE_PATH=$(which zig)" >> "$GITHUB_ENV" - zig version - - name: Checkout code uses: actions/checkout@v4 @@ -273,18 +230,6 @@ jobs: with: elixir-version: "1.19.5" otp-version: "28.3.1" - - name: Install system libraries - run: | - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - - name: Install Zig - uses: goto-bus-stop/setup-zig@v2 - with: - version: 0.16.0 - - name: Verify Zig - run: | - echo "ZIG_EXECUTABLE_PATH=$(which zig)" >> "$GITHUB_ENV" - zig version - run: mix deps.get - name: Read zarr-python fixtures with ExZarr env: diff --git a/.github/workflows/precompile.yml b/.github/workflows/precompile.yml deleted file mode 100644 index 285e765..0000000 --- a/.github/workflows/precompile.yml +++ /dev/null @@ -1,172 +0,0 @@ -name: Build precompiled NIFs - -on: - push: - tags: ["v*"] - workflow_dispatch: - inputs: - # Attach built assets to an existing GitHub Release (needed when re-running - # after a workflow fix without moving the tag). - release_tag: - description: "Existing release tag to upload assets to (e.g. v1.2.0)" - required: false - type: string - -permissions: - contents: write - -jobs: - build: - runs-on: ${{ matrix.os }} - strategy: - fail-fast: false - matrix: - include: - # Native builds — ZigCodecs dynamically links system compression libs - - { target: x86_64-linux-gnu, os: ubuntu-latest } - - { target: aarch64-linux-gnu, os: ubuntu-24.04-arm } - - { target: aarch64-macos-none, os: macos-15 } - - { target: x86_64-windows-gnu, os: windows-latest } - - steps: - - uses: actions/checkout@v4 - - - name: Install compression libraries (Linux) - if: runner.os == 'Linux' - run: | - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - - - name: Install compression libraries (macOS) - if: runner.os == 'macOS' - run: brew install zstd lz4 snappy c-blosc bzip2 - - - name: Install MSYS2 compression libraries (Windows) - if: runner.os == 'Windows' - id: msys2 - uses: msys2/setup-msys2@v2 - with: - # MINGW64 is deprecated; UCRT64 is the current MinGW environment - msystem: UCRT64 - update: true - path-type: inherit - install: >- - mingw-w64-ucrt-x86_64-zstd - mingw-w64-ucrt-x86_64-lz4 - mingw-w64-ucrt-x86_64-snappy - mingw-w64-ucrt-x86_64-blosc - mingw-w64-ucrt-x86_64-bzip2 - mingw-w64-ucrt-x86_64-pkgconf - mingw-w64-ucrt-x86_64-cc - - # Zigler 0.16 requires Zig 0.16 — do not pin to 0.15.x - - name: Install Zig - uses: goto-bus-stop/setup-zig@v2 - with: - version: 0.16.0 - - # Needed for zig translate-c -lc on Windows (MSVC libc) - - name: Setup MSVC environment (Windows) - if: runner.os == 'Windows' - uses: ilammy/msvc-dev-cmd@v1 - - - uses: erlef/setup-beam@v1 - with: - otp-version: "27" - elixir-version: "1.18" - - - name: Build NIF artifact - shell: bash - env: - EX_ZARR_BUILD: "1" - ZIGLER_RELEASE_MODE: fast - # Absolute Windows path from setup-msys2 (Git Bash maps bare /ucrt64 - # under Program Files\Git, which breaks Zig -L/-I). - MSYS2_LOCATION: ${{ steps.msys2.outputs.msys2-location }} - RUNNER_TEMP: ${{ runner.temp }} - run: | - set -euo pipefail - - # Pin ONE Zig. On Windows, `command -v zig` returns a Git Bash path - # (/c/...) that Elixir File.exists?/1 rejects — Zigler then ignores - # ZIG_EXECUTABLE_PATH and uses its own cached Zig. Use cygpath -m. - zig_bash="$(command -v zig)" - if [[ "${{ runner.os }}" == "Windows" ]]; then - export ZIG_EXECUTABLE_PATH="$(cygpath -m "${zig_bash}")" - else - export ZIG_EXECUTABLE_PATH="${zig_bash}" - fi - echo "Using Zig at ${ZIG_EXECUTABLE_PATH}" - zig version - - if [[ "${{ runner.os }}" == "Windows" ]]; then - if [[ -z "${MSYS2_LOCATION:-}" ]]; then - echo "MSYS2_LOCATION is empty; setup-msys2 did not report a path" >&2 - exit 1 - fi - # D:\a\_temp\msys64 -> D:/a/_temp/msys64/ucrt64 (Zig wants drive paths) - prefix="${MSYS2_LOCATION//\\//}/ucrt64" - export COMPRESSION_LIB_DIRS="${prefix}/lib" - export COMPRESSION_INCLUDE_DIRS="${prefix}/include" - export PATH="${prefix}/bin:${PATH}" - echo "COMPRESSION_LIB_DIRS=${COMPRESSION_LIB_DIRS}" - test -d "${COMPRESSION_LIB_DIRS}" - test -f "${COMPRESSION_LIB_DIRS}/libzstd.dll.a" \ - || test -f "${COMPRESSION_LIB_DIRS}/libzstd.a" - - # Fresh Zig caches (default %LOCALAPPDATA%\zig was hitting CacheCheckFailed - # during translate-c of erl_nif_win.h). One Zig + clean temp dirs. - export ZIG_GLOBAL_CACHE_DIR="$(cygpath -m "${RUNNER_TEMP}")/zig-global-cache" - export ZIG_LOCAL_CACHE_DIR="$(cygpath -m "${RUNNER_TEMP}")/zig-local-cache" - rm -rf "${ZIG_GLOBAL_CACHE_DIR}" "${ZIG_LOCAL_CACHE_DIR}" .zig-cache \ - "$(cygpath -u "${LOCALAPPDATA}")/zig" 2>/dev/null || true - mkdir -p "${ZIG_GLOBAL_CACHE_DIR}" "${ZIG_LOCAL_CACHE_DIR}" - echo "ZIG_GLOBAL_CACHE_DIR=${ZIG_GLOBAL_CACHE_DIR}" - echo "ZIG_LOCAL_CACHE_DIR=${ZIG_LOCAL_CACHE_DIR}" - fi - - mix local.hex --force - mix local.rebar --force - mix deps.get - - if [[ "${{ runner.os }}" == "Windows" ]]; then - # The zigler 0.16.0 Hex package only ships priv/beam, but on Windows - # zigler translate-c's priv/erl_nif_win/erl_nif_win.h. Zig reports the - # missing file as "translate-c failure ... CacheCheckFailed". - # Restore the headers from the commit tagged 0.16.0 (keep in sync - # with the zigler version in mix.lock). - zigler_sha=57813d3f571b1fadffe8d1f926f4fe3cd81747dc - grep -q '"zigler": {:hex, :zigler, "0.16.0"' mix.lock \ - || { echo "zigler is no longer 0.16.0; update zigler_sha" >&2; exit 1; } - win_dir=deps/zigler/priv/erl_nif_win - mkdir -p "${win_dir}" - for header in erl_nif_win.h erl_nif_api_funcs_win.h; do - curl -fsSL --retry 3 -o "${win_dir}/${header}" \ - "https://raw.githubusercontent.com/E-xyza/zigler/${zigler_sha}/priv/erl_nif_win/${header}" - done - ls -l "${win_dir}" - fi - - # Prove Elixir can see the pinned Zig (catches Git Bash path bugs) - mix run -e 'path = System.get_env("ZIG_EXECUTABLE_PATH"); unless path && File.exists?(path), do: raise("ZIG_EXECUTABLE_PATH not visible to Elixir: #{inspect(path)}")' - - # Prefer the pinned setup-zig binary; still run zig.get for Zigler bookkeeping - mix zig.get - - mix zigler_precompiled.build lib/ex_zarr/codecs/zig_codecs.ex \ - --target "${{ matrix.target }}" \ - --out artifacts - - - name: Upload build artifact - uses: actions/upload-artifact@v4 - with: - name: ${{ matrix.target }} - path: artifacts/*.tar.gz - if-no-files-found: error - - - name: Upload to release - if: startsWith(github.ref, 'refs/tags/') || inputs.release_tag != '' - uses: softprops/action-gh-release@v2 - with: - tag_name: ${{ inputs.release_tag || github.ref_name }} - files: artifacts/*.tar.gz diff --git a/CHANGELOG.md b/CHANGELOG.md index 6caec19..7939d01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,60 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added +- `compressor_config:` option for `ExZarr.create/1` (e.g. `[level: 9]`; Blosc + also `cname`, `shuffle`, `typesize`), validated per compressor by the new + `ExZarr.Codecs.CompressorConfig`. v2 arrays store the settings in `.zarray` + in numcodecs form and reuse them after reopening (including arrays created + by zarr-python); v3 arrays store them in the codec configuration. The + option was documented before but never implemented. +- `compressor: :gzip` for Zarr v3 arrays. + +### Fixed +- `config :ex_zarr, default_zarr_version` is now applied by `ExZarr.create/1` + and `ExZarr.Nx`. Unset, new arrays remain v2; `ExZarr.Version` previously + documented (and returned) 3 although nothing used it. +- `ExZarr.create/1` rejects unknown compressors, invalid compressor settings + and Zarr versions other than 2 or 3 with an error tuple. Previously an + unknown compressor failed on the first write and an invalid version raised. +- Zarr v3 arrays created with a compressor that has no v3 codec (`:snappy`, + custom codecs without `codecs:`) were silently stored uncompressed; they now + return `{:error, {:unsupported_codec_for_v3, compressor}}`. +- `:zlib` honours a compression `level`. +- v2 `.zarray` compressor levels now record the level ExZarr actually uses + (zlib 6, zstd 3) instead of a fixed 5. + +### Changed +- Compression codecs (zstd, lz4, snappy, blosc, bzip2, crc32c) now come from + [ExCodecs](https://hex.pm/packages/ex_codecs) `~> 0.2.4`: pure-Rust NIFs + with precompiled binaries for macOS, Linux (glibc and musl) and Windows. + ExZarr no longer contains native code; Zig, zigler, zigler_precompiled and + system compression libraries are no longer needed. +- `:lz4` writes the numcodecs `LZ4` format (4-byte little-endian size prefix) + and `:bzip2` writes plain bzip2 streams (numcodecs `BZ2`), so zarr-python can + read them. Data written by earlier ExZarr versions (8-byte size prefix) is + still read. +- v2 `.zarray` compressor metadata uses numcodecs configurations: `lz4` + (`acceleration`), `blosc` (`cname`, `clevel`, `shuffle`, `blocksize`) and + `bz2` (previously `bzip2`). The old `bzip2` id is still read. zarr-python can + now open ExZarr v2 arrays compressed with lz4, blosc and bzip2. +- The v3 `blosc` codec honours `cname`, `clevel`, `shuffle` and `typesize` from + its configuration; without them ExZarr keeps BloscLZ, byte shuffle, + typesize 1. +- Decompressed chunks are capped at 256 MiB (`config :ex_zarr, + max_chunk_bytes: ...`) to guard against decompression bombs. +- Minimum OTP is 26 (ExCodecs NIFs target NIF 2.17). +- `ExZarr.Codecs.compress/3` accepts options as a map as well as a keyword list. + +### Removed +- `ExZarr.Codecs.ZigCodecs`, `ExZarr.Codecs.CompressionConfig`, the + `fix_nif_rpaths` Mix task, the precompile workflow and + `docs/PRECOMPILATION.md` / `docs/COMPRESSION_SETUP.md`. +- `COMPRESSION_LIB_DIRS`, `HOMEBREW_PREFIX`, `BZIP2_STATIC_LIB`, `ZIG_PATH` and + `EX_ZARR_BUILD` no longer have any effect. + ## [1.2.0] - 2026-10-03 ### Zarr 3.1 Interoperability & Range-Aware Cloud I/O @@ -841,6 +895,7 @@ None - Full backward compatibility maintained with v2 arrays --- +[Unreleased]: https://github.com/AnimaLogica/ExZarr/compare/v1.2.0...HEAD [1.2.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.2.0 [1.1.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.1.0 [1.0.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.0.0 diff --git a/README.md b/README.md index 3ae0666..d5aba69 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ interop tested in CI against zarr-python 2.18 / 3.2 / 3.3 / 3.4. See - **Pipeline integrations** - Optional Flow, GenStage, and Broadway support for production pipelines - **Parallel chunk processing** - Automatic parallel I/O and decompression for large operations - **Chunking** along arbitrary dimensions for optimized I/O operations -- **Compression** - Erlang `:zlib` plus Zig NIF codecs (zstd, lz4, snappy, blosc, bzip2, crc32c) +- **Compression** - Erlang `:zlib` plus [ExCodecs](https://hex.pm/packages/ex_codecs) codecs (zstd, lz4, snappy, blosc, bzip2, crc32c), precompiled for macOS, Linux and Windows - **Flexible storage** backends (in-memory, filesystem, and zip archive) - **Custom storage backends** with plugin architecture for S3, databases, and more - **Hierarchical groups** for organizing multiple arrays @@ -322,14 +322,17 @@ ExZarr provides the following built-in compression options: - **`:none`** - No compression (fastest, largest size) - **`:zlib`** - Standard zlib compression (good balance of speed and compression) -- **`:crc32`c** - CRC32C checksum codec (RFC 3720 compatible with Python zarr) -- **`:zstd`** - Zstandard compression (Zig NIF implementation) -- **`:lz4`** - LZ4 compression (Zig NIF implementation) -- **`:snappy`** - Snappy compression (Zig NIF implementation) -- **`:blosc`** - Blosc meta-compressor (Zig NIF implementation) -- **`:bzip2`** - Bzip2 compression (Zig NIF implementation) - -The `:zlib` codec uses Erlang's built-in `:zlib` module for maximum reliability and compatibility. +- **`:crc32c`** - CRC32C checksum codec (Zarr v3 `crc32c`) +- **`:zstd`** - Zstandard compression +- **`:lz4`** - LZ4 compression (numcodecs `LZ4` format) +- **`:snappy`** - Snappy compression +- **`:blosc`** - Blosc meta-compressor (Blosc1 chunks, readable by numcodecs / zarr-python) +- **`:bzip2`** - Bzip2 compression (numcodecs `BZ2` format) + +`:zlib` uses Erlang's built-in `:zlib` module. The other codecs come from +[ExCodecs](https://hex.pm/packages/ex_codecs), pure-Rust NIFs that are +downloaded precompiled for macOS, Linux (glibc and musl) and Windows, so no +system libraries or compilers are needed. ### Custom Codecs @@ -655,9 +658,8 @@ assert_received {:mock_storage, :write_chunk, _} ExZarr uses: - **Erlang :zlib** for zlib/gzip compression -- **Zig NIFs** (`ExZarr.Codecs.ZigCodecs`) for zstd, lz4, snappy, blosc, bzip2, and crc32c — - Hex/Livebook installs use **precompiled** artifacts (Zig not required); see - [`PRECOMPILATION.md`](docs/PRECOMPILATION.md) +- **[ExCodecs](https://hex.pm/packages/ex_codecs)** for zstd, lz4, snappy, blosc, bzip2 and + crc32c (precompiled pure-Rust NIFs; ExZarr itself has no native code) - **GenServer** for array state management - **Lazy streams** (`Stream.resource/3`, `Task.async_stream/3`) for bounded-memory chunk I/O - **Optional pipeline modules** (`ExZarr.Flow`, `ExZarr.GenStage`, `ExZarr.Broadway`) for backpressure and fault tolerance @@ -669,23 +671,14 @@ ExZarr uses: ## Development -Requires **Elixir ~> 1.17** and **OTP 25+**. Codec NIFs are **precompiled** for -published Hex releases (no Zig needed). Local/source builds need **Zig 0.16.0** -(via zigler) and system compression libraries: - -```bash -# macOS -brew install zstd lz4 snappy c-blosc bzip2 - -# Ubuntu/Debian -sudo apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev -``` +Requires **Elixir ~> 1.17** and **OTP 26+**. No native toolchain or system +libraries are needed; the codec NIFs come precompiled with ExCodecs. ```bash # Install dependencies mix deps.get -# Compile (dev/test force a Zig source build; or EX_ZARR_BUILD=1) +# Compile mix compile # Run tests @@ -731,7 +724,6 @@ mix coveralls The project uses GitHub Actions for continuous integration. The CI pipeline: - Tests on Elixir 1.17–1.19, 1.20-rc, and OTP 26–28 (Ubuntu) -- Installs Zig 0.16.0 for codec NIF builds - Runs all test suites (unit, integration, property-based) - Performs code quality checks (Credo, Dialyzer, `mix format`) - Generates test coverage reports and documentation (`mix docs --warnings-as-errors`) @@ -882,7 +874,7 @@ See [ROADMAP.md](docs/ROADMAP.md) for the full release plan. **v1.2.0 (released 2026-10-03)** - Zarr 3.1 interoperability & range-aware cloud I/O: spec `sharding_indexed`, optional byte-range reads, version-aware cloud keys, -AzureSDK migration, precompiled Zig NIFs, Python fixture/CI matrix. +AzureSDK migration, Python fixture/CI matrix. **Upcoming** (high level): diff --git a/config/config.exs b/config/config.exs index 4323385..145c993 100644 --- a/config/config.exs +++ b/config/config.exs @@ -1,23 +1,4 @@ import Config -# Configure Ziggler to use the system Zig installation -if System.get_env("ZIG_PATH") do - config :zigler, - zig_path: System.get_env("ZIG_PATH") -end - -# Compression library paths -# By default, ExZarr auto-detects library paths based on your platform: -# - macOS: Uses Homebrew paths (/opt/homebrew or /usr/local) -# - Linux: Uses standard system library paths -# -# You can override the auto-detection with environment variables: -# -# export COMPRESSION_LIB_DIRS="/custom/path/lib:/another/path/lib" -# export HOMEBREW_PREFIX="/custom/homebrew" -# export BZIP2_STATIC_LIB="/custom/path/to/libbz2.a" -# -# These environment variables must be set before compilation (mix compile). - # Import environment-specific config import_config "#{config_env()}.exs" diff --git a/config/dev.exs b/config/dev.exs index 4c4924b..becde76 100644 --- a/config/dev.exs +++ b/config/dev.exs @@ -1,4 +1 @@ import Config - -# Development configuration — always build Zig NIFs from source locally -config :zigler_precompiled, :force_build, ex_zarr: true diff --git a/config/test.exs b/config/test.exs index 1e1a37b..becde76 100644 --- a/config/test.exs +++ b/config/test.exs @@ -1,4 +1 @@ import Config - -# Test configuration — always build Zig NIFs from source in CI / local tests -config :zigler_precompiled, :force_build, ex_zarr: true diff --git a/docs/COMPRESSION_SETUP.md b/docs/COMPRESSION_SETUP.md deleted file mode 100644 index 913ef70..0000000 --- a/docs/COMPRESSION_SETUP.md +++ /dev/null @@ -1,313 +0,0 @@ -# Compression Codecs Setup - -ExZarr includes Zig NIFs that bind to C compression libraries for high-performance compression. This guide explains how to set up and use these codecs. - -## Available Codecs - -### Compression Codecs - -- **:none** - No compression (always available) -- **:zlib** - Zlib compression via Erlang's built-in `:zlib` (always available) -- **:zstd** - Zstandard compression via libzstd (requires system library) -- **:lz4** - LZ4 compression via liblz4 (requires system library) -- **:snappy** - Snappy compression via libsnappy (requires system library) -- **:blosc** - Blosc meta-compressor via libblosc (requires system library) -- **:bzip2** - Bzip2 compression via libbz2 (requires system library) - -### Checksum Codecs - -- **:crc32c** - CRC32C checksum codec for data integrity (always available, pure Zig implementation) - -## Installation - -### macOS (Homebrew) - -```bash -brew install zstd lz4 snappy c-blosc bzip2 -``` - -### Ubuntu/Debian - -```bash -sudo apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev -``` - -### Fedora/RHEL - -```bash -sudo dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel -``` - -## Compilation - -### First-time Setup - -1. Install system libraries (see above) - -2. Install Zig for zigler: -```bash -mix zig.get -``` - -3. Compile the project: - -**macOS and Linux (Auto-detection):** -```bash -mix deps.get -mix compile -``` - -ExZarr automatically detects compression library paths based on your platform: -- **macOS**: Uses Homebrew paths (detects ARM vs Intel automatically) -- **Linux**: Uses standard system library paths - -**Custom Installation Paths:** - -If your compression libraries are installed in non-standard locations, set environment variables before compiling: - -```bash -# Custom library directories (colon-separated) -export COMPRESSION_LIB_DIRS="/custom/path/lib:/another/path/lib" - -# Custom Homebrew prefix (macOS only) -export HOMEBREW_PREFIX="/custom/homebrew" - -# Custom bzip2 static library path -export BZIP2_STATIC_LIB="/custom/path/to/libbz2.a" - -# Now compile -mix deps.get -mix compile -``` - -### Runtime Configuration (macOS) - -The Zig NIFs need to find the compression libraries at runtime. ExZarr automatically handles this in most cases. - -#### Automatic Configuration (Recommended) - -The compilation process automatically adds rpaths to the compiled NIF library using `install_name_tool`. This works for most standard Homebrew installations on both ARM and Intel Macs. - -Just compile and run: -```bash -mix compile -mix run -``` - -#### Manual Configuration (If Needed) - -If you installed libraries in non-standard locations, or if automatic rpaths fail, set the library path environment variable: - -**Option 1: Set for single session** -```bash -export DYLD_FALLBACK_LIBRARY_PATH="$(brew --prefix)/opt/zstd/lib:$(brew --prefix)/opt/lz4/lib:$(brew --prefix)/opt/snappy/lib:$(brew --prefix)/opt/c-blosc/lib:$(brew --prefix)/opt/bzip2/lib" -``` - -**Option 2: Add to shell profile** - -Add to your shell profile (`~/.zshrc` or `~/.bashrc`): - -```bash -# For ARM Macs -export DYLD_FALLBACK_LIBRARY_PATH="/opt/homebrew/opt/zstd/lib:/opt/homebrew/opt/lz4/lib:/opt/homebrew/opt/snappy/lib:/opt/homebrew/opt/c-blosc/lib:/opt/homebrew/opt/bzip2/lib:$DYLD_FALLBACK_LIBRARY_PATH" - -# For Intel Macs -export DYLD_FALLBACK_LIBRARY_PATH="/usr/local/opt/zstd/lib:/usr/local/opt/lz4/lib:/usr/local/opt/snappy/lib:/usr/local/opt/c-blosc/lib:/usr/local/opt/bzip2/lib:$DYLD_FALLBACK_LIBRARY_PATH" -``` - -Then reload your shell: -```bash -source ~/.zshrc # or ~/.bashrc -``` - -## Usage - -### Check Available Codecs - -```elixir -# Returns list of available codecs at runtime -ExZarr.Codecs.available_codecs() -# => [:none, :zlib, :zstd, :lz4, :snappy, :blosc, :bzip2] - -# Check if specific codec is available -ExZarr.Codecs.codec_available?(:zstd) -# => true or false -``` - -### Compress and Decompress - -```elixir -data = "Hello, World!" - -# Basic compression (uses default level) -{:ok, compressed} = ExZarr.Codecs.compress(data, :zstd) -{:ok, decompressed} = ExZarr.Codecs.decompress(compressed, :zstd) - -# With compression level -{:ok, compressed} = ExZarr.Codecs.compress(data, :zstd, level: 9) - -# Different codecs -{:ok, lz4_compressed} = ExZarr.Codecs.compress(data, :lz4) -{:ok, blosc_compressed} = ExZarr.Codecs.compress(data, :blosc, level: 5) -{:ok, bzip2_compressed} = ExZarr.Codecs.compress(data, :bzip2, level: 9) - -# CRC32C checksum (adds 4-byte checksum for data integrity) -{:ok, checksummed} = ExZarr.Codecs.compress(data, :crc32c) -{:ok, validated} = ExZarr.Codecs.decompress(checksummed, :crc32c) -# Will return {:error, {:checksum_validation_failed, :crc32c_checksum_mismatch}} if corrupted -``` - -### Use with Zarr Arrays - -```elixir -# Create array with compression -{:ok, array} = ExZarr.create( - shape: {1000, 1000}, - chunks: {100, 100}, - dtype: :float64, - compressor: :zstd, # Use Zstandard compression - storage: :filesystem, - path: "/data/my_array" -) -``` - -## Troubleshooting - -### NIFs Not Loading - -If you see errors like: -``` -Failed to load NIF library: 'symbol not found in flat namespace' -``` - -**Solution 1:** Set the library path environment variable (see Option 1 above) - -**Solution 2:** Recompile with `-Wl,-headerpad_max_install_names` (see Option 2 above) - -### Codec Not Available - -If `codec_available?/1` returns `false` for a codec: - -1. **Check system library is installed:** - ```bash - # macOS - brew list | grep -E "zstd|lz4|snappy|blosc|bzip2" - - # Linux - ldconfig -p | grep -E "zstd|lz4|snappy|blosc|bz2" - ``` - -2. **Verify library paths are set** (macOS): - ```bash - echo $DYLD_FALLBACK_LIBRARY_PATH - ``` - -3. **Recompile** with proper LDFLAGS and CPPFLAGS - -### Compression Fails - -If compression returns an error: - -1. **Check the codec is available:** - ```elixir - ExZarr.Codecs.codec_available?(:your_codec) - ``` - -2. **Try a simpler codec** to isolate the issue: - ```elixir - # ZLIB always works - {:ok, compressed} = ExZarr.Codecs.compress(data, :zlib) - ``` - -3. **Check library loading:** - ```bash - # macOS - check what libraries the NIF links to - otool -L _build/dev/lib/ex_zarr/priv/lib/Elixir.ExZarr.Codecs.ZigCodecs.so - ``` - -## Performance Comparison - -Typical compression ratios and speeds (on text data): - -| Codec | Compression Ratio | Speed | Best For | Overhead | -|---------|------------------|-----------|---------------------------|----------| -| none | 1.0x | Fastest | Already compressed data | 0 bytes | -| snappy | 2-3x | Very Fast | Real-time compression | Varies | -| lz4 | 2-3x | Very Fast | Real-time compression | Varies | -| zlib | 3-5x | Medium | General purpose (default) | Varies | -| zstd | 3-7x | Fast | Best ratio/speed tradeoff | Varies | -| blosc | 2-10x | Fast | Numerical/scientific data | Varies | -| bzip2 | 4-8x | Slow | Maximum compression | Varies | -| crc32c | 1.0x (no compression) | Very Fast | Data integrity checking | 4 bytes | - -**Recommendations:** -- Use `:zstd` for most cases - excellent compression with good speed -- Use `:crc32c` when you need data integrity validation without compression -- Combine codecs in a pipeline for both compression and integrity (e.g., zstd + crc32c) - -## Development Setup Script - -Create a file `setup_compression.sh`: - -```bash -#!/bin/bash -set -e - -echo "Setting up ExZarr compression codecs..." - -# Install system libraries -if [[ "$OSTYPE" == "darwin"* ]]; then - echo "Installing libraries via Homebrew..." - brew install zstd lz4 snappy c-blosc bzip2 - -elif [[ -f /etc/debian_version ]]; then - echo "Installing libraries via apt..." - sudo apt-get update - sudo apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - -elif [[ -f /etc/redhat-release ]]; then - echo "Installing libraries via dnf..." - sudo dnf install -y zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel -fi - -# Get Zig for zigler -echo "Installing Zig for zigler..." -mix zig.get - -# Compile -echo "Compiling ExZarr with compression codecs..." -mix deps.get -mix compile - -echo "✓ Setup complete!" -echo "" -echo "Test codecs with: mix run -e 'IO.inspect(ExZarr.Codecs.available_codecs())'" -``` - -Make it executable: -```bash -chmod +x setup_compression.sh -./setup_compression.sh -``` - -## Environment Variables Reference - -ExZarr supports the following environment variables for customization: - -| Variable | Purpose | Example | -|----------|---------|---------| -| `COMPRESSION_LIB_DIRS` | Custom library directories (colon-separated) | `/custom/lib:/another/lib` | -| `HOMEBREW_PREFIX` | Override Homebrew installation path (macOS) | `/custom/homebrew` | -| `BZIP2_STATIC_LIB` | Custom path to bzip2 static library | `/custom/libbz2.a` | -| `DYLD_FALLBACK_LIBRARY_PATH` | Runtime library search path (macOS) | See above | -| `ZIG_PATH` | Custom Zig installation path | `/custom/zig` | - -**Note**: `COMPRESSION_LIB_DIRS`, `HOMEBREW_PREFIX`, and `BZIP2_STATIC_LIB` must be set **before** running `mix compile`, as they affect compilation. `DYLD_FALLBACK_LIBRARY_PATH` is only needed at runtime if automatic rpaths don't work. - -## References - -- [Zigler Documentation](https://hexdocs.pm/zigler/) -- [ZSTD Documentation](http://facebook.github.io/zstd/) -- [LZ4 Documentation](https://github.com/lz4/lz4) -- [Snappy Documentation](https://github.com/google/snappy) -- [Blosc Documentation](https://www.blosc.org/) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 1a3db43..079f9db 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -1,279 +1,88 @@ # Configuration Guide -This guide explains how to configure ExZarr's compression libraries and other configurable options. +ExZarr needs no build-time configuration: it has no native code of its own, +and its compression codecs come from [ExCodecs](https://hex.pm/packages/ex_codecs), +which downloads precompiled NIFs for your platform. This guide lists the +runtime settings ExZarr reads. -## Overview +## Application settings -ExZarr automatically detects compression library paths based on your platform, but also supports environment variable overrides for custom installations. +Set these in `config/config.exs` (or `runtime.exs`): -## Compression Library Configuration - -### Automatic Detection - -By default, ExZarr automatically detects where compression libraries are installed: - -- **macOS**: Detects Homebrew installation automatically - - ARM Macs (Apple Silicon): `/opt/homebrew` - - Intel Macs: `/usr/local` - - Falls back to `brew --prefix` command if available - -- **Linux**: Uses standard system library paths - - `/usr/lib` - - `/usr/local/lib` - - `/usr/lib/x86_64-linux-gnu` (Debian/Ubuntu x64) - - `/usr/lib/aarch64-linux-gnu` (Debian/Ubuntu ARM) - -### Environment Variables - -You can override automatic detection using these environment variables: - -#### `COMPRESSION_LIB_DIRS` - -Specifies custom library directories as a colon-separated list. - -**When to use**: When compression libraries are installed in non-standard locations. - -**Example**: -```bash -export COMPRESSION_LIB_DIRS="/opt/custom/lib:/usr/local/custom/lib" -rm -rf _build -mix compile -``` - -**Important**: Must be set **before** compilation, as paths are embedded in the compiled NIF. - -#### `HOMEBREW_PREFIX` - -Overrides the Homebrew installation prefix (macOS only). - -**When to use**: When using a custom Homebrew installation location. - -**Example**: -```bash -export HOMEBREW_PREFIX="/custom/homebrew" -rm -rf _build -mix compile -``` - -#### `BZIP2_STATIC_LIB` - -Specifies the full path to the bzip2 static library file. - -**When to use**: When bzip2's static library is in a non-standard location. - -**Example**: -```bash -export BZIP2_STATIC_LIB="/custom/path/to/libbz2.a" -rm -rf _build -mix compile -``` - -#### `DYLD_FALLBACK_LIBRARY_PATH` (macOS Runtime) - -macOS-specific environment variable for runtime library loading. - -**When to use**: Only needed if automatic rpaths don't work (rare). - -**Example**: -```bash -export DYLD_FALLBACK_LIBRARY_PATH="/opt/homebrew/opt/zstd/lib:/opt/homebrew/opt/lz4/lib:/opt/homebrew/opt/snappy/lib:/opt/homebrew/opt/c-blosc/lib:/opt/homebrew/opt/bzip2/lib" -``` - -**Note**: This is a runtime variable, not a compile-time variable. - -## Zig Configuration - -#### `ZIG_PATH` - -Specifies a custom Zig installation path for zigler. - -**When to use**: When using a custom Zig installation instead of the cached version. - -**Example**: -```bash -export ZIG_PATH="/custom/zig/installation/bin/zig" -mix compile -``` - -## Configuration Check - -To verify your configuration, run: - -```bash -mix run -e "ExZarr.Codecs.CompressionConfig.library_dirs() |> IO.inspect(label: \"Library dirs\")" -``` - -This will show the detected library paths. - -## Common Scenarios - -### Scenario 1: Standard Homebrew Installation (macOS) - -No configuration needed! Just install libraries and compile: - -```bash -brew install zstd lz4 snappy c-blosc bzip2 -mix compile -``` - -### Scenario 2: Custom Library Location (Any Platform) - -```bash -# Set custom paths -export COMPRESSION_LIB_DIRS="/my/custom/libs:/another/path" - -# Clean and recompile -rm -rf _build -mix compile -``` - -### Scenario 3: Docker/Container with Custom Paths - -```dockerfile -FROM elixir:1.19 - -# Install libraries to custom location -RUN mkdir -p /opt/compression-libs -# ... install libraries to /opt/compression-libs ... - -# Set environment variable -ENV COMPRESSION_LIB_DIRS=/opt/compression-libs/lib - -# Compile application -WORKDIR /app -COPY . . -RUN mix deps.get && mix compile -``` - -### Scenario 4: Multiple Homebrew Installations (macOS) - -```bash -# Use specific Homebrew installation -export HOMEBREW_PREFIX="/opt/homebrew-custom" - -# Clean and recompile -rm -rf _build -mix compile -``` - -### Scenario 5: Intel Mac (x86_64) - -ExZarr automatically detects Intel Macs and uses `/usr/local` instead of `/opt/homebrew`. - -To verify: -```bash -mix run -e "ExZarr.Codecs.CompressionConfig.homebrew_prefix() |> IO.inspect()" +```elixir +config :ex_zarr, + default_zarr_version: 2, + max_chunk_bytes: 256 * 1024 * 1024, + range_read_concurrency: 4, + range_read_timeout: 60_000, + custom_codecs: [MyApp.MyCodec], + custom_storage_backends: [MyApp.MyBackend] ``` -### Scenario 6: Linux Package Managers - -Standard package manager installations work automatically: +| Setting | Default | Meaning | +|---------|---------|---------| +| `:default_zarr_version` | `2` | Zarr version for `ExZarr.create/1` and `ExZarr.Nx` when no `zarr_version:` option is given (`2` or `3`). | +| `:max_chunk_bytes` | `268_435_456` (256 MiB) | Largest decompressed chunk any codec may produce. Protects against decompression bombs; raise it if your chunks are bigger. Blosc chunks are additionally capped at 1 GiB by the Blosc format. | +| `:range_read_concurrency` | `4` | Parallel inner-chunk range requests per shard when reading sharded v3 arrays from range-capable storage. | +| `:range_read_timeout` | `60_000` | Timeout in milliseconds for each of those range requests. A timed-out read falls back to downloading the whole shard. | +| `:custom_codecs` | `[]` | Codec modules (implementing `ExZarr.Codecs.Codec`) registered at startup. See `ExZarr.Codecs.Registry`. | +| `:custom_storage_backends` | `[]` | Storage backend modules (implementing `ExZarr.Storage.Backend`) registered at startup. See the [custom storage backend guide](guides/custom_storage_backend.md). | -**Ubuntu/Debian**: -```bash -sudo apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev -mix compile -``` +## Compressor settings -**Fedora/RHEL**: -```bash -sudo dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel -mix compile -``` +Compressor settings are per array, not application config. Pass them to +`ExZarr.create/1`: -**Arch Linux**: -```bash -sudo pacman -S zstd lz4 snappy blosc bzip2 -mix compile +```elixir +ExZarr.create( + shape: {1000, 1000}, + chunks: {100, 100}, + compressor: :blosc, + compressor_config: [cname: :zstd, level: 5, shuffle: :bit, typesize: 8] +) ``` -## Troubleshooting - -### Libraries Not Found at Compile Time +See `ExZarr.Codecs.CompressorConfig` for the settings each compressor accepts. -**Symptom**: Compilation fails with "library not found" errors. +## Codec availability -**Solution**: Set `COMPRESSION_LIB_DIRS` to the correct paths: -```bash -# Find where libraries are installed -find /usr /opt -name "libzstd.*" 2>/dev/null +All built-in codecs are available on every platform ExCodecs ships binaries +for: macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64, glibc and +musl) and Windows (x86_64). Check at runtime with: -# Set the directory (without the filename) -export COMPRESSION_LIB_DIRS="/path/to/lib" -rm -rf _build && mix compile +```elixir +ExZarr.Codecs.available_codecs() +# => [:none, :zlib, :crc32c, :zstd, :lz4, :snappy, :blosc, :bzip2] ``` -### Libraries Not Found at Runtime (macOS) - -**Symptom**: Application crashes with "symbol not found" or "library not loaded" errors. +If only `:none` and `:zlib` are listed, the ExCodecs NIF did not load, +usually because the precompiled download failed during `mix deps.compile`. +Re-fetch with network access: -**Solution**: Set runtime library path: ```bash -export DYLD_FALLBACK_LIBRARY_PATH="$(brew --prefix)/opt/zstd/lib:$(brew --prefix)/opt/lz4/lib:$(brew --prefix)/opt/snappy/lib:$(brew --prefix)/opt/c-blosc/lib:$(brew --prefix)/opt/bzip2/lib" +mix deps.clean ex_codecs --build +mix deps.get && mix deps.compile ex_codecs ``` -Or add to your shell profile (`~/.zshrc` or `~/.bashrc`). +### Building ExCodecs from source -### Codec Shows as Unavailable - -**Symptom**: `ExZarr.Codecs.codec_available?(:codec_name)` returns `false`. - -**Steps to debug**: - -1. Check if library is installed: - ```bash - # macOS - brew list | grep codec_name - - # Linux - ldconfig -p | grep codec_name - ``` - -2. Check if library can be loaded: - ```bash - otool -L _build/dev/lib/ex_zarr/priv/lib/Elixir.ExZarr.Codecs.ZigCodecs.so # macOS - ldd _build/dev/lib/ex_zarr/priv/lib/Elixir.ExZarr.Codecs.ZigCodecs.so # Linux - ``` - -3. Recompile with correct paths: - ```bash - export COMPRESSION_LIB_DIRS="/correct/path" - rm -rf _build - mix compile - ``` - -## Configuration Module API - -The `ExZarr.Codecs.CompressionConfig` module provides functions to inspect configuration: +On a platform without a precompiled binary, build the NIF from source. This +needs a Rust toolchain: ```elixir -# Get library directories for compilation -ExZarr.Codecs.CompressionConfig.library_dirs() -# => ["/opt/homebrew/opt/zstd/lib", ...] - -# Get rpath directories (includes both ARM and Intel paths on macOS) -ExZarr.Codecs.CompressionConfig.rpath_dirs() -# => ["/opt/homebrew/opt/zstd/lib", ..., "/usr/local/opt/zstd/lib", ...] +# mix.exs +{:rustler, ">= 0.0.0", optional: true} -# Get Homebrew prefix -ExZarr.Codecs.CompressionConfig.homebrew_prefix() -# => "/opt/homebrew" - -# Get bzip2 static library path -ExZarr.Codecs.CompressionConfig.bzip2_static_lib() -# => "/opt/homebrew/opt/bzip2/lib/libbz2.a" +# config/config.exs +config :rustler_precompiled, :force_build, ex_codecs: true ``` -## Files Involved - -The configuration system consists of: +## Telemetry -- `lib/ex_zarr/codecs/compression_config.ex` - Configuration detection and management -- `lib/ex_zarr/codecs/zig_codecs.ex` - Uses configuration at compile time -- `lib/mix/tasks/fix_nif_rpaths.ex` - Uses configuration to add rpaths -- `config/config.exs` - Documents environment variables +Range reads emit `[:ex_zarr, :shard, :range_read]` events; see the +[telemetry guide](guides/telemetry.md). ## See Also -- [COMPRESSION_SETUP.md](COMPRESSION_SETUP.md) - Installation and setup guide -- [README.md](README.md) - General usage documentation +- [Compression and Codecs](guides/compression_codecs.md) +- [Installation](guides/installation.md) diff --git a/docs/PRECOMPILATION.md b/docs/PRECOMPILATION.md deleted file mode 100644 index 6adc5d8..0000000 --- a/docs/PRECOMPILATION.md +++ /dev/null @@ -1,65 +0,0 @@ -# Precompiled Zig NIFs - -ExZarr ships precompiled `ZigCodecs` NIFs via -[ZiglerPrecompiled](https://hex.pm/packages/zigler_precompiled) so Hex and -Livebook users do not need a Zig toolchain. - -## Supported targets - -- `x86_64-linux-gnu` -- `aarch64-linux-gnu` -- `aarch64-macos-none` -- `x86_64-windows-gnu` - -Artifacts are built **natively** on each OS (no cross-link against system -compression libs). They still **dynamically link** `zstd`, `lz4`, `snappy`, -`blosc`, and `bz2` at runtime — install those packages on the host. -Windows CI uses MSYS2 UCRT64 packages for the link step. - -## Force a source build - -```bash -EX_ZARR_BUILD=1 mix compile -``` - -Or in config: - -```elixir -config :zigler_precompiled, :force_build, ex_zarr: true -``` - -Dev/test configs already force a source build. If -`checksum-ExZarr.Codecs.ZigCodecs.exs` is missing, source build is also forced. - -Until checksums are published, `zigler` is a **required** dependency so -`Mix.install` / path Livebooks pull it (optional deps are not fetched for -dependencies). After Hex ships with checksums, `zigler` can be marked -`optional: true` again. - -## Release flow - -1. Tag: `git tag v1.2.0 && git push origin v1.2.0` -2. Wait for `.github/workflows/precompile.yml` to upload `.tar.gz` assets to the - GitHub Release -3. Generate checksums (from a machine that can download the assets): - - ```bash - EX_ZARR_BUILD=1 mix deps.get - mix zigler_precompiled.download ExZarr.Codecs.ZigCodecs --all --print - ``` - -4. Commit `checksum-ExZarr.Codecs.ZigCodecs.exs` -5. Publish to Hex (`mix hex.publish`) — the checksum file is included in - `package.files` - -## Local artifact (smoke) - -```bash -EX_ZARR_BUILD=1 mix zigler_precompiled.build \ - lib/ex_zarr/codecs/zig_codecs.ex \ - --target aarch64-macos-none \ - --out artifacts -``` - -Replace the target with your host triple from -`ZiglerPrecompiled.current_target_triple()`. diff --git a/docs/ZARR_V3_STATUS.md b/docs/ZARR_V3_STATUS.md index cfd78c6..561cb5d 100644 --- a/docs/ZARR_V3_STATUS.md +++ b/docs/ZARR_V3_STATUS.md @@ -12,7 +12,7 @@ against zarr-python **2.18.3** and **3.2.1 / 3.3.0 / 3.4.0** (fixtures in | Feature | Status | Notes | |---------|--------|-------| | **Zarr v3.1 core metadata** | Supported | Scalars, zero-length dims, dim names, extensions | -| **Unified Codec Pipeline** | Supported | Core codecs + Zig NIF compressors | +| **Unified Codec Pipeline** | Supported | Core codecs + ExCodecs compressors (precompiled NIFs) | | **v3 Metadata Format** | Supported | `zarr.json` | | **Chunk key encoding** | Supported | `default` and `v2`, `/` or `.` separator | | **Automatic Version Detection** | Supported | Filesystem and cloud backends (probes `zarr.json`, then `.zarray`) | diff --git a/docs/guides/compression_codecs.md b/docs/guides/compression_codecs.md index 0312c15..ee27c3d 100644 --- a/docs/guides/compression_codecs.md +++ b/docs/guides/compression_codecs.md @@ -127,11 +127,13 @@ Read Path: Reverse order (Storage → Decompression → ... → Array Data) **Configuration:** ```elixir compressor: :zlib -compressor_config: [level: 6] # 1-9, default 6 +compressor_config: [level: 6] # 0-9, default 6 ``` +In Zarr v3, `:zlib` arrays use the `gzip` codec (default level 5). + **Characteristics:** -- **Compression level:** 1 (fast) to 9 (best compression) +- **Compression level:** 0 (stored) to 9 (best compression) - **Compression speed:** ~10-50 MB/s - **Decompression speed:** ~100-300 MB/s - **Compression ratio:** 2-4× (typical) @@ -140,7 +142,7 @@ compressor_config: [level: 6] # 1-9, default 6 **Best for:** - Compatibility (works everywhere) - Reliable default choice -- Systems without Zig NIFs +- Systems where the ExCodecs NIF cannot load **Example:** ```elixir @@ -155,14 +157,17 @@ compressor_config: [level: 6] # 1-9, default 6 ) ``` -### gzip (Erlang Built-in) +### gzip (Erlang Built-in, Zarr v3) -**Implementation:** Erlang `:zlib` module with gzip headers +**Implementation:** Erlang `:zlib` module with gzip headers. This is the Zarr +v3 `gzip` codec; Zarr v2 arrays use `:zlib` instead (`compressor: :gzip` is +rejected for v2). **Configuration:** ```elixir -compressor: :gzip -compressor_config: [level: 6] +zarr_version: 3, +compressor: :gzip, +compressor_config: [level: 6] # 0-9, default 5 ``` **Characteristics:** @@ -174,9 +179,9 @@ compressor_config: [level: 6] - Compatibility with gzip tools - Archives that need CLI decompression -### zstd (Zig NIF, Requires libzstd) +### zstd -**Implementation:** Zig NIF calling libzstd +**Implementation:** ExCodecs (pure-Rust Zstandard), standard zstd frames **Configuration:** ```elixir @@ -209,36 +214,25 @@ compressor_config: [level: 3] # 1-22, default 3 ) ``` -**Availability:** -```bash -# macOS -brew install zstd - -# Ubuntu/Debian -sudo apt-get install libzstd-dev - -# Fedora/RHEL -sudo dnf install zstd-devel -``` - -Check at runtime: +**Availability:** included with ExZarr (precompiled ExCodecs NIF). Check at +runtime: ```elixir ExZarr.Codecs.codec_available?(:zstd) -# => true (if libzstd installed) +# => true ``` -### lz4 (Zig NIF, Requires liblz4) +### lz4 -**Implementation:** Zig NIF calling liblz4 +**Implementation:** ExCodecs (`lz4_flex`), LZ4 block with a 4-byte little-endian size prefix, the numcodecs `LZ4` format **Configuration:** ```elixir compressor: :lz4 -compressor_config: [level: 1] # 1-12, default 1 ``` +LZ4 takes no settings; `compressor_config` must be empty. + **Characteristics:** -- **Compression level:** 1 (fastest) to 12 (higher compression) - **Compression speed:** ~200-500 MB/s - **Decompression speed:** ~1-3 GB/s (extremely fast) - **Compression ratio:** 1.5-2.5× (typical) @@ -262,15 +256,18 @@ compressor_config: [level: 1] # 1-12, default 1 ) ``` -### snappy (Zig NIF, Requires libsnappy) +### snappy -**Implementation:** Zig NIF calling libsnappy +**Implementation:** ExCodecs (`snap`), raw Snappy blocks **Configuration:** ```elixir compressor: :snappy # No level parameter ``` +Zarr v2 only: there is no Zarr v3 snappy codec, so `compressor: :snappy` +with `zarr_version: 3` is rejected. + **Characteristics:** - **Compression level:** Fixed (no configuration) - **Compression speed:** ~200-500 MB/s @@ -294,27 +291,44 @@ compressor: :snappy # No level parameter ) ``` -### blosc (Zig NIF, Requires libblosc) +### blosc -**Implementation:** Zig NIF calling libblosc (meta-compressor) +**Implementation:** ExCodecs, Blosc1 chunks (the c-blosc 1.x / numcodecs / Zarr `blosc` format) -**Configuration:** +**Configuration (Zarr v3):** set the inner compressor, shuffle and typesize in +the `blosc` codec configuration, as zarr-python does: ```elixir -compressor: :blosc -compressor_config: [ - compressor: :zstd, # Internal compressor: :lz4, :zstd, :zlib - level: 5, - blocksize: 0, # 0 = automatic - shuffle: true # Built-in shuffle +codecs: [ + %{name: "bytes", configuration: %{endian: "little"}}, + %{ + name: "blosc", + configuration: %{ + cname: "zstd", # blosclz | lz4 | lz4hc | zlib | zstd + clevel: 5, # 0-9 + shuffle: "bitshuffle", # noshuffle | shuffle | bitshuffle + typesize: 8 # element size in bytes + } + } ] ``` +**Configuration (Zarr v2 and v3):** the same settings through +`compressor_config`, which ExZarr turns into the numcodecs `.zarray` entry +(v2) or the codec configuration above (v3): +```elixir +compressor: :blosc, +compressor_config: [cname: :zstd, level: 5, shuffle: :bit, typesize: 8] +``` + +Without settings, ExZarr uses BloscLZ, level 5, byte shuffle and typesize 1. +numcodecs' v2 `Blosc` entry has no `typesize`, so for v2 arrays a configured +`typesize` applies only until the array is reopened. + **Characteristics:** -- **Meta-compressor:** Uses lz4/zstd/zlib internally -- **SIMD acceleration:** Optimized for numerical arrays -- **Built-in shuffle:** Automatically reorders bytes -- **Multithreaded:** Can use multiple cores -- **Compression ratio:** Varies by internal compressor (2-6×) +- **Meta-compressor:** BloscLZ, LZ4, zlib or zstd inside each block +- **Shuffle filters:** byte or bit shuffle group bytes of equal significance +- **Compression ratio:** Varies by inner compressor and data (2-6× typical) +- **Interoperable:** Blosc1 chunks are read by numcodecs and zarr-python **Best for:** - Scientific/numerical data (float arrays) @@ -327,20 +341,19 @@ compressor_config: [ shape: {10000, 10000}, chunks: {1000, 1000}, dtype: :float64, - compressor: :blosc, - compressor_config: [ - compressor: :zstd, - level: 5, - shuffle: true # Optimize for typed arrays + zarr_version: 3, + codecs: [ + %{name: "bytes", configuration: %{endian: "little"}}, + %{name: "blosc", configuration: %{cname: "zstd", clevel: 5, shuffle: "shuffle", typesize: 8}} ], storage: :filesystem, path: "/data/scientific_array" ) ``` -### bzip2 (Zig NIF, Requires libbz2) +### bzip2 -**Implementation:** Zig NIF calling libbz2 +**Implementation:** ExCodecs (pure-Rust bzip2), plain bzip2 streams, the numcodecs `BZ2` format **Configuration:** ```elixir @@ -376,9 +389,9 @@ compressor_config: [level: 9] # 1-9, default 9 **Warning:** Use only for archival. Slow compression and decompression make it unsuitable for active workloads. -### crc32c (Zig NIF, Checksum) +### crc32c (Checksum) -**Implementation:** Zig NIF with CRC32C algorithm +**Implementation:** ExCodecs, appends a little-endian CRC32C (Zarr v3 `crc32c`) **Configuration:** ```elixir @@ -976,46 +989,6 @@ Uncompressed: 7.63 MB ## Troubleshooting Codec Issues -### Issue: Zig Codec Compilation Fails - -**Symptom:** -``` -** (Mix) Could not compile dependency :ex_zarr -... -error: unable to find library -lzstd -``` - -**Cause:** Missing system libraries (libzstd, liblz4, libsnappy, etc.) - -**Fix:** -Install required libraries for your platform: - -```bash -# macOS -brew install zstd lz4 snappy c-blosc bzip2 - -# Ubuntu/Debian -sudo apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - -# Fedora/RHEL -sudo dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel -``` - -Then recompile: -```bash -mix deps.clean ex_zarr --build -mix deps.get -mix compile -``` - -**Workaround:** Use `:zlib` (always available, no dependencies): -```elixir -{:ok, array} = ExZarr.create( - compressor: :zlib, # Erlang built-in, always works - # ... other options -) -``` - ### Issue: Codec Not Available at Runtime **Symptom:** @@ -1023,49 +996,28 @@ mix compile {:error, {:unsupported_codec, :zstd}} ``` -**Cause:** Zig NIF failed to load or system library missing at runtime. +**Cause:** the ExCodecs NIF did not load, usually because its precompiled +binary was not downloaded during `mix deps.compile` (offline build) or the +platform has no binary. **Diagnosis:** ```elixir -# Check which codecs are available ExZarr.Codecs.available_codecs() -# => [:zlib, :gzip] # Only Erlang built-ins +# => [:none, :zlib] # ExCodecs NIF not loaded -# Check specific codec -ExZarr.Codecs.codec_available?(:zstd) +ExCodecs.Native.nif_loaded?() # => false ``` -**Fix:** -1. Verify system libraries are installed -2. Check NIF file exists: - ```bash - ls -la _build/dev/lib/ex_zarr/priv/zig_codecs.so - ``` -3. Check library dependencies (Linux): - ```bash - ldd _build/dev/lib/ex_zarr/priv/zig_codecs.so - ``` - -**Workaround:** Fallback to zlib in application code: -```elixir -defmodule MyApp.ArrayFactory do - def create_array(opts) do - # Prefer zstd, fallback to zlib - compressor = if ExZarr.Codecs.codec_available?(:zstd) do - :zstd - else - :zlib - end - - ExZarr.create( - compressor: compressor, - # ... other options - ) - end -end +**Fix:** re-fetch with network access: +```bash +mix deps.clean ex_codecs --build +mix deps.get && mix deps.compile ex_codecs ``` +See the [Troubleshooting guide](troubleshooting.md#issue-codec-not-available-at-runtime) +for building from source on other platforms. + ### Issue: Decompression Fails on Existing Array **Symptom:** @@ -1073,27 +1025,16 @@ end {:error, :decompression_failed} ``` -**Cause:** Array created with codec not available on reading system. +**Causes:** +- The codec is unavailable on the reading system (see above). +- The chunk is larger than the decompression limit + (`{:decompression_failed, :output_limit_exceeded}`); raise it with + `config :ex_zarr, max_chunk_bytes: ...` (default 256 MiB). +- The chunk is corrupt. -**Example:** -```elixir -# Created on system A with zstd -{:ok, array} = ExZarr.create(compressor: :zstd, path: "/data/array") - -# Opened on system B without libzstd -{:ok, array} = ExZarr.open(path: "/data/array") -{:error, :decompression_failed} = ExZarr.Array.get_slice(...) -``` - -**Fix:** Install required codec library on reading system. - -**Prevention:** Use widely available codecs for portability: -```elixir -# Portable: zlib works everywhere -{:ok, array} = ExZarr.create(compressor: :zlib, ...) - -# Or document codec requirements -``` +**Prevention:** for arrays other tools must read, use codecs those tools +support. zstd, blosc, lz4, bzip2, zlib/gzip and crc32c are all readable by +zarr-python. ### Issue: Poor Compression Ratio diff --git a/docs/guides/contributing.md b/docs/guides/contributing.md index 5ee92e1..6fbeebb 100644 --- a/docs/guides/contributing.md +++ b/docs/guides/contributing.md @@ -25,9 +25,9 @@ Before you begin, ensure you have: # Elixir 1.19.5 (compiled with Erlang/OTP 28) ``` -- **Zig toolchain** (for codec development, optional) - - Zigler will automatically download Zig on first compile - - Or install manually: `brew install zig` (macOS), download from [ziglang.org](https://ziglang.org) (other platforms) +- No native toolchain is needed. Compression codecs come from + [ExCodecs](https://hex.pm/packages/ex_codecs), whose NIFs are downloaded + precompiled. Codec implementation work happens in that repository. - **Git** for version control ```bash @@ -61,7 +61,7 @@ git remote add upstream https://github.com/AnimaLogica/ExZarr.git # Install dependencies mix deps.get -# Compile (includes Zig NIFs - may take a few minutes on first run) +# Compile mix compile # Run tests to verify setup @@ -75,7 +75,7 @@ mix credo mix dialyzer ``` -**Note**: First compilation may take 5-10 minutes as Zigler downloads the Zig toolchain and compiles NIFs. Subsequent compilations are much faster. +**Note**: `mix deps.get` downloads the precompiled ExCodecs NIF for your platform; no native code is compiled. ### Development Workflow @@ -147,7 +147,7 @@ ExZarr/ │ │ ├── codecs/ │ │ │ ├── codec.ex # Codec behavior definition │ │ │ ├── registry.ex # Codec discovery and registration -│ │ │ ├── zig_codecs.ex # Zig NIF codec implementations +│ │ │ ├── sharding_indexed.ex # sharding_indexed codec (v3) │ │ │ ├── pipeline_v3.ex # v3 codec pipeline orchestration │ │ │ └── filters/ # v3 filter implementations │ │ ├── storage/ @@ -185,8 +185,6 @@ ExZarr/ │ ├── test_helper.ex │ ├── setup_python_tests.sh │ └── fixtures/ -├── native/ # Zig NIFs source code -│ └── codecs/ # Compression codec implementations ├── docs/guides/ # User documentation (markdown) ├── examples/ # Example scripts (runnable .exs files) ├── benchmarks/ # Performance benchmarks @@ -1271,6 +1269,6 @@ Thank you for contributing to ExZarr! Every contribution helps make the library - [Elixir Style Guide](https://github.com/christopheradams/elixir_style_guide) - [Writing Documentation](https://hexdocs.pm/elixir/writing-documentation.html) - [Zarr Specification](https://zarr-specs.readthedocs.io/) -- [Zigler Documentation](https://hexdocs.pm/zigler/) +- [ExCodecs Documentation](https://hexdocs.pm/ex_codecs/) For questions, open an issue on GitHub: [https://github.com/AnimaLogica/ExZarr/issues](https://github.com/AnimaLogica/ExZarr/issues) diff --git a/docs/guides/core_concepts.md b/docs/guides/core_concepts.md index 3c3f8f2..2eeb754 100644 --- a/docs/guides/core_concepts.md +++ b/docs/guides/core_concepts.md @@ -358,14 +358,14 @@ Codecs transform chunk data before storage. This includes compression (reduce si **Compression codecs** (reduce size): - `:zlib` - Standard compression (Erlang built-in, always available) - `:gzip` - Compatible with gzip tools (Erlang built-in) -- `:zstd` - Fast compression with excellent ratio (Zig NIF) -- `:lz4` - Very fast compression, lower ratio (Zig NIF) -- `:blosc` - Multi-threaded compression (Zig NIF) -- `:bzip2` - High compression ratio, slower (Zig NIF) -- `:snappy` - Fast compression, moderate ratio (Zig NIF) +- `:zstd` - Fast compression with excellent ratio +- `:lz4` - Very fast compression, lower ratio +- `:blosc` - Shuffle filters plus an inner compressor; strong on numeric data +- `:bzip2` - High compression ratio, slower +- `:snappy` - Fast compression, moderate ratio **Checksum codecs** (verify integrity): -- `:crc32c` - CRC32 checksum for error detection (Zig NIF) +- `:crc32c` - CRC32C checksum for error detection **Filter codecs** (transform before compression): - `Shuffle` - Reorder bytes by element position (improves compression for numeric data) @@ -460,8 +460,8 @@ ExZarr internally converts v2 configuration to v3 pipeline for consistent execut ### Codec Selection Guidance **Prioritize availability:** -- `:zlib` is always available (Erlang built-in) -- Other codecs require Zig NIFs (see [Installation Guide](../../README.md#installation)) +- `:zlib` uses Erlang's built-in `:zlib` +- The other codecs come from ExCodecs' precompiled NIFs and are available on all supported platforms **Compression vs speed trade-offs:** - **Fastest**: `:lz4` (good for hot data, frequent access) diff --git a/docs/guides/glossary.md b/docs/guides/glossary.md index 88ebd90..e29110b 100644 --- a/docs/guides/glossary.md +++ b/docs/guides/glossary.md @@ -121,7 +121,7 @@ NumPy's in-memory N-dimensional array type. ExZarr arrays are persistent, chunke ### numcodecs -Python package providing compression codecs for zarr-python. Includes Blosc, Zstd, LZ4, etc. ExZarr uses Zig NIFs for equivalent codec implementations. +Python package providing compression codecs for zarr-python. Includes Blosc, Zstd, LZ4, etc. ExZarr uses ExCodecs (precompiled pure-Rust NIFs) for equivalent codec implementations. ### Slicing @@ -165,7 +165,7 @@ Elixir's `[key: value]` syntax for function options. Used throughout ExZarr API. ### NIF (Native Implemented Function) -Erlang mechanism for calling native code (C, Zig, Rust) from BEAM. ExZarr uses Zig NIFs for high-performance compression codecs. NIFs run in the same OS process as BEAM (fast but can crash VM if buggy). +Erlang mechanism for calling native code (C, Zig, Rust) from BEAM. ExZarr's compression codecs run as NIFs from the ExCodecs package. NIFs run in the same OS process as BEAM (fast but can crash VM if buggy). ### Process @@ -187,9 +187,9 @@ Elixir pattern for parallel map operations with controlled concurrency. ExZarr u Elixir's immutable ordered collection with fixed size. ExZarr uses tuples for shapes (`{1000, 500}`), chunk coordinates (`{0, 2}`), and nested data representation. Access by index is O(1). -### Zigler +### ExCodecs -Elixir package for writing Zig NIFs. Handles compilation, type conversion, and resource management. ExZarr uses Zigler for codec implementations. Automatically downloads Zig toolchain on first compile. +Elixir package of compression and checksum codecs implemented as pure-Rust NIFs with precompiled binaries. ExZarr uses it for zstd, lz4, snappy, blosc, bzip2 and crc32c. ## Storage and I/O Terms @@ -350,12 +350,12 @@ Note: `<` indicates little-endian byte order (standard on most modern systems). |-------|-------------------|---------------|--------------| | Zlib | `numcodecs.Zlib` | `:zlib` | Always (Erlang built-in) | | Gzip | `numcodecs.GZip` | `:gzip` | Always (Erlang built-in) | -| Zstd | `numcodecs.Zstd` | `:zstd` | Requires Zig NIFs + libzstd | -| LZ4 | `numcodecs.LZ4` | `:lz4` | Requires Zig NIFs + liblz4 | -| Blosc | `numcodecs.Blosc` | `:blosc` | Requires Zig NIFs + libblosc | -| Snappy | `numcodecs.Snappy` | `:snappy` | Requires Zig NIFs + libsnappy | -| Bzip2 | `numcodecs.BZ2` | `:bzip2` | Requires Zig NIFs + libbz2 | -| CRC32C | `numcodecs.CRC32C` | `:crc32c` | Requires Zig NIFs | +| Zstd | `numcodecs.Zstd` | `:zstd` | Via ExCodecs | +| LZ4 | `numcodecs.LZ4` | `:lz4` | Via ExCodecs | +| Blosc | `numcodecs.Blosc` | `:blosc` | Via ExCodecs (Blosc1 chunks) | +| Snappy | none | `:snappy` | ExZarr-only; no numcodecs equivalent | +| Bzip2 | `numcodecs.BZ2` | `:bzip2` | Via ExCodecs | +| CRC32C | `numcodecs.CRC32C` | `:crc32c` | Via ExCodecs | | Shuffle | `numcodecs.Shuffle` | `ExZarr.Filters.Shuffle` | Built-in (v3) | | Delta | `numcodecs.Delta` | `ExZarr.Filters.Delta` | Built-in (v3) | diff --git a/docs/guides/installation.md b/docs/guides/installation.md index 8b057e9..a9cfe57 100644 --- a/docs/guides/installation.md +++ b/docs/guides/installation.md @@ -1,6 +1,6 @@ # Installation and Build Toolchain -This guide covers installing ExZarr, setting up the Zig toolchain for high-performance codecs, and troubleshooting common build issues. Follow these steps to get ExZarr running reliably on macOS, Linux, or in CI environments. +This guide covers installing ExZarr, checking that its compression codecs are available, and troubleshooting common build issues. Follow these steps to get ExZarr running reliably on macOS, Linux, or in CI environments. ## Prerequisites @@ -8,29 +8,28 @@ This guide covers installing ExZarr, setting up the Zig toolchain for high-perfo **Elixir and Erlang/OTP:** - Elixir 1.17 or later -- Erlang/OTP 25 or later +- Erlang/OTP 26 or later Check your versions: ```bash elixir --version -# Erlang/OTP 25 [erts-13.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] +# Erlang/OTP 26 [erts-14.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] # Elixir 1.17.0 (compiled with Erlang/OTP 26) ``` -These versions are required because ExZarr uses modern Elixir features and requires OTP 25+ for certain BEAM capabilities. +These versions are required because ExZarr uses modern Elixir features, and the precompiled codec NIFs (from ExCodecs) target NIF version 2.17, which needs OTP 26 or later. **Operating System Support:** - macOS (Intel and Apple Silicon) - Linux (Ubuntu/Debian, Fedora/RHEL, Arch) -- Windows (not officially tested, may work with WSL2) +- Windows (x86_64) -### Optional Dependencies - -**Zig Toolchain:** -Required only if you want high-performance codecs (zstd, lz4, snappy, blosc, bzip2). If you skip this, ExZarr will use Erlang's built-in zlib codec, which is reliable and works everywhere. +### Native Dependencies -**System Libraries for Codecs:** -Required for Zig NIFs to compile. See [Zig Toolchain Setup](#zig-toolchain-setup) section below. +No native toolchain or system libraries are needed. All codecs other than +`:zlib` and `:gzip` come from [ExCodecs](https://hex.pm/packages/ex_codecs), +whose pure-Rust NIFs are downloaded precompiled for your platform (macOS, +Linux glibc and musl, Windows x86_64) when you run `mix deps.get`. ## Mix Dependency Installation @@ -51,7 +50,7 @@ Then fetch dependencies: mix deps.get ``` -This installs ExZarr with all required dependencies (Jason for JSON, Zigler for Zig compilation). +This installs ExZarr with its required dependencies: Jason (JSON), Telemetry, and ExCodecs (compression codecs, precompiled NIFs). ### Optional Dependencies @@ -109,103 +108,23 @@ Compile the project: mix compile ``` -First compilation will: -1. Download Zigler (if not cached) -2. Download Zig compiler (if not cached, ~200MB) -3. Compile Zig NIFs (if system libraries present) -4. Compile Elixir modules - -This can take 2-5 minutes on first run. Subsequent compilations are much faster (seconds). - -## Zig Toolchain Setup - -Zig is used for high-performance compression codecs. If you skip this section, ExZarr will work fine using Erlang's built-in zlib codec. - -### Automatic Setup (Recommended) - -Zigler automatically downloads the Zig compiler when you run `mix compile`. No manual installation needed. - -The Zig compiler (~200MB) is cached in `~/.cache/zigler/` for reuse across projects. - -### Manual Zig Installation (Optional) - -You can pre-install Zig manually to avoid automatic downloads: - -**macOS:** -```bash -brew install zig -``` - -**Linux (from official binaries):** -```bash -# Download Zig 0.13.0 (or later) -wget https://ziglang.org/download/0.13.0/zig-linux-x86_64-0.13.0.tar.xz -tar xf zig-linux-x86_64-0.13.0.tar.xz -sudo mv zig-linux-x86_64-0.13.0 /usr/local/zig -export PATH=/usr/local/zig:$PATH - -# Verify -zig version -``` - -**Ubuntu/Debian (from apt):** -```bash -# Note: apt version may be older than required -sudo apt install zig -``` - -### System Libraries for Codecs - -For Zig NIFs to compile and link properly, install compression libraries: +No native code is compiled: ExCodecs downloads its precompiled NIF during +`mix deps.compile` and verifies it against the checksums shipped in the +package. The first build takes well under a minute. -**macOS:** -```bash -brew install zstd lz4 snappy c-blosc bzip2 -``` - -**Ubuntu/Debian:** -```bash -sudo apt-get update -sudo apt-get install -y \ - libzstd-dev \ - liblz4-dev \ - libsnappy-dev \ - libblosc-dev \ - libbz2-dev -``` - -**Fedora/RHEL/Rocky Linux:** -```bash -sudo dnf install -y \ - zstd-devel \ - lz4-devel \ - snappy-devel \ - blosc-devel \ - bzip2-devel -``` - -**Arch Linux:** -```bash -sudo pacman -S zstd lz4 snappy blosc bzip2 -``` - -### Verifying Codec Availability +## Verifying Codec Availability After compilation, check which codecs are available: ```elixir -# Start IEx iex -S mix -# Check available codecs iex> ExZarr.Codecs.available_codecs() -[:zlib, :gzip, :zstd, :lz4, :snappy, :blosc, :bzip2, :crc32c] - -# If Zig NIFs didn't compile, you'll see: -[:zlib, :gzip] # Only Erlang built-in codecs +[:none, :zlib, :crc32c, :zstd, :lz4, :snappy, :blosc, :bzip2] ``` -If you see only `:zlib` and `:gzip`, the Zig NIFs didn't compile. This is fine for basic usage, but you won't have access to faster compression algorithms. See [Build Troubleshooting](#build-troubleshooting) below. +If only `:none` and `:zlib` are listed, the ExCodecs NIF failed to load; see +[Codec Not Available](#issue-codec-not-available). ## Verifying Installation @@ -267,183 +186,37 @@ iex> Application.spec(:ex_zarr, :vsn) |> to_string() ## Build Troubleshooting -### Issue: Zig Compilation Fails - -**Symptom:** -``` -** (Mix) Could not compile dependency :ex_zarr -... -error: unable to find library -lzstd -``` - -**Cause:** Missing system libraries for compression codecs. - -**Fix:** -Install the required libraries for your platform (see [System Libraries for Codecs](#system-libraries-for-codecs) above). - -After installing libraries: -```bash -# Clean and recompile -mix deps.clean ex_zarr --build -mix deps.get -mix compile -``` - -**Workaround:** -If you can't install system libraries, ExZarr will work with Erlang's built-in zlib: - -```elixir -# Just use zlib - always available -{:ok, array} = ExZarr.create( - shape: {1000, 1000}, - chunks: {100, 100}, - dtype: :float64, - compressor: :zlib # No Zig NIFs required -) -``` - -### Issue: Mix Hangs During Compilation - -**Symptom:** -`mix compile` appears to hang with no output for several minutes. - -**Cause:** -Zigler is downloading the Zig compiler (~200MB) on first compilation. On slow networks, this takes time. - -**Diagnosis:** -Check if download is happening: -```bash -# In another terminal -ls -lh ~/.cache/zigler/ - -# Watch for growing files -watch -n 1 "du -sh ~/.cache/zigler/" -``` - -**Fix:** -Wait patiently. The download happens only once and is cached. - -**Alternative:** -Pre-install Zig manually (see [Manual Zig Installation](#manual-zig-installation-optional)) to avoid automatic download. - -### Issue: Compilation Timeout in CI - -**Symptom:** -CI build times out during `mix deps.compile`. - -**Cause:** -First compilation is slow (downloading Zig, compiling NIFs). Default CI timeouts may be too short. - -**Fix:** -Increase timeout in CI configuration: - -```yaml -# GitHub Actions -- name: Compile dependencies - run: mix deps.compile - timeout-minutes: 15 # Increase from default 5 -``` - -**Better solution - Cache Zigler:** -```yaml -# GitHub Actions -- name: Cache Zigler - uses: actions/cache@v3 - with: - path: ~/.cache/zigler - key: ${{ runner.os }}-zigler-${{ hashFiles('**/mix.lock') }} - -- name: Cache build - uses: actions/cache@v3 - with: - path: _build - key: ${{ runner.os }}-build-${{ hashFiles('**/mix.lock') }} -``` +### Issue: Codec Not Available -After caching, subsequent builds are fast (30-60 seconds). +**Symptom:** `ExZarr.Codecs.available_codecs()` lists only `:none` and `:zlib`. -### Issue: Architecture Mismatch (Apple Silicon) +**Cause:** the ExCodecs NIF did not load. Usually the precompiled download +failed (offline build, proxy) or the platform has no precompiled binary. -**Symptom on M1/M2 Mac:** -``` -dyld: Symbol not found: _lz4_compress -Architecture error -``` +**Solutions:** -**Cause:** -System libraries installed for wrong architecture (x86_64 instead of ARM64). +1. Re-fetch and rebuild the dependency with network access: + ```bash + mix deps.clean ex_codecs --build + mix deps.get && mix deps.compile ex_codecs + ``` +2. On a platform without a precompiled binary, build ExCodecs from source + (needs a Rust toolchain): + ```elixir + # mix.exs + {:rustler, ">= 0.0.0", optional: true} -**Diagnosis:** -```bash -# Check your architecture -uname -m -# Should output: arm64 - -# Check library architecture -file /opt/homebrew/lib/liblz4.dylib -# Should contain: arm64 -``` - -**Fix:** -Reinstall libraries for ARM64: -```bash -# Ensure using ARM64 Homebrew -which brew -# Should be: /opt/homebrew/bin/brew (not /usr/local/bin/brew) - -# Reinstall libraries -brew reinstall zstd lz4 snappy c-blosc bzip2 - -# Clean and recompile -mix deps.clean ex_zarr --build -mix compile -``` - -### Issue: ExZarr Compiles But Codec Not Available - -**Symptom:** -Compilation succeeds, but codec returns unavailable at runtime: - -```elixir -iex> ExZarr.Codecs.available_codecs() -[:zlib, :gzip] # Missing zstd, lz4, etc. -``` - -**Cause:** -NIF compiled but failed to load at runtime due to missing runtime libraries. - -**Diagnosis:** -```bash -# Check if NIF file exists -ls -la _build/dev/lib/ex_zarr/priv/*.so - -# Check NIF dependencies (Linux) -ldd _build/dev/lib/ex_zarr/priv/zig_codecs.so - -# Check NIF dependencies (macOS) -otool -L _build/dev/lib/ex_zarr/priv/zig_codecs.so -``` - -**Fix (Linux):** -```bash -# Ensure library path includes codec libraries -export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH - -# Or install libraries system-wide -sudo ldconfig -``` - -**Fix (macOS):** -```bash -# Usually Homebrew handles this, but if needed: -export DYLD_LIBRARY_PATH=/opt/homebrew/lib:$DYLD_LIBRARY_PATH -``` + # config/config.exs + config :rustler_precompiled, :force_build, ex_codecs: true + ``` +3. Meanwhile `:zlib` and `:gzip` keep working; they use Erlang's built-in + `:zlib`. ### Issue: Permission Denied Errors **Symptom:** ``` -** (File.Error) could not write to file "priv/zig_codecs.so": permission denied +** (File.Error) could not write to file "_build/dev/lib/ex_zarr/ebin/Elixir.ExZarr.beam": permission denied ``` **Cause:** @@ -467,7 +240,8 @@ Never run `mix` commands with `sudo`. If you need system-wide installation, use ### GitHub Actions Example -Complete workflow for testing ExZarr in CI: +Complete workflow for testing ExZarr in CI. No system packages or native +toolchains are needed. ```yaml name: CI @@ -487,7 +261,7 @@ jobs: otp: ['26', '27', '28'] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Elixir uses: erlef/setup-beam@v1 @@ -495,33 +269,13 @@ jobs: elixir-version: ${{ matrix.elixir }} otp-version: ${{ matrix.otp }} - - name: Install system dependencies - run: | - sudo apt-get update - sudo apt-get install -y \ - libzstd-dev \ - liblz4-dev \ - libsnappy-dev \ - libblosc-dev \ - libbz2-dev - - - name: Cache deps - uses: actions/cache@v3 - with: - path: deps - key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }} - - - name: Cache Zigler - uses: actions/cache@v3 + - name: Cache deps and build + uses: actions/cache@v4 with: - path: ~/.cache/zigler - key: ${{ runner.os }}-zigler-${{ hashFiles('**/mix.lock') }} - - - name: Cache build - uses: actions/cache@v3 - with: - path: _build - key: ${{ runner.os }}-build-${{ matrix.otp }}-${{ matrix.elixir }}-${{ hashFiles('**/mix.lock') }} + path: | + deps + _build + key: ${{ runner.os }}-mix-${{ matrix.otp }}-${{ matrix.elixir }}-${{ hashFiles('**/mix.lock') }} - name: Install dependencies run: mix deps.get @@ -541,50 +295,30 @@ jobs: ### Docker Example -Dockerfile for ExZarr application: +Dockerfile for an ExZarr application. The ExCodecs NIF is precompiled for +both glibc and musl (Alpine), so no compression packages are needed: ```dockerfile -FROM elixir:1.16-alpine AS builder +FROM elixir:1.18-alpine AS builder -# Install build dependencies -RUN apk add --no-cache \ - build-base \ - git \ - zstd-dev \ - lz4-dev \ - snappy-dev \ - bzip2-dev +RUN apk add --no-cache build-base git WORKDIR /app -# Install Hex and Rebar RUN mix local.hex --force && \ mix local.rebar --force -# Copy mix files COPY mix.exs mix.lock ./ RUN mix deps.get --only prod -# Copy source COPY . . -# Compile (includes Zig NIFs) -RUN mix deps.compile RUN MIX_ENV=prod mix compile -# Runtime stage -FROM elixir:1.16-alpine - -# Install runtime dependencies -RUN apk add --no-cache \ - zstd \ - lz4-libs \ - snappy \ - bzip2 +FROM elixir:1.18-alpine WORKDIR /app -# Copy compiled artifacts COPY --from=builder /app/_build/prod /app/_build/prod COPY --from=builder /app/deps /app/deps @@ -594,17 +328,14 @@ CMD ["iex", "-S", "mix"] ### GitLab CI Example ```yaml -image: elixir:1.16 +image: elixir:1.18 cache: paths: - deps/ - _build/ - - ~/.cache/zigler/ before_script: - - apt-get update - - apt-get install -y libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - mix local.hex --force - mix local.rebar --force - mix deps.get @@ -619,19 +350,14 @@ test: ### Caching Best Practices **Always cache:** -- `deps/` - Mix dependencies +- `deps/` - Mix dependencies (including the downloaded ExCodecs NIF) - `_build/` - Compiled artifacts -- `~/.cache/zigler/` - Zig compiler download **Cache keys should include:** - OS/platform - Elixir/OTP versions - `mix.lock` hash (invalidate on dependency changes) -**Typical cache savings:** -- Without cache: 5-10 minutes (downloading Zig, compiling NIFs) -- With cache: 30-90 seconds (only recompile changed files) - ## Next Steps Now that ExZarr is installed: diff --git a/docs/guides/overview.md b/docs/guides/overview.md index ef6d669..04c9f1f 100644 --- a/docs/guides/overview.md +++ b/docs/guides/overview.md @@ -1,6 +1,6 @@ # Overview and Philosophy -ExZarr is a pure Elixir implementation of the Zarr storage specification for compressed, chunked, N-dimensional arrays. This guide explains what ExZarr is, why it exists, and how its architecture leverages BEAM concurrency and Zig acceleration to provide high-performance array storage for scientific computing and machine learning workloads. +ExZarr is a pure Elixir implementation of the Zarr storage specification for compressed, chunked, N-dimensional arrays. This guide explains what ExZarr is, why it exists, and how its architecture leverages BEAM concurrency and precompiled native codecs to provide high-performance array storage for scientific computing and machine learning workloads. ## What is ExZarr @@ -81,16 +81,12 @@ Parallel S3 reads (10 concurrent): The BEAM's process model makes this parallelism natural to express and safe to execute. -## Zig Acceleration Explained +## Native Codecs -ExZarr uses Zig for performance-critical compression codecs while keeping coordination logic in Elixir. - -### Why Zig Over Alternatives - -- **Simpler C interop**: Zig can directly import C libraries (`@cImport`) without bindings -- **Zigler integration**: The Zigler library provides seamless Zig NIF compilation in Mix -- **Smaller footprint**: Zig produces compact binaries compared to alternative approaches -- **No runtime dependency**: Unlike Rustler (which requires Rust toolchain at compile time), Zig is more lightweight +ExZarr itself is pure Elixir. Compression runs in native code supplied by +[ExCodecs](https://hex.pm/packages/ex_codecs), a separate package of +pure-Rust NIFs that ships precompiled binaries, so installing ExZarr needs no +compiler, Zig or Rust toolchain, or system compression libraries. ### What Runs Where @@ -100,52 +96,32 @@ ExZarr uses Zig for performance-critical compression codecs while keeping coordi - Storage I/O coordination (backend selection, error handling) - Concurrent task scheduling (Task.async_stream orchestration) - Array lifecycle management (GenServer state) +- zlib and gzip, through Erlang's built-in `:zlib` -**Zig NIFs handle:** -- Compression: zstd, lz4, snappy, blosc, bzip2 -- Decompression: reverse operations -- CRC32C checksum calculation -- Direct calls to C libraries (libzstd, liblz4, etc.) - -This division keeps the hot path (compression/decompression) in native code while maintaining high-level logic in readable Elixir. +**ExCodecs NIFs handle:** +- zstd, lz4, snappy, blosc (Blosc1 chunks) and bzip2 +- CRC32C checksums -### Fallback Behavior +This keeps the hot path (compression and decompression) in native code while +the rest stays in readable Elixir. -ExZarr gracefully degrades when Zig NIFs are unavailable: - -```elixir -# Zig codecs fail to compile (missing system libraries) -# ExZarr falls back to Erlang's built-in :zlib +### Availability -{:ok, array} = ExZarr.create( - shape: {1000, 1000}, - chunks: {100, 100}, - dtype: :float64, - compressor: :zstd # Attempts Zig NIF first -) - -# If zstd NIF unavailable, uses :zlib automatically -# Or specify fallback explicitly: -{:ok, array} = ExZarr.create( - shape: {1000, 1000}, - chunks: {100, 100}, - dtype: :float64, - compressor: :zlib # Always available (Erlang built-in) -) -``` - -The Erlang `:zlib` module is always available and provides reliable compression without any native dependencies. This ensures ExZarr works even in environments where Zig compilation is not possible. +ExCodecs publishes binaries for macOS (Apple Silicon and Intel), Linux +(x86_64 and aarch64, glibc and musl) and Windows (x86_64). If its NIF cannot +load, ExZarr does not silently switch codecs: `ExZarr.Codecs.available_codecs/0` +lists only `:none` and `:zlib`, and compressing with another codec returns +`{:error, {:unsupported_codec, codec}}` or a compression error. `:zlib` and +`:gzip` always work. ### NIF Safety -Native code in the BEAM requires care to avoid crashing the VM. ExZarr's Zig NIFs follow safety practices: - -- **Memory management**: Uses BEAM allocator for all allocations -- **Error handling**: Returns `{:error, reason}` tuples rather than crashing -- **Bounded execution**: Compression operations are bounded by chunk size -- **No blocking**: NIFs release scheduler during long operations where possible +Native code in the BEAM must not crash or stall the VM. The ExCodecs NIFs: -The Zigler library handles much of this complexity automatically, generating safe NIF wrappers from Zig code. +- run on dirty CPU schedulers, so large chunks do not block normal schedulers +- return `{:error, reason}` tuples instead of crashing +- cap decompressed output (`config :ex_zarr, max_chunk_bytes: ...`, default + 256 MiB) to guard against decompression bombs ## Zarr in the Elixir Stack @@ -333,7 +309,7 @@ ExZarr brings Zarr array storage to the BEAM with: - **Pure Elixir implementation**: No Python dependencies, native BEAM integration - **BEAM concurrency**: True parallelism for multi-chunk operations without GIL constraints -- **Zig acceleration**: High-performance codecs via NIFs, graceful fallback to Erlang zlib +- **Native codecs**: zstd, lz4, snappy, blosc and bzip2 via precompiled ExCodecs NIFs; zlib and gzip via Erlang - **Ecosystem integration**: Complements Nx, integrates with Broadway/GenStage, cloud-native design - **Specification compliance**: Full Zarr v2 and v3 support with automatic format detection diff --git a/docs/guides/python_interop.md b/docs/guides/python_interop.md index ae8f805..a841f9e 100644 --- a/docs/guides/python_interop.md +++ b/docs/guides/python_interop.md @@ -581,11 +581,11 @@ Override with `dimension_separator` option: |--------|----------------|---------------------|---------------|-------| | **zlib** | Always (Erlang) | Always | ✓ Yes | **Recommended for compatibility** | | **gzip** | Always (Erlang) | Always | ✓ Yes | Same as zlib with gzip headers | -| **zstd** | Optional (Zig NIF) | Yes (numcodecs) | ✓ Yes | Requires libzstd on both sides | -| **lz4** | Optional (Zig NIF) | Yes (numcodecs) | ✓ Yes | Requires liblz4 on both sides | -| **blosc** | Optional (Zig NIF) | Yes (numcodecs) | ✓ Yes | Requires libblosc on both sides | -| **snappy** | Optional (Zig NIF) | Yes (numcodecs) | Partial | Less common in Python ecosystem | -| **bzip2** | Optional (Zig NIF) | Yes (numcodecs) | ✓ Yes | Requires libbz2 on both sides | +| **zstd** | Always (ExCodecs) | Yes (numcodecs) | ✓ Yes | v2 and v3 | +| **lz4** | Always (ExCodecs) | Yes (numcodecs) | ✓ Yes | v2 (`LZ4`); numcodecs size-prefixed block | +| **blosc** | Always (ExCodecs) | Yes (numcodecs) | ✓ Yes | v2 and v3; Blosc1 chunks; v3 `cname`/`shuffle`/`typesize` honoured | +| **snappy** | Always (ExCodecs) | No | ✗ No | No standard Zarr/numcodecs Snappy codec | +| **bzip2** | Always (ExCodecs) | Yes (numcodecs) | ✓ Yes | v2 (`BZ2`, id `"bz2"`) | | **Custom** | Via behavior | Via numcodecs | ✗ No | Requires identical codec implementation | ### Recommendation for Maximum Compatibility @@ -611,10 +611,10 @@ end **In Elixir:** ```elixir ExZarr.Codecs.codec_available?(:zstd) -# => true (if libzstd installed) +# => true ExZarr.Codecs.available_codecs() -# => [:zlib, :gzip, :zstd, :lz4, :blosc, :bzip2, :crc32c] +# => [:none, :zlib, :crc32c, :zstd, :lz4, :snappy, :blosc, :bzip2] ``` **In Python:** diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 37005fe..ec7d73a 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -16,187 +16,73 @@ This guide helps diagnose and fix common ExZarr issues. Problems are organized b ## Build and Installation Issues -### Issue: Zig compilation fails during `mix compile` +ExZarr has no native code of its own. Compression codecs other than zlib +and gzip come from [ExCodecs](https://hex.pm/packages/ex_codecs), which +downloads a precompiled NIF for your platform during `mix deps.compile`. No +Zig, Rust, or system compression libraries are involved. -**Symptom**: Error messages mentioning "zig", "NIF", or "zigler" during compilation. +### Issue: codec not available at runtime -**Cause**: Missing system libraries for optional codecs (zstd, lz4, snappy, blosc, bzip2). +**Symptom**: `ExZarr.Codecs.available_codecs()` lists only `[:none, :zlib]`, +or creating an array with `compressor: :zstd` (or lz4, snappy, blosc, bzip2, +crc32c) fails. **Diagnosis**: -```bash -# macOS: Check installed libraries -brew list | grep -E "zstd|lz4|snappy|blosc|bzip2" - -# Ubuntu/Debian: Check installed packages -dpkg -l | grep -E "zstd|lz4|snappy|blosc|bzip2" +```elixir +ExZarr.Codecs.available_codecs() +# Expected: [:none, :zlib, :crc32c, :zstd, :lz4, :snappy, :blosc, :bzip2] -# Arch Linux -pacman -Q | grep -E "zstd|lz4|snappy|blosc|bzip2" +ExCodecs.Native.nif_loaded?() +# false means the ExCodecs NIF did not load ``` +**Cause**: the precompiled NIF was not downloaded (offline build, blocked +proxy) or there is no binary for this platform. + **Fix**: ```bash -# macOS -brew install zstd lz4 snappy c-blosc bzip2 - -# Ubuntu/Debian -sudo apt-get update -sudo apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev +# Re-fetch with network access +mix deps.clean ex_codecs --build +mix deps.get +mix deps.compile ex_codecs +``` -# Arch Linux -sudo pacman -S zstd lz4 snappy blosc bzip2 +Binaries exist for macOS (Apple Silicon, Intel), Linux (x86_64, aarch64; +glibc and musl) and Windows (x86_64). On other platforms, build the NIF from +source (requires a Rust toolchain): -# Fedora/RHEL -sudo dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel +```elixir +# mix.exs +{:rustler, ">= 0.0.0", optional: true} -# After installing libraries, recompile -mix deps.clean ex_zarr --build -mix deps.get -mix compile +# config/config.exs +config :rustler_precompiled, :force_build, ex_codecs: true ``` -**Workaround**: If you can't install these libraries, ExZarr will still work with zlib-only compression (always available, no dependencies): +**Workaround**: `:zlib` (and `:gzip` for v3) always work: ```elixir -# Use zlib or gzip (no Zig NIFs needed) {:ok, array} = ExZarr.create( shape: {1000, 1000}, chunks: {100, 100}, dtype: :float64, - compressor: :zlib # Always available + compressor: :zlib ) ``` --- -### Issue: Mix hangs during compilation - -**Symptom**: `mix compile` stalls with no output for several minutes. - -**Cause**: Zigler is downloading the Zig toolchain (first-time only). This can take 5-10 minutes on slow connections. - -**Diagnosis**: - -```bash -# Check if Zig is being downloaded -ls -lh ~/.cache/zigler/ - -# Monitor network activity (macOS) -nettop -m tcp - -# Monitor network activity (Linux) -iftop -``` - -**Fix**: Wait patiently. The download happens once and is cached. - -**Alternative**: Pre-install Zig manually to skip automatic download: - -```bash -# macOS -brew install zig - -# Ubuntu (download from ziglang.org) -wget https://ziglang.org/download/0.11.0/zig-linux-x86_64-0.11.0.tar.xz -tar xf zig-linux-x86_64-0.11.0.tar.xz -sudo mv zig-linux-x86_64-0.11.0 /usr/local/zig -export PATH=$PATH:/usr/local/zig -``` - ---- - -### Issue: Architecture mismatch on Apple Silicon (M1/M2/M3) - -**Symptom**: Errors like "Architecture not supported", "dyld: Library not loaded", or "wrong architecture". - -**Cause**: System libraries installed for wrong architecture (x86_64 instead of ARM64). - -**Diagnosis**: - -```bash -# Check current architecture -uname -m # Should show "arm64" - -# Check library architecture -file /opt/homebrew/lib/libzstd.dylib # Should show "arm64" - -# If using Rosetta accidentally -arch # Should show "arm64", not "i386" -``` - -**Fix**: - -```bash -# Ensure using native Homebrew (not Rosetta) -which brew # Should be /opt/homebrew/bin/brew - -# Reinstall libraries for ARM64 -brew reinstall zstd lz4 snappy c-blosc bzip2 - -# Clean and recompile -mix deps.clean ex_zarr --build -mix compile -``` - ---- - -### Issue: ExZarr compiles but codec not available at runtime - -**Symptom**: `{:error, :codec_not_available}` when creating array, even after successful compilation. - -**Diagnosis**: - -```elixir -# Check which codecs are available -ExZarr.Codecs.available_codecs() -# Expected: [:zlib, :gzip, :zstd, :lz4, :snappy, :blosc, :bzip2, :crc32c] -# If missing codecs, NIF failed to load - -# Check NIF load status -:code.which(ExZarr.Codecs.ZigCodecs) -# Should return path to .beam file, not :non_existing -``` - -**Cause**: Runtime library paths not configured correctly. - -**Fix**: - -```bash -# Linux: Check LD_LIBRARY_PATH -echo $LD_LIBRARY_PATH -# Should include /usr/lib, /usr/local/lib - -export LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH - -# macOS: Check DYLD_LIBRARY_PATH (rarely needed) -# Homebrew libraries are usually found automatically - -# Verify libraries are loadable -ldd /path/to/_build/dev/lib/ex_zarr/priv/zig_codecs.so # Linux -otool -L /path/to/_build/dev/lib/ex_zarr/priv/zig_codecs.so # macOS -``` - ---- - -### Issue: Compilation fails with "zigler not found" +### Issue: precompiled NIF download fails in CI or behind a proxy -**Symptom**: Error message "Could not find zigler" during compilation. +**Symptom**: `mix deps.compile` fails while compiling `ex_codecs` with an +error about downloading a precompiled NIF. -**Cause**: Zigler dependency not installed. - -**Fix**: - -```bash -# Ensure dependencies are fetched -mix deps.get - -# If problem persists, clean and retry -mix deps.clean --all -mix deps.get -mix compile -``` +**Fix**: allow access to `github.com` release downloads, or cache `deps/` +and `_build/` between CI runs so the NIF is downloaded once. RustlerPrecompiled +also honours `HTTP_PROXY` / `HTTPS_PROXY`. As a last resort, build from source +as shown above. ## Runtime Errors (Codec/Storage) @@ -262,9 +148,9 @@ ExZarr.Codecs.available_codecs() |> Enum.member?(:zstd) # Or whatever codec is used ``` -**Cause 1**: Codec not available on reading system. +**Cause 1**: Codec not available on the reading system. -**Fix**: Install required codec library (see [Build and Installation Issues](#build-and-installation-issues)). +**Fix**: See [codec not available at runtime](#issue-codec-not-available-at-runtime). **Cause 2**: Corrupted chunk data. @@ -278,14 +164,13 @@ ExZarr.Codecs.available_codecs() **Fix**: Re-write corrupted chunk or restore from backup. -**Cause 3**: Codec version mismatch (rare). +**Cause 3**: The chunk is larger than the decompression limit. The error is +`{:decompression_failed, :output_limit_exceeded}`. -**Fix**: Ensure same codec library versions on read/write systems: +**Fix**: Raise the limit (default 256 MiB per chunk): -```bash -# Check version -dpkg -l | grep libzstd # Debian/Ubuntu -brew list --versions zstd # macOS +```elixir +config :ex_zarr, max_chunk_bytes: 1_073_741_824 ``` --- @@ -1260,7 +1145,8 @@ Avoid these common mistakes: aws_access_key_id: System.get_env("AWS_ACCESS_KEY_ID") ``` -- [ ] **Not installing Zig codecs then wondering why they're unavailable** +- [ ] **Building offline and wondering why codecs are unavailable** + - The ExCodecs NIF is downloaded during `mix deps.compile` - Check: `ExZarr.Codecs.available_codecs()` - [ ] **Expecting Python NumPy arrays when ExZarr returns nested tuples** @@ -1408,7 +1294,7 @@ Available codecs: [:zlib, :gzip] Most issues fall into these categories: -1. **Build/Installation**: Install system libraries for codecs +1. **Build/Installation**: Make sure the ExCodecs precompiled NIF downloaded 2. **Runtime Errors**: Check paths, permissions, codec availability 3. **Memory**: Reduce chunk size or parallelism, use streaming 4. **Cloud Storage**: Verify credentials, check regions, increase chunk size diff --git a/lib/ex_zarr.ex b/lib/ex_zarr.ex index 67979fe..721fe1d 100644 --- a/lib/ex_zarr.ex +++ b/lib/ex_zarr.ex @@ -127,7 +127,11 @@ defmodule ExZarr do - `:chunks` - Tuple specifying chunk dimensions (required). Must match shape dimensionality. - `:dtype` - Data type (default: `:float64`). One of: `:int8`, `:int16`, `:int32`, `:int64`, `:uint8`, `:uint16`, `:uint32`, `:uint64`, `:float32`, `:float64`. - - `:compressor` - Compression codec (default: `:zstd`). One of: `:none`, `:zlib`, `:zstd`, `:lz4`. + - `:compressor` - Compression codec (default: `:zstd`). One of: `:none`, `:zlib`, `:zstd`, + `:lz4`, `:snappy`, `:blosc`, `:bzip2`, `:crc32c`, or a registered custom codec. + - `:compressor_config` - Compressor settings, e.g. `[level: 9]` (default: `[]`). See + `ExZarr.Codecs.CompressorConfig` for the settings each compressor accepts. + - `:zarr_version` - `2` or `3` (default: `config :ex_zarr, default_zarr_version`, or `2`). - `:filters` - List of filter tuples to apply before compression (default: `nil`). Filters are transformation codecs that pre-process data to improve compression. Format: `[{:filter_id, [opt: value, ...]}]` @@ -162,6 +166,15 @@ defmodule ExZarr do fill_value: 255 ) + # zstd at a higher compression level + {:ok, array} = ExZarr.create( + shape: {1000, 1000}, + chunks: {100, 100}, + dtype: :float64, + compressor: :zstd, + compressor_config: [level: 9] + ) + # Create array with Delta filter for sequential data {:ok, array} = ExZarr.create( shape: {10000}, @@ -190,6 +203,10 @@ defmodule ExZarr do - `{:error, :chunks_required}` if chunks is missing - `{:error, :invalid_shape}` if shape is malformed - `{:error, :invalid_chunks}` if chunks is malformed or doesn't match shape + - `{:error, {:unsupported_codec, compressor}}` if the compressor is not registered + - `{:error, {:invalid_compressor_config, compressor, reason}}` if a compressor setting is + unknown or out of range + - `{:error, {:invalid_zarr_version, version}}` if the version is not 2 or 3 - `{:error, {:unknown_filter, filter_id}}` if a filter is not registered - `{:error, {:invalid_filter_config, filter_id, reason}}` if filter configuration is invalid """ diff --git a/lib/ex_zarr/array.ex b/lib/ex_zarr/array.ex index 2f63d72..27440a6 100644 --- a/lib/ex_zarr/array.ex +++ b/lib/ex_zarr/array.ex @@ -50,6 +50,7 @@ defmodule ExZarr.Array do use GenServer alias ExZarr.ChunkGrid.Irregular alias ExZarr.{Codecs, DataType, Indexing, Metadata, MetadataV3, Storage} + alias ExZarr.Codecs.CompressorConfig alias ExZarr.Codecs.PipelineV3 alias ExZarr.Codecs.Registry alias ExZarr.Codecs.ShardingIndexed @@ -106,6 +107,9 @@ defmodule ExZarr.Array do - `:chunks` - Tuple specifying chunk dimensions (required) - `:dtype` - Data type (default: `:float64`) - `:compressor` - Compression codec (default: `:zstd`) + - `:compressor_config` - Compressor settings, e.g. `[level: 9]` (see + `ExZarr.Codecs.CompressorConfig`) + - `:zarr_version` - `2` or `3` (default: `config :ex_zarr, default_zarr_version`, or `2`) - `:storage` - Storage backend (default: `:memory`) - `:path` - Path for filesystem storage - `:fill_value` - Fill value for uninitialized chunks (default: `0`) @@ -145,12 +149,11 @@ defmodule ExZarr.Array do def create(opts) do require Logger - # Get version from opts (default to 2 for backward compatibility) - version = Keyword.get(opts, :zarr_version, 2) enable_server = Keyword.get(opts, :enable_server, false) enable_cache = Keyword.get(opts, :enable_cache, false) - with {:ok, config} <- validate_config(opts), + with {:ok, version} <- zarr_version(opts), + {:ok, config} <- validate_config(opts, version), {:ok, storage} <- Storage.init(Map.put(config, :zarr_format, version)), {:ok, metadata} <- create_metadata(config, version), {:ok, chunks} <- io_chunk_shape(metadata, config.chunks) do @@ -1952,15 +1955,48 @@ defmodule ExZarr.Array do :ok end - defp validate_config(opts) do + # `zarr_version:` wins; otherwise `config :ex_zarr, default_zarr_version: ...` + # (default 2, for backward compatibility). + defp zarr_version(opts) do + case Keyword.get_lazy(opts, :zarr_version, &ExZarr.Version.default_version/0) do + version when version in [2, 3] -> {:ok, version} + other -> {:error, {:invalid_zarr_version, other}} + end + end + + # v2 compresses chunks through ExZarr.Codecs, so any registered codec works. + defp validate_compressor(compressor, 2, _opts) do + case Registry.get(compressor) do + {:ok, _} -> {:ok, compressor} + {:error, :not_found} -> {:error, {:unsupported_codec, compressor}} + end + end + + # v3 turns the compressor into a codec in zarr.json; it must have one rather + # than be dropped silently. An explicit `codecs:` list replaces it entirely. + defp validate_compressor(compressor, 3, opts) do + cond do + Keyword.has_key?(opts, :codecs) -> {:ok, compressor} + compressor == :none -> {:ok, compressor} + CompressorConfig.to_v3(compressor, []) != nil -> {:ok, compressor} + true -> {:error, {:unsupported_codec_for_v3, compressor}} + end + end + + defp validate_config(opts, version) do with {:ok, shape} <- validate_shape(opts[:shape]), - {:ok, chunks} <- validate_chunks(opts[:chunks], shape) do + {:ok, chunks} <- validate_chunks(opts[:chunks], shape), + {:ok, compressor} <- + validate_compressor(Keyword.get(opts, :compressor, :zstd), version, opts), + {:ok, compressor_config} <- + CompressorConfig.normalize(compressor, Keyword.get(opts, :compressor_config)) do # Base config with validated/default values base_config = %{ shape: shape, chunks: chunks, dtype: Keyword.get(opts, :dtype, :float64), - compressor: Keyword.get(opts, :compressor, :zstd), + compressor: compressor, + compressor_config: compressor_config, fill_value: Keyword.get(opts, :fill_value, 0), filters: Keyword.get(opts, :filters, nil), storage_type: Keyword.get(opts, :storage, :memory), @@ -1969,7 +2005,10 @@ defmodule ExZarr.Array do # Pass through all backend-specific options by merging with opts # Backend-specific keys like :array_id, :table_name, :bucket, etc. will be preserved - config = Enum.into(opts, base_config) + config = + opts + |> Keyword.delete(:compressor_config) + |> Enum.into(base_config) {:ok, config} end @@ -3742,7 +3781,7 @@ defmodule ExZarr.Array do 2 -> # v2 path: apply filters in forward order, then compress with {:ok, filtered} <- apply_filters_encode(chunk_data, array.metadata.filters) do - Codecs.compress(filtered, array.compressor) + Codecs.compress(filtered, array.compressor, array.metadata.compressor_config || []) end 3 -> diff --git a/lib/ex_zarr/codecs.ex b/lib/ex_zarr/codecs.ex index 1a4bfbb..f4607ba 100644 --- a/lib/ex_zarr/codecs.ex +++ b/lib/ex_zarr/codecs.ex @@ -6,46 +6,48 @@ defmodule ExZarr.Codecs do ## Built-in Codecs - Provides compression and decompression operations for chunk data using - the following codecs: - - - `:none` - No compression (fastest, but largest storage size) - - `:zlib` - Standard zlib compression using Erlang's built-in `:zlib` module - - `:zstd` - Zstandard compression using `ezstd` (optional dependency) - - `:lz4` - LZ4 compression using `nimble_lz4` (optional dependency) - - `:snappy` - Snappy compression using `snappyer` (optional dependency) - - `:blosc` - Blosc meta-compressor using Zig NIF (optional) - - `:bzip2` - Bzip2 compression using Zig NIF (optional) - - `:crc32c` - CRC32C checksum codec (bytes-to-bytes, adds 4-byte checksum) + - `:none` - No compression + - `:zlib` - zlib via Erlang's built-in `:zlib` + - `:zstd` - Zstandard frames + - `:lz4` - LZ4 block with a 4-byte little-endian size prefix (numcodecs `LZ4`) + - `:snappy` - Raw Snappy + - `:blosc` - Blosc1 chunks (c-blosc 1.x / numcodecs `Blosc` / Zarr `blosc`) + - `:bzip2` - bzip2 stream (numcodecs `BZ2`) + - `:crc32c` - Appends / verifies a little-endian CRC32C (Zarr v3 `crc32c`) + + Everything except `:none` and `:zlib` is provided by + [ExCodecs](https://hex.pm/packages/ex_codecs): pure-Rust NIFs that ship + precompiled for Linux (glibc and musl), macOS and Windows. No system + libraries or compilers are needed. ## Compression Performance - Different codecs offer different trade-offs between compression speed, - decompression speed, and compression ratio: - - - `:none` - Fastest (no CPU overhead), largest files - - `:lz4` - Very fast compression and decompression, moderate compression ratio - - `:snappy` - Very fast compression, good for real-time data - - `:zlib` - Balanced performance and compression ratio (default level 6) - - `:zstd` - Best compression ratio, fast decompression, configurable levels - - `:blosc` - Meta-compressor with SIMD acceleration, excellent for numerical data - - `:bzip2` - High compression ratio, slower speed - - `:crc32c` - Checksum codec for data integrity (not compression, adds 4-byte overhead) - - ## Optional Dependencies - - Some codecs require optional dependencies. To use them, add to your `mix.exs`: - - def deps do - [ - {:ex_zarr, "~> 1.2"}, - {:ezstd, "~> 1.1"}, # For :zstd - {:nimble_lz4, "~> 0.1.3"}, # For :lz4 - {:snappyer, "~> 1.2"} # For :snappy - ] - end + - `:none` - Fastest, largest + - `:lz4` / `:snappy` - Very fast, moderate ratio + - `:zlib` - Balanced + - `:zstd` - High ratio, fast decompression, configurable level + - `:blosc` - Shuffle filters plus a compressor; strong on numeric data + - `:bzip2` - High ratio, slow + - `:crc32c` - Integrity check, not compression (4 bytes overhead) + + ## Options + + `compress/3` accepts a keyword list (or map): + + - `:level` - compression level (`:zlib` 0-9, `:zstd` 1-22, `:blosc` 0-9, + `:bzip2` 1-9) + - `:cname`, `:shuffle`, `:typesize` - `:blosc` only; defaults `:blosclz`, + `:byte` and `1` + + Decompressed chunks are capped at 256 MiB as protection against + decompression bombs. Raise the cap for larger chunks: - Check codec availability at runtime with `codec_available?/1`. + config :ex_zarr, max_chunk_bytes: 1_073_741_824 + + ## Compatibility + + `:lz4` and `:bzip2` data written by ExZarr 1.1 (an 8-byte size prefix) is + still read; new data uses the numcodecs formats, so zarr-python can read it. ## Custom Codecs @@ -92,22 +94,10 @@ defmodule ExZarr.Codecs do # data == "hello" # Check codec availability - ExZarr.Codecs.codec_available?(:zlib) # => true (always) - ExZarr.Codecs.codec_available?(:zstd) # => true if libzstd is installed - ExZarr.Codecs.codec_available?(:blosc) # => true if libblosc is installed - ExZarr.Codecs.codec_available?(:crc32c) # => true (always available) - - ## Compatibility Notes - - - All codecs are compatible with zarr-python when using the same compression - - `:zlib` is always available (uses Erlang's built-in module) - - Other codecs require system libraries and compiled Zig NIFs - - If a codec is not available, compression/decompression will return an error - - Use `codec_available?/1` to check availability before use + ExZarr.Codecs.codec_available?(:zstd) # => true """ alias ExZarr.Codecs.Registry - alias ExZarr.Codecs.ZigCodecs # Changed from fixed atoms to allow custom codecs @type codec :: atom() @@ -124,8 +114,10 @@ defmodule ExZarr.Codecs do - `data` - Binary data to compress - `codec` - Compression codec (`:none`, `:zlib`, `:zstd`, `:lz4`, `:snappy`, `:blosc`, `:bzip2`, or `:crc32c`) - - `opts` - Optional keyword list with compression options: + - `opts` - Optional keyword list (or map) with compression options: - `:level` - Compression level (codec-specific, typically 1-9) + - `:cname`, `:shuffle`, `:typesize` - `:blosc` only (defaults `:blosclz`, + `:byte`, `1`) ## Examples @@ -153,6 +145,8 @@ defmodule ExZarr.Codecs do def compress(data, codec, opts \\ []) # First try to use custom codec from registry + def compress(data, codec, opts) when is_map(opts), do: compress(data, codec, Map.to_list(opts)) + def compress(data, codec, opts) when is_binary(data) and is_atom(codec) do case Registry.get(codec) do {:ok, builtin} when is_atom(builtin) -> @@ -187,96 +181,68 @@ defmodule ExZarr.Codecs do end end - # Built-in codec implementations + # Built-in codec implementations. zlib uses Erlang's :zlib; everything else + # goes through ExCodecs (pure-Rust NIFs, precompiled). defp compress_builtin(data, :none, _opts), do: {:ok, data} - defp compress_builtin(data, :zlib, _opts) when is_binary(data) do - ZigCodecs.zlib_compress(data) - rescue - e -> {:error, {:compression_failed, e}} - end - - defp compress_builtin(data, :zstd, opts) when is_binary(data) do - level = Keyword.get(opts, :level, 3) - - case ZigCodecs.zstd_compress(data, level) do - {:ok, compressed} -> {:ok, compressed} - compressed when is_binary(compressed) -> {:ok, compressed} - {:error, reason} -> {:error, {:compression_failed, reason}} + defp compress_builtin(data, :zlib, opts) do + case Keyword.get(opts, :level) do + nil -> {:ok, :zlib.compress(data)} + level -> {:ok, zlib_compress(data, level)} end rescue e -> {:error, {:compression_failed, e}} end - defp compress_builtin(data, :lz4, _opts) when is_binary(data) do - # LZ4 requires original size for decompression, so we prepend it (8 bytes) - original_size = byte_size(data) + defp compress_builtin(data, :zstd, opts), + do: ex_codecs_encode(:zstd, data, level: Keyword.get(opts, :level, 3)) - case ZigCodecs.lz4_compress(data) do - {:ok, compressed} -> - {:ok, <>} + # numcodecs LZ4 format: 4-byte little-endian size + LZ4 block. + defp compress_builtin(data, :lz4, _opts), do: ex_codecs_encode(:lz4, data, []) - compressed when is_binary(compressed) -> - {:ok, <>} + defp compress_builtin(data, :snappy, _opts), do: ex_codecs_encode(:snappy, data, []) - {:error, reason} -> - {:error, {:compression_failed, reason}} - end - rescue - e -> {:error, {:compression_failed, e}} + # Blosc1 chunk, readable by c-blosc 1.x / numcodecs / zarr-python. Defaults + # match ExZarr 1.1 (BloscLZ, byte shuffle, typesize 1). + defp compress_builtin(data, :blosc, opts) do + ex_codecs_encode(:blosc, data, + cname: Keyword.get(opts, :cname, :blosclz), + clevel: Keyword.get(opts, :level, 5), + shuffle: Keyword.get(opts, :shuffle, :byte), + typesize: Keyword.get(opts, :typesize, 1) + ) end - defp compress_builtin(data, :snappy, _opts) when is_binary(data) do - case ZigCodecs.snappy_compress(data) do - {:ok, compressed} -> {:ok, compressed} - compressed when is_binary(compressed) -> {:ok, compressed} - {:error, reason} -> {:error, {:compression_failed, reason}} - end - rescue - e -> {:error, {:compression_failed, e}} - end - - defp compress_builtin(data, :blosc, opts) when is_binary(data) do - level = Keyword.get(opts, :level, 5) + # Plain bzip2 stream (numcodecs BZ2 format). + defp compress_builtin(data, :bzip2, opts), + do: ex_codecs_encode(:bzip2, data, block_size: Keyword.get(opts, :level, 9)) - case ZigCodecs.blosc_compress(data, level) do - {:ok, compressed} -> {:ok, compressed} - compressed when is_binary(compressed) -> {:ok, compressed} - {:error, reason} -> {:error, {:compression_failed, reason}} + # Appends a little-endian CRC32C. + defp compress_builtin(data, :crc32c, _opts) do + case ExCodecs.encode(:crc32c, data) do + {:ok, checksummed} -> {:ok, checksummed} + {:error, error} -> {:error, {:checksum_failed, error.reason}} end - rescue - e -> {:error, {:compression_failed, e}} end - defp compress_builtin(data, :bzip2, opts) when is_binary(data) do - level = Keyword.get(opts, :level, 9) - # Bzip2 also requires original size, prepend it (8 bytes) - original_size = byte_size(data) - - case ZigCodecs.bzip2_compress(data, level) do - {:ok, compressed} -> - {:ok, <>} - - compressed when is_binary(compressed) -> - {:ok, <>} + defp zlib_compress(data, level) do + z = :zlib.open() - {:error, reason} -> - {:error, {:compression_failed, reason}} + try do + :ok = :zlib.deflateInit(z, level) + compressed = :zlib.deflate(z, data, :finish) + :ok = :zlib.deflateEnd(z) + IO.iodata_to_binary(compressed) + after + :zlib.close(z) end - rescue - e -> {:error, {:compression_failed, e}} end - defp compress_builtin(data, :crc32c, _opts) when is_binary(data) do - # CRC32C is a checksum codec, not compression - # Appends 4-byte CRC32C checksum to data - case ZigCodecs.crc32c_encode(data) do - {:ok, checksummed} -> {:ok, checksummed} - checksummed when is_binary(checksummed) -> {:ok, checksummed} - {:error, reason} -> {:error, {:checksum_failed, reason}} + defp ex_codecs_encode(codec, data, opts) do + case ExCodecs.encode(codec, data, opts) do + {:ok, compressed} -> {:ok, compressed} + {:error, error} -> {:error, {:compression_failed, error.reason}} end - rescue - e -> {:error, {:checksum_failed, e}} end @doc """ @@ -317,8 +283,8 @@ defmodule ExZarr.Codecs do ## Notes - For `:lz4` and `:bzip2`, the original size is stored in the first 8 bytes - of the compressed data (prepended during compression). + `:lz4` and `:bzip2` also read the 8-byte size-prefixed layout written by + ExZarr 1.1. """ @spec decompress(binary(), codec()) :: {:ok, binary()} | {:error, term()} @@ -360,100 +326,93 @@ defmodule ExZarr.Codecs do # Built-in codec implementations defp decompress_builtin(data, :none), do: {:ok, data} - defp decompress_builtin(data, :zlib) when is_binary(data) do - ZigCodecs.zlib_decompress(data) + defp decompress_builtin(data, :zlib) do + {:ok, :zlib.uncompress(data)} rescue e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :zstd) when is_binary(data) do - case ZigCodecs.zstd_decompress(data) do + defp decompress_builtin(data, :zstd), do: ex_codecs_decode(:zstd, data) + + defp decompress_builtin(data, :lz4) do + case ex_codecs_decode(:lz4, data) do {:ok, decompressed} -> {:ok, decompressed} - decompressed when is_binary(decompressed) -> {:ok, decompressed} - {:error, reason} -> {:error, {:decompression_failed, reason}} + error -> decode_legacy_lz4(data, error) end - rescue - e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :lz4) when is_binary(data) do - # Extract original size from first 8 bytes - case data do - <> -> - case ZigCodecs.lz4_decompress(compressed, original_size) do - {:ok, decompressed} -> {:ok, decompressed} - decompressed when is_binary(decompressed) -> {:ok, decompressed} - {:error, reason} -> {:error, {:decompression_failed, reason}} - end + defp decompress_builtin(data, :snappy), do: ex_codecs_decode(:snappy, data) + + # Reads Blosc1 (and Blosc2) chunks; the compressor settings are in the header. + defp decompress_builtin(data, :blosc), do: ex_codecs_decode(:blosc, data) - _ -> - {:error, {:decompression_failed, :invalid_lz4_format}} + defp decompress_builtin(data, :bzip2) do + case ex_codecs_decode(:bzip2, data) do + {:ok, decompressed} -> {:ok, decompressed} + error -> decode_legacy_bzip2(data, error) end - rescue - e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :snappy) when is_binary(data) do - case ZigCodecs.snappy_decompress(data) do - {:ok, decompressed} -> {:ok, decompressed} - decompressed when is_binary(decompressed) -> {:ok, decompressed} - {:error, reason} -> {:error, {:decompression_failed, reason}} + # Verifies and strips a trailing little-endian CRC32C. + defp decompress_builtin(data, :crc32c) do + case ExCodecs.decode(:crc32c, data) do + {:ok, validated} -> {:ok, validated} + {:error, error} -> {:error, {:checksum_validation_failed, crc32c_reason(error.reason)}} end - rescue - e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :blosc) when is_binary(data) do - case ZigCodecs.blosc_decompress(data) do - {:ok, decompressed} -> {:ok, decompressed} - decompressed when is_binary(decompressed) -> {:ok, decompressed} - {:error, reason} -> {:error, {:decompression_failed, reason}} + # ExZarr 1.1 wrote LZ4 as an 8-byte native-endian size followed by a raw LZ4 + # block. A valid block never starts with a zero token after a 4-byte size, so + # the standard decode fails on this layout and we retry with the 4-byte + # prefix numcodecs (and ExCodecs) use. + defp decode_legacy_lz4(<>, _error) + when size <= 0xFFFFFFFF do + case ex_codecs_decode(:lz4, <>) do + {:ok, decompressed} when byte_size(decompressed) == size -> {:ok, decompressed} + _ -> {:error, {:decompression_failed, :invalid_lz4_format}} end - rescue - e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :bzip2) when is_binary(data) do - # Extract original size from first 8 bytes - case data do - <> -> - case ZigCodecs.bzip2_decompress(compressed, original_size) do - {:ok, decompressed} -> {:ok, decompressed} - decompressed when is_binary(decompressed) -> {:ok, decompressed} - {:error, reason} -> {:error, {:decompression_failed, reason}} - end + defp decode_legacy_lz4(_data, error), do: error - _ -> - {:error, {:decompression_failed, :invalid_bzip2_format}} + # ExZarr 1.1 wrote bzip2 as an 8-byte native-endian size followed by the stream. + defp decode_legacy_bzip2(<> = data, _error) do + <<_::binary-size(8), stream::binary>> = data + + case ex_codecs_decode(:bzip2, stream) do + {:ok, decompressed} when byte_size(decompressed) == size -> {:ok, decompressed} + _ -> {:error, {:decompression_failed, :invalid_bzip2_format}} end - rescue - e -> {:error, {:decompression_failed, e}} end - defp decompress_builtin(data, :crc32c) when is_binary(data) do - # CRC32C is a checksum codec - # Validates and removes 4-byte CRC32C checksum from end of data - case ZigCodecs.crc32c_decode(data) do - {:ok, validated} -> {:ok, validated} - validated when is_binary(validated) -> {:ok, validated} - {:error, reason} -> {:error, {:checksum_validation_failed, reason}} + defp decode_legacy_bzip2(_data, error), do: error + + # Keep the error atoms ExZarr has always returned for CRC32C. + defp crc32c_reason(:truncated_input), do: :crc32c_invalid_data + defp crc32c_reason(:checksum_mismatch), do: :crc32c_checksum_mismatch + defp crc32c_reason(reason), do: reason + + defp ex_codecs_decode(codec, data) do + case ExCodecs.decode(codec, data, max_output_size: max_chunk_bytes()) do + {:ok, decompressed} -> {:ok, decompressed} + {:error, error} -> {:error, {:decompression_failed, error.reason}} end - rescue - e -> {:error, {:checksum_validation_failed, e}} end + # Upper bound on one decoded chunk, guarding against decompression bombs. + defp max_chunk_bytes, do: Application.get_env(:ex_zarr, :max_chunk_bytes, 256 * 1024 * 1024) + @doc """ Returns the list of available codecs. - This function checks which codecs are actually available at runtime. - `:none` and `:zlib` are always available. Other codecs require the - Zig NIFs to be compiled with the system libraries installed. + This function checks which codecs are available at runtime. `:none` and + `:zlib` are always available; the others are available whenever the + ExCodecs NIF is loaded. ## Examples ExZarr.Codecs.available_codecs() - # => [:none, :zlib, :zstd, :lz4, :snappy, :blosc, :bzip2] - # (if all system libraries are installed) + # => [:none, :zlib, :crc32c, :zstd, :lz4, :snappy, :blosc, :bzip2] ## Returns @@ -471,8 +430,7 @@ defmodule ExZarr.Codecs do Returns `true` if the codec can be used with `compress/3` and `decompress/2`, `false` otherwise. - This function actually tests if the codec can be used by checking if the - necessary functions are exported from the ZigCodecs module. + Custom codecs answer through their `available?/0` callback. ## Examples @@ -480,7 +438,7 @@ defmodule ExZarr.Codecs do # => true ExZarr.Codecs.codec_available?(:zstd) - # => true (if libzstd is installed and NIFs are compiled) + # => true ExZarr.Codecs.codec_available?(:unknown) # => false @@ -503,7 +461,7 @@ defmodule ExZarr.Codecs do true {:ok, :builtin_crc32c} -> - true + nif_codec_available?(:crc32c) {:ok, :builtin_zstd} -> nif_codec_available?(:zstd) @@ -533,62 +491,9 @@ defmodule ExZarr.Codecs do end end - # Private helper to check if a NIF codec is available - # Tests actual functionality instead of checking exports, since zigler - # NIFs may not show up correctly with function_exported? - defp nif_codec_available?(codec) do - test_data = "test" - - case codec do - :zstd -> - case ZigCodecs.zstd_compress(test_data, 1) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - :lz4 -> - case ZigCodecs.lz4_compress(test_data) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - :snappy -> - case ZigCodecs.snappy_compress(test_data) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - :blosc -> - case ZigCodecs.blosc_compress(test_data, 1) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - :bzip2 -> - case ZigCodecs.bzip2_compress(test_data, 1) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - :crc32c -> - # CRC32C is always available (pure Zig implementation) - case ZigCodecs.crc32c_encode(test_data) do - bin when is_binary(bin) -> true - {:ok, _} -> true - _ -> false - end - - _ -> - false - end - rescue - _ -> false - end + # The ExCodecs NIF ships precompiled for every supported target; a codec is + # unavailable only if the NIF failed to load. + defp nif_codec_available?(codec), do: codec in ExCodecs.available_codecs() # === Custom Codec Support === diff --git a/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigCodecs.zig b/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigCodecs.zig deleted file mode 100644 index 993b8d0..0000000 --- a/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigCodecs.zig +++ /dev/null @@ -1,366 +0,0 @@ -const std = @import("std"); -const beam = @import("beam"); - -// Import C libraries -const c_zstd = @cImport(@cInclude("zstd.h")); -const c_lz4 = @cImport({ - @cInclude("lz4.h"); - @cInclude("lz4hc.h"); -}); -const c_snappy = @cImport(@cInclude("snappy-c.h")); -const c_blosc = @cImport(@cInclude("blosc.h")); -const c_bz2 = @cImport(@cInclude("bzlib.h")); - -// CRC32C lookup table (Castagnoli polynomial 0x1EDC6F41) -const crc32c_table = [256]u32{ - 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB, - 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24, - 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384, - 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B, - 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35, - 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA, - 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A, - 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595, - 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957, - 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198, - 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38, - 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7, - 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789, - 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46, - 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6, - 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829, - 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93, - 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C, - 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC, - 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033, - 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D, - 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982, - 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622, - 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED, - 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F, - 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0, - 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540, - 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F, - 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1, - 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E, - 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E, - 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351, -}; - -/// Compresses data using ZSTD -pub fn zstd_compress(data: []const u8, level: i32) !beam.term { - const max_size = c_zstd.ZSTD_compressBound(data.len); - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - const size = c_zstd.ZSTD_compress( - buffer.ptr, - max_size, - data.ptr, - data.len, - level - ); - - if (c_zstd.ZSTD_isError(size) != 0) { - return beam.make(.{.@"error", .zstd_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, size); - @memcpy(result, buffer[0..size]); - - return beam.make(result, .{}); -} - -/// Decompresses ZSTD data -pub fn zstd_decompress(data: []const u8) !beam.term { - const content_size = c_zstd.ZSTD_getFrameContentSize(data.ptr, data.len); - - // Check for errors (ZSTD_CONTENTSIZE_ERROR or UNKNOWN are very large values) - if (content_size >= 0xFFFFFFFFFFFFFFFE) { - return beam.make(.{.@"error", .zstd_invalid_data}, .{}); - } - - const buffer = try beam.allocator.alloc(u8, content_size); - - const result_size = c_zstd.ZSTD_decompress( - buffer.ptr, - content_size, - data.ptr, - data.len - ); - - if (c_zstd.ZSTD_isError(result_size) != 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .zstd_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using LZ4 -pub fn lz4_compress(data: []const u8) !beam.term { - const max_size = c_lz4.LZ4_compressBound(@intCast(data.len)); - const buffer = try beam.allocator.alloc(u8, @intCast(max_size)); - defer beam.allocator.free(buffer); - - const size = c_lz4.LZ4_compress_default( - @ptrCast(data.ptr), - @ptrCast(buffer.ptr), - @intCast(data.len), - max_size - ); - - if (size <= 0) { - return beam.make(.{.@"error", .lz4_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, @intCast(size)); - @memcpy(result, buffer[0..@intCast(size)]); - - return beam.make(result, .{}); -} - -/// Decompresses LZ4 data -pub fn lz4_decompress(data: []const u8, original_size: usize) !beam.term { - const buffer = try beam.allocator.alloc(u8, original_size); - - const result_size = c_lz4.LZ4_decompress_safe( - @ptrCast(data.ptr), - @ptrCast(buffer.ptr), - @intCast(data.len), - @intCast(original_size) - ); - - if (result_size < 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .lz4_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Snappy -pub fn snappy_compress(data: []const u8) !beam.term { - const max_size = c_snappy.snappy_max_compressed_length(data.len); - - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - var output_length: usize = max_size; - const status = c_snappy.snappy_compress( - @ptrCast(data.ptr), - data.len, - @ptrCast(buffer.ptr), - &output_length - ); - - if (status != c_snappy.SNAPPY_OK) { - return beam.make(.{.@"error", .snappy_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, output_length); - @memcpy(result, buffer[0..output_length]); - - return beam.make(result, .{}); -} - -/// Decompresses Snappy data -pub fn snappy_decompress(data: []const u8) !beam.term { - var uncompressed_length: usize = undefined; - var status = c_snappy.snappy_uncompressed_length( - @ptrCast(data.ptr), - data.len, - &uncompressed_length - ); - - if (status != c_snappy.SNAPPY_OK) { - return beam.make(.{.@"error", .snappy_invalid_input}, .{}); - } - - const buffer = try beam.allocator.alloc(u8, uncompressed_length); - - status = c_snappy.snappy_uncompress( - @ptrCast(data.ptr), - data.len, - @ptrCast(buffer.ptr), - &uncompressed_length - ); - - if (status != c_snappy.SNAPPY_OK) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .snappy_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Blosc -pub fn blosc_compress(data: []const u8, level: i32) !beam.term { - c_blosc.blosc_init(); - - const max_size = data.len + c_blosc.BLOSC_MAX_OVERHEAD; - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - const size = c_blosc.blosc_compress( - level, - 1, - 1, - data.len, - data.ptr, - buffer.ptr, - max_size - ); - - if (size <= 0) { - return beam.make(.{.@"error", .blosc_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, @intCast(size)); - @memcpy(result, buffer[0..@intCast(size)]); - - return beam.make(result, .{}); -} - -/// Decompresses Blosc data -pub fn blosc_decompress(data: []const u8) !beam.term { - c_blosc.blosc_init(); - - var nbytes: usize = undefined; - var cbytes: usize = undefined; - var blocksize: usize = undefined; - c_blosc.blosc_cbuffer_sizes( - data.ptr, - &nbytes, - &cbytes, - &blocksize - ); - - const buffer = try beam.allocator.alloc(u8, nbytes); - - const size = c_blosc.blosc_decompress( - data.ptr, - buffer.ptr, - nbytes - ); - - if (size <= 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .blosc_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Bzip2 -pub fn bzip2_compress(data: []const u8, level: i32) !beam.term { - const max_size = data.len + (data.len / 100) + 600; - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - var dest_len: c_uint = @intCast(max_size); - const result = c_bz2.BZ2_bzBuffToBuffCompress( - @ptrCast(buffer.ptr), - &dest_len, - @ptrCast(@constCast(data.ptr)), - @intCast(data.len), - level, - 0, - 30 - ); - - if (result != c_bz2.BZ_OK) { - return beam.make(.{.@"error", .bzip2_compression_failed}, .{}); - } - - const output = try beam.allocator.alloc(u8, dest_len); - @memcpy(output, buffer[0..dest_len]); - - return beam.make(output, .{}); -} - -/// Decompresses Bzip2 data -pub fn bzip2_decompress(data: []const u8, original_size: usize) !beam.term { - const buffer = try beam.allocator.alloc(u8, original_size); - - var dest_len: c_uint = @intCast(original_size); - const result = c_bz2.BZ2_bzBuffToBuffDecompress( - @ptrCast(buffer.ptr), - &dest_len, - @ptrCast(@constCast(data.ptr)), - @intCast(data.len), - 0, - 0 - ); - - if (result != c_bz2.BZ_OK) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .bzip2_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Computes CRC32C checksum using Castagnoli polynomial -fn compute_crc32c(data: []const u8) u32 { - var crc: u32 = 0xFFFFFFFF; - for (data) |byte| { - const index = @as(u8, @truncate(crc ^ byte)); - crc = (crc >> 8) ^ crc32c_table[index]; - } - return crc ^ 0xFFFFFFFF; -} - -/// Encodes data with CRC32C checksum appended (bytes-to-bytes codec) -/// Appends 4-byte checksum in little-endian format to the end of data -pub fn crc32c_encode(data: []const u8) !beam.term { - // Calculate checksum - const checksum = compute_crc32c(data); - - // Allocate buffer for data + 4-byte checksum - const result = try beam.allocator.alloc(u8, data.len + 4); - - // Copy original data - @memcpy(result[0..data.len], data); - - // Append checksum as little-endian u32 - result[data.len] = @truncate(checksum); - result[data.len + 1] = @truncate(checksum >> 8); - result[data.len + 2] = @truncate(checksum >> 16); - result[data.len + 3] = @truncate(checksum >> 24); - - return beam.make(result, .{}); -} - -/// Decodes data with CRC32C checksum validation (bytes-to-bytes codec) -/// Validates and removes the 4-byte checksum from the end of data -pub fn crc32c_decode(data: []const u8) !beam.term { - // Need at least 4 bytes for checksum - if (data.len < 4) { - return beam.make(.{.@"error", .crc32c_invalid_data}, .{}); - } - - // Extract the stored checksum (little-endian) - const stored_checksum: u32 = - @as(u32, data[data.len - 4]) | - (@as(u32, data[data.len - 3]) << 8) | - (@as(u32, data[data.len - 2]) << 16) | - (@as(u32, data[data.len - 1]) << 24); - - // Extract inner data (everything except last 4 bytes) - const inner_data = data[0..data.len - 4]; - - // Compute checksum of inner data - const computed_checksum = compute_crc32c(inner_data); - - // Validate checksums match - if (stored_checksum != computed_checksum) { - return beam.make(.{.@"error", .crc32c_checksum_mismatch}, .{}); - } - - // Allocate and return validated data (without checksum) - const result = try beam.allocator.alloc(u8, inner_data.len); - @memcpy(result, inner_data); - - return beam.make(result, .{}); -} \ No newline at end of file diff --git a/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigNifs.zig b/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigNifs.zig deleted file mode 100644 index ed55672..0000000 --- a/lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigNifs.zig +++ /dev/null @@ -1,94 +0,0 @@ -// this code is autogenerated, do not check it into to your code repository - -// ref lib/ex_zarr/codecs/zig_nifs.ex:11 -const std = @import("std"); -const beam = @import("beam"); - -/// Compresses data using flate (deflate) algorithm from Zig standard library -pub fn flate_compress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - // Estimate maximum compressed size - const max_size = data.len + (data.len / 10) + 32; - const compressed_buffer = try allocator.alloc(u8, max_size); - defer allocator.free(compressed_buffer); - - var fbs = std.io.fixedBufferStream(compressed_buffer); - var compressor = std.compress.flate.compressor(fbs.writer(), .{}); - - _ = try compressor.write(data); - try compressor.close(); - - const compressed_len = fbs.pos; - - // Allocate result that BEAM will own - const result = try beam.allocator.alloc(u8, compressed_len); - @memcpy(result, compressed_buffer[0..compressed_len]); - - return result; -} - -/// Decompresses flate-compressed data -pub fn flate_decompress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - var fbs = std.io.fixedBufferStream(data); - var decompressor = std.compress.flate.decompressor(fbs.reader()); - - const decompressed = try decompressor.reader().readAllAlloc(allocator, std.math.maxInt(usize)); - defer allocator.free(decompressed); - - // Copy to BEAM-managed memory - const result = try beam.allocator.alloc(u8, decompressed.len); - @memcpy(result, decompressed); - - return result; -} - -/// Compresses data using ZSTD algorithm -pub fn zstd_compress(data: []const u8, level: i32) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - const max_size = std.compress.zstd.compressBound(data.len); - const compressed_buffer = try allocator.alloc(u8, max_size); - defer allocator.free(compressed_buffer); - - var fbs = std.io.fixedBufferStream(compressed_buffer); - var compressor = std.compress.zstd.compressor(fbs.writer(), .{ - .level = @intCast(level), - }); - - _ = try compressor.write(data); - try compressor.finish(); - - const compressed_len = fbs.pos; - - const result = try beam.allocator.alloc(u8, compressed_len); - @memcpy(result, compressed_buffer[0..compressed_len]); - - return result; -} - -/// Decompresses ZSTD-compressed data -pub fn zstd_decompress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - var fbs = std.io.fixedBufferStream(data); - var decompressor = std.compress.zstd.decompressor(fbs.reader(), .{}); - - const decompressed = try decompressor.reader().readAllAlloc(allocator, std.math.maxInt(usize)); - defer allocator.free(decompressed); - - const result = try beam.allocator.alloc(u8, decompressed.len); - @memcpy(result, decompressed); - - return result; -} diff --git a/lib/ex_zarr/codecs/compression_config.ex b/lib/ex_zarr/codecs/compression_config.ex deleted file mode 100644 index 5520efd..0000000 --- a/lib/ex_zarr/codecs/compression_config.ex +++ /dev/null @@ -1,235 +0,0 @@ -defmodule ExZarr.Codecs.CompressionConfig do - @moduledoc """ - Configuration helper for compression library paths. - - Automatically detects library paths based on the platform and supports - environment variable overrides for custom installations. - - ## Environment Variables - - You can override the automatic detection by setting these environment variables: - - - `COMPRESSION_LIB_DIRS` - Colon-separated list of library directories - - `COMPRESSION_INCLUDE_DIRS` - Colon-separated list of header include directories - - `HOMEBREW_PREFIX` - Override Homebrew installation path (macOS only) - - ## Examples - - # Get library directories for linking - ExZarr.Codecs.CompressionConfig.library_dirs() - # => ["/opt/homebrew/opt/zstd/lib", "/opt/homebrew/opt/lz4/lib", ...] - - # Get include directories for C headers (needed by Livebook.app / GUI PATH) - ExZarr.Codecs.CompressionConfig.include_dirs() - # => ["/opt/homebrew/opt/zstd/include", "/opt/homebrew/opt/lz4/include", ...] - - # Get library paths for runtime (rpaths) - ExZarr.Codecs.CompressionConfig.rpath_dirs() - # => ["/opt/homebrew/opt/zstd/lib", "/opt/homebrew/opt/lz4/lib", ...] - - # Get static library path - ExZarr.Codecs.CompressionConfig.bzip2_static_lib() - # => "/opt/homebrew/opt/bzip2/lib/libbz2.a" - - ## Platform Support - - - **macOS (ARM)**: Uses `/opt/homebrew` prefix by default - - **macOS (Intel)**: Uses `/usr/local` prefix by default - - **Linux**: Uses standard system library paths - - **Custom**: Set `COMPRESSION_LIB_DIRS` / `COMPRESSION_INCLUDE_DIRS` - """ - - @compression_libs [:zstd, :lz4, :snappy, :"c-blosc", :bzip2] - - @doc """ - Returns the list of library directories to use for compilation linking. - - This is used by the Zig NIF compiler to find compression libraries. - """ - def library_dirs do - case System.get_env("COMPRESSION_LIB_DIRS") do - nil -> detect_library_dirs() - paths -> split_path_list(paths) - end - end - - @doc """ - Returns the list of include directories for compression C headers. - - Required so Zig `@cImport` / translate-c can find Homebrew headers when - `pkg-config` is unavailable (common for Livebook.app with a minimal PATH). - """ - def include_dirs do - case System.get_env("COMPRESSION_INCLUDE_DIRS") do - nil -> detect_include_dirs() - paths -> split_path_list(paths) - end - end - - @doc """ - Returns the list of library directories to use for runtime rpaths. - - This is used by the Mix task to add rpaths to the compiled NIF. - Includes both ARM and Intel Homebrew paths on macOS for maximum compatibility. - """ - def rpath_dirs do - case System.get_env("COMPRESSION_LIB_DIRS") do - nil -> detect_rpath_dirs() - paths -> split_path_list(paths) - end - end - - @doc """ - Returns the path to the bzip2 static library. - - Bzip2 only provides a static library on many systems, so we need the full path. - """ - def bzip2_static_lib do - case System.get_env("BZIP2_STATIC_LIB") do - nil -> - prefix = homebrew_prefix() - "#{prefix}/opt/bzip2/lib/libbz2.a" - - path -> - path - end - end - - @doc """ - Returns the Homebrew prefix path. - - Detects automatically based on platform, or uses HOMEBREW_PREFIX env var. - """ - def homebrew_prefix do - case System.get_env("HOMEBREW_PREFIX") do - nil -> detect_homebrew_prefix() - prefix -> prefix - end - end - - # Private functions - - # Unix uses ':' ; Windows uses ';' so drive letters like C: are preserved. - defp split_path_list(paths) do - sep = - case :os.type() do - {:win32, _} -> ";" - _ -> ":" - end - - paths - |> String.split(sep) - |> Enum.map(&String.trim/1) - |> Enum.reject(&(&1 == "")) - end - - defp detect_library_dirs do - case :os.type() do - {:unix, :darwin} -> - # macOS - use Homebrew - prefix = detect_homebrew_prefix() - build_lib_paths(prefix) - - {:unix, _} -> - # Linux - use standard system paths - # The linker will find these automatically, but we can specify common locations - [ - "/usr/lib", - "/usr/local/lib", - "/usr/lib/x86_64-linux-gnu", - "/usr/lib/aarch64-linux-gnu" - ] - |> Enum.filter(&File.dir?/1) - - _ -> - [] - end - end - - defp detect_include_dirs do - case :os.type() do - {:unix, :darwin} -> - prefix = detect_homebrew_prefix() - build_include_paths(prefix) - - {:unix, _} -> - [ - "/usr/include", - "/usr/local/include" - ] - |> Enum.filter(&File.dir?/1) - - _ -> - [] - end - end - - defp detect_rpath_dirs do - case :os.type() do - {:unix, :darwin} -> - # macOS - include both ARM and Intel paths for compatibility - arm_paths = build_lib_paths("/opt/homebrew") - intel_paths = build_lib_paths("/usr/local") - arm_paths ++ intel_paths - - {:unix, _} -> - # Linux doesn't need rpaths - uses standard library paths - [] - - _ -> - [] - end - end - - defp detect_homebrew_prefix do - # Prefer env / known brew binaries so Livebook.app (minimal PATH) still works - brew_bins = [ - System.find_executable("brew"), - "/opt/homebrew/bin/brew", - "/usr/local/bin/brew" - ] - - brew_bins - |> Enum.find(&(&1 && File.regular?(&1))) - |> case do - nil -> - architecture_homebrew_prefix() - - brew -> - case System.cmd(brew, ["--prefix"], stderr_to_stdout: true) do - {prefix, 0} -> String.trim(prefix) - _ -> architecture_homebrew_prefix() - end - end - rescue - _ -> architecture_homebrew_prefix() - end - - defp architecture_homebrew_prefix do - case :erlang.system_info(:system_architecture) do - arch when is_list(arch) -> - arch_str = List.to_string(arch) - - if String.contains?(arch_str, "aarch64") or String.contains?(arch_str, "arm64") do - "/opt/homebrew" - else - "/usr/local" - end - - _ -> - "/opt/homebrew" - end - end - - defp build_lib_paths(prefix) do - Enum.map(@compression_libs, fn lib -> - "#{prefix}/opt/#{lib}/lib" - end) - end - - defp build_include_paths(prefix) do - @compression_libs - |> Enum.map(fn lib -> "#{prefix}/opt/#{lib}/include" end) - |> Enum.filter(&File.dir?/1) - end -end diff --git a/lib/ex_zarr/codecs/compressor_config.ex b/lib/ex_zarr/codecs/compressor_config.ex new file mode 100644 index 0000000..87bae60 --- /dev/null +++ b/lib/ex_zarr/codecs/compressor_config.ex @@ -0,0 +1,284 @@ +defmodule ExZarr.Codecs.CompressorConfig do + @moduledoc """ + Validates `compressor_config:` options and maps them to and from Zarr + metadata. + + `ExZarr.create/1` accepts a compressor and its settings: + + ExZarr.create(shape: {1000}, chunks: {100}, compressor: :zstd, compressor_config: [level: 9]) + + Supported settings per built-in compressor: + + | Compressor | Settings | + |------------|----------| + | `:zlib` | `level` 0-9 (default 6; 5 for the v3 `gzip` codec) | + | `:gzip` (Zarr v3 only) | `level` 0-9 (default 5) | + | `:zstd` | `level` 1-22 (default 3) | + | `:bzip2` | `level` 1-9 (default 9) | + | `:blosc` | `cname` (`:blosclz`, `:lz4`, `:lz4hc`, `:zlib`, `:zstd`; default `:blosclz`), `level` 0-9 (default 5; `clevel` also accepted), `shuffle` (`:none`, `:byte`, `:bit`; default `:byte`), `typesize` 1-255 (default 1) | + | `:lz4`, `:snappy`, `:crc32c`, `:none` | none | + + Settings for custom codecs are passed through to the codec unchanged. + + For Zarr v2 the settings are stored in `.zarray` in numcodecs form and read + back on open, so later writes use the same settings. numcodecs' `Blosc` has + no `typesize` field; a configured Blosc `typesize` applies to the creating + process only (reopened arrays use the default). For Zarr v3 the settings + become the compressor codec's `configuration` in `zarr.json`. + """ + + @blosc_cnames [:blosclz, :lz4, :lz4hc, :zlib, :zstd] + @blosc_shuffles [:none, :byte, :bit] + @no_settings [:none, :lz4, :snappy, :crc32c] + + @doc """ + Validates `config` (a keyword list or map) for `compressor`. + + Returns `{:ok, keyword}` with normalised keys and values, or + `{:error, {:invalid_compressor_config, compressor, reason}}`. + + ## Examples + + iex> ExZarr.Codecs.CompressorConfig.normalize(:zstd, level: 9) + {:ok, [level: 9]} + + iex> ExZarr.Codecs.CompressorConfig.normalize(:blosc, %{cname: "zstd", clevel: 7}) + {:ok, [cname: :zstd, level: 7]} + + iex> ExZarr.Codecs.CompressorConfig.normalize(:lz4, level: 1) + {:error, {:invalid_compressor_config, :lz4, {:unknown_option, :level}}} + """ + @spec normalize(atom(), keyword() | map() | nil) :: + {:ok, keyword()} | {:error, {:invalid_compressor_config, atom(), term()}} + def normalize(_compressor, nil), do: {:ok, []} + + def normalize(compressor, config) when is_map(config), + do: normalize(compressor, Map.to_list(config)) + + def normalize(compressor, config) when is_list(config) do + if Keyword.keyword?(config) do + config + |> Enum.reduce_while({:ok, []}, fn {key, value}, {:ok, acc} -> + case option(compressor, key, value) do + {:ok, {k, v}} -> {:cont, {:ok, Keyword.put(acc, k, v)}} + {:error, reason} -> {:halt, {:error, {:invalid_compressor_config, compressor, reason}}} + end + end) + |> case do + {:ok, opts} -> {:ok, Enum.reverse(opts)} + error -> error + end + else + {:error, {:invalid_compressor_config, compressor, :not_a_keyword_list}} + end + end + + def normalize(compressor, other), + do: {:error, {:invalid_compressor_config, compressor, {:not_a_keyword_list, other}}} + + # String keys (from JSON) become atoms only when they are known options. + defp option(compressor, key, value) when is_binary(key) do + case key do + k when k in ~w(level clevel cname shuffle typesize) -> + option(compressor, String.to_atom(k), value) + + _ -> + {:error, {:unknown_option, key}} + end + end + + defp option(compressor, :level, l) + when compressor in [:zlib, :gzip] and is_integer(l) and l in 0..9, + do: {:ok, {:level, l}} + + defp option(:zstd, :level, l) when is_integer(l) and l in 1..22, do: {:ok, {:level, l}} + defp option(:bzip2, :level, l) when is_integer(l) and l in 1..9, do: {:ok, {:level, l}} + + defp option(:blosc, key, l) when key in [:level, :clevel] and is_integer(l) and l in 0..9, + do: {:ok, {:level, l}} + + defp option(:blosc, :cname, cname) when is_binary(cname) do + case Enum.find(@blosc_cnames, &(Atom.to_string(&1) == cname)) do + nil -> {:error, {:invalid_value, :cname, cname}} + atom -> {:ok, {:cname, atom}} + end + end + + defp option(:blosc, :cname, cname) when cname in @blosc_cnames, do: {:ok, {:cname, cname}} + defp option(:blosc, :shuffle, s) when s in @blosc_shuffles, do: {:ok, {:shuffle, s}} + + defp option(:blosc, :typesize, t) when is_integer(t) and t in 1..255, + do: {:ok, {:typesize, t}} + + defp option(compressor, key, value) + when compressor in [:zlib, :gzip, :zstd, :bzip2, :blosc] and + key in [:level, :clevel, :cname, :shuffle, :typesize] do + if key in allowed(compressor), + do: {:error, {:invalid_value, key, value}}, + else: {:error, {:unknown_option, key}} + end + + defp option(compressor, key, _value) when compressor in [:zlib, :gzip, :zstd, :bzip2, :blosc], + do: {:error, {:unknown_option, key}} + + defp option(compressor, key, _value) when compressor in @no_settings, + do: {:error, {:unknown_option, key}} + + # Custom codec: pass through. + defp option(_compressor, key, value) when is_atom(key), do: {:ok, {key, value}} + defp option(_compressor, key, _value), do: {:error, {:unknown_option, key}} + + defp allowed(:blosc), do: [:level, :clevel, :cname, :shuffle, :typesize] + defp allowed(_), do: [:level] + + @doc """ + numcodecs compressor JSON (Zarr v2 `.zarray`) for `compressor` and its + normalised options. Unset values are the ones ExZarr uses by default. + + ## Examples + + iex> ExZarr.Codecs.CompressorConfig.to_numcodecs(:zstd, level: 7) + %{id: "zstd", level: 7} + + iex> ExZarr.Codecs.CompressorConfig.to_numcodecs(:blosc, cname: :lz4, shuffle: :bit) + %{id: "blosc", cname: "lz4", clevel: 5, shuffle: 2, blocksize: 0} + """ + @spec to_numcodecs(atom(), keyword()) :: map() | nil + def to_numcodecs(:none, _opts), do: nil + def to_numcodecs(:zlib, opts), do: %{id: "zlib", level: Keyword.get(opts, :level, 6)} + def to_numcodecs(:zstd, opts), do: %{id: "zstd", level: Keyword.get(opts, :level, 3)} + def to_numcodecs(:bzip2, opts), do: %{id: "bz2", level: Keyword.get(opts, :level, 9)} + def to_numcodecs(:lz4, _opts), do: %{id: "lz4", acceleration: 1} + + def to_numcodecs(:blosc, opts) do + %{ + id: "blosc", + cname: Atom.to_string(Keyword.get(opts, :cname, :blosclz)), + clevel: Keyword.get(opts, :level, 5), + shuffle: shuffle_to_int(Keyword.get(opts, :shuffle, :byte)), + blocksize: 0 + } + end + + def to_numcodecs(compressor, opts) when is_atom(compressor) do + opts + |> Enum.filter(fn {_k, v} -> is_integer(v) or is_binary(v) or is_atom(v) end) + |> Map.new(fn {k, v} -> + {k, if(is_atom(v) and not is_boolean(v), do: Atom.to_string(v), else: v)} + end) + |> Map.put(:id, Atom.to_string(compressor)) + end + + @doc """ + Reads a numcodecs compressor JSON map (atom or string keys) back into an + ExZarr compressor and options. + + ## Examples + + iex> ExZarr.Codecs.CompressorConfig.from_numcodecs(%{"id" => "bz2", "level" => 5}) + {:bzip2, [level: 5]} + + iex> ExZarr.Codecs.CompressorConfig.from_numcodecs(%{id: "blosc", cname: "zstd", clevel: 3, shuffle: 2, blocksize: 0}) + {:blosc, [cname: :zstd, level: 3, shuffle: :bit]} + """ + @spec from_numcodecs(map() | nil) :: {atom(), keyword()} + def from_numcodecs(nil), do: {:none, []} + + def from_numcodecs(json) when is_map(json) do + get = fn key -> Map.get(json, key, Map.get(json, Atom.to_string(key))) end + + case get.(:id) do + # numcodecs calls bzip2 "bz2"; ExZarr 1.1 wrote "bzip2". + id when id in ["bz2", "bzip2"] -> {:bzip2, level_opts(get.(:level), 1..9)} + "zlib" -> {:zlib, level_opts(get.(:level), 0..9)} + "zstd" -> {:zstd, level_opts(get.(:level), 1..22)} + "lz4" -> {:lz4, []} + "blosc" -> {:blosc, blosc_opts(get)} + id when is_binary(id) -> {String.to_atom(id), []} + _ -> {:none, []} + end + end + + def from_numcodecs(_), do: {:none, []} + + defp level_opts(level, range) when is_integer(level) do + if level in range, do: [level: level], else: [] + end + + defp level_opts(_level, _range), do: [] + + defp blosc_opts(get) do + [ + cname: blosc_cname(get.(:cname)), + level: blosc_level(get.(:clevel)), + shuffle: shuffle_from_int(get.(:shuffle)) + ] + |> Enum.reject(fn {_k, v} -> is_nil(v) end) + end + + defp blosc_cname(cname) when is_binary(cname) do + case option(:blosc, :cname, cname) do + {:ok, {:cname, atom}} -> atom + _ -> nil + end + end + + defp blosc_cname(_), do: nil + + defp blosc_level(level) when is_integer(level) and level in 0..9, do: level + defp blosc_level(_), do: nil + + @doc """ + Zarr v3 compressor codec configuration for `compressor` and its options. + + ## Examples + + iex> ExZarr.Codecs.CompressorConfig.to_v3(:zstd, level: 9) + %{name: "zstd", configuration: %{level: 9}} + + iex> ExZarr.Codecs.CompressorConfig.to_v3(:snappy, []) + nil + """ + @spec to_v3(atom(), keyword()) :: map() | nil + # zlib arrays use the v3 gzip codec. + def to_v3(compressor, opts) when compressor in [:zlib, :gzip], + do: %{name: "gzip", configuration: %{level: Keyword.get(opts, :level, 5)}} + + def to_v3(:zstd, opts), + do: %{name: "zstd", configuration: %{level: Keyword.get(opts, :level, 5)}} + + def to_v3(:lz4, _opts), do: %{name: "lz4", configuration: %{}} + + def to_v3(:bzip2, opts), + do: %{name: "bz2", configuration: Map.new(Keyword.take(opts, [:level]))} + + def to_v3(:crc32c, _opts), do: %{name: "crc32c", configuration: %{}} + + def to_v3(:blosc, opts) do + config = + Enum.reduce(opts, %{}, fn + {:cname, c}, acc -> Map.put(acc, :cname, Atom.to_string(c)) + {:level, l}, acc -> Map.put(acc, :clevel, l) + {:shuffle, s}, acc -> Map.put(acc, :shuffle, shuffle_to_v3(s)) + {:typesize, t}, acc -> Map.put(acc, :typesize, t) + _, acc -> acc + end) + + %{name: "blosc", configuration: config} + end + + def to_v3(_compressor, _opts), do: nil + + defp shuffle_to_int(:none), do: 0 + defp shuffle_to_int(:byte), do: 1 + defp shuffle_to_int(:bit), do: 2 + + defp shuffle_from_int(0), do: :none + defp shuffle_from_int(1), do: :byte + defp shuffle_from_int(2), do: :bit + defp shuffle_from_int(_), do: nil + + defp shuffle_to_v3(:none), do: "noshuffle" + defp shuffle_to_v3(:byte), do: "shuffle" + defp shuffle_to_v3(:bit), do: "bitshuffle" +end diff --git a/lib/ex_zarr/codecs/pipeline_v3.ex b/lib/ex_zarr/codecs/pipeline_v3.ex index 80d956b..696b574 100644 --- a/lib/ex_zarr/codecs/pipeline_v3.ex +++ b/lib/ex_zarr/codecs/pipeline_v3.ex @@ -44,6 +44,7 @@ defmodule ExZarr.Codecs.PipelineV3 do """ alias ExZarr.Codecs + alias ExZarr.Codecs.CompressorConfig @type codec_spec :: %{required(:name) => String.t(), optional(:configuration) => map()} @type codec_stage :: :array_to_array | :array_to_bytes | :bytes_to_bytes @@ -601,13 +602,13 @@ defmodule ExZarr.Codecs.PipelineV3 do Codecs.compress(data, :zstd, level: level) "blosc" -> - Codecs.compress(data, :blosc) + Codecs.compress(data, :blosc, blosc_options(config)) "lz4" -> Codecs.compress(data, :lz4) "bz2" -> - Codecs.compress(data, :bzip2) + Codecs.compress(data, :bzip2, config_options(config, [:level])) "crc32c" -> Codecs.compress(data, :crc32c) @@ -617,6 +618,42 @@ defmodule ExZarr.Codecs.PipelineV3 do end end + # Zarr v3 blosc configuration -> ExZarr.Codecs options. Missing keys keep + # the codec defaults. + defp blosc_options(config) do + config + |> config_options([:cname, :clevel, :shuffle, :typesize]) + |> Enum.flat_map(fn + {:cname, cname} when cname in ~w(blosclz lz4 lz4hc zlib zstd) -> + [cname: String.to_existing_atom(cname)] + + {:clevel, level} when is_integer(level) -> + [level: level] + + {:shuffle, shuffle} -> + case shuffle do + "noshuffle" -> [shuffle: :none] + "shuffle" -> [shuffle: :byte] + "bitshuffle" -> [shuffle: :bit] + _ -> [] + end + + {:typesize, typesize} when is_integer(typesize) -> + [typesize: typesize] + + _ -> + [] + end) + end + + # Reads atom- or string-keyed configuration values. + defp config_options(config, keys) do + for key <- keys, + value = Map.get(config, key, Map.get(config, Atom.to_string(key))), + value != nil, + do: {key, value} + end + @doc false defp apply_compression_decode(%{name: name}, data, _opts) do case name do @@ -651,6 +688,8 @@ defmodule ExZarr.Codecs.PipelineV3 do * `filters` - List of v2 filter tuples `{:filter_id, opts}` * `compressor` - v2 compressor atom (`:zlib`, `:zstd`, etc.) + * `compressor_config` - normalised compressor settings (see + `ExZarr.Codecs.CompressorConfig`), placed in the codec's configuration ## Returns @@ -663,8 +702,8 @@ defmodule ExZarr.Codecs.PipelineV3 do iex> length(codecs) 4 """ - @spec from_v2(list() | nil, atom()) :: [codec_spec()] - def from_v2(filters, compressor) do + @spec from_v2(list() | nil, atom(), keyword()) :: [codec_spec()] + def from_v2(filters, compressor, compressor_config \\ []) do # Convert filters to v3 array→array codecs filter_codecs = case filters do @@ -683,17 +722,11 @@ defmodule ExZarr.Codecs.PipelineV3 do # Required bytes codec bytes_codec = %{name: "bytes", configuration: %{}} - # Convert compressor to v3 bytes→bytes codec + # Convert compressor (and its compressor_config) to a v3 bytes→bytes codec compressor_codec = - case compressor do - :none -> [] - :zlib -> [%{name: "gzip", configuration: %{level: 5}}] - :zstd -> [%{name: "zstd", configuration: %{level: 5}}] - :lz4 -> [%{name: "lz4", configuration: %{}}] - :blosc -> [%{name: "blosc", configuration: %{}}] - :bzip2 -> [%{name: "bz2", configuration: %{}}] - :crc32c -> [%{name: "crc32c", configuration: %{}}] - _ -> [] + case CompressorConfig.to_v3(compressor, compressor_config) do + nil -> [] + codec -> [codec] end filter_codecs ++ [bytes_codec] ++ compressor_codec diff --git a/lib/ex_zarr/codecs/registry.ex b/lib/ex_zarr/codecs/registry.ex index 291baaf..90ad5f6 100644 --- a/lib/ex_zarr/codecs/registry.ex +++ b/lib/ex_zarr/codecs/registry.ex @@ -342,7 +342,6 @@ defmodule ExZarr.Codecs.Registry do :builtin_blosc, :builtin_bzip2 ] -> - # These will be checked through the old system until phase 2 check_builtin_available(atom) _module -> @@ -396,25 +395,18 @@ defmodule ExZarr.Codecs.Registry do ## Private Helpers - # Temporary helper for checking built-in codec availability (Phase 1) - defp check_builtin_available(:builtin_zstd) do - function_exported?(ExZarr.Codecs.ZigCodecs, :zstd_compress, 2) - end - - defp check_builtin_available(:builtin_lz4) do - function_exported?(ExZarr.Codecs.ZigCodecs, :lz4_compress, 1) - end - - defp check_builtin_available(:builtin_snappy) do - function_exported?(ExZarr.Codecs.ZigCodecs, :snappy_compress, 1) - end - - defp check_builtin_available(:builtin_blosc) do - function_exported?(ExZarr.Codecs.ZigCodecs, :blosc_compress, 2) - end - - defp check_builtin_available(:builtin_bzip2) do - function_exported?(ExZarr.Codecs.ZigCodecs, :bzip2_compress, 2) + # Built-in codecs other than :none and :zlib come from ExCodecs; they are + # unavailable only if its NIF failed to load. + @ex_codecs_names %{ + builtin_zstd: :zstd, + builtin_lz4: :lz4, + builtin_snappy: :snappy, + builtin_blosc: :blosc, + builtin_bzip2: :bzip2 + } + + defp check_builtin_available(builtin) when is_map_key(@ex_codecs_names, builtin) do + Map.fetch!(@ex_codecs_names, builtin) in ExCodecs.available_codecs() end # Temporary helper for built-in codec info (Phase 1) @@ -450,7 +442,7 @@ defmodule ExZarr.Codecs.Registry do name: "Zstandard", version: "1.0.0", type: :compression, - description: "Zstandard compression via Zig NIF" + description: "Zstandard compression via ExCodecs (pure-Rust NIF)" } end @@ -459,7 +451,7 @@ defmodule ExZarr.Codecs.Registry do name: "LZ4", version: "1.0.0", type: :compression, - description: "LZ4 compression via Zig NIF" + description: "LZ4 compression via ExCodecs (pure-Rust NIF)" } end @@ -468,7 +460,7 @@ defmodule ExZarr.Codecs.Registry do name: "Snappy", version: "1.0.0", type: :compression, - description: "Snappy compression via Zig NIF" + description: "Snappy compression via ExCodecs (pure-Rust NIF)" } end @@ -477,7 +469,7 @@ defmodule ExZarr.Codecs.Registry do name: "Blosc", version: "1.0.0", type: :compression, - description: "Blosc meta-compressor via Zig NIF" + description: "Blosc meta-compressor (Blosc1 chunks) via ExCodecs" } end @@ -486,7 +478,7 @@ defmodule ExZarr.Codecs.Registry do name: "Bzip2", version: "1.0.0", type: :compression, - description: "Bzip2 compression via Zig NIF" + description: "Bzip2 compression via ExCodecs (pure-Rust NIF)" } end end diff --git a/lib/ex_zarr/codecs/zig_codecs.ex b/lib/ex_zarr/codecs/zig_codecs.ex deleted file mode 100644 index 17b1408..0000000 --- a/lib/ex_zarr/codecs/zig_codecs.ex +++ /dev/null @@ -1,132 +0,0 @@ -defmodule ExZarr.Codecs.ZigCodecs do - @moduledoc """ - High-performance compression codec implementations using Zig NIFs. - - This module provides compression and decompression functions for various codecs - by binding to C libraries through Zig's C interop. - - ## Available Codecs - - - **ZLIB**: Erlang's built-in `:zlib` (always available) - - **ZSTD**: Zstandard via libzstd (requires system library) - - **LZ4**: LZ4 via liblz4 (requires system library) - - **Snappy**: Snappy via libsnappy (requires system library) - - **Blosc**: Blosc via libblosc (requires system library) - - **Bzip2**: Bzip2 via libbz2 (requires system library) - - **CRC32C**: Pure Zig checksum (always available when NIF loads) - - ## Precompiled NIFs - - Hex / Livebook installs download a precompiled NIF for your platform when - `checksum-ExZarr.Codecs.ZigCodecs.exs` is present (no Zig required). - - Force a local Zig build (developers / CI artifact builders): - - EX_ZARR_BUILD=1 mix compile - - Or: - - config :zigler_precompiled, :force_build, ex_zarr: true - - ## System Library Requirements - - Precompiled and source builds dynamically link compression libraries. - Install them for codecs other than zlib / crc32c: - - ### macOS - ```bash - brew install zstd lz4 snappy c-blosc bzip2 - ``` - - ### Ubuntu/Debian - ```bash - apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev - ``` - - ### Fedora/RHEL - ```bash - dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel - ``` - - See `docs/PRECOMPILATION.md` for releasing new precompiled artifacts. - """ - - alias ExZarr.Codecs.CompressionConfig - - @version Mix.Project.config()[:version] - - @checksum_file Path.expand( - "../../../checksum-ExZarr.Codecs.ZigCodecs.exs", - __DIR__ - ) - - # Homebrew / COMPRESSION_*_DIRS / platform defaults (see CompressionConfig) - @library_dirs CompressionConfig.library_dirs() - @include_dirs CompressionConfig.include_dirs() - - @force_build System.get_env("EX_ZARR_BUILD") in ["1", "true"] or - not File.exists?(@checksum_file) or - Application.compile_env(:zigler_precompiled, [:force_build, :ex_zarr], false) == - true - - use ZiglerPrecompiled, - otp_app: :ex_zarr, - base_url: "https://github.com/AnimaLogica/ExZarr/releases/download/v#{@version}", - version: @version, - force_build: @force_build, - targets: ~w( - x86_64-linux-gnu - aarch64-linux-gnu - aarch64-macos-none - x86_64-windows-gnu - ), - zig_code_path: "zig_codecs.zig", - optimize: :env, - c: [ - include_dirs: @include_dirs, - library_dirs: @library_dirs, - link_lib: [ - {:system, "zstd"}, - {:system, "lz4"}, - {:system, "snappy"}, - {:system, "blosc"}, - # Use dynamic linking for bzip2 to avoid PIC issues on Linux - {:system, "bz2"} - ] - ], - nifs: [ - zstd_compress: 2, - zstd_decompress: 1, - lz4_compress: 1, - lz4_decompress: 2, - snappy_compress: 1, - snappy_decompress: 1, - blosc_compress: 2, - blosc_decompress: 1, - bzip2_compress: 2, - bzip2_decompress: 2, - crc32c_encode: 1, - crc32c_decode: 1 - ] - - @doc """ - Compresses data using ZLIB. - Uses Erlang's battle-tested :zlib for maximum compatibility. - """ - def zlib_compress(data) when is_binary(data) do - compressed = :zlib.compress(data) - {:ok, compressed} - rescue - e -> {:error, {:zlib_compress_failed, e}} - end - - @doc """ - Decompresses ZLIB data. - """ - def zlib_decompress(data) when is_binary(data) do - decompressed = :zlib.uncompress(data) - {:ok, decompressed} - rescue - e -> {:error, {:zlib_decompress_failed, e}} - end -end diff --git a/lib/ex_zarr/codecs/zig_codecs.zig b/lib/ex_zarr/codecs/zig_codecs.zig deleted file mode 100644 index 993b8d0..0000000 --- a/lib/ex_zarr/codecs/zig_codecs.zig +++ /dev/null @@ -1,366 +0,0 @@ -const std = @import("std"); -const beam = @import("beam"); - -// Import C libraries -const c_zstd = @cImport(@cInclude("zstd.h")); -const c_lz4 = @cImport({ - @cInclude("lz4.h"); - @cInclude("lz4hc.h"); -}); -const c_snappy = @cImport(@cInclude("snappy-c.h")); -const c_blosc = @cImport(@cInclude("blosc.h")); -const c_bz2 = @cImport(@cInclude("bzlib.h")); - -// CRC32C lookup table (Castagnoli polynomial 0x1EDC6F41) -const crc32c_table = [256]u32{ - 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB, - 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24, - 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384, - 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B, - 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35, - 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA, - 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A, - 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595, - 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957, - 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198, - 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38, - 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7, - 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789, - 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46, - 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6, - 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829, - 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93, - 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C, - 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC, - 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033, - 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D, - 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982, - 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622, - 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED, - 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F, - 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0, - 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540, - 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F, - 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1, - 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E, - 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E, - 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351, -}; - -/// Compresses data using ZSTD -pub fn zstd_compress(data: []const u8, level: i32) !beam.term { - const max_size = c_zstd.ZSTD_compressBound(data.len); - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - const size = c_zstd.ZSTD_compress( - buffer.ptr, - max_size, - data.ptr, - data.len, - level - ); - - if (c_zstd.ZSTD_isError(size) != 0) { - return beam.make(.{.@"error", .zstd_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, size); - @memcpy(result, buffer[0..size]); - - return beam.make(result, .{}); -} - -/// Decompresses ZSTD data -pub fn zstd_decompress(data: []const u8) !beam.term { - const content_size = c_zstd.ZSTD_getFrameContentSize(data.ptr, data.len); - - // Check for errors (ZSTD_CONTENTSIZE_ERROR or UNKNOWN are very large values) - if (content_size >= 0xFFFFFFFFFFFFFFFE) { - return beam.make(.{.@"error", .zstd_invalid_data}, .{}); - } - - const buffer = try beam.allocator.alloc(u8, content_size); - - const result_size = c_zstd.ZSTD_decompress( - buffer.ptr, - content_size, - data.ptr, - data.len - ); - - if (c_zstd.ZSTD_isError(result_size) != 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .zstd_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using LZ4 -pub fn lz4_compress(data: []const u8) !beam.term { - const max_size = c_lz4.LZ4_compressBound(@intCast(data.len)); - const buffer = try beam.allocator.alloc(u8, @intCast(max_size)); - defer beam.allocator.free(buffer); - - const size = c_lz4.LZ4_compress_default( - @ptrCast(data.ptr), - @ptrCast(buffer.ptr), - @intCast(data.len), - max_size - ); - - if (size <= 0) { - return beam.make(.{.@"error", .lz4_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, @intCast(size)); - @memcpy(result, buffer[0..@intCast(size)]); - - return beam.make(result, .{}); -} - -/// Decompresses LZ4 data -pub fn lz4_decompress(data: []const u8, original_size: usize) !beam.term { - const buffer = try beam.allocator.alloc(u8, original_size); - - const result_size = c_lz4.LZ4_decompress_safe( - @ptrCast(data.ptr), - @ptrCast(buffer.ptr), - @intCast(data.len), - @intCast(original_size) - ); - - if (result_size < 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .lz4_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Snappy -pub fn snappy_compress(data: []const u8) !beam.term { - const max_size = c_snappy.snappy_max_compressed_length(data.len); - - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - var output_length: usize = max_size; - const status = c_snappy.snappy_compress( - @ptrCast(data.ptr), - data.len, - @ptrCast(buffer.ptr), - &output_length - ); - - if (status != c_snappy.SNAPPY_OK) { - return beam.make(.{.@"error", .snappy_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, output_length); - @memcpy(result, buffer[0..output_length]); - - return beam.make(result, .{}); -} - -/// Decompresses Snappy data -pub fn snappy_decompress(data: []const u8) !beam.term { - var uncompressed_length: usize = undefined; - var status = c_snappy.snappy_uncompressed_length( - @ptrCast(data.ptr), - data.len, - &uncompressed_length - ); - - if (status != c_snappy.SNAPPY_OK) { - return beam.make(.{.@"error", .snappy_invalid_input}, .{}); - } - - const buffer = try beam.allocator.alloc(u8, uncompressed_length); - - status = c_snappy.snappy_uncompress( - @ptrCast(data.ptr), - data.len, - @ptrCast(buffer.ptr), - &uncompressed_length - ); - - if (status != c_snappy.SNAPPY_OK) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .snappy_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Blosc -pub fn blosc_compress(data: []const u8, level: i32) !beam.term { - c_blosc.blosc_init(); - - const max_size = data.len + c_blosc.BLOSC_MAX_OVERHEAD; - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - const size = c_blosc.blosc_compress( - level, - 1, - 1, - data.len, - data.ptr, - buffer.ptr, - max_size - ); - - if (size <= 0) { - return beam.make(.{.@"error", .blosc_compression_failed}, .{}); - } - - const result = try beam.allocator.alloc(u8, @intCast(size)); - @memcpy(result, buffer[0..@intCast(size)]); - - return beam.make(result, .{}); -} - -/// Decompresses Blosc data -pub fn blosc_decompress(data: []const u8) !beam.term { - c_blosc.blosc_init(); - - var nbytes: usize = undefined; - var cbytes: usize = undefined; - var blocksize: usize = undefined; - c_blosc.blosc_cbuffer_sizes( - data.ptr, - &nbytes, - &cbytes, - &blocksize - ); - - const buffer = try beam.allocator.alloc(u8, nbytes); - - const size = c_blosc.blosc_decompress( - data.ptr, - buffer.ptr, - nbytes - ); - - if (size <= 0) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .blosc_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Compresses data using Bzip2 -pub fn bzip2_compress(data: []const u8, level: i32) !beam.term { - const max_size = data.len + (data.len / 100) + 600; - const buffer = try beam.allocator.alloc(u8, max_size); - defer beam.allocator.free(buffer); - - var dest_len: c_uint = @intCast(max_size); - const result = c_bz2.BZ2_bzBuffToBuffCompress( - @ptrCast(buffer.ptr), - &dest_len, - @ptrCast(@constCast(data.ptr)), - @intCast(data.len), - level, - 0, - 30 - ); - - if (result != c_bz2.BZ_OK) { - return beam.make(.{.@"error", .bzip2_compression_failed}, .{}); - } - - const output = try beam.allocator.alloc(u8, dest_len); - @memcpy(output, buffer[0..dest_len]); - - return beam.make(output, .{}); -} - -/// Decompresses Bzip2 data -pub fn bzip2_decompress(data: []const u8, original_size: usize) !beam.term { - const buffer = try beam.allocator.alloc(u8, original_size); - - var dest_len: c_uint = @intCast(original_size); - const result = c_bz2.BZ2_bzBuffToBuffDecompress( - @ptrCast(buffer.ptr), - &dest_len, - @ptrCast(@constCast(data.ptr)), - @intCast(data.len), - 0, - 0 - ); - - if (result != c_bz2.BZ_OK) { - beam.allocator.free(buffer); - return beam.make(.{.@"error", .bzip2_decompression_failed}, .{}); - } - - return beam.make(buffer, .{}); -} - -/// Computes CRC32C checksum using Castagnoli polynomial -fn compute_crc32c(data: []const u8) u32 { - var crc: u32 = 0xFFFFFFFF; - for (data) |byte| { - const index = @as(u8, @truncate(crc ^ byte)); - crc = (crc >> 8) ^ crc32c_table[index]; - } - return crc ^ 0xFFFFFFFF; -} - -/// Encodes data with CRC32C checksum appended (bytes-to-bytes codec) -/// Appends 4-byte checksum in little-endian format to the end of data -pub fn crc32c_encode(data: []const u8) !beam.term { - // Calculate checksum - const checksum = compute_crc32c(data); - - // Allocate buffer for data + 4-byte checksum - const result = try beam.allocator.alloc(u8, data.len + 4); - - // Copy original data - @memcpy(result[0..data.len], data); - - // Append checksum as little-endian u32 - result[data.len] = @truncate(checksum); - result[data.len + 1] = @truncate(checksum >> 8); - result[data.len + 2] = @truncate(checksum >> 16); - result[data.len + 3] = @truncate(checksum >> 24); - - return beam.make(result, .{}); -} - -/// Decodes data with CRC32C checksum validation (bytes-to-bytes codec) -/// Validates and removes the 4-byte checksum from the end of data -pub fn crc32c_decode(data: []const u8) !beam.term { - // Need at least 4 bytes for checksum - if (data.len < 4) { - return beam.make(.{.@"error", .crc32c_invalid_data}, .{}); - } - - // Extract the stored checksum (little-endian) - const stored_checksum: u32 = - @as(u32, data[data.len - 4]) | - (@as(u32, data[data.len - 3]) << 8) | - (@as(u32, data[data.len - 2]) << 16) | - (@as(u32, data[data.len - 1]) << 24); - - // Extract inner data (everything except last 4 bytes) - const inner_data = data[0..data.len - 4]; - - // Compute checksum of inner data - const computed_checksum = compute_crc32c(inner_data); - - // Validate checksums match - if (stored_checksum != computed_checksum) { - return beam.make(.{.@"error", .crc32c_checksum_mismatch}, .{}); - } - - // Allocate and return validated data (without checksum) - const result = try beam.allocator.alloc(u8, inner_data.len); - @memcpy(result, inner_data); - - return beam.make(result, .{}); -} \ No newline at end of file diff --git a/lib/ex_zarr/metadata.ex b/lib/ex_zarr/metadata.ex index 9ab91d9..1d96b51 100644 --- a/lib/ex_zarr/metadata.ex +++ b/lib/ex_zarr/metadata.ex @@ -14,6 +14,8 @@ defmodule ExZarr.Metadata do - **chunks**: The size of each chunk - **dtype**: The data type of elements - **compressor**: The compression codec used + - **compressor_config**: Compressor settings, e.g. `[level: 9]` (see + `ExZarr.Codecs.CompressorConfig`) - **fill_value**: Default value for uninitialized elements - **order**: Memory layout order ("C" for row-major) - **zarr_format**: Version of the Zarr specification (2) @@ -48,6 +50,7 @@ defmodule ExZarr.Metadata do chunks: tuple(), dtype: ExZarr.dtype(), compressor: ExZarr.compressor(), + compressor_config: keyword(), fill_value: number(), order: String.t(), zarr_format: integer(), @@ -59,6 +62,7 @@ defmodule ExZarr.Metadata do :chunks, :dtype, :compressor, + compressor_config: [], fill_value: 0, order: "C", zarr_format: 2, @@ -107,6 +111,7 @@ defmodule ExZarr.Metadata do chunks: config.chunks, dtype: config.dtype, compressor: config.compressor, + compressor_config: Map.get(config, :compressor_config, []), fill_value: config.fill_value, order: Map.get(config, :order, "C"), zarr_format: 2, diff --git a/lib/ex_zarr/metadata_v3.ex b/lib/ex_zarr/metadata_v3.ex index 6a3cd96..27a72c7 100644 --- a/lib/ex_zarr/metadata_v3.ex +++ b/lib/ex_zarr/metadata_v3.ex @@ -929,7 +929,12 @@ defmodule ExZarr.MetadataV3 do @spec from_v2(ExZarr.Metadata.t()) :: {:ok, t()} def from_v2(%ExZarr.Metadata{} = v2_metadata) do # Convert filters and compressor to v3 codec pipeline - codecs = PipelineV3.from_v2(v2_metadata.filters || [], v2_metadata.compressor) + codecs = + PipelineV3.from_v2( + v2_metadata.filters || [], + v2_metadata.compressor, + Map.get(v2_metadata, :compressor_config) || [] + ) metadata = %__MODULE__{ zarr_format: 3, @@ -1228,7 +1233,7 @@ defmodule ExZarr.MetadataV3 do # Convert v2 style to v3 codecs filters = Map.get(config, :filters, []) compressor = Map.get(config, :compressor, :zstd) - PipelineV3.from_v2(filters, compressor) + PipelineV3.from_v2(filters, compressor, Map.get(config, :compressor_config, [])) codecs -> # Use provided v3 codecs diff --git a/lib/ex_zarr/nx.ex b/lib/ex_zarr/nx.ex index 32aad73..3bbe30e 100644 --- a/lib/ex_zarr/nx.ex +++ b/lib/ex_zarr/nx.ex @@ -175,7 +175,8 @@ defmodule ExZarr.Nx do - `:path` - Path for filesystem storage - `:compressor` - Compression codec (default: `:zlib`) - `:fill_value` - Fill value for uninitialized chunks (default: 0) - - `:zarr_version` - Zarr format version, 2 or 3 (default: 2) + - `:zarr_version` - Zarr format version, 2 or 3 (default: + `config :ex_zarr, default_zarr_version`, or 2) ## Returns @@ -219,7 +220,7 @@ defmodule ExZarr.Nx do storage: Keyword.get(opts, :storage, :memory), compressor: Keyword.get(opts, :compressor, :zlib), fill_value: Keyword.get(opts, :fill_value, 0), - zarr_version: Keyword.get(opts, :zarr_version, 2) + zarr_version: Keyword.get_lazy(opts, :zarr_version, &ExZarr.Version.default_version/0) ] |> maybe_put_path(opts) diff --git a/lib/ex_zarr/storage.ex b/lib/ex_zarr/storage.ex index b7b0c54..e847a3b 100644 --- a/lib/ex_zarr/storage.ex +++ b/lib/ex_zarr/storage.ex @@ -50,6 +50,7 @@ defmodule ExZarr.Storage do {:ok, metadata} = ExZarr.Storage.read_metadata(storage) """ + alias ExZarr.Codecs.CompressorConfig alias ExZarr.Storage.Backend @type backend :: :memory | :filesystem | :zip | :http | :s3 @@ -438,11 +439,14 @@ defmodule ExZarr.Storage do # Parse v2 metadata defp parse_metadata_v2(metadata) do + {compressor, compressor_config} = CompressorConfig.from_numcodecs(metadata.compressor) + parsed_metadata = %ExZarr.Metadata{ shape: List.to_tuple(metadata.shape), chunks: List.to_tuple(metadata.chunks), dtype: string_to_dtype(metadata.dtype), - compressor: parse_compressor(metadata.compressor), + compressor: compressor, + compressor_config: compressor_config, fill_value: metadata.fill_value, order: Map.get(metadata, :order, "C"), zarr_format: metadata.zarr_format, @@ -558,7 +562,8 @@ defmodule ExZarr.Storage do shape: Tuple.to_list(metadata.shape), chunks: Tuple.to_list(metadata.chunks), dtype: dtype_to_string(metadata.dtype), - compressor: compressor_to_json(metadata.compressor), + compressor: + CompressorConfig.to_numcodecs(metadata.compressor, metadata.compressor_config || []), fill_value: metadata.fill_value, order: metadata.order, filters: encode_filters(metadata.filters) @@ -645,19 +650,6 @@ defmodule ExZarr.Storage do ## Private Functions - defp parse_compressor(nil), do: :none - defp parse_compressor(%{id: id}), do: String.to_atom(id) - defp parse_compressor(_), do: :none - - defp compressor_to_json(:none), do: nil - - defp compressor_to_json(compressor) when is_atom(compressor) do - %{ - id: Atom.to_string(compressor), - level: 5 - } - end - # Little-endian (< prefix) defp string_to_dtype(" ExZarr.Version.default_version() - 3 + 2 """ @spec default_version() :: version() def default_version do - Application.get_env(:ex_zarr, :default_zarr_version, 3) + Application.get_env(:ex_zarr, :default_zarr_version, 2) end @doc """ diff --git a/lib/mix/tasks/fix_nif_rpaths.ex b/lib/mix/tasks/fix_nif_rpaths.ex deleted file mode 100644 index 39b84b5..0000000 --- a/lib/mix/tasks/fix_nif_rpaths.ex +++ /dev/null @@ -1,192 +0,0 @@ -defmodule Mix.Tasks.FixNifRpaths do - @moduledoc """ - Adds rpaths to compiled Zig NIF libraries for macOS dynamic library loading. - - This task runs after compilation to ensure that the Zig NIFs can find their - dependent compression libraries at runtime on macOS. - - ## Background - - On macOS, dynamic libraries need to know where to find their dependencies at runtime. - The Zig NIFs link against system compression libraries (zstd, lz4, snappy, blosc, bzip2), - but without proper rpaths, the dynamic linker can't find them. - - This task uses `install_name_tool` to add rpaths for each compression library to the - compiled .so file. - - ## Usage - - This task is automatically run after compilation via a Mix alias: - - mix compile - - Or run manually: - - mix fix_nif_rpaths - - ## Platform Support - - - **macOS**: Fully supported, adds rpaths using `install_name_tool` - - **Linux**: Not needed (uses standard library paths) - - **Windows**: Not supported (different dynamic library mechanism) - """ - - use Mix.Task - - @shortdoc "Adds rpaths to Zig NIF libraries on macOS" - - alias ExZarr.Codecs.CompressionConfig - - # Get compression library paths from configuration - # This supports environment variable overrides via COMPRESSION_LIB_DIRS - defp macos_lib_paths do - CompressionConfig.rpath_dirs() - end - - @impl Mix.Task - def run(_args) do - # Only run on macOS - case :os.type() do - {:unix, :darwin} -> - fix_macos_rpaths() - - _ -> - Mix.shell().info("Skipping rpath fix (not macOS)") - :ok - end - end - - defp fix_macos_rpaths do - nif_path = find_nif_library() - - case nif_path do - nil -> - # No NIF library found, probably not compiled yet - Mix.shell().info("No Zig NIF library found, skipping rpath fix") - :ok - - path -> - Mix.shell().info("Adding rpaths to #{Path.basename(path)}...") - add_rpaths(path) - end - end - - defp find_nif_library do - # Look for the compiled Zig NIF library - build_dir = Mix.Project.build_path() - app_name = Mix.Project.config()[:app] - - nif_path = - Path.join([ - build_dir, - "lib", - to_string(app_name), - "priv", - "lib", - "Elixir.ExZarr.Codecs.ZigCodecs.so" - ]) - - if File.exists?(nif_path) do - nif_path - else - nil - end - end - - defp add_rpaths(nif_path) do - # Get existing rpaths to avoid duplicates - existing_rpaths = get_existing_rpaths(nif_path) - - # Filter out paths that already exist and paths that don't exist on the system - new_paths = - macos_lib_paths() - |> Enum.filter(&File.dir?/1) - |> Enum.reject(&(&1 in existing_rpaths)) - - case new_paths do - [] -> - Mix.shell().info("All rpaths already present ✓") - :ok - - paths -> - # Try to add rpaths - results = - Enum.map(paths, fn path -> - {path, add_single_rpath(nif_path, path)} - end) - - successful = Enum.count(results, fn {_, result} -> result == :ok end) - failed = Enum.count(results, fn {_, result} -> result == :error end) - - if failed > 0 do - Mix.shell().info("") - - Mix.shell().info( - "WARNING: Some rpaths could not be added due to Mach-O header size limits." - ) - - Mix.shell().info( - " To use all compression codecs, set the library path before running:" - ) - - Mix.shell().info("") - - Mix.shell().info( - " export DYLD_FALLBACK_LIBRARY_PATH=\"#{Enum.join(macos_lib_paths() |> Enum.filter(&File.dir?/1), ":")}\"" - ) - - Mix.shell().info("") - Mix.shell().info(" Or add it to your shell profile (~/.zshrc or ~/.bashrc)") - Mix.shell().info("") - end - - if successful > 0 do - Mix.shell().info("Added #{successful} rpath(s) ✓") - end - - :ok - end - end - - defp get_existing_rpaths(nif_path) do - case System.cmd("otool", ["-l", nif_path], stderr_to_stdout: true) do - {output, 0} -> - # Parse otool output to extract existing rpaths - output - |> String.split("\n") - |> Enum.chunk_every(3, 1, :discard) - |> Enum.filter(fn [line1, _line2, _line3] -> - String.contains?(line1, "LC_RPATH") - end) - |> Enum.map(fn [_line1, _line2, line3] -> - # Extract path from " path /some/path (offset 12)" - line3 - |> String.trim() - |> String.split(" ") - |> Enum.at(1) - end) - |> Enum.reject(&is_nil/1) - - {error, _} -> - Mix.shell().error("Failed to read existing rpaths: #{error}") - [] - end - end - - defp add_single_rpath(nif_path, rpath) do - case System.cmd("install_name_tool", ["-add_rpath", rpath, nif_path], stderr_to_stdout: true) do - {_output, 0} -> - Mix.shell().info(" + #{rpath}") - :ok - - {error, _} -> - # Check if error is just "would duplicate path" which is fine - if String.contains?(error, "would duplicate path") do - :ok - else - Mix.shell().error(" ✗ Failed to add rpath #{rpath}: #{error}") - :error - end - end - end -end diff --git a/mix.exs b/mix.exs index e439e46..5c1daa7 100644 --- a/mix.exs +++ b/mix.exs @@ -36,7 +36,6 @@ defmodule ExZarr.MixProject do defp aliases do [ - compile: ["compile", "fix_nif_rpaths"], verify: &verify/1 ] end @@ -66,11 +65,9 @@ defmodule ExZarr.MixProject do # JSON encoding/decoding for metadata {:jason, "~> 1.4"}, - # Zig NIFs: zigler required until precompiled checksums ship (Mix.install / - # path deps do not pull optional deps). After checksum-*.exs is published, - # zigler can become optional: true again for Hex-only consumers. - {:zigler_precompiled, "~> 0.1.6"}, - {:zigler, "~> 0.16", runtime: false}, + # Compression codecs (zstd, lz4, snappy, bzip2, blosc, crc32c): pure-Rust + # NIFs with precompiled binaries, so no system libraries or toolchains. + {:ex_codecs, "~> 0.2.4"}, # Cloud storage backends (optional) {:ex_aws, "~> 2.5", optional: true}, @@ -94,9 +91,7 @@ defmodule ExZarr.MixProject do {:flow, "~> 1.2", optional: true}, {:gen_stage, "~> 1.2", optional: true}, {:broadway, "~> 1.0", optional: true}, - - # Documentation — no :only: zig_doc (zigler transitive dep) requires ex_doc in prod - {:ex_doc, "~> 0.39", runtime: false}, + {:ex_doc, "~> 0.39", only: :dev, runtime: false}, {:credo, "~> 1.7", only: [:dev, :test], runtime: false}, {:ex_slop, "~> 0.4.5", only: [:dev, :test], runtime: false}, {:doctor, "~> 0.23.0", only: :dev, runtime: false}, @@ -127,13 +122,8 @@ defmodule ExZarr.MixProject do "Zarr Specification" => "https://zarr.dev" }, maintainers: ["Thanos Vassilakis"], - # Zigler regenerates this ignored intermediate during source builds. - exclude_patterns: ["lib/ex_zarr/codecs/.Elixir.ExZarr.Codecs.ZigCodecs.zig"], files: [ "lib", - "native", - "checksum-*.exs", - "docs/PRECOMPILATION.md", ".formatter.exs", "mix.exs", "README.md", @@ -212,7 +202,6 @@ defmodule ExZarr.MixProject do "docs/V2_TO_V3_MIGRATION.md", "docs/migration_guide_v1_1_0.md", "benchmarks/README.md", - "docs/PRECOMPILATION.md", "LICENSE", # Reference @@ -284,7 +273,6 @@ defmodule ExZarr.MixProject do "docs/PERFORMANCE_IMPROVEMENTS.md", "docs/V2_TO_V3_MIGRATION.md", "benchmarks/README.md", - "docs/PRECOMPILATION.md", "LICENSE" ], "Release Notes & Migration Guides": [ diff --git a/mix.lock b/mix.lock index fb9d2fe..b0f566b 100644 --- a/mix.lock +++ b/mix.lock @@ -3,8 +3,6 @@ "benchee": {:hex, :benchee, "1.5.1", "b95cbc36c4b98969a5c592a246e171041eb683c56bad1cb4f49a3b081ba66087", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}, {:statistex, "~> 1.1", [hex: :statistex, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "a539301f8dfd4efc5c5123bfb9d47ebde20092a863a5b5b16c2a60d2243dfce7"}, "broadway": {:hex, :broadway, "1.3.0", "f75f6376159b74f55c5ba2629dac613e4fd79d9e71148ab5fbac8fdd7c999d2a", [:mix], [{:gen_stage, "~> 1.0", [hex: :gen_stage, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.3.7 or ~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "bef3b4c5512d0072917b70239cbecf8f76a2587465a5b7c3e2b9ae18b4bc405b"}, "bunt": {:hex, :bunt, "1.0.0", "081c2c665f086849e6d57900292b3a161727ab40431219529f13c4ddcf3e7a44", [:mix], [], "hexpm", "dc5f86aa08a5f6fa6b8096f0735c4e76d54ae5c9fa2c143e5a1fc7c1cd9bb6b5"}, - "castore": {:hex, :castore, "1.0.21", "0a0e8330dc267a40a3b7ad86d39302764bb71758172904e6a59d5ad6443ce307", [:mix], [], "hexpm", "e42e22723e25dbd46876d056a03f685513d6e98f6b5e555dc551321decd76c5c"}, - "certifi": {:hex, :certifi, "2.15.0", "0e6e882fcdaaa0a5a9f2b3db55b1394dba07e8d6d9bcad08318fb604c6839712", [:rebar3], [], "hexpm", "b147ed22ce71d72eafdad94f055165c1c182f61a2ff49df28bcc71d1d5b94a60"}, "complex": {:hex, :complex, "0.7.0", "695632ef9487517aa5d57edd1697801079d622414cb2e1a7cf538b1f9a50f205", [:mix], [], "hexpm", "0ee39c0803129f546e7f3f640da8f021c9e659402bf59da6f7f2c4848f068f8d"}, "credo": {:hex, :credo, "1.7.19", "cc52129665fc7c15143d47838fda0f9cd6dac9ceced7bf4da6f85fcbfe64b12a", [:mix], [{:bunt, "~> 0.2.1 or ~> 1.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2 or ~> 1.0", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "2d8bc95d5a7bb99dd2613621d4f08c6a3575c3fd4b62e6a2b48a100352a557b8"}, "db_connection": {:hex, :db_connection, "2.10.2", "ae391e803a5adff104da913c2fc1c0c14a37f8b10001dcef568796e1fb7bf95c", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "510b14482330f1af6490a2fa0efd8d4f1435d1529b165647df22ac0f2df0fa93"}, @@ -16,6 +14,7 @@ "erlex": {:hex, :erlex, "0.2.9", "7debbbaa9f4f368b8cd648983e0f1d7963028508e9c59e9d4ed504e94ef52a55", [:mix], [], "hexpm", "8cfffc0ec7159e6d73de2ab28a588064de80f88b2798d5cbe4482cbbc200178b"}, "ex_aws": {:hex, :ex_aws, "2.7.0", "e6bfd4b5fb8c791aa6c7d57fc7c45f050bb89de9576f1f6104aa974b234c01ec", [:mix], [{:configparser_ex, "~> 5.0", [hex: :configparser_ex, repo: "hexpm", optional: true]}, {:hackney, "~> 4.0", [hex: :hackney, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:jsx, "~> 2.8 or ~> 3.0", [hex: :jsx, repo: "hexpm", optional: true]}, {:mime, "~> 1.2 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:req, "~> 0.5.10 or ~> 0.6 or ~> 1.0", [hex: :req, repo: "hexpm", optional: true]}, {:sweet_xml, "~> 0.7", [hex: :sweet_xml, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "bfe9d744d4fd4c1f40314ee7fab504d5547d1f01cd377fff1568cbe630b06d65"}, "ex_aws_s3": {:hex, :ex_aws_s3, "2.5.9", "862b7792f2e60d7010e2920d79964e3fab289bc0fd951b0ba8457a3f7f9d1199", [:mix], [{:ex_aws, "~> 2.0", [hex: :ex_aws, repo: "hexpm", optional: false]}, {:sweet_xml, ">= 0.0.0", [hex: :sweet_xml, repo: "hexpm", optional: true]}], "hexpm", "a480d2bb2da64610014021629800e1e9457ca5e4a62f6775bffd963360c2bf90"}, + "ex_codecs": {:hex, :ex_codecs, "0.2.4", "31ae12cd98ce6d6e50ec02f1a09550995e2069322e00a026307031ff05f68763", [:mix], [{:rustler, "~> 0.36", [hex: :rustler, repo: "hexpm", optional: true]}, {:rustler_precompiled, "~> 0.8", [hex: :rustler_precompiled, repo: "hexpm", optional: false]}], "hexpm", "b4e020ca8d94897cd7785b0d5b56e5e30dc36d3fc10928765ddd1f3b12a15706"}, "ex_doc": {:hex, :ex_doc, "0.39.1", "e19d356a1ba1e8f8cfc79ce1c3f83884b6abfcb79329d435d4bbb3e97ccc286e", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "8abf0ed3e3ca87c0847dfc4168ceab5bedfe881692f1b7c45f4a11b232806865"}, "ex_slop": {:hex, :ex_slop, "0.4.5", "4e3ea38397d08d422f8e92e4cef16877e4a31a93c6abd07b494f4458e37c3ce2", [:mix], [{:credo, "~> 1.7", [hex: :credo, repo: "hexpm", optional: false]}], "hexpm", "174c1ce4da612f7d3e93734d08d81d93774bacb6a8e86123e7d6876369a2df81"}, "excoveralls": {:hex, :excoveralls, "0.18.5", "e229d0a65982613332ec30f07940038fe451a2e5b29bce2a5022165f0c9b157e", [:mix], [{:castore, "~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "523fe8a15603f86d64852aab2abe8ddbd78e68579c8525ae765facc5eae01562"}, @@ -24,18 +23,13 @@ "flow": {:hex, :flow, "1.2.4", "1dd58918287eb286656008777cb32714b5123d3855956f29aa141ebae456922d", [:mix], [{:gen_stage, "~> 1.0", [hex: :gen_stage, repo: "hexpm", optional: false]}], "hexpm", "874adde96368e71870f3510b91e35bc31652291858c86c0e75359cbdd35eb211"}, "gen_stage": {:hex, :gen_stage, "1.3.2", "7c77e5d1e97de2c6c2f78f306f463bca64bf2f4c3cdd606affc0100b89743b7b", [:mix], [], "hexpm", "0ffae547fa777b3ed889a6b9e1e64566217413d018cabd825f786e843ffe63e7"}, "goth": {:hex, :goth, "1.4.5", "ee37f96e3519bdecd603f20e7f10c758287088b6d77c0147cd5ee68cf224aade", [:mix], [{:finch, "~> 0.17", [hex: :finch, repo: "hexpm", optional: false]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:jose, "~> 1.11", [hex: :jose, repo: "hexpm", optional: false]}], "hexpm", "0fc2dce5bd710651ed179053d0300ce3a5d36afbdde11e500d57f05f398d5ed5"}, - "hackney": {:hex, :hackney, "1.25.0", "390e9b83f31e5b325b9f43b76e1a785cbdb69b5b6cd4e079aa67835ded046867", [:rebar3], [{:certifi, "~> 2.15.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.4", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.4.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "7209bfd75fd1f42467211ff8f59ea74d6f2a9e81cbcee95a56711ee79fd6b1d4"}, "hpax": {:hex, :hpax, "1.1.0", "782931867cc23217c68fb5f68fe1a11f5e7544c7fda82c8a7019a5df5a4a1cdf", [:mix], [], "hexpm", "0b8d0f05832f55571d65ac720f79bf8994138ffbb133209dc4685eae0ad456a8"}, - "httpoison": {:hex, :httpoison, "1.8.2", "9eb9c63ae289296a544842ef816a85d881d4a31f518a0fec089aaa744beae290", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "2bb350d26972e30c96e2ca74a1aaf8293d61d0742ff17f01e0279fef11599921"}, - "idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"}, "jason": {:hex, :jason, "1.4.5", "2e3a008590b0b8d7388c20293e9dcc9cf3e5d642fd2a114e4cbbb52e595d940a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0 or ~> 3.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "b0c823996102bcd0239b3c2444eb00409b72f6a140c1950bc8b457d836b30684"}, "jose": {:hex, :jose, "1.11.12", "06e62b467b61d3726cbc19e9b5489f7549c37993de846dfb3ee8259f9ed208b3", [:mix, :rebar3], [], "hexpm", "31e92b653e9210b696765cdd885437457de1add2a9011d92f8cf63e4641bab7b"}, "makeup": {:hex, :makeup, "1.2.2", "882d46dc0905e9ff7abf2aab61a7e6b3dcc555533977d8a23b06019e6c89ac94", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "9a1a24e5b343b8ae16abea0822c10a6f75da27af7fa802ada5251f7579bfccfa"}, "makeup_elixir": {:hex, :makeup_elixir, "1.0.1", "e928a4f984e795e41e3abd27bfc09f51db16ab8ba1aebdba2b3a575437efafc2", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.2.3 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "7284900d412a3e5cfd97fdaed4f5ed389b8f2b4cb49efc0eb3bd10e2febf9507"}, "makeup_erlang": {:hex, :makeup_erlang, "1.1.0", "835f7e60792e08824cda445639555d7bf1bbbddb1b60b306e33cb6f6db24dc74", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "1cd6780fb1dd1a03979abaed0fe82712b0625118fd5257d3ebbf73f960c73c3c"}, - "metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"}, "mime": {:hex, :mime, "2.0.7", "b8d739037be7cd402aee1ba0306edfdef982687ee7e9859bee6198c1e7e2f128", [:mix], [], "hexpm", "6171188e399ee16023ffc5b76ce445eb6d9672e2e241d2df6050f3c771e80ccd"}, - "mimerl": {:hex, :mimerl, "1.5.0", "f35aca6f23242339b3666e0ac0702379e362b469d0aea167f6cc713547e777ed", [:rebar3], [], "hexpm", "db648ce065bae14ea84ca8b5dd123f42f49417cef693541110bf6f9e9be9ecc4"}, "mint": {:hex, :mint, "1.11.0", "a713551624815c0435237b93d90ea8b9b14254690c66f732d0ef8930f76ff1d9", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 1.1", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "c6279ba2d6aa3a383a1d4cfbe7b59f42e6efd400f58d8e2acfeac48a438693ab"}, "mongodb_driver": {:hex, :mongodb_driver, "1.6.4", "d7a36f4083ff1a91b65eba86fdfacaaa2870aa791bfe226ff2af9d28c90a6dcf", [:mix], [{:db_connection, "~> 2.6", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, ">= 2.1.1 and < 3.0.0-0 or ~> 3.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:ezstd, "~> 1.1", [hex: :ezstd, repo: "hexpm", optional: true]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "bcac423730223e5dbffbc8397f9131e03eec1ad53a2d809c1395882bb43e597c"}, "mox": {:hex, :mox, "1.3.2", "f34ca4331b1cce3125609c6de674739fe5f3df0d68df25510d64b6a94b2246de", [:mix], [{:nimble_ownership, "~> 1.0", [hex: :nimble_ownership, repo: "hexpm", optional: false]}], "hexpm", "97918a185e727f3128a826f01827b5b8168e1b815bda19eb4192c3ffdb6a8054"}, @@ -44,20 +38,11 @@ "nimble_parsec": {:hex, :nimble_parsec, "1.4.2", "8efba0122db06df95bfaa78f791344a89352ba04baedd3849593bfce4d0dc1c6", [:mix], [], "hexpm", "4b21398942dda052b403bbe1da991ccd03a053668d147d53fb8c4e0efe09c973"}, "nimble_pool": {:hex, :nimble_pool, "1.1.0", "bf9c29fbdcba3564a8b800d1eeb5a3c58f36e1e11d7b7fb2e084a643f645f06b", [:mix], [], "hexpm", "af2e4e6b34197db81f7aad230c1118eac993acc0dae6bc83bac0126d4ae0813a"}, "nx": {:hex, :nx, "0.13.1", "f96311a573dcd57d11ff889958ee518f52f7c16dbf6d22191020d322887856fc", [:mix], [{:complex, "~> 0.7", [hex: :complex, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "20556ea36c1e9732c1d87eebbf7c0773517e876e92af4ff592b974f80db0b9d9"}, - "parse_trans": {:hex, :parse_trans, "3.4.1", "6e6aa8167cb44cc8f39441d05193be6e6f4e7c2946cb2759f015f8c56b76e5ff", [:rebar3], [], "hexpm", "620a406ce75dada827b82e453c19cf06776be266f5a67cff34e1ef2cbb60e49a"}, - "pegasus": {:hex, :pegasus, "0.2.6", "b4af6522326fbb2ffd1bb706e78ec05854fbcb4b03a7fe08f8db2e9de1d0be67", [:mix], [{:nimble_parsec, "~> 1.2", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "0ac159f0ccab7967cf90208327cc8a35788874814c8d78e19d47104d3fc049b9"}, - "protoss": {:hex, :protoss, "1.1.0", "853533313989751c7da5b0e1ce71501a23f3dc41f128709478af40daccc6e959", [:mix], [], "hexpm", "c2f874383dd047fcfdf467b814dd33a208a4d24a467aef90d86185b41a0752ad"}, "req": {:hex, :req, "0.6.3", "7fe5e68792ff0546e45d5919104fa1764a13694cfe3e48c8a0f32ad051ae77e4", [:mix], [{:brotli, "~> 0.3.1", [hex: :brotli, repo: "hexpm", optional: true]}, {:ezstd, "~> 1.0", [hex: :ezstd, repo: "hexpm", optional: true]}, {:finch, "~> 0.21", [hex: :finch, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mime, "~> 2.0.6 or ~> 2.1", [hex: :mime, repo: "hexpm", optional: false]}, {:nimble_csv, "~> 1.0", [hex: :nimble_csv, repo: "hexpm", optional: true]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "e85b5c6c990e6c3f52bbba68e6f099118f2b8252825f96c7c3636b97a3de307d"}, + "rustler_precompiled": {:hex, :rustler_precompiled, "0.10.0", "8f07ff0865cb09a6a0f75e11701e9a46e331e05f737da4e5b01374dcd25b68c0", [:mix], [{:rustler, "~> 0.28", [hex: :rustler, repo: "hexpm", optional: true]}], "hexpm", "c68c9dc732c12acfdf4a7fa06f4e6f48a641179e6be49a474c8a60809929c430"}, "sobelow": {:hex, :sobelow, "0.16.0", "13e3f761bc7fcc88e81ae23ddef9140617a12a3efb26601e5516453290539398", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "00a0260edf9aeac35d9116366f55dc9451d11e0e696e28ec5570c6a56bd41f1e"}, - "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"}, "statistex": {:hex, :statistex, "1.1.1", "73612aa7f79e53c30569be065fd121e380f1cf57bc4c2da5b41be9246da18df9", [:mix], [], "hexpm", "310c4b49b34adf683de3103639006bed233ab54c08a4add65a531448e653857c"}, "stream_data": {:hex, :stream_data, "1.4.0", "026f929db613aabea6208012ae9b8970d3fd5f88b3bdf26831bc536f98c42036", [:mix], [], "hexpm", "2b0ee3a340dcce1c8cf6302a763ee757d1e01c54d6e16d9069062509d68b1dc9"}, "sweet_xml": {:hex, :sweet_xml, "0.7.5", "803a563113981aaac202a1dbd39771562d0ad31004ddbfc9b5090bdcd5605277", [:mix], [], "hexpm", "193b28a9b12891cae351d81a0cead165ffe67df1b73fe5866d10629f4faefb12"}, "telemetry": {:hex, :telemetry, "1.4.2", "a0cb522801dffb1c49fe6e30561badffc7b6d0e180db1300df759faa22062855", [:rebar3], [], "hexpm", "928f6495066506077862c0d1646609eed891a4326bee3126ba54b60af61febb1"}, - "unicode_util_compat": {:hex, :unicode_util_compat, "0.7.1", "a48703a25c170eedadca83b11e88985af08d35f37c6f664d6dcfb106a97782fc", [:rebar3], [], "hexpm", "b3a917854ce3ae233619744ad1e0102e05673136776fb2fa76234f3e03b23642"}, - "zig_doc": {:hex, :zig_doc, "0.7.0", "812948dbac9761a55426788952b9df7209b3462f3d2f027a3f9b000ede7e8108", [:mix], [{:ex_doc, "0.39.1", [hex: :ex_doc, repo: "hexpm", optional: false]}, {:zig_parser, "~> 0.7.0", [hex: :zig_parser, repo: "hexpm", optional: false]}], "hexpm", "28b57b0bc12f241fc855b6514fe5198f3067fbab2555a3e3a83f6f9ed4699f4c"}, - "zig_get": {:hex, :zig_get, "0.16.0", "de0d5ef81a4917c142496d4f26e12f3fbbe209b3ed7b3af25a57437ac3af6542", [:mix], [], "hexpm", "2841127cf2005da74fb28d3cc68397a393d3d71df5f45752b974345e696beb01"}, - "zig_parser": {:hex, :zig_parser, "0.7.0", "be5972c78387a56833599cf9358c498baf7a8ab71959721e0e9ede224b676f52", [:mix], [{:pegasus, "~> 0.2.4", [hex: :pegasus, repo: "hexpm", optional: false]}], "hexpm", "fd0b2fc343820a202787a78a40feae372386b4303717b099223d5f5c5f21f0bb"}, - "zigler": {:hex, :zigler, "0.16.0", "d872da4571a43e400c084005c214dbe88d1ed8841025766d9d3967c4799abfc8", [:mix], [{:protoss, "~> 1.0", [hex: :protoss, repo: "hexpm", optional: false]}, {:zig_doc, "~> 0.7.0", [hex: :zig_doc, repo: "hexpm", optional: false]}, {:zig_get, "~> 0.16.0", [hex: :zig_get, repo: "hexpm", optional: false]}, {:zig_parser, "~> 0.7.0", [hex: :zig_parser, repo: "hexpm", optional: false]}], "hexpm", "867ce49289568a7fabff400cb9f1636a429defcd488f3302a82c4b51b2bc7741"}, - "zigler_precompiled": {:hex, :zigler_precompiled, "0.1.6", "1c5c889ff30164eb340e599ae987b6c51c3ba0654117ec7a68a6b83d4df39310", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: false]}, {:zigler, "~> 0.13", [hex: :zigler, repo: "hexpm", optional: true]}], "hexpm", "5754d342533b2fb8e767c0ef216ee13c928e17f317cac7d6a01a0972ba68ce4a"}, } diff --git a/native/codecs/compression.zig b/native/codecs/compression.zig deleted file mode 100644 index 3294fba..0000000 --- a/native/codecs/compression.zig +++ /dev/null @@ -1,79 +0,0 @@ -const std = @import("std"); -const beam = @import("beam"); - -/// Compresses data using Zig's standard library deflate compression -pub fn deflate_compress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - // Estimate compressed size - const max_size = data.len + (data.len / 10) + 16; - var compressed = try allocator.alloc(u8, max_size); - - var fbs = std.io.fixedBufferStream(compressed); - var comp = try std.compress.deflate.compressor(fbs.writer(), .{}); - - _ = try comp.write(data); - try comp.finish(); - - const compressed_len = fbs.pos; - - // Return a new allocation that beam owns - const result = try allocator.alloc(u8, compressed_len); - @memcpy(result, compressed[0..compressed_len]); - - allocator.free(compressed); - return result; -} - -/// Decompresses deflate-compressed data -pub fn deflate_decompress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - var fbs = std.io.fixedBufferStream(data); - var decomp = std.compress.deflate.decompressor(fbs.reader()); - - const decompressed = try decomp.reader().readAllAlloc(allocator, std.math.maxInt(usize)); - return decompressed; -} - -/// Compresses data using gzip format -pub fn gzip_compress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - // Estimate compressed size - const max_size = data.len + (data.len / 10) + 32; - var compressed = try allocator.alloc(u8, max_size); - - var fbs = std.io.fixedBufferStream(compressed); - var comp = try std.compress.gzip.compressor(fbs.writer(), .{}); - - _ = try comp.write(data); - try comp.finish(); - - const compressed_len = fbs.pos; - - const result = try allocator.alloc(u8, compressed_len); - @memcpy(result, compressed[0..compressed_len]); - - allocator.free(compressed); - return result; -} - -/// Decompresses gzip-compressed data -pub fn gzip_decompress(data: []const u8) ![]u8 { - var gpa = std.heap.GeneralPurposeAllocator(.{}){}; - defer _ = gpa.deinit(); - const allocator = gpa.allocator(); - - var fbs = std.io.fixedBufferStream(data); - var decomp = try std.compress.gzip.decompressor(fbs.reader()); - - const decompressed = try decomp.reader().readAllAlloc(allocator, std.math.maxInt(usize)); - return decompressed; -} diff --git a/test/ex_zarr/codecs/codec_formats_test.exs b/test/ex_zarr/codecs/codec_formats_test.exs new file mode 100644 index 0000000..5014383 --- /dev/null +++ b/test/ex_zarr/codecs/codec_formats_test.exs @@ -0,0 +1,190 @@ +defmodule ExZarr.Codecs.CodecFormatsTest do + @moduledoc """ + On-disk codec formats after moving the built-in codecs to ExCodecs. + + * `legacy_codecs/` holds bytes written by ExZarr's Zig NIFs (1.1.x and the + 1.2 development line). They must keep decoding. + * `numcodecs/` holds bytes written by numcodecs (zarr-python); see + `test/fixtures/numcodecs/generate.py`. + """ + # async: false because one test changes the :max_chunk_bytes app env + use ExUnit.Case, async: false + + alias ExZarr.Codecs + + @legacy_dir Path.expand("../../fixtures/legacy_codecs", __DIR__) + @numcodecs_dir Path.expand("../../fixtures/numcodecs", __DIR__) + + # Same payload both fixture generators use. + @payload for i <- 0..39_999, into: <<>>, do: <> + + defp fixture(dir, name), do: File.read!(Path.join(dir, name <> ".bin")) + + describe "data written by the Zig NIF codecs" do + for codec <- [:zstd, :lz4, :snappy, :blosc, :bzip2, :crc32c, :zlib] do + test "#{codec} still decodes" do + bin = fixture(@legacy_dir, unquote(Atom.to_string(codec))) + assert {:ok, @payload} = Codecs.decompress(bin, unquote(codec)) + end + end + + test "legacy lz4 and bzip2 used an 8-byte size prefix" do + for codec <- ["lz4", "bzip2"] do + assert <<40_000::64-little, _::binary>> = fixture(@legacy_dir, codec) + end + end + end + + describe "data written by numcodecs (zarr-python)" do + for {name, codec} <- [ + {"zstd", :zstd}, + {"lz4", :lz4}, + {"bz2", :bzip2}, + {"blosc_blosclz_shuffle", :blosc}, + {"blosc_lz4_bitshuffle", :blosc}, + {"blosc_zstd_noshuffle", :blosc} + ] do + test "#{name} decodes" do + bin = fixture(@numcodecs_dir, unquote(name)) + assert {:ok, @payload} = Codecs.decompress(bin, unquote(codec)) + end + end + end + + describe "formats written now" do + test "lz4 uses numcodecs' 4-byte little-endian size prefix" do + assert {:ok, <<40_000::32-little, _block::binary>>} = Codecs.compress(@payload, :lz4) + end + + test "bzip2 is a plain bzip2 stream" do + assert {:ok, <<"BZh9", _::binary>>} = Codecs.compress(@payload, :bzip2) + assert {:ok, <<"BZh1", _::binary>>} = Codecs.compress(@payload, :bzip2, level: 1) + end + + test "blosc writes Blosc1 chunks with ExZarr's historical defaults" do + assert {:ok, <<2, 1, flags, 1, 40_000::32-little, _::binary>>} = + Codecs.compress(@payload, :blosc) + + # BloscLZ (format 0), byte shuffle + assert Bitwise.bsr(flags, 5) == 0 + assert Bitwise.band(flags, 0x01) == 0x01 + end + + test "blosc honours cname, shuffle and typesize" do + assert {:ok, <<2, 1, flags, 8, _::binary>> = chunk} = + Codecs.compress(@payload, :blosc, cname: :zstd, shuffle: :bit, typesize: 8) + + assert Bitwise.bsr(flags, 5) == 4 + assert Bitwise.band(flags, 0x04) == 0x04 + assert {:ok, @payload} = Codecs.decompress(chunk, :blosc) + end + + test "crc32c appends a little-endian checksum" do + assert {:ok, "123456789" <> <<0x83, 0x92, 0x06, 0xE3>>} = + Codecs.compress("123456789", :crc32c) + end + + test "map options are accepted like keyword options" do + assert {:ok, <<"BZh1", _::binary>>} = Codecs.compress(@payload, :bzip2, %{level: 1}) + end + end + + describe "v3 pipeline" do + alias ExZarr.Codecs.PipelineV3 + + test "blosc configuration from zarr.json is applied" do + {:ok, pipeline} = + PipelineV3.parse_codecs([ + %{name: "bytes", configuration: %{endian: "little"}}, + %{ + name: "blosc", + configuration: %{ + "cname" => "lz4", + "clevel" => 7, + "shuffle" => "bitshuffle", + "typesize" => 4 + } + } + ]) + + {:ok, <<2, 1, flags, 4, _::binary>> = chunk} = PipelineV3.encode(@payload, pipeline) + assert Bitwise.bsr(flags, 5) == 1 + assert Bitwise.band(flags, 0x04) == 0x04 + assert {:ok, @payload} = PipelineV3.decode(chunk, pipeline) + end + end + + describe "v2 compressor metadata" do + setup do + path = Path.join(System.tmp_dir!(), "ex_zarr_v2meta_#{System.unique_integer([:positive])}") + on_exit(fn -> File.rm_rf!(path) end) + {:ok, path: path} + end + + defp zarray(path), do: path |> Path.join(".zarray") |> File.read!() |> Jason.decode!() + + defp write_v2(path, compressor) do + {:ok, array} = + ExZarr.create( + shape: {20}, + chunks: {10}, + dtype: :float64, + compressor: compressor, + storage: :filesystem, + path: path + ) + + :ok = ExZarr.Array.save(array, path: path) + array + end + + test "uses numcodecs ids and configurations", %{path: path} do + for {compressor, expected} <- [ + lz4: %{"id" => "lz4", "acceleration" => 1}, + bzip2: %{"id" => "bz2", "level" => 9}, + blosc: %{ + "id" => "blosc", + "cname" => "blosclz", + "clevel" => 5, + "shuffle" => 1, + "blocksize" => 0 + }, + zstd: %{"id" => "zstd", "level" => 3} + ] do + dir = Path.join(path, Atom.to_string(compressor)) + write_v2(dir, compressor) + assert zarray(dir)["compressor"] == expected + end + end + + test "arrays written with the old \"bzip2\" id still open", %{path: path} do + data = for i <- 0..19, into: <<>>, do: <> + array = write_v2(path, :bzip2) + :ok = ExZarr.Array.set_slice(array, data, start: {0}, stop: {20}) + + meta = zarray(path) |> Map.put("compressor", %{"id" => "bzip2", "level" => 5}) + File.write!(Path.join(path, ".zarray"), Jason.encode!(meta)) + + {:ok, reopened} = ExZarr.open(path: path) + assert reopened.compressor == :bzip2 + assert {:ok, ^data} = ExZarr.Array.get_slice(reopened, start: {0}, stop: {20}) + end + end + + describe "decoded chunk size limit" do + test "max_chunk_bytes caps decompressed output" do + {:ok, compressed} = Codecs.compress(@payload, :zstd) + previous = Application.get_env(:ex_zarr, :max_chunk_bytes) + Application.put_env(:ex_zarr, :max_chunk_bytes, 1_000) + + try do + assert {:error, {:decompression_failed, :output_limit_exceeded}} = + Codecs.decompress(compressed, :zstd) + after + if previous, + do: Application.put_env(:ex_zarr, :max_chunk_bytes, previous), + else: Application.delete_env(:ex_zarr, :max_chunk_bytes) + end + end + end +end diff --git a/test/ex_zarr/codecs/compression_config_test.exs b/test/ex_zarr/codecs/compression_config_test.exs deleted file mode 100644 index 51e15f9..0000000 --- a/test/ex_zarr/codecs/compression_config_test.exs +++ /dev/null @@ -1,379 +0,0 @@ -defmodule ExZarr.Codecs.CompressionConfigTest do - use ExUnit.Case, async: true - - alias ExZarr.Codecs.CompressionConfig - - describe "library_dirs/0" do - test "returns list of library directories" do - dirs = CompressionConfig.library_dirs() - assert is_list(dirs) - # All entries should be strings - assert Enum.all?(dirs, &is_binary/1) - end - - test "respects COMPRESSION_LIB_DIRS environment variable" do - original = System.get_env("COMPRESSION_LIB_DIRS") - - try do - System.put_env("COMPRESSION_LIB_DIRS", "/custom/path1:/custom/path2") - dirs = CompressionConfig.library_dirs() - - assert dirs == ["/custom/path1", "/custom/path2"] - after - if original do - System.put_env("COMPRESSION_LIB_DIRS", original) - else - System.delete_env("COMPRESSION_LIB_DIRS") - end - end - end - - test "returns platform-specific defaults when env var not set" do - original = System.get_env("COMPRESSION_LIB_DIRS") - - try do - System.delete_env("COMPRESSION_LIB_DIRS") - dirs = CompressionConfig.library_dirs() - - case :os.type() do - {:unix, :darwin} -> - # macOS should return Homebrew paths - assert Enum.empty?(dirs) == false - assert Enum.any?(dirs, &String.contains?(&1, "homebrew")) - - {:unix, _} -> - # Linux should return system paths - assert Enum.any?(dirs, &String.contains?(&1, "/usr")) - - _ -> - # Other platforms may return empty - assert is_list(dirs) - end - after - if original do - System.put_env("COMPRESSION_LIB_DIRS", original) - end - end - end - end - - describe "include_dirs/0" do - test "returns list of include directories" do - dirs = CompressionConfig.include_dirs() - assert is_list(dirs) - assert Enum.all?(dirs, &is_binary/1) - end - - test "respects COMPRESSION_INCLUDE_DIRS environment variable" do - original = System.get_env("COMPRESSION_INCLUDE_DIRS") - - try do - System.put_env("COMPRESSION_INCLUDE_DIRS", "/custom/inc1:/custom/inc2") - dirs = CompressionConfig.include_dirs() - - assert dirs == ["/custom/inc1", "/custom/inc2"] - after - if original do - System.put_env("COMPRESSION_INCLUDE_DIRS", original) - else - System.delete_env("COMPRESSION_INCLUDE_DIRS") - end - end - end - - test "returns platform-specific defaults when env var not set" do - original = System.get_env("COMPRESSION_INCLUDE_DIRS") - - try do - System.delete_env("COMPRESSION_INCLUDE_DIRS") - dirs = CompressionConfig.include_dirs() - - case :os.type() do - {:unix, :darwin} -> - assert Enum.empty?(dirs) == false - assert Enum.any?(dirs, &String.contains?(&1, "include")) - assert Enum.any?(dirs, &String.contains?(&1, "zstd")) - - {:unix, _} -> - assert Enum.any?(dirs, &String.contains?(&1, "/usr")) - - _ -> - assert is_list(dirs) - end - after - if original do - System.put_env("COMPRESSION_INCLUDE_DIRS", original) - end - end - end - end - - describe "rpath_dirs/0" do - test "returns list of rpath directories" do - dirs = CompressionConfig.rpath_dirs() - assert is_list(dirs) - assert Enum.all?(dirs, &is_binary/1) - end - - test "respects COMPRESSION_LIB_DIRS environment variable" do - original = System.get_env("COMPRESSION_LIB_DIRS") - - try do - System.put_env("COMPRESSION_LIB_DIRS", "/rpath1:/rpath2:/rpath3") - dirs = CompressionConfig.rpath_dirs() - - assert dirs == ["/rpath1", "/rpath2", "/rpath3"] - after - if original do - System.put_env("COMPRESSION_LIB_DIRS", original) - else - System.delete_env("COMPRESSION_LIB_DIRS") - end - end - end - - test "includes both ARM and Intel paths on macOS" do - case :os.type() do - {:unix, :darwin} -> - original = System.get_env("COMPRESSION_LIB_DIRS") - - try do - System.delete_env("COMPRESSION_LIB_DIRS") - dirs = CompressionConfig.rpath_dirs() - - # Should include both ARM (/opt/homebrew) and Intel (/usr/local) paths - arm_paths = Enum.filter(dirs, &String.contains?(&1, "/opt/homebrew")) - intel_paths = Enum.filter(dirs, &String.contains?(&1, "/usr/local")) - - assert Enum.empty?(arm_paths) == false, "Should include ARM Homebrew paths" - assert Enum.empty?(intel_paths) == false, "Should include Intel Homebrew paths" - after - if original, do: System.put_env("COMPRESSION_LIB_DIRS", original) - end - - _ -> - :ok - end - end - - test "returns empty list on Linux by default" do - case :os.type() do - {:unix, :linux} -> - original = System.get_env("COMPRESSION_LIB_DIRS") - - try do - System.delete_env("COMPRESSION_LIB_DIRS") - dirs = CompressionConfig.rpath_dirs() - - assert dirs == [] - after - if original, do: System.put_env("COMPRESSION_LIB_DIRS", original) - end - - _ -> - :ok - end - end - end - - describe "bzip2_static_lib/0" do - test "returns path to bzip2 static library" do - path = CompressionConfig.bzip2_static_lib() - assert is_binary(path) - assert String.ends_with?(path, "libbz2.a") - end - - test "respects BZIP2_STATIC_LIB environment variable" do - original = System.get_env("BZIP2_STATIC_LIB") - - try do - System.put_env("BZIP2_STATIC_LIB", "/custom/bzip2/libbz2.a") - path = CompressionConfig.bzip2_static_lib() - - assert path == "/custom/bzip2/libbz2.a" - after - if original do - System.put_env("BZIP2_STATIC_LIB", original) - else - System.delete_env("BZIP2_STATIC_LIB") - end - end - end - - test "includes homebrew prefix on macOS" do - case :os.type() do - {:unix, :darwin} -> - original = System.get_env("BZIP2_STATIC_LIB") - - try do - System.delete_env("BZIP2_STATIC_LIB") - path = CompressionConfig.bzip2_static_lib() - - assert String.contains?(path, "opt/bzip2") - assert String.ends_with?(path, "lib/libbz2.a") - after - if original, do: System.put_env("BZIP2_STATIC_LIB", original) - end - - _ -> - :ok - end - end - end - - describe "homebrew_prefix/0" do - test "returns homebrew prefix path" do - prefix = CompressionConfig.homebrew_prefix() - assert is_binary(prefix) - end - - test "respects HOMEBREW_PREFIX environment variable" do - original = System.get_env("HOMEBREW_PREFIX") - - try do - System.put_env("HOMEBREW_PREFIX", "/custom/homebrew") - prefix = CompressionConfig.homebrew_prefix() - - assert prefix == "/custom/homebrew" - after - if original do - System.put_env("HOMEBREW_PREFIX", original) - else - System.delete_env("HOMEBREW_PREFIX") - end - end - end - - test "detects homebrew prefix on macOS" do - case :os.type() do - {:unix, :darwin} -> - original = System.get_env("HOMEBREW_PREFIX") - - try do - System.delete_env("HOMEBREW_PREFIX") - prefix = CompressionConfig.homebrew_prefix() - - # Should be either ARM or Intel homebrew path - assert prefix in ["/opt/homebrew", "/usr/local"] or - String.starts_with?(prefix, "/") - after - if original, do: System.put_env("HOMEBREW_PREFIX", original) - end - - _ -> - :ok - end - end - - test "handles brew command failure gracefully" do - # This tests the fallback when brew command is not available - # The function should still return a valid path - prefix = CompressionConfig.homebrew_prefix() - assert is_binary(prefix) - assert String.length(prefix) > 0 - end - end - - describe "platform detection" do - test "detects correct operating system" do - os_type = :os.type() - - case os_type do - {:unix, :darwin} -> - # macOS - should have Homebrew paths - dirs = CompressionConfig.library_dirs() - assert Enum.empty?(dirs) == false - - {:unix, :linux} -> - # Linux - should have system paths - dirs = CompressionConfig.library_dirs() - assert Enum.any?(dirs, &String.contains?(&1, "/usr")) - - _ -> - # Other platforms - :ok - end - end - - test "handles different compression libraries" do - case :os.type() do - {:unix, :darwin} -> - dirs = CompressionConfig.library_dirs() - - # Should include paths for all compression libraries - expected_libs = [:zstd, :lz4, :snappy, :"c-blosc", :bzip2] - - for lib <- expected_libs do - lib_str = Atom.to_string(lib) - - assert Enum.any?(dirs, &String.contains?(&1, lib_str)), - "Should include path for #{lib}" - end - - _ -> - :ok - end - end - end - - describe "environment variable parsing" do - test "handles platform path lists correctly" do - original = System.get_env("COMPRESSION_LIB_DIRS") - sep = if match?({:win32, _}, :os.type()), do: ";", else: ":" - - try do - System.put_env("COMPRESSION_LIB_DIRS", Enum.join(["/path1", "/path2", "/path3"], sep)) - assert CompressionConfig.library_dirs() == ["/path1", "/path2", "/path3"] - - System.put_env("COMPRESSION_LIB_DIRS", "/single/path") - assert CompressionConfig.library_dirs() == ["/single/path"] - - System.put_env("COMPRESSION_LIB_DIRS", "") - assert CompressionConfig.library_dirs() == [] - - System.put_env( - "COMPRESSION_LIB_DIRS", - Enum.join(["/path1", "", "/path2", ""], sep) - ) - - assert CompressionConfig.library_dirs() == ["/path1", "/path2"] - after - if original do - System.put_env("COMPRESSION_LIB_DIRS", original) - else - System.delete_env("COMPRESSION_LIB_DIRS") - end - end - end - end - - describe "integration" do - test "library_dirs and rpath_dirs can be different" do - # On some platforms (like macOS), rpath_dirs includes more paths - lib_dirs = CompressionConfig.library_dirs() - rpath_dirs = CompressionConfig.rpath_dirs() - - case :os.type() do - {:unix, :darwin} -> - # macOS rpath should include both ARM and Intel paths - assert length(rpath_dirs) >= length(lib_dirs) - - {:unix, :linux} -> - # Linux: library_dirs has paths, rpath_dirs is empty - assert Enum.empty?(lib_dirs) == false - assert rpath_dirs == [] - - _ -> - :ok - end - end - - test "all paths are absolute" do - dirs = CompressionConfig.library_dirs() ++ CompressionConfig.rpath_dirs() - - for dir <- dirs do - assert String.starts_with?(dir, "/"), "Path #{dir} should be absolute" - end - end - end -end diff --git a/test/ex_zarr/compressor_config_test.exs b/test/ex_zarr/compressor_config_test.exs new file mode 100644 index 0000000..7e5550c --- /dev/null +++ b/test/ex_zarr/compressor_config_test.exs @@ -0,0 +1,205 @@ +defmodule ExZarr.CompressorConfigTest do + # async: false because some tests change the :default_zarr_version app env + use ExUnit.Case, async: false + + doctest ExZarr.Codecs.CompressorConfig + + alias ExZarr.Array + alias ExZarr.Codecs.CompressorConfig + + setup do + path = Path.join(System.tmp_dir!(), "ex_zarr_ccfg_#{System.unique_integer([:positive])}") + on_exit(fn -> File.rm_rf!(path) end) + {:ok, path: path} + end + + defp data(n \\ 4_000), + do: for(i <- 0..(n - 1), into: <<>>, do: <>) + + defp create(path, opts) do + ExZarr.create( + Keyword.merge( + [shape: {4_000}, chunks: {4_000}, dtype: :float64, storage: :filesystem, path: path], + opts + ) + ) + end + + defp json(path, file), do: path |> Path.join(file) |> File.read!() |> Jason.decode!() + + defp stored_chunk(array), do: ExZarr.Storage.read_chunk(array.storage, {0}) + + describe "default_zarr_version" do + setup do + previous = Application.get_env(:ex_zarr, :default_zarr_version) + + on_exit(fn -> + if previous, + do: Application.put_env(:ex_zarr, :default_zarr_version, previous), + else: Application.delete_env(:ex_zarr, :default_zarr_version) + end) + end + + test "unset: new arrays are v2" do + Application.delete_env(:ex_zarr, :default_zarr_version) + assert ExZarr.Version.default_version() == 2 + assert {:ok, %{version: 2}} = ExZarr.create(shape: {10}, chunks: {5}) + end + + test "configured: new arrays use it" do + Application.put_env(:ex_zarr, :default_zarr_version, 3) + + assert {:ok, %{version: 3, metadata: %ExZarr.MetadataV3{}}} = + ExZarr.create(shape: {10}, chunks: {5}) + end + + test "the zarr_version option still wins" do + Application.put_env(:ex_zarr, :default_zarr_version, 3) + assert {:ok, %{version: 2}} = ExZarr.create(shape: {10}, chunks: {5}, zarr_version: 2) + end + + test "invalid versions are errors, not crashes" do + assert {:error, {:invalid_zarr_version, 4}} = + ExZarr.create(shape: {10}, chunks: {5}, zarr_version: 4) + + Application.put_env(:ex_zarr, :default_zarr_version, "3") + + assert {:error, {:invalid_zarr_version, "3"}} = ExZarr.create(shape: {10}, chunks: {5}) + end + end + + describe "compressor_config on v2 arrays" do + test "settings reach .zarray and survive reopen", %{path: path} do + {:ok, array} = create(path, compressor: :zstd, compressor_config: [level: 19]) + :ok = Array.save(array, path: path) + + assert json(path, ".zarray")["compressor"] == %{"id" => "zstd", "level" => 19} + assert {:ok, reopened} = ExZarr.open(path: path) + assert reopened.metadata.compressor_config == [level: 19] + end + + test "blosc settings are applied and kept for later writes", %{path: path} do + {:ok, array} = + create(path, + compressor: :blosc, + compressor_config: [cname: :zstd, shuffle: :bit, typesize: 8] + ) + + :ok = Array.save(array, path: path) + :ok = Array.set_slice(array, data(), start: {0}, stop: {4_000}) + + # Blosc1 header: compressor format in bits 5-7 (zstd = 4), bitshuffle flag 0x04 + {:ok, <<2, 1, flags, 8, _::binary>>} = stored_chunk(array) + assert Bitwise.bsr(flags, 5) == 4 + assert Bitwise.band(flags, 0x04) == 0x04 + + assert json(path, ".zarray")["compressor"] == + %{ + "id" => "blosc", + "cname" => "zstd", + "clevel" => 5, + "shuffle" => 2, + "blocksize" => 0 + } + + # A later writer picks the settings up from .zarray. + {:ok, reopened} = ExZarr.open(path: path) + :ok = Array.set_slice(reopened, data(), start: {0}, stop: {4_000}) + {:ok, <<2, 1, flags2, _::binary>>} = stored_chunk(reopened) + assert Bitwise.bsr(flags2, 5) == 4 + assert {:ok, data()} == Array.get_slice(reopened, start: {0}, stop: {4_000}) + end + + test "zlib honours the level", %{path: path} do + sizes = + for level <- [0, 9] do + dir = Path.join(path, "l#{level}") + {:ok, array} = create(dir, compressor: :zlib, compressor_config: [level: level]) + :ok = Array.set_slice(array, data(), start: {0}, stop: {4_000}) + {:ok, chunk} = stored_chunk(array) + byte_size(chunk) + end + + assert [stored, compressed] = sizes + assert stored > byte_size(data()) + assert compressed < div(byte_size(data()), 10) + end + + test "maps are accepted", %{path: path} do + assert {:ok, array} = create(path, compressor: :bzip2, compressor_config: %{level: 3}) + assert array.metadata.compressor_config == [level: 3] + end + end + + describe "compressor_config on v3 arrays" do + test "settings become the codec configuration", %{path: path} do + {:ok, _} = + create(path, + zarr_version: 3, + compressor: :blosc, + compressor_config: [cname: :lz4, level: 7, shuffle: :bit, typesize: 8] + ) + + assert %{"name" => "blosc", "configuration" => config} = + List.last(json(path, "zarr.json")["codecs"]) + + assert config == %{ + "cname" => "lz4", + "clevel" => 7, + "shuffle" => "bitshuffle", + "typesize" => 8 + } + end + + test "gzip is a v3 compressor", %{path: path} do + {:ok, array} = + create(path, zarr_version: 3, compressor: :gzip, compressor_config: [level: 8]) + + assert %{"name" => "gzip", "configuration" => %{"level" => 8}} = + List.last(json(path, "zarr.json")["codecs"]) + + :ok = Array.set_slice(array, data(), start: {0}, stop: {4_000}) + {:ok, <<0x1F, 0x8B, _::binary>>} = stored_chunk(array) + end + end + + describe "validation" do + test "unknown or out-of-range settings are rejected" do + for {compressor, config, reason} <- [ + {:zstd, [level: 30], {:invalid_value, :level, 30}}, + {:zlib, [level: -1], {:invalid_value, :level, -1}}, + {:lz4, [level: 1], {:unknown_option, :level}}, + {:snappy, [level: 1], {:unknown_option, :level}}, + {:zstd, [cname: :lz4], {:unknown_option, :cname}}, + {:blosc, [shuffle: :sideways], {:invalid_value, :shuffle, :sideways}}, + {:blosc, [cname: "snappy"], {:invalid_value, :cname, "snappy"}}, + {:zstd, [:level], :not_a_keyword_list} + ] do + assert {:error, {:invalid_compressor_config, ^compressor, ^reason}} = + ExZarr.create( + shape: {10}, + chunks: {5}, + compressor: compressor, + compressor_config: config + ) + end + end + + test "unknown compressors fail at create, not at the first write" do + assert {:error, {:unsupported_codec, :gzip}} = + ExZarr.create(shape: {10}, chunks: {5}, compressor: :gzip) + + assert {:error, {:unsupported_codec, :brotli}} = + ExZarr.create(shape: {10}, chunks: {5}, compressor: :brotli, zarr_version: 2) + end + + test "v3 rejects compressors with no v3 codec instead of storing data uncompressed" do + assert {:error, {:unsupported_codec_for_v3, :snappy}} = + ExZarr.create(shape: {10}, chunks: {5}, compressor: :snappy, zarr_version: 3) + end + + test "custom codec settings pass through unchanged" do + assert {:ok, [threshold: 10]} = CompressorConfig.normalize(:my_codec, threshold: 10) + end + end +end diff --git a/test/ex_zarr_codecs_test.exs b/test/ex_zarr_codecs_test.exs index 71d7e1f..d0f3f7a 100644 --- a/test/ex_zarr_codecs_test.exs +++ b/test/ex_zarr_codecs_test.exs @@ -271,26 +271,4 @@ defmodule ExZarr.CodecsExtendedTest do assert compressed == decompressed end end - - describe "ZigCodecs module" do - alias ExZarr.Codecs.ZigCodecs - - test "zlib_compress works directly" do - data = "Direct zlib test" - assert {:ok, compressed} = ZigCodecs.zlib_compress(data) - assert byte_size(compressed) > 0 - end - - test "zlib_decompress works directly" do - data = "Direct zlib test" - {:ok, compressed} = ZigCodecs.zlib_compress(data) - assert {:ok, ^data} = ZigCodecs.zlib_decompress(compressed) - end - - test "handles compression errors gracefully" do - # zlib_decompress with invalid data should handle errors - result = ZigCodecs.zlib_decompress("invalid compressed data") - assert match?({:error, _}, result) - end - end end diff --git a/test/fixtures/legacy_codecs/blosc.bin b/test/fixtures/legacy_codecs/blosc.bin new file mode 100644 index 0000000000000000000000000000000000000000..ccaf72d7a163a4b7b5938c2276af690419c98f09 GIT binary patch literal 961 zcmd6m({>(c0094}vCS`N(AYMc#!edBwrw}I*%*y&+qUi8>~`N`=bSn71Op8P1tppb zf&AwH3~H%2N(Bmu*g_M6m+aS!qN)rdZsom-a%1` z>G@?fEnS0Pa$$Y{?DiEJ5t9JWGII$@%BlS{ws!IijEGOmE&1I9ItIq))^<;Bo*@Y6 zcx2R!oC4xMRJ4sO9X$NQ;!?7UK~+P0-{|bh_VLx@=ND955-J9EKC$mgS_bC!ZoZ*0 z$snt+qW(|M$n?_I;l;x{EDAOeJt*3lvz+-*V;WiwYYh3e*Xr8j73O6#{zhSr4-b4O>LaL zgCY~t^U7+Py9Or~*7weCU!V~&2mqjA<`R^YQ~had<>VO<5uch{Qr*K`n#!fU>wY^?VjE|Lm{B!lTkBq3P}7=(J``g z^zaXlOUWs&YHSC6W3wwe$5&6EU(j$#zA~`$i+xwtGO)0B^9zmnl~q^?>i_hPOfPR8 zUOv3Tpt=&UYiyH^$cW*FAScK$sEZo9U^6I)K zHqKr_ksu*Gue7GQb8upQeed-41qu-ZAJ8yy2};PR>KI!&dSV2G$EW6$R5y0?kIk*_ SoLoO0{_}rj`yTr5O8){7auVPG literal 0 HcmV?d00001 diff --git a/test/fixtures/legacy_codecs/bzip2.bin b/test/fixtures/legacy_codecs/bzip2.bin new file mode 100644 index 0000000000000000000000000000000000000000..f1506f1ef196fb972c0d9d4ea86d220a787f0277 GIT binary patch literal 897 zcmaitYe-XZ7{>oQ?w~tVg!3PfR2Cw+W9DQ9O687-mo`P$Ih&T38FoRuR3Mq%bV*dG zNRirw7#wG3|D0zfv&)Q*l%_^y$|}upX=a{o_QA|<78-_q>G|-y&+p;oebpDhv?90c z3R6-k0h?2nEpxvGV9C_~G?s{O5Ld2PcpTIg&)k@GRo85s4jz$n8Gs)3+-APlTm~NT z{}d`2LY4n?)A;w7Doe5nnP+x9)H74^hiB-wdsDl9*-+kJ8jDS@gl^$FN$_2kjczpR^N5F%lm~R9S-+178iN;InvM8Q zw+%pZ*Ez2M0;&As7a-Lp?r}C7EqyGqvC?izg}82!)uM8hJOP!jmnWj~D7g;LHrtZn zWtS!yRb3^XAdAi95cD^Rr^v#anh0yoTHUjft&7PlJjbEyE!nz(ZYtLIFsnJ}Ff|GK zK3Nl%>qxqW<&Gy^XQGD~-^zdy#@8Gm%A8E}N5;36&i6a-qVtXChv|HC1Av0*x}U+5 z3HC;*tH@)hdI2#gNsT2>qUuG&kiydD(p2Dk+zo2j?$VsX_ZH}eC8;+0hS3mAyq1I+ z6H?_hs3J>#4pro#uug7-3KO}JmT-`8423JWHzHdr9nEIzq@yCs$qI_=y+|8wq|yrL zn`4GdCipJw#hf9AUBr7T%|SbhtouO2EBFF)+6{%o$85_0=KN|c1iTc!#F|44Czv%p zaEQ^|!}kN;9}Z*9cEed_%{r96n|y>90X_rg7|jr>ZYL(Gj1Y1JKLq$Xl#xijMb*!UZ&b#fq|dIHe%XX8#x46& zdpj@)(@IZ|-05Fx F`~~6ET^;}c literal 0 HcmV?d00001 diff --git a/test/fixtures/legacy_codecs/crc32c.bin b/test/fixtures/legacy_codecs/crc32c.bin new file mode 100644 index 0000000000000000000000000000000000000000..8975dd46fbbfba50aafd82bd267e4fb05e557ce6 GIT binary patch literal 40004 zcmeI5cRP@67>6l)Wv@_H_RPpEg^Y@1@3L2BMnVZiA+sd1LiVQYQIsv4>@A~E>96+_ zyx-vW9iHPk?)$pW>pTfaX_z_rM9(W;*48(*alCu)LBzAf*V%;?^(~+KM`xF}ejO82 zGoI!ZmQuKMRnOSU!QD4FJUZdk+k&#%=C0mv(~Fz?e~2jQ*|`NJ=ZK>zo?ATWu2R*woZ5bLn59hz0P@G(a`#NU~G0}``14bY9svnckIpP_ z{X8b7Vm!?wEOp`1)f>iE_U^v-!=qokdYfNX+tk%NGQGI5|K}tnJsY>6guJT8bwdj~ zSMQ*QPvTQE-j!50c6=G0oL}EPIzdj$dgh$Ctg^bUf%$D`uYk~~*yK04MO7c#dIrbm zR(~84ke*`Zsmhb zjm|7>{rpQz#c-NONb17Ht2d0S?A?6tKZ<_w@@;-;ZBu9O$kgJ--k*~c^lWDZCFE2! zt{Ga~arF*-_#`ehBd?_TW5<`F$@#V2BSLaomNVzXWRef7ry0X}$q@c7{IhsPfte|Q{eaHPSJ24{Gj;cRb9l_*F^9*)6CR%M@PwCEytLw_6=(s_0-yz8X~5Eer2#w!cnq{8Xi3nLz=MGY z0}lod4;~&oJQhH(0D=V&Y&>D(2^&vX@xzKAR{XHjik(*Mv|@=HOWauEhP43J0$2;6 z>42sKnhtnr;H80=1|lYim|z}*c?{+;sP>@RgK7_MNw_89mV`_dGF8Y_VGo8q81`W3 zvZ2d{E*pM$_~GG)M-Tx)1OyRK06_r+1rQ`}ki0?i28|~)p3rzg)C^HGRQyo!L&Xo- zN@OdMtwg64omO;O5hg~M7-3?RxKZLpi5uy8r00>I?-`t!TipRm1C|CX4OkkmG+=4K z(txD_O9Pe$EDcy1ury$4z|w%F0ZRjx1}qI&8n85AX~5Eer2$I=mgfIh8kMB*|&p@D&um~v_ z*!jd{0oq2EjvoHuajCf_)y-XllM8Emr#CO(kcb%g^Goaw^)!R*s$l;qhs?r8Ujn zLz9c^duO*VKTwF61myH=yrMG7T80)5?tWphDLKVeO`QW1^Q*fj*Uw)N27=60^WA<@ZMg_RBM{bRE$+ecTAAK-APxFpofT!NAcYI>$NF5W>=i5Ugub#1*P zGr-Nmi~Dy_7!+(GDnS^gNJ`uTxw29byL^i#KPL{>CN*uBmxFL867K+h_sTXfw{e#Z)i+%c2Q+xNB{WT z%FglC(l%j^d znVqXoNOV$GVMRlG-`MQ(_R;0z2N)bG4hc0Am!O1#ny#shvv*KrVn%*>U2E^iG;s6a z;{FX31{s@(l97X7Tuw#D*viQ>AR<03ue7G6duVELegEwC^#=+GlYoMrjZahtpk-+3 z;O-w5o0?Nx-PAcaF~7EZa`XHJiGYqzM$5`0EUlzrU~cE;8xoV8RaDu~(LXk~vVDB@ z_z4b&hD$=j%q1kLpsr_X>*5_8m6%yjUfAESA1(h8&Hw-a literal 0 HcmV?d00001 diff --git a/test/fixtures/legacy_codecs/snappy.bin b/test/fixtures/legacy_codecs/snappy.bin new file mode 100644 index 0000000000000000000000000000000000000000..61b8c0e70d481e9e9b275cc8f5883b6c9630db67 GIT binary patch literal 2601 zcmeIuQ*tZ_006*bV%xTD+qP}nwr$(CZQHgnaq{tMZ?Gqrsy;!F_5wa$0bnr5*hG|! z9Q@*Ps=6lD&R&6$3F-M|wXHqF(@UF&7x!uiuae==fx`tUSWfN}7fi_U?Y6u_@WbRgIkkA8)xsJ$i$5N^19aE zk?G~F!^?-aKWHQ@LJ9_UJ~3Gp9b+pe&wz;dw7k-qmhPdc#f^jWJKzr_A_hJ=9V@Sh zw6d0=rGvYFSZr!eadlJY;Kcm;-s$bj7X&;S9w`kAw~&;ghJm@An{P-=a#m4gLr4GE z-0IHB_0uOfEGiB$H4~?xguJ?*sjZ85a8zPuL3w>!@950(_R;0z`yUK477--_2fvt{ zs*Z`ZlUHCwLV8|VZA;Ja)Y9g`#XayB3K5floSuzWR7P3b$kNfnKRhlqx1_qcYjAR5 zegEwC^&0{K9gmEbg-2LQNz=f>-pwyGCMCP5sX zZ)WT26C9nCSy)lu-ZwhCvVDB@_yGokf=xul$RQvur>1LSk1%OAxC8c5J7Lrud&^Nbp^$m$m&MK^IXzw4JUEMjp ze)<51MZqDaV&W8#kXO?)wQ=zdib~8VD6ebl9hq6)I=Xy#|AR)xBBW$s=NFSz)iJhq z@(hfKPtPl@ZRsAKTHHK1zX$$6Az~1a)3Nc2$S7+WSvq+5hsCAllvFo$4Nfeq@15Pg zd_%ya^r5wt!9BbBv_F&5MlWh8>A$uXtJh9Y0AXC-|toDb4_jGnyRsS zQQckh9sVEYrdzP2Y|)8HcTJsQ@|8zd_D(I+A-SW=Q=Uz~y7gO+kA6+W|GE#;TiRY; zkxbNdEtO?1w%#uNZz2DqN%2pnT{9`?_dU1&|CX)gES@7_ZX>j!P*W2H_ z9kcU1uemPkmCTHaukRjTpI`46Fel})puqIty%k4~+&X#MTtmsqPREq0%aanFJI`HGr<&?ZM_N_=V7XU4<%7=V1$)-5El)}Nc=*`2gZ4pEY3GcUE_F%m zViWt8Xa2s$u4c}hiQ!SVFWK6d?*CJn{eQ-cSrZ>`nee1?tM{+8ZR_rBcYb%=u6AB< z(A_Jh79Xo#KHh%U-erpV%}Q34C!t%{&$?kz{gc)Gh|1EvI;(QtypW$?z5CnK-*pRDp!1#`*C~E4ZnWTe0%ZUhw`5k-o5O;Gilbdbqh~k ze{WxV+gfQ+Qo?iLDFHhc)SkU^?CGxicTdbvf2rTSTFRRJSp44T-@iV8=j5s~ldCJV zoZ21Uvv#gcZS$eU!oi!lwWV(TK9+WWP2RgdN(zgUA6ltvEnc%SJ>}(thr$K#1OIG3 z^XJp9YxgF;Jj(z2HhaBi<_wGT9NyZiUQ5lmz5DJC;a?95KXvc9G;7+sg)4u4pTGTE zU*}2*%O5Q&qF3}{jW?Hm+*kX(?#<-qmPfDp&TST6Kh3W0?%(33Pb$*u8+Pn@b#%4b zwR3U%4hmlM%)IG3l~?v}a&dn6{y#rlJ^YTJs`c{;i+sIl;?%{**B+=2(uMp zRqp9x;bz{W$V-P_rLdhu8{Pf zD>lh${j$aRY0qE1l`pb?_ww_dNw1!*J9zTC_xyctUUad X@$(4`4+Z6hJNozk@PGC`3-3PwISCnc literal 0 HcmV?d00001 diff --git a/test/fixtures/legacy_codecs/zstd.bin b/test/fixtures/legacy_codecs/zstd.bin new file mode 100644 index 0000000000000000000000000000000000000000..ca84c68da82811a7d9dcbf3b944bc37a9b26dabc GIT binary patch literal 774 zcmV+h1NrB)ZF9h z^7#b~6dNNdF*!g-PgY%JY;${piIJF}r>?cV#LLs#;py=A1Pc-xAt*36K1NMcTw`i- zd4PzImY$`ow7bH}(b(YU?)Cx+5f~pQE;T(wOH*25X>fOchK-e+qpPyH!N||o-sSD| z00<8j9wjU^J3>lPSYT&vc7245lbWKbu(-d*&eh!H>+<>r4HX6AjNKaQ@Wo>kO zgNl)vpr^04y~WJa+v4f*_yi0S8X_q$I6p>CRa|6ia(aP?ke8mOt+c$u%F@~3=iOSYKyub$x`3lbWHau(!X)&DGrF>+<;q4HO(BDl$1hNKaN?Wo&eNf{Kxt zpr@|3y~NDZ+T!W(_yh|R8X+k#H$O&ARa|3ha(RJ>kC&dMt+czt%F)^2=kN9c3K19} zCoVQUL`+j!Vrg)Ae};~goTRI=y1~fM*WTvs^Z^JD7ak=oG&@5|QCVPTZ+3l!jFX$9 zsj;}f$IjK=bL1`ZV*Br7sHK}b+nUS@4{e1nRTnV_hzw!X#8)Z60f@%RM{6B{Ba zF*rX*PE}oGYjb*miIA6{rmeNS!^_gy;pp%80}2uuASf?3K1EDaTViT)cz}nFmYt-m zv%12`(AeMR?)3o)5EmaNEj2ttOHx^3Xm59Zg^iS(qpGpFz{k(l-R11_0RZusGW2wj E22uid`v3p{ literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/blosc_blosclz_shuffle.bin b/test/fixtures/numcodecs/blosc_blosclz_shuffle.bin new file mode 100644 index 0000000000000000000000000000000000000000..ccaf72d7a163a4b7b5938c2276af690419c98f09 GIT binary patch literal 961 zcmd6m({>(c0094}vCS`N(AYMc#!edBwrw}I*%*y&+qUi8>~`N`=bSn71Op8P1tppb zf&AwH3~H%2N(Bmu*g_M6m+aS!qN)rdZsom-a%1` z>G@?fEnS0Pa$$Y{?DiEJ5t9JWGII$@%BlS{ws!IijEGOmE&1I9ItIq))^<;Bo*@Y6 zcx2R!oC4xMRJ4sO9X$NQ;!?7UK~+P0-{|bh_VLx@=ND955-J9EKC$mgS_bC!ZoZ*0 z$snt+qW(|M$n?_I;l;x{EDAOeJt*3lvz+-*V;WiwYYh3e*Xr8j73O6#{zhSr4-b4O>LaL zgCY~t^U7+Py9Or~*7weCU!V~&2mqjA<`R^YQ~had<>VO<5uch{Qr*K`n#!fU>wY^?VjE|Lm{B!lTkBq3P}7=(J``g z^zaXlOUWs&YHSC6W3wwe$5&6EU(j$#zA~`$i+xwtGO)0B^9zmnl~q^?>i_hPOfPR8 zUOv3Tpt=&UYiyH^$cW*FAScK$sEZo9U^6I)K zHqKr_ksu*Gue7GQb8upQeed-41qu-ZAJ8yy2};PR>KI!&dSV2G$EW6$R5y0?kIk*_ SoLoO0{_}rj`yTr5O8){7auVPG literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/blosc_lz4_bitshuffle.bin b/test/fixtures/numcodecs/blosc_lz4_bitshuffle.bin new file mode 100644 index 0000000000000000000000000000000000000000..e881ca03e0f5fa1aca307600c133897da3c851a3 GIT binary patch literal 4607 zcmaKwZEVxm6~@o?OALYJ|94(iq8dADlA@*?5WJ$myrj~aw4pT`hGZM7w2ll%gE9hC z$WkK+%Cro|s0yTw(^QsdXd9<|pn{HlT0Ao{Dl1#3qE=;6oduD7XtjSx)*3Ctd!B2X zl@YDdsqf1!C(~zqZoXS%-Y{zFo3~jWWLARk1=S)AQX`K^}CS%Eo5C~T^$C$#o!$4nErCU zZawQh!OCXVA@e#C+J1(?qZk|wh~+wu^j{-uq)_)E1_v*Q_+`Fg6SDUs`-d!h9)lW| z@kpL@5rfwQ;yE=V5|E|(-+r3A2q1~M9fT$X^1j$ze zf)5ptLH4Ef)hNq#k1*X2g>|SE7j>DlMmcB1^L$>`>nPPq(22HEUqe6~lgfM$?T)d5 zCB?QC>vf%@MqzHAyC9BYn@lR4{-s%P^6trZoz#}pD03u$+ol`l!3de4qmoVR*()V8 z@_qdPCKbdvW~8LpW?n|#ULK-mBzLGWYs>!6KkMe|PE0qRNbd7Ye1Mp3#Ejvc*ynK} zCa^i3MnpWAF#h}M?OIC^xu%p%pNkB$ulW&woQbXjrkLwB!X zA=!r{ikM~)`vS2(g=Aw)Y^LjLB0Izp5%XCP*3c{Gm{c@j;unGSe@}4Pw{lFJW~o+y zl!{A!@jNglW{+Nxd8Sr;fT9{jtTT5Ne!+!E4`o$kB4n#cCGRV81D^CqdOLGlDF%wMHNc=}Q4AB%0GP#SNT*2Z3|%zu#Y4gNM?tLyeZbc`c$lbHtaq zjhG!b7RDLaOiv?b?9Emm!j8=&{fJMoX)qiBlp>n2+1!L!0SF_4%>_m`AkPtmO=}J@ zO$Z|azy#t?*mN;$fEbA1VbkYDECIlS=y^-F_JFY#S5|LfL(7~h$ef}?!~`WW`T%$d z5u?OX1Z8^L#Hz+-a{yri(nt_C7cvM+)G}xDGLw`zk0+9(M8r#!Si}It*Ct!3(t}G5 z;5v55yek4kHQ3$kRQm`-q65Df+C!B07X;G781&dEa5)3qCd8qTvg&vt6d0kve?y#w zcpRH)hV~GkJ_=-L521mj)K|(T8MIK}2M*|Aqx6vXL6sKcU#Oq5_y8>%uS2`YJ}GoN zlxr8+C|#sb4@J()+-kbWSF@HP8Mkpjx*(KS@+BZif+&(#qFrRiMR3g<;+L%4(`qvt z^$_|6ljmA(mQ7y*$`ULEb36tu{{XX};`mNnh!}-+sPg3<_`bZQzX|ZbwfI}^P%5UL zfSf*jRbtBgGGutl4x3!;*x3?hb&egY{dj}EU z)jrDDwW)nCPW7yLBuyk0eG7rK#mc#9grUo zbS)lYTssZG$pRz}$lCy%MO3x}paYOL#3PU}Mi8pwLj;8Y*h0`iyrU8^Z~=gK35f&1 zcm#mD0tCIZZvo_K#G{1h0OTY==%s_8c9`)P;^GJZX9$5A^^Bhau#*sN1|VAj*i?X^ zm-rb#ULgn)wh>gK=2zFjl(X8=ysNwDDjjVpxVjAxI{H2#+6+M61myHEftHn#b^gH@gKN9V#@yYtX1} zWRD_3%3Z0yhzS(6(J{nWi8@8|=~t+QuXLo6R?+t}n)G`*o-=85S4jOFMNgx-`q%K7 z&!Fri-C@uj#$!-gM5}4DwwmtH>X5R(V>=x}wYH^0%~rpO%2(*E_7`pa{}=nY4ZSV$ z=6B9zpBUWW?+RZ@MWY-1;nvAiw4uQto=hE&Huw*OCyyVG`VZ_6V;c2$hQq1hXqCV7 z=8e>FRk^?GW-2vU<@a}8N&OL!RRsvDqb~!pc@mHYKv3-mEOmSskR5qQ_$DA#fZQlR zQh@jYxe6r@69NfgNC50Wcyj>)3I3W+K!%}Y2Ov+ZyDVo7eb&~H(V%@k`l>P~e`K@1Nn_}8XTfC{LzmIq z;2>QF0=0BGx`8fh1Wkfum;;wvq&l1min}wWUMSbj4u@1(RcNsne0u)R=|r->-(`5c zO@T6Z4O7v+}9;oQnn(u;8n4WUtYJ8MTP9n#_G@$!WyeMYrF;PJ=lW^4+l-tf+ z<9#lzhK;AwiJ+8()I*6rfsC6Yg;| zXM&+kxX00+peU#E8sPQ3DBdZ2x)6;{Wyde>Y7wY|Laf<7lJ2}KPF3x4T@tmyR)lBh zR1Q>t05vvUbe=x){?R}9=)h&511l&?Ldv0gDvl|v^ME1UR?~=Wk>A3DR=wP@XIvR0 z?Hh0IJuF#uM=iK4!^%&wyI{{?n#OK#R_N#o z(i$Ne;nIlape%PE9+K(>BsfL|EjVu#k2jA)M(!X3G)!}FognUPP^vR7RE;<9W{IoB zsmgjUiE~Iw;Q?;E+QPEqS+NgENl?n*u=);-tIsOEf*VJ|64bfxgjA7U!xtUa3G#9& z*jvDSMqqoT5mOboM0X~n@K)tQ$|643%U1DW@GM+swawn@K z1HaPQX6emlxoh{zyICwVFP-&yX@1tOv&NAn*bB;TpJo|e#&aqW~pJ)I4uI}tV D3y6;@ literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/blosc_zstd_noshuffle.bin b/test/fixtures/numcodecs/blosc_zstd_noshuffle.bin new file mode 100644 index 0000000000000000000000000000000000000000..59bcdb10d4c0863ca0781493e01d49ac83bcef58 GIT binary patch literal 798 zcmV+(1L6Dv0gwSeoB#kooB#kG0{{RN000060{{RhwJ-f(K%3B)ZF9h^7#b~6dNNdF*!g-PgY%JY;${piIJF}r>?cV z#LLs#;py=A1Pc-xAt*36K1NMcTw`i-d4PzImY$`ow7bH}(b(YU?)Cx+5f~pQE;T(w zOH*25X>fOchK-e+qpPyH!N||o-sSD|00<8j9wjU^J3>lPSYT&vc7245lbWKbu(-d* z&eh!H>+<>r4HX6AjNKaQ@Wo>kOgNl)vpr^04y~WJa+v4f*_yi0S8X_q$I6p>C zRa|6ia(aP?ke8mOt+c$u%F@~3=iOSYKyub$x`3lbWHau(!X)&DGrF z>+<;q4HO(BDl$1hNKaN?Wo&eNf{Kxtpr@|3y~NDZ+T!W(_yh|R8X+k#H$O&ARa|3h za(RJ>kC&dMt+czt%F)^2=kN9c3K19}CoVQUL`+j!Vrg)Ae};~goTRI=y1~fM*WTvs z^Z^JD7ak=oG&@5|QCVPTZ+3l!jFX$9sj;}f$IjK=bL1`ZV*Br7sHK}b+nUS@4{ ze1nRTnV_hzw!X#8)Z60f@%RM{6B{BaF*rX*PE}oGYjb*miIA6{rmeNS!^_gy;pp%8 z0}2uuASf?3K1EDaTViT)cz}nFmYt-mv%12`(AeMR?)3o)5EmaNEj2ttOHx^3Xm59Z cg^iS(qpGpFz{k(l-R11_0RZusGW2wj1}uzvg#Z8m literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/bz2.bin b/test/fixtures/numcodecs/bz2.bin new file mode 100644 index 0000000000000000000000000000000000000000..3ace1c627ff50698a60b224766a6214f5d1d3173 GIT binary patch literal 889 zcmajYYe-XJ7zglo#~pHql4!mWNo66DJEcxm;HdOOcxg*?owIqFy236d|#~rFYYp>h&t(W|z{TEk?>-ZK-?-sxhOOg~yP1UqIZKWZa`$E&9F8ib;6hqgjXV@Y(^i^qvh2 zCy*+1QxH;Zhk1g zQC$Iwy~B-Cu_8Cxr3NY0NUh*ri)^EGET3(Xj)^QME2(G>B<;9~$}XYXCkxecSWuk(ikI~%1_W?c( zj$j=Q!x<)OHOkpTKETTWzX6V6*VMMM3SN(L9OM9A4){Dc!Dyb~FWx3yAm#;JA6{ho3Ey8_LI3~& literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/generate.py b/test/fixtures/numcodecs/generate.py new file mode 100644 index 0000000..42e0e89 --- /dev/null +++ b/test/fixtures/numcodecs/generate.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""Regenerate numcodecs (zarr-python) codec fixtures for ExZarr. + + pip install numcodecs + python test/fixtures/numcodecs/generate.py + +Payload: 40_000 bytes, byte i = (i * 7 + i // 3) % 251 (same as the tests). +""" +from pathlib import Path + +import numcodecs +from numcodecs import BZ2, LZ4, Blosc, Zstd + +HERE = Path(__file__).parent +payload = bytes((i * 7 + i // 3) % 251 for i in range(40_000)) + +codecs = { + "zstd": Zstd(level=3), + "lz4": LZ4(), + "bz2": BZ2(level=9), + "blosc_blosclz_shuffle": Blosc(cname="blosclz", clevel=5, shuffle=Blosc.SHUFFLE), + "blosc_lz4_bitshuffle": Blosc(cname="lz4", clevel=5, shuffle=Blosc.BITSHUFFLE), + "blosc_zstd_noshuffle": Blosc(cname="zstd", clevel=5, shuffle=Blosc.NOSHUFFLE), +} + +for name, codec in codecs.items(): + (HERE / f"{name}.bin").write_bytes(bytes(codec.encode(payload))) + +print("numcodecs", numcodecs.__version__, "->", sorted(codecs)) diff --git a/test/fixtures/numcodecs/lz4.bin b/test/fixtures/numcodecs/lz4.bin new file mode 100644 index 0000000000000000000000000000000000000000..59b660fbe178dea732e12f6ea73f9a0e7a410fa2 GIT binary patch literal 923 zcmd7RQ+6Bx06@`+ZQHh;Ol;e>ZQHhO+qP}n=1wz7+X7e21dlE=atsBbPrE0ZXTT9zX70;FbOH>+4)3dl(md39X$NQ;!<-;s++n7 zCl=QC&Te0RAQ3R|$>>;lgryWU4b1J`d_!ZBvx_PlJNn1xR(DUXpTEH2(Qrv=n7M@{ z6x8+1Y+ZbUqmnWUD(c(%MrT&GkFFj+z+h2uh^d%31;pi4bxo|Dy#gZ>((}t|TYHA5 zmo^VC?%zS6k+BFV8QA$nWmL3{tQmZ=Sy);L-6&X<4{|k_sC7W_GSVA<;=$g%u6$ePgpL zJI7Z~pI~sPI3(0eT!P~AYPzO2&fYZZ=YiTU-t z)7zJCNCb3zGFnz1pp>G9fw`TVZ%9mXR#9a`NB`K|>dwja(-$}#8ZHS9GnbHrytYw+}iHR&GQ!oJQ^M;4GXuBq=LG>nXRi&aCA~;VMTp=-{|bh_VLx@ zCm1X$4ly+or=Yl;ny!hBvsX}LLPmaBU2D(C^wQSh#lt%Y3^EoGB?AY)sH}>%v6Z7| gKzMvwZfQ+(_t50x#{T);>oe$I_Wx%~O8>t311gIV&Hw-a literal 0 HcmV?d00001 diff --git a/test/fixtures/numcodecs/zstd.bin b/test/fixtures/numcodecs/zstd.bin new file mode 100644 index 0000000000000000000000000000000000000000..ca84c68da82811a7d9dcbf3b944bc37a9b26dabc GIT binary patch literal 774 zcmV+h1NrB)ZF9h z^7#b~6dNNdF*!g-PgY%JY;${piIJF}r>?cV#LLs#;py=A1Pc-xAt*36K1NMcTw`i- zd4PzImY$`ow7bH}(b(YU?)Cx+5f~pQE;T(wOH*25X>fOchK-e+qpPyH!N||o-sSD| z00<8j9wjU^J3>lPSYT&vc7245lbWKbu(-d*&eh!H>+<>r4HX6AjNKaQ@Wo>kO zgNl)vpr^04y~WJa+v4f*_yi0S8X_q$I6p>CRa|6ia(aP?ke8mOt+c$u%F@~3=iOSYKyub$x`3lbWHau(!X)&DGrF>+<;q4HO(BDl$1hNKaN?Wo&eNf{Kxt zpr@|3y~NDZ+T!W(_yh|R8X+k#H$O&ARa|3ha(RJ>kC&dMt+czt%F)^2=kN9c3K19} zCoVQUL`+j!Vrg)Ae};~goTRI=y1~fM*WTvs^Z^JD7ak=oG&@5|QCVPTZ+3l!jFX$9 zsj;}f$IjK=bL1`ZV*Br7sHK}b+nUS@4{e1nRTnV_hzw!X#8)Z60f@%RM{6B{Ba zF*rX*PE}oGYjb*miIA6{rmeNS!^_gy;pp%80}2uuASf?3K1EDaTViT)cz}nFmYt-m zv%12`(AeMR?)3o)5EmaNEj2ttOHx^3Xm59Zg^iS(qpGpFz{k(l-R11_0RZusGW2wj E22uid`v3p{ literal 0 HcmV?d00001 diff --git a/test/mix/tasks/fix_nif_rpaths_test.exs b/test/mix/tasks/fix_nif_rpaths_test.exs deleted file mode 100644 index 26b7876..0000000 --- a/test/mix/tasks/fix_nif_rpaths_test.exs +++ /dev/null @@ -1,175 +0,0 @@ -defmodule Mix.Tasks.FixNifRpathsTest do - use ExUnit.Case, async: false - - import ExUnit.CaptureIO - - alias Mix.Tasks.FixNifRpaths - - describe "run/1" do - @tag :skip - test "runs on macOS" do - case :os.type() do - {:unix, :darwin} -> - output = capture_io(fn -> FixNifRpaths.run([]) end) - assert output =~ ~r/(Adding rpaths|No Zig NIF library found|All rpaths already present)/ - - _ -> - :skip - end - end - - test "skips on non-macOS platforms" do - output = - capture_io(fn -> - case :os.type() do - {:unix, :darwin} -> - FixNifRpaths.run([]) - - _ -> - Mix.shell().info("Skipping rpath fix (not macOS)") - end - end) - - assert is_binary(output) - end - - test "handles missing NIF library gracefully" do - case :os.type() do - {:unix, :darwin} -> - output = capture_io(fn -> FixNifRpaths.run([]) end) - assert output =~ ~r/(Adding rpaths|No Zig NIF library found|All rpaths already present)/ - - _ -> - output = capture_io(fn -> FixNifRpaths.run([]) end) - assert output =~ "Skipping rpath fix (not macOS)" - end - end - - test "accepts any arguments" do - output = capture_io(fn -> FixNifRpaths.run(["arg1", "arg2"]) end) - assert is_binary(output) - end - end - - describe "platform detection" do - test "correctly identifies platform" do - os_type = :os.type() - - case os_type do - {:unix, :darwin} -> - assert elem(os_type, 0) == :unix - assert elem(os_type, 1) == :darwin - - {:unix, _} -> - assert elem(os_type, 0) == :unix - refute elem(os_type, 1) == :darwin - - {:win32, _} -> - assert elem(os_type, 0) == :win32 - - _ -> - assert is_tuple(os_type) - end - end - - test "task behaves correctly on current platform" do - output = capture_io(fn -> FixNifRpaths.run([]) end) - - case :os.type() do - {:unix, :darwin} -> - assert output =~ ~r/(Adding rpaths|No Zig NIF library found|All rpaths already present)/ - - _ -> - assert output =~ "Skipping rpath fix (not macOS)" - end - end - end - - describe "documentation" do - test "has module documentation" do - {:docs_v1, _, :elixir, _, module_doc, _, _} = Code.fetch_docs(Mix.Tasks.FixNifRpaths) - assert module_doc != :hidden - assert module_doc != :none - end - - test "has shortdoc" do - shortdoc = Mix.Task.shortdoc(Mix.Tasks.FixNifRpaths) - assert is_binary(shortdoc) - assert String.length(shortdoc) > 0 - end - end - - describe "idempotency" do - test "running multiple times is safe" do - output1 = capture_io(fn -> FixNifRpaths.run([]) end) - output2 = capture_io(fn -> FixNifRpaths.run([]) end) - - assert is_binary(output1) - assert is_binary(output2) - - case :os.type() do - {:unix, :darwin} -> - if String.contains?(output2, "All rpaths already present") do - assert output2 =~ "All rpaths already present ✓" - end - - _ -> - assert output1 =~ "Skipping rpath fix (not macOS)" - assert output2 =~ "Skipping rpath fix (not macOS)" - end - end - end - - describe "Mix.Project integration" do - test "can access Mix.Project.build_path" do - build_path = Mix.Project.build_path() - assert is_binary(build_path) - assert String.length(build_path) > 0 - end - - test "can access Mix.Project.config" do - config = Mix.Project.config() - assert is_list(config) - assert Keyword.has_key?(config, :app) - assert config[:app] == :ex_zarr - end - - test "constructs valid NIF path" do - build_dir = Mix.Project.build_path() - app_name = Mix.Project.config()[:app] - - nif_path = - Path.join([ - build_dir, - "lib", - to_string(app_name), - "priv", - "lib", - "Elixir.ExZarr.Codecs.ZigCodecs.so" - ]) - - assert is_binary(nif_path) - assert String.ends_with?(nif_path, ".so") - assert String.contains?(nif_path, "ex_zarr") - end - end - - describe "error handling" do - test "handles invalid paths gracefully" do - output = capture_io(fn -> FixNifRpaths.run([]) end) - assert is_binary(output) - end - - test "handles missing dependencies gracefully" do - case :os.type() do - {:unix, :darwin} -> - assert System.find_executable("otool") != nil - assert System.find_executable("install_name_tool") != nil - - _ -> - output = capture_io(fn -> FixNifRpaths.run([]) end) - assert output =~ "Skipping rpath fix (not macOS)" - end - end - end -end diff --git a/test/support/python_fixtures/generate_fixtures.py b/test/support/python_fixtures/generate_fixtures.py index 2b88a56..3ae39a3 100644 --- a/test/support/python_fixtures/generate_fixtures.py +++ b/test/support/python_fixtures/generate_fixtures.py @@ -20,11 +20,31 @@ def write_v2(out: Path) -> list[dict]: + from numcodecs import BZ2, LZ4, Blosc, Zstd + + fixtures = [] path = out / "v2_float64_2d" data = np.arange(20, dtype="float64").reshape(4, 5) arr = zarr.open(str(path), mode="w", shape=data.shape, chunks=(2, 5), dtype="float64") arr[:] = data - return [entry("v2_float64_2d", path, data)] + fixtures.append(entry("v2_float64_2d", path, data)) + + # One array per numcodecs compressor ExZarr supports. + data = (np.arange(400, dtype=" list[dict]: