diff --git a/CHANGELOG.md b/CHANGELOG.md index 332a1a3..1e4ce90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.1] - 2026-10-09 + +Patch release from the v1.3.0 review: faster Nx chunk and batch streams, a +shuffle that leaves the caller's random state alone, livebooks that run, and +docs that match measured behaviour. No public API was removed or renamed. + ### Fixed - `ExZarr.Nx.stream_chunk_tensors/2` crops padded edge chunks with one contiguous copy per row instead of one copy per element. A 999x1000 float32 @@ -32,6 +38,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 environment carried forward, like Livebook, names the failing cell, and keeps the end of long output. `--local` runs notebooks against this checkout instead of the Hex release. CI runs `mix livebook.test --local`. +- Livebooks, notebooks and guide install snippets use `{:ex_zarr, "~> 1.3"}`. +- The Nx guide and `ExZarr.Nx` docs replace unbenchmarked speed claims + ("5-10x", "400-800 MB/s") with measured timings, and the chunk-reduction + example uses `include_missing: true` and `on_error: :halt` so its total is + correct. +- The v1.3.0 entries in this changelog, the release notes, and the roadmap now + say that Explorer streaming was planned for 1.3 and dropped. + +### Tests +- The streaming memory test measures the test process instead of the whole VM, + so other async tests no longer move the result; the limit is now 2 MB. +- New tests cover `stream_chunk_tensors/2` options (`:include_missing`, + `:on_error`, `:backend`, unordered concurrency), 3D and 4D edge crops, seeded + shuffle orders matching 1.3.0, and cell-by-cell livebook evaluation. ## [1.3.0] - 2026-10-08 @@ -932,7 +952,8 @@ None - Full backward compatibility maintained with v2 arrays --- -[Unreleased]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.0...HEAD +[Unreleased]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.1...HEAD +[1.3.1]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.0...v1.3.1 [1.3.0]: https://github.com/AnimaLogica/ExZarr/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.2.0 [1.1.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.1.0 diff --git a/README.md b/README.md index df92477..5d26ad3 100644 --- a/README.md +++ b/README.md @@ -874,6 +874,10 @@ Key modules: See [ROADMAP.md](docs/ROADMAP.md) for the full release plan. +**v1.3.1 (released 2026-10-09)** - Patch release: faster Nx chunk crops and +shuffled batches, livebooks checked in CI, measured performance figures. See +[release notes](docs/release_notes_v1_3_1.md). + **v1.3.0 (released 2026-10-08)** - Data science interop: `ExZarr.Nx.stream_chunk_tensors/2`, DataLoader batch recipes, livebook MVP (metadata, Nx conversion, minibatches, a small Axon train). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b486a80..cbace5f 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -35,6 +35,12 @@ Released 2026-10-08. - [x] Livebook MVP: metadata, Zarr↔Nx, minibatches, Axon toy train - [ ] ~~Explorer direct streaming~~ Dropped from 1.3: columnar data and Explorer belong in [ex_arrow](https://hex.pm/packages/ex_arrow), and Binsparse in [ex_graphblas](https://hex.pm/packages/ex_graphblas) +### v1.3.1 + +Released 2026-10-09. Patch release: faster edge-chunk crops and shuffled +batching, shuffling that leaves the caller's `:rand` state alone, runnable +livebooks checked in CI, and measured performance figures in the Nx docs. + --- ## v1.4.0 (Planned) - Performance & Packaging diff --git a/docs/guides/installation.md b/docs/guides/installation.md index ddd9042..b7c406a 100644 --- a/docs/guides/installation.md +++ b/docs/guides/installation.md @@ -181,7 +181,7 @@ If tests pass, your installation is complete and working correctly. ```elixir iex> Application.spec(:ex_zarr, :vsn) |> to_string() -"1.3.0" +"1.3.1" ``` ## Build Troubleshooting diff --git a/docs/guides/migration_from_python.md b/docs/guides/migration_from_python.md index 3b326f8..e32bb6b 100644 --- a/docs/guides/migration_from_python.md +++ b/docs/guides/migration_from_python.md @@ -38,7 +38,7 @@ pip install zarr # In mix.exs def deps do [ - {:ex_zarr, "~> 1.2"} + {:ex_zarr, "~> 1.3"} ] end ``` diff --git a/docs/guides/storage_providers.md b/docs/guides/storage_providers.md index dc9b974..9b38844 100644 --- a/docs/guides/storage_providers.md +++ b/docs/guides/storage_providers.md @@ -157,7 +157,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:ex_aws, "~> 2.5"}, {:ex_aws_s3, "~> 2.5"}, {:sweet_xml, "~> 0.7"} # For XML response parsing @@ -353,7 +353,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:goth, "~> 1.4"}, # Authentication {:req, "~> 0.6.1"} # HTTP client ] @@ -490,7 +490,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:azure_sdk, "~> 0.4.1"} ] end @@ -641,7 +641,7 @@ Stores large chunks in MongoDB using GridFS. ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:mongodb_driver, "~> 1.4"} ] end diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 568eccd..736d6a4 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -1244,7 +1244,7 @@ GitHub Issues: [https://github.com/AnimaLogica/ExZarr/issues](https://github.com ```markdown ## Environment -- ExZarr version: 1.3.0 +- ExZarr version: 1.3.1 - Elixir version: 1.19.5 - Erlang/OTP version: 28.1 - OS: macOS 15.2 (arm64) diff --git a/docs/livebooks/06_cookbook/06_05_geospatial.livemd b/docs/livebooks/06_cookbook/06_05_geospatial.livemd index 0653f45..0f24b6b 100644 --- a/docs/livebooks/06_cookbook/06_05_geospatial.livemd +++ b/docs/livebooks/06_cookbook/06_05_geospatial.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"} + {:ex_zarr, "~> 1.3"} # {:ex_zarr, path: Path.join(__DIR__, "../../..")}, ]) ``` diff --git a/docs/livebooks/PLAN.md b/docs/livebooks/PLAN.md index 516854e..f5da698 100644 --- a/docs/livebooks/PLAN.md +++ b/docs/livebooks/PLAN.md @@ -1,6 +1,6 @@ -## Status (v1.3.0) +## Status (v1.3.1) -Shipped for 1.3.0: +Shipped in 1.3.0; in 1.3.1 every notebook below runs in CI against the current code: - `01_core_zarr/01_01` through `01_04` - `03_nx_ml/03_01_zarr_to_nx.livemd` @@ -13,7 +13,7 @@ Shipped for 1.3.0: Deferred past 1.3.0: `02_concurrency`, `04_02`–`04_15`, `05_02`–`05_12`, `06_crypto` through `09_systems`, and the rest of the table below. Arrow and Explorer livebooks belong with ex_arrow. Binsparse livebooks belong with ex_graphblas. -The livebooks directry should be a curated collection of **Elixir Livebooks** demonstrating how to use +The livebooks directory should be a curated collection of **Elixir Livebooks** demonstrating how to use **ExZarr** — a Zarr v3–compatible array storage library — to build scalable data systems for **AI / GenAI, finance, crypto, and scientific workloads**. diff --git a/docs/livebooks/scripts/00_01_welcome.exs b/docs/livebooks/scripts/00_01_welcome.exs index 8ae5f60..4ad179f 100644 --- a/docs/livebooks/scripts/00_01_welcome.exs +++ b/docs/livebooks/scripts/00_01_welcome.exs @@ -3,7 +3,7 @@ # Title: Welcome to ExZarr # Mix.install([ -# # {:ex_zarr, "~> 1.2"}, +# # {:ex_zarr, "~> 1.3"}, # {:ex_zarr, path: "/Users/thanos/work/ExZarr"}, # {:nx, "~> 0.7"}, # {:kino, "~> 0.13"}, diff --git a/docs/livebooks/scripts/01_01_first_zarr_array.exs b/docs/livebooks/scripts/01_01_first_zarr_array.exs index 0b4ce5e..c69b58a 100644 --- a/docs/livebooks/scripts/01_01_first_zarr_array.exs +++ b/docs/livebooks/scripts/01_01_first_zarr_array.exs @@ -3,7 +3,7 @@ # Title: Your First Zarr Array Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:kino, "~> 0.13"} ]) diff --git a/docs/livebooks/scripts/01_02_metadata_and_chunks.exs b/docs/livebooks/scripts/01_02_metadata_and_chunks.exs index 03ff6be..101ad3b 100644 --- a/docs/livebooks/scripts/01_02_metadata_and_chunks.exs +++ b/docs/livebooks/scripts/01_02_metadata_and_chunks.exs @@ -3,7 +3,7 @@ # Title: Metadata and Chunks: Inside a Zarr Array Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:kino, "~> 0.13"}, {:jason, "~> 1.4"} ]) diff --git a/docs/notebooks/earthmover_datacube.livemd b/docs/notebooks/earthmover_datacube.livemd index 77c35b4..7b093f9 100644 --- a/docs/notebooks/earthmover_datacube.livemd +++ b/docs/notebooks/earthmover_datacube.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"}, {:kino_vega_lite, "~> 0.1"} diff --git a/docs/notebooks/xarray_zarr_intro.livemd b/docs/notebooks/xarray_zarr_intro.livemd index a1d2435..aef7bf8 100644 --- a/docs/notebooks/xarray_zarr_intro.livemd +++ b/docs/notebooks/xarray_zarr_intro.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"} ]) diff --git a/docs/notebooks/zarr_fundamentals.livemd b/docs/notebooks/zarr_fundamentals.livemd index 6d94c5e..aed155f 100644 --- a/docs/notebooks/zarr_fundamentals.livemd +++ b/docs/notebooks/zarr_fundamentals.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"}, {:kino_vega_lite, "~> 0.1"} diff --git a/docs/release_notes_v1_3_1.md b/docs/release_notes_v1_3_1.md new file mode 100644 index 0000000..a30525b --- /dev/null +++ b/docs/release_notes_v1_3_1.md @@ -0,0 +1,48 @@ +# v1.3.1 Release Notes + +ExZarr v1.3.1 is a patch release. It fixes performance and correctness issues +found reviewing v1.3.0, and makes every shipped livebook run. Upgrading from +1.3.0 needs no code changes. + +```elixir +{:ex_zarr, "~> 1.3"} +``` + +## Nx + +- `ExZarr.Nx.stream_chunk_tensors/2` crops padded edge chunks one contiguous + row at a time. A 999x1000 float32 edge chunk took about 300 ms in 1.3.0 and + now takes under 1 ms. +- An array whose dtype has no Nx type yields a single `{:error, reason}` from + `stream_chunk_tensors/2` instead of one error per chunk. +- `ExZarr.Nx.DataLoader.shuffled_batch_stream/3` and + `paired_shuffled_batch_stream/4` group the shuffled indices in one pass. In + 1.3.0 the index slicing alone grew with the square of the sample count. +- Shuffling no longer changes the calling process's `:rand` state. The same + `:seed` still gives the same order as in 1.3.0. +- `to_tensor_chunked/3` applies `:names`, as `to_tensor/2` does. + +## Livebooks + +- `01_04_codecs_and_pipelines` ran into `:gzip` with a Zarr v2 array and + `compressor: nil`; it now uses `:zlib` and `:none`. `:gzip` is a Zarr v3 + codec. +- `03_03_training_from_zarr` trains on an independent second feature from + zero-initialized weights with Adam, so it reliably reaches the line + (`weights` about `[2.0, 0.0]`, `bias` about `1.0`) and prints the same numbers + on every run. It uses Axon 0.8's `Axon.ModelState`. +- `mix livebook.test` runs cells one at a time, as Livebook does, and + `--local` tests notebooks against the checkout. CI now runs every shipped + livebook. + +## Documentation + +- The guides no longer say `:gzip` always works; it applies to Zarr v3 arrays. +- The Nx guide's speed figures are measured: for an 8 MB float64 array in + memory storage on an Apple M1 Max, `to_tensor/2` takes about 75 ms + uncompressed and 117 ms with the default zlib. +- The v1.3.0 notes and roadmap record that Explorer streaming, planned for + 1.3, was dropped. Columnar data and Explorer belong in + [ex_arrow](https://hex.pm/packages/ex_arrow). + +Full list: [CHANGELOG](https://github.com/AnimaLogica/ExZarr/blob/main/CHANGELOG.md). diff --git a/mix.exs b/mix.exs index cb25e2c..5cd937c 100644 --- a/mix.exs +++ b/mix.exs @@ -1,7 +1,7 @@ defmodule ExZarr.MixProject do use Mix.Project - @version "1.3.0" + @version "1.3.1" @source_url "https://github.com/AnimaLogica/ExZarr" def project do @@ -200,6 +200,7 @@ defmodule ExZarr.MixProject do "docs/release_notes_v1_1_0.md", "docs/release_notes_v1_2_0.md", "docs/release_notes_v1_3_0.md", + "docs/release_notes_v1_3_1.md", "docs/ROADMAP.md", "docs/ZARR_V3_STATUS.md", "docs/INTEROPERABILITY.md", @@ -290,6 +291,7 @@ defmodule ExZarr.MixProject do "docs/release_notes_v1_1_0.md", "docs/release_notes_v1_2_0.md", "docs/release_notes_v1_3_0.md", + "docs/release_notes_v1_3_1.md", "docs/migration_guide_v1_1_0.md" ], Reference: [