From 396b941f73abaedf8a499b7422a57e6fe2f4488b Mon Sep 17 00:00:00 2001 From: thanos Date: Fri, 9 Oct 2026 15:05:20 -0400 Subject: [PATCH] What changed (17 files) - Version: mix.exs is now 1.3.1, and the new docs/release_notes_v1_3_1.md is registered in the docs under the release notes group. - CHANGELOG: - The [Unreleased] fixes are now ## [1.3.1] - 2026-10-09, with a short summary and an empty [Unreleased] section above. - I added entries for your post-merge commit e2abf91: the steadier streaming memory test and the livebook pin updates. - The comparison links are updated: [Unreleased] compares from v1.3.1, and [1.3.1] compares v1.3.0 to v1.3.1. - Release notes: written for users, covering the Nx speedups, the shuffle fix, the livebooks and the doc corrections. It says no code changes are needed to upgrade. - ROADMAP and README: both have a v1.3.1 entry. - Version references: "1.3.0" became "1.3.1" in the installation guide's version check and the troubleshooting bug-report template. The livebook status page now says every shipped notebook runs in CI, and I fixed a typo there ("directry"). - Install pins: the remaining ~> 1.2 pins are now ~> 1.3. That covers one shipped livebook (06_05), three notebooks, three livebook scripts, and the storage-providers and migrating-from-Python guides. The 1.1 migration guide keeps ~> 1.1, since it documents that upgrade. - I left the code comments and test names that say "1.3.0" alone. They describe the shuffle order that 1.3.0 produced, which is still accurate. Verified - Format, compile, credo and docs pass, and the test suite passes (1,679 tests, 0 failures). - mix hex.build produces a version 1.3.1 package. I built it in the scratchpad and deleted it. - I simulated the release workflow's checks: - The tag v1.3.1 will match mix.exs, and ## [1.3.1] exists in the CHANGELOG. - The GitHub release body will come from docs/release_notes_v1_3_1.md. - The CHANGELOG fallback also extracts the 1.3.1 section correctly. - All 17 shipped livebooks pass against this code, and 06_05 passes from Hex with its new pin. To release: commit this branch, merge it to main, then push the tag v1.3.1. The release workflow verifies the build, creates the GitHub release and publishes to Hex, as it did for 1.3.0. --- CHANGELOG.md | 23 ++++++++- README.md | 4 ++ docs/ROADMAP.md | 6 +++ docs/guides/installation.md | 2 +- docs/guides/migration_from_python.md | 2 +- docs/guides/storage_providers.md | 8 ++-- docs/guides/troubleshooting.md | 2 +- .../06_cookbook/06_05_geospatial.livemd | 2 +- docs/livebooks/PLAN.md | 6 +-- docs/livebooks/scripts/00_01_welcome.exs | 2 +- .../scripts/01_01_first_zarr_array.exs | 2 +- .../scripts/01_02_metadata_and_chunks.exs | 2 +- docs/notebooks/earthmover_datacube.livemd | 2 +- docs/notebooks/xarray_zarr_intro.livemd | 2 +- docs/notebooks/zarr_fundamentals.livemd | 2 +- docs/release_notes_v1_3_1.md | 48 +++++++++++++++++++ mix.exs | 4 +- 17 files changed, 100 insertions(+), 19 deletions(-) create mode 100644 docs/release_notes_v1_3_1.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 332a1a3..1e4ce90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.1] - 2026-10-09 + +Patch release from the v1.3.0 review: faster Nx chunk and batch streams, a +shuffle that leaves the caller's random state alone, livebooks that run, and +docs that match measured behaviour. No public API was removed or renamed. + ### Fixed - `ExZarr.Nx.stream_chunk_tensors/2` crops padded edge chunks with one contiguous copy per row instead of one copy per element. A 999x1000 float32 @@ -32,6 +38,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 environment carried forward, like Livebook, names the failing cell, and keeps the end of long output. `--local` runs notebooks against this checkout instead of the Hex release. CI runs `mix livebook.test --local`. +- Livebooks, notebooks and guide install snippets use `{:ex_zarr, "~> 1.3"}`. +- The Nx guide and `ExZarr.Nx` docs replace unbenchmarked speed claims + ("5-10x", "400-800 MB/s") with measured timings, and the chunk-reduction + example uses `include_missing: true` and `on_error: :halt` so its total is + correct. +- The v1.3.0 entries in this changelog, the release notes, and the roadmap now + say that Explorer streaming was planned for 1.3 and dropped. + +### Tests +- The streaming memory test measures the test process instead of the whole VM, + so other async tests no longer move the result; the limit is now 2 MB. +- New tests cover `stream_chunk_tensors/2` options (`:include_missing`, + `:on_error`, `:backend`, unordered concurrency), 3D and 4D edge crops, seeded + shuffle orders matching 1.3.0, and cell-by-cell livebook evaluation. ## [1.3.0] - 2026-10-08 @@ -932,7 +952,8 @@ None - Full backward compatibility maintained with v2 arrays --- -[Unreleased]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.0...HEAD +[Unreleased]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.1...HEAD +[1.3.1]: https://github.com/AnimaLogica/ExZarr/compare/v1.3.0...v1.3.1 [1.3.0]: https://github.com/AnimaLogica/ExZarr/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.2.0 [1.1.0]: https://github.com/AnimaLogica/ExZarr/releases/tag/v1.1.0 diff --git a/README.md b/README.md index df92477..5d26ad3 100644 --- a/README.md +++ b/README.md @@ -874,6 +874,10 @@ Key modules: See [ROADMAP.md](docs/ROADMAP.md) for the full release plan. +**v1.3.1 (released 2026-10-09)** - Patch release: faster Nx chunk crops and +shuffled batches, livebooks checked in CI, measured performance figures. See +[release notes](docs/release_notes_v1_3_1.md). + **v1.3.0 (released 2026-10-08)** - Data science interop: `ExZarr.Nx.stream_chunk_tensors/2`, DataLoader batch recipes, livebook MVP (metadata, Nx conversion, minibatches, a small Axon train). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b486a80..cbace5f 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -35,6 +35,12 @@ Released 2026-10-08. - [x] Livebook MVP: metadata, Zarr↔Nx, minibatches, Axon toy train - [ ] ~~Explorer direct streaming~~ Dropped from 1.3: columnar data and Explorer belong in [ex_arrow](https://hex.pm/packages/ex_arrow), and Binsparse in [ex_graphblas](https://hex.pm/packages/ex_graphblas) +### v1.3.1 + +Released 2026-10-09. Patch release: faster edge-chunk crops and shuffled +batching, shuffling that leaves the caller's `:rand` state alone, runnable +livebooks checked in CI, and measured performance figures in the Nx docs. + --- ## v1.4.0 (Planned) - Performance & Packaging diff --git a/docs/guides/installation.md b/docs/guides/installation.md index ddd9042..b7c406a 100644 --- a/docs/guides/installation.md +++ b/docs/guides/installation.md @@ -181,7 +181,7 @@ If tests pass, your installation is complete and working correctly. ```elixir iex> Application.spec(:ex_zarr, :vsn) |> to_string() -"1.3.0" +"1.3.1" ``` ## Build Troubleshooting diff --git a/docs/guides/migration_from_python.md b/docs/guides/migration_from_python.md index 3b326f8..e32bb6b 100644 --- a/docs/guides/migration_from_python.md +++ b/docs/guides/migration_from_python.md @@ -38,7 +38,7 @@ pip install zarr # In mix.exs def deps do [ - {:ex_zarr, "~> 1.2"} + {:ex_zarr, "~> 1.3"} ] end ``` diff --git a/docs/guides/storage_providers.md b/docs/guides/storage_providers.md index dc9b974..9b38844 100644 --- a/docs/guides/storage_providers.md +++ b/docs/guides/storage_providers.md @@ -157,7 +157,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:ex_aws, "~> 2.5"}, {:ex_aws_s3, "~> 2.5"}, {:sweet_xml, "~> 0.7"} # For XML response parsing @@ -353,7 +353,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:goth, "~> 1.4"}, # Authentication {:req, "~> 0.6.1"} # HTTP client ] @@ -490,7 +490,7 @@ Add to `mix.exs`: ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:azure_sdk, "~> 0.4.1"} ] end @@ -641,7 +641,7 @@ Stores large chunks in MongoDB using GridFS. ```elixir def deps do [ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:mongodb_driver, "~> 1.4"} ] end diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 568eccd..736d6a4 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -1244,7 +1244,7 @@ GitHub Issues: [https://github.com/AnimaLogica/ExZarr/issues](https://github.com ```markdown ## Environment -- ExZarr version: 1.3.0 +- ExZarr version: 1.3.1 - Elixir version: 1.19.5 - Erlang/OTP version: 28.1 - OS: macOS 15.2 (arm64) diff --git a/docs/livebooks/06_cookbook/06_05_geospatial.livemd b/docs/livebooks/06_cookbook/06_05_geospatial.livemd index 0653f45..0f24b6b 100644 --- a/docs/livebooks/06_cookbook/06_05_geospatial.livemd +++ b/docs/livebooks/06_cookbook/06_05_geospatial.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"} + {:ex_zarr, "~> 1.3"} # {:ex_zarr, path: Path.join(__DIR__, "../../..")}, ]) ``` diff --git a/docs/livebooks/PLAN.md b/docs/livebooks/PLAN.md index 516854e..f5da698 100644 --- a/docs/livebooks/PLAN.md +++ b/docs/livebooks/PLAN.md @@ -1,6 +1,6 @@ -## Status (v1.3.0) +## Status (v1.3.1) -Shipped for 1.3.0: +Shipped in 1.3.0; in 1.3.1 every notebook below runs in CI against the current code: - `01_core_zarr/01_01` through `01_04` - `03_nx_ml/03_01_zarr_to_nx.livemd` @@ -13,7 +13,7 @@ Shipped for 1.3.0: Deferred past 1.3.0: `02_concurrency`, `04_02`–`04_15`, `05_02`–`05_12`, `06_crypto` through `09_systems`, and the rest of the table below. Arrow and Explorer livebooks belong with ex_arrow. Binsparse livebooks belong with ex_graphblas. -The livebooks directry should be a curated collection of **Elixir Livebooks** demonstrating how to use +The livebooks directory should be a curated collection of **Elixir Livebooks** demonstrating how to use **ExZarr** — a Zarr v3–compatible array storage library — to build scalable data systems for **AI / GenAI, finance, crypto, and scientific workloads**. diff --git a/docs/livebooks/scripts/00_01_welcome.exs b/docs/livebooks/scripts/00_01_welcome.exs index 8ae5f60..4ad179f 100644 --- a/docs/livebooks/scripts/00_01_welcome.exs +++ b/docs/livebooks/scripts/00_01_welcome.exs @@ -3,7 +3,7 @@ # Title: Welcome to ExZarr # Mix.install([ -# # {:ex_zarr, "~> 1.2"}, +# # {:ex_zarr, "~> 1.3"}, # {:ex_zarr, path: "/Users/thanos/work/ExZarr"}, # {:nx, "~> 0.7"}, # {:kino, "~> 0.13"}, diff --git a/docs/livebooks/scripts/01_01_first_zarr_array.exs b/docs/livebooks/scripts/01_01_first_zarr_array.exs index 0b4ce5e..c69b58a 100644 --- a/docs/livebooks/scripts/01_01_first_zarr_array.exs +++ b/docs/livebooks/scripts/01_01_first_zarr_array.exs @@ -3,7 +3,7 @@ # Title: Your First Zarr Array Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:kino, "~> 0.13"} ]) diff --git a/docs/livebooks/scripts/01_02_metadata_and_chunks.exs b/docs/livebooks/scripts/01_02_metadata_and_chunks.exs index 03ff6be..101ad3b 100644 --- a/docs/livebooks/scripts/01_02_metadata_and_chunks.exs +++ b/docs/livebooks/scripts/01_02_metadata_and_chunks.exs @@ -3,7 +3,7 @@ # Title: Metadata and Chunks: Inside a Zarr Array Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:kino, "~> 0.13"}, {:jason, "~> 1.4"} ]) diff --git a/docs/notebooks/earthmover_datacube.livemd b/docs/notebooks/earthmover_datacube.livemd index 77c35b4..7b093f9 100644 --- a/docs/notebooks/earthmover_datacube.livemd +++ b/docs/notebooks/earthmover_datacube.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"}, {:kino_vega_lite, "~> 0.1"} diff --git a/docs/notebooks/xarray_zarr_intro.livemd b/docs/notebooks/xarray_zarr_intro.livemd index a1d2435..aef7bf8 100644 --- a/docs/notebooks/xarray_zarr_intro.livemd +++ b/docs/notebooks/xarray_zarr_intro.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"} ]) diff --git a/docs/notebooks/zarr_fundamentals.livemd b/docs/notebooks/zarr_fundamentals.livemd index 6d94c5e..aed155f 100644 --- a/docs/notebooks/zarr_fundamentals.livemd +++ b/docs/notebooks/zarr_fundamentals.livemd @@ -2,7 +2,7 @@ ```elixir Mix.install([ - {:ex_zarr, "~> 1.2"}, + {:ex_zarr, "~> 1.3"}, {:nx, "~> 0.7"}, {:kino, "~> 0.13"}, {:kino_vega_lite, "~> 0.1"} diff --git a/docs/release_notes_v1_3_1.md b/docs/release_notes_v1_3_1.md new file mode 100644 index 0000000..a30525b --- /dev/null +++ b/docs/release_notes_v1_3_1.md @@ -0,0 +1,48 @@ +# v1.3.1 Release Notes + +ExZarr v1.3.1 is a patch release. It fixes performance and correctness issues +found reviewing v1.3.0, and makes every shipped livebook run. Upgrading from +1.3.0 needs no code changes. + +```elixir +{:ex_zarr, "~> 1.3"} +``` + +## Nx + +- `ExZarr.Nx.stream_chunk_tensors/2` crops padded edge chunks one contiguous + row at a time. A 999x1000 float32 edge chunk took about 300 ms in 1.3.0 and + now takes under 1 ms. +- An array whose dtype has no Nx type yields a single `{:error, reason}` from + `stream_chunk_tensors/2` instead of one error per chunk. +- `ExZarr.Nx.DataLoader.shuffled_batch_stream/3` and + `paired_shuffled_batch_stream/4` group the shuffled indices in one pass. In + 1.3.0 the index slicing alone grew with the square of the sample count. +- Shuffling no longer changes the calling process's `:rand` state. The same + `:seed` still gives the same order as in 1.3.0. +- `to_tensor_chunked/3` applies `:names`, as `to_tensor/2` does. + +## Livebooks + +- `01_04_codecs_and_pipelines` ran into `:gzip` with a Zarr v2 array and + `compressor: nil`; it now uses `:zlib` and `:none`. `:gzip` is a Zarr v3 + codec. +- `03_03_training_from_zarr` trains on an independent second feature from + zero-initialized weights with Adam, so it reliably reaches the line + (`weights` about `[2.0, 0.0]`, `bias` about `1.0`) and prints the same numbers + on every run. It uses Axon 0.8's `Axon.ModelState`. +- `mix livebook.test` runs cells one at a time, as Livebook does, and + `--local` tests notebooks against the checkout. CI now runs every shipped + livebook. + +## Documentation + +- The guides no longer say `:gzip` always works; it applies to Zarr v3 arrays. +- The Nx guide's speed figures are measured: for an 8 MB float64 array in + memory storage on an Apple M1 Max, `to_tensor/2` takes about 75 ms + uncompressed and 117 ms with the default zlib. +- The v1.3.0 notes and roadmap record that Explorer streaming, planned for + 1.3, was dropped. Columnar data and Explorer belong in + [ex_arrow](https://hex.pm/packages/ex_arrow). + +Full list: [CHANGELOG](https://github.com/AnimaLogica/ExZarr/blob/main/CHANGELOG.md). diff --git a/mix.exs b/mix.exs index cb25e2c..5cd937c 100644 --- a/mix.exs +++ b/mix.exs @@ -1,7 +1,7 @@ defmodule ExZarr.MixProject do use Mix.Project - @version "1.3.0" + @version "1.3.1" @source_url "https://github.com/AnimaLogica/ExZarr" def project do @@ -200,6 +200,7 @@ defmodule ExZarr.MixProject do "docs/release_notes_v1_1_0.md", "docs/release_notes_v1_2_0.md", "docs/release_notes_v1_3_0.md", + "docs/release_notes_v1_3_1.md", "docs/ROADMAP.md", "docs/ZARR_V3_STATUS.md", "docs/INTEROPERABILITY.md", @@ -290,6 +291,7 @@ defmodule ExZarr.MixProject do "docs/release_notes_v1_1_0.md", "docs/release_notes_v1_2_0.md", "docs/release_notes_v1_3_0.md", + "docs/release_notes_v1_3_1.md", "docs/migration_guide_v1_1_0.md" ], Reference: [