thanos · Oct 8, 2026 · Oct 8, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -182,6 +182,43 @@ jobs:
           echo "Checking for undocumented public functions..."
           mix docs --warnings-as-errors
 
+  livebooks:
+    name: Livebooks
+    runs-on: ubuntu-latest
+    env:
+      MIX_ENV: dev
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Elixir
+        uses: erlef/setup-beam@v1
+        with:
+          elixir-version: "1.19.5"
+          otp-version: "28.1"
+
+      - name: Restore dependencies cache
+        uses: actions/cache@v4
+        with:
+          path: |
+            deps
+            _build
+          key: ${{ runner.os }}-mix-livebooks-${{ hashFiles('**/mix.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-mix-livebooks-
+
+      - name: Install dependencies
+        run: mix deps.get
+
+      # Runs every shipped notebook cell by cell against this checkout, so a
+      # change that breaks a livebook fails here rather than after release.
+      - name: Run livebooks
+        run: >-
+          mix livebook.test --local --max 2 --timeout 900
+          --paths "docs/livebooks/0[1-6]*/*.livemd"
+          --paths docs/livebooks/nx_streaming.livemd
+
   python-interop:
     name: Python Interop (zarr ${{ matrix.zarr }})
     runs-on: ubuntu-latest

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,9 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+- `ExZarr.Nx.stream_chunk_tensors/2` crops padded edge chunks with one
+  contiguous copy per row instead of one copy per element. A 999x1000 float32
+  edge chunk went from about 300 ms to under 1 ms.
+- `stream_chunk_tensors/2` on an array whose dtype has no Nx type now yields a
+  single `{:error, reason}` and reads nothing, instead of one error per chunk.
+- `DataLoader.shuffled_batch_stream/3` and `paired_shuffled_batch_stream/4`
+  group shuffled indices in one pass. Slicing the index list per batch was
+  quadratic in the sample count (about 1 s of indexing at 200,000 samples).
+  A short final batch is no longer read when `drop_remainder: true` drops it.
+- Shuffling no longer reseeds the caller's `:rand` state, with or without
+  `:seed`. A given `:seed` still produces the same order as 1.3.0.
+- `to_tensor_chunked/3` now applies `:names`, like `to_tensor/2`.
+- Livebooks: `01_04_codecs_and_pipelines` no longer uses `:gzip` with Zarr v2
+  (gzip is a Zarr v3 codec; v2 uses `:zlib`) or `compressor: nil` (use
+  `:none`). `03_03_training_from_zarr` uses an independent second feature,
+  zero-initialized weights and Adam, so it reliably learns the line, and uses
+  Axon 0.8's `Axon.ModelState`.
+- Guides no longer say `:gzip` always works; it is accepted for Zarr v3 arrays.
+
+### Changed
+- `mix livebook.test` evaluates notebook cells one at a time with bindings and
+  environment carried forward, like Livebook, names the failing cell, and keeps
+  the end of long output. `--local` runs notebooks against this checkout
+  instead of the Hex release. CI runs `mix livebook.test --local`.
+
 ## [1.3.0] - 2026-10-08
 
-### Data Science Interop
+Data science interop for dense arrays.
+
+Not included, although planned for 1.3: Explorer direct streaming. Columnar
+data and Explorer integration belong in [ex_arrow](https://hex.pm/packages/ex_arrow);
+Binsparse belongs in [ex_graphblas](https://hex.pm/packages/ex_graphblas).
 
 ### Added
 - `ExZarr.Nx.stream_chunk_tensors/2` maps `Array.stream_chunks/2` to Nx tensors.

diff --git a/README.md b/README.md
@@ -833,8 +833,8 @@ Practical examples demonstrating real-world usage:
   - Intuitive slicing by name instead of index
   - Real-world examples (climate, medical imaging)
   - Validation and best practices
-- **[Using Nx](docs/guides/getting_started.md#using-nx)** — what Nx is, when it fits, and how tensors and Zarr chunks exchange bytes
-- **[Nx Integration](docs/guides/nx_integration.md)** — conversion API, chunk streams, and DataLoader
+- **[Using Nx](docs/guides/getting_started.md#using-nx)**: what Nx is, when it fits, and how tensors and Zarr chunks exchange bytes
+- **[Nx Integration](docs/guides/nx_integration.md)**: conversion API, chunk streams, and DataLoader
   - Converting between Nx tensors and Zarr arrays
   - Machine learning workflows
   - Streaming large arrays

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -1,6 +1,6 @@
 # Roadmap
 
-## v1.2.0 — Zarr 3.1 Interoperability & Range-Aware Cloud I/O
+## v1.2.0 - Zarr 3.1 Interoperability & Range-Aware Cloud I/O
 
 Released 2026-10-04. Backward compatible with v1.1 (rewrite pre-1.2 private shards; OTP 26+).
 
@@ -16,7 +16,7 @@ Released 2026-10-04. Backward compatible with v1.1 (rewrite pre-1.2 private shar
 
 ---
 
-## v1.1.0 — BEAM-Native Streaming
+## v1.1.0 - BEAM-Native Streaming
 
 Released 2026-06-12. Backward compatible with v1.0.
 
@@ -26,18 +26,18 @@ Released 2026-06-12. Backward compatible with v1.0.
 
 ---
 
-## v1.3.0 — Data Science Interop
+## v1.3.0 - Data Science Interop
 
 Released 2026-10-08.
 
 - [x] `ExZarr.Nx.stream_chunk_tensors/2` (stored chunks → tensors, edge crops)
 - [x] DataLoader documented against its real shuffle (full index list)
 - [x] Livebook MVP: metadata, Zarr↔Nx, minibatches, Axon toy train
-- [x] Arrow/Explorer stay in ex_arrow; Binsparse stays in ex_graphblas
+- [ ] ~~Explorer direct streaming~~ Dropped from 1.3: columnar data and Explorer belong in [ex_arrow](https://hex.pm/packages/ex_arrow), and Binsparse in [ex_graphblas](https://hex.pm/packages/ex_graphblas)
 
 ---
 
-## v1.4.0 (Planned) — Performance & Packaging
+## v1.4.0 (Planned) - Performance & Packaging
 
 - [ ] Async codec pipeline
 - [ ] Vendored/static codec libraries
@@ -46,14 +46,14 @@ Released 2026-10-08.
 
 ---
 
-## v2.0.0 (Future) — Distributed Processing
+## v2.0.0 (Future) - Distributed Processing
 
 - [ ] Multi-node chunk processing
 - [ ] Cross-node telemetry
 
 ---
 
-## Deferred from v1.2 (explicit)
+## Deferred (explicit)
 
 - Unified retry layer across cloud SDKs
 - Native multi-range / coalescing heuristics

diff --git a/docs/guides/core_concepts.md b/docs/guides/core_concepts.md
@@ -357,7 +357,7 @@ Codecs transform chunk data before storage. This includes compression (reduce si
 
 **Compression codecs** (reduce size):
 - `:zlib` - Standard compression (Erlang built-in, always available)
-- `:gzip` - Compatible with gzip tools (Erlang built-in)
+- `:gzip` - Compatible with gzip tools (Erlang built-in; Zarr v3 arrays only, use `:zlib` for v2)
 - `:zstd` - Fast compression with excellent ratio
 - `:lz4` - Very fast compression, lower ratio
 - `:blosc` - Shuffle filters plus an inner compressor; strong on numeric data

diff --git a/docs/guides/getting_started.md b/docs/guides/getting_started.md
@@ -471,7 +471,7 @@ Skip it when the access pattern is a few scalar lookups. A Zarr read loads a chu
 
 Skip it for `{:bf, 16}`, `{:f, 16}`, and complex tensors. Those types are not in the Zarr dtypes ExZarr stores. Keep float32 on disk and cast after `to_tensor/2`.
 
-### Quick Guide to Nx functionality
+### Nx functions at a glance
 
 | Function | What you get |
 | --- | --- |
@@ -495,7 +495,7 @@ array
 
 The livebooks under `docs/livebooks/03_nx_ml/` and `docs/livebooks/nx_streaming.livemd` run these calls. The rest of the API is in the [Nx integration guide](nx_integration.md).
 
-### Deep Dive in how data is stored in Nx versus Zarr and how we exchange data between the two
+### How data moves between Nx and Zarr
 
 An Nx tensor is one buffer. For `Nx.iota({6, 4}, type: {:f, 32})` that buffer is `6 * 4 * 4` bytes, row-major (C order), with the shape stored beside the binary. Names and the backend are metadata in the tensor struct. Nothing is compressed. Nothing is split into files.
 

diff --git a/docs/guides/glossary.md b/docs/guides/glossary.md
@@ -349,7 +349,7 @@ Note: `<` indicates little-endian byte order (standard on most modern systems).
 | Codec | Python (numcodecs) | ExZarr/Elixir | Availability |
 |-------|-------------------|---------------|--------------|
 | Zlib | `numcodecs.Zlib` | `:zlib` | Always (Erlang built-in) |
-| Gzip | `numcodecs.GZip` | `:gzip` | Always (Erlang built-in) |
+| Gzip | `numcodecs.GZip` | `:gzip` | Zarr v3 only (Erlang built-in) |
 | Zstd | `numcodecs.Zstd` | `:zstd` | Via ExCodecs |
 | LZ4 | `numcodecs.LZ4` | `:lz4` | Via ExCodecs |
 | Blosc | `numcodecs.Blosc` | `:blosc` | Via ExCodecs (Blosc1 chunks) |

diff --git a/docs/guides/installation.md b/docs/guides/installation.md
@@ -209,8 +209,8 @@ failed (offline build, proxy) or the platform has no precompiled binary.
    # config/config.exs
    config :rustler_precompiled, :force_build, ex_codecs: true
    ```
-3. Meanwhile `:zlib` and `:gzip` keep working; they use Erlang's built-in
-   `:zlib`.
+3. Meanwhile `:zlib` (and `:gzip` on Zarr v3 arrays) keep working; they use
+   Erlang's built-in `:zlib`.
 
 ### Issue: Permission Denied Errors
 

diff --git a/docs/guides/nx_integration.md b/docs/guides/nx_integration.md
@@ -4,7 +4,7 @@ This guide shows how to integrate ExZarr with Nx (Numerical Elixir) for numerica
 
 For what Nx is, when to use the bridge, and how tensor bytes relate to Zarr chunks, start with [Using Nx](getting_started.md#using-nx) in the getting started guide. This page is the API: conversion, chunk streams, and DataLoader.
 
-** Performance Note:** ExZarr provides an optimized `ExZarr.Nx` module with **5-10x faster** conversion compared to manual approaches. Always use `ExZarr.Nx` for best performance.
+**Performance note:** `ExZarr.Nx` converts through one binary instead of nested lists, so it skips the cost of building and walking millions of list cells. Measured numbers are in [Optimized Conversion](#optimized-conversion-recommended).
 
 ## Table of Contents
 
@@ -42,7 +42,7 @@ tensor = Nx.iota({1000, 1000})
 )
 ```
 
-**Performance:** 10-20ms for 8MB (400-800 MB/s)
+**Performance:** an 8 MB float64 array converts in tens of milliseconds without compression; codec choice dominates. See the table under [Optimized Conversion](#optimized-conversion-recommended).
 
 ## Nx and ExZarr Overview
 
@@ -89,12 +89,19 @@ ExZarr provides the `ExZarr.Nx` module for efficient conversion using direct bin
 
 ### Performance Comparison
 
-| Approach | Time (8MB) | Throughput | Status |
-|----------|-----------|------------|---------|
-| **`ExZarr.Nx` (optimized)** | **10-20ms** | **400-800 MB/s** | **Recommended** |
-| Nested tuples (legacy) | 80-150ms | 50-100 MB/s | Deprecated |
+Measured on an Apple M1 Max: a `{1000, 1000}` float64 array (8 MB), chunks
+`{100, 100}`, memory storage, median of seven runs.
 
-**Speedup: 5-10x faster**
+| Compressor | `to_tensor/2` | `from_tensor/2` |
+|------------|---------------|-----------------|
+| `:none` | 75 ms | 50 ms |
+| `:zstd` | 87 ms | 98 ms |
+| `:zlib` (the default) | 117 ms | 640 ms |
+
+The legacy nested-list path does the same storage work and adds the list
+conversion on top: about 400 ms for `Nx.to_list/1` on the write side and about
+100 ms for `Nx.tensor/2` on the read side for this array. Compression is the
+largest cost, so choose the codec before tuning anything else.
 
 ### Converting ExZarr to Nx
 
@@ -397,9 +404,12 @@ trained_state =
 ### Chunk-aligned reduction
 
 ```elixir
+# include_missing: never-written chunks count as the fill value.
+# on_error: :halt: a chunk that fails to read stops the sum instead of
+# silently leaving it out.
 sums =
   array
-  |> ExZarr.Nx.stream_chunk_tensors(concurrency: 4, on_error: :skip)
+  |> ExZarr.Nx.stream_chunk_tensors(concurrency: 4, include_missing: true, on_error: :halt)
   |> Enum.map(fn {:ok, tensor} -> Nx.sum(tensor) |> Nx.to_number() end)
 
 total = Enum.sum(sums)
@@ -723,7 +733,7 @@ result = tensor
 
 **Problem**: Conversion requires copying data (binary → tensor).
 
-**Impact**: ~10-20ms overhead per 8MB.
+**Impact**: one copy of the data; for 8 MB, tens of milliseconds (see the table under [Optimized Conversion](#optimized-conversion-recommended)).
 
 **Workarounds:**
 1. Process in chunks (amortize overhead)
@@ -805,7 +815,7 @@ end
 
 ### WARNING: Deprecated: Nested Tuple Conversion
 
-**Note:** This approach is **5-10x slower** than `ExZarr.Nx`. Only use for compatibility with old code.
+**Note:** This approach is slower than `ExZarr.Nx` because it builds nested lists (see the measurements above). Only use it for compatibility with old code.
 
 <details>
 <summary>Click to expand legacy nested tuple approach</summary>
@@ -817,7 +827,7 @@ defmodule ExZarr.Nx.LegacyHelpers do
   @moduledoc """
   Legacy conversion helpers using nested tuples.
 
-  WARNING: DEPRECATED: Use ExZarr.Nx module instead (5-10x faster).
+  WARNING: DEPRECATED: Use the ExZarr.Nx module instead.
   """
 
   def nested_list_to_tuple(list) when is_list(list) do
@@ -839,7 +849,7 @@ end
 #### Legacy Write Pattern
 
 ```elixir
-# WARNING: SLOW: 80-150ms for 8MB
+# WARNING: SLOW: adds a nested-list conversion on top of the storage work
 tensor = Nx.iota({1000, 1000})
 
 {:ok, array} = ExZarr.create(
@@ -864,7 +874,7 @@ data = tensor
 #### Legacy Read Pattern
 
 ```elixir
-# WARNING: SLOW: 80-150ms for 8MB
+# WARNING: SLOW: adds a nested-list conversion on top of the storage work
 {:ok, array} = ExZarr.open(path: "/data/array")
 
 {:ok, data} = ExZarr.Array.get_slice(array,
@@ -885,13 +895,12 @@ tensor = data
 
 ExZarr provides first-class Nx integration via the `ExZarr.Nx` module:
 
-**5-10x faster** than manual conversion (400-800 MB/s vs 50-100 MB/s)
+**No nested lists** - conversion goes through one binary
 **Simple API** - Single function calls for conversion
 **Full type support** - All 10 standard numeric types
 **Chunked processing** - Constant memory for large arrays
 **Backend agnostic** - Works with CPU, EXLA, Torchx
 **ML-ready** - Efficient batch loading, checkpointing
-**Production-tested** - Used in real-world workflows
 
 **Next steps:**
 - Try the example: `elixir examples/nx_optimized_conversion.exs`

diff --git a/docs/guides/overview.md b/docs/guides/overview.md
@@ -111,8 +111,8 @@ ExCodecs publishes binaries for macOS (Apple Silicon and Intel), Linux
 (x86_64 and aarch64, glibc and musl) and Windows (x86_64). If its NIF cannot
 load, ExZarr does not silently switch codecs: `ExZarr.Codecs.available_codecs/0`
 lists only `:none` and `:zlib`, and compressing with another codec returns
-`{:error, {:unsupported_codec, codec}}` or a compression error. `:zlib` and
-`:gzip` always work.
+`{:error, {:unsupported_codec, codec}}` or a compression error. `:zlib` always
+works, and so does `:gzip` on Zarr v3 arrays.
 
 ### NIF Safety
 

diff --git a/docs/guides/performance.md b/docs/guides/performance.md
@@ -315,7 +315,7 @@ Data integrity critical (checksums required)?
   → Add crc32c codec to pipeline
 
 Data already compressed (images, video)?
-  → Use no compression (compressor: nil)
+  → Use no compression (compressor: :none)
 ```
 
 ### Compression Level Tuning

diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md
@@ -1011,7 +1011,7 @@ python3 -c "import zarr; z = zarr.open('/path/to/array', mode='r'); print(z.info
    ```elixir
    # Python zarr always supports zlib/gzip
    compressor: :zlib
-   # Or
+   # Or, for a Zarr v3 array
    compressor: :gzip
    ```
 

diff --git a/docs/livebooks/01_core_zarr/01_01_first_zarr_array.livemd b/docs/livebooks/01_core_zarr/01_01_first_zarr_array.livemd
@@ -2,7 +2,7 @@
 
 ```elixir
 Mix.install([
-  {:ex_zarr, "~> 1.2"},
+  {:ex_zarr, "~> 1.3"},
   # {:ex_zarr, path: Path.join(__DIR__, "../../..")}
 ])
 ```