Repository navigation
V1.3.0 - Nx support - #314
Merged
Merged
Conversation
…sparse stay in ex_arrow and ex_graphblas. **Nx.** `ExZarr.Nx.stream_chunk_tensors/2` turns each stored chunk from `Array.stream_chunks/2` into a tensor. Edge chunks that are stored at the full chunk size are cropped to the array bounds. `:concurrency` and `:on_error` pass through. `DataLoader` docs now match the code: shuffled batches permute the full sample index, and `:shuffle_buffer_size` is ignored. 51 Nx tests passed, and `mix docs` built cleanly. **Docs and livebooks.** The Nx guide and `nx_streaming.livemd` use DataLoader and chunk streams. New notebooks cover metadata, Zarr-to-Nx conversion, minibatches, and a small Axon train. Mix.install cells in the embeddings and tick-data notebooks are fixed. The Axon notebook was not executed here because Axon is not a project dependency. **Release files.** `@version` is `1.3.0`, with changelog, release notes, roadmap, and README updated. The existing tag workflow already checks that `v1.3.0` matches `mix.exs` and `CHANGELOG.md`. I did not create or push that tag, so Hex is unchanged until you tag and push `v1.3.0`. - closed #310 - closed #311 - closed #312 - closed #313
…of numbers at once, instead of looping in Elixir one element at a time.
You use it for:
- **Summaries** over an image, a table, or a cube of measurements: sum, mean, min, max
- **Linear algebra**: dot products and matrix multiplies
- **Training**: the prediction step and the gradient of a loss. Axon builds models on top of Nx
- **Compiled runs**: `defn` turns those expressions into CPU code, or GPU code through EXLA or Torchx
The grid stays in memory as a tensor: one binary buffer, a type such as `{:f, 32}`, and a shape. ExZarr does not run that math. It stores the bytes. `ExZarr.Nx` copies an array into a tensor when you want to compute, and copies a tensor back when you want to keep the result.
The getting-started section **What is Nx** now says this directly, without assuming you know NumPy.
`03_03_training_from_zarr.livemd` now explains each step before the code runs.
* **The dataset** is 64 rows of two features. The target is `2 * x0 + 1`. Column 1 is stored and does not enter that formula. Chunks of 16 match the batch size, so each step reads one chunk. The cell prints the first row `[0.0, 0.05]` and the first target `1.0`.
* **The model** is one dense layer: `y_hat = w0 * x0 + w1 * x1 + b`. A good fit is weights near `[2, 0]` and bias near `1`. The batch cell shows shapes `{16, 2}` and `{16, 1}`, and how that pair is wrapped as `%{"features" => x}` for Axon.
* **Training** is mean squared error and SGD for 3 passes of 4 batches, 12 updates from random weights. The two probe rows are `[0.0, 0.0]` and `[1.0, 0.0]`, whose line values are `1` and `3`. The notebook says those predictions move toward `1` and `3` and will not land on them after this short run.
The first target is stored as one column, so `y0[0]` has shape `{1}` and `Nx.to_number/1` rejects it. The cell now indexes that remaining column with `y0[0][0]` before converting to a number.
ExDoc only links to pages listed in `mix.exs`. The Using Nx links pointed at `docs/guides/getting_started.md`, and that file was not registered, so the anchors were reported as missing files. It is now included in the docs extras and in the Getting Started group.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ExZarr 1.3.0 is packaged as dense-array Nx interop. Arrow and Bin…sparse stay in ex_arrow and ex_graphblas.
Nx.
ExZarr.Nx.stream_chunk_tensors/2turns each stored chunk fromArray.stream_chunks/2into a tensor. Edge chunks that are stored at the full chunk size are cropped to the array bounds.:concurrencyand:on_errorpass through.DataLoaderdocs now match the code: shuffled batches permute the full sample index, and:shuffle_buffer_sizeis ignored. 51 Nx tests passed, andmix docsbuilt cleanly.Docs and livebooks. The Nx guide and
nx_streaming.livemduse DataLoader and chunk streams. New notebooks cover metadata, Zarr-to-Nx conversion, minibatches, and a small Axon train. Mix.install cells in the embeddings and tick-data notebooks are fixed. The Axon notebook was not executed here because Axon is not a project dependency.Release files.
@versionis1.3.0, with changelog, release notes, roadmap, and README updated. The existing tag workflow already checks thatv1.3.0matchesmix.exsandCHANGELOG.md. I did not create or push that tag, so Hex is unchanged until you tag and pushv1.3.0.