Skip to content

v1.3.1 - various fixes - #315

Merged
thanos merged 1 commit into
mainfrom
v1.3.1/review-fixes
Oct 8, 2026
Merged

thanos merged 1 commit into
mainfrom
v1.3.1/review-fixes

Conversation

@thanos

@thanos thanos commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator
  • Tests: 1,679 tests, 70 properties and 155 doctests, plus the performance tests.
  • Checks: format, compile with warnings as errors, credo, doctor, sobelow, docs, and Dialyzer. I rebuilt your stale local Dialyzer cache, which is a gitignored build file.
  • Livebooks: all 17 shipped livebooks run against this branch's code.

To show the new tests check something real, I ran them against the released 1.3.0 code. They fail exactly where behaviour changed: the dtype error, crop speed, the caller's random state, and the livebook harness.

Code

  • Edge-chunk crop (F2): it now copies one contiguous run per row. A 999×1000 edge chunk takes under 1 ms instead of 305 ms. The results match to_tensor/2 on 1D through 4D arrays with edges on every axis. While making this change I found that the review's "duplicate shape computation" point was wrong: the stream's metadata.shape is the nominal chunk shape, not the cropped one. Using it broke every edge chunk, so the code still computes the cropped shape from the chunk bounds.
  • Shuffled batching (F3):
    • Batches are grouped in a single pass, which removes the quadratic slowdown.
    • The caller's random state is untouched, with or without a seed. The default seed was quietly using the caller's random generator, which I only caught by testing.
    • A given seed produces exactly the same order as 1.3.0, pinned by tests.
    • A short final batch that drop_remainder will drop is no longer read first.
  • Smaller fixes: an unsupported dtype now gives one error instead of one per chunk (F13), the unreachable clause is gone (F12), and one shared helper handles :names and :backend. That last change means to_tensor_chunked/3 now honours :names too.

Livebooks and tooling

  • Codecs livebook (F5): it crashed on gzip, which is a Zarr v3 codec only, and then on compressor: nil, which needs to be :none. Both are fixed, and it runs end to end.
  • Axon notebook (F6): the second feature was an exact copy of the first plus 0.05, so the "right weights" the notebook described eren't learnable. It now uses an independent second feature, zero starting weights, Adam and Axon 0.8's model-state API. It prints weights: [2.0, -0.0], bias: [1.0], predictions: [1.0, 3.0], the same on every run, with no deprecation warning.
  • mix livebook.test (F7): it now runs cells one at a time like Livebook, names the failing cell, and keeps the end of long output where the error is. A new --local option tests against the checkout, and a new CI job runs every shipped livebook that way.

Docs

  • Explorer scope (F1): the CHANGELOG, release notes and ROADMAP now say Explorer streaming was planned and dropped, with links to ex_arrow and ex_graphblas. The plan file's statuses are updated, with Explorer marked cancelled.
  • Speed claims: the "5-10× faster, 400-800 MB/s" claims in the Nx guide and module docs didn't hold up. Measured on your M1 Max with an 8 MB array, to_tensor/2 takes 75 ms uncompressed and 117 ms with the default zlib. I replaced the claims with those labelled numbers.
  • Remaining review itms: the reduction example uses include_missing and on_error: :halt (F9), the stale shuffle example is fixed (F8), and the pin (F10), comma (F11), headings (F15), broken bold (F16), new long dashes and CHANGELOG wording (F14) are corrected.

Beyond the review, found while fixing:

  • Six guides said :gzip "always works". It only works for Zarr v3 arrays, so they now say so.
  • compressor: nil also appeared in two livebook formance.md; those are fixed too.

- Tests: 1,679 tests, 70 properties and 155 doctests, plus the performance tests.
- Checks: format, compile with warnings as errors, credo, doctor, sobelow, docs, and Dialyzer. I rebuilt your stale local Dialyzer cache, which is a gitignored build file.
- Livebooks: all 17 shipped livebooks run against this branch's code.

To show the new tests check something real, I ran them against the released 1.3.0 code. They fail exactly where behaviour changed: the dtype error, crop speed, the caller's random state, and the livebook harness.

Code
- Edge-chunk crop (F2): it now copies one contiguous run per row. A 999×1000 edge chunk takes under 1 ms instead of 305 ms. The results match to_tensor/2 on 1D through 4D arrays with edges on every axis. While making this change I found that the review's "duplicate shape computation" point was wrong: the stream's metadata.shape is the nominal chunk shape, not the cropped one. Using it broke every edge chunk, so the code still computes the cropped shape from the chunk bounds.
- Shuffled batching (F3):
  - Batches are grouped in a single pass, which removes the quadratic slowdown.
  - The caller's random state is untouched, with or without a seed. The default seed was quietly using the caller's random generator, which I only caught by testing.
  - A given seed produces exactly the same order as 1.3.0, pinned by tests.
  - A short final batch that drop_remainder will drop is no longer read first.
- Smaller fixes: an unsupported dtype now gives one error instead of one per chunk (F13), the unreachable clause is gone (F12), and one shared helper handles :names and :backend. That last change means to_tensor_chunked/3 now honours :names too.

Livebooks and tooling
- Codecs livebook (F5): it crashed on gzip, which is a Zarr v3 codec only, and then on compressor: nil, which needs to be :none. Both are fixed, and it runs end to end.
- Axon notebook (F6): the second feature was an exact copy of the first plus 0.05, so the "right weights" the notebook described eren't learnable. It now uses an independent second feature, zero starting weights, Adam and Axon 0.8's model-state API. It prints weights: [2.0, -0.0], bias: [1.0], predictions: [1.0, 3.0], the same on every run, with no deprecation warning.
- mix livebook.test (F7): it now runs cells one at a time like Livebook, names the failing cell, and keeps the end of long output where the error is. A new --local option tests against the checkout, and a new CI job runs every shipped livebook that way.

Docs
- Explorer scope (F1): the CHANGELOG, release notes and ROADMAP now say Explorer streaming was planned and dropped, with links to ex_arrow and ex_graphblas. The plan file's statuses are updated, with Explorer marked cancelled.
- Speed claims: the "5-10× faster, 400-800 MB/s" claims in the Nx guide and module docs didn't hold up. Measured on your M1 Max with an 8 MB array, to_tensor/2 takes 75 ms uncompressed and 117 ms with the default zlib. I replaced the claims with those labelled numbers.
- Remaining review itms: the reduction example uses include_missing and on_error: :halt (F9), the stale shuffle example is fixed (F8), and the pin (F10), comma (F11), headings (F15), broken bold (F16), new long dashes and CHANGELOG wording (F14) are corrected.

Beyond the review, found while fixing:
- Six guides said :gzip "always works". It only works for Zarr v3 arrays, so they now say so.
- compressor: nil also appeared in two livebook formance.md; those are fixed too.
@thanos
thanos merged commit 016042d into main Oct 8, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant