Repository navigation
v1.3.1 - various fixes - #315
Merged
Merged
Conversation
- Tests: 1,679 tests, 70 properties and 155 doctests, plus the performance tests. - Checks: format, compile with warnings as errors, credo, doctor, sobelow, docs, and Dialyzer. I rebuilt your stale local Dialyzer cache, which is a gitignored build file. - Livebooks: all 17 shipped livebooks run against this branch's code. To show the new tests check something real, I ran them against the released 1.3.0 code. They fail exactly where behaviour changed: the dtype error, crop speed, the caller's random state, and the livebook harness. Code - Edge-chunk crop (F2): it now copies one contiguous run per row. A 999×1000 edge chunk takes under 1 ms instead of 305 ms. The results match to_tensor/2 on 1D through 4D arrays with edges on every axis. While making this change I found that the review's "duplicate shape computation" point was wrong: the stream's metadata.shape is the nominal chunk shape, not the cropped one. Using it broke every edge chunk, so the code still computes the cropped shape from the chunk bounds. - Shuffled batching (F3): - Batches are grouped in a single pass, which removes the quadratic slowdown. - The caller's random state is untouched, with or without a seed. The default seed was quietly using the caller's random generator, which I only caught by testing. - A given seed produces exactly the same order as 1.3.0, pinned by tests. - A short final batch that drop_remainder will drop is no longer read first. - Smaller fixes: an unsupported dtype now gives one error instead of one per chunk (F13), the unreachable clause is gone (F12), and one shared helper handles :names and :backend. That last change means to_tensor_chunked/3 now honours :names too. Livebooks and tooling - Codecs livebook (F5): it crashed on gzip, which is a Zarr v3 codec only, and then on compressor: nil, which needs to be :none. Both are fixed, and it runs end to end. - Axon notebook (F6): the second feature was an exact copy of the first plus 0.05, so the "right weights" the notebook described eren't learnable. It now uses an independent second feature, zero starting weights, Adam and Axon 0.8's model-state API. It prints weights: [2.0, -0.0], bias: [1.0], predictions: [1.0, 3.0], the same on every run, with no deprecation warning. - mix livebook.test (F7): it now runs cells one at a time like Livebook, names the failing cell, and keeps the end of long output where the error is. A new --local option tests against the checkout, and a new CI job runs every shipped livebook that way. Docs - Explorer scope (F1): the CHANGELOG, release notes and ROADMAP now say Explorer streaming was planned and dropped, with links to ex_arrow and ex_graphblas. The plan file's statuses are updated, with Explorer marked cancelled. - Speed claims: the "5-10× faster, 400-800 MB/s" claims in the Nx guide and module docs didn't hold up. Measured on your M1 Max with an 8 MB array, to_tensor/2 takes 75 ms uncompressed and 117 ms with the default zlib. I replaced the claims with those labelled numbers. - Remaining review itms: the reduction example uses include_missing and on_error: :halt (F9), the stale shuffle example is fixed (F8), and the pin (F10), comma (F11), headings (F15), broken bold (F16), new long dashes and CHANGELOG wording (F14) are corrected. Beyond the review, found while fixing: - Six guides said :gzip "always works". It only works for Zarr v3 arrays, so they now say so. - compressor: nil also appeared in two livebook formance.md; those are fixed too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To show the new tests check something real, I ran them against the released 1.3.0 code. They fail exactly where behaviour changed: the dtype error, crop speed, the caller's random state, and the livebook harness.
Code
Livebooks and tooling
Docs
Beyond the review, found while fixing: