Rebuild the Atlas reader on the single-file format - #490
Merged
Merged
Conversation
Atlas 0.16 replaced the directory of per-array files with one write-once container, `data.atlas`, holding every dataset and a footer that describes them all. The reader here was written against the old layout, every call it made is gone, and the crate had been excluded from the workspace since the morsel scan landed. `STORED AS ATLAS` and `read_atlas` failed. The crate is rewritten on the new format and registered again. A `LOCATION` names the container rather than a marker beside it. A collection written before 0.16 is not read at all: its registry is not a marker, so a listing passes over it. There is no compatibility path. One dataset is one unit of work. The format lists a collection at plan time and emits one entry per dataset, those entries go into the shared morsel queue, and the scan sits under the nd spine like netCDF and Zarr. A worker takes the next dataset when it is free and helps drain an open one when none is left, so a collection of a million small datasets and one of four large ones both divide over every core. Level two of the queue follows the chunk shape the writer chose, so one pop reads one stored chunk. A predicate skips whole datasets. The footer records the minimum, the maximum and the null count of every array, so the first scan of a collection pivots those into one index -- one row per dataset, one typed Arrow column per column the predicate names -- and judges every dataset in a single vectorised pass. A million datasets cost one pass rather than a million decisions. A dataset-level attribute is exact in the footer, so a predicate on one prunes too. Every path fails open, and the filter above the scan still decides each row. Three behaviour changes come with the rebuild. A dataset attribute is a column under a leading dot, matching netCDF and Zarr rather than the bare key. A column two datasets type in two families refuses the merge by name instead of silently becoming text; `keep_first` settles it the other way. And collections are crawlable now, because a collection is one file whose extension is its format. Atlas applies no CF decoding: it has a native timestamp type and `atlas create` applies scale, offset and time units before the write. Verified against a collection built by the real `atlas create`: its per-dataset statistics are present, so pruning works on collections built the normal way, and xarray's NaN fills and marker attributes read as documented.
The plan served the rebuild and is finished. What it decided is in the code and its doc comments, what it changed for a user is in the changelog and the format page, and the four requests it made of atlas-rust belong upstream rather than in this repository.
Atlas 0.17 changes where the bytes live. A container used to hold one segment per dataset, with the footer carrying every dataset's shapes, attribute values and statistics. It now holds one segment per *variable*: a segment holds one array name across the whole collection, and each dataset's copy sits inside it under the dataset's own name. The footer therefore names things and nothing else. An array's layout (shape, chunking, dimension names, fill value), its statistics, and every attribute value moved into the variable's segment, and reading one is async. One open answers for every dataset of the collection, so a column costs one request whether the collection holds ten datasets or a million. What that changes here: - The dataset build asks for a layout per array instead of reading it off the footer, and skips a dtype Beacon cannot surface before it opens anything. - The pruning index gathers a column with one call — `array_stats_by_dataset` for an array, `attributes_by_dataset` for an attribute — and pivots what comes back. That drops the 100k-dataset limit on indexing an attribute, which existed only because an attribute had needed a view per dataset. - Schema inference keys a dataset on its arrays, its attribute keys and its dimension names. The interned schema is coarser than it was — it names types and no longer implies a grid — so the names come from the segments to keep two datasets with different grids apart. - `COUNT(*)` picks its driving array by element count from the layouts. - An attribute can no longer be a timestamp: atlas stores none, because one would go to disk as an i64 and could not come back. Verified against the fixtures atlas-rust 0.17 ships, including the one its Python layer writes: schema, values, timestamps, string arrays and both attribute scopes all read back.
A read that projects no column loaded no array, so every dataset sat on a rank-0 grid and `count(*)` returned the dataset count. The view now drives such a read with each dataset's widest array, and a chunk states its row count through `DatasetSource::chunk_rows`, so a count reads no cell. The changelog and the format, tuning, configuration and external-table pages named four `BEACON_ATLAS_*` settings and two metrics that no longer exist. They now describe the reader cache, pruning and the metrics as the code has them. The beacon-core atlas tests referenced the removed `AtlasConfig`. The pruning test now compares a predicate against a full read filtered in memory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Atlas moved to one write-once container,
data.atlas. The old reader used the directory layout and was out of the build. This PR rebuilds the reader on the 0.17 container and registersSTORED AS ATLASandread_atlasagain.Changes
LOCATIONnames the container, or a glob such asobs/**/data.atlas. Beacon does not read a collection from before 0.17.count(*)returns the full row count and reads no cell.Docs and tests
Known gaps
read_dimensionsbut does not apply it.