Skip to content

Index.from_chunks(): manifest-built indices for the parquet-primary sidecar store#2

Merged
espg merged 3 commits into
mainfrom
from-chunks
Jul 4, 2026
Merged

Index.from_chunks(): manifest-built indices for the parquet-primary sidecar store#2
espg merged 3 commits into
mainfrom
from-chunks

Conversation

@espg

@espg espg commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Closes #1. Implemented exactly per the plan posted there
(#1 (comment)).

What

  • Index.from_chunks(source, datasets) (staticmethod): construct a
    decode-capable index from an externally-supplied chunk manifest — no
    granule file access, no stored bincode. Plain python/numpy inputs only;
    parquet decoding stays caller-side.
  • Index.filters(dataset){"gzip": int|None, "shuffle": bool, "byte_order": "little"|"big"|"unknown"} — the missing extractor-side
    getter; datasets()/shape()/chunk_shape()/dtype()/filters()/chunks() now
    produce everything from_chunks consumes.
  • README: usage, the extractor metadata contract, the filter_mask
    limitation.

Construction route (hybrid, as planned)

The per-dataset payload is built entirely with hidefix 0.12 public API
(Chunk::new, Dataset::new, DatasetD::D1..D9, Datatype, Order).
The two non-constructible shells (Index, GroupIndex — private fields)
go through in-memory serde shims with bincode-identical layout, serialized
and immediately deserialized through the same code path load() uses
(shared holder_from_bytes helper). Bincode never touches disk.

Validation / errors

ValueError with dataset name + offending key on: missing keys, rank
mismatches, chunk-table length mismatches, wrong chunk count vs
ceil(shape/chunk_shape), non-chunk-aligned or out-of-bounds offsets,
duplicate offsets, unsupported dtype, gzip given as bool, and any nonzero
filter_mask (hidefix's chunk model is (addr, size, offset) — per-chunk
filter masks are unrepresentable; ATL03 masks are all zero per
englacial/zagg PR #159). Chunk rows are sorted internally.

Tests

45 total (was 37), all green both tiers:

  • hermetic: extract-via-getters → from_chunks → byte-identical
    read/read_plan/read_from_buffers vs the directly-built index
    (incl. empty + boundary-crossing ranges), filters() values vs h5py,
    save/load coherence of a from_chunks index, unsorted rows, plain-list
    inputs, no-file-access construction (phantom source path), and 15+ error
    paths.
  • local (NEON ATL03 granule): extract → from_chunks → byte-identical direct
    and buffer-fed reads.

clippy -D warnings, fmt, and both CI workflows green.

@espg espg merged commit 004fabc into main Jul 4, 2026
2 checks passed
@espg espg deleted the from-chunks branch July 4, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Index.from_chunks(): build a decode-capable index from an external chunk manifest (parquet-primary store)

1 participant