π€ from Claude
Problem
The ragged t-digest output bypasses the Zarr v3 sharding codec, so it is written as one CSR subgroup per inner chunk β ~1,792 objects per shard β while the fixed-width companion arrays collapse to one object per shard. This is documented as intentional in src/zagg/processing/write.py:279 ("companions and ragged (CSR) fields are NOT sharded β they are written per inner chunk"), but it inflates the write phase badly as shards get denser, and it was invisible until a full-AOI / CONUS-scale run surfaced it (refs #202).
Evidence
For a single o9 tdigest shard (atl03_tdigest_healpix_o9.yaml, sharded: true, chunk_inner: 13 β K=256 inner chunks), object counts in the written store:
| array |
objects / shard |
sharded? |
cell_ids, count, morton (dense, fixed-width) |
1 each |
yes β ShardingCodec collapses the K=256 inner chunks |
h_tdigest (ragged CSR) |
~1,792 |
no β 256 inner-chunk subgroups Γ 7 objects (cell_ids/offsets/values Γ {chunk + zarr.json} + group zarr.json) |
So a 4-shard NEON full-AOI write emitted 7,186 objects, ~7,168 of them the t-digest.
The write-time cost scales with occupied cells / observations because each occupied inner chunk is its own set of tiny PUTs:
| shard |
obs |
cells |
write phase |
shard total |
| NEON o9 (63 gran) |
6.2 M |
118 k |
~25 s |
130 s |
| CONUS o9 (144 gran) |
4.6 M |
122 k |
~153 s |
505 s |
| CONUS o9 (62 gran, dense) |
23.0 M |
176 k |
~165 s |
448 s |
At NEON scale the write is dwarfed by the ~105 s read, which is why the existing single-shard benchmarks (tdigest_healpix_o9_sharded etc.) never exposed it β benchmark.md reports runtime/cost/memory, not object counts. At CONUS scale the write is ~a third of the shard.
Impact
Proposed options
We need something "sharding-codec-like" for the t-digest. Candidates:
- Fixed-length centroid vectors (pad each cell's digest to
delta = 512 centroids) β a dense, fixed-shape array that rides the default ShardingCodec unchanged. Cost is larger raw writes from the padding β but the ShardingCodec's inner compression codec should squeeze the padded/empty tail, so the on-disk and PUT-count cost may be modest. This is the option to investigate first: measure padded-and-compressed size + object count vs the current ragged CSR, across a sparse NEON shard and a dense CONUS shard. If compression eats the padding, this is the cheapest fix (no new machinery, just a fixed-width digest layout).
- One CSR blob per shard β serialize the whole shard's ragged digest (all K inner chunks) as a single object, shard-granular like the dense arrays, decoded on read.
- A Zarr extension for sharded variable-length arrays β the most general, the most work.
Open question to settle
Does the ShardingCodec's compression make fixed-length padding cheap enough that option (1) wins on simplicity? That is the key measurement β a fixed-length digest that compresses well would let us drop the per-inner-chunk CSR entirely and reuse the existing sharded write path. Worth measuring padded+compressed bytes before committing to (2) or (3).
Refs: #202 (where this surfaced), write.py:279 (the current per-inner-chunk CSR path), #30/#82/#142/#186 (the ragged/sharded write history). espg to triage/label.
π€ from Claude
Problem
The ragged t-digest output bypasses the Zarr v3 sharding codec, so it is written as one CSR subgroup per inner chunk β ~1,792 objects per shard β while the fixed-width companion arrays collapse to one object per shard. This is documented as intentional in
src/zagg/processing/write.py:279("companions and ragged (CSR) fields are NOT sharded β they are written per inner chunk"), but it inflates the write phase badly as shards get denser, and it was invisible until a full-AOI / CONUS-scale run surfaced it (refs #202).Evidence
For a single o9 tdigest shard (
atl03_tdigest_healpix_o9.yaml,sharded: true,chunk_inner: 13β K=256 inner chunks), object counts in the written store:cell_ids,count,morton(dense, fixed-width)h_tdigest(ragged CSR)cell_ids/offsets/valuesΓ {chunk +zarr.json} + groupzarr.json)So a 4-shard NEON full-AOI write emitted 7,186 objects, ~7,168 of them the t-digest.
The write-time cost scales with occupied cells / observations because each occupied inner chunk is its own set of tiny PUTs:
At NEON scale the write is dwarfed by the ~105 s read, which is why the existing single-shard benchmarks (
tdigest_healpix_o9_shardedetc.) never exposed it βbenchmark.mdreports runtime/cost/memory, not object counts. At CONUS scale the write is ~a third of the shard.Impact
zarr.jsonmetadata bloat).Proposed options
We need something "sharding-codec-like" for the t-digest. Candidates:
delta= 512 centroids) β a dense, fixed-shape array that rides the default ShardingCodec unchanged. Cost is larger raw writes from the padding β but the ShardingCodec's inner compression codec should squeeze the padded/empty tail, so the on-disk and PUT-count cost may be modest. This is the option to investigate first: measure padded-and-compressed size + object count vs the current ragged CSR, across a sparse NEON shard and a dense CONUS shard. If compression eats the padding, this is the cheapest fix (no new machinery, just a fixed-width digest layout).Open question to settle
Does the ShardingCodec's compression make fixed-length padding cheap enough that option (1) wins on simplicity? That is the key measurement β a fixed-length digest that compresses well would let us drop the per-inner-chunk CSR entirely and reuse the existing sharded write path. Worth measuring padded+compressed bytes before committing to (2) or (3).
Refs: #202 (where this surfaced), write.py:279 (the current per-inner-chunk CSR path), #30/#82/#142/#186 (the ragged/sharded write history). espg to triage/label.