Optimize large S3 dataset listings - #416

Open

robinskil wants to merge 26 commits into

mainfrom

features/optimize-large-listings-s3

Commits on Aug 19, 2026

Request large listing pages from S3, and repair the S3 back-compat flag
robinskil
committed
9300fd4
Shard a recursive listing across sub-directories, and size the pages
robinskil
committed
1a0890b
Propagate listing errors, and correct why the page size is capped
robinskil
committed
7153912
Browse the datasets store one directory at a time
robinskil
committed
78e81ec
Narrow dataset listings to what the caller may read
robinskil
committed
3af79c7

Commits on Aug 20, 2026

Put path visibility behind one type
robinskil
committed
db2b6ea
List datasets from a table provider, not from the table function
robinskil
committed
df01388
Browse through SQL, and stop filtering listings by grants
robinskil
committed
1c1613d
Merge origin/main into features/optimize-large-listings-s3
robinskil
committed
4ba4a24
Drop the listing authorization work from this branch
robinskil
committed
5dd4e82
Give listing its own module, and wrap the store where it is built
robinskil
committed
0125e45
Detect a Zarr store from its own path
robinskil
committed
1ec315f
Resolve a Zarr store the same way everywhere
robinskil
committed
6a369d1
Stream a listing, and give it a plan node
robinskil
committed
e170d8d
Stream objects from the listing, not datasets
robinskil
committed
f8477b2
Hand out a resolved listing instead of three ways to ask for one
robinskil
committed
6c7e438

Commits on Aug 21, 2026

Fix the two failures CI found
robinskil
committed
ae4a8dc
Test the listing, and fix the duplicate it found
robinskil
committed
30c4d34
Merge branch 'main' into features/optimize-large-listings-s3
robinskil
authored
6a7408c

Commits on Sep 4, 2026