Conversation
beacon-arrow-tiff was the last nd format still on the v1 broadcast, which built the pixel cross-product per column inside the opener. It now runs the same spine as netCDF, HDF5 and Zarr. - The opener emits `beacon.nd`-encoded batches through `any_dataset_as_encoded_stream` and adapts them in the encoded struct domain. - `create_physical_plan` sets the file source schema to `encoded_schema(file_schema)` and returns `NdBroadcastExec(NdSourceExec(DataSourceExec))`. - `COUNT(*)` keeps the v1 zero-column driver path, and now also drives with the predicate columns, which the old `any_dataset_as_row_size` path ignored. A raster is a grid over `y` (image rows) and `x` (image columns): bands on (y, x), `geo.lat` on y, `geo.lon` on x, TIFF tags as rank-0 scalars. The two nd optimizer rules therefore now reach a raster. `read_tiff` also gains the optional `dimensions` argument the other nd readers take, with `OPTIONS (read_dimensions '...')` as the external-table equivalent, and `resolve_read_dimensions` in both schema inference and the opener.
Conflict in beacon-arrow-tiff's `infer_schema`: main bounded the schema fetch with `buffered(meta_fetch_concurrency)` (#361, file-descriptor exhaustion) while this branch added the `read_dimensions` argument to `fetch_schema`. Resolved by keeping main's bounded stream and passing the dimensions through it. The test module conflicted only because main reformatted the opener-level tests this branch replaces with SessionContext ones; kept this branch's. `table_function.rs` merged cleanly: main's `FastObjectTable::try_new` (#364) with this branch's optional `dimensions` argument.
The scan-source docs list which formats return more than a bare `DataSourceExec`, because that is why both the metric recorder and the pruner descend the single-child chain. GeoTIFF joined that set in this branch.
…features/tiff-nd-pipeline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beacon-arrow-tiffwas the last nd format still on the v1 broadcast: it built the pixelcross-product per column inside the opener. It now runs the same spine as netCDF, HDF5 and Zarr.
What changed
The spine. The opener emits
beacon.nd-encoded batches throughany_dataset_as_encoded_streamand adapts them in the encoded struct domain.TiffFormat::create_physical_plansets the file source schema toencoded_schema(file_schema)and returns
NdBroadcastExec(NdSourceExec(DataSourceExec)).A raster is a grid over
y(image rows) andx(image columns): bands on(y, x),geo.latony,geo.lononx, and the TIFF tags as rank-0 scalars. The cross-product is therefore builtonce, at the top of the plan, and the two nd optimizer rules now reach a raster:
WHERE "geo.lat" > 40sinks into anNdFilterExecand selects the grid before it materializes.SELECT "geo.lat" * 2sinks into anNdProjectionExecand evaluates over the 380-valuelatitude axis instead of all 489060 pixels.
COUNT(*)keeps the v1 zero-column driver path, and now also drives with the predicatecolumns. The old
any_dataset_as_row_sizepath ignored the predicate, so aCOUNT(*)with aWHEREover-counted.Dimension parity.
read_tiff('raster.tif', ['y'])returns one row per image row, matchingread_netcdf/read_hdf5/read_zarr. The same list works asOPTIONS (read_dimensions 'y')onCREATE EXTERNAL TABLE ... STORED AS TIFF, andresolve_read_dimensionsnow runs in both schema inference and the opener.read_tiff_schemainherits the argument automatically.
Behaviour change worth reviewing
The nd encoding does not carry the target grid;
decode_nd_record_batch_rowre-infers it from theprojected columns. So a projection can change the row count:
SELECT count(*) FROM tiffSELECT count("geo.lat"), count("band.0") FROM tiffSELECT count("geo.lat") FROM tiffThe last row is the change: with the full-rank band projected away, the grid narrows to the axes
the surviving columns carry. Same-rank columns also leave the row order unspecified, because
infer_targettakes the axis order from the widest column.This is a property of the shared encoding, not of the TIFF reader. netCDF, HDF5 and Zarr already
behave this way, and the zarr test
attribute_is_single_distinct_value_across_griddocuments it.Fixing it means carrying the target grid through
encode_nd_record_batch, which changes all fourformats, so it is out of scope here.
Tests
beacon-arrow-tiff: 23 pass. New coverage for the plan shape, projection pushdown, filterpushdown,
COUNT(*), scalar broadcast, andread_dimensions(schema and row count).beacon-core/tests/read_functions.rs: two end-to-end tests through a real runtime, beside theHDF5 ones.
cargo test --workspace --lib --bins --testspasses.cargo clippy --workspace --lib --bins --tests: no new warnings; the crate drops from 7 to 5,all pre-existing.
Docs (
formats/geotiff.md,sql/table-functions.md) and the CHANGELOG record the new argumentand the grid semantics.