Skip to content

Read GeoTIFF and TIFF through the nd pipeline - #363

Open
robinskil wants to merge 6 commits into
mainfrom
features/tiff-nd-pipeline
Open

robinskil wants to merge 6 commits into
mainfrom
features/tiff-nd-pipeline

Conversation

@robinskil

Copy link
Copy Markdown
Collaborator

beacon-arrow-tiff was the last nd format still on the v1 broadcast: it built the pixel
cross-product per column inside the opener. It now runs the same spine as netCDF, HDF5 and Zarr.

What changed

The spine. The opener emits beacon.nd-encoded batches through
any_dataset_as_encoded_stream and adapts them in the encoded struct domain.
TiffFormat::create_physical_plan sets the file source schema to encoded_schema(file_schema)
and returns NdBroadcastExec(NdSourceExec(DataSourceExec)).

A raster is a grid over y (image rows) and x (image columns): bands on (y, x), geo.lat on
y, geo.lon on x, and the TIFF tags as rank-0 scalars. The cross-product is therefore built
once, at the top of the plan, and the two nd optimizer rules now reach a raster:

  • WHERE "geo.lat" > 40 sinks into an NdFilterExec and selects the grid before it materializes.
  • SELECT "geo.lat" * 2 sinks into an NdProjectionExec and evaluates over the 380-value
    latitude axis instead of all 489060 pixels.

COUNT(*) keeps the v1 zero-column driver path, and now also drives with the predicate
columns. The old any_dataset_as_row_size path ignored the predicate, so a COUNT(*) with a
WHERE over-counted.

Dimension parity. read_tiff('raster.tif', ['y']) returns one row per image row, matching
read_netcdf / read_hdf5 / read_zarr. The same list works as
OPTIONS (read_dimensions 'y') on CREATE EXTERNAL TABLE ... STORED AS TIFF, and
resolve_read_dimensions now runs in both schema inference and the opener. read_tiff_schema
inherits the argument automatically.

Behaviour change worth reviewing

The nd encoding does not carry the target grid; decode_nd_record_batch_row re-infers it from the
projected columns. So a projection can change the row count:

query before after
SELECT count(*) FROM tiff 489060 489060
SELECT count("geo.lat"), count("band.0") FROM tiff 489060 489060
SELECT count("geo.lat") FROM tiff 489060 380

The last row is the change: with the full-rank band projected away, the grid narrows to the axes
the surviving columns carry. Same-rank columns also leave the row order unspecified, because
infer_target takes the axis order from the widest column.

This is a property of the shared encoding, not of the TIFF reader. netCDF, HDF5 and Zarr already
behave this way, and the zarr test attribute_is_single_distinct_value_across_grid documents it.
Fixing it means carrying the target grid through encode_nd_record_batch, which changes all four
formats, so it is out of scope here.

Tests

  • beacon-arrow-tiff: 23 pass. New coverage for the plan shape, projection pushdown, filter
    pushdown, COUNT(*), scalar broadcast, and read_dimensions (schema and row count).
  • beacon-core/tests/read_functions.rs: two end-to-end tests through a real runtime, beside the
    HDF5 ones.
  • cargo test --workspace --lib --bins --tests passes.
  • cargo clippy --workspace --lib --bins --tests: no new warnings; the crate drops from 7 to 5,
    all pre-existing.

Docs (formats/geotiff.md, sql/table-functions.md) and the CHANGELOG record the new argument
and the grid semantics.

beacon-arrow-tiff was the last nd format still on the v1 broadcast, which
built the pixel cross-product per column inside the opener. It now runs the
same spine as netCDF, HDF5 and Zarr.

- The opener emits `beacon.nd`-encoded batches through
  `any_dataset_as_encoded_stream` and adapts them in the encoded struct
  domain.
- `create_physical_plan` sets the file source schema to
  `encoded_schema(file_schema)` and returns
  `NdBroadcastExec(NdSourceExec(DataSourceExec))`.
- `COUNT(*)` keeps the v1 zero-column driver path, and now also drives with
  the predicate columns, which the old `any_dataset_as_row_size` path
  ignored.

A raster is a grid over `y` (image rows) and `x` (image columns): bands on
(y, x), `geo.lat` on y, `geo.lon` on x, TIFF tags as rank-0 scalars. The two
nd optimizer rules therefore now reach a raster.

`read_tiff` also gains the optional `dimensions` argument the other nd
readers take, with `OPTIONS (read_dimensions '...')` as the external-table
equivalent, and `resolve_read_dimensions` in both schema inference and the
opener.
@robinskil robinskil self-assigned this Aug 9, 2026
@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.34572% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.06%. Comparing base (0ac879e) to head (e067706).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...-db/beacon-datafusion-ext/src/fast_object/prune.rs 79.32% 43 Missing ⚠️
...ile-formats/beacon-arrow-odv/src/datafusion/mod.rs 0.00% 24 Missing ⚠️
...formats/beacon-arrow-tiff/src/datafusion/source.rs 71.79% 22 Missing ⚠️
...beacon-arrow-tiff/src/datafusion/table_function.rs 59.37% 13 Missing ⚠️
...-db/beacon-datafusion-ext/src/fast_object/table.rs 94.15% 9 Missing ⚠️
...le-formats/beacon-arrow-tiff/src/datafusion/mod.rs 97.34% 7 Missing ⚠️
...formats/beacon-arrow-tiff/src/datafusion/reader.rs 71.42% 4 Missing ⚠️
...eacon-arrow-atlas/src/datafusion/table_function.rs 0.00% 3 Missing ⚠️
.../beacon-arrow-bbf/src/datafusion/table_function.rs 0.00% 3 Missing ⚠️
...-arrow-geoparquet/src/datafusion/table_function.rs 0.00% 3 Missing ⚠️
... and 6 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #363      +/-   ##
==========================================
+ Coverage   80.87%   81.06%   +0.19%     
==========================================
  Files         371      370       -1     
  Lines       57527    57811     +284     
==========================================
+ Hits        46525    46865     +340     
+ Misses      11002    10946      -56     
Files with missing lines Coverage Δ
beacon-db/beacon-common/src/file_descriptors.rs 89.47% <100.00%> (+0.58%) ⬆️
beacon-db/beacon-common/src/super_typing.rs 69.20% <ø> (-3.05%) ⬇️
beacon-db/beacon-core/src/runtime_builder.rs 88.46% <100.00%> (+0.08%) ⬆️
beacon-db/beacon-core/src/statement_plan/authz.rs 93.41% <100.00%> (+1.84%) ⬆️
...acon-db/beacon-datafusion-ext/src/ordered_union.rs 76.11% <ø> (ø)
...acon-db/beacon-datafusion-ext/src/unique_values.rs 70.50% <ø> (ø)
...e-formats/beacon-arrow-atlas/src/datafusion/mod.rs 92.20% <100.00%> (+0.45%) ⬆️
.../beacon-arrow-csv/src/datafusion/table_function.rs 87.50% <100.00%> (ø)
...eacon-file-formats/beacon-arrow-hdf5/src/format.rs 93.23% <100.00%> (ø)
...le-formats/beacon-arrow-hdf5/src/table_function.rs 63.63% <100.00%> (ø)
... and 21 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Conflict in beacon-arrow-tiff's `infer_schema`: main bounded the schema fetch
with `buffered(meta_fetch_concurrency)` (#361, file-descriptor exhaustion)
while this branch added the `read_dimensions` argument to `fetch_schema`.
Resolved by keeping main's bounded stream and passing the dimensions through
it.

The test module conflicted only because main reformatted the opener-level
tests this branch replaces with SessionContext ones; kept this branch's.

`table_function.rs` merged cleanly: main's `FastObjectTable::try_new` (#364)
with this branch's optional `dimensions` argument.
The scan-source docs list which formats return more than a bare
`DataSourceExec`, because that is why both the metric recorder and the pruner
descend the single-child chain. GeoTIFF joined that set in this branch.
@robinskil robinskil mentioned this pull request Aug 17, 2026
63 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant