enhancement(packaging): vendor libgdal-arrow-parquet (OGR Parquet driver) into the platform wheel

[MAfarrag]

Context

Pyramids ships a fat platform wheel that vendors GDAL/osgeo (so pip users get GDAL without installing it
separately) and lets conda-forge users get GDAL from the feedstock. That wheel already bundles GDAL driver
plugins (NetCDF / HDF4 / HDF5) into src/pyramids/_data/gdalplugins/.

This proposal surfaced from PR #455 (LabeledDataset). While fixing that PR's CI, the parquet pixi feature
(pyarrow) was temporarily added to the py311-py314 test-matrix envs. That un-skipped a pre-existing
core test, tests/feature/test_tile_strategy.py::TestRowGroupParquet::test_row_group_parquet, which then
failed on Linux/macOS with:

pyogrio.errors.DataSourceError: '.../points.parquet' not recognized as being in a supported file format.
It could have been recognized by driver Parquet, but plugin ogr_Parquet.so is not available in your
installation. You may install it with 'conda install -c conda-forge libgdal-arrow-parquet'.

It passed on Windows because the conda-forge Windows GDAL build bundles the Arrow/Parquet driver in-tree;
on Linux/macOS conda-forge splits it into a separate libgdal-arrow-parquet package that is not installed.
PR #455 resolved this by dropping parquet from those envs (keeping only lazy), restoring the prior skip.
This issue tracks doing it properly, decoupled from #455.

Problem / Current Behaviour

There are two distinct Parquet capabilities, and they are independent - having one does not give the
other:

1. Python pyarrow (the [parquet] extra) - used for writing Parquet from pandas/geopandas
   (df.to_parquet, gdf.to_parquet) and LabeledDataset.to_parquet. Already available and pip-portable
   (pyarrow is a normal PyPI dependency).
2. GDAL OGR Parquet driver (libgdal-arrow-parquet / ogr_Parquet.so) - used for reading a
   .parquet as a vector layer through GDAL/pyogrio, e.g.
   FeatureCollection.iter_features(p, bbox=..., tile_strategy="row_group") (streaming + bbox + row-group
   pushdown). This is the piece missing on Linux/macOS conda envs without the driver and on all pip
   installs (the standard gdal wheel does not bundle the Arrow/Parquet driver).

Installing libgdal-arrow-parquet does not provide Python pyarrow, and vice-versa; the row-group test
needs both. The result is a real conda-vs-pip (and Windows-vs-Linux/macOS) asymmetry in GDAL-side Parquet
vector reading.

Affected locations

File	Symbol	Notes
ci/install-and-vendor-osgeo.py	plugin-copy step (~L365-375)	Already copies lib/gdalplugins/ to _data/gdalplugins/; would pick up ogr_Parquet automatically
pyproject.toml	[tool.pixi.dependencies] (gdal, libgdal-netcdf, libgdal-hdf4)	Where libgdal-arrow-parquet would be added for the wheel build
.github/workflows/build-wheels.yml	wheel repair step	Must explicitly include the Arrow C++ libs (dlopen'd plugin)
src/pyramids/feature*	FeatureCollection.iter_features	The consumer of the OGR Parquet read driver
tests/feature/test_tile_strategy.py	TestRowGroupParquet::test_row_group_parquet	Skips today; would run skip-free with the driver present
ci/check-wheel-size.sh	WHEEL_SIZE_BUDGET_MB (120)	Size gate the added libs eat into

Motivation Example

from pyramids.featurecollection import FeatureCollection

# Today: works on conda-Windows; on Linux/macOS conda without libgdal-arrow-parquet, and on ALL pip
# installs, this raises DataSourceError ("plugin ogr_Parquet.so is not available").
feats = list(
    FeatureCollection.iter_features("reaches.parquet", bbox=(0, 0, 10, 10), tile_strategy="row_group")
)

Proposed Solution

Vendor libgdal-arrow-parquet into the platform wheel using the existing osgeo-vendoring machinery:

1. Add libgdal-arrow-parquet to the wheel-build conda deps (next to libgdal-netcdf / libgdal-hdf4).
   The vendor script's existing plugin-copy step then bundles ogr_Parquet.so (and the Arrow driver
   .dll/.dylib) into _data/gdalplugins/ with no change to the copy logic.
2. Catch - the plugin is dlopen'd at runtime, so the wheel-repair tools
   (auditwheel / delvewheel / delocate) will not auto-discover its Arrow C++ deps via the osgeo._gdal
   extension's DT_NEEDED. The Arrow runtime libs (libarrow, libparquet, libarrow-dataset,
   libarrow-acero, libarrow-compute) must be explicitly added to the vendor/repair step so they land
   in pyramids_gis.libs/.
3. ABI: conda-forge solves libgdal-arrow-parquet against the pinned libgdal 3.12, so the binaries
   match.

Size cost (measured)

The plugin itself is tiny (~0.8 MB); the weight is the Arrow C++ runtime it links. Measured uncompressed
on-disk:

Library	Uncompressed	Needed by the OGR Parquet driver?
libgdal-arrow-parquet (plugin)	~0.8 MB	yes (driver glue)
arrow	~13.2 MB	yes
arrow_compute	~8.8 MB	yes (dataset/acero depend on it)
parquet	~2.6 MB	yes
arrow_dataset	~1.3 MB	yes (row-group/dataset API)
arrow_acero	~1.2 MB	yes (dataset engine)
arrow_python*	~1.4 MB	no - pyarrow-only, not the GDAL driver

• ~28 MB uncompressed total for the driver + its Arrow runtime.
• Compressed conda downloads: libgdal-arrow-parquet 0.8 MB + libarrow 6.5 MB + libparquet 1.4 MB +
  libarrow-acero 0.6 MB ~= 9.3 MB.
• Native libs compress ~2.5-3x in the .whl, so this adds ~10-12 MB to every platform wheel's download
  (Linux/macOS/Windows x py311-py314), paid by all users - including those who never read Parquet.
• Wheel budget is 120 MB per .whl (WHEEL_SIZE_BUDGET_MB, rationale in
  planning/bundle/wheel-size-analysis.md). ~10-12 MB fits within headroom but is a permanent, meaningful
  chunk of it.

Pros

• pip/PyPI users gain GDAL-side Parquet vector reading - closes the conda-vs-pip asymmetry that exists
  today.
• FeatureCollection.iter_features row-group pushdown works uniformly across all install paths.
• test_row_group_parquet (and LabeledDataset.to_parquet) can run skip-free on the Linux/macOS matrix when
  the env has the driver.
• Reuses the existing vendoring design; minimal new infra (a conda dep + explicit repair-step inclusion).

Cons

• +~10-12 MB to every wheel for all users, eating wheel-size-budget headroom, for a feature many users
  never use.
• More native libs to keep ABI-aligned with libgdal across version bumps (another pin/maintenance
  surface).
• Repair-step needs explicit handling because the driver is dlopen'd (the repair tools won't auto-bundle
  the Arrow deps).
• conda-forge already provides this via the feedstock; the only net-new beneficiaries are pip users and
  Linux/macOS conda envs lacking the driver.

Alternative

Move FeatureCollection's Parquet reading off the GDAL OGR driver onto the pure-Python
geopandas/pyarrow path (gpd.read_parquet(bbox=...)). pyarrow is a normal pip dependency (not vendored), so
this adds ~0 to the wheel and is fully pip/conda portable - at the cost of a Parquet-specific code path
that diverges from the unified pyogrio/OGR reader and may have different row-group pushdown semantics. This is
the lighter-weight alternative to bundling ~28 MB.

Decision asked

Choose between:

• (A) Vendor libgdal-arrow-parquet + Arrow runtime into the wheel - skip-free, uniform GDAL reader,
  +~12 MB/wheel. Requires: add the conda dep, extend the repair/vendor step to include the Arrow libs, add
  libgdal-arrow-parquet to the relevant test envs, and re-enable the row-group test on Linux/macOS.
• (B) Keep the GDAL driver conda-only and optionally add a geopandas/pyarrow read path for pip
  portability - no wheel growth.

Out of Scope

• The LabeledDataset PR (feat(netcdf): label-indexed (non-gridded) NetCDF/Zarr reader — PY-G subset (P-A…P-F, P-B) #455) itself - that ships with parquet excluded from the matrix and is unaffected
  by this decision.
• Any change to the Python pyarrow write path (to_parquet), which already works pip-portably.

Effort Estimate

Size: M
Rationale: small conda-dep + explicit repair-step change, but cross-platform wheel verification
(auditwheel/delvewheel/delocate must actually bundle and load the Arrow libs at runtime on
Linux/macOS/Windows) plus a size-budget re-check is the bulk of the work.

Definition of Done

• Decision recorded (A or B) on this issue.
• If (A): libgdal-arrow-parquet added to the wheel-build conda deps; Arrow runtime libs explicitly
  vendored into pyramids_gis.libs/.
• If (A): a built wheel on Linux, macOS, and Windows can dlopen the OGR Parquet driver and read a
  .parquet via FeatureCollection.iter_features at runtime (verified in wheel-test.yml).
• If (A): test_row_group_parquet runs (not skipped) on the Linux/macOS matrix and passes.
• Wheel sizes re-measured and confirmed within the WHEEL_SIZE_BUDGET_MB ceiling.
• Docs/installation notes updated to state GDAL-side Parquet vector reading is available on pip installs.
• All existing tests continue to pass.

[MAfarrag]

Cost update — the "one conda dep + existing machinery" premise no longer holds

This proposal was written when every platform wheel was conda-extract, so vendoring the driver meant adding
libgdal-arrow-parquet next to libgdal-netcdf and letting the plugin-copy step pick it up. Since #680 and
#695 there are three build models, and each would need Arrow separately:

• macOS + win_amd64 (conda-extract): the original plan still works — one conda dep + explicit Arrow libs
  in the delocate/delvewheel repair step.
• Linux (from-source, ci/source-build/config.sh): Arrow C++ (arrow, parquet, arrow_dataset,
  arrow_acero, arrow_compute) plus the GDAL driver would have to join the pinned tarball table and build
  from source. Arrow is a heavy CMake build — real cold-release wall-clock on every release (Linux release
  builds are always cold by design), plus five more version+SHA256 pins to maintain.
• win_arm64 (vcpkg, ci/vcpkg.json): needs the Arrow/Parquet feature through the vcpkg manifest and the
  single delvewheel pass picking up the dlopen'd plugin's deps.

The dlopen catch (repair tools cannot auto-discover the plugin's Arrow deps) now has to be solved three times,
once per repair tool. And a partial rollout (conda platforms only) is off the table by design:
ci/verify-wheel.py asserts the exact same driver set on every platform wheel, so the driver ships everywhere
or nowhere.

Unchanged from the original analysis: ~10–12 MB added to every wheel download for all users, for the GDAL-side
Parquet vector read path only (Parquet writing via pyarrow already works on every install path).

Recommendation recorded here for when this gets picked up: defer until there is actual user demand for
FeatureCollection.iter_features over Parquet on pip installs. If demand materializes, the better shape is
likely an optional companion plugin wheel wired behind the existing [parquet] extra (driver + Arrow libs on
GDAL_DRIVER_PATH), so the cost lands only on users who opt in — at the price of new packaging infra that
must stay ABI-aligned with the bundled libgdal.