Problem
Large prediction exports (e.g. dataset 665 with ~374k geometries) are currently generated via Python feature materialization and GeoPandas/Fiona writes. This can take a long time and may remain in processing for extended periods.
Goal
Add an optional GDAL-native export backend for labels downloads to improve throughput and stability for large datasets, while keeping the current Python exporter as a fallback.
Proposed approach
- Add a configurable exporter backend flag (e.g.
LABEL_EXPORT_BACKEND=python|gdal), defaulting to python initially.
- Implement a GDAL-based exporter path in the API background task using
ogr2ogr.
- Build deterministic SQL layer exports for a dataset into one GPKG:
deadwood_model_prediction
deadwood_visual_interpretation
forest_cover_model_prediction
forest_cover_visual_interpretation
aoi
- Keep existing route/error contract (
.error marker file, status endpoint returns failed with message).
- Add timeout + robust subprocess logging (stdout/stderr + elapsed time).
Configuration
- Do not read DSN from local MCP config files.
- Introduce explicit environment variable for export DSN (e.g.
GDAL_EXPORT_PG_DSN) managed via deployment secrets.
- Prefer direct local Postgres endpoint if available for large export workloads.
Testing plan
- Unit tests:
- command/SQL builder for each layer
- dataset/source filtering correctness
- Integration tests (api-test container):
- export completes
- expected layers exist in output GPKG
- feature counts match DB counts for fixture dataset
- Failure-path tests:
- invalid DSN / timeout ->
.error marker + failed status response
- Performance smoke benchmark:
- compare current Python path vs GDAL path on a large dataset and record wall-clock time.
Acceptance criteria
- New backend can be enabled with config toggle.
- Endpoints remain backward compatible.
- Export succeeds with correct layer schema/content.
- Clear error propagation on failure.
- Documented rollout plan to switch default backend after validation.
Out of scope
- Pre-generating/caching exports.
- Changing download response format away from GPKG.
Problem
Large prediction exports (e.g. dataset 665 with ~374k geometries) are currently generated via Python feature materialization and GeoPandas/Fiona writes. This can take a long time and may remain in
processingfor extended periods.Goal
Add an optional GDAL-native export backend for labels downloads to improve throughput and stability for large datasets, while keeping the current Python exporter as a fallback.
Proposed approach
LABEL_EXPORT_BACKEND=python|gdal), defaulting topythoninitially.ogr2ogr.deadwood_model_predictiondeadwood_visual_interpretationforest_cover_model_predictionforest_cover_visual_interpretationaoi.errormarker file, status endpoint returnsfailedwith message).Configuration
GDAL_EXPORT_PG_DSN) managed via deployment secrets.Testing plan
.errormarker +failedstatus responseAcceptance criteria
Out of scope