Add a GeoParquet map source (stacked on #166 EF5 floodmaps) - #167
Closed
JakeGimenes wants to merge 24 commits into
Closed
Add a GeoParquet map source (stacked on #166 EF5 floodmaps)#167JakeGimenes wants to merge 24 commits into
JakeGimenes wants to merge 24 commits into
Conversation
Add a /tethysdash/floodmap/cog controller that reads a public Zarr flood-depth store over HTTPS, slices one storm scenario, and streams a Cloud-Optimized GeoTIFF generated in memory (nothing persisted). The existing GeoTIFF map layer consumes the endpoint URL directly. - floodmap.py: Django-free zarr->COG conversion + store metadata; uses direct member access so it works over non-listable HTTP-backed stores. - url_safety.py: SSRF guard rejecting internal/loopback/link-local hosts. - controllers.py: floodmap_cog endpoint (400/422/500/502 error mapping). - hermetic unit tests for the conversion logic and the SSRF guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…parent OpenLayers masks nodata cells via `value == nodata`, and `NaN == NaN` is always false, so NaN dry cells could never be masked -- they rendered as the color ramp's minimum color instead of dropping out. Emit a concrete -9999 nodata value instead: OpenLayers reads it from the COG metadata, adds the alpha band, and the GeoTIFF layer's nodata style renders dry ground transparent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add /tethysdash/floodmap/meta, which returns a public Zarr floodmap store's variables, storm count, per-storm labels, CRS, extent, and grid shape -- so a storm/variable selector can be populated without downloading raster data. read_metadata now derives human-readable storm labels from a `magnitude_mm` array when present (e.g. "47 mm"), falling back to the storm index. Same SSRF guard and 400/422/502 error mapping as the COG endpoint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a map GeoTIFF layer's source URL changed (e.g. a variable-driven floodmap storm selector), the reconciliation kept the old layer visible until the new layer's tiles loaded, leaving the previous raster on screen; rapid changes also raced overlapping reconciliations. Hide the outgoing GeoTIFF layer immediately and guard against superseded runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A slider's initial value is populated by its own debounced preview, but the preview only mounted once every field including initial_value was filled - a deadlock that left the value null and blocked save with a false "Initial value must be selected" error (then crashed on null derefs). Exempt a slider's empty initial value from the preview and save gates, and null-safe the arg builders and saved source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This reverts commit 0ab6bc7.
A Number slider left with Output Format blank emitted null metadata, so
"All arguments must be filled out" blocked the save. Default it to {{n}}
when Number is selected so it isn't required.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The map layer's COG reader fetches byte ranges, but the endpoint returned the whole file (200) for Range requests, corrupting geotiff.js's block reads and failing the layer. Serve 206 partial content, cache the COG per storm (in-memory, no disk), and retry the S3 read on transient failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New GeoTIFF layer sources pre-fill the Nodata field with -9999 (the value our floodmap COGs use); existing saved layers are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New GeoTIFF layers default to the turbo ramp, and the picker shows only color swatches. The ramp range now auto-fits the source: the editor pre-fills Min/Max from the COG's embedded band statistics (read via geotiff.js, http(s) only), and clearing them opts into per-storm auto scaling where OpenLayers normalizes band 1 to [0,1] from those stats. The floodmap backend embeds per-storm min/max as STATISTICS_MINIMUM/ MAXIMUM so the range fits each storm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Animated slider playback flashed white each time a GeoTIFF layer's source URL changed: the replacement layer was added visible and cleared to white before its tiles painted. Load the replacement hidden via opacity 0 (which still renders, so its tiles load), fade it in over 250ms once painted, then drop the old layer. Superseded frames are dropped so fast playback skips instead of flashing or stalling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The zarr->COG conversion is not flood-specific, so rename floodmap.py to zarr_utils.py (and its tests) and make it data-agnostic: - Masking is opt-in: cells are masked only when the store declares extent_threshold_m/source_nodata or the caller passes mask_below. The old default threshold of 0.0 silently erased zeros/negatives in non-flood data. - Variable discovery lists the store, falling back to caller-supplied candidates instead of a hardcoded flood name list. - Accept 2-D [y, x] grids as well as 3-D [n, y, x] stacks. Flood specifics (depth default, storm labels, candidate names) move into the floodmap endpoints in controllers.py; the HTTP contract is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the flood-named floodmap/cog and floodmap/meta endpoints with data-agnostic tethysdash/zarr/cog and tethysdash/zarr/meta over the now-generic zarr_utils module: - zarr/cog: src, variable (required), index, optional mask_below; serves a COG with HTTP Range support. - zarr/meta: src, optional variable/candidates/label_var; returns generic slice_count/slice_labels metadata. read_cog gains an optional mask_below (added to the cache key). Adds direct-call (RequestFactory) controller tests for both endpoints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add docs/source/maps/zarr_cog.rst covering the zarr/cog and zarr/meta HTTP endpoints -- parameters, an animated-slider example, and the metadata response -- and link it into the Maps toctree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a first-class "Zarr" source to the map layer editor so authors point at a
public Zarr store instead of hand-writing the zarr/cog endpoint URL. The source
takes url + variable (required) and optional index / mask_below fields; index
accepts a ${...} variable input for slider-driven animation.
It renders as a WebGLTile COG: ModuleLoader assembles the zarr/cog URL from the
fields (zarrSourceToGeoTIFF), and the layer reuses the GeoTIFF ramp styling,
per-slice auto-scaling, crossfade, and click-to-query. The endpoint is no longer
user-facing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A generic Cloud-Optimized GeoTIFF should not assume a -9999 nodata sentinel; that default only made sense when GeoTIFF was the floodmap path. Floodmaps now use the Zarr source, which bakes -9999 in server-side, so the GeoTIFF source leaves its Nodata field blank by default. Authors can still type a value when their COG uses one. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Zarr" source entry to the Source Tab docs (fields + example) now that the zarr/cog endpoint is internal, and remove the standalone endpoint reference page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The map's auto-zoom-to-extent (and load-error surfacing) fired only for GeoTIFF sources; a Zarr layer has source type "Zarr", so it rendered without zooming to the raster's extent. Include "Zarr" in the WebGLTile auto-fit gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add geoparquet.py: read a public GeoParquet file with geopandas, reproject to EPSG:4326, and return it as a GeoJSON FeatureCollection, with retry and FileOpenError/GeoParquetError. Expose it at tethysdash/geoparquet/geojson, SSRF-guarded like the zarr endpoints, and declare geopandas/pyarrow/fsspec in install.yml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "GeoParquet" vector source (single url field) that renders via the geoparquet/geojson endpoint: ModuleLoader.geoParquetToGeoJSON assembles the endpoint URL into a GeoJSON vector source, getLayerType maps it to a VectorLayer, and it joins the client-vector query + StylePane styling lists so it can be styled (stroke/fill) and clicked like a GeoJSON layer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "GeoParquet" entry to the Source Tab docs (url field, example config, and the moderate-size caveat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
controllers.py imports zarr_utils and geoparquet, whose read/convert stack (numpy, zarr, rasterio, rio-cogeo, fsspec, aiohttp, geopandas, pyarrow) was not declared for pip. CI installs with `pip install -e .`, so declare them there and drop the redundant install.yml conda entries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run prettier --write so the branch passes the CI pretty:check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
Superseded by a client-side (frontend) GeoParquet implementation that removes the geopandas/pyarrow backend dependencies and the conversion endpoint, reading the file in-browser with hyparquet + proj4. The replacement is prepared in JakeGimenes:feature/geoparquet-upstream (stacked on the GeoPackage PR #175) and will open against main once #175 merges. Closing this backend version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a GeoParquet map source to TethysDash:
geoparquet.py, wired intocontrollers.py)geoparquetmap source type on the frontendurl_safety.py) with unit tests (test_geoparquet.py,test_url_safety.py)geopandas,pyarrow) inpyproject.tomlThis branch is built on top of the EF5 / zarr floodmap line in #166, so this PR's diff currently includes all of #166's changes in addition to the GeoParquet work. #166 is intended to merge first; once it lands on
main, this PR's diff will reduce to only the GeoParquet-specific changes.Testing
The full frontend (eslint + prettier + jest) and backend (ruff + pytest) CI suite passes on this branch's content (verified on the author's fork).
🤖 Generated with Claude Code