diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9c3c1ed..1e88ab5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -45,6 +45,13 @@ jobs: - name: Type-check (pyright) run: uv run pyright + # The other half of the static gate. No path argument and no `select`: the rule set is + # ruff's own default, configured in [tool.ruff.lint], and the scan follows .gitignore so the + # data store and the venv are out without being named. Placed after pyright because the two + # overlap least at the top — a type error is usually the more informative first failure. + - name: Lint (ruff) + run: uv run ruff check + # The venv is Python 3.14, but palette.py + the two bpy scripts must stay importable # under Blender's bundled 3.13.9. This re-checks exactly those files at 3.13 so # 3.14-only syntax can't leak into a Blender-shared module. File list lives in the @@ -88,9 +95,22 @@ jobs: - name: Install dependencies run: pnpm install --frozen-lockfile - - name: Type-check (astro check) + # TWO PROGRAMS, and `pnpm check` runs both. `astro check` excludes worker/ on purpose — + # @cloudflare/workers-types redefines fetch/Response and breaks the site's program — so the + # Worker has its own tsconfig, and for a while nothing ran it. Nothing would have noticed + # either: wrangler's esbuild strips types without checking them, so a deploy ships regardless. + - name: Type-check (astro check + the tile Worker) run: pnpm check + # The web half of the static gate, and the counterpart to `ruff` in the job above: the two + # type-checkers ask whether the types line up, oxlint asks whether the code says what it + # means. No path argument — it walks from web/ and follows .gitignore, so it reaches all 95 + # sources including the eight .astro pages and the Worker, which `astro check` excludes. + # Placed after the type-check for the same reason ruff sits after pyright, and before the + # ~100 MB browser download because it takes milliseconds. + - name: Lint (oxlint) + run: pnpm lint + # The vitest `browser` project launches a real chromium; `pnpm install` fetches the # playwright NPM package but never its browser binaries, so without this the project # cannot start and the run reports `28 passed (30)` with an unhandled launch error. diff --git a/ART.md b/ART.md index 60ac507..4c66bf9 100644 --- a/ART.md +++ b/ART.md @@ -6,7 +6,11 @@ bathymetry, data-driven snow. - The system of record is code: shared constants in `pipeline/render/palette.py`, hero-only constants in `pipeline/render/scene_build.py`, tile levers in `pipeline/tile/shade.py` - (`KNOBS`). Canonical renders: `blender/renders/heroes/.png` (203 posters — 204 countries are in scope, Kiribati deferred). + (`KNOBS`). The ramp constants below are Earth's unless a row says otherwise, and they are + assembled into a named `Look` that the shading path resolves **per body** — every lever's cost in + this file is Earth's cost. `KNOBS` is the exception that is NOT per body: tone, saturation and the + highlight ceiling apply to every planet alike, which is a shared house style rather than an + oversight, but it means a body cannot soften its own highlights without that changing first. Canonical renders: `blender/renders/heroes/.png` (203 posters — 204 countries are in scope, Kiribati deferred). - The A/Bs and rationale behind every value live in HISTORY (cited by § heading); this file is the operational view. @@ -54,14 +58,15 @@ Cost: hero sweep **~10–13 h** + tile restage **~29 min** + caps auto-restage * | Lever | Value | Section | |---|---|---| -| `LAND_STOPS` (+ `LAND_MAX_M`) | 6 stops, 0–6000 m | § Land color ramp | +| `LAND_STOPS` (+ `LAND_MAX_M`) | 6 stops; Earth's domain 0–6000 m, but the ENDS are the body's — a `Surface` carries `origin_m` and `extreme_m`, and only Earth's origin is the datum | § Land color ramp | +| `MARS_LAND_STOPS` | 6 stops over −6000…+6100 m, sharing no colour object with Earth. Hue measured against a global Viking mosaic (G/R 0.654, vs the 0.780 Earth's ramp was shipping); level taken only in part, landing at **0.71× Earth's shipped land**, because the mosaic's 2.07× darker reading is mostly its own tone curve and an uncorrected haze floor. Monotone in luminance BY CHOICE — Mars is really brightest at both ends, so fidelity would put Hellas and Olympus in one colour | § Land color ramp | | `SEA_STOPS` (+ `SEA_MIN_M`) | 6 shelf-weighted stops, 0…−6000 m | § Sea color ramp | | `LAKE_STOPS` (+ `LAKE_MAX_M`) | 3 stops; stop 0 == `WATER_RGB`, far end 1642 m (Baikal) | § Inland water | | `WATER_RGB` | `8EC6C4` — pinned relationally: sea surface +7% | § Inland water | | `SNOW_RGB` / `SNOW_SHADOW_RGB` | `E8F1F6` / `B0C7DB` | § Snow | | `ICE_RGB` / `ICE_SHADOW_RGB` | `D4E4F0` / `9CB8D2` | § Sea ice | | `SUN_ALT_DEG` | 45.0 — hero X-tilt and tile `alt` both derive from it | § Sun altitude | -| `EXAGGERATION` | 15.0 — hero displacement + tile `EXAG` both import it (the copy pair was collapsed) | § Vertical exaggeration | +| `EXAGGERATION` | 15.0 — the hero's and the region preview's; the tiles read `Body.exaggeration`, pinned equal to it for Earth | § Vertical exaggeration | | `LUT_STEP_M` | 1.0 m — ramp LUT resolution (fidelity, not hue) | § Land color ramp | ### Hero only (`scene_build.py`, `render_prep.py`) — the 203-country sweep, ~10–13 h @@ -87,7 +92,7 @@ Cost: hero sweep **~10–13 h** + tile restage **~29 min** + caps auto-restage * |---|---|---|---| | `fill_strength` | 0.15 | hillshade ~46 min | § Fill sun — TILES | | `shadow_strength` / `shadow_reach` | 0.0 (off; +~2.1 h when on) / 300 px | hillshade | § Fill sun — TILES | -| tile `EXAG` (shade_planet.py) | 15 — imports `palette.EXAGGERATION` (shared, no longer a copy pair) | hillshade | § Vertical exaggeration | +| `exaggeration` (bodies.py) | 15 on Earth — a per-body field; tiles AND caps read it, pinned equal to `palette.EXAGGERATION` | hillshade | § Vertical exaggeration | | `ambient` / `ambient_knee` | 0.50 / 0.30 | composite ~29 min | § Ambient floor | | `hi` / `exposure` / `saturation` / `warmth` | 1.12 / 1.05 / 1.18 / 0.06 | composite | § Ambient floor | | `shadow_warmth` | 0.55 | composite | § Hero → tile parameter map | @@ -117,12 +122,12 @@ Cost: hero sweep **~10–13 h** + tile restage **~29 min** + caps auto-restage * | `SIMPLIFY_DEG` (countries_geojson.py) | 0.002° — guard-tested sub-pixel vs `Z8_RES` | regen, seconds | | `COORDINATE_PRECISION` (both geojson emitters) | 4 (countries) / 5 (borders) | regen, seconds | | highlight look (web/src/lib/countryHighlight.ts) | `HIGHLIGHT_GOLD` `#eca834`, casing `#1c140c`, wash 0.16 | `pnpm build` | § Borders | -| border line ramps (earth.astro `inkWidth`/`casingWidth`) | 0.6→1.7 px ink, 2.2→5.0 px casing over z1–8 | `pnpm build` | § Borders | -| hover name chip (earth.astro `.country-chip`) | top-centre pill on `.globe-home`'s treatment, `var(--serif)` 1.05 rem, 0.4 rem dot taking `HIGHLIGHT_GOLD` from the constant (never a CSS copy); no fade, hidden under `(hover: none)` and while the detail card is open | `pnpm build` | -| globe camera flatness (earth.astro `VERTICAL_FIELD_OF_VIEW_DEG`) | 15° — tested band 5–15 (MapLibre default 36.87° reads as low-orbit fisheye; 5° ≈ the hero's orthographic camera) | `pnpm build` | -| on-map credit + chrome placement (earth.astro `CREDITS`, global.css `.view-bar`) | credit is an **ⓘ** linking to /about, folded into the centred `.view-bar` as its own segment — the bottom edge carries ONE floating element, not three. Still a MapLibre `AttributionControl`, constructed **`compact: false`**: any source declaring `attribution` is still picked up automatically (joined with `\|`), while MapLibre's own ⓘ — a baked-in black SVG that CSS cannot recolour — is never shown at any width. Ours is the same glyph redrawn with `currentColor`, its **viewBox cropped to the ink** (`4 4 12 12`, not `0 0 20 20`): the untrimmed box is 40% margin and renders ~60% the size of the octicon beside it. `title` + `aria-label` are mandatory — an icon-only link has no accessible name without them | `pnpm build` | +| border line ramps (Globe.astro `inkWidth`/`casingWidth`) | 0.6→1.7 px ink, 2.2→5.0 px casing over z1–8 | `pnpm build` | § Borders | +| hover name chip (Globe.astro `.country-chip`) | top-centre pill on `.globe-home`'s treatment, `var(--serif)` 1.05 rem, 0.4 rem dot taking `HIGHLIGHT_GOLD` from the constant (never a CSS copy); no fade, hidden under `(hover: none)` and while the detail card is open | `pnpm build` | +| globe camera flatness (Globe.astro `VERTICAL_FIELD_OF_VIEW_DEG`) | 15° — tested band 5–15 (MapLibre default 36.87° reads as low-orbit fisheye; 5° ≈ the hero's orthographic camera) | `pnpm build` | +| on-map credit + chrome placement (Globe.astro `CREDITS`, global.css `.view-bar`) | credit is an **ⓘ** linking to /about, folded into the centred `.view-bar` as its own segment — the bottom edge carries ONE floating element, not three. Still a MapLibre `AttributionControl`, constructed **`compact: false`**: any source declaring `attribution` is still picked up automatically (joined with `\|`), while MapLibre's own ⓘ — a baked-in black SVG that CSS cannot recolour — is never shown at any width. Ours is the same glyph redrawn with `currentColor`, its **viewBox cropped to the ink** (`4 4 12 12`, not `0 0 20 20`): the untrimmed box is 40% margin and renders ~60% the size of the octicon beside it. `title` + `aria-label` are mandatory — an icon-only link has no accessible name without them | `pnpm build` | | view bar geometry + phone collapse (global.css `.view-bar`, Base.astro) | centres by `inset-inline: 0` + auto margins, **never** `left: 50%` + `translateX` — the transform applies after layout, so that form caps the bar's width at half the viewport and wraps it early. Below **600 px** it collapses to `[tune \| ⓘ]`, persisted, defaulting closed; the credit sits OUTSIDE `.view-bar-items` so a collapse can never hide it. `is-collapsible` must track the trigger's render condition exactly, or pages without a trigger lose their controls. An expanded bar reaches **141 px at 320 px**, so the scale bar drops via `body.viewbar-open` rather than the lift chasing it. MapLibre's CSS is injected at runtime, so overrides must out-specify, not just follow | `pnpm build` | -| globe atmosphere (web/src/lib/skyAtmosphere.ts) | `atmosphere-blend` **0.7 at z≤3 → 0.15 at z≥6**, geometric, as a zoom expression MapLibre evaluates per frame. Not a halo: the same uniform lays aerial perspective over the ground, and the overview-tuned 0.7 clips 23.8% of a pitched z7 frame to ≥254. `?sky=off\|` sweeps it | `pnpm build` | +| globe atmosphere (web/src/lib/skyAtmosphere.ts) | `atmosphere-blend` **0.7 at z≤3 → 0.15 at z≥6**, geometric, as a zoom expression MapLibre evaluates per frame. Not a halo: the same uniform lays aerial perspective over the ground, and the overview-tuned 0.7 clips 23.8% of a pitched z7 frame to ≥254. `?sky=off\|` sweeps it. The three COLOURS are the body's (`lib/bodies.ts`) and a body may declare none, in which case no sky is set at all; the numbers here are the ramp's | `pnpm build` | | terrain displacement (web/src/lib/terrainSource.ts) | **15× at z≤3 decaying geometrically to 2.5× at z8**, written to `terrain.exaggeration` per frame — **never** via `setTerrain`, which rebuilds the Terrain + RenderToTexture and only destroys on removal. Rides the **`full` tier**; `?terrain=N` forces it on at any tier and `?terrain=off` removes only the geometry. The floor is **derived, not preferred**: elevation change goes as ~`width^0.5` while the facet halves per level, so slope holds constant only at 0.707 per level. Declared **`tileSize: 128` over a z0–8 pyramid** (one decision — 512's DEM sits at `camera − 2` and could never reach z7), and **`terrainSkirtLength: "none"`**, which trades MapLibre's seam-tearing for tiny black specks that land only on drastic elevation change | `pnpm build` | | terrain polar feather (pipeline/tile/terrain_rgb.py `FEATHER_LAT_LO/HI`) | encoded elevation ramps to zero over **78°–85°** by smoothstep, the geometric twin of the alpha feather `polarCaps.ts` already applies across the same band. Not optional: MapLibre does **not** drape custom layers onto the terrain mesh, so displaced tiles under an undisplaced cap would open a geometric seam — worst in the south, where that band is 2–3 km of Antarctic ice. Smoothstep rather than linear because this multiplies *geometry*, and a slope discontinuity is a visible crease where an alpha one is not | terrain re-cut, ~41 min | @@ -130,11 +135,12 @@ Cost: hero sweep **~10–13 h** + tile restage **~29 min** + caps auto-restage * ### Vertical exaggeration — global constant, per-country number -- Baseline **15× globally** (`EXAGGERATION`, `palette.py` — imported by `render_prep` + `shade_planet`). The number in the scene is +- Baseline **15× globally** (`EXAGGERATION`, `palette.py` — imported by `render_prep`; the tiles and caps read `Body.exaggeration`). The number in the scene is per-frame: `displacement_scale = 15 ÷ (extent_w_m / 2)` because the plane is always 2 units wide (India 8.0e-6, Nepal 3.3e-5, Sri Lanka 1.0e-4 — docs/framing-math.md). Copying one country's scale onto another multiplies exaggeration by the frame-width ratio. -- **Adjust globally:** `EXAGGERATION` in `palette.py`, then regenerate frame.json per country +- **Adjust globally:** `EXAGGERATION` in `palette.py` AND `exaggeration` on the body in `bodies.py` — a + test fails if you move one alone — then regenerate frame.json per country (delete + rerun; hand-edit pinned ones). **Adjust one country:** edit `displacement_scale` in its frame.json — legitimate only as a recorded pathology override; per-country drama re-litigates the series promise (same border, two posters, same mountain height). @@ -221,7 +227,7 @@ Cost: hero sweep **~10–13 h** + tile restage **~29 min** + caps auto-restage * | Tile | Hero | Relationship | |---|---|---| -| `EXAG` (shade_planet.py) | `render_prep` displacement (both import `palette.EXAGGERATION` 15.0) | **now import-shared** — the copy pair was collapsed into `palette` | +| `exaggeration` (bodies.py) | `render_prep` displacement (imports `palette.EXAGGERATION` 15.0) | **pinned, not shared** — the tiles must vary per body, so a test holds Earth's field equal to the hero's constant | | `KNOBS["alt"]` 45°, azimuth 315° | `SUN_ROTATION` X = `90 − SUN_ALT_DEG`, NW | both derive from `palette.SUN_ALT_DEG` | | `KNOBS["fill_strength"]` 0.15 | `FILL_STRENGTH 0.45 / SUN_STRENGTH 3.0` | the ratio, ported exactly | | `FILL_ALTITUDE 60°` / `FILL_AZIMUTH 135°` | `FILL_ROTATION` | identical geometry | diff --git a/ATTRIBUTIONS.md b/ATTRIBUTIONS.md index 7de07cd..bd2c36d 100644 --- a/ATTRIBUTIONS.md +++ b/ATTRIBUTIONS.md @@ -5,10 +5,14 @@ Terrella is built entirely from open data and open-source tools. This file is th ## Terrella's own outputs - **Code** (this repository): MIT — see `LICENSE`. -- **Rendered imagery** (hero renders, map tiles, polar caps — everything the pipeline draws): **CC BY-NC 4.0**. Free to share and adapt for non-commercial use — education, entertainment, personal projects — with attribution to "Terrella (Rohan Bansal)". Commercial use is reserved; separate commercial grants can be issued case-by-case, and individual images can additionally be released under a free-culture license (e.g. CC BY-SA for Wikimedia use) at the author's discretion — multi-licensing is explicitly permitted. Chosen deliberately over plain CC BY: the known trade-off is that Wikimedia projects reject NC-licensed media. +- **Rendered imagery** (hero renders, map tiles, polar caps — everything the pipeline draws, for every body): **CC BY-SA 4.0**. Free to share and adapt for any purpose, commercial use included, with attribution to "Terrella (Rohan Bansal)" and on the condition that adaptations carry the same license. **One license covers both planets rather than one per body.** The Mars blend's publisher labels part of its input share-alike (below), and share-alike is the only output license that complies under both readings of that label; extending it to Earth is a choice rather than an obligation, and it buys a single sentence that is true of every image on the site. The accepted trade is that commercial use is no longer reserved, and that Creative Commons licenses are irrevocable — nothing already published under this can be narrowed later. What it buys back is Wikimedia, which rejects NC-licensed media and accepts BY-SA. ## Data sources +Grouped by body, because the two planets are built from different data under different terms and a merged table invites the reader to assume one set of obligations covers both. The About page carries the same split for the same reason. + +### Earth + | Dataset | Role in the pipeline | License | |---|---|---| | [Copernicus DEM GLO-30](https://dataspace.copernicus.eu/explore-data/data-collections/copernicus-contributing-missions/collections-description/COP-DEM) | land elevation (the heightfield) | Copernicus WorldDEM-30 licence — free & open incl. derived-product distribution; required notices below ([licence text](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM/resources/license/License-COPDEM-30.pdf)) | @@ -20,6 +24,39 @@ Terrella is built entirely from open data and open-source tools. This file is th | [GLOBathy](https://doi.org/10.1038/s41597-022-01132-9) | tile lake depth: modelled bathymetry that shades lakes instead of leaving them flat plates | **CC0** (public domain dedication) | | [Natural Earth](https://www.naturalearthdata.com) | borders, coastlines, country bounding boxes | Public domain | +### Mars + +| Dataset | Role in the pipeline | License | +|---|---|---| +| [Mars MGS MOLA – MEX HRSC Blended DEM Global 200m v2](https://astrogeology.usgs.gov/search/map/mars_mgs_mola_mex_hrsc_blended_dem_global_200m) | the entire heightfield — Mars enters the pipeline already fused, so there is no acquire or fuse tier | Publisher states: **access constraints "MOLA (CC0) and HRSC (CC BY-SA 3.0 IGO)"**, use constraints "Please cite authors" | + +| [Geologic Map of Mars (SIM 3292)](https://pubs.usgs.gov/sim/3292/) | where the permanent polar ice is — the mapped units the ice layer is drawn from | Read at the product: `Access_Constraints: None`, `Use_Constraints: please cite authors.` The one Mars source with a stated obligation, and it is an exact-string one | +| [Viking Global Color Mosaic 925m](https://astrogeology.usgs.gov/search/map/mars_viking_colorized_global_mosaic_925m) | how bright each part of the ice is, which grades the white; and the hue the land ramp is measured against | Read at the product's own constraint fields: `Access Constraints: public domain`, `Use Constraints: None` — credited by courtesy, since no constraint requires it | +| [IAU/USGS Gazetteer of Planetary Nomenclature — Mars geometries](https://planetarynames.wr.usgs.gov/) | every named feature the globe draws and a visitor can point at — 1,717 areal and 203 linear | Read at the product, in the archive's own FGDC record: `useconst: Public domain.`, `distliab: none`. `download_nomenclature.assert_licence` re-reads that field on every acquisition, so a republished archive with different terms stops the pipeline rather than flowing into a CC BY-SA 4.0 site unnoticed | + +Mars has four sources and no others are used. There is no bathymetry, no glacier, no sea-ice and no boundary dataset, because none of those exist as a Mars product we ship. The polar white is the one surface layer Mars declares: it is not a snow product but a mapped permanent cap, graded by brightness; the gazetteer is the only vector source, and it names features rather than dividing territory. + +#### The share-alike finding, which contradicts an earlier reading and is answered by taking the strict one + +**The USGS product page for the blend states its access constraints as "MOLA (CC0) and HRSC (CC BY-SA 3.0 IGO)".** That was read verbatim off the publisher's own metadata for the exact file this project downloads — the page's "Online File Link" is `Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif`, the 11 GB mosaic in `data/raw/mars/`. + +It mattered because the rendered imagery used to be CC BY-NC 4.0, and **share-alike input cannot flow into a non-commercial output** — the two conditions are incompatible in that direction, and the permitted direction is the other one. That incompatibility is what moved the output license to BY-SA, above. + +The legal question is genuinely open and both readings are recorded so neither gets lost. **The response no longer depends on which is right**, which is the point of settling it this way: + +- **It may be descriptive rather than a grant.** In the FGDC metadata vocabulary the page uses, *Access* constraints and *Use* constraints are separate fields, and USGS put the licences under *Access* while its *Use* constraint says only "Please cite authors". Read that way, the line names the provenance of the two inputs and the blend itself — a US Geological Survey work — carries no share-alike. +- **It may be a real term on the HRSC half.** HRSC contributes 44% of the blend's coverage; if that fraction is BY-SA, the derived tiles are a derivative of BY-SA material regardless of which metadata field records it. + +**This supersedes MARS.md's earlier conclusion** that the share-alike trap applies only to ESA's published *pictures* and that the archive route imposes nothing. That conclusion was reached from ESA's own terms and is correct about ESA; what it did not account for is the USGS blend's own metadata making the same claim about the data. + +**The resolution is to assume the strict reading and license the output share-alike**, which is correct whichever reading is right, and which needs no legal determination to act on. Three consequences worth stating rather than re-deriving: + +- **The output is CC BY-SA 4.0, not 3.0 IGO**, and that is permitted by the input license's own text rather than by analogy: BY-SA 3.0 IGO § 4(b) allows an Adaptation to be distributed under *"a later version of this License with the same License Elements as this License"*. Verified against the legal code, not recalled. +- **Its § 4(c) wants "a credit identifying the use of the Work in the Adaptation"**, which the Fergason citation below already carries. Attribution was never the hard half; share-alike was. +- **The alternative was to change the source rather than the license, and it was declined on the merits.** MOLA MEGDR is CC0 at 463 m/px against a z6 cut of 651 m/px, so swapping it would remove the question outright at no visible cost today — but only today. The trade bites from z7 (326 m/px) onward, where MEGDR is upsampled and the blend is not, and at z8 (163 m/px) it is a 2.8x upsample. The swap therefore forecloses the finer cut, and the blend's detail is wanted more than the permissive license. + +The About page continues to state the publisher's own words for the source (`MOLA CC0 · HRSC CC BY-SA 3.0 IGO`) rather than our reading of them — quoting cannot be wrong where a paraphrase would assert something the source contradicts. + ### Required / requested attribution strings - **Copernicus DEM GLO-30** — for adapted/modified data (our case — Article 6(b) of the licence), the exact required notice is: *"produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved"*. The About page (or its legal notice) must also carry the Article 6(c) liability sentence: *"The organisations in charge of the Copernicus programme by law or by delegation do not incur any liability for any use of the Copernicus WorldDEM-30"* — and must not imply official endorsement by ESA/Copernicus (6(d)). @@ -30,10 +67,12 @@ Terrella is built entirely from open data and open-source tools. This file is th - **GLOBathy** — "Lake depth from GLOBathy: Khazaei, B., Read, L.K., Casali, M., Sampson, K.M., Yates, D.N. (2022), *GLOBathy, the Global Lakes Bathymetry Dataset*, Scientific Data 9, 36 (doi:10.1038/s41597-022-01132-9)." **CC0**, so no attribution is legally required — this is an academic-citation courtesy, and we chose it deliberately: the tint-only architecture avoided a HydroLAKES join, which is what kept the whole depth layer CC0. The About page must also carry the *epistemics* (the depth shape is a modelled cone, surveyed scale for only 647 of 83,357 lakes) — that is an honesty obligation, not a licensing one. - **OSI SAF** — "Sea-ice climatology derived from the OSI SAF Global sea ice concentration climate data record 1978–2020 (v3.0, 2022), OSI-450-a, EUMETSAT Ocean and Sea Ice Satellite Application Facility (doi:10.15770/EUM_SAF_OSI_0013)." CC-BY 4.0. The About page should note the reference period actually used (1991–2020 annual ice frequency). - **Natural Earth** — public domain; a courtesy credit to naturalearthdata.com. +- **SIM 3292 (Geologic Map of Mars)** — the citation the product's own metadata asks for, quoted from its FGDC `Data_Set_Credit` field rather than composed by us: *"K.L. Tanaka, J.A. Skinner, Jr., J.M. Dohm, R.P. Irwin, III, E.J. Kolb, C.M. Fortezzo, Thomas Platz, G.G. Michael, and T.M. Hare, 2014, Geologic Map of Mars, Scale 1:20,000,000, U.S. Geological Survey Scientific Investigations Map SIM 3292, http://pubs.usgs.gov/sim/3292"*. Read at the product: `Access_Constraints: None`, `Use_Constraints: please cite authors.`, `Distribution_Liability: None`, `Fees: None` — attribution-only, which flows into CC BY-SA 4.0 without an argument. **Carried on the About page and swept by `REQUIRED_STRINGS`**, since this map now decides where Mars's polar white is drawn — the obligation attaches to what is published, and it moved the moment the map first painted a pixel. +- **MOLA / HRSC blend** — the publisher's own recommended citation, quoted from its product page: *"Fergason, R. L, Hare, T. M., & Laura, J. (2018). HRSC and MOLA Blended Digital Elevation Model at 200m v2. Astrogeology PDS Annex, U.S. Geological Survey."* Its stated use constraint is "Please cite authors", so unlike the public-domain Earth sources this citation is **requested by the publisher rather than a pure courtesy** — carry it. It does double duty: on the strict reading of the share-alike finding above, this is also the credit BY-SA 3.0 IGO § 4(c) asks an adaptation to carry. ### Licensing posture -GEBCO (public domain), Natural Earth (public domain), NSIDC-0791 / MODIS (NASA, public domain), ESA WorldCover (CC-BY 4.0), RGI 7.0 (CC-BY 4.0), and OSI SAF OSI-450-a (CC-BY 4.0) all unambiguously permit derivative works, public display, and redistribution with attribution. Copernicus GLO-30 is the only source under a bespoke licence rather than PD/CC, and its terms have been **verified against the primary licence text** ([License-COPDEM-30.pdf](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM/resources/license/License-COPDEM-30.pdf)): Article 4 grants **reproduction, distribution, communication to the public, and adaptation/combination**, worldwide, without time limit or purpose restriction — commercial use of derived products included; Article 9 confirms the IPR in work produced *using* the DEM (our renders) belongs to us. The obligations are the two exact notices and the liability sentence above, plus no implied endorsement. The higher-resolution WorldDEM-10 is expressly outside this licence — this project uses only GLO-30. +GEBCO (public domain), Natural Earth (public domain), NSIDC-0791 / MODIS (NASA, public domain), ESA WorldCover (CC-BY 4.0), RGI 7.0 (CC-BY 4.0), and OSI SAF OSI-450-a (CC-BY 4.0) all unambiguously permit derivative works, public display, and redistribution with attribution. Copernicus GLO-30 is the only source under a bespoke licence rather than PD/CC, and its terms have been **verified against the primary licence text** ([License-COPDEM-30.pdf](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM/resources/license/License-COPDEM-30.pdf)): Article 4 grants **reproduction, distribution, communication to the public, and adaptation/combination**, worldwide, without time limit or purpose restriction — commercial use of derived products included; Article 9 confirms the IPR in work produced *using* the DEM (our renders) belongs to us. **Those two articles are what make a share-alike output license possible on Earth at all**, and the check is not optional: BY-SA promises downstream recipients the right to redistribute and adapt commercially, and a promise like that can only be made where the inputs already permit it. Every other Earth source is PD, CC0 or CC-BY, all of which flow into BY-SA. The obligations are the two exact notices and the liability sentence above, plus no implied endorsement — and they now travel further than they used to, because BY-SA invites the redistribution that NC discouraged. The higher-resolution WorldDEM-10 is expressly outside this licence — this project uses only GLO-30. ## Tools & technique diff --git a/CLAUDE.md b/CLAUDE.md index 0d5e44a..69c887c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,7 @@ # Terrella — project memory -A static site of ray-traced relief maps of every country, navigable as an interactive globe. +A static site of ray-traced relief maps, navigable as an interactive globe — every country on Earth, +and Mars as a second body. Look target: Frank Ramspott's "3D Render Topographic Map — Neutral" — soft raytraced shadows, heavy vertical exaggeration, warm sand land, desaturated teal sea with bathymetry, white vector borders, minimal typography. The aesthetic decisions live in ART.md. @@ -43,12 +44,13 @@ probe; degrade at runtime if frame rate tanks; honour `Save-Data`, `prefers-redu - **Heroes:** headless Blender Cycles (bpy), RTX 4070 Super, OptiX backend + OpenImageDenoise — but **CPU denoise for 8K**, or render and denoise contend for the 12 GB VRAM and the driver throws an Xid 31 MMU fault. - One scene rig for every country: DEM displacement, low sun, two-ramp material (elevation-keyed land, depth-keyed sea), ortho camera framed from Natural Earth bounds. -- **Vertical exaggeration 15×**, locked and shared by import (`palette.EXAGGERATION`) so hero and tile cannot drift. +- **Vertical exaggeration belongs to the body** — 15× on Earth. The hero imports `palette.EXAGGERATION`, the tiles and caps read `Body.exaggeration`, and a test pins Earth's field equal to the constant; unpinned, the tiles drift away from the heroes they must match. - **Tiles approximate the Cycles look:** single-NW hillshade (multidirectional rejected) + sky-view factor from our own `sky_view.py` (WhiteboxTools dropped) + the same ramps, composited with GDAL. -- **z0–8, and z8 is LOCKED.** z9/z10 are parked in FUTURE and blocked on disk — a planet re-fuse at ~2.5″, never a tiling flag. +- **Tile depth belongs to the body too** — `Body.tile_max_zoom`, z8 on Earth and z7 on Mars. **Earth's z8 is LOCKED**: z9/z10 are parked in FUTURE and blocked on disk — a planet re-fuse at ~2.5″, never a tiling flag. - **Tiles are 512px**, declared to MapLibre as `tileSize: 256`, which centres the scheme on DPR 2. → FUTURE § raster tile resolution vs device pixel ratio - **Delivery encoding is a policy, not one constant** — masters stay lossless, delivery does not. → ART § Delivery encoding · § The srcset ladder - **Every writer records its recipe beside its output**, because existence cannot see a settings change. +- **A producer declares what it emitted; no consumer infers it from what is on disk.** A missing raster cannot distinguish "this body has none" from "the producer crashed", and an absent path scores nothing in an mtime comparison — so switching an input off leaves the output that used it looking fresh. `pipeline/planet_seam.py`. - Baked NW-ish lighting globally (cartographic convention); no per-region sun position. ## Serving & deployment @@ -65,21 +67,24 @@ probe; degrade at runtime if frame rate tanks; honour `Save-Data`, `prefers-redu - Pipeline Python is the uv-managed venv (`source .venv/bin/activate`); `uv sync` rebuilds it exactly, upgrades only via `uv lock --upgrade`. Blender's bundled Python is a **separate interpreter** — bpy scripts cannot import the venv's packages. - Dev/render box: dual-boot desktop, RTX 4070 Super, 12 GB VRAM. **All work happens in the Ubuntu boot** — never suggest Windows paths, WSL, or PowerShell. - **OptiX crash recipe:** `OPTIX_ERROR_UNKNOWN` at context creation → check `journalctl -k` for NVRM **Xid** lines; if the Xid's pid is Blender the driver is fine, just restart Blender to clear the dead CUDA context. -- **One heavy job at a time under a 12 G cgroup cap, no third-party exemptions.** Keep project data and temp on ext4 — never tmpfs `/tmp`, never large rasters on NTFS. +- **One heavy job at a time under a 16 G cgroup cap, no exemptions** — not for third-party tools, and not for an ad-hoc measurement either; the category that matters is "touches a full-planet raster", not "is a pipeline stage". `run_pass.sh` sizes the cap per body from `pipeline/profile/pass_cap.py`. 12 G is the value that failed: the shade pass invokes `cap_render` in its own cgroup, so a 12 G pass completed every tile stage and died at the last one. Keep project data and temp on ext4 — never tmpfs `/tmp`, never large rasters on NTFS. - A separate home server runs the pipeline; it is not the site's origin. The site is served entirely from the CDN. -- Budget ~8–10 GB of DEM per large country and tens of GB for the full pyramid; keep intermediates out of backups. +- **Keep intermediates out of backups.** INVENTORY.md is the storage map — current sizes, what each store feeds, and which are reclaimable. ## Working conventions - Pipeline stages are **idempotent and resumable** — a crash at tile N must not restart the world. Cache intermediates, validate per stage. - Python for pipeline code; boring debuggable scripts over frameworks. (Upheld on measurement, not taste: numpy releases the GIL, so threads reach the same ceiling xarray/dask would.) -- **`uv run pyright` stays at 0 and `pytest` stays green** — there is no "pre-existing error" allowance. rasterio call sites take a targeted `# pyright: ignore[reportCallIssue]`; GDAL creation-option dicts are `dict[str, Any]`. +- **Every gate stays at zero and there is no "pre-existing error" allowance.** From the root: `uv run pytest`, `uv run pyright`, `uv run ruff check` — pyright asks whether the types line up, ruff whether the code says what it means, and neither substitutes for the other. From `web/`: `pnpm test`, `pnpm lint`, `pnpm check` (which is `astro check` plus the worker's own tsconfig, since a second tsconfig is a second program the project check cannot see) and `pnpm check:test-collection`. rasterio call sites take a targeted `# pyright: ignore[reportCallIssue]`; GDAL creation-option dicts are `dict[str, Any]`. - **Docs in this repo state current truth, not history** — if a row and reality disagree, the row is the bug. Dated decisions live in a decision archive kept outside the repo. -- **A learning goes where it will be met:** a fact about one function into that function's docstring, a general work heuristic into the agent's memory. One claim, one home; if it must appear twice, make one copy executable so drift fails loudly. +- **A learning goes where it will be met:** a fact about one function into that function's docstring, a general work heuristic into the agent's memory. +- **A second reader with no owner is the defect, and the KIND of thing is incidental** — a path, a procedure, a constant, a header, an explanation in a comment. With no home to import from, the second module copies, and *every copy is correct where it sits*: Natural Earth reached eight spellings of one path, all resolving identically on this machine, before `pipeline/naturalearth.py`; `download_one` reached two, and the copy drifted a timeout and a 404 branch, each exercised only by its own callers. **The trigger is "change one copy — what goes red?"** Nothing red means it needs an owner; where one owner is impossible — a latitude that must exist in Python and in TypeScript — make one copy executable so the drift fails loudly instead. +- **For an EXPLANATION that trigger is inert — nothing ever goes red — so ask "if this concept changed, how many places would I edit?"** `bodies.py` re-established *why there are three radii* at each radius field and again in its module docstring, and every copy passed the comment rule below on its own: a per-comment test cannot see a cross-comment property, which is why the bloat was invisible from inside and obvious to a reader. A concept gets one block; the sites that need it get one line pointing at it. Measure with `scripts/prose_report.py` rather than by eye — but as an instrument you run, never a gate, since a dense constants file is legitimately dense. +- **A comment explains its own subject and makes no claim that can rot behind its back.** The test: *could this sentence go false without anyone touching this function?* Counts, measurements taken elsewhere and system-wide properties all answer yes — they keep reading as specification long after they stop being true, which is how `composite_deps`' "eight look constants" and a test docstring's "53.8 min composite" each sent a decision the wrong way. Those belong in a test, in PROCESS.md, or nowhere; what stays is the concept, the context needed to read the code, and the anti-redo guard. - **A superseded path is deleted the same day**, or moved out of the production package — prose calling it "retired" does not disarm a runnable entry point. Exception: under gitignored `data/`, where deletion is permanent. - Never commit rendered assets or DEM data — code and config only. - Plan first (Plan Mode) before any multi-file or architectural task. -- The other docs, so facts are looked up rather than re-guessed: **PROCESS.md** measured runtimes (the authority — read it before estimating), **INVENTORY.md** the storage map, **ART.md** the aesthetic decisions, **FUTURE.md** the v2 parking lot (check it before designing a "new" feature), **docs/*.mmd** the pipeline diagrams. +- The other docs, so facts are looked up rather than re-guessed: **PROCESS.md** measured runtimes (the authority — read it before estimating), **INVENTORY.md** the storage map, **ART.md** the aesthetic decisions, **FUTURE.md** the v2 parking lot (check it before designing a "new" feature), **MARS.md** the standing brief for the second body (read it before touching a body seam), **docs/*.mmd** the pipeline diagrams. ## Skills context diff --git a/FUTURE.md b/FUTURE.md index 6263da9..46e610c 100644 --- a/FUTURE.md +++ b/FUTURE.md @@ -63,14 +63,24 @@ grep HISTORY before re-arguing anything an entry says was already decided. page full-screen, where q95 is right. - **Compute is not the obstacle, storage might be.** Measured: hero variants **6 min** at `--jobs 8`, spotlight **1m45s**, borders **7m21s** (and a border rung costs a full regeneration). But a - portrait country gets TALLER files for the same width, so the 3.5 GB served store could grow into - 870 MB of R2 headroom — that has to be measured before committing, not estimated. + portrait country gets TALLER files for the same width, so the served store grows — and **there is + no R2 headroom left to grow into**; the free tier is spent, so any growth is overage from the + first byte. It rounds up to whole GB-months at $0.015, which makes this cheap rather than free: + measure the growth before committing, and price it, rather than treating storage as headroom. - **It would close the border gap too**, which is real and currently exempted in the ladder guard: `gen_borders` stops at 1920, so a portrait border jumps to native — a lossless PNG at ~3× the width the panel draws. Off the cold path only because the layer is hidden until Borders is on. -## No test ever drives a real map, and the scale ruler showed what that costs (analysed 2026-08-02) +## No test ever drives a real map, and the scale ruler showed what that costs (analysed 2026-08-02, **SHIPPED 2026-08-03**, one shape still open) +- **What shipped:** `testing/mountGlobe.ts` mounts a real MapLibre globe in the browser project, and + the two named shapes are asserted against it — the ruler takes a **distinct** label at every zoom + (`scaleRuler.browser.test.ts`), and the hover chip **follows the camera under a parked pointer** + (`hoverTracking.browser.test.ts`). The scale-linked tier readout is the one shape not yet covered. +- **The fixture question was answered the other way.** This entry framed it as "stub the tile route + or point at the dev server"; the fixture mounts **no sources at all**, because every assertion it + exists for is camera-derived and the camera is fully real with zero tiles loaded. Wiring it to the + asset stores would have coupled the suite to a multi-gigabyte archive being on the machine. - **State at analysis:** every frontend guard is a unit test over a pure function, a source-text assertion over `earth.astro?raw`, or a canary over the shipped bundle. **Nothing instantiates a MapLibre map and checks what it does.** Grepped, not assumed. @@ -92,8 +102,8 @@ grep HISTORY before re-arguing anything an entry says was already decided. here is much smaller and sharper — **assert that outputs which must track the camera actually track it.** The ruler is one; the hovered-country chip and the scale-linked tier readout are the same shape. -- **Adjacent:** `forced-colors` below also needs Playwright. One browser-fixture pass would carry - both. +- **Adjacent:** `forced-colors` below also needs Playwright, and the toolchain it was waiting on is + now here — that entry is no longer blocked on infrastructure, only unwritten. ## `forced-colors` is unhandled, and the rail's icons are the thing it breaks (analysed 2026-08-02) @@ -114,6 +124,57 @@ grep HISTORY before re-arguing anything an entry says was already decided. - **Adjacent, same sweep:** the tier picker's `radiogroup` a11y defect is already parked here. If either is ever picked up, do both — one accessibility pass, one round of judgement. +## A cold page load at high zoom paints a flat fill and never recovers (observed 2026-08-11, not analysed) + +- **Reproduced three times while checking the antimeridian**: loading `/mars/?…#map=8/-20/180/0/0` + from scratch leaves the whole viewport one flat sand colour. The UI, the scale bar and the tier + pill all render, so the page is alive; only the map surface is empty, and waiting does not fix it. +- **Changing only the HASH from an already-loaded overview works every time**, which is the whole + observation and also the workaround — load an overview first, then jump. +- **Nothing here is measured**: no console read, no network read, no check of whether tiles were + requested at all. Worker-thread fetches do not appear in the main thread's `performance` entries, + so the obvious first probe has to be `read_network_requests` rather than a page script. +- The shape suggests there is no lower-zoom tile to overzoom from while the first z8 requests are in + flight, but that is a guess with nothing behind it. **Not antimeridian-specific** — it reproduced + at 180° only because that is where the camera happened to be. + +## Tiles "jump" a little when panning around a pole (observed 2026-08-11, not analysed) + +- **Observed by eye on Mars**, after the polar seam fix landed and was judged stable: panning around + the pole shows the tiles shifting slightly rather than sliding. Judged not a big deal at the time, + and recorded so it is not re-discovered as new. +- **Nothing here is measured yet** — no camera, no magnitude, no frame capture. Treat every sentence + below as a hypothesis to test, not a finding. +- Likely candidates, in the order worth checking: the render-tile covering set churning as the globe + reassigns zoom near the limb (`terrainZoomsFor` records that a pitched view drops a DEM level); + the cap-to-tile alpha crossfade re-evaluating per frame; and `TERRAIN_SKIRT_DEFAULT = "none"`, + which we ratified knowing it trades skirt artifacts for hairline gaps at zoom boundaries. +- **The cheapest first move is to tell those apart, not to fix any of them** — `?skirt=auto` isolates + the third in one page load, and it is a control that can fail. +- **Not Mars-specific until shown to be.** Everything named above is body-independent, so check + Earth's poles before scoping this as a Mars defect. + +## The polar caps are a texture because MapLibre allows nothing else, and the ceiling is WebP's (analysed 2026-08-07) + +- **State at analysis:** each pole ships one AEQD texture with a four-rung ladder (1024/2048/4096/8192) + picked from the cap's measured on-screen size. It is not a tile pyramid, and the reason has been + assumed rather than recorded. +- **GDAL is not the constraint.** `gdal raster tile --tiling-scheme` offers `APSTILE` and + `LINZAntarticaMapTilegrid` alongside `WebMercatorQuad` — both polar stereographic, both able to + cut a real pyramid over a pole. +- **MapLibre is.** Its raster and vector sources are Web Mercator only; `scheme` chooses `xyz` vs + `tms` and that is the whole vocabulary. Consuming a polar pyramid means a custom loader, LOD + selector and stitcher — most of what the custom cap layer already does, with 8 files instead of + thousands. +- **The texture ceiling is 16,383 px, and it is a file-format limit, not a taste one.** WebP cannot + encode a larger side at all; GPU `MAX_TEXTURE_SIZE` is typically 16,384 on desktop and the mobile + budget already clamps to 4096. So the largest cap that could ever ship is 2× today's linear size. +- **What that would buy, measured against each body's own source:** Mars nothing — its cap already + interpolates its 200 m/px blend. Earth's south cap is the one real gap, sitting several times + coarser than the land DEM beneath it and than the tiles it feathers into at the seam. +- **Verdict: parked, and the gap is Earth's, not Mars's.** Revisit only if MapLibre gains a + TileMatrixSet source, or if Antarctic detail is judged short on the sphere — never from the number. + ## MapLibre's WebGPU backend — irrelevant to our memory problem, and NOT the no-op we recorded (analysed 2026-07-29) Prompted by the graphics-modernization roadmap. Read it against the DEM-cache work rather than in @@ -402,14 +463,15 @@ which is why it never blocked anything. bathymetry-dominant seamount fields (Maldives especially). So an atoll hero is on-aesthetic; the *only* real blocker is the antimeridian split. - **The render pipeline is single-frame end-to-end** — one slug → one bbox → one `frame.json` → - one ortho render → one hero (`country_config.py:100-102` unpacks exactly one `[W,S,E,N]`; - `scene_build.py` one camera/one render; there is no `montage()` anywhere in the tree, so a - multi-frame hero has no existing machinery to extend). + one ortho render → one hero (every `west, south, east, north =` unpack in + `pipeline/frame/country_config.py` takes exactly one `[W,S,E,N]`; `scene_build.py` one + camera/one render; there is no `montage()` anywhere in the tree, so a multi-frame hero has no + existing machinery to extend). - **The frontend already degrades gracefully for a hero-less country** — `rendered:false`/`sizes:[]` - is a first-class manifest state (`gen_manifest.py:97-98`, `lib/manifest.ts`), and both the - gallery card (`index.astro:110-115`) and detail page (`[slug].astro:57-61`) render a placeholder. - It is dead code today because Kiribati is dropped at the manifest step (`gen_manifest.py:82-83` - `continue`s on `resolve()==None`). + is a first-class manifest state (`gen_manifest.py`'s `rendered=bool(sizes)`, `lib/manifest.ts`), + and both the gallery card and the detail page branch on `country.rendered` to render a + placeholder. It is dead code today because Kiribati is dropped at the manifest step — + `gen_manifest.py`'s `main()` `continue`s on `resolve() is None`. ### Viable option A — composited twin-panel hero (keeps Kiribati as one country) @@ -417,7 +479,8 @@ which is why it never blocked anything. normal non-crossing frame rendering like Maldives/Marshall. Preserves country integrity (one sovereign nation = one gallery card) — the reason it beats sub-heroes (below). - **Effort: HIGH.** The single-frame pipeline has no seam for it — needs new code at ~every stage: - a `panels=[...]` config key + list validation (`country_config.py:67,100-102`); per-panel + a `panels=[...]` config key + list validation (`country_config.py`'s `COUNTRY_KEYS` and its + `[W,S,E,N]` unpacks); per-panel work/render subdirs through `stage_commands` (each panel is a *different* AEA projection with its own `frame.json`/heightfield/masks); **a brand-new compositor stage** (the keystone — nothing composites two RGBA renders today); a batch loop over panels; and per-panel border/overlay mapping @@ -429,12 +492,13 @@ which is why it never blocked anything. - Kiribati appears as a placeholder card + gazetteer + detail page, no relief hero — honest about a permanent deferral. Keeps it as one entry. - **Effort: LOW, and entirely in the data/manifest layer** (presentation already exists): (1) emit an - `rendered:false` manifest entry for antimeridian-deferred countries instead of dropping them - (`gen_manifest.py:82-83`); (2) author a `bbox` — Kiribati has `status`/`notes` but no `frame`, and - the gazetteer + globe fly-to read `country.bbox`; (3) guard the globe's `openPanel()` - (`earth.astro:685`) which unconditionally requests `…-${sizes[0]}.webp` → a broken - `kiribati-undefined.webp` for an unrendered entry; (4) optional distinct "deferred" copy — today's - only placeholder string is "still rendering," which misrepresents a permanent state. + `rendered:false` manifest entry for antimeridian-deferred countries instead of dropping them — + `gen_manifest.py`'s `main()` skips them with `if r is None: continue`; (2) author a `bbox` — + Kiribati has `status`/`notes` but no `frame`, and the gazetteer + globe fly-to read `country.bbox`; + (3) guard the globe's `openPanel()` in `Globe.astro`, which sets `heroImg.src` from + `country.sizes[0]` unconditionally → a broken `kiribati-undefined.webp` for an unrendered entry; + (4) optional distinct "deferred" copy — today's only placeholder string is "still rendering," + which misrepresents a permanent state. ### Ruled out (do not re-litigate) @@ -543,7 +607,7 @@ conclusion below, which rests only on there being no value between 2 and 4. - **Trigger:** the capability probe looks like it protects weak devices. Measured against the spec and the code, it barely does. Deferred rather than fixed — the question is a product one (*is `full` the right default for these visitors?*), and nobody has reported a bad experience. -- **`capability.ts:119` is `lowMemory = deviceMemory < 4`** — but **`navigator.deviceMemory` is +- **`capability.ts` then read `lowMemory = deviceMemory < 4`** — but **`navigator.deviceMemory` is spec-quantised to powers of two** (0.25 / 0.5 / 1 / 2 / 4 / 8, clamped at both ends). So `< 4` **cannot** mean "under 4 GB". It means **2 GB or less**. There is no 3. - **It is Chromium-only.** Absent → `Infinity` → never `lowMemory`, so **every Safari and Firefox @@ -552,7 +616,7 @@ conclusion below, which rests only on there being no value between 2 and 4. i.e. the mobile score is measured on a device the ladder treats as healthy. - **A second, independent gap:** `Base.astro`'s pre-paint guard gates `/earth/` on `webgl2()` alone, while `decideTier`'s `capable()` also requires `!softwareGpu`. A software-rasterizer visitor who - deep-links `/earth/` is therefore never bounced to the gallery, and `earth.astro` reads + deep-links `/earth/` is therefore never bounced to the gallery, and `Globe.astro` reads `currentTier()` only to decide whether to spin — so they get a full globe on SwiftShader. - **If reopened, decide these separately:** the memory threshold is a *tuning* question (2 GB is a very low bar; `<= 4` would catch mid-range Android), the Safari/Firefox blindness is a *coverage* question @@ -724,7 +788,7 @@ conclusion below, which rests only on there being no value between 2 and 4. - an **object** sets `params.data`, and `Actor.sendAsync` then calls `serialize(message.data)`, which **recursively rebuilds every array and object**, before `postMessage` structured-clones that rebuilt copy. Two full deep walks of the geometry, on the main thread. -- **We pass objects for all three country sources** (`earth.astro`, `addCountries`) while +- **We pass objects for all three country sources** (`Globe.astro`, `addCountries`) while `boundary_lines.geojson` in the same file is passed as a **URL**. The asymmetry inside one file is the defect; the geometry is only the multiplier. - **Measured** (Node 24 / V8, warm, ×3 — same engine as Chrome, different host, so a proxy): @@ -823,8 +887,8 @@ conclusion below, which rests only on there being no value between 2 and 4. | 8 | 31,972 | 5,091,265 | 159 | Whole pyramid **11.19 MB gzip stored**, never all fetched. The globe opens at **zoom 1.6**, so - the cold window is z1–z2: **~50–180 KB against 2.51 MB today.** Storage is a non-issue against - R2's remaining headroom, and **no tippecanoe dependency** — the stage is a Node script using the + the cold window is z1–z2: **~50–180 KB against 2.51 MB today.** Storage is a non-issue at 11 MB whatever + R2 holds — not because there is headroom, there is none — and **no tippecanoe dependency** — the stage is a Node script using the same tiler MapLibre already runs at runtime. - **The stray-gold-meridian fix survives by construction:** ship the rings as a separate LINE layer in the same tileset. Clipping a line trims it; clipping a polygon closes the ring along the cut. diff --git a/INVENTORY.md b/INVENTORY.md index 4890735..d7fa102 100644 --- a/INVENTORY.md +++ b/INVENTORY.md @@ -8,7 +8,7 @@ `blender/renders/` (the hero products); no assets or DEM data are in git. Free space: **~466 GB** of a 1.8 TB ext4 root. Sizes approximate. -## Raw sources — `data/raw/` (~677 GB) +## Raw sources — `data/raw/` (~688 GB) | Store | Size | What it is | Used by | Reclaim? | |---|---|---|---|---| @@ -20,8 +20,9 @@ | `seaice/` | 640 MB | OSI SAF OSI-450-a monthly EASE2 files + the derived 1991–2020 ice-frequency climatology (`seaice_frequency_1991-2020_4326.tif`) + native `freq_{nh,sh}_ease2.tif` | tile sea ice (`render/seaice.py`) + both caps | Keep (climatology is tiny); `monthly/` regenerable from anonymous THREDDS | | `cop30_void/` | 1.2 GB | Cop30 void-fill DEM | fusion void-fill | Keep | | `naturalearth/` | 38 MB | NE vectors (borders, framing polygons, coastline oracle) | framing, borders, countries/boundary GeoJSON | Keep (tiny) | +| `mars/` | 12 GB | Two whole-planet downloads, no per-tile machinery. The MOLA/HRSC blended DEM (`Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif`, 11,384,463,908 B, 106694 × 53347 int16) is the heightfield. `Mars_Viking_ClrMosaic_global_925m.tif` (797,888,177 B, 23059 × 11530 RGB, SimpleCylindrical metres) is **an acquired input** — it is the field Mars's polar ice alpha is graded from, and `mars_ice.ALPHA_LEVELS` was measured over these exact bytes; it is also what the land ramp's hue was measured against | the DEM feeds Mars's planet seam (`fuse/relabel_mars.py`); the mosaic is acquired by `acquire/download_viking_mosaic.py` and read so far only by the ice-level scripts, no render stage yet | Keep the DEM — re-downloadable, but a ~23 min single-stream fetch with its edition pinned by size and Last-Modified. The mosaic is **re-fetchable exactly**, its acquirer pinning the publisher's own md5, so a deleted copy returns byte-identical in ~90 s — but it is no longer spare, and deleting it now costs a re-fetch rather than nothing | -## Work / intermediates — `data/work/` (~320 GB) +## Work / intermediates — `data/work/` (~330 GB) | Store | Size | What it is | Reclaim? | |---|---|---|---| @@ -30,10 +31,15 @@ | `globathy/` | 16 GB | GLOBathy extracted: `rasters/` = **83,357** per-lake 1″ TIFFs (~15 GB, 83 k inodes) + `lakedepth.vrt` (the Caspian is excluded — watermask class 1, takes GEBCO) | Keep — the VRT is the lake-depth warp's only dependency, and the raw zips it came from are gone, so this IS the store now (re-downloadable via `acquire.download_globathy`, pinned md5) | | `planet/` | 14 GB | Fused planet heightfield + masks, 648 cells of 10° (36 lon × 18 lat, pole to pole) | Keep — input to the tiler | | `planet_terrain/` | **7.6 GB** | Terrain-RGB (Tier 3 displacement), built by `tile/terrain_rgb.py` from `height_3857.tif`. Shipping pyramid is `bathy_s8_webp/` (**2.63 GB, 87,381 tiles, z0–8**), stamped `tiles.done` + `terrain_params.json` so it will not restage, packed to **`terrain.pmtiles` (2.63 GB)** via **`terrain.mbtiles` (2.69 GB)**. **Both the 60 GB `elev/` chain and the spike A/B builds are gone** — this store is now only the shipping pyramid, its archive, and the bridge | Mixed — `terrain.mbtiles` (2.69 GB) is the bridge format and is dead now that the archive is live in production; it rebuilds from `tiles/` in 12 s, the same standing exception `planet.mbtiles` takes. Keep `bathy_s8_webp/` (the pack source) and `terrain.pmtiles` (the deployment artifact). A rebuild re-derives the chain from `height_3857.tif` and no longer copies the master, so it costs ~13 GB transiently, not 60 | -| `planet_countries/` | **10.2 MB** | Country VECTOR tiles (MVT), cut by `compose/countries_pmtiles.py` from `borders/countries.geojson` plus the two layers it derives. One archive, three source-layers (`country_fill`, `country_outline`, `country_hit`), z0–8, stamped `countries_tiles_params.json`. **Three orders of magnitude smaller than the raster pyramids** — it is geometry, not pixels | Keep. Re-cuts from `countries.geojson` in **17 s**, so the archive is cheap to regenerate; the recipe sidecar is what makes a settings change visible, since the filename cannot carry one | +| `planet_vector/` | **10.2 MB** | Earth's VECTOR tiles (MVT), cut by `compose/countries_pmtiles.py` from `borders/countries.geojson` plus the two layers it derives. One archive, three source-layers (`country_fill`, `country_outline`, `country_hit`), z0–8, stamped `countries_tiles_params.json` — the sidecar keeps its producer's name, the archive takes the layer's. **Three orders of magnitude smaller than the raster pyramids** — it is geometry, not pixels | Keep. Re-cuts from `countries.geojson` in **17 s**, so the archive is cheap to regenerate; the recipe sidecar is what makes a settings change visible, since the filename cannot carry one | | `cap/` | 1.3 GB | Both caps' render intermediates (`tile/cap_render.py`): AEQD warps + `cap_{north,south}.tif` + **the freshness sidecars `cap_{north,south}_params.json`** + the A/B rung archives (decision records, ~10 MB). Served outputs live at `web/public/caps/` (two WebP rungs per pole + `caps.json`) | Reclaimable — regenerated by a cap render (deleting the sidecars merely forces one), but budget **≥16 G**: the render peaks ~14 GB and OOMs under the standard 12 G cap (PROCESS § Polar cap render) | | `borders/` | <1 GB | `countries.geojson` + `boundary_lines.geojson` (NE → GeoJSON emitters), served at `/borders/` | Keep (tiny); regenerable from `naturalearth/` | +| `mars/` | 4.2 GB | **The second body's whole work tree**, nested under its own prefix where Earth's stages sit un-prefixed at the root. `planet/` is 12 KB — a CRS-relabelled VRT over the raw blend plus its seam declaration, no copy of the 11 GB. `planet_tiles/` holds the two products of the z7 cut — `tiles/` 1.4 GB (21,845 tiles, z0–7) and **`planet.pmtiles` 1.40 GB** (the deployment artifact, 20,950 unique tile bodies) — plus the four recipe sidecars and the burnt ice GeoJSONs. **Its intermediates are deliberately absent**: `height_3857.tif` (11 GB at 65536²), `planet_rgb.tif` 4.1 GB, `hs_3857.tif` 3.2 GB, the six ice rasters and the alpha, and `planet.mbtiles` were all reclaimed once the cut was accepted, because no remaining phase reads them — vectors are a web overlay and heroes render from the raw DEM | Mixed — the archive and `tiles/` are the products and `tiles/` stays until R2 holds a second copy of the archive. Everything reclaimed rebuilds from raw in ~16:10; the `.done` markers left vouching for absent outputs are safe, since every guard returns "rebuild" for a missing file | +| `mars/ice/` | 205 MB | **Live.** `viking_luma_4326.tif` — the Viking mosaic collapsed to one Float32 brightness band on a 4326 grid covering the whole sphere, which is the field BOTH ice tiers grade against, beside the two VRTs that reach it and `viking_luma_params.json`. Whole-planet on purpose though only the poles are read: a polar crop would save ~160 MB and cost a crop latitude whose failure is ice quietly missing at the band edge | Keep — a 45 s rebuild from the raw mosaic, and the sidecar makes a re-run a skip. `mars_ice.ALPHA_LEVELS` is four percentiles OF THIS FILE, so rebuilding it on a different grid means re-measuring them | +| `mars/cap/` | 1.3 GB | **Live.** The cap stage's intermediates for both poles — the AEQD height warps and the full-size colour renders, beside the `*_params.json` sidecars that decide whether a re-run restages them. `MARS.renders_polar_caps` is `True`, so the shade pass refreshes these on every run | Keep — deleting them costs a ~1:15 re-render, and the sidecars are what make it a skip | | `_profile_tiles/` | 6 MB | The latest `run_pass.sh --tiles` run: `pass.log` (stage timings) + `samples.jsonl`. **Truncated on every run** — only ever the most recent pass | Keep — the source of PROCESS.md's numbers | +| `_profile_mars_tiles/` | 540 KB | The same two files for the FIRST Mars pass, under its own name because that one predates the harness knowing about bodies and was run by hand. Mars runs through `run_pass.sh` now and lands in `_profile_tiles/` beside Earth — the 16 G cap that forced the detour is derived from `renders_polar_caps` and answers 12 G for a capless body | Keep — the source of PROCESS.md's first Mars row, which no later run reproduces | +| `_profile_pass/` | 17 MB | The same two files for the most recent `run_pass.sh` with NO `--tiles`, kept separate from `_profile_tiles/` so a composite-only look iteration does not overwrite the timings of the last full cut. **Truncated on every run** | Keep — the source of PROCESS.md's warm-loop row | | `_*/` experiment scratch | 0 now | A/B and investigation output, by convention leading-underscore (`_ab_shadow`, `_pinecone_exp`, …) | **Reclaim as soon as the decision lands in HISTORY** — the finding is the product, the pixels are not | ### `planet_tiles/` breakdown @@ -86,9 +92,11 @@ ## What the browser loads (dev vs prod) - The wire view — which stores actually reach a visitor, and how dev differs from the deploy - target. Dev serves stores through three routes in `web/astro.config.ts`, pointed by `web/.env` - (`HERO_STORE` → `blender/renders/variants`, `BORDERS_STORE` → `work/borders`, - `PMTILES_STORE` → `planet_tiles/`); the deploy target is Cloudflare — a **site Worker** serving + target. Dev serves stores through three routes in `web/astro.config.ts`: two pointed by + `web/.env` (`HERO_STORE` → `blender/renders/variants`, `BORDERS_STORE` → `work/borders`) and + `/tiles`, whose three archives are derived from the work tree itself + (`work//planet_tiles/planet.pmtiles` and siblings, rooted at `MAPS_DATA`); the deploy + target is Cloudflare — a **site Worker** serving `web/dist` as static assets (`web/wrangler.jsonc`, *not* Pages), R2 for the hero and border stores, and a **separate tile Worker** for tiles. The site addresses all three through `web/src/lib/assetBase.ts`, whose defaults are the same-origin dev paths. @@ -101,10 +109,10 @@ | page CSS | **inlined into every document** (`build.inlineStylesheets: 'always'`), so it costs document bytes and no request — 12 KB on the globe, 5 KB on the gallery, uncompressed | dev injects it as ` + + +
+
+

Terrain covering tiles: a narrow-fov cliff on the globe

+

+ With terrain on the globe projection, coveringTiles returns thousands of renderable + terrain tiles at ordinary cameras. The count is not a smooth function of field of view: it has a + band, and the sweep below finds it. All defaults — no setSourceTileLodParams, no + setMaxPitch, stock maxZoomLevelsOnScreen. +

+

+ + Keep this tab focused — a background tab throttles requestAnimationFrame and every + arm will read the same stale number. Each arm is gated on the WebGL context: if it is lost the + run aborts rather than reporting the last good frame forever. + +

+

+ + +
+ + + +

+

…

+

+ +

+ + + +
loading terrain…
+
+ + + + +
+ + + + diff --git a/docs/upstream/maplibre-terrain-coords-allocation/README.md b/docs/upstream/maplibre-terrain-coords-allocation/README.md new file mode 100644 index 0000000..7b01456 --- /dev/null +++ b/docs/upstream/maplibre-terrain-coords-allocation/README.md @@ -0,0 +1,132 @@ +# Terrain RTT: `_getTerrainCoordsForRegularTile` allocates for every renderable tile, keeps ~4% + +A performance report prepared for upstream [maplibre-gl-js](https://github.com/maplibre/maplibre-gl-js). +Everything here is reproducible with the library alone — `repro.html` in this directory uses public +demo tiles and no application code. + +Observed on **maplibre-gl 6.3.0**, Chrome 151, Linux, RTX 4070 Super. + +## Summary + +`TerrainTileManager._getTerrainCoordsForRegularTile` allocates a tile-ID clone and a `Float64Array(16)` +for **every** renderable terrain tile, then discards both for every tile that is not the same tile, a +parent, or a child. The discard branch is reached *after* both allocations. + +Measured on a live map during a 5 s camera ease, the ratio of allocated to kept is **27.8 : 1 at +pitch 0** — about 5,700 discarded pairs per frame, roughly 11,000 objects per frame that exist only +to be thrown away. + +The fix is mechanical, changes no rendered output, and is largest at low pitch where frame rates are +highest. + +## Where + +`src/tile/terrain_tile_manager.ts`: + +```ts +_getTerrainCoordsForRegularTile(tileID: OverscaledTileID): Record { + const coords: Record = {}; + for (const key of this._renderableTilesKeys) { + const terrainTileID = this._tiles[key].tileID; + const coord = tileID.clone(); // allocated for every renderable tile + const mat = createMat4f64(); // allocated for every renderable tile + if (terrainTileID.canonical.equals(tileID.canonical)) { + ... + } else if (terrainTileID.canonical.isChildOf(tileID.canonical)) { + ... + } else if (tileID.canonical.isChildOf(terrainTileID.canonical)) { + ... + } else { + continue; // ...and discarded here, for most tiles + } + coord.terrainRttPosMatrix32f = new Float32Array(mat); + coords[key] = coord; + } + return coords; +} +``` + +The caller, `src/webgl/render_to_texture.ts`, runs it once per visible coordinate for each source +carrying a draped layer: + +```ts +for (const tileID of tileManager.getVisibleCoordinates()) { + const keys = this.terrain.tileManager.getTerrainCoords(tileID, terrainTileRanges); +``` + +So the count is *visible tiles × renderable terrain tiles*, while the related tiles per call are a +handful. (PR #7863, already merged, narrowed the outer loop to RTT-eligible sources; this is the +inner product that remains.) + +## Measurements + +Taken by wrapping `map.terrain.tileManager.getTerrainCoords` on a live map, counting +`_renderableTilesKeys.length` per call against `Object.keys(result).length`, over one 5 s `easeTo`. +Globe projection, zoom 5.61, terrain on. `repro.html` performs exactly this. + +`repro.html`, globe + terrarium DEM over the Alps: + +| | pitch 0 | pitch 60 | +|---|---|---| +| frames sampled | 825 | 825 | +| `getTerrainCoords` calls per frame | 30.0 | 53.9 | +| renderable terrain tiles per call | 15.0 | 27.0 | +| **allocation pairs per frame** | **450** | **1,454** | +| allocated : kept | **15.0 : 1** | **27.0 : 1** | + +A different scene (a globe over Mars relief, 512 px tiles, three draped sources) gives 5,716 pairs +per frame at pitch 0 against 496 at pitch 60 — **27.8 : 1 and 9.5 : 1**. + +**Note the two scenes disagree about which pitch is worse, so the direction is not the finding.** +What holds across every arm measured is the ratio: between 9 and 28 allocations for each one kept. +In both scenes `kept` equals the call count almost exactly — a visible tile typically has exactly +**one** related terrain tile, so the loop allocates N pairs to keep one. + +Independently, a JS self-profile of a pitched pan attributes **92%** of samples whose leaf frame is +the `Float64Array(16)` allocator to `_getTerrainCoordsForRegularTile < getTerrainCoords < +prepareForRender`, with 4% under `_calcMatrices`. In the heaviest run sampled, that allocator was +27.4% of main-thread busy time. + +## Proposed change + +Two independent, behaviour-preserving edits: + +1. **Decide before allocating.** Compute the relationship first and `continue` before `clone()` and + `createMat4f64()`. This removes the discarded allocations entirely. +2. **Reuse a scratch matrix.** Only `new Float32Array(mat)` escapes the loop, so the `Float64Array` + can be a single module-scope scratch that is re-initialised per hit rather than reallocated. + +Both preserve the returned coords and matrices exactly; existing tests in +`src/tile/terrain_tile_manager.test.ts` should pass unchanged, which is itself the argument that no +rendered output moves. + +## What this does NOT claim + +- **It is not a fix for high-pitch terrain slowness.** That was the hypothesis this investigation + started from and the measurements refute it: in the scene where high pitch was pathological, pitch 0 + allocated 11.5× *more* per frame and was the smooth arm. The waste is real and largely + pitch-independent, and the browser absorbs it when the rest of the frame is cheap. Expect a general + reduction in GC pressure, not a fix for a specific stall. +- **No patched build has been measured.** The expected win is inferred from the allocation counts and + the profile attribution, not demonstrated. A maintainer should treat the numbers above as the + problem statement and the patch as unbenchmarked. + +## Related upstream work + +- **#7863** (merged) — limits RTT preparation to useful sources. Narrows the outer loop; the inner + per-tile allocation is untouched by it. +- **#8048** (open) — stops source LOD settings from reaching internal terrain tiles. +- **#8049** (open) — terrain tile detail near a pitched camera. + +None of these change the allocate-then-discard pattern. + +## Running the reproduction + +`repro.html` is standalone. Serve the directory over http (the DEM source needs a real origin): + +``` +python3 -m http.server 8099 +``` + +then open `http://localhost:8099/repro.html`. Press **Measure at pitch 0** and **Measure at pitch 60**; +each runs a 4 s settle followed by a 5 s eased pan and prints the table above for your machine. diff --git a/docs/upstream/maplibre-terrain-coords-allocation/repro.html b/docs/upstream/maplibre-terrain-coords-allocation/repro.html new file mode 100644 index 0000000..b7fb467 --- /dev/null +++ b/docs/upstream/maplibre-terrain-coords-allocation/repro.html @@ -0,0 +1,161 @@ + + + + + +maplibre-gl — terrain RTT coords allocate-then-discard + + + + +
+
+

Terrain RTT coords: allocated vs kept

+

+ _getTerrainCoordsForRegularTile allocates a tile-ID clone and a + Float64Array(16) per renderable terrain tile, then continues past both + for every tile that is not the same tile, a parent, or a child. +

+

+ Each button settles the camera for 4 s, then measures during a 5 s eased pan. + Run both: the waste is present at every pitch, and is largest at pitch 0. +

+ + +
loading terrain…
+ + + + +
+ + + + diff --git a/pipeline/acquire/download_cop30_void.py b/pipeline/acquire/download_cop30_void.py index 9fa46cd..da7b8d2 100644 --- a/pipeline/acquire/download_cop30_void.py +++ b/pipeline/acquire/download_cop30_void.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Fetch the GLO-30 DEM tiles that OpenTopography holds but AWS GLO-30 Public withholds. The AWS "GLO-30 Public" bucket (download_glo30.py's source) omits ~two dozen tiles over @@ -14,7 +13,7 @@ The water mask for these tiles is NOT here (OpenTopography serves DEM only) — it comes from build_void_wbm.py, which synthesises a WBM from ESA WorldCover over the same extent. -Idempotent/resumable: reuses download_glo30.download_one (streams to .part, size-checks +Idempotent/resumable: reuses fetch.download_one (streams to .part, size-checks against Content-Length, atomic rename); held files are skipped, failures retried next run. Delete data/raw/cop30_void/COP30_hh.vrt to re-check the index against a newer OT edition. @@ -27,8 +26,12 @@ from pathlib import Path from pipeline import paths -from pipeline.acquire.download_glo30 import (TILE_LIST, WORKERS, download_one, - fetch_tile_list) +from pipeline.acquire.download_glo30 import ( + TILE_LIST, + WORKERS, + fetch_tile_list, +) +from pipeline.fetch import download_one OT_ENDPOINT = "https://opentopography.s3.sdsc.edu" OT_PREFIX = "raster/COP30/COP30_hh" # keyless public COG mirror, DGED 2023_1 diff --git a/pipeline/acquire/download_gebco.py b/pipeline/acquire/download_gebco.py index 75377c1..1c4eba0 100644 --- a/pipeline/acquire/download_gebco.py +++ b/pipeline/acquire/download_gebco.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Download the global GEBCO_2026 bathymetry grid and build its mosaic VRT. The one-time global bootstrap for the sea half of the heightfield (the land @@ -9,7 +8,7 @@ The grid is GEBCO_2026 "ice surface elevation" (land + ice-sheet *surface*, not sub-ice bedrock) at 15 arc-seconds, distributed as 8 tiles of 90x90 degrees that tile the globe. Files stream from CEDA over plain HTTPS, -WORKERS at a time, reusing download_glo30.download_one: each streams to a +WORKERS at a time, reusing fetch.download_one: each streams to a .part name, is size-checked against Content-Length, then atomically renamed, so a file under its final name is always complete and re-runs skip it. Unlike GLO-30's unversioned S3 bucket (which needs an ETag oracle), the edition here @@ -26,12 +25,11 @@ import concurrent.futures as cf import subprocess import sys -from pathlib import Path import rasterio from pipeline import paths -from pipeline.acquire.download_glo30 import download_one +from pipeline.fetch import download_one BASE_URL = ("https://dap.ceda.ac.uk/bodc/gebco/global/gebco_2026" "/ice_surface_elevation/geotiff") diff --git a/pipeline/acquire/download_glo30.py b/pipeline/acquire/download_glo30.py index c2efe70..879c700 100644 --- a/pipeline/acquire/download_glo30.py +++ b/pipeline/acquire/download_glo30.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Download Copernicus GLO-30 elevation tiles for one lat/lon extent. For every 1x1 degree land tile intersecting --extent (per the bucket's @@ -28,13 +27,12 @@ import hashlib import json import os -import shutil import sys -import urllib.request -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path -from pipeline import paths +from pipeline import fetch, paths +from pipeline.fetch import download_one BUCKET_URL = "https://copernicus-dem-30m.s3.amazonaws.com" DATA_DIR = paths.DATA / "raw/glo30" @@ -78,7 +76,7 @@ def preflight_cached_age_hours() -> float | None: checked = datetime.fromisoformat(stamp["checked_utc"]) except (OSError, ValueError, KeyError, TypeError): return None - age_hours = (datetime.now(timezone.utc) - checked).total_seconds() / 3600.0 + age_hours = (datetime.now(UTC) - checked).total_seconds() / 3600.0 if not 0.0 <= age_hours < PREFLIGHT_TTL_HOURS: return None return age_hours @@ -95,9 +93,8 @@ def bucket_preflight(): sample = [held[0], held[len(held) // 2], held[-1]] if held else [] for path in dict.fromkeys(sample): name = path.stem - req = urllib.request.Request(f"{BUCKET_URL}/{name}/{name}.tif", - method="HEAD") - with urllib.request.urlopen(req, timeout=30) as resp: + with fetch.open_url(f"{BUCKET_URL}/{name}/{name}.tif", + method="HEAD", timeout=30) as resp: etag = resp.headers["ETag"].strip('"') local = hashlib.md5(path.read_bytes()).hexdigest() if local != etag: @@ -110,7 +107,7 @@ def bucket_preflight(): # replace): the stamp's existence must mean a completed pass. stamp_tmp = preflight_stamp_path().with_suffix(".json.tmp") stamp_tmp.write_text(json.dumps({ - "checked_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "checked_utc": datetime.now(UTC).isoformat(timespec="seconds"), "tiles": [path.stem for path in dict.fromkeys(sample)], }, indent=1) + "\n") os.replace(stamp_tmp, preflight_stamp_path()) @@ -119,27 +116,6 @@ def bucket_preflight(): flush=True) -def download_one(url: str, dest: Path) -> str: - """Download url to dest. Returns 'ok', 'skipped', or 'failed: '.""" - if dest.exists(): - return "skipped" - part = dest.with_suffix(".part") - try: - with urllib.request.urlopen(url, timeout=60) as resp: - expected = int(resp.headers.get("Content-Length", -1)) - with open(part, "wb") as out: - shutil.copyfileobj(resp, out) - actual = part.stat().st_size - if expected != -1 and actual != expected: - part.unlink() - return f"failed: size mismatch ({actual} of {expected} bytes)" - os.replace(part, dest) - return "ok" - except Exception as exc: - part.unlink(missing_ok=True) - return f"failed: {exc}" - - def tile_files(name: str) -> list[tuple[str, Path]]: """(url, local destination) pairs for one tile: the DEM and its water-body mask.""" wbm = name.replace("_DEM", "_WBM") + ".tif" diff --git a/pipeline/acquire/download_globathy.py b/pipeline/acquire/download_globathy.py index eaab655..0be46e8 100644 --- a/pipeline/acquire/download_globathy.py +++ b/pipeline/acquire/download_globathy.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Download the GLOBathy lake-bathymetry dataset (Khazaei 2022, CC0). The general answer to lake depth: every lake except the Caspian and the Great Lakes fuses @@ -24,7 +23,7 @@ API URL (the self-pinning download_gebco.py model) cross-checked against the size + md5 baked in below. If figshare ever serves different bytes under v1, preflight aborts loudly rather than silently swapping the dataset underneath us. Downloads reuse -download_glo30.download_one (.part -> Content-Length check -> atomic rename), so a file +fetch.download_one (.part -> Content-Length check -> atomic rename), so a file under its final name is always complete and re-runs skip it; md5 is verified on fresh downloads only, so an idempotent re-run stays instant instead of re-hashing 16.7 GB. @@ -38,11 +37,10 @@ import hashlib import json import sys -import urllib.request from pathlib import Path -from pipeline import paths -from pipeline.acquire.download_glo30 import download_one +from pipeline import fetch, paths +from pipeline.fetch import download_one DATA_DIR = paths.DATA / "raw/globathy" API = "https://api.figshare.com/v2/articles/{article}/versions/1" @@ -78,7 +76,7 @@ def preflight(entry: dict) -> None: contract broke -- abort rather than download a silently different edition. """ url = API.format(article=entry["article"]) - with urllib.request.urlopen(url, timeout=60) as response: + with fetch.open_url(url, timeout=60) as response: article = json.load(response) served = {item["name"]: item for item in article.get("files", [])} if entry["name"] not in served: @@ -95,7 +93,7 @@ def preflight(entry: dict) -> None: f"no-attribution assumption in ATTRIBUTIONS.md no longer holds") -def fetch(entry: dict, verify_existing: bool) -> str: +def fetch_archive(entry: dict, verify_existing: bool) -> str: """Download one archive if absent; md5-check whatever is new (or --verify).""" dest = DATA_DIR / entry["name"] gigabytes = entry["size"] / 1e9 @@ -136,7 +134,7 @@ def main() -> int: failures = [] for entry in wanted: - status = fetch(entry, args.verify) + status = fetch_archive(entry, args.verify) if status.startswith("failed"): failures.append(f"{entry['name']} {status}") diff --git a/pipeline/acquire/download_mars_dem.py b/pipeline/acquire/download_mars_dem.py new file mode 100644 index 0000000..37f3898 --- /dev/null +++ b/pipeline/acquire/download_mars_dem.py @@ -0,0 +1,214 @@ +"""Download the MOLA/HRSC blended global DEM — the whole of Mars's elevation input. + +ONE FILE, AND THAT IS THE POINT. Earth's heightfield is assembled from ~26,000 Copernicus tiles, +a void-fill edition, a bathymetry grid and a fusion pass; Mars arrives pre-blended as a single +raster, so there is no acquire tier to mirror and no fuse tier to run. Mirroring either would be +inventing work that has no input. What this module owes instead is everything the Earth machinery +gave us for free: proof that the bytes are the edition we designed against, and a loud failure when +they are not. + +Source: the USGS Astrogeology `Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2` mosaic — MOLA (Mars Global +Surveyor) upsampled from its native 463 m/px, with Mars Express HRSC stereo DTMs blended in where +they exist. 200 m/px is a compromise between upsampling MOLA and downsampling HRSC, not a resolution +the whole planet actually carries. + + 200 m/px IS NOT 200 m OF INFORMATION, and this is the most consequential fact about the product. + HRSC covers roughly 44% of the surface; the other 56% is MOLA at 463 m, interpolated. Edges are + feathered over ~5 km. So a straight edge or a texture change in this DEM is a provenance boundary, + not a defect — do not smooth it reflexively, and verify at the data's own pixel scale first. It is + also the argument against a z8 Mars pyramid: z8 would buy four times the disk for a 2.8x upsample + over most of the planet. + +LICENCE: read the product page's own CONSTRAINT FIELDS, never the publisher's organisational status. +This block once said "none to satisfy", reasoning that a USGS Astrogeology PDS Annex product is a US +government work. That is true and it is not the answer: the page states Access Constraints of "MOLA +(CC0) and HRSC (CC BY-SA 3.0 IGO)" and Use Constraints of "Please cite authors" — the publisher +labelling its own inputs, about this exact file, one page load away the whole time. Whether the +share-alike half binds is a legal question; the site answers it by licensing its renders CC BY-SA +4.0, which complies either way. Note where this had to be caught: the obligation attaches to what is +PUBLISHED, so fetching and rendering succeed regardless and no pipeline gate can ever see it. The +constraint-field read belongs here, at acquisition, not months later when a page needs a credit. +ATTRIBUTIONS.md carries both readings and the citation the use constraint asks for. + +A separate trap, unchanged: ESA's published HRSC *pictures* — the colour perspective views — are a +different product under ESA's own terms. Take HRSC from the archive, never from the picture gallery. +Nothing here needs the gallery. + +WHAT WAS MEASURED, rather than read off a landing page (every number below is pinned in code): + 106694 x 53347 px, Int16, nodata -32768, 11,384,463,908 bytes on the wire. It is UNCOMPRESSED and + STRIPED ONE ROW PER BLOCK, with no overviews, and the byte count accounts for itself exactly: + 106694 x 53347 x 2 = 11,383,609,636 bytes of pixels, and the remaining 854,272 is the per-strip + directory that layout costs -- 53347 rows x 16 bytes of BigTIFF offset and byte-count entries = + 853,552, plus a 720-byte header. One row per strip also means it is read SEQUENTIALLY: a windowed + read is not cheap here, and neither is a second pass over it. + Its CRS is `GCS_Mars_2000_Sphere` on a SPHERE of 3,396,190 m — flattening exactly 0, and the same + number `bodies.MARS.ground_radius_m` carries, which `assert_grid` below ties together. + +WHY THE SPHERE MATTERS MORE THAN IT LOOKS. The pipeline projects every body on Earth's spheres +because PROJ refuses to build an operation between two celestial bodies, so this file enters by +having its CRS DECLARED as EPSG:4326 — an identity on the angles, since the grid is already +degrees on an unflattened sphere and only the body label changes. That relabel is only honest while +the source really is a sphere: on an ellipsoid the same declaration would silently shift latitudes. + +THE URL IS A FRONT DOOR, NOT THE ORIGIN, and both halves of that cost a failed run to learn. It is +served through Cloudflare, which 302s to an S3 bucket (`asc-pds-services`, us-west-2) and refuses +the default Python user agent with a 403 before the redirect is ever followed — hence `pipeline +.fetch`, and hence a 403 here means the agent, not a rotted URL. The S3 leg advertises +`Accept-Ranges`, so a resumable fetch is possible if this transfer ever proves unreliable; it is not +built, because nothing has yet measured a need for it. + +Output (data/raw/mars/): + Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif the blend, exactly as published + +Idempotency: the file streams to a `.part` name, is size-checked against Content-Length and only +then atomically renamed (`fetch.download_one`, one home for that rule), so a file under its +final name is always complete and a re-run skips it. + +Edition oracle: `preflight` HEADs the URL and refuses to download unless the size AND the +Last-Modified date still match the pin. USGS republishes mosaics in place under the same filename — +the `_v2` in the name is the product version, not an immutable release — so a same-name file with a +different date is a different planet's worth of pixels arriving under our recipe. `assert_grid` +then re-checks the raster itself, because a byte count is not a grid. + +Usage: + python3 -m pipeline.acquire.download_mars_dem --check # preflight only, downloads nothing + python3 -m pipeline.acquire.download_mars_dem # preflight, then ~10.6 GiB + python3 -m pipeline.acquire.download_mars_dem --verify # re-check the file already on disk +""" + +import argparse +import sys +from pathlib import Path + +import pyproj +import rasterio + +from pipeline import bodies, fetch, paths +from pipeline.fetch import download_one + +DATA_DIR = paths.DATA / "raw/mars" + +BLEND_NAME = "Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif" +BLEND_URL = ("https://planetarymaps.usgs.gov/mosaic/Mars/HRSC_MOLA_Blend/" + f"{BLEND_NAME}") + +#: The edition this pipeline was designed against, pinned so a republished mosaic cannot arrive +#: unannounced. Both fields, not just the size: a re-blend that happens to land on the same byte +#: count is unlikely but a re-upload of the same bytes is not, and only the date separates them. +EXPECTED_BYTES = 11_384_463_908 +EXPECTED_LAST_MODIFIED = "Wed, 09 Nov 2022 14:59:19 GMT" + +#: The grid the entry seam is built on. Restated here as a CONTRACT rather than derived at read +#: time: every one of these is a number some later stage assumes, and a source that quietly changed +#: shape would otherwise be discovered as a wrong-looking planet rather than as an error. +EXPECTED_WIDTH = 106_694 +EXPECTED_HEIGHT = 53_347 +EXPECTED_DTYPE = "int16" +EXPECTED_NODATA = -32768.0 + + +def blend_path() -> Path: + """Where the blend lives once fetched. A function, not a constant, per `paths`.""" + return DATA_DIR / BLEND_NAME + + +def preflight(url: str = BLEND_URL) -> None: + """Assert the server still offers the exact edition this module is pinned to, or exit. + + A HEAD, so this costs no bandwidth and can be run before committing to ~10.6 GiB. It is the + only check that can run BEFORE the download, which is precisely when a drifted edition is + cheapest to discover. + """ + with fetch.open_url(url, method="HEAD", timeout=60) as response: + served_bytes = int(response.headers.get("Content-Length", -1)) + served_date = response.headers.get("Last-Modified", "") + for field, served, expected in (("size", served_bytes, EXPECTED_BYTES), + ("Last-Modified", served_date, EXPECTED_LAST_MODIFIED)): + if served != expected: + sys.exit(f"{BLEND_NAME}: the server now reports {field}={served!r}, pinned to " + f"{expected!r} — the mosaic was republished under the same name. Stop and " + f"re-check the product before re-pinning: the resolution ceiling, the " + f"provenance split and the ground radius all key off this edition.") + + +def assert_grid(path: Path) -> None: + """Assert the raster on disk is the grid the rest of the pipeline assumes, or exit. + + SEPARATE FROM `preflight` BECAUSE A BYTE COUNT IS NOT A GRID. The size pin proves the transfer + matched what was advertised; this proves the advertised thing is still shaped the way the entry + seam, the zoom ceiling and the registry were all written against. + + The sphere check is the load-bearing one. `bodies.MARS.ground_radius_m` is what converts this + pipeline's map units back into Martian ground metres — the hillshade z-factor, the sky-view + search radius, the cap relief all divide by it — and it was taken FROM this product. Holding the + two together here means a source published on the 3,389,500 m mean sphere instead of the + 3,396,190 m IAU sphere is an error at acquisition rather than a 0.2% relief error nobody sees. + """ + with rasterio.open(path) as dataset: + checks = [ + ("width", dataset.width, EXPECTED_WIDTH), + ("height", dataset.height, EXPECTED_HEIGHT), + ("dtype", dataset.dtypes[0], EXPECTED_DTYPE), + ("nodata", dataset.nodata, EXPECTED_NODATA), + ] + crs = dataset.crs + for field, actual, expected in checks: + if actual != expected: + sys.exit(f"{path.name}: {field} is {actual!r}, expected {expected!r} — this is not " + f"the grid the Mars entry seam was built against") + # A geographic CRS in degrees is what makes the EPSG:4326 relabel an identity on the angles. + if crs is None or not crs.is_geographic: + sys.exit(f"{path.name}: CRS is {crs!r}, expected a geographic (degree) CRS — the entry " + f"seam RELABELS this to EPSG:4326, which only holds for lon/lat degrees") + # VIA THE ELLIPSOID, NOT `to_dict()["a"]`, and the difference is not cosmetic: PROJ + # serialises an unflattened body as `+R=`, so a sphere has no `a` key at all and the + # tempting spelling reads None for exactly the products this check exists to inspect. + ellipsoid = pyproj.CRS.from_user_input(crs.to_wkt()).ellipsoid + semi_major = ellipsoid.semi_major_metre if ellipsoid is not None else None + if semi_major is None or abs(semi_major - bodies.MARS.ground_radius_m) > 1.0: + sys.exit(f"{path.name}: published on a sphere of {semi_major!r} m, but " + f"bodies.MARS.ground_radius_m is {bodies.MARS.ground_radius_m} — every ground " + f"metre this pipeline computes for Mars divides by that number, and it was " + f"taken from this product. Re-check the source before changing either.") + + +def build_parser() -> argparse.ArgumentParser: + """The CLI, split out of `main` so its contract is testable without touching the network.""" + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--check", action="store_true", + help="run the edition preflight and stop; downloads nothing") + parser.add_argument("--verify", action="store_true", + help="re-check the grid of the file already on disk; downloads nothing") + return parser + + +def main() -> int: + args = build_parser().parse_args() + destination = blend_path() + + if args.verify: + if not destination.exists(): + sys.exit(f"nothing to verify: {destination} is not on disk") + assert_grid(destination) + print(f"verified {destination} ({destination.stat().st_size:,} bytes)", flush=True) + return 0 + + preflight() + print(f"preflight ok: {BLEND_NAME} is the pinned edition " + f"({EXPECTED_BYTES:,} bytes, {EXPECTED_LAST_MODIFIED})", flush=True) + if args.check: + return 0 + + DATA_DIR.mkdir(parents=True, exist_ok=True) + print(f"downloading {BLEND_URL} -> {destination} (~10.6 GiB) ...", flush=True) + result = download_one(BLEND_URL, destination) + if result.startswith("failed"): + sys.exit(f"{BLEND_NAME}: {result}") + print(f"{result}: {destination}", flush=True) + assert_grid(destination) + print("grid verified against the Mars entry seam's contract", flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/acquire/download_naturalearth.sh b/pipeline/acquire/download_naturalearth.sh index b77d75a..419570f 100755 --- a/pipeline/acquire/download_naturalearth.sh +++ b/pipeline/acquire/download_naturalearth.sh @@ -20,7 +20,12 @@ set -euo pipefail TAG="v5.1.2" BASE="https://raw.githubusercontent.com/nvkelso/natural-earth-vector/$TAG" -DEST="$(cd "$(dirname "$0")/../.." && pwd)/data/raw/naturalearth" +# MAPS_DATA overrides the data root (tests, alternate checkouts); the default is the repo's data/. +# Same idiom as build_mosaics.sh, and load-bearing rather than tidy: this script is the WRITER for +# seven Python readers that all resolve the directory through pipeline/naturalearth.py. Left +# checkout-rooted, a relocated store would have this write where nothing reads. +DATA="${MAPS_DATA:-$(cd "$(dirname "$0")/../.." && pwd)/data}" +DEST="$DATA/raw/naturalearth" LAYERS=( "10m_cultural/ne_10m_admin_0_boundary_lines_land" diff --git a/pipeline/acquire/download_nomenclature.py b/pipeline/acquire/download_nomenclature.py new file mode 100644 index 0000000..612fb39 --- /dev/null +++ b/pipeline/acquire/download_nomenclature.py @@ -0,0 +1,427 @@ +"""Download the IAU/USGS gazetteer geometries — the named features a visitor can point at. + +WHY THIS FILE AND NOT THE ONE THE DOWNLOAD PAGE OFFERS. The gazetteer publishes Mars twice. The +`GIS_Downloads` page lists `MARS_nomenclature_center_pts.zip`, which is one point per name; +`MARS_nomenclature_geometries.zip` is not listed anywhere on it and carries the OUTLINES. Only the +second can answer "what is under this finger", because a point has no interior to hit-test against. +It was found by paging the bucket rather than by reading the page, and a bucket listing is the only +route that names it. + +LICENCE: public domain, and read at the product rather than off a web page. The archive ships +`metadata_nomenclature_polygons_MARS.xml`, whose `useconst` is "Public domain." and whose `distliab` +is "none"; `assert_licence` re-reads that field on every acquisition rather than trusting this +sentence, so a republished archive with different terms stops the pipeline instead of flowing into a +CC BY-SA 4.0 site unnoticed. The access constraint is about tooling, not rights: "GIS software is +required to view these data." + +THE ARCHIVE IS REGENERATED AND ITS BYTES ARE NOT THE THING TO PIN. The bucket rebuilds on a nightly +cadence, and a zip carries member timestamps, so two builds of identical nomenclature differ as +bytes while describing the same 1,717 features. Pinning the zip would re-acquire forever, which is +the `download_sim3292` timeStamp trap in a different costume. Freshness is keyed on the sha256 of +each EXTRACTED MEMBER instead: those are the shapefile's own bytes, and they hold still across a +rebuild that changed nothing. (Measured when this was written: the live object's `Last-Modified` was +the previous day's build and its `Content-Length` matched the copy on disk exactly.) + +THE ENCODING FIX IS WRITTEN BESIDE THE DATA, NOT LEFT AS A RULE FOR EVERY READER. The DBF is UTF-8 +and the archive ships no `.cpg`, so GDAL falls back to Latin-1 and 64 features decode as mojibake -- +`Eugène` becomes `Eugène` -- silently, in the strings that end up on screen. `SHAPE_ENCODING=UTF-8` +fixes it per reader; a `.cpg` file fixes it for every reader that will ever open this shapefile, +including ones nobody has written. So `extract` writes one, and that is the whole reason it does. + +LONGITUDE: THE CENTRES AND THE GEOMETRY DO NOT SHARE A RANGE, and only the second is surprising. +Every `center_lon` is east-positive 0-360. The POLYGONS reach -180 to +360.34, because a feature +crossing the prime meridian is drawn with its vertices continuing past the seam rather than wrapped +-- so the outlines span 540 degrees of a 360-degree planet. A consumer that rasterises on either +convention alone silently drops the features that cross; span 540 and fold onto one window. The +publisher's own `min_lon`/`max_lon` fields reproduce that extent exactly, which is what +`assert_layer` checks rather than re-deriving it from geometry. + +Output (data/raw/mars/nomenclature/): + MARS_nomenclature_geometries.zip the published archive, kept as fetched + MARS_nomenclature_poly.{shp,shx,dbf,prj,cpg} 1,717 named areal features + MARS_nomenclature_line.{shp,shx,dbf,prj,cpg} 203 named linear features + metadata_nomenclature_polygons_MARS.xml the FGDC record carrying the licence + nomenclature_params.json the recipe: host, member digests, counts + +Idempotency: an acquisition whose extracted members all match their pinned digests is skipped. +`--verify` re-asserts what is on disk without touching the network; `--check` reads the live +object's headers and compares them to the archive on disk, downloading nothing. + +Usage: + python3 -m pipeline.acquire.download_nomenclature --verify # re-assert on-disk copy, no network + python3 -m pipeline.acquire.download_nomenclature --check # HEAD the source, write nothing + python3 -m pipeline.acquire.download_nomenclature # fetch and extract if not fresh +""" + +import argparse +import csv +import hashlib +import io +import json +import re +import subprocess +import sys +import zipfile +from pathlib import Path + +from pipeline import fetch, paths + +DATA_DIR = paths.DATA / "raw/mars/nomenclature" + +#: The regional endpoint rather than the global alias. Both answer today; naming the region means a +#: recorded recipe says WHERE the bytes came from, which the global alias leaves ambiguous. +HOST = "https://asc-planetarynames-data.s3.us-west-2.amazonaws.com" +ARCHIVE = "MARS_nomenclature_geometries.zip" + +#: The FGDC record, and the only member that is neither geometry nor its index. Shipped TWICE in the +#: archive under one name -- once per layer group -- and verified byte-identical, so extraction +#: overwrites it with itself and the duplicate is harmless rather than ambiguous. +METADATA_MEMBER = "metadata_nomenclature_polygons_MARS.xml" + +#: sha256 of each extracted member. THE FRESHNESS KEY, and deliberately not the zip's own hash -- +#: see the module note on regeneration. The two `.prj` members are byte-identical to each other, +#: which is why one digest appears twice. +MEMBER_DIGESTS = { + "MARS_nomenclature_poly.shp": "551ab23cffae4891ac23d76645323c0f99a856a5358ae8056dbcecae70f86907", + "MARS_nomenclature_poly.shx": "77e5b6d27433d1c3eaeb4d3502422419c0cb257d9d7e164d245286f6699999a1", + "MARS_nomenclature_poly.dbf": "b3c2f701ef484cf7fd79e7cd59a5669e9265e6ff64a6fedd2601c7ed8706293b", + "MARS_nomenclature_poly.prj": "50ddbbfcb902fe00f6cfe5be39c1f1e64d53ddb1102bfde75d587fc3c2ff91b3", + "MARS_nomenclature_line.shp": "0c28f2e1403e0e5a0de3b4adb453fd2aa0913d7e990cd03990a9e4ddee5fa97c", + "MARS_nomenclature_line.shx": "2da80097c91854fe2da4a5541bca2c58e324b17be99400d870d267e862a7ddf2", + "MARS_nomenclature_line.dbf": "10b931cfc5903a9c347d6f18755dfe8a7c0218cf52991acd1ab46a6e6990d13d", + "MARS_nomenclature_line.prj": "50ddbbfcb902fe00f6cfe5be39c1f1e64d53ddb1102bfde75d587fc3c2ff91b3", + METADATA_MEMBER: "5e0c303fa94618f127e13ef02b3ca1b2da83f0fff8a436c0bbdf54b2c5cb6f1e", +} + +#: The two layers, and what each is for. Areal features are what hit-testing lands on; the linear +#: ones (valles, fossae, catenae) carry names a search index wants and no interior to point at. +LAYERS = ("poly", "line") + +#: Pinned so a truncated extraction or a republished catalogue is an error rather than a quietly +#: smaller map. Every feature is IAU-adopted, so this only moves when the IAU adopts names. +FEATURE_COUNTS = {"poly": 1717, "line": 203} + +#: The publisher's own longitude bounds, summarised per layer as (min of `min_lon`, max of +#: `max_lon`). AN INDEPENDENT ORACLE OVER THE DIGEST, and the one that matters most: it is the +#: 540-degree span from the module note, so a republished file that normalised longitudes into +#: 0-360 would pass every digest-shaped check and silently break the fold every consumer does. +LONGITUDE_BOUNDS = {"poly": (-180.0000, 360.3366), "line": (-138.2220, 358.9187)} + +#: Centres east of 180, per layer. A second, cheaper reading of the same convention question: the +#: centres are uniformly east-positive where the geometry is not, and a file that switched the +#: CENTRES to -180..180 would leave the bounds above untouched. +CENTRES_EAST_OF_180 = {"poly": 1047, "line": 118} + +#: The DBF schema this pipeline reads. `origin` is here because it is the card's entire content on a +#: body with no heroes -- the IAU's own etymology, populated for every feature in both layers -- and +#: a schema change that dropped it would otherwise surface as blank panels. +REQUIRED_FIELDS = frozenset({ + "name", "clean_name", "origin", "diameter", "type", "code", "link", + "center_lon", "center_lat", "min_lon", "max_lon", "approval", +}) + +#: The shape of the gazetteer's own per-feature page. PINNED BECAUSE THE SITE SENDS READERS THERE: +#: `web/scripts/gen_feature_index.py` copies this field verbatim into a tracked artifact, so a +#: republished catalogue that moved its host or its path would ship a dead link per feature -- and a +#: URL is the one field whose breakage is invisible to everything downstream of it, since a wrong +#: address serialises, type-checks and renders exactly like a right one. +FEATURE_URL = re.compile(r"^http://planetarynames\.wr\.usgs\.gov/Feature/\d+$") + +#: The use constraint the FGDC record must still carry. Asserted rather than remembered, because a +#: licence claim in a docstring is exactly the kind that outlives its source. +USE_CONSTRAINT = "Public domain." + + +def archive_path() -> Path: + """Where the published zip lives once fetched. A function rather than a constant, per `paths`.""" + return DATA_DIR / ARCHIVE + + +def layer_path(layer: str, suffix: str = "shp") -> Path: + """One extracted layer's component, e.g. `MARS_nomenclature_poly.dbf`.""" + return DATA_DIR / f"MARS_nomenclature_{layer}.{suffix}" + + +def recipe_path() -> Path: + """The recipe sidecar, beside the outputs it describes.""" + return DATA_DIR / "nomenclature_params.json" + + +def archive_url() -> str: + """The object's URL. Recorded here because a zip on disk cannot say where it came from.""" + return f"{HOST}/{ARCHIVE}" + + +def digest_of(data: bytes) -> str: + """sha256, hex.""" + return hashlib.sha256(data).hexdigest() + + +def extract(archive: Path) -> list[Path]: + """Unpack every pinned member, then write the `.cpg` files the archive does not ship. + + EVERY DIGEST IS CHECKED BEFORE ANY MEMBER LANDS, in two passes, and the second pass is the + point. Verifying and writing in one loop refuses the tampered archive just as loudly, but by + then every member sorting ahead of the bad one has already overwritten a good extraction — + measured, with a one-bit flip in `poly.dbf`: four `line` files landed before the guard fired. + That leaves a directory holding half of one edition and half of another, which is the shape this + repo has been bitten by before: a partial state that reads as a complete one. + + The `.cpg` is written LAST and is this function's only invention: everything else on disk is the + publisher's, and the one file that is ours exists to stop every future reader of this shapefile + from having to know about `SHAPE_ENCODING`. See the module note. + """ + written: list[Path] = [] + with zipfile.ZipFile(archive) as bundle: + members = {info.filename for info in bundle.infolist()} + unexpected = members - set(MEMBER_DIGESTS) + missing = set(MEMBER_DIGESTS) - members + if missing or unexpected: + sys.exit(f"{archive.name}: members are not the published set — missing " + f"{sorted(missing)}, unexpected {sorted(unexpected)}. The archive was " + f"restructured; re-scout it before re-pinning.") + verified: dict[str, bytes] = {} + for name, expected in sorted(MEMBER_DIGESTS.items()): + data = bundle.read(name) + actual = digest_of(data) + if actual != expected: + sys.exit(f"{name}: sha256 {actual} != pinned {expected} — the gazetteer was " + f"republished. Re-measure the counts and bounds in this module before " + f"re-pinning; every coverage figure on record assumes this edition. " + f"Nothing was extracted, so what is on disk is still the pinned edition.") + verified[name] = data + for name, data in verified.items(): + destination = DATA_DIR / name + part = destination.with_suffix(destination.suffix + ".part") + part.write_bytes(data) + part.replace(destination) + written.append(destination) + for layer in LAYERS: + codepage = layer_path(layer, "cpg") + codepage.write_text("UTF-8", encoding="ascii") + written.append(codepage) + return written + + +def read_attributes(layer: str) -> list[dict[str, str]]: + """The layer's DBF rows, decoded through the `.cpg` this module wrote. + + CSV rather than GeoJSON on purpose: every property checked here is an attribute, and asking OGR + for geometry would serialise 1,717 outlines to answer questions none of them bear on. + + DECODED EXPLICITLY RATHER THAN BY `text=True`, because the failure this module exists to prevent + is an encoding failure. Letting the subprocess wrapper decode raises `UnicodeDecodeError` from + inside `subprocess.py`, pointing at a line of the standard library instead of at the missing + `.cpg` that actually caused it -- which is precisely the diagnosis a reader needs here and the + one a traceback destroys. Found by mutation: every layer guard below was reachable only after + this, because a mutated shapefile crashed here first and exited non-zero for the wrong reason. + """ + path = layer_path(layer) + if not path.exists(): + sys.exit(f"nothing to read: {path} is not on disk") + codepage = layer_path(layer, "cpg") + if not codepage.exists(): + sys.exit(f"{codepage.name} is missing — without it GDAL reads this UTF-8 DBF as Latin-1 and " + f"64 features decode to mojibake. Re-run the acquirer; it writes this file.") + # EVERY FIELD, NOT `-select REQUIRED_FIELDS`. Naming the fields makes OGR fail the whole read + # when one is absent, which is the exact condition `assert_layer`'s schema check exists to + # report -- so the guard could never fire, and a dropped field surfaced as a CalledProcessError + # traceback instead of the sentence naming it. Reading everything costs one more column set and + # leaves the diagnosis where it belongs. + completed = subprocess.run( + ["ogr2ogr", "-f", "CSV", "/vsistdout/", str(path)], capture_output=True, check=False, + ) + if completed.returncode != 0: + sys.exit(f"{path.name}: ogr2ogr could not read the layer " + f"({completed.stderr.decode('utf-8', 'replace').strip()[:300]})") + raw = completed.stdout + try: + decoded = raw.decode("utf-8") + except UnicodeDecodeError as exc: + sys.exit(f"{path.name}: OGR returned bytes that are not UTF-8 ({exc}) — the DBF's declared " + f"encoding disagrees with its contents, so every name and origin string read from " + f"it is suspect. Check {codepage.name} against the shapefile it sits beside.") + return list(csv.DictReader(io.StringIO(decoded))) + + +def assert_layer(layer: str, rows: list[dict[str, str]]) -> None: + """Assert one layer is the edition every downstream figure was measured over, or exit. + + FIVE CHECKS THAT CAN EACH PASS WHILE ANOTHER FAILS. The count refuses a truncated or grown + catalogue; the schema identifies the product by its own field list; `origin` being populated + everywhere is what a body with no heroes puts in its detail card, so an empty one is a blank + panel rather than an error; `link` is where that card sends a reader next, and a URL is the one + field whose breakage renders exactly like a working one; and the two longitude readings catch a + re-projection that every byte-shaped check would wave through. + """ + expected_count = FEATURE_COUNTS[layer] + if len(rows) != expected_count: + sys.exit(f"{layer}: {len(rows)} features, pinned to {expected_count} — a short count is a " + f"truncated extraction and a long one is a catalogue the IAU has added to. Either " + f"way the coverage measurements on record describe a different file.") + + missing = REQUIRED_FIELDS - set(rows[0]) + if missing: + sys.exit(f"{layer}: DBF is missing {sorted(missing)} — this is not the gazetteer's " + f"published schema, so nothing downstream can trust the fields it does have") + + blank = [row["name"] for row in rows if not (row.get("origin") or "").strip()] + if blank: + sys.exit(f"{layer}: {len(blank)} feature(s) carry no `origin`, e.g. {blank[:3]} — that " + f"field is the entire content of Mars's detail card, and a blank one is a panel " + f"that opens saying nothing") + + astray = [row["name"] for row in rows if not FEATURE_URL.match((row.get("link") or "").strip())] + if astray: + sys.exit(f"{layer}: {len(astray)} feature(s) carry a `link` that is not a gazetteer feature " + f"page, e.g. {astray[:3]} — the detail card sends readers to that address, and a " + f"wrong one renders identically to a right one all the way to the click") + + low, high = min(float(row["min_lon"]) for row in rows), max(float(row["max_lon"]) for row in rows) + pinned_low, pinned_high = LONGITUDE_BOUNDS[layer] + if abs(low - pinned_low) > 0.001 or abs(high - pinned_high) > 0.001: + sys.exit(f"{layer}: longitude bounds {low:.4f}..{high:.4f}, pinned to " + f"{pinned_low:.4f}..{pinned_high:.4f} — the outlines no longer span what they did. " + f"A file normalised into 0-360 reads as correct everywhere else and silently drops " + f"every feature that crosses the seam.") + + east = sum(1 for row in rows if float(row["center_lon"]) > 180.0) + if east != CENTRES_EAST_OF_180[layer]: + sys.exit(f"{layer}: {east} centres east of 180, pinned to {CENTRES_EAST_OF_180[layer]} — " + f"the centres have changed convention, which the bounds check above cannot see") + + +def assert_licence() -> None: + """Re-read the shipped FGDC record's use constraint, or exit. + + THE TERMS COME FROM THE PRODUCT ON EVERY RUN, not from this module's docstring. A republished + archive that changed its licence is otherwise indistinguishable from one that did not, and the + consequence lands in the site's own CC BY-SA 4.0 rather than here. + """ + path = DATA_DIR / METADATA_MEMBER + if not path.exists(): + sys.exit(f"nothing to verify: {path} is not on disk") + record = path.read_text(encoding="utf-8", errors="replace") + found = re.search(r"(.*?)", record, re.DOTALL) + if not found: + sys.exit(f"{path.name}: no element — the FGDC record no longer states its use " + f"constraints, so this archive's terms are undocumented rather than permissive") + stated = found.group(1).strip() + if stated != USE_CONSTRAINT: + sys.exit(f"{path.name}: use constraint is {stated!r}, pinned to {USE_CONSTRAINT!r} — the " + f"gazetteer's terms changed. Nothing may be published from it until the new terms " + f"are read against the site's CC BY-SA 4.0.") + + +def build_recipe() -> str: + """Everything the outputs depend on besides the source itself, serialised for the sidecar.""" + return json.dumps({ + "host": HOST, + "archive": ARCHIVE, + "url": archive_url(), + "member_sha256": dict(sorted(MEMBER_DIGESTS.items())), + "layers": {layer: {"features": FEATURE_COUNTS[layer], + "longitude_bounds": list(LONGITUDE_BOUNDS[layer]), + "centres_east_of_180": CENTRES_EAST_OF_180[layer]} + for layer in LAYERS}, + "dbf_encoding": "UTF-8, declared by a .cpg this pipeline writes; the archive ships none", + "use_constraints": USE_CONSTRAINT, + }, indent=2, sort_keys=True) + "\n" + + +def is_fresh() -> bool: + """Whether the acquisition can be skipped: every pinned member on disk and matching its digest. + + Reads the FILES rather than the sidecar, for `download_sim3292`'s reason: the sidecar records + what the producer meant to emit, the files are what a consumer will actually open, and a + half-written extraction must not be called fresh because a JSON note beside it agrees. + """ + if not recipe_path().exists(): + return False + for name, expected in MEMBER_DIGESTS.items(): + member = DATA_DIR / name + if not member.exists() or digest_of(member.read_bytes()) != expected: + return False + return all(layer_path(layer, "cpg").exists() for layer in LAYERS) + + +def check_source() -> int: + """HEAD the live object and report it against the copy on disk. Downloads nothing.""" + with fetch.open_url(archive_url(), method="HEAD", timeout=60) as response: + length = response.headers.get("Content-Length") + modified = response.headers.get("Last-Modified") + local = archive_path() + local_size = local.stat().st_size if local.exists() else None + print(f"source: {archive_url()}", flush=True) + print(f" Content-Length {length}, Last-Modified {modified}", flush=True) + print(f" on disk: {local_size if local_size is not None else 'absent'}", flush=True) + # A size match is NOT proof the bytes are the same build -- the archive is regenerated and a + # rebuild of unchanged nomenclature is the same length. It is only a cheap disagreement check. + if local_size is not None and length is not None and int(length) != local_size: + print(" DIFFERS in length — the catalogue changed; re-acquire and re-pin", flush=True) + return 1 + return 0 + + +def build_parser() -> argparse.ArgumentParser: + """The CLI, split out of `main` so its contract is testable without touching the network.""" + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--check", action="store_true", + help="read the live object's headers and compare; write nothing") + parser.add_argument("--verify", action="store_true", + help="re-assert the files already on disk; touches no network") + return parser + + +def main() -> int: + args = build_parser().parse_args() + + if args.verify: + # THE SEMANTIC CHECKS RUN EVEN WHEN A DIGEST HAS MOVED, and the order is the point. A digest + # says "this is the edition we pinned"; the layer checks say "this edition still has the + # properties everything downstream assumes". Exiting on the first drifted digest would make + # the second question unanswerable exactly when it is being asked -- on a re-pin, where what + # you need is the full delta rather than the first difference. + assert_licence() + for layer in LAYERS: + assert_layer(layer, read_attributes(layer)) + print(f"verified {layer_path(layer)} ({FEATURE_COUNTS[layer]} features)", flush=True) + drifted = [name for name, expected in sorted(MEMBER_DIGESTS.items()) + if not (DATA_DIR / name).exists() + or digest_of((DATA_DIR / name).read_bytes()) != expected] + if drifted: + print(f"✗ {len(drifted)} member(s) do not match their pinned digest:", flush=True) + for name in drifted: + print(f" {name}", flush=True) + print(" The assertions above describe what is ON DISK, which is not the pinned " + "edition. Re-pin only after reading what moved.", flush=True) + return 1 + print(f"✓ {len(MEMBER_DIGESTS)} member(s) match their pinned digest", flush=True) + return 0 + + if args.check: + return check_source() + + DATA_DIR.mkdir(parents=True, exist_ok=True) + if is_fresh(): + print(f"{ARCHIVE} fresh -> skip", flush=True) + return 0 + + status = fetch.download_one(archive_url(), archive_path(), timeout=300) + if status.startswith("failed"): + sys.exit(f"{ARCHIVE}: {status}") + print(f"{ARCHIVE}: {status}", flush=True) + + for path in extract(archive_path()): + print(f"wrote {path}", flush=True) + assert_licence() + for layer in LAYERS: + assert_layer(layer, read_attributes(layer)) + + recipe_path().write_text(build_recipe(), encoding="utf-8") # AFTER the members, so a crash + print(f"wrote {recipe_path()}", flush=True) # leaves them stale rather than fresh + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/acquire/download_rgi.py b/pipeline/acquire/download_rgi.py index 51aa96c..37373c0 100644 --- a/pipeline/acquire/download_rgi.py +++ b/pipeline/acquire/download_rgi.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Fetch RGI 7.0 'G' glacier shapefiles (GTN-G regions) from the open UNESCO IHP-WINS mirror and merge them into one EPSG:3857 GeoPackage for rasterizing into the tile snow layer. @@ -14,11 +13,11 @@ import json import subprocess -import urllib.request +import sys import zipfile -from pathlib import Path -from pipeline import paths +from pipeline import fetch, paths +from pipeline.fetch import download_one CKAN = "https://ihp-wins.unesco.org/api/3/action/package_search?q=Randolph+Glacier+Inventory+7.0&rows=5" DATASET = "randolph-glacier-inventory-rgi-7-0-glacier-product" @@ -27,7 +26,7 @@ def resource_urls(): - with urllib.request.urlopen(CKAN, timeout=60) as response: + with fetch.open_url(CKAN, timeout=60) as response: data = json.load(response) pkg = next(p for p in data["result"]["results"] if p["name"] == DATASET) urls = [r["url"] for r in pkg["resources"] if (r.get("format") or "").upper() == "SHP"] @@ -44,7 +43,14 @@ def main(): zip_path = OUT / name if not zip_path.exists(): print(f"downloading {name} ...", flush=True) - urllib.request.urlretrieve(url, zip_path) + # `download_one` rather than a direct fetch, and not only for the User-Agent: it + # streams to `.part` and renames atomically, where the `urlretrieve` this replaced + # wrote straight to the final name. An interrupted download therefore looked exactly + # like a finished one to the `exists()` check above, and the truncation surfaced later + # as a corrupt zip — one stage away from the thing that actually went wrong. + status = download_one(url, zip_path) + if status.startswith("failed"): + sys.exit(f"{name}: {status}") unzip_dir = OUT / zip_path.stem if not unzip_dir.exists(): with zipfile.ZipFile(zip_path) as zf: diff --git a/pipeline/acquire/download_seaice.py b/pipeline/acquire/download_seaice.py index c5cd253..9c8af49 100644 --- a/pipeline/acquire/download_seaice.py +++ b/pipeline/acquire/download_seaice.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Download OSI SAF sea-ice concentration and build a frequency-of-occurrence climatology. The sea half of the polar look: a static, timeless "how often is this sea-water frozen" @@ -29,7 +28,7 @@ global EPSG:4326, packed UInt16 (0..10000 = 0..1 frequency, PACK_FILL where undefined) Idempotency: each monthly file streams to a .part name, is size-checked, then atomically -renamed (download_glo30.download_one), so a file under its final name is always complete and +renamed (fetch.download_one), so a file under its final name is always complete and re-runs skip it. The reduction rebuilds only if its output is missing (or --force). Edition oracle: the versioned THREDDS path (conc_450a_files) + the cdr-v3p0 filename self-pin the release -- a renamed/withdrawn file 404s and aborts the run -- and assert_edition() reads the @@ -53,7 +52,7 @@ from scipy.ndimage import gaussian_filter from pipeline import paths -from pipeline.acquire.download_glo30 import download_one +from pipeline.fetch import download_one DATA_DIR = paths.DATA / "raw/seaice" MONTHLY_DIR = DATA_DIR / "monthly" @@ -237,9 +236,8 @@ def main() -> int: help="rebuild the climatology even if the output exists") args = parser.parse_args() - if not args.build_only: - if download_all() != 0: - return 1 + if not args.build_only and download_all() != 0: + return 1 if not args.download_only: if FINAL.exists() and not args.force: print(f"{FINAL} exists -- pass --force to rebuild", flush=True) diff --git a/pipeline/acquire/download_sim3292.py b/pipeline/acquire/download_sim3292.py new file mode 100644 index 0000000..e2a9e7b --- /dev/null +++ b/pipeline/acquire/download_sim3292.py @@ -0,0 +1,304 @@ +"""Download the SIM 3292 polar map units — the EXTENT of Mars's ice, and only the extent. + +WHAT THIS IS AND IS NOT. The white on Mars's poles is two fields from two sources: an extent (where +white is drawn at all) and an alpha (how white each pixel inside it is). This module acquires the +extent. The alpha comes from elsewhere and nothing here depends on which source that is — the two +are independent, which is what let the alpha's source be replaced outright without touching this. + +WHY A MAPPED SOURCE RATHER THAN A MEASURED ONE, since Earth's ice comes from a measured field. +NSIDC-0791 is a *classification* — how many years out of N a pixel was snow — so on Earth the extent +and the alpha fall out of one number. Mars publishes no equivalent; a scout over the PDS4 registry +and Astropedia found none, and that was a census rather than a sample. TES bolometric albedo is the +obvious substitute and is a trap twice over: it is hard-clipped to 0.06-0.32, destroying every +gradation inside the ice, and its polar holes are FILLED at the floor value rather than flagged +nodata, which makes "nothing was measured" and "the darkest surface on the planet" the same number. + + THE MAPPED BOUNDARY IS NOT A MEASUREMENT OF ICE, and the source says so in its own metadata: + linework drawn while viewing at 1:5,000,000, vertex spacing 5 km, minimum mappable outcrop 40 km + wide by 100 km long. A hard edge between two of those vertices reads as faceted, so any feather + over this boundary is DRAWN and must be described as drawn -- a distance transform wearing a + measurement's clothes is the `antarctic_snow_mask` mistake. How wide such a feather is in PIXELS + belongs to `render/mars_ice.FEATHER_KM`, whose note holds why no count may be written here. + +LICENCE: attribution-only, and read at the product rather than at the publisher. The FGDC metadata +states `Access_Constraints: None`, `Use_Constraints: please cite authors.`, `Distribution_Liability: +None`, `Fees: None`. The citation the use constraint asks for is the metadata's own +`Data_Set_Credit`, reproduced in ATTRIBUTIONS.md and pinned by `tests/test_attributions.py` -- not +composed by us, so there is no guessed string anywhere in the chain. This composes with the site's +CC BY-SA 4.0 without an argument, which is the whole reason this source was reachable when the +first choice of alpha source turned out to be CC BY-NC and therefore unusable. + +WHICH DISTRIBUTION, AND WHY NOT THE PUBLISHED ZIP. The same geometry ships two ways. The +Publications Warehouse offers `sim3292_database.zip` at 808,910,442 bytes, and its shapefile is +Robinson on the Mars 2000 sphere in METRES, carrying a `.prj` that makes PROJ refuse outright: +"Source and target ellipsoid do not belong to the same celestial body". Working with it needs +`Robinson(Mars) -> longlat(Mars) -> -a_srs EPSG:4326 -> EPSG:3857`, where `-a_srs` is the CURE. +(The same flag used INSTEAD of reprojecting is the classic bug, whose failure is a silent all-zero +raster. Any guard over this has to tell the two apart.) + + This module takes the pygeoapi route instead: ~4.3 MB against 809 MB, verified byte-equal on + `features` and `links` against copies fetched independently. Its GeoJSON declares NO CRS, so GDAL + assumes CRS84 and the Mars degrees pass through as Earth degrees -- which is CORRECT here by + design rather than by luck, because the master grid is Earth-radius Web Mercator and the + heightfield is warped into it identically. It stops being correct the moment a consumer declares + the Mars sphere, which is exactly what the zip does. + +THE RESPONSE IS NOT BYTE-STABLE, AND THIS IS THE TRAP THE MODULE IS SHAPED AROUND. pygeoapi stamps +every response with a `timeStamp` field carrying the request time. It is fixed-width ISO-8601, so +two fetches of identical data have the SAME byte length and a DIFFERENT hash -- the one shape that +defeats a size check and a content hash simultaneously. An acquirer pinning raw bytes would read +"the source changed" on every single run and re-acquire forever, which is precisely the idempotency +this pipeline requires. So freshness is keyed on `geometry_digest`, over `features` alone. + +Output (data/raw/mars/sim3292/): + lapc_sim3292.json Late Amazonian polar cap unit, 2 features (one per pole) + apu_sim3292.json Amazonian polar undivided unit, 5 features, both hemispheres + sim3292_params.json the recipe: host, query, and each unit's digest and feature count + +Idempotency: a unit whose file is on disk and whose recorded digest still matches the module's pin +is skipped. `--verify` re-reads the files without touching the network; `--check` reaches the host +and compares digests without writing. + +Usage: + python3 -m pipeline.acquire.download_sim3292 --verify # re-check what is on disk, no network + python3 -m pipeline.acquire.download_sim3292 --check # fetch and compare, write nothing + python3 -m pipeline.acquire.download_sim3292 # fetch what is missing or drifted +""" + +import argparse +import hashlib +import json +import sys +from pathlib import Path +from typing import Any + +from pipeline import fetch, paths + +DATA_DIR = paths.DATA / "raw/mars/sim3292" + +#: The pygeoapi instance. RECORDED HERE BECAUSE THE DATA CANNOT RECORD IT: every `links[].href` in +#: a response comes back RELATIVE, so a file re-fetched from this host still does not name it. That +#: gap is what made the scout copies unreproducible until it was probed. +HOST = "https://astrogeology.usgs.gov/pygeoapi" +COLLECTION = "mars/sim3292_global_geologic_map/units" + +#: `limit` is the SOURCE's paging control, not a cap of ours — it must exceed every unit's feature +#: count or a fetch truncates silently, the response carrying no `numberMatched` to notice with. +#: `FEATURE_COUNTS` below is what actually refuses a truncated fetch. +QUERY_LIMIT = 200 + +#: The units this pipeline draws, and what each one is. `Apu` is in the NORTH extent only, on +#: measurement rather than symmetry: the albedo puts it +0.13 to +0.16 over ordinary ground at matched +#: latitude in the north and within ±0.04 in the south, where it covers 68.7% of the disc — so +#: painting it there would whiten two-thirds of the view on no evidence. That decision belongs to +#: the consumer; this module acquires both and takes no view. +UNITS = ("lApc", "Apu") + +#: Feature counts, pinned so a truncated or re-published response is an error rather than a smaller +#: ice cap. The map is `Maintenance_and_Update_Frequency: None planned`, so drift here means the +#: distribution changed and every extent figure downstream needs re-measuring. +FEATURE_COUNTS = {"lApc": 2, "Apu": 5} + +#: sha256 over `features` alone, canonically serialised — the freshness key, and the answer to the +#: `timeStamp` trap in the module docstring. Pinned to the edition every extent measurement on +#: record was taken over. +GEOMETRY_DIGESTS = { + "lApc": "12c66f17fa53722d02f0b23e94f8e8f857ab6f418bd80f0ae5deb118795b55ad", + "Apu": "b1eb2a86a12b5cdf704fd0bc0a28b1be69157d13c4b7126c91a9c52f02887bd2", +} + +#: The publisher's own geodesic areas, summed per unit, in km². An INDEPENDENT oracle over the +#: digest: a digest proves the bytes are what we saw, and this proves they describe the polygons the +#: publisher measured. `spharea_km` is computed by USGS on the Mars sphere, so it also catches a +#: response that parsed cleanly into the wrong geometry. +SPHERE_AREAS_KM2 = {"lApc": 700_535.0, "Apu": 2_005_394.2} + +#: The property schema, lowercased. The FGDC metadata declares the layer's fields as +#: `Unit` / `UnitDesc` / `SphArea_km`; pygeoapi serves them lowercased, and that correspondence is +#: what identifies these bytes as the published SIM 3292 product rather than a lookalike. +REQUIRED_PROPERTIES = frozenset({"unit", "unitdesc", "spharea_km"}) + + +def unit_path(unit: str) -> Path: + """Where one unit's GeoJSON lives once fetched. + + A function rather than a constant, per `paths`. + """ + return DATA_DIR / f"{unit.lower()}_sim3292.json" + + +def recipe_path() -> Path: + """The recipe sidecar, beside the outputs it describes.""" + return DATA_DIR / "sim3292_params.json" + + +def unit_url(unit: str) -> str: + """The query for one unit — the same one the scout files record in their own `links[].href`.""" + return f"{HOST}/collections/{COLLECTION}/items?f=json&limit={QUERY_LIMIT}&unit={unit}" + + +def geometry_digest(document: dict[str, Any]) -> str: + """sha256 over `features` alone, canonically serialised. + + EXCLUDES `timeStamp` DELIBERATELY, and that exclusion is the point of the function. pygeoapi + stamps each response with the request time in fixed-width ISO-8601, so a re-fetch of unchanged + data differs in hash while matching in length — a raw-bytes pin would re-acquire forever and a + size check would agree that nothing was wrong. + + `links` is excluded too, for a weaker reason: it carries paging hrefs that describe the request + rather than the data. Nothing downstream reads them, and they are relative in every response. + """ + canonical = json.dumps(document["features"], sort_keys=True, separators=(",", ":")) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def assert_document(unit: str, document: dict[str, Any]) -> None: + """Assert a parsed response is the SIM 3292 unit this pipeline was measured against, or exit. + + FOUR INDEPENDENT CHECKS, because each can pass while another fails. The digest proves the bytes + are the edition on record; the feature count refuses a truncated page, which nothing else can + see since the response carries no `numberMatched`; the property schema identifies the product + against its own FGDC field list; and the summed `spharea_km` is the publisher's own geodesic + measurement, so it catches a document that parsed cleanly into geometry we did not expect. + """ + if document.get("type") != "FeatureCollection": + sys.exit(f"{unit}: response type is {document.get('type')!r}, expected 'FeatureCollection'") + + features = document.get("features", []) + expected_count = FEATURE_COUNTS[unit] + if len(features) != expected_count: + sys.exit(f"{unit}: {len(features)} features, pinned to {expected_count} — a short count is " + f"a TRUNCATED page (the response carries no numberMatched to say so), and a long " + f"one is a republished map. Every extent figure on record assumes this count.") + + for index, feature in enumerate(features): + missing = REQUIRED_PROPERTIES - set(feature.get("properties", {})) + if missing: + sys.exit(f"{unit}: feature {index} is missing {sorted(missing)} — the FGDC layer " + f"SIM3292_Global_Geology declares Unit/UnitDesc/SphArea_km, so this is not " + f"the published product's schema") + if feature.get("geometry", {}).get("type") != "MultiPolygon": + sys.exit(f"{unit}: feature {index} is a " + f"{feature.get('geometry', {}).get('type')!r}, expected 'MultiPolygon' — the " + f"ice extent is areal, and a point or line here would rasterise to nothing") + + total_area = sum(feature["properties"]["spharea_km"] for feature in features) + expected_area = SPHERE_AREAS_KM2[unit] + if abs(total_area - expected_area) > 1.0: + sys.exit(f"{unit}: geodesic area sums to {total_area:,.1f} km², pinned to " + f"{expected_area:,.1f} — the publisher's own measurement disagrees, so this is a " + f"different map even if it parsed") + + digest = geometry_digest(document) + if digest != GEOMETRY_DIGESTS[unit]: + sys.exit(f"{unit}: geometry digest {digest} != pinned {GEOMETRY_DIGESTS[unit]} — the " + f"polygons changed. This is NOT a timeStamp difference (the digest excludes it); " + f"re-measure the extents before re-pinning.") + + +def fetch_unit(unit: str) -> dict[str, Any]: + """GET one unit and parse it. Network only; every assertion lives in `assert_document`.""" + with fetch.open_url(unit_url(unit), timeout=120) as response: + return json.loads(response.read().decode("utf-8")) + + +def read_unit(unit: str) -> dict[str, Any]: + """Parse the copy already on disk, or exit saying which file is missing.""" + path = unit_path(unit) + if not path.exists(): + sys.exit(f"nothing to verify: {path} is not on disk") + return json.loads(path.read_text(encoding="utf-8")) + + +def build_recipe() -> str: + """Everything the outputs depend on besides the source itself, serialised for the sidecar. + + A PRODUCER DECLARES WHAT IT EMITTED. The host and query are here because a file on disk cannot + say where it came from — pygeoapi's own `links` are relative — so without this the outputs are + exactly as unreproducible as the hand-fetched scout copies they replace. + """ + return json.dumps({ + "host": HOST, + "collection": COLLECTION, + "query_limit": QUERY_LIMIT, + "units": {unit: {"features": FEATURE_COUNTS[unit], + "geometry_sha256": GEOMETRY_DIGESTS[unit], + "spharea_km2": SPHERE_AREAS_KM2[unit]} + for unit in UNITS}, + }, indent=2, sort_keys=True) + "\n" + + +def is_fresh(unit: str) -> bool: + """Whether this unit can be skipped: on disk, parseable, and matching the pinned digest. + + Reads the FILE rather than the sidecar. The sidecar records what the producer meant to emit; the + file is what a consumer will actually read, and a half-written or hand-edited one must not be + called fresh because a JSON note beside it agrees with the module. + """ + path = unit_path(unit) + if not path.exists() or not recipe_path().exists(): + return False + try: + document = json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError): + return False + return geometry_digest(document) == GEOMETRY_DIGESTS[unit] + + +def write_unit(unit: str, document: dict[str, Any]) -> Path: + """Write one unit's GeoJSON atomically, `timeStamp` stripped. + + The stamp is dropped rather than kept because it is the only part of the response that is not + about Mars: keeping it would make two byte-identical acquisitions differ on disk, which is the + same trap one layer down. What lands is stable, so a plain `diff` between runs means something. + + Written to `.part` and renamed, so a file under its final name is always complete — a crashed + fetch must not leave a truncated document that parses. + """ + stable = {key: value for key, value in document.items() if key != "timeStamp"} + path = unit_path(unit) + part = path.with_suffix(".json.part") + part.write_text(json.dumps(stable, indent=1, sort_keys=True) + "\n", encoding="utf-8") + part.replace(path) + return path + + +def build_parser() -> argparse.ArgumentParser: + """The CLI, split out of `main` so its contract is testable without touching the network.""" + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--check", action="store_true", + help="fetch and assert, but write nothing") + parser.add_argument("--verify", action="store_true", + help="re-assert the files already on disk; touches no network") + return parser + + +def main() -> int: + args = build_parser().parse_args() + + if args.verify: + for unit in UNITS: + assert_document(unit, read_unit(unit)) + print(f"verified {unit_path(unit)} ({FEATURE_COUNTS[unit]} features)", flush=True) + return 0 + + DATA_DIR.mkdir(parents=True, exist_ok=True) + for unit in UNITS: + if not args.check and is_fresh(unit): + print(f"{unit} fresh -> skip", flush=True) + continue + document = fetch_unit(unit) + assert_document(unit, document) + if args.check: + print(f"{unit} ok: {FEATURE_COUNTS[unit]} features, digest matches", flush=True) + continue + print(f"wrote {write_unit(unit, document)}", flush=True) + + if not args.check: + recipe_path().write_text(build_recipe(), encoding="utf-8") # AFTER the units, so a crash + print(f"wrote {recipe_path()}", flush=True) # leaves them stale not fresh + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/acquire/download_viking_mosaic.py b/pipeline/acquire/download_viking_mosaic.py new file mode 100644 index 0000000..b9ce57c --- /dev/null +++ b/pipeline/acquire/download_viking_mosaic.py @@ -0,0 +1,293 @@ +"""Download the Viking colour mosaic — the FIELD Mars's polar ice is graded against. + +WHAT THIS IS AND IS NOT. The white on Mars's poles is two fields from two sources: an extent (where +white is drawn at all) and an alpha (how white each pixel inside it is). `download_sim3292` acquires +the extent; this module acquires the field the alpha is measured from. The two are independent, and +nothing in either one depends on the other's source. + +WHY THE 925 m COLOUR MOSAIC AND NOT A FINER ONE. Two finer global Viking products sit in the same +directory and both are traps for this use, so reaching for one is a redo waiting to happen: + `Mars_Viking_MDIM21_ClrMosaic_global_232m.tif` is RELIEF IMAGERY, not an albedo field — its base + is high-pass filtered to strip regional albedo and emphasise topography, and its colour is this + 925 m mosaic warped on top. It buys registration and spends the very signal an ice ramp grades on. + `Mars_Viking_MDIM21_Mosaic_global_232m.tif` is that same filtered base without the colour. + This product is the one USGS built FOR albedo: Minnaert normalisation, condensate-haze modelling, + and source frames chosen for least atmospheric obscuration and seasonal frost. + +LICENCE: read at the product page's own CONSTRAINT FIELDS, per the rule `download_mars_dem` records. +`Access Constraints: public domain`, `Use Constraints: None` — no attribution obligation at all, +weaker than the Mars DEM's "please cite authors" and weaker than SIM 3292's. The operative reason is +17 U.S.C. §105 rather than age: Viking is NASA and the mosaic is USGS, both federal, and unlike the +MOLA/HRSC blend there is no non-federal contributor to carry terms in. ATTRIBUTIONS.md credits it +anyway, which no constraint requires and courtesy does. + + THE PRODUCT PAGE IS NOT READABLE BY THIS PIPELINE'S HTTP CLIENT, and it fails in the shape that + looks like success. `astrogeology.usgs.gov` renders from JavaScript and serves the same shell for + every slug INCLUDING ones that do not exist, so a fetch of it can neither read the constraints nor + tell a real product from a typo. The reading above was taken in a browser. Do not add a licence + check here that appears to work. + +WHAT THE PUBLISHER SHIPS, enumerated rather than guessed: the bucket behind the mosaic host permits +anonymous listing, so the product's file set is known exactly, and it is four files — the `.tif`, a +detached ISIS `.lbl`, a `_pds3.lbl`, and `.tif.md5`. This module takes the GeoTIFF alone. It is +self-describing, and no stage reads either label. + + THE LABELS AND THE GEOTIFF DISAGREE ABOUT THE SHAPE OF MARS, and only one of them can be acted on. + Both detached labels declare a flattened body — `PolarRadius = 3376200`, `C_AXIS_RADIUS = 3376.2 + ` — while the GeoTIFF's own CRS declares `SPHEROID["Mars",3396190,0]`, flattening + exactly 0. The GeoTIFF governs, and its sphere is what makes the EPSG:4326 relabel every Mars + raster passes through an identity on the angles; the labels' polar radius is the body's physical + shape carried as metadata, and the latitudes are planetocentric, which is the same angle on any + figure. `assert_grid` pins the sphere, so an edition published on the ellipsoid is an error here + rather than a silent latitude shift downstream. + +FRESHNESS IS KEYED ON THE PUBLISHER'S OWN DIGEST, which this product offers and a size pin cannot +match. `.tif.md5` sits beside the mosaic, so `--check` proves the CONTENT is the edition on +record before spending ~761 MiB — where a size-and-date preflight can only prove that something of +the right length is on offer. + +WHAT WAS MEASURED, rather than read off a landing page (every number below is pinned in code): + 23059 x 11530 px, 3 bands, UnsignedByte, nodata 0, 925.406 m/px at 64.05264 px/degree. Its CRS is + SimpleCylindrical in METRES — PROJECTED, unlike the DEM's degrees, so a consumer warps out of map + metres rather than relabelling in place. + The byte count accounts for itself exactly: 23059 x 11530 x 3 = 797,610,810 bytes of pixels, and + the remaining 277,367 is what the layout costs — it is STRIPED ONE ROW PER BAND-ROW with no + overviews, so 34,590 strips x 8 bytes of classic-TIFF offset and byte-count entries = 276,720, + plus 647 of header and IFD. One row per strip also means it is read SEQUENTIALLY: a windowed read + is not cheap here, and neither is a second pass over it. + + NODATA IS 0 ON EVERY BAND AND THAT IS NOT A SCALAR SENTINEL. Invalidity is all three channels zero + together; one channel reading 0 is black, not absent. `mars_ice.albedo_alpha` carries the same + seam from the consumer's side. + +Output (data/raw/mars/): + Mars_Viking_ClrMosaic_global_925m.tif the mosaic, exactly as published + +Idempotency: the file streams to a `.part` name, is size-checked against Content-Length and only +then atomically renamed (`fetch.download_one`, one home for that rule), so a file under its final +name is always complete and a re-run skips the transfer — but still re-digests what it found. + +Usage: + python3 -m pipeline.acquire.download_viking_mosaic --check # preflight only, downloads nothing + python3 -m pipeline.acquire.download_viking_mosaic # preflight, then ~761 MiB + python3 -m pipeline.acquire.download_viking_mosaic --verify # digest the file already on disk +""" + +import argparse +import hashlib +import sys +from pathlib import Path + +import pyproj +import rasterio + +from pipeline import bodies, fetch, paths +from pipeline.fetch import download_one + +DATA_DIR = paths.DATA / "raw/mars" + +MOSAIC_NAME = "Mars_Viking_ClrMosaic_global_925m.tif" + +#: Note the path: this product sits at the mosaic ROOT, where the Mars DEM is one directory deeper +#: under `Mars/HRSC_MOLA_Blend/`. The host's layout is per-product, not per-body. +MOSAIC_URL = f"https://planetarymaps.usgs.gov/mosaic/{MOSAIC_NAME}" + +#: The publisher's checksum sidecar. Fetching it costs a few dozen bytes and is the only edition +#: check that can inspect CONTENT before the transfer is committed to. +CHECKSUM_URL = f"{MOSAIC_URL}.md5" + +#: The edition every Mars ice level on record was measured over. The digest is the PUBLISHER's, so +#: this pin is checkable against the source itself rather than only against a previous download. +EXPECTED_MD5 = "a0e0bbf33ecb0ff65ece9cfa8e08813e" +EXPECTED_BYTES = 797_888_177 +EXPECTED_LAST_MODIFIED = "Thu, 10 Nov 2022 04:35:38 GMT" + +#: The grid, restated as a CONTRACT rather than derived at read time: each of these is a number some +#: later stage assumes, and a source that quietly changed shape would otherwise be discovered as a +#: wrong-looking pole rather than as an error. +EXPECTED_WIDTH = 23_059 +EXPECTED_HEIGHT = 11_530 +EXPECTED_BANDS = 3 +EXPECTED_DTYPE = "uint8" +EXPECTED_NODATA = 0.0 +EXPECTED_PIXEL_METRES = 925.406 + + +def mosaic_path() -> Path: + """Where the mosaic lives once fetched. A function, not a constant, per `paths`.""" + return DATA_DIR / MOSAIC_NAME + + +def file_md5(path: Path) -> str: + """md5 of a file on disk, streamed a megabyte at a time. + + md5 RATHER THAN SOMETHING MODERN because the digest has to be comparable to the publisher's, and + md5 is what they publish. It is an integrity check against a truncated or substituted download, + not a security boundary, and choosing sha256 here would simply make the pin uncheckable. + """ + digest = hashlib.md5() + with open(path, "rb") as handle: + for block in iter(lambda: handle.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +def published_md5(url: str = CHECKSUM_URL) -> str: + """The digest the publisher currently advertises, read from the `.md5` sidecar. + + THE SIDECAR NAMES ITS OWN SUBJECT AND THAT NAME IS CHECKED. The format is ` `, + so a sidecar describing some other product would otherwise hand back a digest that fails the pin + for a reason the error message would misreport — a rotted URL reading as a republished mosaic. + """ + with fetch.open_url(url, timeout=60) as response: + text = response.read().decode("ascii").strip() + fields = text.split() + if len(fields) != 2 or fields[1] != MOSAIC_NAME: + sys.exit(f"{url}: expected ' {MOSAIC_NAME}', got {text!r} — the checksum sidecar does " + f"not describe this product, so its digest cannot be compared to ours") + return fields[0] + + +def preflight(url: str = MOSAIC_URL) -> None: + """Assert the server still offers the exact edition this module is pinned to, or exit. + + A HEAD plus a tiny GET, so this costs no meaningful bandwidth and can run before committing to + ~761 MiB — which is precisely when a drifted edition is cheapest to discover. THE DIGEST IS THE + LOAD-BEARING CHECK and the other two are cheap corroboration: size and date can both survive a + re-render that moved pixels, and the digest cannot. + """ + with fetch.open_url(url, method="HEAD", timeout=60) as response: + served_bytes = int(response.headers.get("Content-Length", -1)) + served_date = response.headers.get("Last-Modified", "") + checks: tuple[tuple[str, object, object], ...] = ( + ("size", served_bytes, EXPECTED_BYTES), + ("Last-Modified", served_date, EXPECTED_LAST_MODIFIED), + ("md5", published_md5(), EXPECTED_MD5), + ) + for field, served, expected in checks: + if served != expected: + sys.exit(f"{MOSAIC_NAME}: the server now reports {field}={served!r}, pinned to " + f"{expected!r} — the mosaic was republished under the same name. Stop and " + f"re-check the product before re-pinning: every polar alpha level was measured " + f"over these pixels, and a re-render moves them.") + + +def assert_grid(path: Path) -> None: + """Assert the raster on disk is the grid the ice levels were measured over, or exit. + + SEPARATE FROM `preflight`, AND NOT REDUNDANT WITH IT EVEN THOUGH A MATCHING DIGEST IMPLIES EVERY + NUMBER BELOW. What this guards is the RE-PIN: when the mosaic is republished the cheap fix is to + update `EXPECTED_MD5` and move on, and at that moment the digest check agrees with whatever + arrived. This function is what still refuses a different planet's shape. + + The sphere check is the load-bearing one, and it is load-bearing for a reason the labels + actively argue against — see this module's docstring. `bodies.MARS.ground_radius_m` is what + converts map units back into Martian ground metres, and the EPSG:4326 relabel this raster passes + through is an identity on the angles only while the source really is unflattened. + """ + with rasterio.open(path) as dataset: + checks: tuple[tuple[str, object, object], ...] = ( + ("width", dataset.width, EXPECTED_WIDTH), + ("height", dataset.height, EXPECTED_HEIGHT), + ("band count", dataset.count, EXPECTED_BANDS), + ("dtype", dataset.dtypes[0], EXPECTED_DTYPE), + ("nodata", dataset.nodata, EXPECTED_NODATA), + ) + for field, actual, expected in checks: + if actual != expected: + sys.exit(f"{path.name}: {field} is {actual!r}, expected {expected!r} — this is not " + f"the mosaic the Mars ice levels were measured over") + + pixel_metres = abs(dataset.transform.a) + if abs(pixel_metres - EXPECTED_PIXEL_METRES) > 0.001: + sys.exit(f"{path.name}: {pixel_metres} m/px, expected {EXPECTED_PIXEL_METRES} — a " + f"resampled edition grades the same ice through a different filter") + + crs = dataset.crs + # PROJECTED, not geographic: this product is SimpleCylindrical METRES where the Mars DEM is + # degrees, so a consumer that relabels in place instead of warping gets Mars-sized numbers + # read as degrees rather than an error. + if crs is None or not crs.is_projected: + sys.exit(f"{path.name}: CRS is {crs!r}, expected a PROJECTED (metre) CRS — consumers " + f"warp this out of SimpleCylindrical map metres, and a geographic edition " + f"would need a different chain entirely") + + # VIA THE ELLIPSOID, NOT `to_dict()["a"]`: PROJ serialises an unflattened body as `+R=`, so + # a sphere has no `a` key at all and the tempting spelling reads None for exactly the + # products this check exists to inspect. + ellipsoid = pyproj.CRS.from_user_input(crs.to_wkt()).ellipsoid + semi_major = ellipsoid.semi_major_metre if ellipsoid is not None else None + semi_minor = ellipsoid.semi_minor_metre if ellipsoid is not None else None + if semi_major is None or abs(semi_major - bodies.MARS.ground_radius_m) > 1.0: + sys.exit(f"{path.name}: published on a body of semi-major {semi_major!r} m, but " + f"bodies.MARS.ground_radius_m is {bodies.MARS.ground_radius_m} — every ground " + f"metre this pipeline computes for Mars divides by that number. Re-check the " + f"source before changing either.") + if semi_minor is None or abs(semi_minor - semi_major) > 1.0: + sys.exit(f"{path.name}: published on an ELLIPSOID (semi-minor {semi_minor!r} m against " + f"semi-major {semi_major!r}), not the sphere this product has always carried. " + f"The EPSG:4326 relabel downstream is an identity on the angles only on a " + f"sphere; on an ellipsoid it shifts latitudes silently. The detached PDS " + f"labels declare a polar radius of 3376200 and are NOT the authority here.") + + +def build_parser() -> argparse.ArgumentParser: + """The CLI, split out of `main` so its contract is testable without touching the network.""" + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--check", action="store_true", + help="run the edition preflight and stop; downloads nothing") + parser.add_argument("--verify", action="store_true", + help="re-digest and re-check the file already on disk; downloads nothing") + return parser + + +def assert_digest(path: Path) -> str: + """Assert the file on disk carries the pinned digest, returning it, or exit.""" + digest = file_md5(path) + if digest != EXPECTED_MD5: + sys.exit(f"{path.name}: md5 {digest} != pinned {EXPECTED_MD5} — the bytes on disk are not " + f"the published edition. Delete the file and re-run rather than re-pinning: a " + f"truncated or substituted mosaic looks exactly like this.") + return digest + + +def main() -> int: + args = build_parser().parse_args() + destination = mosaic_path() + + if args.verify: + if not destination.exists(): + sys.exit(f"nothing to verify: {destination} is not on disk") + digest = assert_digest(destination) + assert_grid(destination) + print(f"verified {destination} ({destination.stat().st_size:,} bytes, md5 {digest})", + flush=True) + return 0 + + preflight() + print(f"preflight ok: {MOSAIC_NAME} is the pinned edition ({EXPECTED_BYTES:,} bytes, " + f"{EXPECTED_LAST_MODIFIED}, md5 {EXPECTED_MD5})", flush=True) + if args.check: + return 0 + + DATA_DIR.mkdir(parents=True, exist_ok=True) + if destination.exists(): + print(f"{destination} is already on disk — re-checking, not re-fetching", flush=True) + else: + print(f"downloading {MOSAIC_URL} -> {destination} (~761 MiB) ...", flush=True) + result = download_one(MOSAIC_URL, destination) + if result.startswith("failed"): + sys.exit(f"{MOSAIC_NAME}: {result}") + if result == "ok": + print(f"wrote {destination}", flush=True) + # Re-digested even when the transfer was SKIPPED, which is the case that matters: a file already + # under its final name is complete by construction but says nothing about which edition it is. + assert_digest(destination) + assert_grid(destination) + print("digest matches the publisher's checksum; grid verified against the ice levels' contract", + flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/acquire/extract_globathy.py b/pipeline/acquire/extract_globathy.py index ce7d2cc..1b1b282 100644 --- a/pipeline/acquire/extract_globathy.py +++ b/pipeline/acquire/extract_globathy.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Extract the GLOBathy rasters worth reading at z8 and build their mosaic VRT. GLOBathy ships 1,427,688 per-lake 1" GeoTIFFs in one 16.73 GB zip. At 306 m/px most are a @@ -32,7 +31,6 @@ import subprocess import sys import zipfile -from pathlib import Path from pipeline import paths diff --git a/pipeline/batch.py b/pipeline/batch.py index d0b2835..a33769f 100644 --- a/pipeline/batch.py +++ b/pipeline/batch.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Batch runner: drive the whole pipeline across every in-scope country. Reuses country_config's resolver (scope, frames, per-country stage commands) @@ -42,14 +41,25 @@ import subprocess import sys import time -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path -from pipeline.frame.country_config import (build_scope, load_config, - load_ne_rows, preflight_gebco, - resolve, stage_commands) - -ROOT = Path(__file__).resolve().parent.parent +from pipeline import paths +from pipeline.frame.country_config import ( + build_scope, + country_render_dir, + country_work_dir, + load_config, + load_ne_rows, + preflight_gebco, + resolve, + stage_commands, +) + +#: The CHECKOUT, and the working directory every stage subprocess is run from — so the +#: checkout-relative paths in those commands (`pipeline/…`, `blender/…`) resolve. Data paths do NOT +#: hang off it; they come from `country_work_dir`, which follows the store. +ROOT = paths.ROOT # Stage commands say "python …" assuming a venv-active shell; put this runner's # own interpreter dir first on PATH so the subprocesses use the venv, not system. ENV = {**os.environ, @@ -77,7 +87,7 @@ def detect_cgroup_cap() -> bool: return False probe = ("systemd-run --user --scope -q -p MemoryMax=256M " "-p MemorySwapMax=0 -- true") - return subprocess.run(probe, shell=True, capture_output=True).returncode == 0 + return subprocess.run(probe, shell=True, capture_output=True, check=False).returncode == 0 def wait_for_mem(floor_gib: float) -> bool: @@ -96,7 +106,7 @@ def wait_for_mem(floor_gib: float) -> bool: def log_failure(slug, stage_index, cmd, returncode, kind) -> None: FAIL_LOG.parent.mkdir(parents=True, exist_ok=True) - ts = datetime.now(timezone.utc).isoformat(timespec="seconds") + ts = datetime.now(UTC).isoformat(timespec="seconds") with open(FAIL_LOG, "a") as log_file: log_file.write(json.dumps(dict( ts=ts, slug=slug, stage_index=stage_index, @@ -108,7 +118,7 @@ def bootstrap() -> None: for cmd in ("bash pipeline/acquire/download_naturalearth.sh", "python -m pipeline.acquire.download_gebco"): print(f"[bootstrap] {cmd}", flush=True) - if subprocess.run(cmd, shell=True, cwd=ROOT, env=ENV).returncode != 0: + if subprocess.run(cmd, shell=True, cwd=ROOT, env=ENV, check=False).returncode != 0: sys.exit(f"bootstrap failed: {cmd} — cannot proceed without it") @@ -117,7 +127,7 @@ def prune_intermediates(slug: str) -> None: fused rasters, warp dir, and scene file. The hero PNG and the shared raw GLO-30/GEBCO tiles are kept. Used by --clean to keep a full sweep within disk (a near-global run accretes ~500 GB of tiles + fusions otherwise).""" - work = ROOT / f"data/work/{slug}" + work = country_work_dir(slug) if work.exists(): shutil.rmtree(work) (ROOT / f"blender/{slug}_hero.blend").unlink(missing_ok=True) @@ -129,7 +139,7 @@ def run_country(slug, resolved, through, force, dry, cap_gib, use_cap, floor, """Run one country's stages; return a short outcome string.""" do_clean = clean and through == "render" and not dry target = (ROOT / f"blender/renders/heroes/{slug}.png" if through == "render" - else ROOT / f"data/work/{slug}/render/lakedepth_aea.tif") + else country_render_dir(slug) / "lakedepth_aea.tif") if target.exists() and not force: if do_clean: prune_intermediates(slug) @@ -165,7 +175,7 @@ def run_country(slug, resolved, through, force, dry, cap_gib, use_cap, floor, flush=True) else: rc = subprocess.run(prefix + run_cmd, shell=True, cwd=ROOT, - env=ENV).returncode + env=ENV, check=False).returncode if rc != 0: kind = "oom" if rc in (137, -9) else "error" if raw_tmp: @@ -180,10 +190,10 @@ def run_country(slug, resolved, through, force, dry, cap_gib, use_cap, floor, # writes the shaded hero as a SEPARATE file (atomic, internal .tmp), so # the raw stays pristine and post-look tweaks never re-render. sv = subprocess.run( - f"python -m pipeline.render.sky_view --render-dir data/work/{slug}/render" + f"python -m pipeline.render.sky_view --render-dir {country_render_dir(slug)}" f" --hero {raw} --out {final}" f" --strength {resolved['sky_view_strength']}", shell=True, - cwd=ROOT, env=ENV).returncode + cwd=ROOT, env=ENV, check=False).returncode if sv != 0: log_failure(slug, idx, "sky_view", sv, "error") return f"FAIL@{idx} (sky_view)" diff --git a/pipeline/bodies.py b/pipeline/bodies.py index fa4ec36..115ff59 100644 --- a/pipeline/bodies.py +++ b/pipeline/bodies.py @@ -7,28 +7,63 @@ produce a different planet. Geometry (how big the sphere is), the vertical exaggeration its relief is drawn at, and how deep its pyramid is cut. Everything else about a planet is data. -WHY THIS EXISTS BEFORE THERE IS A SECOND BODY. Every one of these values is currently a module-level -constant sized for Earth, and two of them are already written out twice with nothing relating them -(`EARTH_RADIUS` in `render/hillshade.py` and `render/snow.py`). Adding a planet turns each into a -cross-body bug of the worst kind: the wrong sphere radius does not raise, it scales the per-row -hillshade z-factor by latitude and produces a relief that is plausible everywhere and true nowhere. - -THE VALUES HERE ARE STILL DUPLICATED ELSEWHERE, ON PURPOSE AND UNDER GUARD. This module is a pure -addition — nothing reads it yet — so every constant it states also still lives at its original call -site. `tests/test_bodies.py` pins each pair, so the interim cannot drift, and each bridge assertion -dies with the copy it holds. Copied look constants have already cost this project one overnight -re-render of every hero; the only safe copy is one a test refuses to let diverge. +WHAT A BODY IS NOT, so the neighbours are findable: its colours are a `palette.Look`, its optional +surface layers are a `layers.Layer` vocabulary (a body answers for them in `surface_layers`, but the +vocabulary itself is a pipeline fact), and what its planet stage emitted is a `planet_seam` +declaration. + +WHY A REGISTRY AND NOT CONSTANTS. A wrong sphere radius does not raise: it scales the per-row +hillshade z-factor by latitude and produces relief that is plausible everywhere and true nowhere. +Copied look constants have already cost this project one overnight re-render of every hero, so a +value that still has a second home is pinned to this module by a bridge test in +`tests/test_bodies.py`, and each bridge dies with the copy it holds. + +EVERY PROJECTION HERE IS EARTH-SPHERED, WHATEVER PLANET THE ELEVATIONS DESCRIBE. Stated once +because it is the shape of three fields below and of `fuse/relabel_mars.py`, not a fact about any +one of them. PROJ refuses to build an operation between two celestial bodies, and `gdal raster +tile` reprojects into WebMercatorQuad — EPSG:3857 — so a Mars-radius Mercator raster cannot be cut +into tiles at all without disabling that guard globally. Measured twice, because the obvious +objection is that a hand-written proj4 string names no body and might slip past: `gdalwarp -t_srs +EPSG:3857` from IAU_2015:49900 exits 1 with "Source and target ellipsoid do not belong to the same +celestial body (Earth vs Mars)", and so does a warp to an AEQD spelled `+a=3396190 +b=3396190`, +with no EPSG code anywhere — while the identical warp to `+a=6371000` succeeds. A non-Earth +heightfield therefore enters by having its CRS DECLARED as EPSG:4326, an identity on angles where +only the sphere label changes. + +WHICH IS WHY THERE ARE THREE RADII. Each does the one job a radius does — turn an angle into a +length — and they are separate because that job is asked in three coordinate systems. Two name +projections and are Earth's for every body by the rule above; the third names the planet, and is +the single fact that converts map units back into ground metres: + + mercator_radius_m the tile grid's sphere | forced to Earth's, every body + aeqd_radius_m the polar caps' sphere | + ground_radius_m the planet itself what a ground metre is actually worth + +EARTH HIDES THE DISTINCTION, which is why nothing noticed it until a second body: EPSG:3857 is +defined on a sphere of 6378137 m, which is also Earth's own equatorial radius, so +`mercator_radius_m` had been answering two questions with one number. Earth's Mercator ground ratio +is therefore exactly 1.0 by construction of the projection rather than by rounding, while its cap +ratio is 1.0011202 — which is why the two conversions at the foot of this module are separate +functions and must never be collapsed into one `ground_metres_per_map_unit`. NO FIELD MAY CARRY A DEFAULT. A default would let a field added later be inherited unexamined by every planet but the one it was written for — invisible in the diff that adds it. Without defaults, adding a field is a hard error at every construction until each body answers for it. +WHICH IS ALSO WHY THIS IS A FLAT REGISTRY OF FROZEN DATACLASSES AND NOT A CLASS HIERARCHY, a +question worth answering once rather than each time it is asked. Inheritance is the mechanism for +acquiring an unexamined answer, so a `class Mars(Body)` would reintroduce exactly what the rule +above exists to refuse. It would also have nothing to dispatch: no consumer anywhere branches on +which body it holds — every one of them reads a FIELD (`body.exaggeration`, `body.ground_radius_m`, +`"perennial_ice" in body.surface_layers`), so a subclass would carry no overridden behaviour and be a +constructor call spelled longer. A body's facts are DATA, and a frozen dataclass is how Python +states data. THIS ARGUMENT IS ABOUT BODIES AND DOES NOT CARRY TO PRODUCERS — those are behaviour and +they do dispatch, which is why `render/perennial_ice.py` is a registry of functions instead. + from pipeline import bodies body = bodies.get("earth") # raises on an unknown name; never falls back """ -from __future__ import annotations - from dataclasses import dataclass from pathlib import Path @@ -45,21 +80,42 @@ class Body: #: Registry key and path segment. Lowercase, no spaces — it names directories and archive keys. name: str - #: Sphere radius used for Web-Mercator ground-metre arithmetic, in metres. + #: The tile grid's sphere, in metres — one of the three the module note sets out. #: - #: This is the projection's sphere, NOT the body's mean or equatorial radius, and the two are - #: different questions. Consumers use it to turn a Mercator y back into a latitude and to size - #: the per-row hillshade z-factor, both of which must agree with whatever radius the raster was - #: warped with. Mixing two radii yields a latitude-varying error that renders plausibly. + #: Consumers turn a Mercator y back into a latitude with it and size the per-row hillshade + #: z-factor from it, both of which must agree with the radius the raster was actually warped + #: with. Mixing two of the three yields a latitude-varying error that renders plausibly. mercator_radius_m: float - #: Sphere radius for the polar caps' azimuthal-equidistant projection, in metres. + #: The polar caps' azimuthal-equidistant sphere, in metres — NOT the Mercator radius above. #: - #: A SECOND RADIUS, AND DELIBERATELY NOT THE FIRST. Three are in play on Earth: Web Mercator's - #: 6378137 (the tile grid), this AEQD sphere at 6371000, and MapLibre's own globe radius of - #: 6371008.8 on the frontend. The last two sit 8.8 m apart, and that gap is load-bearing — the - #: cap texture is projected on one and blended against tiles drawn on another, so collapsing - #: them puts the polar seam exactly that far out. One `radius_m` field would invite the collapse. + #: THE 8.8 m BETWEEN THIS AND THE FRONTEND IS LOAD-BEARING, which is the one thing about this + #: field the module note does not cover. MapLibre's globe radius is 6371008.8; the cap texture + #: is projected on this sphere and blended against tiles drawn on another, so collapsing any + #: two of the three puts the polar seam exactly that far out. One `radius_m` field would invite + #: the collapse, and nothing downstream would report it as anything but a seam. aeqd_radius_m: float + #: The body's own sphere, in metres — what a ground metre is worth on this planet, and the only + #: one of the three that is physics rather than a projection. See the module note for why the + #: other two are Earth's and this one is what converts their map units back. + ground_radius_m: float + #: Size of one pixel of the EPSG:3857 raster the pyramid is cut from, in MAP UNITS. + #: + #: Map units, not ground metres — they are metres on `mercator_radius_m`'s sphere, so a ground + #: distance is this times `ground_metres_per_mercator_unit(body)`. Writing the conversion out at + #: each call site is deliberate: the units then cancel visibly, and a site that forgot it reads + #: wrong. + #: + #: STORED RATHER THAN DERIVED FROM `tile_max_zoom`, and the reason is measured. Earth's live + #: 46 GB `height_3857.tif` was warped at 305.7483, a rounded value: the exact figure is + #: 305.748113, and `-tap` snapped the grid 12.2 m past the true Mercator edge on every side. + #: Deriving would not restage anything today — `height_3857` is gated on its sources' mtimes and + #: every sibling raster compares against height's ACTUAL grid rather than against this number — + #: it would instead sit inert until the next unrelated re-fuse re-warped height at a new + #: resolution, moving the grid under all six siblings at once and restaging the planet under + #: someone else's change. A latent trap that misattributes itself is worse than a recorded + #: asymmetry, so Earth keeps the number its pixels were actually built at, and + #: `tests/test_bodies.py` pins every body's value against its own ceiling relationally. + map_units_per_pixel: float #: Vertical exaggeration the relief is drawn at, shared by the hero scene and the tile shading. #: #: A look constant rather than a physical one: it is chosen so the planet reads well, and it is @@ -79,26 +135,152 @@ class Body: #: a full composite and cut, ~26 minutes — to produce pixels identical to the ones sitting there. #: A second body pays no such cost, so it nests properly from the start. path_prefix: str + #: Which of `layers.SURFACE_LAYERS` this body actually has, by name. Empty is a real answer. + #: + #: NAMES AND NOT `Layer` OBJECTS, because this set is serialised: `layers.layers_off` turns it + #: into the `layers_off` list inside `composite_params.json`, and anything whose JSON differs + #: restages a 33-minute Earth composite for identical pixels. + #: + #: Spelled out per body rather than defaulting to "all of them", so adding a sixth layer is a + #: decision for every planet including Earth. `tests/test_bodies.py` refuses a name outside the + #: vocabulary — a typo would otherwise turn a layer off silently, which is the same failure this + #: field exists to close. + #: + #: THE ANTARCTIC LAND-ICE RULE RIDES WITH `perennial_ice`, and that is not a conflation. The rule + #: exists only because the snow dataset has a hole — NSIDC-0791 is northern-hemisphere-only and + #: RGI region 19 is excluded — so the continent would render on the tan LAND ramp. It is a patch + #: on that layer, so a body without it has nothing to patch. On a body with no sea it would + #: instead whiten every piece of land below 60 degrees south. + surface_layers: frozenset[str] + #: Whether this body PUBLISHES rendered polar-cap textures. + #: + #: NOT A STATEMENT ABOUT THE PLANET'S CRYOSPHERE, and the name says `renders` for exactly that + #: reason: Mars has real polar ice caps, and `False` here would be a plain factual error read + #: that way. What it describes is the AEQD disc that repairs Web Mercator, which dies at ~85 + #: degrees and leaves a hole at each pole that the tiles cannot fill. + #: + #: SO `False` COSTS A VISIBLE HOLE, and that is the trade rather than a free saving. It is the + #: right answer only while a body's ramps are unratified, because a cap is shaded by the same + #: `shade.composite` as the tiles: rendering one publishes a look decision. Measured on Mars, + #: the cap pass runs happily today off the heightfield alone — one source, nothing missing, no + #: refusal — so without this field a first tile run quietly spends ~14 GB per pole to ship two + #: discs in a palette nobody has agreed to. + #: + #: A body fact rather than a look constant because the two consumers are in different processes: + #: the shade pass decides whether to invoke the cap pass at all, and the cap pass must give the + #: same answer when an operator runs it directly. Absence on disk cannot carry that — it cannot + #: tell "this body publishes none" from "the render died", which is the distinction + #: `planet_seam` exists to preserve one tier up. + renders_polar_caps: bool EARTH = Body( name="earth", - # Web Mercator's sphere. Duplicated today in render/hillshade.py and render/snow.py. + # Web Mercator's sphere. `mercator.WEB_MERCATOR_RADIUS_M` is the projection's own statement of + # the same number and is bridged to this field; the two agreeing is a coincidence with its own + # test, not one value read twice. mercator_radius_m=6378137.0, # The caps' AEQD sphere. NOT the Mercator one above, and not MapLibre's globe radius. aeqd_radius_m=6371000.0, + # The SAME number as its Mercator grid, and NOT a copy of the field above: EPSG:3857 is defined + # on Earth's equatorial radius, which is what makes Earth's ground ratio exactly 1.0 and every + # existing pixel byte-identical through each call site that adopts the conversion. + ground_radius_m=6378137.0, + # The ONE home now: the shade pass reads this, and a test scans it for a regrown literal. The + # number is what the live 46 GB raster was actually warped at, and it is a rounded value — the + # exact z8 figure is 305.748113. See the field's note for why the rounding stays. + map_units_per_pixel=305.7483, # Duplicated today in render/palette.py, which the hero scene imports directly. exaggeration=15.0, - # Duplicated today in tile/shade_planet.py's TILE_CUT and compose/countries_pmtiles.py. + # The raster cut reads this; compose/countries_pmtiles.py still carries its own copy, because + # the vector pyramid is Earth-only until a Mars layer is designed. That last copy is bridged. tile_max_zoom=8, # Empty on purpose — see the field's note. Earth's intermediates stay exactly where they are. path_prefix="", + # All of them, written out rather than spelled `SURFACE_LAYERS`: Earth is the reference body, and + # "whatever the vocabulary happens to contain" is how it would inherit the next layer unexamined. + surface_layers=frozenset({"lake_depth", "perennial_ice", "glaciers", "sea_ice", "coastline"}), + # The reference body, and the caps are a signature feature rather than a detail: both poles + # ship a full rung ladder, feathered into the tiles at the seam. + renders_polar_caps=True, +) + + +MARS = Body( + name="mars", + # BOTH PROJECTION SPHERES ARE EARTH'S, ON PURPOSE, AND NEITHER IS A COPY-PASTE SLIP — the module + # note holds the PROJ constraint that forces it and the two warps that measured it. The tempting + # "fix" is Mars's own radius here; `tests/test_bodies.py` asserts this sameness so that fix fails + # at the gate rather than months later at the tiler. + mercator_radius_m=6378137.0, + # The same constraint, separately measured against a hand-written proj4 string naming no + # celestial body — which does not escape the check either. See the module note. + aeqd_radius_m=6371000.0, + # The IAU 2015 Mars sphere, which is also what the source DEM's own CRS declares — so our + # ground metres agree with the grid the data was published on. It is the equatorial radius used + # as a sphere, NOT the 3389500 m mean; the two differ by 0.2%. The ratio against the grid sphere + # is 0.532474, so a hillshade z-factor comes out 1.878x Earth's for the same physical + # exaggeration. + ground_radius_m=3396190.0, + # Exactly 2*pi*6378137 / (512 * 2**7). Stored rather than derived for the reason the field + # states, and pinned against `tile_max_zoom` relationally — so moving the ceiling without moving + # this is a red test rather than a pyramid cut at a zoom its raster was not built for. + map_units_per_pixel=611.49622628141, + # JUDGED ON THE SPHERE, which is how Earth's own 15x was settled and the only way this number + # was ever going to be. The arithmetic that opened at 10x is kept because it is worth knowing it + # was wrong: MapLibre's globe shader draws every body on one Earth-sized sphere and displaces in + # metres, so only metres matter, and Mars's ~30 km range is ~1.5x Earth's ~20 km — hence + # 15 / 1.5 ~ 10. Looking at 10x and 20x side by side at the same camera settled it at 20x. + # + # THE SATURATION WORRY WAS MEASURED AND DOES NOT APPLY HERE, and it is recorded because it is + # the reason not to fear the next step up. Earth is already saturated at 15x, so more steepness + # there buys less than the number suggests. Mars at 20x is not: on the real hillshade raster, + # 0.00% of pixels sit at DN 0 or DN 255 and the tonal spread is 48.05 against Earth's 45.59. + # + # A DEEPER CUT SPENDS SOME OF THAT HEADROOM, AND FAR LESS THAN HALVING THE PIXEL SUGGESTS. + # The saturating term is the gradient PER PIXEL, which would double with the sampling rate only + # if relief were scale-free in amplitude. The blend says otherwise — self-affine at a Hurst + # exponent of 0.875, so RMS slope grows 1.09x per rung. Do not re-derive a 2x from the pixel + # size: what a rung asks of this number is a trim, and only the sphere may decide it. + exaggeration=20.0, + # CUT BUT NOT YET RATIFIED — a ceiling is settled by being served and looked at, which is how + # Earth's z8 was settled and the only way this one will be. z6 came first as the cheapest + # lookable thing rather than as an answer. + # + # THE SOURCE'S HALF IS MEASURED, AND IT IS NOT THE COVERAGE FIGURE IT LOOKS LIKE. A rung + # unlocks one octave of wavelength: z7's is 652-1302 m, of which MOLA's own 463 m grid resolves + # 926-1302 m — so of the 4.28 m RMS z7 adds, 3.25 m is measurement. z8's octave lies wholly + # below MOLA's Nyquist, and the blend's detail there correlates 0.99 with a bilinear upsample of + # its own coarse grid, where Earth's fused field returns 0.75 through the identical test. HRSC + # therefore reaches nearer 5-12% of this grid than the 44% coverage it is published with, and + # the ceiling must not be re-argued from that 44%. + tile_max_zoom=7, + # Nests, where Earth's is empty. A second body pays no relocation cost, so it starts correct. + path_prefix="mars", + # ONE OF THEM, AND THE OTHER FOUR ARE REFUSED FOR THE REASON THE VOCABULARY WAS WRITTEN FOR: + # every source behind lakes, glaciers, sea ice and coastline is an Earth dataset present on this + # box, so left unstated a Mars pass would paint Earth's onto Mars at the same latitudes and raise + # nothing. Naming a layer here is a claim that a MARTIAN producer answers for it — Mars's ice + # grades Viking albedo inside units the USGS mapped, sharing only a name with Earth's snow. + # + # Each of the four still absent is a statement about our DATA rather than about Mars: it has + # seasonal CO2 frost and a cryosphere of its own that no product here describes. + surface_layers=frozenset({"perennial_ice"}), + # ON, AND IT PREDATES THE ICE BY SEVERAL COMMITS. What it buys first is a projection repair: Web + # Mercator carries no data past ~85 degrees and brutally smears the band below it, so these two + # discs would exist as bare relief in the same ramps even with nothing white to paint on them. + # + # Held False until the M2a ramp was ratified, per the field note. What the False cost meanwhile + # was not a hole but something worse — `shade_planet.CAP_RGB`, the flat pale plug the cap + # textures exist to be drawn over, which MapLibre stretched across the pole and which was tested + # on Earth's globe and rejected. Do not reach for False again as a cheap way to skip a render. + renders_polar_caps=True, ) #: Every body the pipeline knows. Keyed by `Body.name`, which a test pins so one planet cannot #: acquire two spellings. -BODIES: dict[str, Body] = {EARTH.name: EARTH} +BODIES: dict[str, Body] = {EARTH.name: EARTH, MARS.name: MARS} def get(name: str) -> Body: @@ -116,6 +298,50 @@ def get(name: str) -> Body: raise KeyError(f"unknown body {name!r}; known bodies are: {known}") from None +def ground_metres_per_mercator_unit(body: Body) -> float: + """How many real ground metres one map unit of this body's Mercator raster is worth. + + NAMED FOR ITS PROJECTION, and there is a second function below named for the other one. An + unqualified `ground_metres_per_map_unit` would read as the general answer and be adopted by the + cap path, which divides by a different sphere and gets a different number. One name per concept. + + THE WHOLE OF WHAT A NON-EARTH BODY COSTS, in one number: because every projection here is + Earth-sphered (module note), a raster's map units are Earth metres whatever planet the + elevations came from. Anything that mixes the two — a hillshade dividing a rise in body metres + by a run in map units, a horizon search, a shadow length — must pass through here or it computes + a slope that is plausible at every latitude and correct at none. It does not raise. + + Earth's is exactly 1.0, so every call site that adopts this keeps its pixels byte-identical — + which is what let it be adopted one stage at a time. + + Composes so the units cancel where it is read: + + ground_metres_per_pixel = body.map_units_per_pixel * ground_metres_per_mercator_unit(body) + """ + return body.ground_radius_m / body.mercator_radius_m + + +def ground_metres_per_aeqd_unit(body: Body) -> float: + """How many real ground metres one map unit of this body's polar-cap AEQD grid is worth. + + THE CAP'S OWN CONVERSION, deliberately not the Mercator one above — see that function for why + the two are named apart. Earth's is 1.0011202 rather than 1.0, so adopting this MOVED Earth's cap + pixels where adopting the Mercator one moved none. + + EXACT FOR A BODY PUBLISHED ON A SPHERE, PARTIAL FOR EARTH — worth stating, because the residual + is larger than the correction. Measured with `pyproj.Geod` on WGS84: the true meridian arc from + 78N to the pole is 1,340,131 m where this AEQD grid calls it 1,334,339 m, a true ratio of + 1.004341. The 1.001120 here therefore closes about a quarter of that gap and leaves three + quarters of sphere-versus-ellipsoid, which nothing in this pipeline models. Mars's DEM is + published on a sphere, so for Mars there is no residual at all and this is simply right. + + Composes the same way, and the units cancel where it is read: + + ground_metres_per_pixel = (2 * grid.edge_m / grid.px) * ground_metres_per_aeqd_unit(body) + """ + return body.ground_radius_m / body.aeqd_radius_m + + def _require_directory_name(stage: str) -> None: """A stage is a single directory name, never a path expression. @@ -144,20 +370,29 @@ def work_dir(body: Body, stage: str) -> Path: own directory makes every one of those sidecars body-specific for free, because they are different files. The identity is carried by location, which costs nothing. - `stage` is a DIRECTORY NAME, never a path expression. A caller that assembled one by - concatenation could otherwise walk out of this body's tree and land in another's — the single - place a mistake here stops being wrong and starts being unrecoverable. + `stage` is a DIRECTORY NAME, never a path expression — enforced by `_require_directory_name`, + which holds the reason. """ _require_directory_name(stage) # An empty prefix collapses, which is what keeps Earth on its historical layout. return paths.DATA / "work" / body.path_prefix / stage -#: The directory the site serves at its URL root. Named because two things need it: where a served -#: asset is WRITTEN (`public_dir`) and what its URL IS — a path under here, minus this prefix. A -#: caller that assembled the URL from a literal instead would be right for Earth, whose segment is -#: empty, and quietly advertise a 404 for every body that nests. -PUBLIC_ROOT = paths.ROOT / "web/public" +def public_root() -> Path: + """The directory the site serves at its URL root. + + Named because two things need it: where a served asset is WRITTEN (`public_dir`) and what its + URL IS — a path under here, minus this prefix. A caller that assembled the URL from a literal + instead would be right for Earth, whose segment is empty, and quietly advertise a 404 for every + body that nests. + + A FUNCTION, NOT A CONSTANT, per `paths` — and this module is where that rule was paid for. As a + constant it bound `paths.ROOT` at import while `work_dir` read `paths.DATA` at call time, so + redirecting both roots isolated the working tree and left the served tree pointing at the real + checkout, writing test output into `web/public/` for the next `astro build` to copy into + `dist/`. + """ + return paths.ROOT / "web/public" def public_dir(body: Body, stage: str) -> Path: @@ -172,4 +407,4 @@ def public_dir(body: Body, stage: str) -> Path: an empty prefix is what stops a second body rewriting it. Mars nests one level in. """ _require_directory_name(stage) - return PUBLIC_ROOT / stage / body.path_prefix + return public_root() / stage / body.path_prefix diff --git a/pipeline/compose/borders_geojson.py b/pipeline/compose/borders_geojson.py index 741e2b0..ed80f28 100644 --- a/pipeline/compose/borders_geojson.py +++ b/pipeline/compose/borders_geojson.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Convert the Natural Earth admin-0 land boundary LINES -> one WGS84 GeoJSON for the globe's vector border overlay (solid international + dashed disputed/LoC, split downstream by FEATURECLA). @@ -21,11 +20,11 @@ import sys from pathlib import Path -from pipeline import paths +from pipeline import bodies, naturalearth -ROOT = paths.ROOT -NE = ROOT / "data/raw/naturalearth" -OUT_DIR = ROOT / "data/work/borders" +#: Vectors are Earth's alone for now, so the body is named rather than threaded — but named through +#: the registry, so the day Mars gains borders they nest instead of overwriting these. +OUT_DIR = bodies.work_dir(bodies.EARTH, "borders") # Land classes the hero style renders (compose/overlay_borders.py); the rest is # cartographic scaffolding we drop. @@ -34,7 +33,7 @@ LAYERS = [ { "name": "boundary_lines.geojson", - "src": NE / "ne_10m_admin_0_boundary_lines_land" / "ne_10m_admin_0_boundary_lines_land.shp", + "src": naturalearth.layer("ne_10m_admin_0_boundary_lines_land"), "where": "FEATURECLA NOT IN ('" + "', '".join(LAND_DROP) + "')", }, ] diff --git a/pipeline/compose/countries_geojson.py b/pipeline/compose/countries_geojson.py index a582e84..0ebfc32 100644 --- a/pipeline/compose/countries_geojson.py +++ b/pipeline/compose/countries_geojson.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Simplify the Natural Earth admin-0 country POLYGONS -> one WGS84 GeoJSON shared by ALL of the globe's country layers: the click-to-fly hit fill, the hover wash, and the hover outline that strokes these very rings as the visible @@ -29,22 +28,23 @@ import sys from pathlib import Path -from pipeline import paths +from pipeline import bodies, naturalearth -ROOT = paths.ROOT -NE = ROOT / "data/raw/naturalearth" -OUT_DIR = ROOT / "data/work/borders" +#: Earth's, and named through the registry for the same reason `borders_geojson` does it. +OUT_DIR = bodies.work_dir(bodies.EARTH, "borders") -SRC = NE / "ne_10m_admin_0_countries" / "ne_10m_admin_0_countries.shp" +SRC = naturalearth.layer("ne_10m_admin_0_countries") OUT = OUT_DIR / "countries.geojson" # Douglas-Peucker tolerance in degrees. 0.002 deg ~= 220 m ~= 0.7 px at z8 on -# the equator (shade_planet.Z8_RES = 305.75 m/px) — sub-pixel where the raster +# the equator (Earth's grid pixel, ~305.75 m/px) — sub-pixel where the raster # is sharpest, so the hover outline tracks the coast it traces. N-S error grows # toward the poles in Mercator pixels (~3 px at 75N: latitude degrees don't # shrink with cos(lat) but Mercator pixels do) — accepted; buying it back costs # ~1 MB more gzip for fjord vertices only a Svalbard hover would notice. -# Pinned relationally against Z8_RES in tests/test_countries_geojson.py. +# Pinned relationally in tests/test_countries_geojson.py against EARTH's grid pixel — Earth's +# specifically, because this pyramid is Earth-only and the tolerance is justified against the +# raster it overlays, not against whatever the sharpest body in the registry cuts to. SIMPLIFY_DEG = 0.002 diff --git a/pipeline/compose/countries_pmtiles.py b/pipeline/compose/countries_pmtiles.py index f4a5f9c..f2ef848 100644 --- a/pipeline/compose/countries_pmtiles.py +++ b/pipeline/compose/countries_pmtiles.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Cut the country polygons into a VECTOR tile pyramid (PMTiles), so the globe addresses country geometry by z/x/y instead of handing MapLibre a 9.4 MB parsed object. @@ -51,20 +50,37 @@ from pathlib import Path from typing import Any -from pipeline import paths +from pipeline import bodies, paths +from pipeline.compose import vector_layers +from pipeline.compose.vector_layers import polygon_parts_of +#: What this cut carries into its derived layers. One key, and it is the join key: the frontend +#: matches it against countries.json by name. See `vector_layers.carried` for why the first is the +#: identity. +CARRIED = ("ADMIN",) + +#: The CHECKOUT, not the data store, and it is read for exactly one thing: the frontend constant +#: this module's max zoom is pinned against. Kept because `web/src/` follows the repo wherever the +#: data store goes — the two roots are different seams. ROOT = paths.ROOT -BORDERS = paths.DATA / "work/borders" -OUT_DIR = paths.DATA / "work/planet_countries" +#: What `borders_geojson` and `countries_geojson` write — the same directory, named the same way, +#: so a writer and its reader cannot drift apart by one editing its own copy of the path. +BORDERS = bodies.work_dir(bodies.EARTH, "borders") +#: One stage name per LAYER, under each body's own prefix — the convention `devStores.archivePath` +#: rests on, so Earth's vector cut and Mars's land in directories that differ only by planet. +OUT_DIR = bodies.work_dir(bodies.EARTH, "planet_vector") SRC = BORDERS / "countries.geojson" OUTLINES = BORDERS / "country_outlines.geojson" HITS = BORDERS / "country_hits.geojson" +#: What the two files above were derived UNDER. Beside them rather than beside the archive, because +#: it answers for the derivation and the archive has its own. +OUTLINES_RECIPE = BORDERS / "country_outlines_params.json" STAGED = OUT_DIR / "countries_staged.gpkg" -OUT = OUT_DIR / "countries.pmtiles" +OUT = OUT_DIR / "vector.pmtiles" -# Layer names inside the archive. The frontend reads these as MapLibre `source-layer` values and -# they are pinned on both sides — web/src/lib/countryTiles.ts is the other end. +# Layer names inside the archive. The frontend reads these as MapLibre `source-layer` values, and +# `tests/test_source_layers.py` compares them against web/src/lib/sourceLayers.ts on every suite. FILL_LAYER = "country_fill" OUTLINE_LAYER = "country_outline" HIT_LAYER = "country_hit" @@ -85,48 +101,12 @@ # translucent fill wash double-paints in the default tile-buffer overlap, worst near the pole. # Baked here because a vector source cannot set it at runtime. BUFFER = 0 -EXTENT = 4096 - - -def polygon_parts_of(geometry: dict[str, Any]) -> list[Any]: - """Every polygon PART of a geometry, whatever container Natural Earth wrapped it in. - - SOLE HOME as of the GeoJSON arm's deletion. A TypeScript twin (`polygonPartsOf`) derived the - same two layers in the browser while that arm existed, so a bug here used to show up as the two - disagreeing. It cannot any more — tests/test_countries_pmtiles.py is the whole guard. - - The GeometryCollection branch is not defensive coding. Natural Earth ships **Greenland** as a - collection of 129 polygons plus one stray LineString; the two inline copies this replaced both - returned nothing for it, so an in-scope country had a fill wash and a click but no hover - outline and no hit targets. `-nlt MULTIPOLYGON` does not fix it upstream — GDAL keeps the - collection precisely because the LineString cannot join a MultiPolygon. - """ - kind = geometry["type"] - if kind == "MultiPolygon": - return list(geometry["coordinates"]) - if kind == "Polygon": - return [geometry["coordinates"]] - if kind == "GeometryCollection": - return [part for member in geometry["geometries"] for part in polygon_parts_of(member)] - return [] +EXTENT = vector_layers.EXTENT def outlines_from(countries: dict[str, Any]) -> dict[str, Any]: """Country rings re-expressed as boundary LINES — outer coasts and inner holes alike.""" - features: list[dict[str, Any]] = [] - for feature in countries["features"]: - admin = feature["properties"].get("ADMIN") - if not isinstance(admin, str): - continue - rings = [ring for part in polygon_parts_of(feature["geometry"]) for ring in part] - if not rings: - continue - features.append({ - "type": "Feature", - "properties": {"ADMIN": admin}, - "geometry": {"type": "MultiLineString", "coordinates": rings}, - }) - return {"type": "FeatureCollection", "features": features} + return vector_layers.outlines_from(countries, CARRIED) def hit_points_from(countries: dict[str, Any]) -> dict[str, Any]: @@ -160,33 +140,21 @@ def hit_points_from(countries: dict[str, Any]) -> dict[str, Any]: def stage_command(source: Path, destination: Path, layer: str, update: bool) -> list[str]: - """One layer into the staging GeoPackage. - - A GPKG intermediate exists because the PMTiles driver **cannot append a layer** to an archive - it already wrote — `-update -append` fails with "cannot be created by the output driver". All - three layers therefore have to reach `ogr2ogr` as one multi-layer dataset. - """ - return [ - "ogr2ogr", "-f", "GPKG", - *(["-update"] if update else []), - str(destination), str(source), - "-nln", layer, - ] + """One layer into the staging GeoPackage. See `vector_layers.stage_command` for why it exists.""" + return vector_layers.stage_command(source, destination, layer, update) def pmtiles_command(source: Path, destination: Path) -> list[str]: """The single conversion, exposed pure for tests. Argument order is [options] DEST SOURCE.""" - return [ - "ogr2ogr", "-f", "PMTiles", - str(destination), str(source), - "-dsco", f"MINZOOM={MIN_ZOOM}", - "-dsco", f"MAXZOOM={MAX_ZOOM}", - "-dsco", "NAME=countries", - "-dsco", f"BUFFER={BUFFER}", - "-dsco", f"EXTENT={EXTENT}", - "-dsco", f"SIMPLIFICATION={SIMPLIFICATION}", - "-dsco", f"SIMPLIFICATION_MAX_ZOOM={SIMPLIFICATION_MAX_ZOOM}", - ] + return vector_layers.pmtiles_command( + source, destination, + name="countries", + min_zoom=MIN_ZOOM, + max_zoom=MAX_ZOOM, + buffer=BUFFER, + simplification=SIMPLIFICATION, + simplification_max_zoom=SIMPLIFICATION_MAX_ZOOM, + ) def recipe() -> dict[str, Any]: @@ -205,6 +173,7 @@ def recipe() -> dict[str, Any]: "simplification_max_zoom": SIMPLIFICATION_MAX_ZOOM, "buffer": BUFFER, "extent": EXTENT, + **vector_layers.seam_recipe(), } @@ -212,16 +181,34 @@ def recipe_path() -> Path: return OUT_DIR / "countries_tiles_params.json" +def derivation_is_stamped() -> bool: + """True when the derived layers on disk were written under the seam settings in force now. + + Its own question, asked in two places: `derive` decides whether to rewrite the GeoJSON, and + `is_fresh` decides whether the archive above it can still be believed. Answering it in only the + first is what let a stale derivation hide behind a fresh archive — `main` returns on `is_fresh` + and never reaches `derive` at all. + """ + return OUTLINES_RECIPE.exists() and json.loads( + OUTLINES_RECIPE.read_text()) == vector_layers.seam_recipe() + + def is_fresh() -> bool: """True when the live archive is current: present, non-empty, stamped newer than the source it - descends from, and cut under the recipe on disk. + descends from and the layers derived from it, and cut under both recipes on disk. The recipe comparison is the half that catches a settings change — a re-cut with a different - SIMPLIFICATION leaves an archive that is newer than its source and would otherwise pass. + SIMPLIFICATION leaves an archive that is newer than its source and would otherwise pass. THE + DERIVATION'S RECIPE IS THE OTHER HALF, and it is not the same question: this module's knobs + move the cut, `vector_layers`' knobs move the geometry that is cut, and an archive can be + current under the first while its outlines were drawn under the second's previous answer. """ if not OUT.exists() or OUT.stat().st_size == 0: return False - if not SRC.exists() or OUT.stat().st_mtime <= SRC.stat().st_mtime: + for path in (SRC, OUTLINES, HITS): + if not path.exists() or OUT.stat().st_mtime <= path.stat().st_mtime: + return False + if not derivation_is_stamped(): return False stamped = recipe_path() if not stamped.exists(): @@ -230,12 +217,21 @@ def is_fresh() -> bool: def derive(force: bool) -> None: - """Write the two derived layers beside their source, skipping when already current.""" + """Write the two derived layers beside their source, skipping when already current. + + THE RECIPE IS CHECKED HERE AND NOT ONLY AT THE CUT, because this is the stage `vector_layers` + writes through. Gating only on the source's mtime made a change to shared code unobservable: + the source does not move when that module's constants do, so the derivation skipped, the cut + re-ran on the stale GeoJSON it had always had, produced a byte-identical archive, and stamped + the NEW recipe over it — consuming the one signal that anything was out of date. A producer + records its recipe beside its output; these two files had none. + """ fresh = ( not force and OUTLINES.exists() and HITS.exists() and min(OUTLINES.stat().st_mtime, HITS.stat().st_mtime) > SRC.stat().st_mtime + and derivation_is_stamped() ) if fresh: print(f"{OUTLINES.name} + {HITS.name} current -> skip") @@ -247,6 +243,9 @@ def derive(force: bool) -> None: temporary.replace(path) # atomic promote print(f"wrote {path.name} ({len(collection['features'])} features, " f"{path.stat().st_size / 1e6:.2f} MB)") + # Stamped AFTER both files exist, so a crash between them leaves the derivation stale rather + # than vouched for — the same order every other stage in this pipeline writes its marker in. + OUTLINES_RECIPE.write_text(json.dumps(vector_layers.seam_recipe(), indent=2) + "\n") def stage() -> None: diff --git a/pipeline/compose/features_geojson.py b/pipeline/compose/features_geojson.py new file mode 100644 index 0000000..eb1ccb1 --- /dev/null +++ b/pipeline/compose/features_geojson.py @@ -0,0 +1,276 @@ +"""Fold the IAU gazetteer's outlines into canonical WGS84 GeoJSON — the named things on Mars. + +The coordinates are DECLARED rather than transformed, for the reason `bodies.py` states once for the +whole pipeline — every projection here is Earth-sphered whatever planet it describes, and PROJ will +not build an operation between two celestial bodies. + +WHAT IS SPECIFIC TO THIS FILE IS THAT DECLARING IS ONLY THE CURE WHILE THE SOURCE IS GEOGRAPHIC. +`download_sim3292` records the other half: `-a_srs`/`-s_srs` used INSTEAD of reprojecting is the +classic bug, and its failure is silent — a projected source read as degrees collapses to garbage +rather than raising. The gazetteer ships `ESRI:104905`, which is a GEOGCS, so its numbers are +already lon/lat and relabelling them is an identity. `assert_geographic_source` is what tells the +two apart, because nothing else here would: the flags look identical in both cases. + +THE FOLD IS `-wrapdateline`, AND `RFC7946=YES` IS NOT A SUBSTITUTE EVEN THOUGH IT LOOKS LIKE ONE. +The source draws a feature crossing the prime meridian by continuing past the seam instead of +wrapping, so its outlines span -180..+360.34 while its centres are uniformly east-positive 0-360 — +the 540 degrees `download_nomenclature` pins. Both flags fold that onto one window and both keep all +1,717 features, but they do not agree: measured, `RFC7946=YES` alone emits **two +GeometryCollections**, which is precisely the container that cost Greenland its hover outline on +Earth, while `-wrapdateline` emits only Polygon and MultiPolygon. Both are passed — the second for +spec-compliant output — and `assert_folded` pins the result, so a GDAL release that reorders them +fails here rather than in a browser. + +NO DOUGLAS-PEUCKER RUNS HERE, WHICH IS WHERE THIS PARTS COMPANY WITH ITS EARTH SIBLING. +`countries_geojson` simplifies because its output is a DISPLAY artifact the frontend fetches and +strokes directly, so its tolerance is argued against z8 relief. Nothing fetches this file: the +geometry a visitor sees arrives as vector tiles, and the names arrive as a search index. Its only +readers are `features_pmtiles` and this project. Simplifying here would permanently discard detail +the tile cut's own per-zoom knobs can spend or keep reversibly, so the only reduction applied is +coordinate QUANTISATION, which is not a vertex decision. + +Outputs (data/work/mars/features/): + features.geojson 1,717 named areal features, folded + feature_lines.geojson 203 named linear features, folded + feature_labels.geojson 1,920 label anchors at the IAU's own centres, folded + features_params.json the recipe: flags, precision, per-layer counts + + python -m pipeline.compose.features_geojson # writes if missing + python -m pipeline.compose.features_geojson --force # regenerate +""" + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any + +from pipeline import bodies +from pipeline.acquire import download_nomenclature + +OUT_DIR = bodies.work_dir(bodies.MARS, "features") + +POLYGONS = OUT_DIR / "features.geojson" +LINES = OUT_DIR / "feature_lines.geojson" +LABELS = OUT_DIR / "feature_labels.geojson" + +#: Which gazetteer layer lands in which file. `labels` is derived from both and has no source of +#: its own, so it is deliberately absent here. +GEOMETRY_OUTPUTS = {"poly": POLYGONS, "line": LINES} + +#: What every layer carries into the tiles. `name` first, because it is the identity every derived +#: layer is keyed on — see `vector_layers.carried`. +#: +#: `origin` is the IAU's etymology and it is the whole content of Mars's detail card, so it rides +#: into the tiles rather than being looked up: measured, the full catalogue WITH origin cuts a 71.8 +#: KB z0 tile, under Earth's own 86 KB, and a tap that needs no second fetch is worth more than the +#: bytes. THE CENTRES ARE NOT HERE ON PURPOSE — they are east-positive 0-360 in the source, and a +#: folded geometry carrying an unfolded longitude property is the mixed-convention bug this module +#: exists to end. They reach `feature_labels.geojson` as geometry, folded, and nothing else. +CARRIED_FIELDS = ("name", "clean_name", "type", "origin", "diameter") + +#: Decimal places of longitude/latitude in the written GeoJSON. 1e-4 deg is 5.9 m on Mars, against +#: a z7 tile pixel of 325.6 ground m — 55x finer than the sharpest pixel the pyramid will ever +#: address, so this cannot be seen. It is a quantisation and not a simplification: it moves vertices +#: without deciding which ones survive, which is the distinction the module note draws. +COORDINATE_PRECISION = 4 + + +def fold_longitude(east_positive: float) -> float: + """An east-positive 0-360 longitude onto the -180..180 window the geometry is folded into. + + The source's centres are uniformly east-positive while its outlines are not, so this is applied + to the CENTRES alone; the geometry's fold is `-wrapdateline`'s job and is a different operation + — it must also SPLIT the features that straddle the seam, which a scalar cannot. + """ + return east_positive - 360.0 if east_positive > 180.0 else east_positive + + +def assert_geographic_source(layer: str) -> None: + """Refuse a source whose `.prj` is PROJECTED, because relabelling one is not the same operation. + + THE FLAGS CANNOT TELL THE TWO APART AND NEITHER CAN A REVIEWER. `-s_srs EPSG:4326` over a + geographic Mars CRS is an identity on the numbers — the cure. The same flag over a PROJECTED + Mars CRS reads metres as degrees, which does not raise: it produces coordinates a few hundred + metres from the prime meridian and a tiler that quietly emits almost nothing. SIM 3292 ships + exactly that shape, Robinson on the Mars sphere in metres, so the hazard is not hypothetical — + it is one directory away, and this module would accept it in silence. + """ + projection = download_nomenclature.layer_path(layer, "prj") + if not projection.exists(): + sys.exit(f"{projection.name} is missing — without it nothing here can tell a geographic " + f"source from a projected one, and the two need opposite treatment.") + declared = projection.read_text(encoding="utf-8", errors="replace").lstrip() + if not declared.startswith("GEOGCS"): + sys.exit(f"{projection.name} does not declare a GEOGCS — this module DECLARES its source's " + f"angles as EPSG:4326, which is an identity only while they are already lon/lat. " + f"A projected source has to be reprojected to longlat FIRST; declaring it instead " + f"reads metres as degrees and fails silently. Got: {declared[:80]!r}") + + +def ogr_command(source: Path, destination: Path) -> list[str]: + """The translation, exposed pure for tests. Argument order is [options] DESTINATION SOURCE.""" + return [ + "ogr2ogr", "-f", "GeoJSON", + "-select", ",".join(CARRIED_FIELDS), + # Declare, never transform — the module note holds the celestial-body refusal this avoids. + "-s_srs", "EPSG:4326", "-t_srs", "EPSG:4326", + "-wrapdateline", + "-lco", "RFC7946=YES", + "-lco", f"COORDINATE_PRECISION={COORDINATE_PRECISION}", + str(destination), str(source), + ] + + +def vertices_of(geometry: dict[str, Any]) -> list[list[float]]: + """Every coordinate pair in a geometry, at whatever nesting depth its type implies.""" + found: list[list[float]] = [] + pending: list[Any] = [geometry["coordinates"]] + while pending: + item = pending.pop() + if item and isinstance(item[0], (int, float)): + found.append(item) + else: + pending.extend(item) + return found + + +def assert_folded(path: Path, expected: int) -> None: + """Assert a written layer is folded, whole, and free of the container that hides polygons. + + THREE THINGS THAT FAIL SEPARATELY. A short count means the translation dropped features; a + vertex outside the window means the fold did not run, and every feature crossing the seam will + be clipped away by the tiler without an error; a GeometryCollection means GDAL folded via + RFC 7946 rather than `-wrapdateline`, and `polygon_parts_of` is the only thing standing between + that and a layer of invisible features — which on Earth it once was not. + """ + collection = json.loads(path.read_text(encoding="utf-8")) + features = collection["features"] + if len(features) != expected: + sys.exit(f"{path.name}: {len(features)} features, expected {expected} — the translation " + f"dropped some. Every coverage figure on record describes the whole catalogue.") + + containers = sorted({feature["geometry"]["type"] for feature in features + if feature.get("geometry")}) + if "GeometryCollection" in containers: + sys.exit(f"{path.name}: GDAL emitted GeometryCollection geometries. That is the RFC 7946 " + f"fold rather than -wrapdateline's, and it is the exact shape that left Greenland " + f"with no outline on Earth. Check the flag order in `ogr_command`.") + + for feature in features: + if not feature.get("geometry"): + continue + for longitude, latitude in (point[:2] for point in vertices_of(feature["geometry"])): + if not (-180.0001 <= longitude <= 180.0001 and -90.0001 <= latitude <= 90.0001): + sys.exit(f"{path.name}: {feature['properties'].get('name')!r} has a vertex at " + f"{longitude}, {latitude} — outside the tile grid's window, so the fold " + f"did not run and the tiler will silently clip this feature away.") + + +def label_points() -> dict[str, Any]: + """One point per named feature, at the IAU's own centre, from BOTH layers. + + THE PUBLISHER'S ANCHOR RATHER THAN A COMPUTED CENTROID. A centroid falls outside a crescent and + has to be bought with a geometry library; more to the point, the gazetteer's centre is the + position the IAU adopted the name AT, which is what a label should sit on. It also gives the + 203 linear features an anchor they have no interior to supply. + + Read through `download_nomenclature.read_attributes`, so "how this DBF is decoded" keeps one + owner — the encoding it fixes is the whole reason that function refuses to use `text=True`. + """ + features: list[dict[str, Any]] = [] + for layer in download_nomenclature.LAYERS: + for row in download_nomenclature.read_attributes(layer): + properties = {key: row[key] for key in CARRIED_FIELDS if row.get(key)} + if "name" not in properties: + continue + features.append({ + "type": "Feature", + "properties": properties, + "geometry": {"type": "Point", "coordinates": [ + round(fold_longitude(float(row["center_lon"])), COORDINATE_PRECISION), + round(float(row["center_lat"]), COORDINATE_PRECISION), + ]}, + }) + return {"type": "FeatureCollection", "features": features} + + +def recipe() -> dict[str, Any]: + """What this fold was made with, recorded beside it — the flags leave no trace in the output.""" + return { + "source": {layer: download_nomenclature.layer_path(layer).name + for layer in download_nomenclature.LAYERS}, + "carried_fields": list(CARRIED_FIELDS), + "coordinate_precision": COORDINATE_PRECISION, + "fold": "-wrapdateline", + "srs": "declared EPSG:4326, not transformed from ESRI:104905", + "features": dict(download_nomenclature.FEATURE_COUNTS), + "simplified": False, + } + + +def recipe_path() -> Path: + return OUT_DIR / "features_params.json" + + +def is_fresh() -> bool: + """True when every output is present, newer than its source, and cut under the recipe on disk. + + The recipe half is what catches a flag change, which moves no mtime and leaves no mark on the + geometry a reader would notice. + """ + newest_source = max(download_nomenclature.layer_path(layer).stat().st_mtime + for layer in download_nomenclature.LAYERS + if download_nomenclature.layer_path(layer).exists()) + for path in (*GEOMETRY_OUTPUTS.values(), LABELS): + if not path.exists() or path.stat().st_size == 0: + return False + if path.stat().st_mtime <= newest_source: + return False + stamped = recipe_path() + return stamped.exists() and json.loads(stamped.read_text()) == recipe() + + +def translate(force: bool) -> None: + """Fold each gazetteer layer, then derive the label anchors from both.""" + if is_fresh() and not force: + print("features are current -> skip (use --force to regenerate)") + return + + for layer, destination in GEOMETRY_OUTPUTS.items(): + source = download_nomenclature.layer_path(layer) + if not source.exists(): + sys.exit(f"missing {source} — run pipeline.acquire.download_nomenclature first") + assert_geographic_source(layer) + temporary = destination.with_suffix(".geojson.tmp") + temporary.unlink(missing_ok=True) + command = ogr_command(source, temporary) + print(" ".join(command), flush=True) + subprocess.run(command, check=True) + assert_folded(temporary, download_nomenclature.FEATURE_COUNTS[layer]) + temporary.replace(destination) # atomic promote, and only after the assertions + print(f"wrote {destination} ({destination.stat().st_size / 1e6:.2f} MB)") + + anchors = label_points() + temporary = LABELS.with_suffix(".geojson.tmp") + temporary.write_text(json.dumps(anchors), encoding="utf-8") + temporary.replace(LABELS) + print(f"wrote {LABELS} ({len(anchors['features'])} anchors, " + f"{LABELS.stat().st_size / 1e6:.2f} MB)") + + recipe_path().write_text(json.dumps(recipe(), indent=2) + "\n", encoding="utf-8") + print(f"wrote {recipe_path()}") + + +def main() -> int: + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--force", action="store_true", help="regenerate even if present") + args = parser.parse_args() + OUT_DIR.mkdir(parents=True, exist_ok=True) + translate(args.force) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/pipeline/compose/features_pmtiles.py b/pipeline/compose/features_pmtiles.py new file mode 100644 index 0000000..28d60c4 --- /dev/null +++ b/pipeline/compose/features_pmtiles.py @@ -0,0 +1,236 @@ +"""Cut Mars's named features into a VECTOR tile pyramid (PMTiles) — four layers, one archive. + +WHY ALL 1,717 AND NOT A CUTOFF. The scout measured that the ~200 largest features carry 99.2% of +the union coverage, and that number is real but it answers only "if a pin lands at a uniformly +random point, is something under it". It is AREA-WEIGHTED, so it is decided entirely by a handful of +enormous terrae, and rank 200 turns out to be a **394 km diameter floor** — which deletes Gale (154 +km) and Jezero (47.5 km), the two most recognisable place names on the planet, because between them +they are a hundredth of a percent of the sphere. The deeper mismatch is that a whole-sphere +statistic was being aimed at a ZOOM-ADDRESSED layer: 1,198 of 1,717 features fall between 5 km and +one z7 tile, which is precisely what fills the screen once a visitor is zoomed in and precisely what +the metric cannot weigh. Measured, the cutoff buys 54 KB on the cold window (z0: 71.8 KB whole +against 17.9 KB cut) — and the whole catalogue is already lighter at z0 than Earth's countries are. +So the archive carries everything and declutter is a runtime filter on `diameter`, which is also the +reversible direction: a style filter can always narrow, an archive cannot widen without a re-cut. + +SIMPLIFICATION IS LOAD-BEARING HERE IN A WAY IT IS NOT ON EARTH — IT IS THE DIFFERENCE BETWEEN AN +ARCHIVE AND A CRASH. GDAL 3.12.2's MVT writer **segfaults** on the gazetteer polygons when no +simplification is applied, whether the option is omitted or set to 0. Isolated: it is not the driver +(Earth's countries cut clean through the same code path with none), not this pipeline's coordinate +quantisation (the unquantised geometry crashes identically), not the five circumpolar features (they +crash alone, and so does everything with them removed), and not the linear features (they cut fine). +`vector_layers.pmtiles_command` therefore takes both knobs as required keyword arguments — a caller +that forgets one fails to construct rather than dying at 139 with no output. + +WHY FOUR LAYERS. `feature_outline` carries the polygon rings as LINES rather than letting a `line` +layer stroke the polygons, for the phantom-edge reason `vector_layers.outlines_from` holds. +`feature_line` is the 203 valles, rupes and fossae, which have no interior to fill and so cannot ride +in `feature_fill`. `feature_label` is the IAU's own centres for BOTH source layers — an anchor a +polygon centroid cannot supply for a crescent and a line cannot supply at all. + +A HANDFUL OF POLAR ANCHORS DO NOT REACH THE GRID, AND THAT IS THE PROJECTION RATHER THAN A FAULT. +Web Mercator stops at ±85.0511°, and the northernmost features' adopted centres lie past it, so +their label anchors are clipped away while their geometry still arrives clipped at the limit. Those +features stay tappable and go unlabelled, which is the right way round — and the polar caps are +drawn over that band regardless. + + python -m pipeline.compose.features_pmtiles # writes if missing/stale + python -m pipeline.compose.features_pmtiles --force # re-cut +""" + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any + +from pipeline import bodies +from pipeline.compose import features_geojson, vector_layers + +#: One stage name per LAYER, under each body's own prefix — the convention `devStores.archivePath` +#: rests on, so Mars's vector cut and Earth's land in directories that differ only by planet. +OUT_DIR = bodies.work_dir(bodies.MARS, "planet_vector") + +#: Derived, not acquired, so it lands beside the polygons it is derived FROM rather than beside the +#: archive it feeds — `derive` compares the two mtimes and a split across directories would not +#: change that, but the geojsons are one stage's output and belong together. +OUTLINES = features_geojson.OUT_DIR / "feature_outlines.geojson" +#: What the file above was derived UNDER — see `countries_pmtiles.OUTLINES_RECIPE`. +OUTLINES_RECIPE = features_geojson.OUT_DIR / "feature_outlines_params.json" +STAGED = OUT_DIR / "features_staged.gpkg" +OUT = OUT_DIR / "vector.pmtiles" + +# Layer names inside the archive. The frontend reads these as MapLibre `source-layer` values, and a +# mismatch renders the layer empty with no error — so both ends are pinned by test. +FILL_LAYER = "feature_fill" +OUTLINE_LAYER = "feature_outline" +LINE_LAYER = "feature_line" +LABEL_LAYER = "feature_label" + +MIN_ZOOM = 0 +#: READ FROM THE BODY, NEVER RESTATED. The relief pyramid's ceiling is what the outline is judged +#: against, so vector detail has to reach exactly where the raster does; a literal here would let +#: the two drift with nothing going red until someone compared tiles by eye. +MAX_ZOOM = bodies.MARS.tile_max_zoom + +#: Tile-space units, of `vector_layers.EXTENT`. MEASURED ON THIS CATALOGUE AND FOUND NEARLY INERT: +#: z0 moves 81.6 -> 75.7 -> 70.3 KB across 1 / 2 / 4, an 11 KB spread where Earth's coastlines moved +#: 160 KB. Craters are convex and short-perimeter, so Douglas-Peucker has little to take. Earth's +#: value is kept because no Mars measurement argues for another, not because it was inherited — and +#: the sidecar records it, so a re-judge costs one re-cut. +SIMPLIFICATION = 2.0 +#: Deliberately far below SIMPLIFICATION, and vindicated by the same sweep: z7 came out +#: BYTE-IDENTICAL at 4157.2 KB across every arm, so overview weight is tunable without touching the +#: zoom the outline is actually judged at. That is the entire reason there are two knobs. +SIMPLIFICATION_MAX_ZOOM = 0.5 + +#: The translucent hover wash double-paints in the default tile-buffer overlap. A vector source +#: cannot set this at runtime, so losing it here is unrecoverable in the browser. +BUFFER = 0 + + +def sources() -> dict[str, Path]: + """Each archive layer and the GeoJSON it is cut from, in staging order. + + `feature_outline` is derived rather than acquired, which is why it is the one entry whose path + this module owns instead of reading from `features_geojson`. + """ + return { + FILL_LAYER: features_geojson.POLYGONS, + OUTLINE_LAYER: OUTLINES, + LINE_LAYER: features_geojson.LINES, + LABEL_LAYER: features_geojson.LABELS, + } + + +def pmtiles_command(source: Path, destination: Path) -> list[str]: + """The single conversion. Argument order is [options] DESTINATION SOURCE.""" + return vector_layers.pmtiles_command( + source, destination, + name="features", + min_zoom=MIN_ZOOM, + max_zoom=MAX_ZOOM, + buffer=BUFFER, + simplification=SIMPLIFICATION, + simplification_max_zoom=SIMPLIFICATION_MAX_ZOOM, + ) + + +def recipe() -> dict[str, Any]: + """What this cut was made with, recorded beside it — the archive's name carries none of it.""" + return { + "layers": list(sources()), + "min_zoom": MIN_ZOOM, + "max_zoom": MAX_ZOOM, + "simplification": SIMPLIFICATION, + "simplification_max_zoom": SIMPLIFICATION_MAX_ZOOM, + "buffer": BUFFER, + "extent": vector_layers.EXTENT, + **vector_layers.seam_recipe(), + } + + +def recipe_path() -> Path: + return OUT_DIR / "features_tiles_params.json" + + +def derivation_is_stamped() -> bool: + """True when the outlines on disk were written under the seam settings in force now. + + Asked by both `derive` and `is_fresh` — see `countries_pmtiles.derivation_is_stamped` for what + asking it in only the first one costs. + """ + return OUTLINES_RECIPE.exists() and json.loads( + OUTLINES_RECIPE.read_text(encoding="utf-8")) == vector_layers.seam_recipe() + + +def is_fresh() -> bool: + """True when the archive is present, newer than every layer it was cut from, and stamped with + both recipes on disk — the half that catches a knob change, which moves no mtime. + + BOTH, because they answer for different stages: this module's recipe describes the cut, and + `vector_layers`' describes the geometry handed to it. + """ + if not OUT.exists() or OUT.stat().st_size == 0: + return False + for path in sources().values(): + if not path.exists() or OUT.stat().st_mtime <= path.stat().st_mtime: + return False + if not derivation_is_stamped(): + return False + stamped = recipe_path() + return stamped.exists() and json.loads(stamped.read_text()) == recipe() + + +def derive(force: bool) -> None: + """Write the outline layer beside its source, skipping when already current. + + Gated on the seam recipe as well as the source's mtime — see the note on + `countries_pmtiles.derive`, which is where this was found. Mars had the identical hole and + escaped it only by ordering: its outlines happened to be derived after the seam rule landed, + so nothing here was ever observed to be wrong. The next change to `vector_layers` would have + been the one that skipped. + """ + polygons = features_geojson.POLYGONS + if (not force and OUTLINES.exists() and derivation_is_stamped() + and OUTLINES.stat().st_mtime > polygons.stat().st_mtime): + print(f"{OUTLINES.name} current -> skip") + return + collection = json.loads(polygons.read_text(encoding="utf-8")) + outlines = vector_layers.outlines_from(collection, features_geojson.CARRIED_FIELDS) + temporary = OUTLINES.with_suffix(".geojson.tmp") + temporary.write_text(json.dumps(outlines), encoding="utf-8") + temporary.replace(OUTLINES) # atomic promote + print(f"wrote {OUTLINES.name} ({len(outlines['features'])} features, " + f"{OUTLINES.stat().st_size / 1e6:.2f} MB)") + # After the promote, so a crash leaves the derivation stale rather than vouched for. + OUTLINES_RECIPE.write_text(json.dumps(vector_layers.seam_recipe(), indent=2) + "\n", + encoding="utf-8") + + +def stage() -> None: + """Every layer into one GeoPackage — see `vector_layers.stage_command` for why this exists.""" + STAGED.unlink(missing_ok=True) + for index, (layer, source) in enumerate(sources().items()): + command = vector_layers.stage_command(source, STAGED, layer, update=index > 0) + print(" ".join(command), flush=True) + subprocess.run(command, check=True) + + +def cut() -> None: + temporary = OUT.with_suffix(".pmtiles.tmp") + temporary.unlink(missing_ok=True) + command = pmtiles_command(STAGED, temporary) + print(" ".join(command), flush=True) + subprocess.run(command, check=True) + temporary.replace(OUT) # atomic promote + recipe_path().write_text(json.dumps(recipe(), indent=2) + "\n", encoding="utf-8") + print(f"wrote {OUT} ({OUT.stat().st_size / 1e6:.1f} MB)") + + +def main() -> int: + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--force", action="store_true", help="re-cut even if current") + args = parser.parse_args() + + missing = [path for layer, path in sources().items() + if layer != OUTLINE_LAYER and not path.exists()] + if missing: + sys.exit(f"missing {', '.join(path.name for path in missing)} — " + f"run pipeline.compose.features_geojson first") + OUT_DIR.mkdir(parents=True, exist_ok=True) + + if is_fresh() and not args.force: + print(f"{OUT.name} is current -> skip (use --force to re-cut)") + return 0 + + derive(args.force) + stage() + cut() + STAGED.unlink(missing_ok=True) # a large intermediate with no reader once the archive exists + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/pipeline/compose/gen_borders.py b/pipeline/compose/gen_borders.py index 9d972db..17b57b7 100644 --- a/pipeline/compose/gen_borders.py +++ b/pipeline/compose/gen_borders.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Batch-generate standalone transparent border layers + gallery variants. The gallery's border toggle overlays a per-country transparent PNG of the @@ -22,21 +21,32 @@ from pathlib import Path import cairo - -ROOT = Path(__file__).resolve().parents[2] - -import pyproj # noqa: E402 -from pipeline.frame.country_config import (build_scope, load_config, # noqa: E402 - load_ne_rows) -from pipeline.compose.overlay_borders import (DASHED_CLASSES, DISPUTED_STYLE, # noqa: E402 - LAND_STYLE, MARITIME_STYLE, SOLID_CLASSES, - frame_bbox_lonlat, read_lines, render_mapping, - stroke) - -NE = ROOT / "data/raw/naturalearth" -BORDERS = ROOT / "blender/renders/borders" -VARIANTS = ROOT / "blender/renders/variants" -WORK = ROOT / "data/work" +import pyproj + +from pipeline import naturalearth, paths +from pipeline.compose.overlay_borders import ( + DASHED_CLASSES, + DISPUTED_STYLE, + LAND_STYLE, + MARITIME_STYLE, + SOLID_CLASSES, + frame_bbox_lonlat, + read_lines, + render_mapping, + stroke, +) +from pipeline.frame.country_config import ( + build_scope, + country_render_dir, + country_work_dir, + load_config, + load_ne_rows, +) + +# The rendered layers live in the CHECKOUT beside the heroes they overlay; the per-country inputs +# they read live in the DATA store, reached through `country_work_dir`. Two roots, two seams. +BORDERS = paths.ROOT / "blender/renders/borders" +VARIANTS = paths.ROOT / "blender/renders/variants" # The globe's hero panel is the only surface that overlays this layer, and it declares a 420 px # slot — so the rungs that matter are 420 px at DPR 1/2/3. 1920 alone left that panel pulling an # 85 kB border PNG on top of a 48 kB hero, i.e. the border became the heavier half of the card @@ -46,13 +56,12 @@ TARGETS = (640, 960, 1280, 1920) # each country's native long edge added too -def ne(name: str) -> Path: - return NE / name / f"{name}.shp" def find_render_dir(slug: str) -> Path | None: """work//render, else the first render* dir with the prep outputs.""" - cand = [WORK / slug / "render", *sorted((WORK / slug).glob("render*"))] + work = country_work_dir(slug) + cand = [country_render_dir(slug), *sorted(work.glob("render*"))] for candidate in cand: if (candidate / "frame.json").exists() and (candidate / "heightfield_aea.tif").exists(): return candidate @@ -67,11 +76,11 @@ def draw_layer(rdir: Path): fwd = pyproj.Transformer.from_crs("EPSG:4326", crs, always_xy=True) bbox = frame_bbox_lonlat(bounds, crs) - land = list(read_lines(ne("ne_10m_admin_0_boundary_lines_land"), bbox)) + land = list(read_lines(naturalearth.layer("ne_10m_admin_0_boundary_lines_land"), bbox)) solid = [(record, parts) for record, parts in land if record.as_dict()["FEATURECLA"] in SOLID_CLASSES] dashed = [(record, parts) for record, parts in land if record.as_dict()["FEATURECLA"] in DASHED_CLASSES] maritime = list(read_lines( - ne("ne_10m_admin_0_boundary_lines_maritime_indicator"), bbox)) + naturalearth.layer("ne_10m_admin_0_boundary_lines_maritime_indicator"), bbox)) surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) ctx = cairo.Context(surface) diff --git a/pipeline/compose/gen_spotlight.py b/pipeline/compose/gen_spotlight.py index aa5680f..8607485 100644 --- a/pipeline/compose/gen_spotlight.py +++ b/pipeline/compose/gen_spotlight.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Per-country subject-spotlight overlay for the hero gallery (toggle asset). Parallel to gen_borders.py: emits a standalone transparent overlay that, laid @@ -55,25 +54,38 @@ from rasterio.errors import NotGeoreferencedWarning from rasterio.transform import from_origin from rasterio.warp import reproject -from scipy.ndimage import binary_fill_holes, distance_transform_edt, gaussian_filter, label - +from scipy.ndimage import ( + binary_fill_holes, + distance_transform_edt, + gaussian_filter, + label, +) + +from pipeline import naturalearth, paths from pipeline.compose.overlay_borders import render_mapping +from pipeline.frame.country_config import country_render_dir warnings.filterwarnings("ignore", category=NotGeoreferencedWarning) # hero PNGs -ROOT = Path(__file__).resolve().parents[2] -HEROES = ROOT / "blender/renders/heroes" -VARIANTS = ROOT / "blender/renders/variants" -NE_COUNTRIES = (ROOT / "data/raw/naturalearth/ne_10m_admin_0_countries" - / "ne_10m_admin_0_countries.shp") -WORK = ROOT / "data/work" +# Both roots, as in `gen_borders`: the shapefile and the per-country work tree move with +# `MAPS_DATA`, the rendered layers stay in the checkout beside the heroes they dim. +HEROES = paths.ROOT / "blender/renders/heroes" +VARIANTS = paths.ROOT / "blender/renders/variants" +NE_COUNTRIES = naturalearth.layer("ne_10m_admin_0_countries") PLANE_WIDTH_UNITS = 2.0 # The ladder is IMPORTED, not restated. The gallery layers this overlay directly on the hero and # gives both the same `sizes`, so a rung the overlay lacks makes the browser pull a larger file for # the top layer than for the one underneath it. This used to be a copied tuple kept honest by a # test — which worked for a fixed ladder, but the portrait fill rung is computed per hero from its # aspect, and there is no way to copy a function and stay in step. One definition, no drift. -from pipeline.compose.hero_variants import TARGETS, rungs_for # noqa: E402 (re-exported) +# +# `TARGETS` IS RE-EXPORTED and F401 cannot see that, which is why the suppression is here rather +# than left to be rediscovered. Nothing in this module reads the name; `test_hero_variants.py` +# reaches it as `gen_spotlight.TARGETS`, through the module object, to pin this ladder against the +# hero's and to enumerate all three ladders the pipeline produces. An attribute read is invisible +# to an unused-import check, so the automatic fix deletes the name and takes the guard with it. +from pipeline.compose.hero_variants import TARGETS, rungs_for # noqa: E402, F401 + # Unchanged at q88 by the quality pass, and provably so: build_overlay sets # overlay_alpha to 0 across the subject, so these pixels only ever cover the dimmed surroundings. WEBP_QUALITY = 88 @@ -211,7 +223,7 @@ def load_parts(shp_path, bbox, want_slug, exclude=False): def render_one(slug, dim, desat, force, outline_div=OUTLINE_DIV_DEFAULT, halo=HALO_ALPHA_DEFAULT): hero_path = HEROES / f"{slug}.png" - render_dir = WORK / slug / "render" + render_dir = country_render_dir(slug) ocean_path = render_dir / "oceanmask_aea.tif" heightfield_path = render_dir / "heightfield_aea.tif" if not hero_path.exists() or not ocean_path.exists() or not heightfield_path.exists(): diff --git a/pipeline/compose/hero_variants.py b/pipeline/compose/hero_variants.py index 49fc0d3..bbbb7d4 100644 --- a/pipeline/compose/hero_variants.py +++ b/pipeline/compose/hero_variants.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Responsive WebP variants of the hero renders for the Tier-1 gallery srcset. Each 8K hero PNG (~50 MB, RGBA) is far too heavy to serve directly. This emits @@ -43,11 +42,15 @@ import rasterio from rasterio.errors import NotGeoreferencedWarning +from pipeline import paths + warnings.filterwarnings("ignore", category=NotGeoreferencedWarning) # PNGs -ROOT = Path(__file__).resolve().parents[2] -HEROES = ROOT / "blender/renders/heroes" -VARIANTS = ROOT / "blender/renders/variants" +# Checkout-rooted on purpose: hero products live in the repo tree, not in the relocatable data +# store. Taken from `paths` rather than re-derived, so this module cannot grow its own idea of where +# the root is — which is how three of its siblings acquired a data path that ignored `MAPS_DATA`. +HEROES = paths.ROOT / "blender/renders/heroes" +VARIANTS = paths.ROOT / "blender/renders/variants" RECIPE = VARIANTS / "hero_variants_recipe.json" # The srcset ladder, chosen against MEASURED layout rather than viewport guesses: the gallery is diff --git a/pipeline/compose/overlay_borders.py b/pipeline/compose/overlay_borders.py index a815ec7..a4dc8db 100644 --- a/pipeline/compose/overlay_borders.py +++ b/pipeline/compose/overlay_borders.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Overlay Natural Earth vectors on a rendered hero image. Modes: @@ -32,8 +31,10 @@ --render blender/renders/india_hero_8k_v3_water.png \ --heightfield data/work/india/render/heightfield_aea.tif \ --mask data/work/india/render/oceanmask_aea.tif \ - --ne-dir data/raw/naturalearth \ --outdir data/work/india/render/overlay + +`--ne-dir` defaults to the store's own vector directory and is only worth naming when pointing at +a different Natural Earth release. """ import argparse @@ -47,6 +48,8 @@ import rasterio import shapefile +from pipeline import naturalearth + PLANE_WIDTH_UNITS = 2.0 # Art levers (see ART.md): width in render pixels at 8K, dash = [on, off] px. @@ -286,7 +289,8 @@ def main(): ap.add_argument("--render", type=Path, required=True) ap.add_argument("--heightfield", type=Path, required=True) ap.add_argument("--mask", type=Path) - ap.add_argument("--ne-dir", type=Path, required=True) + ap.add_argument("--ne-dir", type=Path, default=naturalearth.DIR, + help="a different Natural Earth release; defaults to the store's own") ap.add_argument("--outdir", type=Path, required=True) args = ap.parse_args() if args.mode == "oracle" and not args.mask: @@ -300,13 +304,17 @@ def main(): to_px, m_per_px, crs, bounds = render_mapping(args.heightfield, width, height) fwd = pyproj.Transformer.from_crs("EPSG:4326", crs, always_xy=True) bbox = frame_bbox_lonlat(bounds, crs) - print(f"frame lon/lat bbox (padded): {['%.1f' % value for value in bbox]}", flush=True) + print(f"frame lon/lat bbox (padded): {[f'{value:.1f}' for value in bbox]}", flush=True) overlay = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) octx = cairo.Context(overlay) - def ne(name): - return args.ne_dir / name / f"{name}.shp" + def ne(name: str) -> Path: + """This entry point takes its directory from the command line, so the ROOT is the caller's + and only the naming RULE is shared. Both travel through `naturalearth.layer`, which also + refuses a layer the acquirer never fetches — the failure this used to defer to a + missing-file error several frames later.""" + return naturalearth.layer(name, directory=args.ne_dir) if args.mode == "oracle": count = stroke(octx, fwd, to_px, diff --git a/pipeline/compose/vector_layers.py b/pipeline/compose/vector_layers.py new file mode 100644 index 0000000..a519d91 --- /dev/null +++ b/pipeline/compose/vector_layers.py @@ -0,0 +1,249 @@ +"""The vector-tile machinery both bodies' pyramids need, owned once. + +WHAT LIVES HERE IS WHAT DRIFTS SILENTLY IF IT DOES NOT. Earth cuts countries and Mars cuts named +features; the two differ in their sources, their property sets and their zoom ceilings, but the +geometry walk, the staging trick and the conversion's option set are the same code answering the +same questions. Copied, a fix to either copy leaves the other wrong and nothing goes red — the +trigger this repo uses for deciding a thing needs an owner. + +WHAT DELIBERATELY DOES NOT LIVE HERE is anything a body ASSERTS about its own output: the recipe +sidecar, the freshness rule, the simplification constants and their justifications. Those read as +shared and are not — a tolerance is argued against the raster of one particular body, and a recipe +is a producer's claim about what it emitted. Hoisting them would make one body's measurement look +like a law. +""" + +import itertools +from pathlib import Path +from typing import Any + +#: MVT tile-space units. Not a knob either body has had reason to move, but it is part of the +#: conversion's contract, so it is named rather than spelled into a command string. +EXTENT = 4096 + +#: How near ±180 an edge must sit for `seam_closures` to consider it at all. GENEROUS ON PURPOSE, +#: because it only decides what gets ASKED the twin question and never what gets dropped — a wide +#: band costs a comparison and a narrow one silently exempts a publisher who did not snap its cut to +#: the meridian. USGS's Arcadia Planitia closes at 179.711 and 180.091, which is what sets the floor. +SEAM_BAND_DEGREES = 1.0 + +#: How closely two candidate edges' latitude spans must agree to be called the same cut seen from +#: both sides. Tight, because it is a COINCIDENCE test rather than a magnitude one: a real boundary +#: is not mirrored across the meridian at both ends to within a twentieth of a degree by accident. +SEAM_TWIN_LATITUDE_EPSILON = 0.05 + + +def seam_recipe() -> dict[str, Any]: + """The seam drop's settings, for a body's recipe sidecar to carry. + + IT IS SHARED CODE MAKING A CHOICE, WHICH IS THE ONE THING A PER-BODY RECIPE CANNOT NOTICE. The + outlines are gated on their SOURCE's mtime, and a source does not move when this module's + constants do — so without the sidecar, retuning either knob leaves every archive on disk exactly + as it was, reading fresh. Returned as a dict rather than spelled into each recipe so a third knob + reaches both bodies by existing. + """ + return { + "seam_band_degrees": SEAM_BAND_DEGREES, + "seam_twin_latitude_epsilon": SEAM_TWIN_LATITUDE_EPSILON, + } + + +def polygon_parts_of(geometry: dict[str, Any]) -> list[Any]: + """Every polygon PART of a geometry, whatever container the publisher wrapped it in. + + The GeometryCollection branch is not defensive coding. Natural Earth ships **Greenland** as a + collection of 129 polygons plus one stray LineString; the two inline copies this replaced both + returned nothing for it, so an in-scope country had a fill wash and a click but no hover + outline and no hit targets. `-nlt MULTIPOLYGON` does not fix it upstream — GDAL keeps the + collection precisely because the LineString cannot join a MultiPolygon. + + Mars's gazetteer ships no collections today, which is exactly why the branch stays owned here + rather than being trimmed to what the newer caller happens to need: the case is a PUBLISHER's + habit, and the second publisher has not promised anything. + """ + kind = geometry["type"] + if kind == "MultiPolygon": + return list(geometry["coordinates"]) + if kind == "Polygon": + return [geometry["coordinates"]] + if kind == "GeometryCollection": + return [part for member in geometry["geometries"] for part in polygon_parts_of(member)] + return [] + + +def carried(properties: dict[str, Any], keys: tuple[str, ...]) -> dict[str, Any] | None: + """The subset of `properties` a derived layer carries, or None if the first key is unusable. + + THE FIRST KEY IS THE IDENTITY and the rest are payload — a feature with no name cannot be + hovered as one thing, labelled, or joined to anything, so it is dropped rather than carried + anonymously. The remaining keys are copied when present and simply absent when not, because a + missing `origin` is a thinner card and a missing name is a broken layer. + """ + identity = properties.get(keys[0]) + if not isinstance(identity, str) or not identity: + return None + return {key: properties[key] for key in keys if properties.get(key) is not None} + + +def seam_closures(rings: list[Any]) -> set[tuple[int, int]]: + """`(ring index, edge index)` for every edge that is a polygon's cut at ±180 rather than boundary. + + A FEATURE THAT STRADDLES THE ANTIMERIDIAN ARRIVES AS TWO POLYGONS, AND EACH HALF HAS TO CLOSE + ITSELF ALONG THE CUT. Stroked, those two closures draw one straight line down the meridian + through the middle of the feature — Arcadia Planitia's runs 20.8° of latitude. It is the same + phenomenon `outlines_from` already exists for, one stage earlier: there the cut is a tile + boundary, here it is the fold, and neither is anything a reader should see. + + THE TEST IS A TWIN, NOT A THRESHOLD, and that is the whole difficulty. Proximity to ±180 cannot + decide it — Fiji contributes 388 edges inside the band and every one is real coast — and neither + can length, because **Terra Cimmeria's published eastern boundary genuinely follows the meridian + for 57°**, abutting Terra Sirenum's own 45° western boundary on the other side. Those two are + single unsplit polygons and their long meridian edges are the source's real answer. What a cut + has and a boundary does not is a counterpart: the same edge, in the same feature, on the other + side of the seam, spanning the same latitudes because both halves were closed along one line. + + IT MUST ALSO SURVIVE A PUBLISHER THAT DID NOT SNAP ITS OWN CUT. `-wrapdateline` lands both ends + on exactly ±180, but the USGS gazetteer ships Arcadia pre-split and closes it at 179.711 and + 180.091 — so an exact-meridian test would fix Earth completely, look correct, and leave the + worst Mars case untouched. The band is wide and the coincidence test does the discriminating. + + THE KNOWN FALSE POSITIVE is a feature with lobes on both sides whose seam-adjacent edges span + the same latitudes by chance, which is indistinguishable from a cut and is dropped. It has a + test; it appears in neither catalogue. Winding cannot rescue it — publishers do not guarantee + ring orientation — and every threshold that would is one that deletes Terra Cimmeria. + """ + candidates: list[tuple[int, int, bool, float, float]] = [] + for ring_index, ring in enumerate(rings): + for edge_index, (start, end) in enumerate(itertools.pairwise(ring)): + if min(abs(start[0]), abs(end[0])) < 180.0 - SEAM_BAND_DEGREES: + continue + if start[0] * end[0] <= 0: # an edge ACROSS the meridian is not one along it + continue + low, high = sorted((start[1], end[1])) + candidates.append((ring_index, edge_index, start[0] > 0, low, high)) + + closures: set[tuple[int, int]] = set() + for ring_index, edge_index, east, low, high in candidates: + # A degenerate span would twin with any other degenerate span at the same latitude, which is + # every pair of short opposite-side edges rather than a cut. + if high - low <= SEAM_TWIN_LATITUDE_EPSILON: + continue + if any(other_east is not east + and abs(low - other_low) < SEAM_TWIN_LATITUDE_EPSILON + and abs(high - other_high) < SEAM_TWIN_LATITUDE_EPSILON + for _, _, other_east, other_low, other_high in candidates): + closures.add((ring_index, edge_index)) + return closures + + +def arcs_without(ring: list[Any], dropped: set[int]) -> list[list[Any]]: + """`ring` as the open arcs left when `dropped` edge indices are removed. + + A ring is closed, so removing an edge in the MIDDLE of the coordinate list has to rejoin the + tail to the head rather than leave two arcs that meet at a point nothing draws. Walking from + just past a cut is what makes that fall out; without it the seam looks fixed and the feature's + outline silently gains a gap at wherever the publisher happened to start the ring. + """ + edge_count = len(ring) - 1 + if not dropped or edge_count < 1: + return [ring] + order = list(range(edge_count)) + if ring[0] == ring[-1] and edge_count > 1: + resume = (max(dropped) + 1) % edge_count + order = [(resume + step) % edge_count for step in range(edge_count)] + arcs: list[list[Any]] = [] + current: list[Any] = [] + for index in order: + if index in dropped: + if len(current) > 1: + arcs.append(current) + current = [] + continue + if not current: + current = [ring[index]] + current.append(ring[index + 1]) + if len(current) > 1: + arcs.append(current) + return arcs + + +def outlines_from(collection: dict[str, Any], keys: tuple[str, ...]) -> dict[str, Any]: + """Polygon rings re-expressed as boundary LINES — outer edges and inner holes alike. + + WHY A LINE LAYER RATHER THAN STROKING THE POLYGONS. Clipping a line trims it; clipping a + polygon closes the ring along the cut, and a `line` layer strokes that phantom edge — the stray + gold meridian this project already fixed once at runtime. The fix cannot be made in the browser + for a vector source, so it is made here. + + The antimeridian is the same cut made earlier and by someone else, so `seam_closures` drops it + here too. A feature whose every ring is one closure would emit no line at all and is skipped, + which is why the emptiness test moved below the drop. + """ + features: list[dict[str, Any]] = [] + for feature in collection["features"]: + properties = carried(feature["properties"], keys) + if properties is None or not feature.get("geometry"): + continue + rings = [ring for part in polygon_parts_of(feature["geometry"]) for ring in part] + closures = seam_closures(rings) + lines = [arc + for ring_index, ring in enumerate(rings) + for arc in arcs_without( + ring, {edge for cut_ring, edge in closures if cut_ring == ring_index})] + if not lines: + continue + features.append({ + "type": "Feature", + "properties": properties, + "geometry": {"type": "MultiLineString", "coordinates": lines}, + }) + return {"type": "FeatureCollection", "features": features} + + +def stage_command(source: Path, destination: Path, layer: str, update: bool) -> list[str]: + """One layer into a staging GeoPackage. + + A GPKG intermediate exists because the PMTiles driver **cannot append a layer** to an archive + it already wrote — `-update -append` fails with "cannot be created by the output driver". Every + layer therefore has to reach `ogr2ogr` as one multi-layer dataset. + """ + return [ + "ogr2ogr", "-f", "GPKG", + *(["-update"] if update else []), + str(destination), str(source), + "-nln", layer, + ] + + +def pmtiles_command( + source: Path, + destination: Path, + *, + name: str, + min_zoom: int, + max_zoom: int, + buffer: int, + simplification: float, + simplification_max_zoom: float, +) -> list[str]: + """The staged dataset to a PMTiles pyramid. Argument order is [options] DESTINATION SOURCE. + + THAT ORDER READS BACKWARDS AND HAS BEEN GOT WRONG BEFORE, which is most of why this is one + function rather than a string in each caller. + + GDAL SIMPLIFIES NOTHING BY DEFAULT, and that is not a safe default for either body: on Earth's + countries it measured a 4.3x weight regression against a tolerance-3 tiler, invisible unless the + tiles are weighed. Both knobs are therefore required arguments with no defaults — a caller that + forgets one fails to construct rather than quietly shipping the unsimplified cut. + """ + return [ + "ogr2ogr", "-f", "PMTiles", + str(destination), str(source), + "-dsco", f"MINZOOM={min_zoom}", + "-dsco", f"MAXZOOM={max_zoom}", + "-dsco", f"NAME={name}", + "-dsco", f"BUFFER={buffer}", + "-dsco", f"EXTENT={EXTENT}", + "-dsco", f"SIMPLIFICATION={simplification}", + "-dsco", f"SIMPLIFICATION_MAX_ZOOM={simplification_max_zoom}", + ] diff --git a/pipeline/fetch.py b/pipeline/fetch.py new file mode 100644 index 0000000..9e4e4dc --- /dev/null +++ b/pipeline/fetch.py @@ -0,0 +1,91 @@ +"""The pipeline's HTTP identity — one home for the header every acquisition must send. + +THIS MODULE EXISTS BECAUSE AN ANONYMOUS CLIENT IS A BLOCKED CLIENT. `urllib.request` sends +`User-Agent: Python-urllib/3.x` unless told otherwise, and that string is on the default block list +of every bot-protection edge. Measured against the Mars blend on 2026-08-04: the USGS mosaic host +sits behind Cloudflare and answers `Python-urllib/3.14` with **HTTP 403** while serving the identical +URL to any named agent. The failure is total — not a slow path, not a partial read — and it lands at +the first byte of a 10.6 GiB acquisition. + +WHY A MODULE RATHER THAN A HEADER AT EACH CALL SITE. Nine call sites across seven modules reach for +HTTP, and a string that a second module needs is a string that gets copied. `open_url` is therefore +the only spelling: it takes the timeout the caller already had and returns the same context manager +`urlopen` did, so adopting it is a one-line change and *forgetting* it is what the scan in +`tests/test_fetch.py` refuses. A helper that merely offers the header would be a helper each new +acquisition is free to skip, and the skip is invisible until a host turns protection on. + +WHY THE FAILURE IS WORSE THAN A 403 LOOKS. A blocked acquisition is discovered at the moment someone +tries to build a body, which is the least convenient moment there is, and it is indistinguishable at +a glance from the URL having rotted. Sending an identity also makes us legible to the publishers +whose bandwidth this pipeline spends in multi-gigabyte units — the polite reading of the same rule. + +The agent string carries no contact URL yet, deliberately: inventing one would be a claim the project +cannot honour. When the repository is public, the URL belongs here, in this constant, and nowhere +else. +""" + +import os +import shutil +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +#: Sent on every request this pipeline makes. Any descriptive value clears the default-agent block +#: lists; the version is here so a future block can be attributed to a change we made. +USER_AGENT = "terrella-pipeline/1.0" + + +def build_request(url: str, *, method: str = "GET") -> urllib.request.Request: + """The request every caller should send, carrying the pipeline's identity. + + Separate from `open_url` only so a caller that must add its own headers has somewhere to start + that is not a bare `Request`; nothing needs that today. + """ + return urllib.request.Request(url, method=method, headers={"User-Agent": USER_AGENT}) + + +def open_url(url: str, *, method: str = "GET", timeout: float) -> Any: + """Open `url` with the pipeline's User-Agent, returning what `urlopen` returns. + + `timeout` is keyword-only and REQUIRED, which is the second half of what this module is for: the + default is no timeout at all, so an unadorned `urlopen` against a stalled host hangs a pipeline + stage forever rather than failing it. Every existing call site already passed one — making it + required means the next one cannot quietly not. + """ + return urllib.request.urlopen(build_request(url, method=method), timeout=timeout) + + +def download_one(url: str, dest: Path, *, timeout: float = 60, + absent_on_404: bool = False) -> str: + """Stream `url` to `dest` atomically. Returns 'ok', 'skipped', 'absent' or 'failed: '. + + The one home for "stream to .part, size-check against Content-Length, atomically rename", so a + file under its final name is always complete and `exists()` is a valid resume. + + `absent_on_404` MUST STAY DEFAULT-OFF. Eight of its ten callers test + `status.startswith("failed")`, so returning 'absent' to one of them turns a missing file into a + silent success; only the two WorldCover callers, whose ocean cells legitimately 404, pass True. + """ + if dest.exists(): + return "skipped" + part = dest.with_suffix(".part") + try: + with open_url(url, timeout=timeout) as response: + expected = int(response.headers.get("Content-Length", -1)) + with open(part, "wb") as sink: + shutil.copyfileobj(response, sink) + actual = part.stat().st_size + if expected != -1 and actual != expected: + part.unlink() + return f"failed: size mismatch ({actual} of {expected} bytes)" + os.replace(part, dest) + return "ok" + except urllib.error.HTTPError as exc: + part.unlink(missing_ok=True) + if absent_on_404 and exc.code == 404: + return "absent" + return f"failed: {exc}" + except Exception as exc: # noqa: BLE001 — one tile's failure must not kill the pool + part.unlink(missing_ok=True) + return f"failed: {exc}" diff --git a/pipeline/frame/country_config.py b/pipeline/frame/country_config.py index e47746c..ac3a4db 100644 --- a/pipeline/frame/country_config.py +++ b/pipeline/frame/country_config.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Resolve config/countries.toml into per-country pipeline parameters. The config is the committed source of truth for scope (which Natural Earth @@ -39,6 +38,7 @@ import argparse import difflib import filecmp +import itertools import math import re import shutil @@ -51,18 +51,49 @@ import shapefile from rasterio.warp import transform_bounds -from pipeline import paths -from pipeline.acquire.download_glo30 import (TILE_LIST, in_extent, - parse_tile_name, tile_files) -from pipeline.frame.frame_country import NE_DIR, pad_frame +from pipeline import bodies, naturalearth, paths +from pipeline.acquire.download_glo30 import ( + TILE_LIST, + in_extent, + parse_tile_name, + tile_files, +) +from pipeline.frame.frame_country import pad_frame from pipeline.fuse.fuse_heightfield import GEBCO from pipeline.render.render_prep import aea_crs -ROOT = Path(__file__).resolve().parents[2] +#: The CHECKOUT. Both paths below are TRACKED config, so they follow the repo and not the data +#: store — the one place in this module where `paths.ROOT` is the right root. +ROOT = paths.ROOT CONFIG_PATH = ROOT / "config/countries.toml" PIN_DIR = ROOT / "config/frames" BLENDER = paths.BLENDER + +def country_work_dir(slug: str) -> Path: + """Everything the hero pipeline builds for one country, in the data store. + + THE HOME FOR A CONCEPT THAT HAD NINE SPELLINGS across five modules — the stage list that + prints the pipeline, the runner that executes it, the pruner that reclaims it, the oracle that + validates it, and the two overlay passes that read its render dir. Every one of them was + correct, because each resolved to the same directory on a machine with `MAPS_DATA` unset; the + duplication is invisible until it isn't. + + Routed through `bodies.work_dir` rather than composed here, so a country's tree sits under the + same root as every other per-body stage and nests automatically if a second body ever grows + regions of its own. Earth's prefix is empty, so today this is exactly `/work/`. + """ + return bodies.work_dir(bodies.EARTH, slug) + + +def country_render_dir(slug: str) -> Path: + """The warp/render subdirectory `render_prep` fills and the hero stages read. + + Named because it is the argument five commands pass to each other — the point where a typo + stops being a wrong path and starts being a stage silently reading an empty directory. + """ + return country_work_dir(slug) / "render" + DEFAULT_KEYS = {"pad_pct", "hero_long_edge", "warp_long_edge", "fusion", "sky_view_strength", "resolution_floor_m"} COUNTRY_KEYS = {"admin", "frame", "hero_long_edge", "fusion", "status", "notes", @@ -113,7 +144,7 @@ def load_config() -> dict: if (status := tbl.get("status")) not in (None, "antimeridian"): bad.append(f"{where}: unknown status {status!r}") if (fusion := tbl.get("fusion")) not in (None, *FUSION_RES): - bad.append(f"{where}: fusion must be one of {sorted(FUSION_RES)}") + bad.append(f"{where}: fusion must be one of {sorted(FUSION_RES)}, not {fusion!r}") if (sv := tbl.get("sky_view_strength")) is not None \ and not _valid_strength(sv): bad.append(f"{where}: sky_view_strength must be a number in [0, 1]") @@ -137,7 +168,7 @@ def load_ne_rows(): The Reader is returned too and kept open: geometry (polygon parts for the far-flung check) is re-read lazily by index, so --country never pays for parsing the whole world's rings.""" - shp = NE_DIR / "ne_10m_admin_0_countries/ne_10m_admin_0_countries.shp" + shp = naturalearth.layer("ne_10m_admin_0_countries") if not shp.exists(): sys.exit(f"{shp} not found — run pipeline/acquire/download_naturalearth.sh") sf = shapefile.Reader(str(shp)) @@ -195,7 +226,7 @@ def resolve(slug: str, row: dict, cfg: dict) -> dict | None: defaults, tbl = cfg["defaults"], cfg.get("countries", {}).get(slug, {}) if tbl.get("status") == "antimeridian": return None - west, south, east, north = row["bbox"] + west, _south, east, _north = row["bbox"] # A frame override is authoritative, so a raw bbox that spans 180 (the # country has parts on both sides) is fine — the override frame does not. if "frame" not in tbl and west <= -179.99 and east >= 179.99: @@ -246,7 +277,7 @@ def main_part_fraction(sf, row) -> float: pts = np.asarray(shape.points) starts = list(shape.parts) + [len(pts)] best_area, best_bb = -1.0, (0.0, 0.0, 0.0, 0.0) - for start, end in zip(starts, starts[1:]): + for start, end in itertools.pairwise(starts): x_coords, y_coords = pts[start:end, 0], pts[start:end, 1] area = 0.5 * abs(np.sum( x_coords * np.roll(y_coords, -1) - np.roll(x_coords, -1) * y_coords)) @@ -302,8 +333,13 @@ def stage_commands(resolved: dict) -> list[str]: """The exact pipeline for one resolved country, in run order.""" fr = fmt_frame(resolved["frame"]) tag = f"{FUSION_RES[resolved['fusion']]}s" - work = f"data/work/{resolved['slug']}" - rd = f"{work}/render" + # ABSOLUTE, from the store. These strings are handed to `subprocess` with `cwd` set to the + # checkout, so a relative `data/work/…` resolved against the checkout no matter where the store + # actually was — which put stage 3 in the incoherent position of READING its mosaics through + # `MAPS_DATA` and WRITING its output beside the source tree. The checkout-relative paths further + # down (`pipeline/…`, `blender/…`) stay relative on purpose: those really are checkout paths. + work = country_work_dir(resolved["slug"]) + rd = country_render_dir(resolved["slug"]) prep = (f"python -m pipeline.render.render_prep --heightfield {work}/heightfield_{tag}.tif" f" --mask {work}/oceanmask_{tag}.tif" f" --watermask {work}/watermask_{tag}.tif" @@ -314,14 +350,14 @@ def stage_commands(resolved: dict) -> list[str]: return [ f"python -m pipeline.acquire.download_glo30 --extent {fr}", "bash pipeline/fuse/build_mosaics.sh", - f"python -m pipeline.fuse.fuse_heightfield --bounds {fr}" - f" --res-arcsec {FUSION_RES[resolved['fusion']]} --outdir {work}", + (f"python -m pipeline.fuse.fuse_heightfield --bounds {fr}" + f" --res-arcsec {FUSION_RES[resolved['fusion']]} --outdir {work}"), prep, f"python -m pipeline.render.snow_mask --render-dir {rd}", f"python -m pipeline.render.lake_mask --render-dir {rd}", - f"{BLENDER} -b --python pipeline/render/scene_build.py --" - f" --render-dir {rd} --out blender/{resolved['slug']}_hero.blend" - f" --render blender/renders/heroes/{resolved['slug']}.png", + (f"{BLENDER} -b --python pipeline/render/scene_build.py --" + f" --render-dir {rd} --out blender/{resolved['slug']}_hero.blend" + f" --render blender/renders/heroes/{resolved['slug']}.png"), ] @@ -392,7 +428,7 @@ def print_country(sf, scope, cfg, slug: str, emit_pin: bool) -> int: def do_emit_pin(slug: str, resolved: dict): - dest = ROOT / f"data/work/{slug}/render/frame.json" + dest = country_render_dir(slug) / "frame.json" if not resolved["pin"]: sys.exit(f"no committed pin at {PIN_DIR / f'{slug}.json'}") if dest.exists(): diff --git a/pipeline/frame/frame_country.py b/pipeline/frame/frame_country.py index 2bb4cda..aba65c8 100644 --- a/pipeline/frame/frame_country.py +++ b/pipeline/frame/frame_country.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Compute a country's render frame from its Natural Earth bounding box. The frame is the lon/lat window every downstream stage shares: the fusion @@ -29,7 +28,7 @@ import shapefile -NE_DIR = Path(__file__).resolve().parents[2] / "data/raw/naturalearth" +from pipeline import naturalearth def country_bbox(shp_path, name): @@ -69,11 +68,11 @@ def main(): ap.add_argument("--country", required=True, help="ADMIN name in ne_10m_admin_0_countries " "(case-insensitive)") - ap.add_argument("--ne-dir", type=Path, default=NE_DIR) + ap.add_argument("--ne-dir", type=Path, default=naturalearth.DIR) ap.add_argument("--pad-pct", type=float, default=5.0) args = ap.parse_args() - shp = args.ne_dir / "ne_10m_admin_0_countries/ne_10m_admin_0_countries.shp" + shp = naturalearth.layer("ne_10m_admin_0_countries", directory=args.ne_dir) admin, bbox = country_bbox(shp, args.country) frame = pad_frame(bbox, args.pad_pct) print(f"country: {admin}") diff --git a/pipeline/freshness.py b/pipeline/freshness.py new file mode 100644 index 0000000..dddf47a --- /dev/null +++ b/pipeline/freshness.py @@ -0,0 +1,168 @@ +"""Is this output still correct? Three separate questions, each with its own oracle. + + COMPLETED `is_stale`, reading a `.done` marker rather than the output's own mtime + NEWER THAN `newest_mtime` over the sources, recursing into directories + SAME SHAPE `grid_matches`, for warp targets warped ONTO the reference grid + SAME SCALE `resolution_matches`, for the reference raster that DEFINES that grid + +Each names a failure the others cannot see, and each function carries the case for its own — the +point of listing them together is that they are separate questions, not one. They compose: +`warp_needs_rebuild` is the second and third at once, `reference_needs_rebuild` the second and +fourth. `write_if_changed` is here because it is what lets a VALUE be a source at all, so a tunable +can be an input to `is_stale` without every `git checkout` restaging the planet. + +These are general and belong to no stage. They lived in `tile/shade_planet.py`, which two sibling +stages already imported them from — do not fold them back into a stage. +""" + +import math +from pathlib import Path + +import rasterio + + +def done_marker(output: Path) -> Path: + """The completion stamp beside `output` (height_3857.tif -> height_3857.done).""" + return output.with_suffix(".done") + + +def mark_done(output: Path) -> None: + """Stamp `output` complete. Call ONLY after its stage has returned successfully.""" + done_marker(output).touch() + + +def newest_mtime(*inputs: Path) -> float: + """Newest mtime among `inputs`, recursing into directories. Missing paths score 0.0. + + Directories are walked rather than stat'ed because a VRT's own mtime does NOT move when + the chunks it points at are re-fused -- which is exactly how the Caspian re-fuse stayed + invisible to the old guard. The planet is 540 cells x 3 rasters, so this is ~1.6k stats. + """ + newest = 0.0 + for path in inputs: + if not path.exists(): + continue + if path.is_dir(): + for child in path.rglob("*"): + if child.is_file(): + newest = max(newest, child.stat().st_mtime) + else: + newest = max(newest, path.stat().st_mtime) + return newest + + +def is_stale(output: Path, *inputs: Path) -> bool: + """True if `output` must be rebuilt: never completed, REWRITTEN since it was completed, or + older than any of `inputs`. + + Freshness is read from the .done marker, never from `output` itself: GDAL creates its + target at the START of a run, so a crashed pass leaves a full-sized, freshly-stamped, + half-written raster that an mtime test on the raster would happily accept as current. + + A MARKER MUST ALSO BE NEWER THAN THE BYTES IT VOUCHES FOR, which is a different claim from + either of the others and the one that was missing. A marker written by an EARLIER successful + run keeps vouching after a later run overwrites the output and dies before finishing: Mars's + ice alpha was rebuilt onto a new grid, crashed in its first unit burn, and was then skipped by + the next pass because the marker from the run before it was still sitting there. Both other + guards passed, `grid_matches` precisely BECAUSE the crash had created a full-size target on the + new grid — and a planet shipped with an all-zero ice layer and nothing to read. + + COMPARING THE TWO MTIMES IS DERIVED, WHICH IS WHY IT LIVES HERE. The alternative is a rule that + every stage unlinks its own marker before writing, which is a thing to remember at each of them + and to notice missing at none. Equal stamps are fresh: a stage marks done after it writes, so + only a STRICTLY newer output means bytes arrived after the promise. + """ + marker = done_marker(output) + if not output.exists() or not marker.exists(): + return True + stamped = marker.stat().st_mtime + if output.stat().st_mtime > stamped: + return True + return newest_mtime(*inputs) > stamped + + +def grid_matches(path: Path, width: int, height: int, bounds) -> bool: + """True if `path` exists on exactly the reference grid (`width` x `height`, same `bounds`). + + Every 3857 raster below `height_3857` is warped to height's grid (via -te/-ts), but each one's + freshness is gated on its own SOURCE, not on height. A re-fuse that GROWS the grid -- un-skipping + Antarctica takes the planet from 93009 to 131072 rows -- re-warps height while these sit falsely + fresh at the old dimensions, and the composite then reads window slices past their bottom (silent + corruption). A dimension/bounds comparison catches exactly that, and is deliberately NOT an mtime + dependency on height: that would re-warp all of them on a SAME-grid re-fuse (the Caspian + rewrote 4 chunks without moving the grid), which is 30+ min of needless work. + + Bounds are compared with a 1 m tolerance -- far below the 305 m pixel, so a real grid shift always + trips it, while the float noise of a -te repr round-trip never does. + """ + if not path.exists(): + return False + with rasterio.open(path) as dataset: + return (dataset.width == width and dataset.height == height + and all(math.isclose(actual, expected, abs_tol=1.0) + for actual, expected in zip(tuple(dataset.bounds), tuple(bounds)))) + + +def warp_needs_rebuild(out: Path, grid, *inputs: Path) -> bool: + """Whether a 3857 warp target must be rebuilt: `is_stale` (a source moved) OR off `grid` + (a re-fuse resized the planet under it). `grid` is (width, height, bounds). + + Split out so the composed condition is testable on its own. The load-bearing case is the one + `is_stale` alone cannot see: a raster whose SOURCE is unchanged but whose grid shrank beneath it. + """ + return is_stale(out, *inputs) or not grid_matches(out, *grid) + + +def resolution_matches(path: Path, map_units_per_pixel: float) -> bool: + """True if `path` is on exactly the pixel size the registry asks for. + + THE RASTER THAT DEFINES THE GRID HAD NO CHECK THE ONES BELOW IT ALL HAD. `grid_matches` re-warps + every 3857 raster when the reference grid moves beneath it, but the reference itself was gated on + `is_stale` alone — the mtimes of a VRT and a chunk directory, neither of which moves when a body's + tile ceiling does. Raising Mars from z6 to z7 therefore left a 32768 square grid reading FRESH: + the pass composited it and began cutting a z7 pyramid out of z6 pixels, which is the exact + upsample the ceiling was chosen to avoid, arriving with no error anywhere. + + DERIVED, NOT RECORDED, which is why this takes no sidecar. A raster's transform IS its + resolution, so there is no second copy to drift — the same reason `grid_matches` reads dimensions + off the file rather than trusting a recipe. A recipe is what a NON-geometric setting needs, + because nothing in the output can be asked; geometry answers for itself. + + Both axes, because a target built with one `-tr` value is square only if it was built correctly, + and half of a resolution change is not a state worth accepting. + """ + if not path.exists(): + return False + with rasterio.open(path) as dataset: + return all(math.isclose(abs(actual), map_units_per_pixel, rel_tol=1e-9) + for actual in (dataset.transform.a, dataset.transform.e)) + + +def reference_needs_rebuild(out: Path, map_units_per_pixel: float, *inputs: Path) -> bool: + """Whether the reference raster must be rebuilt: `is_stale` (a source moved) OR off the + registry's pixel size (the body's ceiling moved). + + Split out so the composed condition is testable on its own, exactly as `warp_needs_rebuild` is. + The load-bearing case is the one `is_stale` cannot see, and it is the mirror of that function's: + there, a raster whose source is unchanged while the grid moved under it; here, the raster the + others take their grid FROM, unchanged at the wrong scale. + """ + return is_stale(out, *inputs) or not resolution_matches(out, map_units_per_pixel) + + +def write_if_changed(path: Path, text: str) -> Path: + """Write `text` to `path` only when it differs, and return `path`. + + The only-when-different part is load-bearing, not an optimisation: it lets a generated + file stand in as a dependency for `is_stale`. Tunables like KNOBS and the ramp colours + live in source, whose mtime moves on any `git checkout` and would force a full planet + rebuild; materialised here, their mtime moves if and only if a VALUE actually changed. + + THE ORDER EVERY STAGE WITH A RECIPE FOLLOWS: write the recipe, THEN ask the freshness question. + That is what makes changing a setting trigger its own restage, and it is only safe because of + the paragraph above — an unchanged recipe never moves an mtime, so writing first cannot restage + an output that is still correct. A stage that asked first would answer against the old recipe. + """ + if not path.exists() or path.read_text() != text: + path.write_text(text) + return path diff --git a/pipeline/fuse/build_void_wbm.py b/pipeline/fuse/build_void_wbm.py index 331449b..0ad2802 100644 --- a/pipeline/fuse/build_void_wbm.py +++ b/pipeline/fuse/build_void_wbm.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Synthesise a water-body mask for the void DEM tiles from ESA WorldCover. download_cop30_void.py fetches the withheld GLO-30 DEM tiles from OpenTopography, but @@ -37,8 +36,13 @@ class 80 -> 2 (inland lake) everything else -> 0 (land) import rasterio from pipeline import paths -from pipeline.render.snow_mask import (BUCKET_URL, DATA_DIR as WC_DIR, WORKERS, - download_one, tiles_for_bounds) +from pipeline.fetch import download_one +from pipeline.render.snow_mask import ( + BUCKET_URL, + WORKERS, + tiles_for_bounds, +) +from pipeline.render.snow_mask import DATA_DIR as WC_DIR VOID_DIR = paths.DATA / "raw/cop30_void" WATER_CLASS = 80 # ESA WorldCover "permanent water bodies" @@ -66,7 +70,8 @@ def ensure_worldcover(bounds) -> Path: counts = {"ok": 0, "skipped": 0, "absent": 0} failures = [] with cf.ThreadPoolExecutor(WORKERS) as pool: - futs = {pool.submit(download_one, f"{BUCKET_URL}/{nm}", WC_DIR / nm): nm + futs = {pool.submit(download_one, f"{BUCKET_URL}/{nm}", WC_DIR / nm, + timeout=120, absent_on_404=True): nm for nm in names} for fut in cf.as_completed(futs): status = fut.result() diff --git a/pipeline/fuse/fuse_heightfield.py b/pipeline/fuse/fuse_heightfield.py index 76ff129..d6c6ae7 100644 --- a/pipeline/fuse/fuse_heightfield.py +++ b/pipeline/fuse/fuse_heightfield.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Fuse GLO-30 land elevation with GEBCO bathymetry into one heightfield. Recipe: @@ -34,10 +33,10 @@ import numpy as np import rasterio +from rasterio.enums import Resampling from rasterio.transform import from_origin from rasterio.vrt import WarpedVRT from rasterio.windows import Window -from rasterio.enums import Resampling from pipeline import paths diff --git a/pipeline/fuse/fuse_planet.py b/pipeline/fuse/fuse_planet.py index 263893c..5ef45e4 100644 --- a/pipeline/fuse/fuse_planet.py +++ b/pipeline/fuse/fuse_planet.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Fuse the whole planet into a 10-arcsecond heightfield, one 10-degree cell at a time. Phase 2, step 1: the seamless global land+bathymetry heightfield the tile pyramid is @@ -27,7 +26,10 @@ Output: per-cell tiled GTiffs with overviews under data/work/planet/chunks//, then `gdalbuildvrt` three planet VRTs over them (heightfield / oceanmask / watermask) for the -tiler. Idempotent: a cell whose heightfield exists is skipped; delete to redo. +tiler, and finally the seam declaration naming which of the three were actually built +(`pipeline/planet_seam.py` — written last, so its presence means this stage finished). +Idempotent: a cell whose heightfield exists is skipped; delete to redo. Re-running +`--build-vrts` alone is free — a VRT is replaced only when its XML changes. Memory budget: fuse_heightfield holds ~1-1.5 GB per process; --workers W costs ~W*1.5 GB against MemAvailable. Default 12 (~14 GB) suits this 16-core / 18-GiB-free box -- CPU-bound @@ -48,13 +50,24 @@ import rasterio -from pipeline.acquire.download_glo30 import DATA_DIR, TILE_LIST, in_extent, parse_tile_name +from pipeline import bodies, planet_seam +from pipeline.acquire.download_glo30 import ( + DATA_DIR, + TILE_LIST, + in_extent, + parse_tile_name, +) RES_ARCSEC = 10 CELL_DEG = 10 TAG = "10s" # must equal fuse_heightfield's f"{RES_ARCSEC:g}s" -PLANET_DIR = DATA_DIR.parent.parent / "work/planet" # data/work/planet -CHUNKS_DIR = PLANET_DIR / "chunks" +#: EARTH BY CONSTRUCTION, AND DELIBERATELY WITHOUT A `--body`. This driver fuses Copernicus tiles +#: against GEBCO over a Natural-Earth-indexed land list; every input it reads describes one planet, +#: so parameterising it would produce a flag whose only legal value is `earth`. A second body enters +#: the same seam through its own producer (`planet_seam`), which is what makes them interchangeable +#: downstream. Resolved through the registry rather than spelled out so the seam has ONE path home. +EARTH_PLANET_DIR = planet_seam.planet_dir(bodies.EARTH) +CHUNKS_DIR = EARTH_PLANET_DIR / "chunks" DEFAULT_WORKERS = 12 GIB_PER_WORKER = 1.5 # fuse_heightfield peak RSS incl. GDAL cache, rounded up @@ -134,8 +147,8 @@ def enforce_land_guard(outdir: Path) -> bool: "100% ocean. The DEM/WBM mosaics are almost certainly stale (a VRT enumerates its " "sources at build time) — run pipeline/fuse/build_mosaics.sh, then re-run the " "sweep. Outputs were deleted so this cell retries.\n") - for layer in ("heightfield", "oceanmask", "watermask"): - (outdir / f"{layer}_{TAG}.tif").unlink(missing_ok=True) + for raster in planet_seam.PLANET_RASTERS: + (outdir / f"{raster}_{TAG}.tif").unlink(missing_ok=True) return False @@ -148,7 +161,7 @@ def fuse_cell(name: str, bounds, expect_land: bool) -> tuple[str, str]: "--bounds", *map(str, bounds), "--res-arcsec", str(RES_ARCSEC), "--outdir", str(outdir), "--coverage-warn"] result = subprocess.run(cmd, env={**os.environ, **FUSE_ENV}, - capture_output=True, text=True) + capture_output=True, text=True, check=False) if result.returncode != 0: outdir.mkdir(parents=True, exist_ok=True) (outdir / "error.log").write_text(result.stdout + "\n" + result.stderr) @@ -160,17 +173,35 @@ def fuse_cell(name: str, bounds, expect_land: bool) -> tuple[str, str]: return name, "ok" +def _run(command: list[str]) -> None: + subprocess.run(command, check=True) + + def build_vrts(): - """Index the per-cell outputs into three planet-wide VRTs for the tiler.""" - for layer in ("heightfield", "oceanmask", "watermask"): - sources = sorted(CHUNKS_DIR.glob(f"*/{layer}_{TAG}.tif")) + """Index the per-cell outputs into planet-wide VRTs, then declare what was built. + + THE DECLARATION IS WRITTEN LAST, and it is the reason this function ends where it does. Its + presence is this stage's completion stamp, and its contents are what every consumer reads to + learn whether Earth has an ocean mask at all — a question that cannot be answered by looking for + the file, because a missing raster and an unfinished fusion are the same absence. See + `pipeline/planet_seam.py`. + + The `continue` below is why the declaration carries content rather than just existing: this + function has always been able to emit fewer than three rasters, and until now nothing recorded + which. + """ + built = [] + for raster in planet_seam.PLANET_RASTERS: + sources = sorted(CHUNKS_DIR.glob(f"*/{raster}_{TAG}.tif")) if not sources: - print(f"no {layer} chunks yet — skipping {layer} VRT", flush=True) + print(f"no {raster} chunks yet — skipping {raster} VRT", flush=True) continue - vrt = PLANET_DIR / f"planet_{layer}.vrt" - subprocess.run(["gdalbuildvrt", "-overwrite", str(vrt), - *[str(path) for path in sources]], check=True) - print(f"{vrt.name}: {len(sources)} chunks", flush=True) + vrt = planet_seam.vrt_path(bodies.EARTH, raster) + changed = planet_seam.write_vrt_if_changed(vrt, lambda target, sources=sources: _run( + ["gdalbuildvrt", "-overwrite", str(target), *[str(path) for path in sources]])) + built.append(raster) + print(f"{vrt.name}: {len(sources)} chunks{'' if changed else ' (unchanged)'}", flush=True) + print(f"declared {planet_seam.declare(bodies.EARTH, built)}", flush=True) def main() -> int: diff --git a/pipeline/fuse/relabel_mars.py b/pipeline/fuse/relabel_mars.py new file mode 100644 index 0000000..faea9a8 --- /dev/null +++ b/pipeline/fuse/relabel_mars.py @@ -0,0 +1,88 @@ +"""Publish the MOLA/HRSC blend as Mars's planet heightfield — a CRS relabel, and nothing else. + +THE FUSE TIER'S COUNTERPART FOR A BODY THAT NEEDS NO FUSION. `fuse_planet` turns ~26,000 Copernicus +tiles, a void-fill edition and a bathymetry grid into 648 fused cells and indexes them; Mars arrives +as one pre-blended global raster, so the same job is a few lines. It lives beside that driver because +a tier is defined by what it PRODUCES — a body's planet rasters, and the declaration naming them — +not by how much work it takes to produce them. + +WHAT THE RELABEL IS, AND WHY IT IS AN IDENTITY. The source is lon/lat degrees on an unflattened +sphere of 3,396,190 m; we declare that grid to be EPSG:4326. Every pixel keeps the longitude and +latitude it already had — only the label naming which body those angles belong to changes. Nothing +is resampled, nothing moves, and the output is a VRT, so no copy of 10.6 GiB is made. + + IT IS ONLY HONEST BECAUSE THE SOURCE IS A TRUE SPHERE. On an ellipsoid the same declaration would + silently shift every latitude, because a geodetic latitude on one figure is a different angle on + another. `download_mars_dem.assert_grid` is what holds that precondition, and `main` runs it + before this module writes anything. + +WHY RELABEL AT ALL, rather than projecting Mars on its own sphere: PROJ refuses to build an +operation between two celestial bodies, and the tiler reprojects into WebMercatorQuad. A +Mars-radius Mercator raster therefore cannot be cut into tiles without disabling that guard +globally. Every projection downstream is Earth-sphered for every body, and `bodies.MARS.ground_ +radius_m` is the single fact that converts the resulting map units back into Martian ground metres. + +MARS DECLARES A HEIGHTFIELD AND NOTHING ELSE. No ocean mask, no water mask — not empty ones, none +at all. A raster of zeros on disk cannot be told apart from one produced by measuring Mars's oceans +and finding none, and it would be the only body fact in this project written as a fabricated +dataset. The consumers build all-land selectors in memory from the declaration instead +(`pipeline/planet_seam.py`), and a sea at a chosen contour later is a change HERE plus a registry +line — which is what makes that question answerable by rendering rather than by arguing. + +KNOWN AND UNANSWERED UNTIL THE FILE IS ON DISK: the blend declares nodata -32768, where Earth's +fused heightfield declares none. If the mosaic actually contains nodata pixels they will pass +through this VRT, survive the warp, and land at the bottom of the sea ramp as deep blue. Whether it +contains any is not knowable from the header — the first z6 pyramid answers it, unmissably. + +Output (data/work/mars/planet/): + planet_heightfield.vrt the blend, CRS declared EPSG:4326 + planet_rasters.json the seam declaration, written last + +Idempotent, and free to re-run: the VRT is replaced only when its XML actually changes, so a second +run does not move the mtime that gates a 3857 warp. + +Usage: + python3 -m pipeline.fuse.relabel_mars +""" + +import subprocess +import sys +from pathlib import Path + +from pipeline import bodies, planet_seam +from pipeline.acquire import download_mars_dem + + +def relabel(source: Path) -> Path: + """Write Mars's planet heightfield VRT over `source`, declaring its CRS to be EPSG:4326. + + `-a_srs` ASSIGNS rather than reprojects, which is the whole point: `-t_srs` would ask PROJ for + an operation between two celestial bodies and be refused, and any resampling here would cost a + 5.7 Gpx pass to move pixels that are already where they belong. + """ + def build(target: Path) -> None: + subprocess.run(["gdal_translate", "-q", "-of", "VRT", "-a_srs", "EPSG:4326", + str(source), str(target)], check=True) + + vrt = planet_seam.vrt_path(bodies.MARS, "heightfield") + changed = planet_seam.write_vrt_if_changed(vrt, build) + print(f"{vrt}{'' if changed else ' (unchanged)'}", flush=True) + return vrt + + +def main() -> int: + blend = download_mars_dem.blend_path() + if not blend.exists(): + sys.exit(f"{blend} is not on disk — run `python3 -m pipeline.acquire.download_mars_dem` " + f"first (~10.6 GiB)") + # BEFORE the relabel, not after, and not skipped because the file was verified at download time. + # The relabel's honesty rests on the source being a true sphere in degrees; a re-published mosaic + # on an ellipsoid would make it a silent latitude shift, and this is the one check that sees it. + download_mars_dem.assert_grid(blend) + relabel(blend) + print(f"declared {planet_seam.declare(bodies.MARS, ['heightfield'])}", flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/layers.py b/pipeline/layers.py new file mode 100644 index 0000000..d91a5cc --- /dev/null +++ b/pipeline/layers.py @@ -0,0 +1,192 @@ +"""What a surface layer IS, and which stages read it — one table, and the views derived from it. + +A LAYER IS SOMETHING THE RENDER PAINTS OVER THE HEIGHTFIELD. Snow, glaciers, sea ice, lake +bathymetry, the coastline. Whether a given planet HAS one is a body fact (`Body.surface_layers`); +what the set of possible ones is, and which stage reads which, is a pipeline fact and lives here. + +NOT `planet_seam`'S RASTERS, and the two must not be merged. Those are the heightfield and the masks +that classify it, produced by a different tier and answering a different question — `planet_seam`'s +own docstring holds that argument, and one word for both concepts is how a reader concludes that one +guard covers the other. + +WHY A TABLE RATHER THAN THREE FROZENSETS. The stage vocabularies were three hand-kept sets that had +to agree: a layer added to the whole one and forgotten in a stage's is one a body can declare and +nothing will ever build, and a name in a stage's set that is not in the whole one appears in that +stage's `layers_off` for every body alike. Both are silent. One row per layer makes the agreement +structural, and the split itself is still pinned against literals in `tests/test_bodies.py`, because +a table can hold a wrong column as easily as two sets can disagree. + +STAGE MEMBERSHIP IS A FIELD PER STAGE, NOT A SET OF NAMES, for the reason `bodies.Body` carries no +defaults: a fourth stage must be a hard error at every row until each layer answers for it. A set +would let the rows that need it be edited and the rest inherit "not mine" unexamined. + + from pipeline import layers + if layers.layer_is_buildable(body, layers.SEA_ICE, source, "bathymetry bare at the poles"): ... +""" + +from dataclasses import dataclass +from pathlib import Path + +from pipeline import bodies + + +@dataclass(frozen=True) +class Layer: + """One optional thing the render paints over the heightfield, and who reads it. + + Frozen, and every field required — see the module note. A `Layer` is a pipeline fact and says + nothing about any planet; `Body.surface_layers` is where a body answers for it. + """ + + #: The vocabulary word. The one spelling: call sites pass the `Layer`, never the string. + name: str + #: Read by the Mercator tile composite (`tile/shade_planet.py`). + in_composite: bool + #: Read by the polar cap render (`tile/cap_render.py`). + in_cap: bool + #: The planet raster this layer cannot be computed without, or None. + #: + #: Held as a NAME rather than an imported constant so this module never imports `planet_seam`, + #: which imports `bodies` beside us. Nothing at runtime spell-checks the string — the price of + #: keeping that dependency one-way — so `tests/test_bodies.py` pins every one of them to + #: `planet_seam.KNOWN_RASTERS`, a typo being otherwise silent in whichever direction it lands. + requires_raster: str | None + #: The file this layer is built into inside the body's own work directory, or None for a layer + #: the composite never reads back from a raster. NOT `requires_raster` above, which names a + #: raster the PLANET SEAM emits at a different tier and on another body's behalf. + #: + #: A LAYER FACT RATHER THAN A PRODUCER FACT, which is what lets `shade_planet.composite_deps` + #: stay body-independent: it names every layer's raster whatever the planet declared, and its + #: sibling `cap_render.cap_sources` names only what it opens. Independent of `in_composite` too, + #: though the two agree on every row today — a composite layer answered by pure arithmetic would + #: carry None here, so deriving either column from the other loses that case silently. + warped_basename: str | None + + def warped_in(self, work: Path) -> Path: + """This layer's built raster inside one body's work directory — the one place they join. + + Only meaningful for a `WARPED_LAYERS` row, and the assertion is that filter restated where a + type checker can see it rather than a condition anything is expected to reach. + """ + assert self.warped_basename is not None, f"{self.name} builds no raster" + return work / self.warped_basename + + +#: Every layer the pipeline knows, and the whole vocabulary `Body.surface_layers` may draw from. +#: +#: Each comes from a dataset describing exactly one planet — Earth — which is why membership is a +#: body fact and not, as it looks, a question of whether a file happens to be on disk: every source +#: is a module constant at a fixed global path shared by every body. `layer_is_buildable` below is +#: what keeps that ordering honest. +#: +#: NAMED FOR THE CLAIM, NOT FOR EARTH'S DATASET. `perennial_ice` was `snow`, which described the +#: source (NSIDC-0791 persistence) rather than what the layer asserts — the white that is there all +#: year. Earth's own south cap already stretched the old word past breaking, using it for +#: Antarctica's permanent ice SHEET, and Mars's residual cap is CO2 and water ice rather than +#: snowfall. A body-specific name is how a vocabulary grows one entry per planet for one concept. +#: +#: THE BASENAMES BELOW ARE SHIPPED AND MUST NOT BE TIDIED. Each is a dependency of the composite by +#: mtime, so renaming one restages Earth's whole pyramid to reproduce the pixels already on disk. +LAKE_DEPTH = Layer("lake_depth", in_composite=True, in_cap=False, + requires_raster="watermask", warped_basename="lakedepth_3857.tif") +PERENNIAL_ICE = Layer("perennial_ice", in_composite=True, in_cap=True, + requires_raster=None, warped_basename="snow_persistence_3857.tif") +GLACIERS = Layer("glaciers", in_composite=True, in_cap=False, + requires_raster=None, warped_basename="glacier_3857.tif") +SEA_ICE = Layer("sea_ice", in_composite=True, in_cap=True, + requires_raster="oceanmask", warped_basename="seaice_3857.tif") +COASTLINE = Layer("coastline", in_composite=False, in_cap=True, + requires_raster=None, warped_basename=None) + +LAYERS: tuple[Layer, ...] = (LAKE_DEPTH, PERENNIAL_ICE, GLACIERS, SEA_ICE, COASTLINE) + + +#: The whole vocabulary, as names. +SURFACE_LAYERS = frozenset(layer.name for layer in LAYERS) + +#: The layers each stage reads, derived — never hand-kept, and never equal to each other. +#: +#: THE SPLIT IS LOAD-BEARING, which is the whole reason two views exist: each stage records the +#: layers it is MISSING in its own freshness recipe, so turning one off restages it — something file +#: mtimes cannot do, because an unbuilt raster scores 0.0 and is therefore silently not a dependency. +#: Recording a layer a stage never reads inverts the trap instead of closing it: switching the +#: coastline would restage a 46 GB tile composite that cannot contain one. The tiles bake no coast +#: (it is a vector overlay the client draws) and the caps composite no lake bathymetry and no +#: glaciers (`depth=None`, persistence-only snow). Over- and under-tracking are both silent. +COMPOSITE_LAYERS = frozenset(layer.name for layer in LAYERS if layer.in_composite) +CAP_LAYERS = frozenset(layer.name for layer in LAYERS if layer.in_cap) + +#: The composite layers with a file behind them, as rows and IN `LAYERS` ORDER — the three places +#: `shade_planet` handles a built layer walk this, so the warp, the dependency tuple and the window +#: reads cannot disagree about the set or about its order. +#: +#: ORDER IS PART OF THE CONTRACT, not tidiness: `composite_deps` returns a tuple whose contents a +#: test pins, and `LAYERS` is written in the order that tuple has always had. +WARPED_LAYERS: tuple[Layer, ...] = tuple( + layer for layer in LAYERS if layer.in_composite and layer.warped_basename) + +#: Which planet raster each dependent layer needs, derived. Read by `planet_seam._require_coherent`. +#: +#: THESE ARE REAL DATA DEPENDENCIES, NOT BOOKKEEPING. `lake_depth` is zeroed off watermask class 2 +#: (`lake_depth.lakes_only`), so a body declaring that layer with no watermask has nothing to zero +#: against and the composite would read `None` as a class code. `sea_ice` is gated on the ocean mask +#: inside `shade.composite`, so ice on a body with no ocean mask is blended against an all-False +#: selector and paints nothing at all — a layer that is switched on, costs a warp, and cannot reach +#: a pixel. Both are incoherent rather than merely empty, which is why `planet_seam` refuses them +#: where the two facts first meet rather than letting one surface as a `TypeError` in a worker. +LAYER_REQUIRES_RASTER: dict[str, str] = { + layer.name: layer.requires_raster for layer in LAYERS if layer.requires_raster is not None} + + +def layers_off(body: bodies.Body, vocabulary: frozenset[str]) -> list[str]: + """Which of `vocabulary` this body does NOT have, sorted — one stage's freshness record. + + THE LAYERS THAT ARE OFF, NEVER THE ONES THAT ARE ON, and that asymmetry is load-bearing rather + than stylistic. Earth declares every layer, so its list is empty and the caller's conditional + record writes nothing at all, leaving a 46 GB composite and a 14 GB cap render byte-identical. + Recording the layers that are ON would put a list into Earth's recipe for the first time and + restage the planet to produce the pixels already sitting there. + + `vocabulary` is the CALLER'S stage view — `COMPOSITE_LAYERS` or `CAP_LAYERS`, never + `SURFACE_LAYERS` — so that a stage records only what it actually reads. See those two for why a + shared vocabulary here would trade one silent freshness bug for another. + """ + return sorted(vocabulary - body.surface_layers) + + +def body_declares_layer(body: bodies.Body, layer: Layer, consequence: str) -> bool: + """Whether this body has `layer` at all — the body half of the gate, on its own. + + SPLIT OUT BECAUSE ONE RULE HAS NO DATASET BEHIND IT. The forced Antarctic land-ice patch is pure + latitude-and-land arithmetic (`snow.antarctic_snow_mask`), so there is no file whose absence + could ever switch it off — on a sea-less body it would simply whiten every piece of land below + 60 degrees south. It rides the `perennial_ice` layer, and this is what lets it ask that question + with the same words and the same printed consequence as the four layers that do read a file. + """ + if layer.name not in body.surface_layers: + print(f"{body.name} declares no {layer.name} layer -> skipped ({consequence})", flush=True) + return False + return True + + +def layer_is_buildable(body: bodies.Body, layer: Layer, source: Path, consequence: str) -> bool: + """Whether this body's `layer` can be warped — asked of the BODY first, then of the disk. + + THE ORDER IS THE POINT, and it is structural here rather than a convention: the body half is a + separate function and this one calls it first. Each of these sources is a module constant at a + fixed global path, so `source.exists()` answers "have we downloaded Earth's data" for every + planet alike. Asking it first would let a second body pass the check on Earth's file and paint + Earth's cryosphere onto its own grid — at the same latitudes, so it renders as a perfectly + plausible planet. + + Both branches print, and each states the consequence rather than only the cause: a skipped layer + is a look decision, and a pass that goes quiet about one is a pass whose output cannot be read + back. Returning False rather than raising keeps a partial build legal, which is what makes the + layers switchable at all. + """ + if not body_declares_layer(body, layer, consequence): + return False + if not source.exists(): + print(f"no {source.name} -> {layer.name} skipped ({consequence})", flush=True) + return False + return True diff --git a/pipeline/mercator.py b/pipeline/mercator.py index bb81b81..7297ded 100644 --- a/pipeline/mercator.py +++ b/pipeline/mercator.py @@ -16,13 +16,33 @@ catalogue would mean every caller that has a radius but no `Body` had to invent one. """ -from __future__ import annotations - import math import numpy as np from numpy.typing import NDArray +#: The sphere EPSG:3857 is DEFINED on, in metres. A projection constant, not a planet's property — +#: it happens to equal Earth's equatorial radius because that is where the projection came from, +#: and it stays this number for every body: PROJ refuses to build an operation between two celestial +#: bodies, so every raster in this pipeline is EPSG:3857 whatever planet its elevations describe. +#: +#: So a northing on a tile grid names a latitude on THIS sphere for Mars exactly as it does for +#: Earth, and a caller reaching into the body registry for it is asking the wrong question — the +#: latitude of a grid row is a property of the grid, not of the ground under it. `bodies.EARTH` +#: pins the same value as the body's own radius, which is a separate coincidence with its own test. +WEB_MERCATOR_RADIUS_M = 6378137.0 + +#: Half the width of the Web Mercator plane in map units — where the world's east and west edges +#: sit, and therefore what "this raster is global" means when a caller has only bounds to go on. +#: +#: DERIVED, NOT TRANSCRIBED, and that is the whole reason it is here. The digits are what drift: the +#: suite alone held `20037508.34`, `20037508.343` and the full `20037508.342789244`, each correct to +#: its own author's tolerance and none of them able to answer whether a raster's bounds ARE the world +#: or merely near it. Earth's heightfield overshoots this by 12.25 m and Mars lands on it exactly, so +#: any test of globalness has to be written against a pixel size rather than against these decimals — +#: which is a judgement a caller can only make if the exact value has one owner. +MERCATOR_HALF_M = math.pi * WEB_MERCATOR_RADIUS_M + def latitude_at(mercator_y, radius_m: float): """Latitude in degrees of a Web-Mercator northing, on a sphere of `radius_m`. diff --git a/pipeline/naturalearth.py b/pipeline/naturalearth.py new file mode 100644 index 0000000..6457a8a --- /dev/null +++ b/pipeline/naturalearth.py @@ -0,0 +1,65 @@ +"""The one home for Natural Earth's vectors: where they live, and how a layer is addressed in them. + +WHY THIS MODULE EXISTS, since a directory and a two-line join look like they need no home at all. +Natural Earth is the most widely shared input in this pipeline — one acquirer writes it and seven +readers across three tiers consume it — and before this module the directory was spelled **eight** +times and the layer join **five**, in modules that never import each other. Two of those readers had +independently invented the identical helper (`NE / name / f"{name}.shp"`) and kept it local, which +is the tell: the abstraction was found twice and promoted zero times. + +The pattern behind it is worth naming, because it will recur. Every dataset with ONE owning module +is already single-homed here — gebco, rgi, seaice, glo30, globathy, worldcover, mars each have +exactly one constant. Duplication appeared precisely where a dataset became SHARED, because +`paths.py` hands out a machine root and `bodies.work_dir` hands out a per-body stage directory, and +nothing owned a shared *dataset*. So the second module to want one copied the first. When the next +raw source gains a second reader, it wants a module like this one rather than a second constant. + +WHAT IS DELIBERATELY NOT HERE: which layer a caller wants. `layer("ne_10m_coastline")` at the call +site is that module choosing its own dataset, not a fact shared with anyone — collapsing those into +constants would move a local decision somewhere it cannot be read alongside the code that makes it. +The shared facts are the directory and the naming rule. Those are here; the choices stay out there. + +THE VOCABULARY IS SPELLED TWICE AND CANNOT BE SPELLED ONCE. `download_naturalearth.sh` is the +writer, and shell cannot import Python, so `LAYERS` below is a second copy of the list in that +script by necessity. `tests/test_naturalearth.py` holds the two together, in the same shape the body +registry uses across the Python/TypeScript boundary — a reader asking for a layer nobody downloads +is otherwise a `shapefile.Reader` error about a missing file, which reads like a failed download +rather than like a name that was never going to exist. +""" + +from pathlib import Path + +from pipeline import paths + +#: Every layer the acquirer fetches, by its own name — which is also its directory name and the +#: stem of every component file inside it. Pinned against `download_naturalearth.sh` by a test. +LAYERS = frozenset({ + "ne_10m_admin_0_boundary_lines_land", + "ne_10m_admin_0_boundary_lines_maritime_indicator", + "ne_10m_admin_0_countries", + "ne_10m_admin_0_disputed_areas", + "ne_10m_coastline", + "ne_10m_lakes", + "ne_10m_rivers_lake_centerlines", +}) + +#: The unpacked vectors, in the data store. `MAPS_DATA` relocates them, and the acquirer honours the +#: same variable — writer and readers move together or the store is only half real. +DIR = paths.DATA / "raw/naturalearth" + + +def layer(name: str, directory: Path | None = None) -> Path: + """The shapefile for one Natural Earth layer. + + Natural Earth nests each layer in a directory of its own name, so the name appears twice in + every path: `/ne_10m_coastline/ne_10m_coastline.shp`. That doubling is the whole reason + this is a function — written by hand it is two chances to typo, and a typo in the second half + fails at read time with a missing-file error that reads like a failed download. + + `directory` exists for the two entry points that expose `--ne-dir`, and is read at CALL time so + the default follows a relocated store rather than freezing at import. + """ + if name not in LAYERS: + known = ", ".join(sorted(LAYERS)) + raise ValueError(f"unknown Natural Earth layer {name!r}; the acquirer fetches: {known}") + return (DIR if directory is None else directory) / name / f"{name}.shp" diff --git a/pipeline/ot_oracle.py b/pipeline/ot_oracle.py index 1ffa81b..2b06469 100644 --- a/pipeline/ot_oracle.py +++ b/pipeline/ot_oracle.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Pull an OpenTopography Global DEM clip as an independent fusion oracle. Our heightfield is GLO-30 land fused with GEBCO bathymetry (fuse_heightfield.py). @@ -37,12 +36,16 @@ import sys import urllib.error import urllib.parse -import urllib.request from pathlib import Path import rasterio -ROOT = Path(__file__).resolve().parent.parent +from pipeline import fetch, paths +from pipeline.frame.country_config import country_work_dir + +#: The CHECKOUT, read for one thing only: the gitignored `.env` holding the API key. The clips this +#: writes are DATA and come from `country_work_dir`, which follows `MAPS_DATA`. +ROOT = paths.ROOT API = "https://portal.opentopography.org/API/globaldem" # The 17 global rasters the API serves; the commented ones are the useful oracles. @@ -89,7 +92,7 @@ def resolve_frame(slug: str) -> tuple: """The exact frame the pipeline fuses for this country (via country_config).""" import pipeline.frame.country_config as cc cfg = cc.load_config() - sf, rows = cc.load_ne_rows() + _sf, rows = cc.load_ne_rows() scope = cc.build_scope(cfg, rows) if slug not in scope: sys.exit(f"no country with slug {slug!r} in scope (see country_config --all)") @@ -99,7 +102,7 @@ def resolve_frame(slug: str) -> tuple: return tuple(resolved["frame"]) -def fetch(dataset: str, frame: tuple, out: Path, key: str) -> None: +def download_clip(dataset: str, frame: tuple, out: Path, key: str) -> None: """Download the clip to out (crash-safe via a .tmp sibling).""" west, south, east, north = frame query = urllib.parse.urlencode(dict( @@ -108,7 +111,7 @@ def fetch(dataset: str, frame: tuple, out: Path, key: str) -> None: tmp = out.with_name(out.name + ".tmp") try: # key is in the URL — never log it - with urllib.request.urlopen(f"{API}?{query}", timeout=600) as resp, \ + with fetch.open_url(f"{API}?{query}", timeout=600) as resp, \ open(tmp, "wb") as out_file: shutil.copyfileobj(resp, out_file) except urllib.error.HTTPError as ex: @@ -147,7 +150,7 @@ def main() -> int: args = ap.parse_args() frame = resolve_frame(args.country) if args.country else tuple(args.bounds) - out = args.out or (ROOT / f"data/work/{args.country}/oracle/{args.dataset}.tif") + out = args.out or (country_work_dir(args.country) / "oracle" / f"{args.dataset}.tif") if args.bounds is not None and args.out is None: ap.error("--bounds needs --out") @@ -163,13 +166,13 @@ def main() -> int: fr = " ".join(f"{value:g}" for value in frame) print(f"{args.dataset} over [{fr}] ({area:,.0f} km^2, cap {cap:,})", flush=True) - fetch(args.dataset, frame, out, load_api_key()) + download_clip(args.dataset, frame, out, load_api_key()) print(f"wrote {out}", flush=True) print("\nvalues (reference vs our fusion, if present):") summarize(out, f"oracle {args.dataset}") if args.country: - for hf in sorted((ROOT / f"data/work/{args.country}").glob("heightfield_*.tif")): + for hf in sorted(country_work_dir(args.country).glob("heightfield_*.tif")): summarize(hf, f"fusion {hf.stem.split('_')[-1]}") return 0 diff --git a/pipeline/paths.py b/pipeline/paths.py index d46229b..ac002e6 100644 --- a/pipeline/paths.py +++ b/pipeline/paths.py @@ -14,6 +14,13 @@ is `build_mosaics.sh`, which reads the same variable. - BLENDER: the Blender binary for hero renders. `MAPS_BLENDER` overrides; defaults to the documented tarball install (docs/pipeline.md § Environment setup). + +DERIVE AT CALL TIME, NOT AT IMPORT — the rule every consumer of these three follows, stated here +rather than at each of them. A module-level `SOMEWHERE = DATA / "x/y"` freezes the root at import, +so redirecting `MAPS_DATA` moves some of a module's paths and not others. The failure has no error +in it: the frozen readers go stale together, so they still agree with each other and every +assertion between them still passes. It has already cost one run that isolated its working tree and +wrote its served output into the real `web/public/`. Write a function. """ import os diff --git a/pipeline/planet_seam.py b/pipeline/planet_seam.py new file mode 100644 index 0000000..9ba78a7 --- /dev/null +++ b/pipeline/planet_seam.py @@ -0,0 +1,204 @@ +"""What a body's planet stage produced, declared by the stage that produced it. + +THE SEAM EVERY BODY ENTERS THROUGH. Earth's planet rasters come out of a 648-cell fusion; Mars's +come out of a CRS relabel of one published file. Both write the same three-named set into +`data/work//planet/`, and everything downstream — the 3857 warps, the composite, both polar +caps — reads them from there without caring which producer ran. + +WHY A DECLARATION AND NOT A DIRECTORY LISTING, which is the whole reason this module exists. +Downstream stages need to know whether this body HAS an ocean mask, and the tempting answer is +`(planet / "planet_oceanmask.vrt").exists()`. That answer is wrong in a way nothing reports: + + * ABSENCE IS NOT A STATEMENT. A missing mask cannot distinguish "this planet has no sea" from + "the producer died two rasters in". The first is a fact to composite against; the second is a + half-built planet that must never be shaded, and they look identical on disk. + * ABSENCE IS ALSO INVISIBLE TO FRESHNESS. `freshness.newest_mtime` scores a missing path 0.0, + so a raster that goes away stops being a dependency at the same moment it stops being an input. + Turning a sea OFF therefore leaves the old sea-painted composite looking perfectly fresh — + which is exactly the loop Phase 2 runs when it tries a shoreline contour, then another, then + none. + +So the producer states what it emitted, and it states it LAST. The declaration's PRESENCE is the +stage's completion stamp (the `.done` idiom one tier up, carrying content), and its CONTENT is the +body fact. A crashed producer writes none and every consumer refuses to run; a complete one is +trusted about what it does not have. + +RASTERS, NEVER "LAYERS". `layers.SURFACE_LAYERS` already owns that word for the optional things the +render paints OVER the heightfield — snow, glaciers, sea ice, lake depth, the coastline. These three +are the heightfield and the masks that classify it, produced by an entirely different tier and +answering an entirely different question. One word for both concepts is how a reader ends up +believing `layers_off` and this set are the same switch. + + from pipeline import planet_seam + rasters = planet_seam.declared(body) # raises if the producer never finished + if "oceanmask" in rasters: ... +""" + +import json +from collections.abc import Callable, Iterable +from pathlib import Path + +from pipeline import bodies, layers + +#: The rasters a planet stage may emit, in the order a producer builds them. +#: +#: `heightfield` is the elevation the whole pipeline shades. `oceanmask` is the 0/1 land/sea split +#: that chooses between the land and sea ramps; `watermask` is the 4-class code (0 land, 1 ocean, +#: 2 inland lake, 3 inland river) that selects inland water and keys the lake bathymetry off it. +#: +#: A TUPLE, NOT A SET, because producers iterate it and the order they build in is the order they +#: report in. Membership tests take `KNOWN_RASTERS` below. +PLANET_RASTERS = ("heightfield", "oceanmask", "watermask") + +KNOWN_RASTERS = frozenset(PLANET_RASTERS) + +#: The file the producer writes last, beside the rasters it names. +DECLARATION_NAME = "planet_rasters.json" + +#: How to produce each body's planet rasters, keyed by body name — quoted verbatim when a consumer +#: finds no declaration. +#: +#: An error that says what is missing and not how to fix it costs the reader a search through two +#: tiers, and the answer differs per body precisely because the producers do: Earth fuses 648 cells +#: from Copernicus and GEBCO, Mars relabels one published file. `tests/test_planet_seam.py` pins +#: every registered body to an entry here, so adding a third planet cannot leave its error message +#: pointing at Earth's command. +PRODUCER_COMMANDS: dict[str, str] = { + "earth": "python3 -m pipeline.fuse.fuse_planet --build-vrts", + "mars": "python3 -m pipeline.fuse.relabel_mars", +} + + +def planet_dir(body: bodies.Body) -> Path: + """This body's planet stage directory — where the rasters and the declaration live together.""" + return bodies.work_dir(body, "planet") + + +def vrt_path(body: bodies.Body, raster: str) -> Path: + """The VRT for one of this body's planet rasters.""" + _require_known(raster) + return planet_dir(body) / f"planet_{raster}.vrt" + + +def declaration_path(body: bodies.Body) -> Path: + """Where this body's planet stage records what it emitted.""" + return planet_dir(body) / DECLARATION_NAME + + +def _require_known(raster: str) -> None: + """A raster name outside the vocabulary is a typo, and a typo must not read as an absence.""" + if raster not in KNOWN_RASTERS: + known = ", ".join(PLANET_RASTERS) + raise ValueError(f"unknown planet raster {raster!r}; known rasters are: {known}") + + +def _require_coherent(body: bodies.Body, rasters: frozenset[str]) -> None: + """Refuse a declaration this body's own layers cannot be computed against. + + Checked on BOTH sides of the file — when it is written and when it is read — because the two + facts have different lifetimes. The producer knows what it emitted; the registry may gain a + layer months later, and that edit must fail against a declaration that is already on disk + rather than at the first pixel that needs it. + """ + if "heightfield" not in rasters: + raise ValueError( + f"{body.name}: a planet stage must emit a heightfield — it is the elevation every " + f"later stage shades, and a planet without one is not a partial planet but no planet") + for layer, raster in sorted(layers.LAYER_REQUIRES_RASTER.items()): + if layer in body.surface_layers and raster not in rasters: + raise ValueError( + f"{body.name}: declares the {layer!r} surface layer but its planet stage emitted no " + f"{raster!r} — see layers.LAYER_REQUIRES_RASTER for why that layer cannot be " + f"computed without it. Either the producer is incomplete or the body's " + f"surface_layers are") + + +def write_vrt_if_changed(vrt: Path, build: Callable[[Path], None]) -> bool: + """Have `build` write `vrt`, and replace the file on disk only when the XML actually differs. + + NOT AN OPTIMISATION — it is what makes a producer safe to re-run at all. Every 3857 warp + downstream is gated on the VRT's mtime, so an unconditional overwrite restages the whole planet: + on Earth that is a 44 GB re-warp, an 8:28 hillshade, a 53.8 min composite and a 3:44 cut, to + reproduce pixels that were already correct. Re-indexing is the natural thing to do after touching + a producer, so that cost sat one command away from anyone who tried. + + Byte-identity is what makes the comparison mean anything, and it was measured rather than + assumed: rebuilding all three of Earth's planet VRTs from the same 648 chunks, into the same + directory, reproduced the live files' SHA-256 exactly (GDAL 3.12.2). + + THE SCRATCH TARGET SHARES THE VRT'S DIRECTORY, and that is load-bearing rather than tidy: GDAL + writes source paths RELATIVE to the VRT, so building somewhere else and moving the result + rewrites every one of them and the comparison can never come out equal. + + SHARED BY BOTH PRODUCERS. Earth's builds a 648-source mosaic index with `gdalbuildvrt`; Mars's + relabels one file's CRS with `gdal_translate`. The tool differs, the hazard does not, and a + ten-line routine with a subtle directory constraint is exactly the shape that drifts when it is + written out twice. + + Returns True when the file on disk changed, so a caller can report it. + """ + scratch = vrt.with_suffix(".vrt.new") + vrt.parent.mkdir(parents=True, exist_ok=True) + build(scratch) + if vrt.exists() and vrt.read_bytes() == scratch.read_bytes(): + scratch.unlink() + return False + scratch.replace(vrt) + return True + + +def declare(body: bodies.Body, rasters: Iterable[str]) -> Path: + """Record what this body's planet stage emitted. CALL LAST, after every raster is on disk. + + THE ORDERING IS THE CONTRACT, not a convention: this file's existence is what tells every + consumer the stage finished. Written early, it promises rasters a crash may never deliver, and + the guarantee it exists to give is gone. + + Each named raster is checked onto disk before the name is written. A declaration is only worth + trusting about what is MISSING if it can be trusted about what is present, and a producer that + reports a raster it failed to build turns this file from an oracle into a second opinion. + """ + named = sorted(set(rasters)) + for raster in named: + _require_known(raster) + _require_coherent(body, frozenset(named)) + absent = [str(vrt_path(body, raster)) for raster in named + if not vrt_path(body, raster).exists()] + if absent: + raise FileNotFoundError( + f"{body.name}: refusing to declare rasters that are not on disk: {', '.join(absent)}") + path = declaration_path(body) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps({"rasters": named}, indent=2) + "\n") + return path + + +def declared(body: bodies.Body) -> frozenset[str]: + """Which planet rasters this body has, as its producer declared them. Raises if it never ran. + + RAISES RATHER THAN RETURNING AN EMPTY SET, and that is the point of the whole module: an empty + answer would be a statement about the planet, and a missing file is a statement about the + pipeline. The two must not share a return value. + """ + path = declaration_path(body) + if not path.exists(): + raise FileNotFoundError( + f"{path} is missing: {body.name}'s planet stage has not finished. This file is written " + f"last, so its absence means the producer never ran or died partway — the rasters " + f"beside it, if any, cannot be trusted. Run this body's planet producer — " + f"{PRODUCER_COMMANDS.get(body.name, 'this body has no producer registered')}") + rasters = frozenset(json.loads(path.read_text())["rasters"]) + for raster in sorted(rasters): + _require_known(raster) + _require_coherent(body, rasters) + return rasters + + +def rasters_off(rasters: frozenset[str]) -> list[str]: + """Which of the vocabulary this planet stage did NOT emit, sorted — one stage's freshness record. + + THE ONES THAT ARE OFF, NEVER THE ONES THAT ARE ON, exactly as `bodies.layers_off` does it and + for the same measured reason, which that function holds. Earth emits all three, so its list is + empty and nothing enters its recipe. + """ + return sorted(KNOWN_RASTERS - rasters) diff --git a/pipeline/profile/composite_ram.py b/pipeline/profile/composite_ram.py index 3a59bab..2b25c79 100644 --- a/pipeline/profile/composite_ram.py +++ b/pipeline/profile/composite_ram.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Measure composite()'s peak RSS on ONE production-shaped planet window. The planet composite is the stage that has actually OOM-killed this box: windows are the full @@ -33,6 +32,7 @@ import numpy as np +from pipeline.render import palette, seaice from pipeline.tile import shade WIDTH = 131072 # the planet grid's full width -- windows are always full-width @@ -77,8 +77,12 @@ def main() -> int: after_inputs = peak_gib() print(f" peak after allocating inputs : {after_inputs:6.2f} GiB", flush=True) + # Earth's, because the whole point is a PRODUCTION-shaped window: the measurement is only a + # lower bound on the real pass if it allocates what the real pass allocates, and a look with no + # sea ramp skips one full-window LUT lookup. rgb = shade.composite(heights, ocean, water, snow_alpha, hillshade, occlusion, - SVF_SHAPE, shape, depth=depth) + SVF_SHAPE, shape, depth=depth, look=palette.EARTH_LOOK, + snow_paint=(palette.SNOW_RGB, palette.SNOW_SHADOW_RGB), ice_paint=seaice.ice_white()) peak = peak_gib() print(f" PEAK through composite() : {peak:6.2f} GiB", flush=True) print(f" composite's own working set : {peak - after_inputs:6.2f} GiB", flush=True) diff --git a/pipeline/profile/pass_cap.py b/pipeline/profile/pass_cap.py new file mode 100644 index 0000000..00a48bc --- /dev/null +++ b/pipeline/profile/pass_cap.py @@ -0,0 +1,78 @@ +"""How much memory a planet pass may take, sized from the body rather than from Earth. + +WHY THIS IS NOT ONE CONSTANT IN THE SHELL. `run_pass.sh` capped every run at 16 G, and that number +is Earth's: the pass ENDS by invoking `cap_render` as a subprocess, which inherits the scope's +cgroup and peaks at 14.3 GB north / 13.9 GB south on 8192 squared float arrays. A body with +`renders_polar_caps = False` never reaches that stage at all, so on such a body the 16 G is +unbacked rather than protective — and the harness's own `MemAvailable` preflight then refuses to +start a pass the box could comfortably have run. That refusal is the failure this module exists to +remove: measured on this box with a browser open, `MemAvailable` sits near 15 GiB, which clears +every cap Mars actually needs and misses the one it does not. + +THE FLOOR IS THE PROJECT'S STANDING CAP, NOT A NEW NUMBER. 12 G is what every heavy job here runs +under; the pass was raised above it for the caps stage and for nothing else. Dropping back to it +for a capless body also restores a tripwire the raise knowingly gave up — 12 G was an accidental +guard on composite footprint, and a regression there currently hides until 16 G. + +WHAT THE MEASUREMENTS SAY, so the two numbers are traceable rather than chosen: + +- Earth, z8, `--tiles`: the composite peaks at 10.55 GiB and the caps at ~14.3 GB. 16 G. +- Mars, z6, `--tiles`, cold and end to end: summed per-process `VmHWM` **4.01 GiB**, the composite + worker holding 2.90 GiB of it. Read `VmHWM`, not the cgroup's `memory.peak` — that reported + 8.30 GiB for the same pass, of which 4.29 GiB was reclaimable page cache. 12 G leaves 3x. + +THE TILING RUN'S OWN JUSTIFICATION IS WEAKER THAN THE CAPS ONE, AND IT IS WORTH SAYING SO. With +`--tiles` the peak stage becomes `gdal raster tile`, whose workers each inherit `GDAL_CACHEMAX`; +that arithmetic is Earth-shaped but has never actually been measured, where Mars's whole pass has. +So this returns one cap per body rather than one per run label: a measured planet is sized off its +measurement, and an unmeasured stage does not get to out-vote a number taken from a real run. + +NO FALLBACK, and it is the same rule `palette.look_for` states for the ramp. A body that quietly +inherited Earth's cap would run — every pass would simply be capped somewhere arbitrary — and +nothing would ever name it. `bodies.get` raises on an unknown name, so this does too, by delegating. +""" + +import sys + +from pipeline import bodies +from pipeline.tile import shade_planet + +#: The cap a pass needs when it will render polar caps, in GiB. Set by `cap_render`'s measured +#: 14.3 GB peak plus headroom, and reached through the scope's cgroup because `shade_planet` +#: invokes that stage as a subprocess. +CAP_RENDERING_GIB = 16 + +#: The cap a pass needs when it will not, in GiB. The project's standing cap for any heavy job. +STANDING_GIB = 12 + + +def pass_memory_cap_gib(body: bodies.Body) -> int: + """The cgroup cap this body's pass may take, in GiB. + + Derived from `renders_polar_caps` rather than held as a per-body number, because that field + already answers the only question the cap turns on and a second field would be free to + disagree with it. A body that starts publishing caps gets the headroom by construction, on the + same commit that turns them on. + """ + return CAP_RENDERING_GIB if body.renders_polar_caps else STANDING_GIB + + +def cap_for_argv(argv: list[str]) -> int: + """The cap for a pass invoked with `argv`, parsed by the pass's OWN parser. + + Sharing `shade_planet.build_parser` rather than re-reading `--body` in shell is the point: the + harness forwards this exact argv to that module moments later, so a second spelling of the + grammar is a second thing to keep in step. It also makes the harness honour its own documented + contract EARLIER — `--body` is required, and until now a run that omitted it cleared the + preflight, opened a cgroup scope and only then died inside Python. + """ + return pass_memory_cap_gib(bodies.get(shade_planet.build_parser().parse_args(argv).body)) + + +def main() -> None: + """Print the cap in GiB, for `run_pass.sh` to read. Errors go to stderr via argparse.""" + print(cap_for_argv(sys.argv[1:])) + + +if __name__ == "__main__": + main() diff --git a/pipeline/profile/run_pass.sh b/pipeline/profile/run_pass.sh index ca996d1..8b3be07 100755 --- a/pipeline/profile/run_pass.sh +++ b/pipeline/profile/run_pass.sh @@ -20,24 +20,23 @@ # 4. the cgroup -> memory.peak for the whole scope, and the 12 G cap that kills the job not # the box (proven today: a 4-cell region render hit it and died alone). # -# Shade cap 16 G, raised from 12 G. The composite is NOT why: it still peaks at -# 10.55 GiB (opt #5, 128/N4; the serial composite was 6.24 GiB) and COMPOSITE_ROWS=128 -# is a hardcoded constant, not a function of this cap, so raising the cap does not let the -# composite grow -- N stays 4 because 256/N3 and N=6 OOM on their own arithmetic. The cap moved -# because the pass ENDS by rendering the polar caps (shade_planet invokes cap_render as a -# subprocess, which inherits this scope's cgroup) and the caps peak at ~14 GB -- so a 12 G pass -# completed every tile stage and then died at the very last one. Known cost of the raise: 12 G was -# also an accidental tripwire on composite footprint, and a regression there now goes unnoticed -# until 16 G. +# THE CAP IS THE BODY'S AND THIS SCRIPT DOES NOT KNOW IT -- pipeline/profile/pass_cap.py derives +# it from the registry, and holds the whole argument plus the measurements behind both numbers. +# MEMORY_CAP_OVERRIDE_GIB substitutes the number afterwards and says so on stdout when it does; the +# registry is still asked either way, and the branch itself carries why that ordering matters. +# The short version: 16 G is the CAP-RENDERING number, because the pass ENDS by invoking cap_render +# as a subprocess that inherits this scope's cgroup; a body rendering no caps never reaches that +# stage, so on it the 16 G is unbacked rather than protective and the preflight below then refuses +# a pass the box could have run. The composite is NOT why either number is what it is, and +# COMPOSITE_ROWS=128 is a hardcoded constant rather than a function of this cap, so a larger cap +# cannot let it grow. The per-stage peaks are measured in PROCESS.md, not restated here. # -# Tiling cap 16 G, and it is NOT the same calculation. The composite is skipped when planet_rgb is -# fresh, so the peak stage becomes `gdal raster tile`, which spawns -j ALL_CPUS workers that EACH -# inherit GDAL_CACHEMAX -- 16 x 512 MB of block cache alone, before any tile buffers. Tiling's real -# peak has never been measured, so the cap is sized off that per-worker cache math with headroom; -# 16 G still kills the job and not the box (29 G total, ~20 G available). A worker killed mid-write -# still leaves a TRUNCATED png, but build_tiles no longer resumes over a partial staging dir -- it -# removes it and cuts clean, so a bad tile can no longer survive into the pyramid. -# GDAL_CACHEMAX=512 per shade_planet.py's own launch note. +# What this script still owns is GDAL_CACHEMAX=512 (per shade_planet.py's own launch note), which +# `gdal raster tile` multiplies: it spawns -j ALL_CPUS workers that EACH inherit it. That product is +# an UPPER BOUND the cut never reaches -- the block cache fills lazily, and measured the cut is the +# lightest stage of the pass, so it is not what sizes this cap. A worker killed mid-write still +# leaves a TRUNCATED png, but build_tiles no longer resumes over a partial staging dir -- it removes +# it and cuts clean, so a bad tile can no longer survive into the pyramid. set -uo pipefail # Roots derive from this script's own location, never a hardcoded home path: the harness has to @@ -51,14 +50,45 @@ cd "$ROOT" || exit 1 if [[ " $* " == *" --tiles "* ]]; then RUN_LABEL=tiles - MEMORY_CAP=16G else RUN_LABEL=pass - MEMORY_CAP=16G fi PROF=$DATA/work/_profile_$RUN_LABEL # output: data, gitignored UNIT=terrella-$RUN_LABEL +# Ask the registry, through the pass's OWN argument parser, rather than reading --body here: this +# argv is forwarded to that module verbatim seconds later, so a second spelling of the grammar is +# a second thing to keep in step. It also makes this wrapper honour the contract its header states +# -- until now a run that omitted --body cleared the preflight, opened a cgroup scope, and only +# then died inside Python. argparse writes its own message to stderr, so nothing is restated here. +MEMORY_CAP_GIB=$("$VENV" -m pipeline.profile.pass_cap "$@") || exit 1 + +# A DELIBERATE OVERRIDE, READ AFTER THE RESOLVER AND NEVER INSTEAD OF IT, which is the whole design +# of this branch. Written `${MEMORY_CAP_OVERRIDE_GIB:-$(...)}` it would let an exported variable skip +# the resolver entirely, and with it the --body check the line above exists to enforce; written here +# the registry is always asked, the body is always named, and only the NUMBER is substituted. +# +# It exists because the alternative is an untestable wiring. Both bodies render caps now, so the +# resolver answers 16 for every planet in the registry and no real invocation can tell "the shell +# used the number it was given" from "the shell holds a 16" -- a distinction sabotage.py has a case +# for. A synthetic body cannot help: pass_cap runs in a SUBPROCESS, so a monkeypatched registry +# never reaches it. This makes the number a controllable input, which is what a wiring test needs. +# +# ANNOUNCED, BECAUSE A SILENT ONE WOULD BE THE THING pass_cap's "NO FALLBACK" NOTE REFUSES. A pass +# capped at an arbitrary number that nothing names is exactly the failure that module is written to +# prevent; a pass capped at a number it prints is an operator decision, like ALLOW_LOW_MEMORY. +if [[ -n "${MEMORY_CAP_OVERRIDE_GIB:-}" ]]; then + # Validated rather than trusted: a non-numeric value makes the comparison below evaluate it as + # 0, so every box would clear every cap and the preflight would silently stop being a check. + if [[ ! "$MEMORY_CAP_OVERRIDE_GIB" =~ ^[0-9]+$ ]]; then + echo "ABORT: MEMORY_CAP_OVERRIDE_GIB=$MEMORY_CAP_OVERRIDE_GIB is not a whole number of GiB." >&2 + exit 1 + fi + echo "memory cap overridden: ${MEMORY_CAP_OVERRIDE_GIB} G instead of this body's ${MEMORY_CAP_GIB} G" + MEMORY_CAP_GIB=$MEMORY_CAP_OVERRIDE_GIB +fi +MEMORY_CAP=${MEMORY_CAP_GIB}G + # --- memory preflight ------------------------------------------------------------------------- # A cgroup cap kills the job instead of the box -- but only if the box can actually BACK the cap. # Capping at 16 G on a machine with 9 G free does not protect anything; it just relocates the diff --git a/pipeline/profile/sample_tree.py b/pipeline/profile/sample_tree.py index 8b7836f..c665bed 100644 --- a/pipeline/profile/sample_tree.py +++ b/pipeline/profile/sample_tree.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Sample every process in the pass's cgroup once a second: RSS, peak RSS, CPU, threads, disk I/O. Why a sampler at all, when perf is already recording? They answer disjoint questions. perf says @@ -108,47 +107,51 @@ def main() -> int: return 1 print(f"sampling {cgroup}", file=sys.stderr, flush=True) - out = open(args.out, "w", buffering=1) - started = time.time() - empty_rounds = 0 - while True: - now = time.time() - try: - pids = [int(p) for p in cgroup.joinpath("cgroup.procs").read_text().split()] - except OSError: - break # scope torn down -> pass finished - - if not pids: - empty_rounds += 1 - if empty_rounds > 5: - break - else: - empty_rounds = 0 - - procs = [sample for sample in (proc_sample(pid) for pid in pids) if sample] - record = { - "t": round(now - started, 2), - "cg_mem_mb": read_int(cgroup / "memory.current") / 1048576, - "cg_peak_mb": read_int(cgroup / "memory.peak") / 1048576, - "mem_avail_mb": 0, - "swap_used_mb": 0, - "procs": procs, - } - try: - meminfo = Path("/proc/meminfo").read_text() - info = {} - for line in meminfo.splitlines(): - key, _, rest = line.partition(":") - info[key] = int(rest.split()[0]) - record["mem_avail_mb"] = info.get("MemAvailable", 0) / 1024 - record["swap_used_mb"] = (info.get("SwapTotal", 0) - info.get("SwapFree", 0)) / 1024 - except (OSError, ValueError, IndexError): - pass - - out.write(json.dumps(record) + "\n") - time.sleep(max(0.0, args.interval - (time.time() - now))) - - out.close() + # The handle is held for the whole pass, so `with` is doing real work rather than satisfying a + # style: this sampler wraps a 30+ minute planet run, and the two `break`s below were the only + # paths that reached the old `close()`. Anything raised inside the loop left the file open, + # which on a watchdog process is exactly when someone goes looking for what it recorded. + with open(args.out, "w", buffering=1) as out: + started = time.time() + empty_rounds = 0 + while True: + now = time.time() + try: + pids = [int(p) for p in cgroup.joinpath("cgroup.procs").read_text().split()] + except OSError: + break # scope torn down -> pass finished + + if not pids: + empty_rounds += 1 + if empty_rounds > 5: + break + else: + empty_rounds = 0 + + procs = [sample for sample in (proc_sample(pid) for pid in pids) if sample] + record = { + "t": round(now - started, 2), + "cg_mem_mb": read_int(cgroup / "memory.current") / 1048576, + "cg_peak_mb": read_int(cgroup / "memory.peak") / 1048576, + "mem_avail_mb": 0, + "swap_used_mb": 0, + "procs": procs, + } + try: + meminfo = Path("/proc/meminfo").read_text() + info = {} + for line in meminfo.splitlines(): + key, _, rest = line.partition(":") + info[key] = int(rest.split()[0]) + record["mem_avail_mb"] = info.get("MemAvailable", 0) / 1024 + record["swap_used_mb"] = (info.get("SwapTotal", 0) + - info.get("SwapFree", 0)) / 1024 + except (OSError, ValueError, IndexError): + pass + + out.write(json.dumps(record) + "\n") + time.sleep(max(0.0, args.interval - (time.time() - now))) + print("sampler done", file=sys.stderr, flush=True) return 0 diff --git a/pipeline/profile/stamp.py b/pipeline/profile/stamp.py index 1d0c837..8100f03 100644 --- a/pipeline/profile/stamp.py +++ b/pipeline/profile/stamp.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Prefix each stdout line with wall-clock + elapsed seconds. shade_planet.py already prints a marker at every stage boundary ("warp height -> 3857 ...", diff --git a/pipeline/profile/watchdog.py b/pipeline/profile/watchdog.py index 7b0bb08..8653fb6 100644 --- a/pipeline/profile/watchdog.py +++ b/pipeline/profile/watchdog.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Watch the instrumented pass and EXIT when something worth reporting happens. The harness re-invokes Claude when a background task exits, so "exit on event" is how a @@ -104,9 +103,9 @@ def last_sample(samples: Path): def summarise(sample) -> str: if not sample: return " (no sample yet)" - out = [f" t={sample['t']:.0f}s cgroup {sample['cg_mem_mb']:.0f} MB " - f"(peak {sample['cg_peak_mb']:.0f}) avail {sample['mem_avail_mb']:.0f} MB " - f"swap {sample['swap_used_mb']:.0f} MB"] + out = [(f" t={sample['t']:.0f}s cgroup {sample['cg_mem_mb']:.0f} MB " + f"(peak {sample['cg_peak_mb']:.0f}) avail {sample['mem_avail_mb']:.0f} MB " + f"swap {sample['swap_used_mb']:.0f} MB")] for proc in sample["procs"]: if proc["comm"] in ("perf",): continue diff --git a/pipeline/raster_io.py b/pipeline/raster_io.py index ee037c0..1d7ada3 100644 --- a/pipeline/raster_io.py +++ b/pipeline/raster_io.py @@ -1,13 +1,14 @@ """Single home for the raster conventions every windowed writer shares. -one call site and been missed at its siblings four separate times — float32 + -windowed reads (composite had it, hillshade didn't), warp-once (lakedepth had -it, snow didn't), NUM_THREADS (warps had it, writers didn't), and the pyright -ignore for rasterio's untyped Window (fuse/render_prep had it, four new sites -didn't). One fix, one home; tests/test_raster_io.py pins the adoption. +The same fix has landed at one call site and been missed at its siblings four separate times — +float32 + windowed reads (composite had it, hillshade didn't), warp-once (lakedepth had it, snow +didn't), NUM_THREADS (warps had it, writers didn't), and the pyright ignore for rasterio's untyped +Window (fuse/render_prep had it, four new sites didn't). One fix, one home; tests/test_raster_io.py +pins the adoption. """ -from typing import Any, Iterator +from collections.abc import Iterator +from typing import Any from rasterio.windows import Window @@ -39,3 +40,13 @@ def band_window(width: int, row0: int, row1: int) -> Window: no py.typed and its old-style attrs __init__ is invisible to the checker. """ return Window(0, row0, width, row1 - row0) # pyright: ignore[reportCallIssue] + + +def column_window(height: int, col0: int, col1: int) -> Window: + """A full-height rasterio Window over columns col0..col1 — `band_window` transposed. + + Here rather than at its caller for the reason in the module docstring: this is the second shape + of window this pipeline builds, and the pyright ignore above is only a single home while every + shape is built through this module. + """ + return Window(col0, 0, col1 - col0, height) # pyright: ignore[reportCallIssue] diff --git a/pipeline/render/cast_shadow.py b/pipeline/render/cast_shadow.py index e5206c8..4ab6aab 100644 --- a/pipeline/render/cast_shadow.py +++ b/pipeline/render/cast_shadow.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Directional cast shadows — the one term a hillshade structurally cannot produce. `gdaldem hillshade` (and our `hillshade.py`) is a purely LOCAL operator: it sees one pixel's @@ -55,11 +54,17 @@ def shadow_mask(heights: np.ndarray, zfactor: float | np.ndarray, m_per_px: floa reach_px: int = 200) -> np.ndarray: """Occluded fraction of the sun's disc, 0.0 (fully lit) .. 1.0 (fully shadowed). - `heights` is metres on a north-up grid of `m_per_px` map units. `zfactor` is the vertical - exaggeration, scalar or a column vector of shape (rows, 1) for the per-latitude Mercator - correction (`relief.mercator_zfactor` = EXAG / cos(lat)) — the same value and the same shape - the hillshade uses, so the two terms exaggerate identically. They must agree: a shadow cast by - 15x relief onto terrain shaded at some other exaggeration is visibly wrong. + `heights` is metres on a north-up grid of `m_per_px` MAP units, and `zfactor` is what converts + that mismatch away — it is the vertical exaggeration already divided by the map-unit-to-ground + ratio, scalar or a column vector of shape (rows, 1) carrying the per-latitude Mercator term. + Hand it exactly what the hillshade uses, because the two terms must exaggerate identically: a + shadow cast by 15x relief onto terrain shaded at some other exaggeration is visibly wrong. + + THIS FUNCTION THEREFORE NEEDED NO CHANGE FOR A SECOND BODY, and that is worth stating rather + than leaving to be rediscovered. The tangent it accumulates is `zfactor * dh / (d * m_per_px)`, + so a body whose map units are not ground metres is corrected the moment its scale reaches the + z-factor — the same one number fixing the shading also fixes the shadows, and a second + correction applied here would double it. `reach_px` truncates the march. A shadow longer than this is silently cut short, so it is a real quality/cost lever and not a safety limit: cost is O(reach_px) full-array passes. At the @@ -93,8 +98,8 @@ def shadow_mask(heights: np.ndarray, zfactor: float | np.ndarray, m_per_px: floa # horizon at exactly 0 degrees, which is indistinguishable from a sun at the horizon. steepest = np.full(exaggerated.shape, -np.inf, dtype=np.float32) for distance in range(1, reach_px + 1): - row_offset = int(round(row_step * distance)) - column_offset = int(round(column_step * distance)) + row_offset = round(row_step * distance) + column_offset = round(column_step * distance) upsun = padded[reach_px + row_offset:reach_px + row_offset + rows] if column_offset: # np.roll(a, s)[i] == a[i - s], so the shift is negated to sample TOWARD the sun. diff --git a/pipeline/render/hillshade.py b/pipeline/render/hillshade.py index 16512be..e4bf1d0 100644 --- a/pipeline/render/hillshade.py +++ b/pipeline/render/hillshade.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Custom hillshade with a per-row (latitude-varying) z-factor for Web Mercator grids. `gdaldem hillshade` takes a single scalar z-factor. A Web-Mercator planet needs the @@ -28,11 +27,10 @@ import numpy as np import rasterio -from pipeline import bodies, mercator +from pipeline import mercator from pipeline.raster_io import GTIFF_CREATE, band_window, row_bands from pipeline.render import cast_shadow - # The hero's fill sun, ported to the tiles (scene_build.FILL_ROTATION (30, 0, 135) -> 60 deg up # from the SE; FILL_ANGLE 10; use_shadow off). "Shadowless" used to reproduce for free, because a # hillshade has no cast shadows at all; since `cast_shadow` landed it is upheld deliberately -- @@ -101,9 +99,11 @@ def combine_fill(main: np.ndarray, fill: np.ndarray, strength: float, altitude: def _latitude_of_rows(transform, row_indices: np.ndarray) -> np.ndarray: """Latitude (degrees) of pixel-row centres from an EPSG:3857 geotransform.""" merc_y = transform.f + (row_indices + 0.5) * transform.e # transform.e < 0 (north-up) - # Still Earth-bound at this one explicit site, where it used to be a bare literal duplicated in - # snow.py. The conversion itself now lives in `mercator` and takes the sphere it is projected on. - return mercator.latitude_at(merc_y, bodies.EARTH.mercator_radius_m) + # NOT the body's radius, and this is not a site left to parameterise. The row's latitude is a + # property of the GRID, and every grid here is EPSG:3857 whatever planet the heights describe — + # so the sphere is the projection's, permanently. Reading it off the body registry (which this + # did) invited the fix that would have made Mars's rows report latitudes 31 degrees out. + return mercator.latitude_at(merc_y, mercator.WEB_MERCATOR_RADIUS_M) def hillshade_array(heights: np.ndarray, cellsize: float, zfactor, @@ -149,9 +149,21 @@ def hillshade_array(heights: np.ndarray, cellsize: float, zfactor, def per_row_zfactor_hillshade(height_path, out_path, exaggeration: float = 15.0, altitude: float = 45.0, azimuth: float = 315.0, window_rows: int = 256, fill_strength: float = 0.0, - shadow_strength: float = 0.0, shadow_reach_px: int = 0) -> None: + shadow_strength: float = 0.0, shadow_reach_px: int = 0, + *, ground_scale: float) -> None: """Stream a seamless, per-latitude-z hillshade over a whole EPSG:3857 height raster. + `ground_scale` is how many real ground metres one map unit of this raster is worth + (`bodies.ground_metres_per_mercator_unit`), and it is KEYWORD-ONLY AND REQUIRED because there is + no safe default. A slope is a rise in body metres over a run in map units, and every raster here + is EPSG:3857 whatever planet the elevations came from — so on Earth the two agree exactly and on + Mars a map unit is worth 0.53 of a ground metre, making the true relief 1.878x steeper than the + grid says. Defaulting it to 1.0 would hand every future body Earth's answer silently, which is + the one failure mode with no symptom: the shading comes out plausible at every latitude. + + It divides the z-factor for the same reason `cos(lat)` does — both convert a map run into a + ground run, one constant across the raster and one varying down it. + `fill_strength` mixes in the hero's fill sun (see `combine_fill`); 0.0 skips the second hillshade entirely and is bit-identical to a no-fill pass. It defaults OFF so the fill is opt-in at the call site, matching how `exaggeration`/`altitude`/`azimuth` are already passed @@ -201,7 +213,8 @@ def per_row_zfactor_hillshade(height_path, out_path, exaggeration: float = 15.0, mode="edge") out_rows = np.arange(row0, row1) latitude = np.clip(_latitude_of_rows(src.transform, out_rows), -85.05, 85.05) - zfactor = (exaggeration / np.cos(np.radians(latitude))).reshape(-1, 1) + zfactor = (exaggeration + / (ground_scale * np.cos(np.radians(latitude)))).reshape(-1, 1) # hillshade_array's contract is exactly ONE halo row; a deeper shadow halo is # trimmed back to it here rather than by widening that function's contract. local = block if halo == 1 else block[halo - 1:block.shape[0] - (halo - 1)] @@ -210,8 +223,8 @@ def per_row_zfactor_hillshade(height_path, out_path, exaggeration: float = 15.0, block_rows = np.arange(row0 - halo, row1 + halo) block_latitude = np.clip(_latitude_of_rows(src.transform, block_rows), -85.05, 85.05) - block_zfactor = (exaggeration - / np.cos(np.radians(block_latitude))).reshape(-1, 1) + block_zfactor = (exaggeration / (ground_scale * np.cos( + np.radians(block_latitude)))).reshape(-1, 1) shadow = cast_shadow.shadow_mask(block, block_zfactor, cellsize, altitude, azimuth, shadow_reach_px) shaded = shaded * (1.0 - shadow_strength * shadow[halo:halo + (row1 - row0)]) diff --git a/pipeline/render/lake_depth.py b/pipeline/render/lake_depth.py index 34352d5..0b52ccd 100644 --- a/pipeline/render/lake_depth.py +++ b/pipeline/render/lake_depth.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Tile lake-depth layer: GLOBathy modelled depth -> a per-pixel depth field in metres. Sits beside snow.py by design. Lake depth is a TINT-ONLY rendering input, never terrain: at @@ -24,7 +23,6 @@ """ import subprocess -from pathlib import Path import numpy as np import rasterio @@ -64,6 +62,25 @@ def warp_depth(bounds, width, height, out_path, vrt=LAKE_VRT): return np.where(np.isfinite(depth) & (depth > 0.0), depth, 0.0).astype("float32") +def warp_depth_raster(bounds, width, height, out_path, vrt=LAKE_VRT): + """Warp GLOBathy onto a whole Web-Mercator grid, leaving the result on disk. + + The planet-tier twin of `warp_depth` above, which hands the array back for the region path. + bounds = (left, bottom, right, top) in EPSG:3857. No `-s_srs`, unlike the NetCDF and GeoTIFF + warps beside it: the VRT carries its own. Tiled/DEFLATE/BIGTIFF because the target is a global + grid, which is the whole difference between the two. + """ + left, bottom, right, top = bounds + _run(["gdalwarp", "-q", "-t_srs", "EPSG:3857", + "-te", repr(left), repr(bottom), repr(right), repr(top), + "-ts", str(width), str(height), + "-srcnodata", str(GLOBATHY_NODATA), "-dstnodata", "0", + "-r", "bilinear", "-ot", "Float32", "-co", "TILED=YES", + "-co", "COMPRESS=DEFLATE", "-co", "BIGTIFF=YES", + "-co", "NUM_THREADS=ALL_CPUS", vrt, out_path]) + return out_path + + def lakes_only(depth, watercode): """Zero the depth field off watermask class 2 (inland lake). diff --git a/pipeline/render/layer_producers.py b/pipeline/render/layer_producers.py new file mode 100644 index 0000000..c7247b0 --- /dev/null +++ b/pipeline/render/layer_producers.py @@ -0,0 +1,467 @@ +"""Which producer builds a body's surface layer for the tile composite, and what that one reads. + +ONE ANSWER TO "HOW DOES THIS BODY MAKE THAT LAYER" at the Mercator tier. `layers.py` says what a +layer is and which stages read it; `Body.surface_layers` says which ones a planet has; this says who +builds each one, out of what, and how the result becomes a number the composite can blend. + +THE CAP TIER'S REGISTRY IS `render/perennial_ice.py`, whose docstring holds the argument this module +inherits rather than restates: a producer is CODE, so bodies differ in machinery and not in +constants. Two registries and not one because the tiers key differently — that one by +`(body, pole)`, this one by `(body, layer)` — and a cap producer paints an AEQD disc where these +paint one window of a Mercator strip. Merging them is a separate claim. + +THE HOLE THIS CLOSES, AND THE TIDY THAT REOPENS IT. Every source below used to be a module constant +at a fixed global path, asked `source.exists()` at the warp gate. That question reads "have we +downloaded Earth's data" whatever body is being built, so a second planet declaring one of these +layers passed the gate on Earth's file and had Earth's cryosphere warped onto its own grid — same +latitudes, no missing file, no error, a plausible planet. Asked of the body's own producer, the disk +question is about the body's own files. Collapsing it back to a bare `.exists()` looks like removing +a redundant check and is silent on the only body anyone builds. + +A PRODUCER OWNS THE WHOLE ACT — the sources, the build, and the per-window arithmetic — because a +body that registers one half inherits Earth's other half in silence. A Martian raster built by a +Martian producer and then run through Earth's `unpack_persistence` and Earth's latitude ramp is ice +graded by NSIDC's packing convention, which no type and no test could notice. + +`contribution` RUNS ON A WORKER THREAD AND MUST TOUCH NO FILESYSTEM. `shade_planet` gathers every +read on the main thread precisely so the compute stays pure; a producer opening a file here would +put GDAL back where rasterio is not thread-safe. + + from pipeline.render import layer_producers + producer = layer_producers.producer_for(body, layers.SEA_ICE) +""" + +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import numpy as np + +from pipeline import bodies, layers +from pipeline.acquire import download_sim3292 +from pipeline.render import lake_depth, mars_ice, palette, seaice, snow, viking_luma + + +@dataclass(frozen=True) +class LayerBuild: + """The 3857 grid a producer must land its sources on, and where to leave the result. + + Every field is passed to every producer whether or not it reads it — `band_rows` means nothing + to a rasterize — on `perennial_ice.CapIceInputs`' rule: a request whose shape depended on which + producer was registered would have to be built per body, which puts a body branch back into the + caller this seam exists to remove. + """ + + #: (left, bottom, right, top) in EPSG:3857, the height raster's own bounds. + bounds: tuple[float, float, float, float] + width: int + height: int + #: The file to write, already named from `Layer.warped_basename` under this body's work dir. + out: Path + #: The band height for a producer that warps a coarse source in latitude strips. It equals the + #: composite's window height, which is what makes a banded mosaic byte-identical to the + #: per-window warps it replaced (`snow.warp_persistence_raster`). + band_rows: int + + +@dataclass(frozen=True) +class LayerWindow: + """One window of the composite, as a producer is allowed to see it. + + `raw` is this producer's OWN slice and None when its raster was never built; the masks and the + geometry are shared, and are here for the producers that grade against them rather than against + a file. Same all-fields-always rule as `LayerBuild` above. + """ + + raw: np.ndarray | None + #: The planet seam's water classes, or None on a body whose seam emitted no water mask. + watercode: np.ndarray | None + #: `~(ocean | water)` for this window — the composite's own definition of land. + land: np.ndarray + #: True latitude in degrees per ROW (1-D). A Mercator window has rows of constant latitude, + #: which is what separates this from the cap tier's per-pixel field. + latitude: np.ndarray + #: The window's latitude span in 3857 metres, for a producer whose ramp needs the extent rather + #: than the per-row value. + top: float + bottom: float + + +@dataclass(frozen=True) +class LayerProducer: + """One body's answer for one layer at the composite tier: what it reads, builds and contributes. + + `sources` is a CALLABLE and never a tuple literal, for the reason `perennial_ice.CapIce` records + at length: a literal evaluates its paths once, at import, so a caller that redirects the data + root is answered with the path from before the redirect. Every path here hangs off `paths.DATA`. + """ + + sources: Callable[[], tuple[Path, ...]] + #: Land the sources on the grid and leave the file at `LayerBuild.out`. Prints its own line, so + #: the pass says "rasterize" where it rasterizes and "warp" where it warps. + build: Callable[[LayerBuild], None] + #: This layer's number for one window, or None when it has nothing to add. Returning None rather + #: than zeros keeps a layer that contributes nothing out of the union entirely. + contribution: Callable[[LayerWindow], "np.ndarray | None"] + #: The `(sunlit, shadowed)` white `shade.composite` paints this producer's alpha with, or None + #: where the layer's number is not painted as a white at all (lake depth, which takes a ramp). + #: + #: Each end may be a bare RGB triple or an array shaped per row, so a producer whose material + #: changes across the window says so itself. Mars's does: its two poles measure as different + #: colours, 1.053 against 1.291 in red:violet, and one producer covers both hemispheres. + #: + #: WHY THIS IS THE PRODUCER'S AND NOT THE COMPOSITOR'S. It used to be a module global read + #: inside `shade.composite`, on the argument that one white serves every body's perennial ice. + #: That was true while Earth was the only body painting any, and it stopped being true twice + #: over — once when a second planet arrived and inherited Earth's white by omission, and again + #: when that planet turned out to need two of its own. A global read cannot show a reader that + #: either fact exists, which is the whole failure: the code that decided to paint a pixel is the + #: only code that knows what the pixel is made of. + paint: Callable[[LayerWindow], "tuple[Any, Any] | None"] + #: The constants THIS producer's own arithmetic reads, for the composite's freshness recipe. + #: + #: A tunable reaching a pixel and reaching no recipe leaves a stale composite looking fresh; one + #: recorded by a body that cannot evaluate it restages that body for output it could not have + #: changed. Both are silent, and a single gate cannot separate them once two bodies paint the + #: same layer by different arithmetic — the gate asks whether the body paints ice, which is + #: right, and the values behind it answered for Earth, which stopped being right. So the code + #: that READS a constant is the code that declares it. + #: + #: THE WHITES USED TO BE EXCLUDED FROM HERE AND ARE NOW INCLUDED, deliberately. The rule was + #: that `shade.composite` paints any alpha with `palette.SNOW_RGB` on every body alike, so the + #: whites were live per LAYER and `composite_params` gated them on the layer. Mars measures two + #: whites of its own, one per pole, so "one white per layer" is simply false and the gate would + #: record Earth's values for a body that paints neither. What a producer PAINTS with is now as + #: much its own declaration as what it grades with — see `paint` above. + #: + #: Still not here: `palette.LAKE_*`. Lake depth really is one ramp on every body that has lakes, + #: it is not a white, and no second instance has contradicted it — so it stays in + #: `composite_params` until one does. + recipe: Callable[[], dict[str, Any]] + #: The constants `build` bakes INTO the raster, which is a different set from `recipe` above and + #: cannot share its machinery. + #: + #: WHY THE SPLIT IS NOT TIDINESS. `recipe` reaches `composite_params`, so changing one of its + #: values restages the composite — which is exactly right for a constant read per window, because + #: the rerun re-reads it. A constant consumed at BUILD time is already frozen into the file on + #: disk, and `warp_needs_rebuild` is closed over PATHS: no Python value can reach it. Recording a + #: build-time constant in `recipe` alone therefore produces the worst available outcome — a full + #: composite restage that repaints from the unchanged raster and lands the same wrong pixels, + #: looking for all the world like the change applied. + #: + #: Materialised by `warp_inputs` through `freshness.write_if_changed`, whose whole purpose is to + #: let a constant stand in as an mtime dependency: the file moves if and only if a value moved. + #: + #: EMPTY IS THE ANSWER FOR EVERY EARTH PRODUCER and that is a property of them rather than an + #: omission — all four are pure transport, landing a source on the grid and storing it raw, with + #: every grading constant read later in `contribution`. An empty dict writes no file and adds no + #: source, so Earth's gate stays byte-for-byte what it was and the optional-layer warps do not + #: restage. Mars's ice is the first build to grade before it writes. + build_recipe: Callable[[], dict[str, Any]] + + +def _build_lake_depth(request: LayerBuild) -> None: + """GLOBathy onto the grid, warped ONCE here rather than per window. + + It is an 83k-source VRT and a many-source VRT re-reads every source on each touch, the same + reason the tiler materialises before cutting. + """ + print("warp lake depth -> 3857 ...", flush=True) + request.out.unlink(missing_ok=True) + lake_depth.warp_depth_raster(request.bounds, request.width, request.height, request.out) + + +def _build_persistence(request: LayerBuild) -> None: + """NSIDC-0791 onto the grid, storing the RAW PACKED Float32 and banded at the window height. + + Banded so each strip is exactly the per-window warp it replaced; `snow.warp_persistence_raster` + holds why a single whole-grid warp decimates a source this coarse. + """ + print("warp snow persistence -> 3857 (banded) ...", flush=True) + request.out.unlink(missing_ok=True) + snow.warp_persistence_raster(request.bounds, request.width, request.height, request.out, + band_rows=request.band_rows) + + +def _build_glaciers(request: LayerBuild) -> None: + """RGI 7.0 burnt onto the grid as a 0/1 Byte mask — the one build here that is not a warp.""" + print("rasterize RGI glaciers -> 3857 ...", flush=True) + snow.rasterize_glaciers_raster(request.bounds, request.width, request.height, request.out) + + +def _build_sea_ice(request: LayerBuild) -> None: + """The 1991-2020 frequency climatology onto the grid, banded like persistence and for its + reason: a single whole-grid warp of a 0.1 degree source decimates the ice edge.""" + print("warp sea-ice frequency -> 3857 (banded) ...", flush=True) + request.out.unlink(missing_ok=True) + seaice.warp_seaice_raster(request.bounds, request.width, request.height, request.out, + band_rows=request.band_rows) + + +def _earth_lake_depth(window: LayerWindow) -> "np.ndarray | None": + """Depth in metres, zeroed off watermask class 2 — the one contribution that is not an alpha. + + The watercode cannot be None while the depth raster is not: `planet_seam` refuses a body that + declares this layer with no water mask, which is what `Layer.requires_raster` records. + """ + if window.raw is None or window.watercode is None: + return None + return lake_depth.lakes_only(window.raw, window.watercode) + + +def _earth_perennial_ice(window: LayerWindow) -> "np.ndarray | None": + """NSIDC-0791 persistence, plus the forced Antarctic patch that has no dataset behind it. + + BOTH HALVES ARE THIS LAYER'S ANSWER, exactly as the cap tier's two poles are: NSIDC-0791 is + NH-only and RGI region 19 is excluded, so persistence and glaciers are both zero over the + continent and it would render on the tan LAND ramp. The patch therefore rides the layer's + DECLARATION and not its raster — no file could ever switch a latitude-and-land rule off — which + is why this returns an array even when `raw` is None. + + float64 in both branches. `snow_alpha` returns float64 and `antarctic_snow_mask` float32, so the + zeros base is what keeps the two paths feeding `shade.composite` the same dtype. + """ + if window.raw is None: + persistence_alpha = np.zeros(window.land.shape, dtype=float) + else: + persistence_alpha = snow.snow_alpha(snow.unpack_persistence(window.raw), + window.top, window.bottom) + return np.maximum(persistence_alpha, + snow.antarctic_snow_mask(window.land, window.latitude)) + + +def _earth_glaciers(window: LayerWindow) -> "np.ndarray | None": + """RGI's 0/1 mask as float, to be maxed into the same union perennial ice feeds.""" + return None if window.raw is None else window.raw.astype(float) + + +def _earth_sea_ice(window: LayerWindow) -> "np.ndarray | None": + """Frequency -> smoothstep, toned in the southern hemisphere. + + South of the equator the pack takes the cap's fainter, pulled-in fringe (`seaice.SH_ICE_*`), or + the full-strength Antarctic belt reads as a bright halo — proven on the cap. No window straddles + both hemispheres' ice and the equator is ice-free, so the per-row split is exact. + """ + if window.raw is None: + return None + frequency = seaice.unpack_seaice(window.raw) + ice_alpha = seaice.ice_alpha(frequency) + southern = window.latitude < 0.0 + if southern.any(): + toned = seaice.ice_alpha(frequency, ice_lo=seaice.SH_ICE_LO, + ice_max_alpha=seaice.SH_ICE_MAX_ALPHA) + ice_alpha = np.where(southern[:, None], toned, ice_alpha) + return ice_alpha + + +def _no_tunables() -> dict[str, Any]: + """A producer whose arithmetic has no constants of its own. + + Empty is a statement and not an omission, on `CapIce.sources`' rule. Lake depth is a warp and + the glacier mask a rasterize: both land a source on the grid and hand the number through + unmodified, so there is nothing here to re-tune. What paints them — the lake ramp, the whites — + belongs to the compositor. A producer that grows a constant declares it here, and the bodies + that read it restage; that is the field working, not a change of policy. + """ + return {} + + +def _earth_white(_window: "LayerWindow | None" = None) -> tuple[Any, Any]: + """Earth's one ice white, shared by its perennial ice, its glaciers and its cap producers. + + ONE HOME, DECLARED BY EACH READER RATHER THAN COPIED INTO IT. Earth's two composite-tier ice + layers feed a single union and must agree, and the caps must agree with both or the crossfade + changes colour across the seam. Reading `palette` from one function keeps that a fact rather + than a coincidence between four registry entries. + """ + return palette.SNOW_RGB, palette.SNOW_SHADOW_RGB + + +def _earth_perennial_ice_recipe() -> dict[str, Any]: + """What `_earth_perennial_ice` reads: `snow_alpha`'s latitude ramp, and the white it paints in. + + The white is here because this producer declares it — see `LayerProducer.paint`. Earth's + glacier producer records the identical pair from the identical home, and `produced` merges by + key, so the duplicate is one value seen twice rather than two values that can drift. + """ + lit, shadow = _earth_white() + return {"snow_ramp_lat_lo": snow.RAMP_LAT_LO, + "snow_ramp_lat_hi": snow.RAMP_LAT_HI, + "snow_ramp_low_min": snow.RAMP_LOW_MIN, + "snow_ramp_low_max": snow.RAMP_LOW_MAX, + "snow_ramp_band": snow.RAMP_BAND, + "snow_rgb": lit, "snow_shadow_rgb": shadow} + + +def _earth_glaciers_recipe() -> dict[str, Any]: + """The glacier mask is pure transport, so the white is the whole of what this producer reads. + + IT PAINTS WITHOUT GRADING, which is exactly the case the old layer gate got wrong: it recorded + the white under `perennial_ice`, so a body with glaciers and no perennial ice would have painted + a white it never recorded. No such body exists yet; the gate was still one declaration away from + being wrong, and asking the producer removes the question. + """ + lit, shadow = _earth_white() + return {"snow_rgb": lit, "snow_shadow_rgb": shadow} + + +def _earth_sea_ice_recipe() -> dict[str, Any]: + """`ice_alpha`'s frequency ramp, in both hemispheres' tunings — what `_earth_sea_ice` reads. + + The southern pair is here rather than under a hemisphere gate because one producer evaluates + both: `_earth_sea_ice` calls `ice_alpha` twice and selects per row, so a window that straddles + no southern ice still ran on a body whose producer can reach them. + """ + lit, shadow = seaice.ice_white() + return {"ice_lo": seaice.ICE_LO, "ice_band": seaice.ICE_BAND, + "ice_max_alpha": seaice.ICE_MAX_ALPHA, + "sh_ice_lo": seaice.SH_ICE_LO, + "sh_ice_max_alpha": seaice.SH_ICE_MAX_ALPHA, + "ice_rgb": lit, "ice_shadow_rgb": shadow} + + +def _mars_ice_sources() -> tuple[Path, ...]: + """The brightness field and both mapped units — every file the Mars build opens.""" + return (viking_luma.luma_path(), + *(download_sim3292.unit_path(unit) for unit in mars_ice.NORTH_UNITS)) + + +def _build_mars_ice(request: LayerBuild) -> None: + """Viking luma graded, cut to the mapped units and feathered — the one build that is not a warp. + + THE ONLY BUILD HERE THAT GRADES BEFORE IT WRITES. Its siblings land a source on the grid and + store it raw, leaving every constant to `contribution`; this one bakes `FEATHER_KM` and + `ALPHA_LEVELS` into the file, because the feather is a distance transform and a distance + transform cannot be computed from one window — a pixel's nearest ice can lie outside whatever + slice is in hand. That asymmetry is what `build_recipe` exists for, and why this producer's + `recipe` is empty rather than carrying those two. + """ + print("grade Viking luma -> Mars ice alpha (polar bands) ...", flush=True) + request.out.unlink(missing_ok=True) + mars_ice.build_alpha_raster( + field=viking_luma.luma_path(), field_nodata=viking_luma.NODATA, + bounds=request.bounds, width=request.width, height=request.height, out=request.out, + ground_metres_per_map_unit=bodies.ground_metres_per_mercator_unit(bodies.get("mars"))) + + +def _mars_perennial_ice(window: LayerWindow) -> "np.ndarray | None": + """The alpha the build already computed, as float64 — this window and nothing else. + + NOTHING IS GRADED HERE, the mirror image of Earth's producer, and it follows from where the + feather has to run rather than from taste. None when the raster was never built, per the + contract: no file, no ice, and no zeros pushed into the union to be blended against. + + float64 because `shade.composite` blends whichever body's answer it is handed alongside Earth's, + and a narrower dtype from one of them shifts the other's blend sub-DN. + """ + if window.raw is None: + return None + return np.asarray(window.raw, dtype=float) + + +def _mars_ice_white(window: LayerWindow) -> tuple[Any, Any]: + """Mars's white, chosen PER ROW, because its two poles are not the same colour. + + Measured off the Viking mosaic over each pole's own painted extent, weighted by the alpha this + producer hands over: red:violet 1.053 north against 1.291 south. Their own surrounding ground + reads 1.231 and 1.807, and normalising each cap by its own ground leaves most of the gap intact + — so the difference belongs to the ice, not to what surrounds it or to how it was imaged. + + ONE PRODUCER SPANS BOTH HEMISPHERES HERE, unlike the cap tier where the pole is the registry + key, which is exactly why the paint has to be able to vary within a window rather than being one + constant per producer. Returned as `(3, H, 1)`, which the blend broadcasts for free. + + The equator is a hemisphere boundary and never an ice boundary: Mars carries no ice within 76 + degrees of it, so the row this splits on is one no alpha reaches. + """ + northern = np.asarray(window.latitude) >= 0.0 + def per_row(north: Any, south: Any) -> np.ndarray: + return np.where(northern[None, :, None], + np.asarray(north, dtype=np.float32).reshape(3, 1, 1), + np.asarray(south, dtype=np.float32).reshape(3, 1, 1)) + return (per_row(palette.MARS_ICE_WHITE["north"][0], palette.MARS_ICE_WHITE["south"][0]), + per_row(palette.MARS_ICE_WHITE["north"][1], palette.MARS_ICE_WHITE["south"][1])) + + +def _mars_ice_recipe() -> dict[str, Any]: + """Both whites, flat and per pole, so a re-tune of either one restages the composite. + + FOUR KEYS RATHER THAN EARTH'S TWO, and the asymmetry is the point: a recipe records what its own + body evaluates, and Mars evaluates two pairs where Earth evaluates one. Recording Earth's shape + here would have to invent a single Martian white that no pixel is painted with. + + Flat keys rather than a nested dict so a `git log -S "snow_rgb_south"` finds the value's history + the same way it finds Earth's. + """ + return {f"snow_{end}_{pole}": list(value) + for pole, pair in sorted(palette.MARS_ICE_WHITE.items()) + for end, value in (("rgb", pair[0]), ("shadow_rgb", pair[1]))} + + +def _mars_ice_build_recipe() -> dict[str, Any]: + """The two constants `_build_mars_ice` freezes into its raster. + + The luma WEIGHTS are deliberately absent, and covered rather than forgotten: `render/viking_luma` + records them in its own recipe, so a weight change restages that stage, moves the field's mtime, + and reaches this raster as a moved SOURCE. Recording them here as well would rebuild correctly + and claim the coupling lives in two places. + """ + return {"mars_feather_km": mars_ice.FEATHER_KM, + "mars_alpha_levels": {pole: list(levels) + for pole, levels in sorted(mars_ice.ALPHA_LEVELS.items())}} + + +#: Every composite-tier producer that ships, by (body slug, layer name). +#: +#: Five entries and five MECHANISMS — a banded NetCDF warp, a vector rasterize, a banded GeoTIFF +#: warp, a nodata-masked bilinear warp, and Mars's graded-and-feathered polar bands — which is what +#: gives the parameterisation real instances instead of one shape repeated with a different constant. +PRODUCER_BY_BODY_LAYER: dict[tuple[str, str], LayerProducer] = { + ("earth", layers.LAKE_DEPTH.name): LayerProducer( + sources=lambda: (lake_depth.LAKE_VRT,), + # None, not a white: this producer's number is a DEPTH, graded by the lake ramp. The field + # existing and being answered "not applicable" is what keeps that visible. + build=_build_lake_depth, contribution=_earth_lake_depth, paint=lambda _window: None, + recipe=_no_tunables, build_recipe=_no_tunables), + ("earth", layers.PERENNIAL_ICE.name): LayerProducer( + sources=lambda: (snow.SP_NC,), + build=_build_persistence, contribution=_earth_perennial_ice, paint=_earth_white, + recipe=_earth_perennial_ice_recipe, build_recipe=_no_tunables), + ("earth", layers.GLACIERS.name): LayerProducer( + sources=lambda: (snow.RGI_GPKG,), + build=_build_glaciers, contribution=_earth_glaciers, paint=_earth_white, + recipe=_earth_glaciers_recipe, build_recipe=_no_tunables), + ("earth", layers.SEA_ICE.name): LayerProducer( + sources=lambda: (seaice.SEAICE_SRC,), + # `seaice.ice_white`, not a literal: the cap tier reads that same function directly, so the + # sentence "sea ice is painted in this pair" has one home across both tiers. + build=_build_sea_ice, contribution=_earth_sea_ice, + paint=lambda _window: seaice.ice_white(), + recipe=_earth_sea_ice_recipe, build_recipe=_no_tunables), + ("mars", layers.PERENNIAL_ICE.name): LayerProducer( + sources=_mars_ice_sources, + build=_build_mars_ice, contribution=_mars_perennial_ice, paint=_mars_ice_white, + # NO LONGER EMPTY: this producer grades nothing per window, but it does declare what it is + # painted in, and those two whites are re-tunable. What it bakes into the raster is a + # different set again, which is what `build_recipe` tracks. + recipe=_mars_ice_recipe, build_recipe=_mars_ice_build_recipe), +} + + +def producer_for(body: bodies.Body, layer: layers.Layer) -> LayerProducer: + """The producer this body builds `layer` with, at the composite tier. + + RAISES RATHER THAN FALLING BACK, on `perennial_ice.cap_ice`'s rule and for its sharper reason: a + body inheriting Earth's producer by omission warps Earth's data onto another world and paints it + as that world's, with nothing missing and nothing to report. Only asked of a body that declares + the layer, so the raise is unreachable in a correct configuration and is exactly a statement that + the two declarations disagree. + """ + try: + return PRODUCER_BY_BODY_LAYER[(body.name, layer.name)] + except KeyError: + raise KeyError( + f"{body.name} declares the {layer.name} layer but registers no composite producer; " + f"known: {sorted(PRODUCER_BY_BODY_LAYER)}" + ) from None diff --git a/pipeline/render/mars_ice.py b/pipeline/render/mars_ice.py new file mode 100644 index 0000000..5e4e158 --- /dev/null +++ b/pipeline/render/mars_ice.py @@ -0,0 +1,481 @@ +"""Mars's polar ice: which mapped units are ice, how icy each pixel of them is, and how the edge +between that and bare ground is softened. + +TWO FIELDS AND THEY COME FROM DIFFERENT PLACES. The extent says where white is drawn at all and comes +from a geologic map (`acquire/download_sim3292.py` holds why). The alpha says how white, and comes +from the Viking colour mosaic's luma (`ALPHA_LEVELS`). Keeping them separate is what lets the map +limit the CLAIM while the albedo supplies the VARIATION — the arm that let albedo do both was judged +and +rejected, roughly 45% of its ice falling outside the mapped unit and reading as seasonal frost caught +in a 6.5-year average. + +THE EXTENTS ARE CORROBORATED BY PUBLISHED AREAS, which is worth stating because a geologic unit +chosen by us could otherwise be an arbitrary polygon. North `lApc | Apu` integrates to 1,095,880 km² +against a published Planum Boreum / NPLD area of about 1,000,000 km² — and `lApc` ALONE is 596,893, +which would be 40% short, so the published figure independently supports including `Apu` in the +north. South `lApc` is 103,642 km² against a south polar residual cap of about 87,000. + +THE EXTENT IS ASYMMETRIC ON MEASUREMENT, NOT ON SYMMETRY. `lApc` is ice at both poles. `Apu` joins it +in the NORTH only, because the albedo puts northern `Apu` +0.13 to +0.16 above ordinary ground at +matched latitude — 55-80% of the way from bare ground to the residual cap — while southern `Apu` is +within +±0.04 of ordinary ground and covers 68.7% of that disc. Painting it south would whiten two thirds of +the view on no evidence. So a body's hemisphere is a real input here and not a tidy-up. Viking +reproduces that split independently at +4.47 scatters against OMEGA's +4.30, so the asymmetry does +not rest on the source that has since been withdrawn. + +AND THE GEOLOGY SAYS THE SAME THING INDEPENDENTLY, which is why the asymmetry is not a fudge: the +southern `Apu` polygon is 1,495,810 km², the south polar LAYERED DEPOSITS, a dusty stack rather than +a residual ice cap — so a unit the albedo finds indistinguishable from ordinary ground is one the +stratigraphy also says is not surface ice. Two lines of evidence, neither derived from the other. + +THE FEATHER IS DRAWN AND MUST NEVER BE DESCRIBED AS OBSERVED. The published linework was drawn while +viewing at 1:5,000,000 with 5 km vertex spacing, so a hard edge between two of those vertices reads +as faceted. `FEATHER_KM` softens that and nothing measured it — a distance transform wearing a +measurement's clothes is the `antarctic_snow_mask` mistake, one module over. + +HOW WIDE THAT IS ON SCREEN IS NOT A CONSTANT AND MUST NOT BE WRITTEN DOWN AS ONE. A pixel count +belongs to a tile ceiling and a latitude, neither of which this module owns. Both this note and the +acquirer's carried one — "~8 pixels per vertex" — and it was arithmetic at the EQUATOR, here under a +polar label; the ice sits at 76-85 degrees, where a Mercator pixel is a fraction of its equatorial +ground size, so the real count was many times that and doubles with every rung the ceiling gains. +Derive it from `bodies.ground_metres_per_mercator_unit` against the latitude in hand. + +THE ICE EDGE HAS SINCE BEEN MEASURED AND THAT IS A DIFFERENT QUANTITY — do not let the number migrate +into `FEATHER_KM`. OMEGA against signed distance to the south's mapped contact puts real ice fading +over about 12 ground km outside it, and the north has no albedo edge at all across 130 km. That +validates the EXTENT; the feather still anti-aliases a drawn boundary, so its anchor is the linework +and not the ice. The profile is not re-runnable — it was taken against OMEGA, which this project no +longer acquires — so treat the two figures above as the record rather than looking for the script. + +THE FEATHER IS THE ONE THING THAT CANNOT BE COMPUTED PER WINDOW. A distance transform is non-local: +a pixel's distance to the nearest ice can be owned by a pixel outside whatever slice is in hand, and +this project has already paid for that lesson on GLOBathy's shore distances. It is bounded here, +which is what rescues it — nothing past `FEATHER_KM` changes the answer — so a banded pass with a pad +wider than the feather is exact where the result is used, and only there. +""" + +import json +import subprocess +from collections.abc import Iterator +from functools import reduce +from pathlib import Path +from typing import Any + +import numpy as np +import rasterio +import rasterio.transform +from rasterio.windows import Window +from scipy import ndimage + +from pipeline import mercator, vector_raster +from pipeline.acquire import download_sim3292 +from pipeline.raster_io import row_bands + +#: The mapped units drawn as ice, per hemisphere. Both tuples are read by `extent_for`, and the +#: difference between them is the measurement in the module note rather than a spelling choice. +NORTH_UNITS: tuple[str, ...] = ("lApc", "Apu") +SOUTH_UNITS: tuple[str, ...] = ("lApc",) + +#: How far outside the mapped contact the ice fades to nothing, in GROUND kilometres. +#: +#: DRAWN, NOT MEASURED, AND IT IS ONE VERTEX SPACING — the only anchor this constant has ever had. +#: Its whole job is to stop a 5 km-vertex boundary stair-stepping, so it is anti-aliasing over a +#: contact the publisher mapped, not a gradient standing in for one nobody observed. +#: +#: THREE WRONG VALUES PRECEDED IT AND EACH LOOKED SETTLED. 25 km came from a superseded arm where the +#: feather WAS the gradient. 10 km replaced it and was then divided by AEQD MAP metres, which on Mars +#: are Earth metres, so what actually rendered was 5.33 km of ground under a label saying 10. The +#: scale a caller passes must therefore be GROUND metres per pixel — see `feather_alpha_bands`. +#: +#: Do not re-anchor it to the measured 12 km ice edge; that is the width of ice this design +#: deliberately does not paint, and the module note above holds why the two are different quantities. +FEATHER_KM = 5.0 + +#: Viking colour LUMA mapping to alpha 0 and 1, per pole: the median of ground that is neither mapped +#: unit, and the median of `lApc`. 8-bit Rec. 709 luma, so these live on 0..255 and not on 0..1. +#: +#: `lApc` ALONE SETS THE CAP LEVEL AT BOTH POLES, including the north where the extent also takes +#: `Apu`; and GROUND EXCLUDES BOTH UNITS AT BOTH POLES, including the south where `Apu` is not ice. +#: Neither is `extent` or `~extent`, and reading them as such moves the look. +#: +#: PINNED, NEVER RECOMPUTED, AND THAT IS CORRECTNESS RATHER THAN TIDINESS. The two tiers grade the +#: same ice over different pixel sets — an AEQD disc and a Mercator strip — so percentiles taken per +#: grid would disagree, and the cap and the tiles crossfade across 80-84 degrees where a disagreement +#: is visible as a step. That grid-dependence is measured, not feared: moving the cap edge 78 -> 80 +#: moved the north's cap level 11.4% while leaving the south's at 0.03 DN. +#: +#: MEASURED OVER `viking_luma`'s SHIPPED RASTER, not over a band built to answer the question. That +#: distinction moved the north by 0.07 DN and is the rule demonstrating itself once more: the +#: prototype's two polar bands were anchored at their own latitudes with square degree pixels, while +#: the shipped grid covers the sphere exactly at the publisher's pixel count, so their rows drift +#: sub-pixel apart with distance from the north pole. The look does not move — `lApc` mean alpha is +#: 0.813 north and 0.756 south either way — but the levels must describe the field that renders. +#: +#: Re-measure with `scripts/measure_viking_levels.py`, which is the only reproducer of these four +#: numbers, whenever the FIELD or the CAP GRID moves — both of which is what retired the OMEGA pair +#: that stood here. Its `--compare` mode is what refuses a drift between the field the levels were +#: taken from and the field the renderer reads. +ALPHA_LEVELS: dict[str, tuple[float, float]] = { + "north": (60.99, 205.58), + "south": (97.39, 174.01), +} + +#: Rec. 709 luma weights — the QUANTITY the four numbers above are stated in. +#: +#: THEY LIVE HERE BECAUSE THE COUPLING IS TO THE LEVELS, not because luma needs a home. A field +#: graded with different weights from the ones the levels were measured through is a different +#: quantity wearing the same units, and `albedo_alpha` says in as many words that nothing can check +#: that pairing. `scripts/measure_viking_levels.py` re-measures the levels through `luma` below for +#: exactly this reason: the measuring instrument and the render must not be able to drift apart. +#: +#: The other Rec. 709 luma in this repo — `compose/gen_spotlight.py` and two tests — are independent +#: uses of a standard scalarisation, and are deliberately NOT routed through here. Nothing breaks if +#: one of them moves to Rec. 601; this one cannot move without re-measuring `ALPHA_LEVELS`. +LUMA_WEIGHTS: tuple[float, float, float] = (0.2126, 0.7152, 0.0722) + + +def _smoothstep(fraction: np.ndarray) -> np.ndarray: + """Clamp to 0..1 and ease both ends. The module's one spelling of it, read by both alphas. + + Kept private and local rather than shared: the array-form smoothstep has several spellings across + this package and giving it a single owner is its own change, not one to make while adding a + seventh caller. float64 out, matching `snow.snow_alpha` — the composite blends whichever body's + answer it is handed, and a narrower dtype from one of them shifts the other's blend sub-DN. + """ + # The cast is load-bearing and not defensive: the graded field lands as float32, and float32 in + # would give float32 out under numpy's promotion rules, silently breaking the dtype this promises. + clipped = np.clip(np.asarray(fraction, dtype=float), 0.0, 1.0) + return clipped * clipped * (3.0 - 2.0 * clipped) + + +def luma(rgb: np.ndarray) -> np.ndarray: + """Rec. 709 luma of a `(3, ...)` RGB stack, as float64 — the field the ice is graded on. + + ZERO IS EXACT INVALIDITY, AND THAT IS A PROPERTY OF THE WEIGHTS RATHER THAN A CONVENTION. Every + weight is positive and every channel is non-negative, so the sum vanishes only where all three + channels do — which is precisely Viking's nodata, declared 0 on each band and meaning absent only + when they agree. That property is what lets `albedo_alpha` keep a scalar sentinel against an RGB + source: the three bands collapse to one BEFORE anything is graded, and the sentinel survives the + collapse instead of having to be carried beside it. + + FLOAT64 OUT, AND THE DTYPE IS THE WHOLE POINT AT THE DARK END. A pixel of (1, 0, 0) has luma + 0.2126, which any integer round would flatten to 0 and hand to the grader as "not measured". + Rounding here would therefore turn the darkest measured ground into nodata, in the one hemisphere + whose ground level is the lower of the two. + """ + stack = np.asarray(rgb, dtype=float) + weights = np.asarray(LUMA_WEIGHTS, dtype=float).reshape(-1, *([1] * (stack.ndim - 1))) + return (stack * weights).sum(axis=0) + + +def albedo_alpha(albedo: np.ndarray, levels: tuple[float, float], nodata: float) -> np.ndarray: + """How icy each pixel is, from the graded field normalised between this pole's two pinned levels. + + `levels` is `(ground, cap)` from `ALPHA_LEVELS`, passed rather than looked up so the caller names + the pole once and this stays a pure function of its arguments. + + THE FIELD AND THE LEVELS MUST BE THE SAME QUANTITY and nothing here can check that. Levels from + one field applied to another normalise to zero everywhere, which renders a bare cap and raises + nothing — so the pairing is the caller's obligation. `ALPHA_LEVELS` is 8-bit luma. + + UNMEASURED PIXELS BECOME ZERO AND ARE COMPARED BEFORE SCALING. A sentinel fill normalised becomes + an ordinary out-of-range float that the clamp would quietly turn into 0.0 anyway — but only by + arithmetic accident, and a fill that ever landed inside the range would paint as ice. Masking on + the raw value makes it a decision. + + A SCALAR SENTINEL FITS THE RGB SOURCE ONLY BECAUSE `luma` COLLAPSES IT FIRST. Viking's invalidity + is every channel at zero rather than one distinguished value, which no scalar could express — but + Rec. 709 weights are all positive over non-negative channels, so luma is zero exactly where the + three bands are. Pass this the luma and 0.0, never a single colour band and a guessed fill. + + THIS IS THE OWNER OF THE RATIFIED ARITHMETIC, and the shape is what was ratified rather than any + of its inputs: a field normalised between two levels and eased. The prototype it came from + recomputed those levels per run and graded OMEGA reflectance in float32; both changed under it — + pinned levels, Viking luma, float64 — without the look moving, which is what the pinning note on + `ALPHA_LEVELS` records. + """ + ground, cap = levels + alpha = _smoothstep((albedo - ground) / (cap - ground)) + alpha[albedo == nodata] = 0.0 + return alpha + + +def burn_unit(unit: str, target_srs: str, bounds: tuple[float, float, float, float], + width: int, height: int, projected: Path, out: Path, + creation_options: tuple[str, ...] = (), + must_draw: "str | None" = None) -> Path: + """Land one acquired unit on a grid as a 0/1 Byte mask. + + Thin over `vector_raster.burn_onto_grid`, whose module note holds the reprojection trap this + whole stage is shaped around. The source is resolved through `download_sim3292.unit_path` AT CALL + TIME so a redirected data root moves it, per `paths`. + + `must_draw` IS THE CALLER'S CLAIM AND IS DELIBERATELY NOT DEFAULTED. On a planet grid an empty + burn is always breakage, because the grid spans the planet and the polygons exist. On a cap grid + it depends on the disc's edge latitude against that unit's own reach, so a default here would + either miss the trap on one caller or cry wolf on the other. + """ + return vector_raster.burn_onto_grid( + download_sim3292.unit_path(unit), target_srs, bounds, width, height, + projected=projected, out=out, creation_options=creation_options, must_draw=must_draw) + + +def extent_for(unit_masks: dict[str, np.ndarray], northern) -> np.ndarray: + """The ice extent from the burnt unit masks, taking each hemisphere's own union. + + `northern` is anything broadcastable against the masks — a plain bool for a cap disc, which is + all one hemisphere, or a per-row column for a Mercator window, which may straddle the equator. + A missing unit raises `KeyError` rather than reading as absent ice, since the caller not burning + a unit this hemisphere needs is a bug and not an extent. + """ + return np.where(northern, + _union(unit_masks, NORTH_UNITS), + _union(unit_masks, SOUTH_UNITS)) + + +def _union(unit_masks: dict[str, np.ndarray], units: tuple[str, ...]) -> np.ndarray: + """Boolean OR of the named units' masks.""" + return reduce(np.logical_or, (np.asarray(unit_masks[unit], dtype=bool) for unit in units)) + + +def feather_alpha_bands(mask: np.ndarray, ground_metres_per_px, feather_km: float = FEATHER_KM, + band_rows: "int | None" = None) -> "Iterator[tuple[int, int, np.ndarray]]": + """Yield `(row0, row1, alpha)` for each band: 1.0 inside the extent, smoothstepping to 0.0 + `feather_km` GROUND kilometres outside it. + + A GENERATOR BECAUSE THE RESULT IS THE THING THAT DOES NOT FIT, which is a different problem from + the transform's own peak and was briefly confused with it. Mars's planet grid is 32768 square, so + one float64 alpha for it is 8.6 GB — bounding the distance transform to a band and then + materialising the whole answer would still not run. Handing back one band at a time lets the + caller write it and drop it, and it is what makes `band_rows` worth having at all. + + `ground_metres_per_px` converts the transform's pixels into ground metres, as a scalar or a value + per row of the WHOLE mask. It has to be either, because a Mercator pixel is not a fixed ground + distance: across the band where Mars's tiles show ice it runs about 152 m down to 68, so a + feather counted in pixels would be more than twice as wide at one end as the other. A cap's AEQD + grid has no such term and passes one number. The ratio behind it belongs to + `bodies.ground_metres_per_mercator_unit` / `..._aeqd_unit`; only the composition is here. + + Bands are EXACT WHERE THEY ARE READ AND WRONG BEYOND, which is the whole bargain. Each is + transformed with `pad` extra rows on both sides, so any nearest-ice within `pad` pixels is inside + the slice that computed it; a cell whose nearest ice lies further away gets a number too large, + and every such cell is already past the feather and clipped to zero. + + Alpha is float64, matching `snow.snow_alpha` — the composite blends whichever body's answer it is + handed, and a narrower dtype from one of them would shift the other's blend sub-DN. + """ + outside = ~np.asarray(mask, dtype=bool) + feather_m = feather_km * 1000.0 + scale = np.asarray(ground_metres_per_px, dtype=float) + if scale.size == 0 or float(scale.min()) <= 0.0: + raise ValueError("ground_metres_per_px must be positive everywhere — it divides the feather " + f"into pixels; got {'an empty array' if not scale.size else scale.min()}") + # The pad is DERIVED from the feather rather than pinned beside it, so the two cannot drift into + # a banded pass that is quietly wrong at every band edge. The finest pixel needs the most of it. + pad = int(np.ceil(feather_m / float(scale.min()))) + 1 + height = outside.shape[0] + for row0, row1 in row_bands(height, band_rows if band_rows else height): + top, bottom = max(0, row0 - pad), min(height, row1 + pad) + distance = np.asarray(ndimage.distance_transform_edt(outside[top:bottom]), dtype=float) + distance = distance[row0 - top:row1 - top] + rows = scale[row0:row1].reshape(-1, 1) if scale.ndim == 1 else scale + yield row0, row1, _smoothstep(1.0 - distance * rows / feather_m) + + +def feather_alpha(mask: np.ndarray, ground_metres_per_px, + feather_km: float = FEATHER_KM) -> np.ndarray: + """The whole feathered alpha at once, for a grid that fits — a cap disc, or one window. + + Spelled as the single-band case of the generator rather than as its own transform, so there is + one arithmetic here and no second copy to drift. `band_rows=None` is that single band. + """ + (_row0, _row1, alpha), = feather_alpha_bands(mask, ground_metres_per_px, feather_km) + return alpha + + +def graded_alpha(field: np.ndarray, northern, nodata: float) -> np.ndarray: + """The field graded between the pinned levels of whichever pole each row belongs to. + + THE TWIN OF `extent_for` AND BROADCAST THE SAME WAY — a plain bool for a cap disc, a per-row + column for a Mercator strip that may straddle the equator. `np.where` evaluates both branches, + which is why this is cheap only on a slice: hand it a whole planet and it grades one twice. + + The two poles are DIFFERENT QUANTITIES sharing a scale. A single pair of levels applied to both + would grade the south against the north's darker ground and paint its cap several tenths too + white, which is precisely the disagreement the pinning note on `ALPHA_LEVELS` exists to prevent. + """ + return np.where(northern, + albedo_alpha(field, ALPHA_LEVELS["north"], nodata), + albedo_alpha(field, ALPHA_LEVELS["south"], nodata)) + + +def unit_latitude_span(unit: str, northern: bool) -> "tuple[float, float] | None": + """How far one unit's polygons reach in ONE hemisphere, in degrees, or None if it has none there. + + PER HEMISPHERE, AND THAT IS THE WHOLE POINT OF THE ARGUMENT. `lApc` is mapped at BOTH poles, so + a single span over its features runs -90 to +90 and describes a planet rather than an extent — + which would quietly turn the banded build below back into the whole-grid one it exists to avoid. + + READ FROM THE GEOMETRY, NEVER PINNED. The published reaches are recorded in this project's notes + and would serve as constants right up until the publisher revises a polygon, at which point a + hard-coded band clips real ice with nothing raising. The acquirer guarantees the file's edition; + the extent is then the file's own business. + + The GeoJSON carries no `bbox`, so the coordinates are walked — a few MB of parse against a stage + measured in minutes. + """ + def latitudes(node) -> "Iterator[float]": + # A position is [lon, lat, ...]; anything else is a nest of them. Testing the FIRST element + # for a scalar distinguishes the two without assuming a geometry type. + if node and isinstance(node[0], (int, float)): + yield float(node[1]) + return + for child in node: + yield from latitudes(child) + + document = json.loads(download_sim3292.unit_path(unit).read_text(encoding="utf-8")) + values = [value + for feature in document["features"] + for value in latitudes(feature["geometry"]["coordinates"]) + if (value >= 0.0) == northern] + return (min(values), max(values)) if values else None + + +def ice_bands(bounds: tuple[float, float, float, float], height: int, + pad_rows: int) -> list[tuple[int, int, bool]]: + """Each `(row0, row1, northern)` of a 3857 grid a mapped unit can reach. Empty if none can. + + WHY THE BUILD IS BANDS AND NOT A PLANET. This layer is only ever painted where a unit reaches, + which on Mars is a few degrees at each pole; the rest of a 32768-row grid is ice-free by + construction. Deriving the rows from the units rather than from a latitude constant keeps that an + observation about the data instead of a number someone has to maintain. + + ONE BAND PER HEMISPHERE, each carrying which one it is, because the levels and the unit union + both differ per pole and a single band spanning the equator could answer for neither. + + `pad_rows` must cover the feather: a pixel outside every unit still takes alpha from one within + `FEATHER_KM` of it, so a band cut to the polygons alone would clip the feather at its own edge. + """ + _left, bottom, _right, top = bounds + pixel = (top - bottom) / height + bands: list[tuple[int, int, bool]] = [] + for northern, units in ((True, NORTH_UNITS), (False, SOUTH_UNITS)): + spans = [span for span in (unit_latitude_span(unit, northern) for unit in units) if span] + if not spans: + continue + rows = [round((top - float(mercator.northing_at( + np.array([latitude]), mercator.WEB_MERCATOR_RADIUS_M)[0])) / pixel) + for span in spans for latitude in span] + row0, row1 = max(0, min(rows) - pad_rows), min(height, max(rows) + pad_rows) + if row1 > row0: + bands.append((row0, row1, northern)) + return bands + + +#: Rows graded at once. `graded_alpha` evaluates both poles in float64, so a slice costs several +#: times its float32 footprint while it is in hand — sized to stay inside the one-heavy-job cap. +GRADE_ROWS = 512 + + +def _warp_band(field: Path, bounds: tuple[float, float, float, float], width: int, height: int, + out: Path) -> Path: + """The 4326 field on one band of the 3857 grid, in ONE gdalwarp. + + DIRECT RATHER THAN THE SUB-BANDED MOSAIC the two Earth cryosphere warps use, and the difference + is the extent being warped rather than a preference. That mosaic exists because a WHOLE-GRID + warp's pole-inflated average scale makes gdalwarp read the source decimated — the failure has no + error and no symptom beyond structure quietly going missing. A band spans a narrow enough range + of scales that the average is honest, which was measured on this grid against a sub-banded + reference and against a deliberately decimated control; `scripts/` has no home for that probe, so + the numbers live in PROCESS.md. + + DO NOT "FIX" THIS BY ADDING BANDING. It would cost a subprocess per sub-band for output already + shown identical, and it would read as though the decimation question here were open. + """ + left, bottom, right, top = bounds + subprocess.run( + ["gdalwarp", "-q", "-overwrite", "-t_srs", "EPSG:3857", + "-te", repr(left), repr(bottom), repr(right), repr(top), + "-ts", str(width), str(height), "-r", "bilinear", "-ot", "Float32", + "-co", "TILED=YES", "-co", "COMPRESS=DEFLATE", "-co", "BIGTIFF=YES", + str(field), str(out)], check=True) + return out + + +def build_alpha_raster(field: Path, field_nodata: float, + bounds: tuple[float, float, float, float], width: int, height: int, + out: Path, ground_metres_per_map_unit: float) -> Path: + """This body's ice alpha on the 3857 grid: graded between the pinned levels, cut to the mapped + units, feathered outward — and computed only where a unit can reach. + + ZERO IS A VALUE HERE AND NOT AN ABSENCE, which is why the output declares no nodata. Every row + outside a band is ice-free by construction, so the blocks that are never written read back as + alpha 0 and mean it. A nodata of 0 would say "unmeasured" about pixels this stage measured and + found bare. + + THE FEATHER IS WHY THE BANDS ARE PADDED. A pixel outside every polygon still takes alpha from one + within `FEATHER_KM` of it, so a band cut to the polygons alone would clip its own feather at the + band edge — visible as a hard line parallel to no coastline. + + THE SCALE PASSED DOWN IS GROUND METRES PER PIXEL AND IT VARIES PER ROW. A Mercator pixel is not a + fixed ground distance; across the latitudes this layer occupies it changes by more than half, so + a feather counted in pixels would be markedly wider at one end of the band than the other. + `feather_alpha_bands` holds the argument, and `ground_metres_per_map_unit` is the body's own + conversion rather than anything derivable here. + """ + left, bottom, right, top = bounds + pixel = (top - bottom) / height + map_units_per_px = (right - left) / width + edge_latitude = float(mercator.latitude_at(np.array([top]), mercator.WEB_MERCATOR_RADIUS_M)[0]) + # The finest ground pixel the grid can hold sits at its top row, so a pad derived from it covers + # the feather at every latitude a band could occupy. Derived, never a constant: the grid's own + # edge latitude is what sets it. + finest = map_units_per_px * ground_metres_per_map_unit * float( + np.cos(np.radians(edge_latitude))) + pad_rows = int(np.ceil(FEATHER_KM * 1000.0 / finest)) + 1 + + profile: dict[str, Any] = dict( + driver="GTiff", width=width, height=height, count=1, dtype="float32", crs="EPSG:3857", + transform=rasterio.transform.from_bounds(left, bottom, right, top, width, height), + tiled=True, compress="DEFLATE", bigtiff="YES") + work = out.parent + with rasterio.open(out, "w", **profile) as writer: # pyright: ignore[reportCallIssue] + for row0, row1, northern in ice_bands(bounds, height, pad_rows): + pole = "north" if northern else "south" + band_bounds = (left, top - row1 * pixel, right, top - row0 * pixel) + band_height = row1 - row0 + drawn = NORTH_UNITS if northern else SOUTH_UNITS + masks = { + unit: _read_mask(burn_unit( + unit, "EPSG:3857", band_bounds, width, band_height, + projected=work / f"ice_{pole}_{unit.lower()}_3857.geojson", + out=work / f"ice_{pole}_{unit.lower()}.tif", + # Only the units this hemisphere PAINTS are guaranteed to land: the band was + # derived from their own reach. The other is burnt because `extent_for` + # evaluates both unions, and it may legitimately be empty here. + must_draw=(f"{unit} must reach the {pole} band it defines" + if unit in drawn else None))) + for unit in NORTH_UNITS + } + latitude = mercator.latitude_at( + top - (np.arange(row0, row1) + 0.5) * pixel, mercator.WEB_MERCATOR_RADIUS_M) + scale = map_units_per_px * ground_metres_per_map_unit * np.cos(np.radians(latitude)) + warped = _warp_band(field, band_bounds, width, band_height, + work / f"ice_{pole}_field.tif") + extent = extent_for(masks, northern) + with rasterio.open(warped) as reader: + for sub0, sub1, feather in feather_alpha_bands(extent, scale, + band_rows=GRADE_ROWS): + window = Window(0, sub0, width, sub1 - sub0) # pyright: ignore[reportCallIssue] + graded = graded_alpha(reader.read(1, window=window), northern, field_nodata) + writer.write((graded * feather).astype(np.float32), 1, + window=Window(0, row0 + sub0, width, # pyright: ignore[reportCallIssue] + sub1 - sub0)) + return out + + +def _read_mask(path: Path) -> np.ndarray: + """A burnt 0/1 raster as a boolean array.""" + with rasterio.open(path) as dataset: + return dataset.read(1) != 0 diff --git a/pipeline/render/palette.py b/pipeline/render/palette.py index 5cb907a..22d5c13 100644 --- a/pipeline/render/palette.py +++ b/pipeline/render/palette.py @@ -14,11 +14,16 @@ `gdaldem color-relief` consumes. Land and sea are separate ramps chosen later by the ocean mask (not the elevation sign), which keeps the coastline crisp. -The frozen endpoints -for land at 0/6000 m and 85B9B7/3A6E7D for sea at 0/-6000 m; `test_palette.py` guards -against drift off those values. +EVERY RAMP CONSTANT HERE IS ONE BODY'S, and the module holds more than one body's. Earth's land +runs 0/6000 m and its sea 85B9B7/3A6E7D over 0/-6000 m, both frozen and guarded against drift by +`test_palette.py`. Mars answers `MARS_LAND_STOPS` over its own domain and shares no colour object +with Earth, so a re-tune of one planet cannot reach the other — which is the whole reason the ramps +are assembled into a named `Look` rather than read as globals. Nothing outside this module may read +an authored ramp by name; a source scan enforces it, because a module that does renders Earth +perfectly and is wrong only on the planet nobody has looked at. """ +import itertools from dataclasses import dataclass from pathlib import Path @@ -53,8 +58,37 @@ # surface tone (85B9B7) lightened ~7%, so lakes stay in the sea's green-teal family but # read a touch calmer/lighter (the lake convention). Re-synced to the sea # rework, which had deepened the sea surface and left this stranded ~15% brighter. +# EARTH'S ice white, and it is Earth's rather than the project's — a second body measured a +# different one. `shade.composite` no longer reads these directly; the producer that computed the +# alpha declares which white paints it, so a body's ice cannot inherit another body's colour by +# omission. The hero rig still reads them, and the rig is Earth's by decision. +# +# The blue is not decoration: thick clean glacial ice absorbs red, so terrestrial snow really does +# go blue in shadow. That is Earth physics and does not travel — see MARS_ICE_WHITE. SNOW_RGB: RGB8 = (232, 241, 246) # E8F1F6 — sunlit snow (bright glacial white) SNOW_SHADOW_RGB: RGB8 = (176, 199, 219) # B0C7DB — shaded snow (cool blue-white, not grey) + +# MARS, PER POLE, because the two deposits are measurably different colours: north 1.053 and south +# 1.291 in red:violet against their own ground at 1.231 and 1.807. Normalising each cap by its own +# surroundings leaves most of that gap standing, so the difference belongs to the ice rather than to +# what surrounds it or to how it was imaged. +# +# RED AND BLUE ARE MEASURED; GREEN IS NOT AND CANNOT BE. The Viking mosaic's green band is a fixed +# linear combination of its red and violet (R^2 0.9998), so only the red:violet ratio is evidence +# and green is a stated design rule: hold Earth's luminance, and keep the pair on one locus. The +# south could not hold both — sRGB cannot reach Earth's brightness at that warmth with red pinned at +# 255 — so its green is lifted to buy the luminance back, which is the choice that was ratified by +# eye rather than derived. +# +# The derivation is checkable and `scripts/measure_mars_ice_white.py` is the check: it reproduces +# Earth's own shipped pair exactly from Earth's ratio and luminance before it will report anything, +# which is what makes it trustworthy on a pair nobody had seen. Its `--compare` mode re-measures the +# ice and refuses a white that has stopped describing it — re-run it after any change to the alpha +# levels or to the cap grid, both of which have moved once already. +MARS_ICE_WHITE: dict[str, tuple[RGB8, RGB8]] = { # pole -> (sunlit, shadowed) + "north": ((243, 239, 231), (198, 195, 188)), # F3EFE7 / C6C3BC + "south": ((255, 239, 198), (207, 196, 160)), # FFEFC6 / CFC4A0 +} # Sea ice: the same light-keyed white family but a subtle notch COOLER and dimmer than land snow, # so the poles read floating-thin-ice vs thick-ice-sheet without a hard colour split (the coastline # and relief carry the rest). Physically honest: thin sea ice over dark ocean reads less bright than @@ -68,9 +102,12 @@ SUN_ALT_DEG = 45.0 # the shared sun altitude: tile KNOBS["alt"] and the hero SUN_ROTATION # X-tilt (90 - alt) both derive from this (the sea-sync — the cure for the 46/45 # split). Azimuth stays per-side: both are NW by their own conventions (tile 315, hero -45). -EXAGGERATION = 15.0 # the shared vertical exaggeration: hero displacement_scale -# (render_prep.scene_numbers) and tile hillshade (shade_planet.EXAG) both derive from it -# — the last surviving copy-pair, moved here at the freeze-lift. +EXAGGERATION = 15.0 # the HERO's vertical exaggeration: render_prep.scene_numbers derives +# displacement_scale from it, and the region preview shades at it. The tile and cap path reads +# Body.exaggeration instead — relief is a different fraction of the radius on every planet, so it +# cannot be one number — and tests/test_bodies.py holds Earth's field equal to this. Pinned rather +# than shared: shared would be wrong for the second body, and unpinned would let the tiles drift +# away from the heroes they must match. def smoothstep(t: float) -> float: @@ -87,7 +124,7 @@ def lin2srgb(c: float) -> float: def ramp_color(pos: float, stops: list[Stop]) -> RGB: """EASE-interpolated linear RGB at pos in [0, 1] (clamped).""" pos = min(1.0, max(0.0, pos)) - for (p0, c0), (p1, c1) in zip(stops, stops[1:]): + for (p0, c0), (p1, c1) in itertools.pairwise(stops): if pos <= p1: t = 0.0 if p1 == p0 else (pos - p0) / (p1 - p0) blend = smoothstep(min(1.0, max(0.0, t))) @@ -132,17 +169,53 @@ def channel(value: int) -> float: @dataclass(frozen=True) class Surface: - """One ramp, and the elevation at which it reaches position 1.0. + """One ramp, and the two elevations it runs BETWEEN: `origin_m` at position 0.0, `extreme_m` + at position 1.0. `extreme_m` carries the DIRECTION in its sign, which is what lets land and sea share every - formula below instead of each carrying its own transcription: position is `elevation / - extreme_m`, so land runs 0 -> +6000 and sea runs 0 -> -6000 with no branch. The elevation a - ramp's first LUT index sits at is `min(0, extreme_m)`, which is 0 for land and the abyss for sea. + formula below instead of each carrying its own transcription: Earth's land runs 0 -> +6000 and + its sea runs 0 -> -6000 with no branch. Position is `(elevation - origin_m) / + (extreme_m - origin_m)`, and the elevation a ramp's first LUT index sits at is + `min(origin_m, extreme_m)` — the abyss for sea, the origin for land. + + ORIGIN EXISTS BECAUSE ZERO WAS AN EARTH FACT WEARING A CONSTANT'S CLOTHES. This class used to + hardcode 0 as one end, in `min(0, extreme_m)` and in dividing by `extreme_m` alone, and on Earth + that is right for a reason: 0 m is the shoreline, a real boundary where the ramp genuinely + should hinge. On a body with no sea it is nothing of the kind. Mars's 0 m is the areoid — an + equipotential reference with no expression on the ground, sitting at the MEDIAN of the planet's + elevations — so hinging there put 51.6% of Mars below the ramp entirely, clamped to one colour. + A body that declares no sea should not have its ramp shaped as though it did. + + `origin_m` HAS NO DEFAULT. A default of 0.0 would be correct at both of today's construction + sites and silently wrong at the first one that needed otherwise, which is the whole failure this + field exists to end; the type checker naming every site is worth more than the two words saved. + Earth reduces to the old expressions exactly rather than approximately — with `origin_m = 0.0`, + `(e - 0) / (x - 0)` IS `e / x` — so its ramps cannot move, by algebra and not only by test. """ stops: list[Stop] + origin_m: float extreme_m: float + def __post_init__(self): + # A zero-width ramp divides by zero. numpy would hand back nan, `np.rint(nan)` is nan, and + # the cast to int32 makes it an arbitrary index — a planet rendered in one wrong colour with + # no exception anywhere. Loud here, where the ramp is declared, is the only cheap place. + if self.origin_m == self.extreme_m: + raise ValueError( + f"a ramp needs two distinct ends; got origin_m == extreme_m == {self.origin_m}" + ) + + @property + def span_m(self) -> float: + """Signed distance from position 0.0 to 1.0. Negative for a ramp that runs downward.""" + return self.extreme_m - self.origin_m + + @property + def lowest_m(self) -> float: + """The elevation at LUT index 0, which is the lower end whichever way the ramp runs.""" + return min(self.origin_m, self.extreme_m) + @dataclass(frozen=True) class Look: @@ -156,22 +229,132 @@ class Look: entire reason this type exists: today every ramp is a module-level global, and a second planet with globals means either copied constants or mutation — and copied look constants have already cost this project one overnight re-render of all 203 heroes. + + `sea` IS OPTIONAL, AND `None` IS A STATEMENT RATHER THAN A GAP. It says this planet draws no + sea. The alternative — a sea ramp written out for a body that declares no oceanmask, so no pixel + could ever select it — puts a colour nobody chose into the freshness recipe, where it is + indistinguishable from one that was deliberated over. That is the fabricated-fact trap the + planet seam already refuses one tier up, where an all-zero ocean raster would have been unable + to say whether a planet has no sea or its fusion died halfway. Same refusal, same reason. """ land: Surface - sea: Surface + sea: Surface | None #: Terrella's look, assembled from the constants above rather than restating them — those remain #: the authored values, and every consumer still reads them directly. This is the seam a second #: look plugs into, not a second copy of the first. EARTH_LOOK = Look( - land=Surface(stops=LAND_STOPS, extreme_m=LAND_MAX_M), - sea=Surface(stops=SEA_STOPS, extreme_m=SEA_MIN_M), + land=Surface(stops=LAND_STOPS, origin_m=0.0, extreme_m=LAND_MAX_M), + sea=Surface(stops=SEA_STOPS, origin_m=0.0, extreme_m=SEA_MIN_M), ) +#: Mars's land ramp, hand-authored against measurements of the planet's own colour. +#: +#: A HYPSOMETRIC RAMP IS A FICTION ON MARS, AND THIS ONE KNOWS IT. Earth's ramp works because +#: elevation predicts colour there, through climate and vegetation. Mars's albedo is set by +#: WIND-BLOWN DUST, which does not care about height. Measured by joining the global Viking colour +#: mosaic (`Mars_Viking_ClrMosaic_global_925m.tif`, USGS) to the shipped heightfield at 6.48 M +#: co-registered lon/lat samples, area-weighted: across the elevations holding 64% of the surface +#: (-3000..+3000 m) mean colour moves 7.1 luma against a within-place scatter of 17.9 — a ratio of +#: 0.40. Widened to 92% of the surface it is 1.03. The named albedo features are off any such trend +#: by more than the trend's whole range: Syrtis Major sits at +1,369 m and luma 41.5 where its +#: elevation band averages 84. +#: +#: SO THESE COLOURS ARE CARTOGRAPHIC CONVENTION, NOT A PICTURE OF MARS, and the About page says so +#: to visitors. What was taken from the measurement and what was not: +#: - HUE, taken in full. Channel ratios survive an uncalibrated tone curve where an absolute level +#: does not. The mosaic puts Mars at G/R 0.654; Earth's borrowed ramp was shipping 0.780. +#: - LEVEL, taken in part. The mosaic reads 2.07x darker than the borrowed ramp shipped, which +#: would put Mars at 0.45x Earth's shipped land. Most of that is the product's own tone curve +#: and an uncorrected atmospheric haze floor, not the planet: real land albedo is broadly +#: comparable between the two bodies (Earth's deserts 0.30-0.40 and forests 0.10-0.15 against +#: Mars's bright regions ~0.30 and dark ~0.10), so pinning Mars to the mosaic while Earth stays +#: stylised would make the two planets disagree about what a map is. This ships at 0.71x. +#: - SHAPE, deliberately refused. Mars is genuinely brightest at BOTH ends — Hellas is a dust trap +#: and Tharsis is dust-mantled — so a faithful ramp would give the deepest basin and the highest +#: summit the same colour, which is exactly the defect inherited from Earth's shoreline hinge. +#: Rising monotonically with elevation is what makes height readable at all. +#: +#: THE AUTHORED VALUES ARE NOT WHAT SHIPS. `shade.composite` resaturates by `saturation` 1.18, warms +#: by `warmth` 0.06 and multiplies by a light term at ~1.025 on flat ground, so a stop picked to hit +#: a hue target overshoots it. These were specified as the intended SHIPPED colour and inverted +#: through that chain — a first pass authored directly by eye landed at B/R 0.35 against a 0.55 +#: intent. Anything re-tuning them should invert the same way rather than nudging the hex. +MARS_LAND_STOPS: list[Stop] = [ + (0.000, (0.187821, 0.078187, 0.045186)), # -6000 m deepest basin floor, ships #804d35 + (0.150, (0.274677, 0.114435, 0.066626)), # -4185 m lowland plains + (0.350, (0.412543, 0.171441, 0.082283)), # -1765 m the northern lowlands' own tone + (0.550, (0.514918, 0.246201, 0.111932)), # +655 m just above the areoid, the modal elevation + (0.780, (0.597202, 0.366253, 0.187821)), # +3438 m southern highlands + (1.000, (0.658375, 0.520996, 0.337164)), # +6100 m Tharsis and the volcanic summits +] + +#: Mars. Everything here is decided except whether it ever draws a sea. +#: +#: `sea` is None, and that half is a FACT rather than a placeholder: `fuse/relabel_mars.py` declares +#: a heightfield and no oceanmask, so no pixel can select a sea ramp however carefully one is +#: written. Whether Mars ever draws a sea — none, one chosen shoreline contour, or the family of +#: candidates — is a look decision made on the sphere, and this is the seam it lands on. +#: +#: THE DOMAIN IS MARS'S OWN AND IT IS DERIVED, NOT PREFERRED. Measured over the shipped heightfield, +#: area-weighted on the sphere (cos(lat)-corrected, ~1.05M samples): p1 -5,990 m, p50 -260 m, +#: p99 +6,098 m, true range -7,882 .. +21,014 m. The two below are p1 and p99 rounded, so the ramp +#: does real work across 98% of the surface. The extremes were rejected on the same measurement: +#: only 1.1% of Mars sits above +6,000 m, so keying the ceiling to Olympus Mons would spend most of +#: the ramp on almost nothing — and Olympus still reads, because it reads through the HILLSHADE, +#: which no ramp touches. +#: +#: THE DOMAIN NUMBERS SIT IN THE CONSTRUCTOR, not in named module constants, and the asymmetry with +#: `MARS_LAND_STOPS` above is the naming rule working rather than an inconsistency in it. That list +#: has two readers besides the constructor — `RAMP_GLOBALS` and the tests that compare the assembled +#: `Surface` against it — exactly as `LAND_STOPS` does. These two numbers have none: read once, on +#: the line below, by nothing else. A name for them would be two names for one number, which is the +#: drift this file exists to refuse. The dataclass IS the representation; a constant earns its own +#: name only when something other than the constructor needs it. +MARS_LOOK = Look( + land=Surface(stops=MARS_LAND_STOPS, origin_m=-6000.0, extreme_m=6100.0), + sea=None, +) + +#: The look each body draws with today. +#: +#: Keyed by SLUG rather than held as a `Body` field, because `pipeline/bodies.py` opens by saying +#: what a body is — "Not a look and not a dataset" — and geometry and colour are separate axes on +#: purpose. Welding a Look into the descriptor would also foreclose the parked idea of one planet +#: carrying several looks. The cost of the separation is that two modules now know the set of +#: planets, so `tests/test_palette.py` holds this dict to the body registry: a planet registered +#: there with no entry here is the failure that renders rather than raises. +LOOK_BY_BODY: dict[str, Look] = {"earth": EARTH_LOOK, "mars": MARS_LOOK} + + +def look_for(body: str) -> Look: + """The look a body draws with. Raises on an unknown body and never falls back to Earth's. + + The fallback is the whole point of raising. A body that quietly inherited Earth's ramp would + render a complete, plausible, internally consistent pyramid in another planet's colours, and + every gate in the pipeline would pass — the same failure shape `bodies.get` refuses for + geometry, where the wrong sphere is plausible everywhere and true nowhere. + + CALLED WHERE THE BODY IS KNOWN, NEVER THREADED IN BESIDE IT, and that is a decision rather than + an omission. The planet seam's rule is the opposite — its rasters are passed as required + parameters and never looked up — because a raster set is a RUNTIME declaration that varies per + run and cannot be derived from the body. A look can be, and there is exactly one right answer. + So a `look` parameter sitting next to a `body` parameter would add nothing but a way for the + two to disagree: `composite_params(body=MARS, look=EARTH_LOOK)` is a sentence the type checker + accepts and no reviewer would notice. The cost is that a synthetic body in a test needs a look + registered, exactly as it needs a radius, and `tests/test_cap_render.py` pays it in a fixture. + """ + try: + return LOOK_BY_BODY[body] + except KeyError: + raise KeyError( + f"no look registered for body {body!r}; known: {sorted(LOOK_BY_BODY)}" + ) from None + -def surface(kind: str, look: Look = EARTH_LOOK) -> Surface: +def surface(kind: str, *, look: Look) -> Surface: """Resolve `'land'`/`'sea'` to its ramp. THE ONE PLACE THAT DISPATCH LIVES. It used to be transcribed in four functions — @@ -179,12 +362,19 @@ def surface(kind: str, look: Look = EARTH_LOOK) -> Surface: re-deriving which stops and which range a kind meant. Four copies of one mapping is the shape of drift this file exists to prevent, and it was sitting inside the file itself. - `look` is defaulted only because nothing selects a look yet; the moment a second one exists it - becomes a required argument, for the reason the body registry states. + `look` IS KEYWORD-ONLY AND REQUIRED, and removing its default was the first move rather than the + last. With a default, adding `MARS_LOOK` would have left every call site below still drawing + Earth and nothing would have named one of them; without it, the type checker names all of them + at once. That is the same reason no field on `Body` may carry a default. """ if kind == "land": return look.land if kind == "sea": + if look.sea is None: + raise ValueError( + "this look draws no sea, so there is no sea ramp to resolve. A body whose planet " + "seam declares an oceanmask needs one; a body that declares none never asks." + ) return look.sea raise ValueError(f"kind must be 'land' or 'sea', got {kind!r}") @@ -199,28 +389,31 @@ def lake_lut(size: int = 256) -> list[RGB8]: return [_srgb8(ramp_color(index / (size - 1), LAKE_STOPS)) for index in range(size)] -def color_relief_rows(kind: str, step: float = 25.0) -> list[tuple[float, RGB8]]: +def color_relief_rows(kind: str, *, look: Look, step: float = 25.0) -> list[tuple[float, RGB8]]: """(elevation, sRGB) rows for one surface, densely sampled so `gdaldem`'s linear interpolation between rows reproduces the EASE ramp. - 'land' maps elevation 0..6000 m; 'sea' maps depth -6000..0 m (deepest first). Each - ramp only has to be correct on its own side — the ocean mask selects between them.""" - ramp = surface(kind) - count = round(abs(ramp.extreme_m) / step) - # Land starts at 0 and climbs; sea starts at the abyss and rises to 0. One expression, because - # `extreme_m` carries the direction — see Surface. - base = min(0.0, ramp.extreme_m) + On Earth 'land' maps elevation 0..6000 m and 'sea' maps depth -6000..0 m (deepest first); on + another body the ends are that body's, and the ramp only has to be correct on its own side — + the ocean mask selects between them where there are two.""" + ramp = surface(kind, look=look) + count = round(abs(ramp.span_m) / step) + # Earth's land starts at 0 and climbs; its sea starts at the abyss and rises to 0; Mars's land + # starts below the areoid. One expression, because the ramp carries both its ends — see Surface. rows = [] for i in range(count + 1): - elev = base + i * step - rows.append((elev, _srgb8(ramp_color(elev / ramp.extreme_m, ramp.stops)))) + elev = ramp.lowest_m + i * step + rows.append((elev, _srgb8(ramp_color((elev - ramp.origin_m) / ramp.span_m, ramp.stops)))) return rows -LUT_STEP_M = 1.0 # LUT resolution in metres. 6001 entries x 3 B = 18 KB per surface. +# LUT resolution in metres. A surface costs `3 * (|span_m| / step + 1)` bytes, so the table is tens +# of KB for any ramp a body could want and the step is chosen for fidelity alone -- no entry count +# is quoted here, because a body-dependent one would be stale the day a second body disagreed. +LUT_STEP_M = 1.0 -def relief_lut(kind: str, step: float = LUT_STEP_M) -> np.ndarray: +def relief_lut(kind: str, *, look: Look, step: float = LUT_STEP_M) -> np.ndarray: """Elevation -> sRGB LUT for one surface, as a (3, N) uint8 array. This is what lets `gdaldem color-relief` be deleted rather than tuned. Measured: @@ -235,37 +428,40 @@ def relief_lut(kind: str, step: float = LUT_STEP_M) -> np.ndarray: is strictly FINER than the 25 m rows gdaldem interpolates across, so it is if anything the more faithful rendering of the authored ramp -- and it is 18 KB. """ - ramp = surface(kind) - count = round(abs(ramp.extreme_m) / step) - # index 0 is the extreme end for sea (deepest) and 0 m for land, matching color_relief_rows' - # ordering — both fall out of `min(0, extreme_m)` rather than being restated per kind. - base = min(0.0, ramp.extreme_m) - colors = [_srgb8(ramp_color((base + index * step) / ramp.extreme_m, ramp.stops)) + ramp = surface(kind, look=look) + count = round(abs(ramp.span_m) / step) + # index 0 is the deepest end for sea and the origin for land, matching color_relief_rows' + # ordering — both fall out of `Surface.lowest_m` rather than being restated per kind. + colors = [_srgb8(ramp_color((ramp.lowest_m + index * step - ramp.origin_m) / ramp.span_m, + ramp.stops)) for index in range(count + 1)] return np.asarray(colors, dtype=np.uint8).T # (3, N) -def lut_index(kind: str, elevation, step: float = LUT_STEP_M) -> np.ndarray: +def lut_index(kind: str, elevation, *, look: Look, step: float = LUT_STEP_M) -> np.ndarray: """Elevation -> clamped LUT index. The whole optimisation: a divide, not a search. - Clamping is load-bearing, not defensive: the planet height raster spans -10,728 m to - +7,281 m (measured), i.e. past BOTH ramp ends, and `gdaldem` clamps to its first/last row. - Land-classed pixels can also be negative (Dead Sea, -430 m) -- the ocean MASK picks the - ramp, never the sign -- so the land ramp must clamp those to its 0 m colour. + Clamping is load-bearing, not defensive: Earth's height raster spans -10,728 m to +7,281 m + (measured) and Mars's -7,882 m to +21,014 m, i.e. both run past BOTH of their ramp's ends, and + `gdaldem` clamps to its first/last row. Earth's land-classed pixels can also be negative (Dead + Sea, -430 m) -- the ocean MASK picks the ramp, never the sign -- so Earth's land ramp must clamp + those to its 0 m colour. On a body whose ramp starts below the datum there is nothing special + about a negative pixel at all, which is the point of `origin_m`. """ elevation = np.asarray(elevation, dtype=np.float32) - ramp = surface(kind) - raw = (elevation - np.float32(min(0.0, ramp.extreme_m))) / np.float32(step) - limit = round(abs(ramp.extreme_m) / step) + ramp = surface(kind, look=look) + raw = (elevation - np.float32(ramp.lowest_m)) / np.float32(step) + limit = round(abs(ramp.span_m) / step) return np.clip(np.rint(raw), 0, limit).astype(np.int32) -def lut_lookup(lut: np.ndarray, kind: str, elevation, step: float = LUT_STEP_M) -> np.ndarray: +def lut_lookup(lut: np.ndarray, kind: str, elevation, *, look: Look, + step: float = LUT_STEP_M) -> np.ndarray: """(3, ...) uint8 colours for `elevation`, shaped like `elevation`.""" - return lut[:, lut_index(kind, elevation, step)] + return lut[:, lut_index(kind, elevation, look=look, step=step)] -def color_relief_text(kind: str, step: float = 25.0) -> str: +def color_relief_text(kind: str, *, look: Look, step: float = 25.0) -> str: """The exact `gdaldem color-relief` file contents for one surface, incl. the `nv` row. Split out from `write_color_relief` so a caller can compare the ramp a run WOULD use @@ -274,11 +470,11 @@ def color_relief_text(kind: str, step: float = 25.0) -> str: the ramp's mtime alone. """ rows = [f"{elev:.2f} {red} {green} {blue}" - for elev, (red, green, blue) in color_relief_rows(kind, step)] + for elev, (red, green, blue) in color_relief_rows(kind, look=look, step=step)] return "\n".join(rows + ["nv 0 0 0", ""]) -def write_color_relief(path: Path, kind: str, step: float = 25.0) -> None: +def write_color_relief(path: Path, kind: str, *, look: Look, step: float = 25.0) -> None: """Write a `gdaldem color-relief` file for one surface, with an `nv` nodata row.""" with open(path, "w") as handle: - handle.write(color_relief_text(kind, step)) + handle.write(color_relief_text(kind, look=look, step=step)) diff --git a/pipeline/render/perennial_ice.py b/pipeline/render/perennial_ice.py new file mode 100644 index 0000000..06f3506 --- /dev/null +++ b/pipeline/render/perennial_ice.py @@ -0,0 +1,261 @@ +"""Which producer paints a body's perennial ice on a polar cap, and what that producer reads. + +THE LAYER IS ONE NAME AND THE PRODUCERS ARE NOT ONE FUNCTION. `perennial_ice` states a claim — the +white that is there all year — and each body answers it with different machinery. Earth's north +warps a hemispheric NetCDF climatology and smoothsteps its persistence fraction; Earth's south has +no dataset in this pipeline at all and forces the answer out of latitude-and-land arithmetic. So +this is a registry of FUNCTIONS where `palette.LOOK_BY_BODY` is a registry of VALUES, and the +difference is not stylistic: "make X body-derived" is honest only while the instances differ in +their constants. When they differ in mechanism, a swapped constant is a rewrite wearing a seam's +clothes. + +KEYED BY (BODY, POLE), AND THE POLE IS NOT PADDING. It is what gives the seam two real, dissimilar, +shipping instances the day it lands instead of one plus a promise. Keyed by body alone, Earth would +be a single entry that re-dispatches on the pole internally — the parameterisation would then be +exercised by exactly one instance, which is the shape that passes by construction and proves +nothing until a second body arrives. Both bodies genuinely differ per pole, so the pole is a real +axis rather than a device: Earth's poles differ in mechanism, and a body whose two caps sit on +different terrain differs in whatever grades them. + +A PRODUCER DECLARES ITS OWN INPUTS. `sources` rides in the same record as `alpha` because the two +answer one question asked by two callers: the renderer asks what to paint, `cap_render.cap_sources` +asks what would make that paint stale. Split apart, a body can register a producer and leave its +caps frozen against the files that producer reads — `cap_is_fresh` compares mtimes, so a cap whose +source is not listed never notices the source changing. It is the same argument that makes +`bakes_coastline` one predicate read by both the render and the source list. An empty tuple is a +statement and not an omission: Earth's south reads no file, so nothing on disk could make its cap +stale, and nothing should be listed. + +THE WARP IS INJECTED RATHER THAN IMPORTED. A producer that reads a raster has to land it on the +cap's AEQD grid, and that grid — with its sphere, its extent and its pixel count — belongs to +`cap_render`, which imports `shade_planet`, which imports this package. Reaching back for it would +close that cycle. Taking a `WarpToCap` instead keeps the dependency pointing one way and makes a +producer drivable in a test with no GDAL behind it, which is how the alpha arithmetic gets an +oracle at all. + +NOT IN `bodies.py`, which opens by saying it is not a look and not a dataset; a producer is both. +The precedent followed here is `palette.look_for`: the consumer's own tier owns the mapping, keyed +by the body's slug, and refuses an unregistered body rather than falling back to Earth's. +""" + +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Protocol + +import numpy as np + +from pipeline import bodies, layers +from pipeline.acquire import download_sim3292 +from pipeline.render import mars_ice, palette, snow, viking_luma + + +class WarpToCap(Protocol): + """Land one 4326 source on the calling cap's AEQD grid and hand back band 1. + + `name` is the warp's label, not a path: the caller turns it into a file under that body's cap + work directory, prefixed for the pole it belongs to. A producer naming its own output path + would be spelling out a convention that `cap_render.cap_warp` already owns, and a half-labelled + warp set is exactly what that helper exists to prevent. + """ + + def __call__(self, source: str, name: str, resampling: str, dtype: str, + srcnodata: "float | None" = None) -> np.ndarray: ... + + +class BurnToCap(Protocol): + """Land one vector source on the calling cap's AEQD grid and hand back a boolean mask. + + THE TWIN OF `WarpToCap` AND INJECTED FOR ITS REASON, not for symmetry: the grid belongs to + `cap_render`, and a producer reaching back for it would close the import cycle that docstring + describes. `cap_render._bake_coastline` already burns vectors onto this exact grid, so what this + exposes is a capability the module has rather than one being invented for a body. + + A SEPARATE PROTOCOL FROM THE WARP BECAUSE THE TRAP IS DIFFERENT. `vector_raster` exists because + `gdal_rasterize` does not reproject and, handed a mismatched CRS, burns nothing while exiting 0 + and writing a well-formed raster. `must_draw` is the caller's claim that an empty result is + breakage; it has no default here for the reason `mars_ice.burn_unit` records — on a cap it + depends on the disc's edge latitude against that unit's own reach. + """ + + def __call__(self, source: Path, name: str, must_draw: "str | None") -> np.ndarray: ... + + +@dataclass(frozen=True) +class CapIceInputs: + """Everything a cap ice producer is allowed to see. + + Every field is passed to every producer whether or not it reads them, and that is deliberate: a + struct whose fields depend on which producer is registered would have to be built differently per + body, which puts a body branch back in the renderer this seam exists to remove. All are cheap — + the renderer computes the lon/lat grid for the light azimuth regardless, and the masks for the + composite. + """ + + #: `~(ocean | water)` on the cap grid — the tile composite's definition of land, so the two + #: agree across the crossfade rather than by coincidence. + land: np.ndarray + #: True latitude in degrees at each AEQD pixel centre. Per-PIXEL, not per-row: an AEQD disc has + #: no rows of constant latitude, so the Mercator path's 1-D form is wrong here. + latitude: np.ndarray + warp: WarpToCap + burn: BurnToCap + #: GROUND metres per pixel, never AEQD map metres. The two differ by `ground_metres_per_aeqd_unit` + #: — 0.533 on Mars — and a producer converting a ground distance into pixels with the map figure + #: draws it at roughly half the width its own constant claims. That has already happened once, + #: which is why this is supplied rather than left for each producer to derive. + ground_metres_per_px: float + + +@dataclass(frozen=True) +class CapIce: + """One body's perennial-ice answer for one pole: what it reads, and how it paints.""" + + #: Files whose change must restage this cap. Declared here rather than at the gate because only + #: the producer knows them, and `cap_is_fresh` requires every listed source to EXIST — so a + #: source named for a body that never opens it leaves that body's caps permanently stale. + #: + #: A CALLABLE, NOT A TUPLE, AND THE SUITE IS WHAT PROVED IT HAS TO BE. A tuple literal in the + #: registry below evaluates `snow.SP_NC` once, at import — so a caller that redirects that + #: constant is answered with the path from before the redirect, and the gate reports the wrong + #: file's existence. The code this replaced read `snow.SP_NC` at the call site, i.e. at call + #: time, so freezing it here would have been a silent narrowing introduced by the extraction + #: rather than a property of the seam. Every path in this pipeline hangs off `paths.DATA`, which + #: a relocated store moves; a zero-argument callable keeps the read where it was. + sources: Callable[[], tuple[Path, ...]] + alpha: Callable[[CapIceInputs], np.ndarray] + #: The `(sunlit, shadowed)` white `shade.composite` paints this cap's alpha with. + #: + #: PER POLE FOR FREE HERE, because the pole is already half of this registry's key — where the + #: composite tier keys on the layer and has to vary its paint within a window. That asymmetry is + #: why the two tiers declare the same thing by different means, and both must land on the same + #: value at the same pole or the 80–84 crossfade changes colour across the seam. + #: + #: A CALLABLE FOR THE REASON `sources` IS ONE: a tuple literal in the registry freezes whatever + #: `palette` held at import, so a test that swings a body's white would be answered with the + #: value from before the swing. + paint: Callable[[], tuple[Any, Any]] + + +def _earth_north(inputs: CapIceInputs) -> np.ndarray: + """NSIDC-0791 snow persistence, smoothstepped — Earth's Arctic land ice and perennial snow. + + The whole cap is north of `cap_render.CAP_EDGE_LAT` and therefore of `snow.RAMP_LAT_HI`, so + `snow_alpha`'s latitude ramp is CONSTANT across every pixel of it. Reproduced here with the + fixed high-latitude thresholds rather than by calling `snow_alpha`, whose per-row latitude is + Mercator-specific and would be wrong on an AEQD grid. + """ + sp_raw = inputs.warp(f'NETCDF:"{snow.SP_NC}":{snow.SP_VAR}', "sp", "bilinear", "Float32", + srcnodata=snow.SP_FILL) + persistence = snow.unpack_persistence(sp_raw) + low = snow.RAMP_LOW_MAX + high = low + snow.RAMP_BAND + fraction = np.clip((persistence - low) / (high - low), 0.0, 1.0) + return fraction * fraction * (3.0 - 2.0 * fraction) # float64, as before the N/S refactor + + +def _earth_south(inputs: CapIceInputs) -> np.ndarray: + """Antarctic land forced white — the one producer with no file behind it. + + NSIDC-0791 is NH-only and RGI region 19 is excluded, so there is no southern dataset to read and + no missing file that could ever switch this off. It is latitude and land and nothing else, which + is why it rides the body's layer declaration and why its `sources` tuple is empty rather than + unset. `snow.antarctic_snow_mask` is the one home for the rule; the tile composite calls the + same function, so the two agree across the −84 seam by construction. + """ + return snow.antarctic_snow_mask(inputs.land, inputs.latitude) + + +def _mars_sources() -> tuple[Path, ...]: + """The brightness field and both mapped units — every file either Mars pole opens. + + ONE LIST FOR BOTH POLES because both poles burn both units; see `_mars_cap_ice`. Listing a unit + a pole never opens would be the failure `CapIce.sources` warns about in reverse — here the risk + is the other direction, and every path named is genuinely read. + """ + return (viking_luma.luma_path(), + *(download_sim3292.unit_path(unit) for unit in mars_ice.NORTH_UNITS)) + + +def _mars_cap_ice(inputs: CapIceInputs, pole: str) -> np.ndarray: + """Viking luma graded between this pole's pinned levels, inside the mapped units, feathered. + + ONE FUNCTION FOR BOTH POLES, where Earth needs two. Earth's poles differ in MECHANISM — a NetCDF + warp against a latitude rule — and Mars's differ only in which constants they read, which is the + case the registry's own note says a parameterisation is honest for. + + THE LEVELS AND THE FIELD ARE ONE PAIRING AND NOTHING HERE CAN CHECK IT — `albedo_alpha` says so + in as many words. What makes it safe is that both come from the same two modules: the field is + `viking_luma`'s shipped raster, and `ALPHA_LEVELS` was measured over that raster and nothing + else. `scripts/measure_viking_levels.py --compare` is what refuses a drift between them. + + BOTH UNITS ARE BURNT AT BOTH POLES, which looks wasteful at the south and is not optional: + `extent_for` is one function serving this tier and the composite's straddling windows, so its + `np.where` evaluates both hemispheres' unions and a missing mask raises. The south's `Apu` is + computed and then discarded by that `where`, which is the correct extent — southern `Apu` is + layered deposits, not surface ice, and it covers 72% of that disc. + + THE FEATHER TAKES GROUND METRES, NEVER AEQD MAP METRES. `inputs.ground_metres_per_px` is + supplied for exactly this call; deriving it here from the grid is the bug that already shipped + once, drawing 5.33 km of Martian ground under a label saying 10. + """ + field = inputs.warp(str(viking_luma.luma_path()), "viking_luma", "bilinear", "Float32", + srcnodata=viking_luma.NODATA) + graded = mars_ice.albedo_alpha(field, mars_ice.ALPHA_LEVELS[pole], viking_luma.NODATA) + masks = { + unit: inputs.burn(download_sim3292.unit_path(unit), unit.lower(), + f"{unit} must reach the {pole} cap disc") + for unit in mars_ice.NORTH_UNITS + } + extent = mars_ice.extent_for(masks, pole == "north") + return graded * mars_ice.feather_alpha(extent, inputs.ground_metres_per_px) + + +#: Every producer that ships, by (body slug, pole). Earth's two entries are the seam's two real +#: instances — a NetCDF warp and a latitude rule, sharing nothing but their signature. +#: +#: MARS ARRIVED ONCE ITS FIELD HAD AN OWNER, which was the ordering this note used to record as the +#: reason for its absence: a producer cannot declare a path nothing acquired. `download_viking_mosaic` +#: and `render/viking_luma` closed that, and the OMEGA entries that once stood here are gone rather +#: than repointed because the licence blocks the source, not because the seam moved. +def _earth_cap_white() -> tuple[Any, Any]: + """Earth's one white at both poles, and the same pair its composite-tier producers declare. + + Read through `palette` rather than restated, so the cap and the tiles it feathers into cannot + disagree about the colour of the same ice sheet. + """ + return palette.SNOW_RGB, palette.SNOW_SHADOW_RGB + + +CAP_ICE_BY_BODY: dict[tuple[str, str], CapIce] = { + ("earth", "north"): CapIce(sources=lambda: (Path(snow.SP_NC),), alpha=_earth_north, + paint=_earth_cap_white), + ("earth", "south"): CapIce(sources=lambda: (), alpha=_earth_south, paint=_earth_cap_white), + ("mars", "north"): CapIce(sources=_mars_sources, + alpha=lambda inputs: _mars_cap_ice(inputs, "north"), + paint=lambda: palette.MARS_ICE_WHITE["north"]), + ("mars", "south"): CapIce(sources=_mars_sources, + alpha=lambda inputs: _mars_cap_ice(inputs, "south"), + paint=lambda: palette.MARS_ICE_WHITE["south"]), +} + + +def cap_ice(body: bodies.Body, pole: str) -> CapIce: + """The producer this body paints its perennial ice with at this pole. + + RAISES RATHER THAN FALLING BACK, on `palette.look_for`'s rule and for a sharper version of its + reason. A body inheriting Earth's ramp by omission renders a plausible planet in the wrong + colours; a body inheriting Earth's north PRODUCER by omission warps a NetCDF of northern- + hemisphere terrestrial snow persistence onto another world's pole and paints the result as that + world's ice — same latitudes, no missing file, no error, and a cap that looks like an + observation. Only asked of a body that declares the layer, so the raise is unreachable in a + correct configuration and is precisely a report that the two declarations disagree. + """ + try: + return CAP_ICE_BY_BODY[(body.name, pole)] + except KeyError: + raise KeyError( + f"{body.name} declares the {layers.PERENNIAL_ICE.name} layer but registers no " + f"{pole} cap producer; " + f"known: {sorted(CAP_ICE_BY_BODY)}" + ) from None diff --git a/pipeline/render/relief.py b/pipeline/render/relief.py index 11e3a02..b2460a9 100644 --- a/pipeline/render/relief.py +++ b/pipeline/render/relief.py @@ -13,5 +13,11 @@ def mercator_zfactor(latitude_deg: float, exaggeration: float) -> float: by 1/cos(lat), so a slope computed over grid-meters is too gentle by cos(lat). Scaling the vertical exaggeration by 1/cos(lat) restores the intended physical exaggeration (the hero's 15x) at every latitude. Shade in latitude bands, each - band using its mid-latitude here.""" + band using its mid-latitude here. + + THIS HANDLES THE LATITUDE TERM ONLY, which is all Earth needs. A body whose map units are not + ground metres — every planet but Earth, since the grid is EPSG:3857 for all of them — must also + divide by `bodies.ground_metres_per_mercator_unit`. The streaming shader takes that as a required + argument; this helper serves the region gdaldem branch, which is Earth by construction, so it is + stated here rather than parameterised.""" return exaggeration / math.cos(math.radians(latitude_deg)) diff --git a/pipeline/render/render_prep.py b/pipeline/render/render_prep.py index 915b7b7..0aa28c8 100644 --- a/pipeline/render/render_prep.py +++ b/pipeline/render/render_prep.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Prepare fused heightfield + ocean mask for Blender rendering. Warps both rasters from EPSG:4326 (degrees; east-west stretched away from the @@ -46,7 +45,6 @@ from pathlib import Path from typing import Any -import numpy as np import rasterio from rasterio.crs import CRS from rasterio.enums import Resampling diff --git a/pipeline/render/scene_build.py b/pipeline/render/scene_build.py index d1c3c2b..bb5fa31 100644 --- a/pipeline/render/scene_build.py +++ b/pipeline/render/scene_build.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Build the hero-render Blender scene from code (Phase 1 keystone). Reconstructs the hand-built Phase 0 scene — plane + adaptive-subdivision @@ -68,10 +67,24 @@ def _rgba(stops): # surface +7%, pinned relationally (the 98C5C8 drift's cure) SNOW_RGBA = (*palette.srgb8_to_linear(palette.SNOW_RGB), 1.0) # E8F1F6 -LAND_RANGE = (0.0, palette.LAND_MAX_M) # meters -> ramp position 0..1 -SEA_RANGE = (palette.SEA_MIN_M, 0.0) # meters -> ramp position 1..0 (reversed To) -LAND_STOPS = _rgba(palette.LAND_STOPS) -SEA_STOPS = _rgba(palette.SEA_STOPS) # the tile ramp: shelf-weighted stops to -6000 m +# THE HERO RIG IS EARTH'S, AND IT NOW SAYS SO. These four read `EARTH_LOOK` rather than the module +# globals they used to, and the difference is not cosmetic: read as globals they were the ramp, so a +# second body's arrival could not be seen from here at all. Read as a named look, the day heroes are +# wanted for another planet these four lines are exactly what has to answer for it — the rig is +# built once at import, so there is no per-call seam to thread and the honest thing is to name which +# planet it was built for. Heroes stay Earth-only by decision, not by omission. +_HERO_LOOK = palette.EARTH_LOOK +_HERO_SEA = _HERO_LOOK.sea +assert _HERO_SEA is not None, "Earth's look must carry a sea ramp; the hero rig builds one" + +# Both ends come off the Surface now. They used to read `extreme_m` and RESTATE the 0.0, which was +# the same Earth-is-the-datum assumption the ramp itself carried until `origin_m` — a third copy, +# in the one module a type checker was never going to connect to the other two. It is Earth-only +# today so nothing was wrong; it would have gone wrong silently on the first body with heroes. +LAND_RANGE = (_HERO_LOOK.land.origin_m, _HERO_LOOK.land.extreme_m) # meters -> ramp position 0..1 +SEA_RANGE = (_HERO_SEA.extreme_m, _HERO_SEA.origin_m) # meters -> ramp position 1..0 (reversed To) +LAND_STOPS = _rgba(_HERO_LOOK.land.stops) +SEA_STOPS = _rgba(_HERO_SEA.stops) # the tile ramp: shelf-weighted stops to -6000 m LAKE_STOPS = _rgba(palette.LAKE_STOPS) # depth-position ramp; stop 0 IS the water tint RAMP_INTERPOLATION = "EASE" diff --git a/pipeline/render/seaice.py b/pipeline/render/seaice.py index a899b24..dc7b303 100644 --- a/pipeline/render/seaice.py +++ b/pipeline/render/seaice.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Tile sea-ice layer: OSI SAF ice-frequency climatology -> soft white alpha over the sea. The sea-side mirror of pipeline/render/snow.py. Where snow drapes persistent white over LAND by a @@ -27,6 +26,7 @@ from pipeline import paths from pipeline.raster_io import band_window, row_bands +from pipeline.render import palette DATA = paths.DATA SEAICE_SRC = DATA / "raw/seaice/seaice_frequency_1991-2020_4326.tif" @@ -53,6 +53,25 @@ SH_ICE_MAX_ALPHA = 0.55 +def ice_white() -> "tuple[Any, Any]": + """The `(sunlit, shadowed)` pair `shade.composite` paints this layer's alpha with. + + ONE DECLARATION, TWO TIERS. The composite tier reaches it through its producer's `paint`; the + cap tier reads it here directly, the same way it already reads `ICE_LO` and `ICE_MAX_ALPHA` + rather than routing them through a registry. What matters is that the sentence "sea ice is + painted in this pair" has one home — not that both tiers take the same road to it. + + A FUNCTION RATHER THAN A CONSTANT, so a test that swings the palette is answered with the value + it swung to rather than whatever `palette` held when this module was imported. + + The values stay in `palette`, which is where colour lives; what lives here is the claim that + this layer is what they paint. They are a notch cooler and dimmer than land snow on purpose — + thin pack over dark ocean really is less bright than a thick snow sheet, and the two whites + keep the poles reading as floating ice against ice sheet without a hard colour split. + """ + return palette.ICE_RGB, palette.ICE_SHADOW_RGB + + def _run(cmd): subprocess.run([str(part) for part in cmd], check=True, capture_output=True) @@ -80,12 +99,10 @@ def warp_seaice_raster(bounds, width, height, out_path, src=SEAICE_SRC, band_row (`warp_persistence_raster`): the composite unpacks per window in float64, so a window slice of this raster must be bit-identical to warping that window alone. - WHY BANDS: the source is COARSE (0.1 deg, ~11 km) relative to the fine Web-Mercator target - (~305 m). A single whole-grid gdalwarp decimates the source read (the pole-inflated average scale - picks a reduced resolution and applies it everywhere), smoothing the ice edge. A band whose - latitude span is small keeps the local scale honest, so with band_rows == the composite window - height each band IS the per-window warp it replaces -- byte-identical by construction. See the snow - analog for the full argument. band_rows=None (region/cap grids) is a single direct warp. + WHY BANDS: `snow.warp_persistence_raster` holds the argument and the measurement that settled it. + This source is coarser still -- 0.1 deg (~11 km) against the ~305 m target -- so a whole-grid warp + decimates it the same way and smooths the ice edge. band_rows=None (region/cap grids) is a single + direct warp. """ left, bottom, right, top = bounds if band_rows is None or height <= band_rows: diff --git a/pipeline/render/sky_view.py b/pipeline/render/sky_view.py index 29c0e45..3d2c0ff 100644 --- a/pipeline/render/sky_view.py +++ b/pipeline/render/sky_view.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Post-render sky-view-factor shading: add topographic depth to a finished hero. The Cycles render lights terrain with one sun, so subtle relief (drainage, @@ -43,7 +42,8 @@ # at a resolution production does not have. Deriving both from ONE ground scale makes that # divergence unrepresentable rather than merely fixed. # -# 9784 is today's PLANET value (Z8_RES * 32), chosen here to keep production byte-identical while +# 9784 is today's PLANET value (Earth's grid pixel x 32), chosen here to keep production +# byte-identical while # the foundation lands; it is the number the softness work then moves. The hero path is deliberately # NOT on this constant: it shades per-country in an equal-area projection at its own tuned scale, # and folding it in would restage 204 renders to answer a tile question. @@ -88,7 +88,7 @@ def horizon_svf(heights: np.ndarray, m_per_px: float, n_dir: int = 16, dy, dx = np.sin(az), np.cos(az) mh = np.full_like(heights, -1e9) for distance_px in range(1, max_px): - zi = np.roll(np.roll(heights, -int(round(dy * distance_px)), 0), -int(round(dx * distance_px)), 1) + zi = np.roll(np.roll(heights, -round(dy * distance_px), 0), -round(dx * distance_px), 1) np.maximum(mh, (zi - heights) / (distance_px * m_per_px), out=mh) acc += 1.0 - np.sin(np.arctan(np.clip(mh, 0, None))) return acc / n_dir diff --git a/pipeline/render/snow.py b/pipeline/render/snow.py index a737d4c..2e5cbe5 100644 --- a/pipeline/render/snow.py +++ b/pipeline/render/snow.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Tile snow layer: NSIDC-0791 snow persistence -> latitude-ramped soft alpha. Replaces the WorldCover class-70 permanent-ice mask (permanent ice only -> bare mid/high-latitude @@ -9,7 +8,6 @@ stress regions; """ -import math import subprocess from pathlib import Path from typing import Any @@ -18,7 +16,7 @@ import rasterio from rasterio.transform import from_bounds -from pipeline import bodies, mercator, paths +from pipeline import mercator, paths from pipeline.raster_io import band_window, row_bands DATA = paths.DATA @@ -159,8 +157,9 @@ def latitude_per_row(top, bottom, height): rows = np.arange(height) merc_y = top - (rows + 0.5) * (top - bottom) / height # The second of the two copies this module and hillshade.py each used to carry, constant and - # formula alike. One home now, and the Earth binding is named rather than a bare literal. - return mercator.latitude_at(merc_y, bodies.EARTH.mercator_radius_m) + # formula alike. One home now, and it is the PROJECTION's sphere rather than a body's: the + # grid is EPSG:3857 for every planet, so this number is the same on all of them. + return mercator.latitude_at(merc_y, mercator.WEB_MERCATOR_RADIUS_M) def ramp_thresholds(latitude): diff --git a/pipeline/render/snow_mask.py b/pipeline/render/snow_mask.py index d697641..3815d39 100644 --- a/pipeline/render/snow_mask.py +++ b/pipeline/render/snow_mask.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Snow/ice mask stage for the hero shader. Produces snowmask_aea.png (0/255) on an existing render dir's grid from @@ -32,11 +31,8 @@ import concurrent.futures as cf import json import os -import shutil import subprocess import sys -import urllib.error -import urllib.request from pathlib import Path from typing import Any @@ -45,6 +41,7 @@ from rasterio.warp import transform_bounds from pipeline import paths +from pipeline.fetch import download_one BUCKET_URL = "https://esa-worldcover.s3.eu-central-1.amazonaws.com/v200/2021/map" DATA_DIR = paths.DATA / "raw/worldcover" @@ -73,33 +70,6 @@ def tiles_for_bounds(west, south, east, north) -> list[str]: return names -def download_one(url: str, dest: Path) -> str: - """download_glo30.py convention: .part + size check + atomic rename. - 404 -> 'absent': the bucket has no tile for all-ocean cells.""" - if dest.exists(): - return "skipped" - part = dest.with_suffix(".part") - try: - with urllib.request.urlopen(url, timeout=120) as resp: - expected = int(resp.headers.get("Content-Length", -1)) - with open(part, "wb") as out: - shutil.copyfileobj(resp, out) - actual = part.stat().st_size - if expected != -1 and actual != expected: - part.unlink() - return f"failed: size mismatch ({actual} of {expected} bytes)" - os.replace(part, dest) - return "ok" - except urllib.error.HTTPError as exc: - part.unlink(missing_ok=True) - if exc.code == 404: - return "absent" - return f"failed: {exc}" - except Exception as exc: - part.unlink(missing_ok=True) - return f"failed: {exc}" - - def main(): ap = argparse.ArgumentParser() ap.add_argument("--render-dir", type=Path, required=True, @@ -149,8 +119,8 @@ def main(): counts = {"ok": 0, "skipped": 0, "absent": 0} failures = [] with cf.ThreadPoolExecutor(WORKERS) as pool: - futures = {pool.submit(download_one, f"{BUCKET_URL}/{name}", - DATA_DIR / name): name for name in names} + futures = {pool.submit(download_one, f"{BUCKET_URL}/{name}", DATA_DIR / name, + timeout=120, absent_on_404=True): name for name in names} for index, fut in enumerate(cf.as_completed(futures), 1): status = fut.result() if status.startswith("failed"): diff --git a/pipeline/render/viking_luma.py b/pipeline/render/viking_luma.py new file mode 100644 index 0000000..86179a7 --- /dev/null +++ b/pipeline/render/viking_luma.py @@ -0,0 +1,239 @@ +"""Viking's colour mosaic as ONE band of brightness on a 4326 grid — the field both ice tiers read. + +WHY THIS STAGE EXISTS AT ALL. The acquired mosaic cannot be handed to either ice tier as published, +for two independent reasons. It is SimpleCylindrical METRES on the Mars sphere where everything +downstream speaks EPSG:4326 degrees; and it is three colour bands where the grading needs one +number. `cap_render`'s warp takes a 4326 source and returns band 1, and the composite tier warps a +4326 source onto the 3857 grid — so both consumers want the same thing, and this builds it once. + +WHOLE PLANET, DELIBERATELY, THOUGH ONLY THE POLES ARE EVER READ. A polar crop is about 240 MB +against this file's ~1 GB, and it would buy that back at the price of a crop latitude that must stay +looser than everything reading it — the cap frame's corners reach about 75.9 degrees and the tile +band starts at 76.5 — plus the guard that keeps it so. The failure mode of a crop set too tight is +ice that quietly stops appearing at the band edge, with no error anywhere. Against a project that +budgets tens of GB for a pyramid, the disk is the cheaper thing to spend. + +THE CHAIN IS THE ONE THE LEVELS WERE MEASURED THROUGH, and that is a constraint rather than a +preference. `ALPHA_LEVELS` is four percentiles of this exact field; a render that resampled the +mosaic differently would grade against levels taken from a different set of numbers. +`scripts/measure_viking_levels.py --compare` re-measures them through whatever this produces, which +is what makes the equality checkable rather than asserted. + + THE METRES-TO-DEGREES STEP IS A WARP, NOT A RELABEL, and it looks like a relabel until measured. + SimpleCylindrical is a linear function of latitude and longitude, so the conversion is a rescale + of the geotransform and nothing else — except that the publisher rounded `PixelResolution` to + 925.406, and 23059 of those overshoot the sphere's circumference by about 860 m. Taking the + transform at face value therefore puts the grid at -180.008 .. +180.008, not -180 .. +180. So the + grid is stated here and warped onto, rather than declared and hoped for. + +ZERO IS THE FILL AND IT SURVIVES THE COLLAPSE EXACTLY. Viking declares nodata 0 on each band and +means absent only where all three agree, which no scalar can express; `mars_ice.luma` is where the +argument lives, and the short version is that positive weights over non-negative channels vanish +only together. That is why the output can carry a scalar nodata of 0 and why it must be FLOAT and +not 8-bit: a pixel of (1, 0, 0) has luma 0.2126, and an integer round would file the darkest +measured ground on the planet under "never measured". + +Output (data/work/mars/ice/): + viking_luma_4326.tif Float32 brightness, nodata 0, 23059 x 11530 over the whole sphere + viking_luma_params.json the recipe: source edition, weights, grid, and the measured valid share + +Idempotency: the recipe sidecar is written LAST and compared on the next run, so a crash mid-pass +leaves the output stale rather than fresh. `--force` rebuilds regardless. + +Usage: + python3 -m pipeline.render.viking_luma # build if stale, else report and stop + python3 -m pipeline.render.viking_luma --force # rebuild unconditionally + python3 -m pipeline.render.viking_luma --check # report freshness, write nothing +""" + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any + +import numpy as np +import rasterio +from rasterio.windows import Window + +from pipeline import bodies +from pipeline.acquire import download_viking_mosaic +from pipeline.raster_io import row_bands +from pipeline.render import mars_ice + +#: The mosaic's own sphere. Warping out of SimpleCylindrical metres has to stay on one celestial +#: body — PROJ refuses to cross — so this is the intermediate the EPSG:4326 relabel then renames. +MARS_LONGLAT = "+proj=longlat +R=3396190 +no_defs" + +#: The output grid: the publisher's own pixel count over the exact sphere. Stated rather than +#: derived, per the module note on the rounded `PixelResolution`. +WIDTH = download_viking_mosaic.EXPECTED_WIDTH +HEIGHT = download_viking_mosaic.EXPECTED_HEIGHT + +#: Absent, on both the source's terms and the output's. See the module note: this survives the RGB +#: collapse exactly, which is the only reason one number can stand for three bands agreeing. +NODATA = 0.0 + +#: Rows per pass. The luma promotes to float64 internally, so this many rows of three bands is about +#: 1.2 GB of working set — sized to stay well inside the one-heavy-job cap rather than to be fast. +BAND_ROWS = 1024 + + +def work_dir() -> Path: + """This stage's directory, resolved at call time so a redirected data root moves it.""" + return bodies.work_dir(bodies.get("mars"), "ice") + + +def luma_path() -> Path: + """The brightness raster both ice tiers read.""" + return work_dir() / "viking_luma_4326.tif" + + +def recipe_path() -> Path: + """The freshness sidecar, beside the output it describes.""" + return work_dir() / "viking_luma_params.json" + + +def _run(argv: list[str]) -> None: + subprocess.run(argv, check=True) + + +def degrees_vrt(source: Path, out: Path) -> Path: + """The mosaic on a clean global degree grid, labelled EPSG:4326, as a VRT that copies nothing. + + TWO STEPS AND ONLY THE FIRST MOVES A PIXEL. The warp lands SimpleCylindrical metres on lon/lat + degrees of the SAME sphere, which PROJ will do; the translate then declares that grid to be + EPSG:4326, which is an identity on the angles because the latitudes are planetocentric and a + planetocentric latitude is the same angle on any figure. `fuse/relabel_mars` carries the full + argument for the second step and cannot be reused for the first: its precondition is a source + already in degrees, which this product fails. + + Both outputs are VRTs, so nothing here materialises 798 MB a second time. + """ + warped = out.with_name(out.name.replace(".vrt", "_marsdeg.vrt")) + _run(["gdalwarp", "-q", "-overwrite", "-of", "VRT", "-t_srs", MARS_LONGLAT, + "-te", "-180", "-90", "180", "90", "-ts", str(WIDTH), str(HEIGHT), + "-r", "bilinear", str(source), str(warped)]) + _run(["gdal_translate", "-q", "-of", "VRT", "-a_srs", "EPSG:4326", str(warped), str(out)]) + return out + + +def build(source: "Path | None" = None) -> tuple[Path, float]: + """Write the brightness raster, returning it and the share of the sphere that is measured. + + STREAMED IN ROW BANDS BECAUSE THE ANSWER IS WHAT DOES NOT FIT, the same shape as the feather's + banded pass one module over: the whole float64 luma of this grid is 2.1 GB before it is cast + down. Each band is read, collapsed and written, and nothing holds the planet at once. + + The valid share is MEASURED HERE rather than assumed, and returned so the recipe can record it. + A producer declares what it emitted; "how much of this file is data" is not knowable later from + a file that stores absence as a legal value. + """ + mosaic = source if source is not None else download_viking_mosaic.mosaic_path() + work_dir().mkdir(parents=True, exist_ok=True) + vrt = degrees_vrt(mosaic, work_dir() / "viking_4326.vrt") + + out = luma_path() + part = out.with_suffix(".tif.part") + measured = 0 + with rasterio.open(vrt) as reader: + profile: dict[str, Any] = dict( + driver="GTiff", width=reader.width, height=reader.height, count=1, + dtype="float32", nodata=NODATA, crs=reader.crs, transform=reader.transform, + tiled=True, compress="DEFLATE", bigtiff="YES") + with rasterio.open(part, "w", **profile) as writer: # pyright: ignore[reportCallIssue] + for row0, row1 in row_bands(reader.height, BAND_ROWS): + window = Window(0, row0, reader.width, row1 - row0) # pyright: ignore[reportCallIssue] + values = mars_ice.luma(reader.read(window=window)) + measured += int(np.count_nonzero(values)) + writer.write(values.astype(np.float32), 1, window=window) + part.replace(out) + return out, measured / float(WIDTH * HEIGHT) + + +def build_recipe(valid_fraction: float) -> str: + """Everything the output depends on besides the code that wrote it. + + THE SOURCE EDITION IS KEYED BY THE PUBLISHER'S DIGEST, not by a path or an mtime. A republished + mosaic under the same name is the failure the acquirer exists to catch, and this is what makes + that catch reach the derived raster instead of stopping at the download. + + THE WEIGHTS ARE HERE BECAUSE THEY REACH A PIXEL. A luma rebuilt through different weights is a + different field, and a field that changed without its recipe changing leaves a stale raster + looking fresh — the same rule that put every look constant into the composite's recipe. + """ + return json.dumps({ + "source": download_viking_mosaic.MOSAIC_NAME, + "source_md5": download_viking_mosaic.EXPECTED_MD5, + "luma_weights": list(mars_ice.LUMA_WEIGHTS), + "grid": {"width": WIDTH, "height": HEIGHT, "crs": "EPSG:4326", + "bounds": [-180.0, -90.0, 180.0, 90.0]}, + "nodata": NODATA, + "valid_fraction": round(valid_fraction, 6), + }, indent=2, sort_keys=True) + "\n" + + +def recorded_recipe() -> "dict[str, Any] | None": + """The recipe beside the output on disk, or None if there is none to read.""" + path = recipe_path() + if not path.exists(): + return None + try: + return json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError): + return None + + +def is_fresh() -> bool: + """Whether the raster can be skipped: on disk, with a recipe matching everything but the count. + + `valid_fraction` is EXCLUDED from the comparison because it is an OUTPUT of the build and not an + input to it — comparing it would ask the stage to predict its own result before running, and a + recipe that can never match makes an idempotent stage rebuild forever. + """ + recorded = recorded_recipe() + if recorded is None or not luma_path().exists(): + return False + expected = json.loads(build_recipe(0.0)) + return {key: value for key, value in recorded.items() if key != "valid_fraction"} == \ + {key: value for key, value in expected.items() if key != "valid_fraction"} + + +def build_parser() -> argparse.ArgumentParser: + """The CLI, split out of `main` so its contract is testable without running a pass.""" + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--force", action="store_true", + help="rebuild even if the recipe on disk already matches") + parser.add_argument("--check", action="store_true", + help="report freshness and stop; writes nothing") + return parser + + +def main() -> int: + args = build_parser().parse_args() + + if args.check: + print(f"{luma_path()}: {'fresh' if is_fresh() else 'STALE'}", flush=True) + return 0 + + if is_fresh() and not args.force: + print(f"{luma_path()} fresh -> skip", flush=True) + return 0 + + mosaic = download_viking_mosaic.mosaic_path() + if not mosaic.exists(): + sys.exit(f"{mosaic} is not on disk — run " + f"`python3 -m pipeline.acquire.download_viking_mosaic` first") + + print(f"collapsing {mosaic.name} -> Rec. 709 luma on a 4326 grid ...", flush=True) + out, valid_fraction = build(mosaic) + print(f"wrote {out} ({out.stat().st_size:,} bytes, " + f"{100 * valid_fraction:.2f}% measured)", flush=True) + recipe_path().write_text(build_recipe(valid_fraction), encoding="utf-8") # LAST, so a crash + print(f"wrote {recipe_path()}", flush=True) # leaves it stale + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pipeline/tile/cap_ladder.py b/pipeline/tile/cap_ladder.py index c970f10..eddcf89 100644 --- a/pipeline/tile/cap_ladder.py +++ b/pipeline/tile/cap_ladder.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Sweep ONE cap parameter across a ladder of values and archive each rung, for judging by eye. The polar caps are the one surface with no cheap preview. A region can be re-composited from cached @@ -45,7 +44,7 @@ from pathlib import Path from typing import Any, cast -from pipeline import bodies +from pipeline import bodies, planet_seam from pipeline.tile import cap_render, shade os.environ.setdefault("GDAL_CACHEMAX", "512") @@ -150,13 +149,17 @@ def render_rung(body: bodies.Body, axis: str, value: float) -> list[Path]: """ out = ladder_dir(body, axis) out.mkdir(parents=True, exist_ok=True) + # The seam's own answer, not an assumed full planet: a ladder run on a body with no masks must + # paint the same all-land cap the production pass would, or the rung being judged is not the + # picture that ships. + rasters = planet_seam.declared(body) archived: list[Path] = [] with swapped(axis, value): for grid, render in ((cap_render.north_grid(body), cap_render.render_cap_north), (cap_render.south_grid(body), cap_render.render_cap_south)): rung_grid = grid_for_rung(grid, axis, value) started = time.monotonic() - asset = render(rung_grid) + asset = render(rung_grid, rasters) seconds = time.monotonic() - started copy = out / f"cap_{grid.name}_{axis}_{value:g}{asset.suffix}" shutil.copy2(asset, copy) @@ -177,11 +180,12 @@ def restore_live_caps(body: bodies.Body) -> None: The sidecars are stamped AFTER that render, so the recipe on disk describes the pixels beside it, and the next production pass skips a ~14 GB composite instead of repeating it. """ + rasters = planet_seam.declared(body) for grid, render in ((cap_render.north_grid(body), cap_render.render_cap_north), (cap_render.south_grid(body), cap_render.render_cap_south)): - render(grid) + render(grid, rasters) sidecar = cap_render.cap_work_dir(body) / f"cap_{grid.name}_params.json" - sidecar.write_text(cap_render.cap_recipe(grid)) + sidecar.write_text(cap_render.cap_recipe(grid, rasters)) served = cap_render.caps_public_dir(body) served.mkdir(parents=True, exist_ok=True) (served / "caps.json").write_text(cap_render.caps_manifest(body) + "\n") diff --git a/pipeline/tile/cap_render.py b/pipeline/tile/cap_render.py index 9f29016..6d08f40 100644 --- a/pipeline/tile/cap_render.py +++ b/pipeline/tile/cap_render.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Render the polar caps: sea ice + snow over real bathymetry, on AEQD grids reaching the pole. Web-Mercator tiles die at ~85N (1/cos-phi sends the pole to infinity), so each pole is a @@ -11,7 +10,7 @@ Both poles share the projection/warp/coastline machinery but source their inputs differently: - NORTH: the fused planet VRTs (height/ocean/water) + NSIDC-0791 snow persistence + OSI SAF sea - ice. The whole cap is >78N, so snow_alpha's Mercator latitude ramp is CONSTANT here (reproduced + ice. The whole cap is >80N, so snow_alpha's Mercator latitude ramp is CONSTANT here (reproduced with fixed high-latitude thresholds). Inland water via lake_depth.inland_water (NEVER watercode.astype(bool) -- that caught class-1 ocean and flat-filled the Arctic sea, the disc-glow bug). @@ -20,7 +19,7 @@ an interior ring). Ocean -> bathymetry depth ramp + the SH half of the same sea-ice climatology. Snow is FORCED over Antarctic land, not read from a dataset (NSIDC-0791 is NH-only, RGI region 19 is excluded), via snow.antarctic_snow_mask (shared with the tile composite). Since the - pyramid carries Antarctica itself, the cap mirrors the north exactly (edge_lat -78, feathered + pyramid carries Antarctica itself, the cap mirrors the north exactly (edge_lat -80, feathered 81..84 over interior ice) and only covers the last smeared Mercator sliver. Two cap-specific twists vs the Mercator tiles: @@ -42,6 +41,7 @@ import argparse import json import subprocess +import sys from dataclasses import asdict, dataclass from pathlib import Path from typing import Any @@ -49,16 +49,25 @@ import numpy as np import rasterio from pyproj import Transformer -from scipy.ndimage import binary_dilation - -from pipeline import bodies, paths -from pipeline.render import hillshade, lake_depth, seaice, snow +from scipy.ndimage import ( + binary_dilation, + gaussian_filter, + map_coordinates, + uniform_filter1d, +) + +from pipeline import bodies, layers, naturalearth, planet_seam, vector_raster +from pipeline.render import hillshade, lake_depth, palette, perennial_ice, seaice from pipeline.tile import shade, terrain_rgb from pipeline.tile.shade import KNOBS -from pipeline.tile.shade_planet import (ALT, AZ, CAP_NORTH, CAP_SOUTH, EXAG, - composite_params) +from pipeline.tile.shade_planet import ( + ALT, + AZ, + CAP_NORTH, + CAP_SOUTH, + composite_params, +) -ROOT = paths.ROOT CAP_PX = 8192 # square texture side (south is a bigger disc -> coarser per px). 8192 chosen # by eye (crop A/B + /earth): visibly crisper coast/pack/sastrugi # at deep pole zoom; 3.2+2.1 MB WebP @@ -73,6 +82,15 @@ # cap's PROJECTED on-screen size: the untouched default camera paints the cap # at 110 CSS px, so the 8192 was a 74x linear oversupply for every visitor who # never zooms to a pole. Measured both caps: 162 KB / 570 KB / 1.7 MB / 5.1 MB. +CAP_EDGE_LAT = 80.0 # inscribed-circle latitude of the texture disc, north; the south mirrors it. + # EQUALS polarCaps.ts's MESH_EDGE_LAT and must stay >= it: the mesh spans this + # latitude to the pole and samples nothing outside the disc, so the two move + # together. The visible band opens at that file's FEATHER_LO 81 and is fully + # opaque from shade_planet.CAP_NORTH 84, so the whole ladder is + # |edge_lat| <= MESH_EDGE_LAT <= FEATHER_LO < |feather_hi| -- asserted in + # polarCaps.test.ts against the served caps.json rather than restated here. + # DO NOT raise CAP_PX chasing a finer cap: Mars's disc already oversamples its + # 200 m source, and RINGS (polarCaps.ts) makes the MESH the limit, not this. CAP_WEBP_QUALITY = 85 # gdal_translate WEBP quality — hero_variants' proven setting; rides in # cap_recipe because the encoder changes the shipped pixels CAP_ELEV_PX = 512 # elevation texture side; see cap_elev_asset for why there is only one size. @@ -125,7 +143,8 @@ def north_grid(body: bodies.Body) -> CapGrid: two: a module-level grid pins `body` at import, so every caller downstream inherits Earth by construction and no amount of argument-passing anywhere else can undo it. """ - return CapGrid(lat_0=90.0, edge_lat=78.0, px=CAP_PX, name="north", az_sign=-1.0, body=body) + return CapGrid(lat_0=90.0, edge_lat=CAP_EDGE_LAT, px=CAP_PX, name="north", az_sign=-1.0, + body=body) def south_grid(body: bodies.Body) -> CapGrid: @@ -135,32 +154,47 @@ def south_grid(body: bodies.Body) -> CapGrid: -- without a line they merge), but on the south white ice sits on teal ocean, which already separates itself; a dark line there just reads as a cartoonish outline around the continent. - The two sea-ice overrides and that coastline decision are EARTH LOOK CONSTANTS applied to - whatever body is passed. They are honest only while Earth is the only body: a second planet's - cryosphere is a Phase-1 question (the snow / sea-ice / glacier layers are Earth-only sources and - have to become off-switchable, not conditionally patched), and this is where it will surface. + The two sea-ice overrides and that coastline opacity are EARTH LOOK CONSTANTS applied to whatever + body is passed, and they stay that way ON PURPOSE now the layers are switchable. Opacity says how + strongly a line is drawn IF one is drawn at all; whether this body has a coastline dataset is a + separate fact, and it lives in `Body.surface_layers` where `bakes_coastline` reads it. Deriving + the opacity from the body instead would record the same fact twice -- as a 0.0 here and as an + entry in the recipe's `layers_off` -- which is the copy-drift this registry exists to remove. + A second body's look constants are a Phase-2 question, decided with its cap on screen. """ - return CapGrid(lat_0=-90.0, edge_lat=-78.0, px=CAP_PX, name="south", az_sign=1.0, body=body, - coast_opacity=0.0, coast_dilate=0, + return CapGrid(lat_0=-90.0, edge_lat=-CAP_EDGE_LAT, px=CAP_PX, name="south", az_sign=1.0, + body=body, coast_opacity=0.0, coast_dilate=0, ice_lo=seaice.SH_ICE_LO, ice_max_alpha=seaice.SH_ICE_MAX_ALPHA) # The coastline baked into the cap texture -- the land/sea line separating land snow from sea ice # where MapLibre's Mercator vector borders can't reach the pole. It must be DARK, not the globe's # white coast line: a white line vanishes between white snow and white ice. A muted steel-blue reads # delicately on both whites without going harsh. Line strength/width are per-cap (CapGrid). -COAST_SHP = ROOT / "data/raw/naturalearth/ne_10m_coastline/ne_10m_coastline.shp" +COAST_SHP = naturalearth.layer("ne_10m_coastline") COAST_RGB = (96, 122, 142) # muted steel-blue -def planet_work_dir(body: bodies.Body) -> Path: - """The fused planet rasters this body's caps warp from — heightfield, ocean mask, water mask. +def bakes_coastline(grid: CapGrid) -> bool: + """Whether this cap burns the land/sea line into its own pixels: look, then body, then disk. + + THREE QUESTIONS IN ONE PLACE, because they were about to become three answers in three. The look + question is silent -- the south sets `coast_opacity=0.0` deliberately and there is nothing to + announce about a decision that was made. The other two go through `layer_is_buildable`, so a body + without the layer prints the same sentence the four raster layers print. - Resolved per call rather than read out of `shade_planet`'s module scope, which is where it used - to come from. Reaching into another module for a path is the coupling that made this fragile: - that global is gone now the shade pass resolves its own, and a cap that silently sourced another - body's fused heightfield would render a clean Arctic and label it Mars. + THE BODY QUESTION CANNOT BE THE DISK QUESTION, which is the lesson this whole gate carries: + `COAST_SHP` is one global path to a Natural Earth product that is present on this box, so + `.exists()` returns True for Mars exactly as it does for Earth. Ordered the other way, a Martian + north cap would have had Earth's coastline reprojected onto it and blended in steel-blue -- a + crisp, confident, entirely fictional shoreline. + + One predicate, two callers: the render (which draws the line) and `cap_sources` (which makes it a + freshness dependency). Those must agree, or a cap depends on a file it never opens. """ - return bodies.work_dir(body, "planet") + if grid.coast_opacity <= 0.0: + return False + return layers.layer_is_buildable(grid.body, layers.COASTLINE, COAST_SHP, + "the cap ships with no land/sea line") def cap_work_dir(body: bodies.Body) -> Path: @@ -170,8 +204,10 @@ def cap_work_dir(body: bodies.Body) -> Path: their own contents, so a `body` key inside `cap__params.json` would restage a render that peaks ~14.3 GB to emit byte-identical pixels. Two bodies at two paths are two files already. - Also picks up the `MAPS_DATA` seam that the old `ROOT / "data/work/cap"` literal bypassed — a - relocated data store used to write its caps back into the checkout. + Also picks up the `MAPS_DATA` seam that the checkout-rooted literal it replaced bypassed — a + relocated data store used to write its caps back into the checkout. The old spelling is + DESCRIBED and not quoted, because it is exactly what `tests/test_paths.py` scans for and a + comment reproducing it re-creates the needle the scan exists to find. """ return bodies.work_dir(body, "cap") @@ -259,20 +295,21 @@ def write_cap_elevation(grid: CapGrid) -> Path: wherever the low byte wraps, every 256*step metres, so a resampled encode invents cliffs. Same rule build_tiles states for the pyramid's overviews. - No latitude feather: `encode_array`'s ramp is applied per ROW, which is a Mercator statement and - meaningless on an AEQD grid, and the cap must carry true elevation regardless. + True elevation, with no ramp toward the pole. That used to be a difference between this writer + and the tile one, which flattened its own metres from 78 to 85 degrees — so the two surfaces + `polarCaps.ts` crossfades between stood kilometres apart, and the plug rim above 84 rose clear + of the cap it was meant to hide beneath. `encode_array` now refuses any positional term at all, + which is what makes "identical encoding" mean identical heights rather than identical bytes. """ if CAP_PX % CAP_ELEV_PX: raise ValueError(f"CAP_ELEV_PX {CAP_ELEV_PX} must divide CAP_PX {CAP_PX}") warp = cap_height_warp(grid) if not warp.exists(): - _warp(grid, planet_work_dir(grid.body) / "planet_heightfield.vrt", warp, + _warp(grid, planet_seam.vrt_path(grid.body, "heightfield"), warp, "bilinear", "Float32") with rasterio.open(warp) as dataset: raw = dataset.read(1) - # The same nodata convention the composite applies before shading, so the elevation texture and - # the cap's own hillshade describe one surface rather than two. - heights = np.where(raw < -1e4, 0.0, raw).astype(np.float32) + heights = cap_heights(grid, raw) factor = CAP_PX // CAP_ELEV_PX metres = heights.reshape(CAP_ELEV_PX, factor, CAP_ELEV_PX, factor).mean(axis=(1, 3), @@ -298,8 +335,10 @@ def grid_recipe_fields(grid: CapGrid) -> dict: can move a cap pixel. TWO THINGS, BOTH LEARNED THE EXPENSIVE WAY. A bare `asdict` would inline the whole Body — - `path_prefix`, `tile_max_zoom`, `exaggeration` — and bind the caps' freshness to fields that - cannot change a cap pixel, restaging a render that peaks ~14 GB on an entirely unrelated edit. + `path_prefix`, `tile_max_zoom`, `map_units_per_pixel` — and bind the caps' freshness to fields + that cannot change a cap pixel, restaging a render that peaks ~14 GB on an unrelated edit. The + body fields that CAN move a cap pixel are named one at a time, here and in `cap_recipe`'s light + block, so that adding a field to the registry stays free until someone decides it is not. And the AEQD radius must be here: while it was a module constant it reached NO recipe at all (`asdict` serialises fields, and the projection string is a property), so changing it would have left both caps falsely fresh — the same untracked-input trap the composite's params exist to @@ -310,26 +349,160 @@ def grid_recipe_fields(grid: CapGrid) -> dict: """ fields = {key: value for key, value in asdict(grid).items() if key != "body"} fields["aeqd_radius_m"] = grid.body.aeqd_radius_m + smooth = POLE_SMOOTH_BY_BODY.get(grid.body.name) + # CONDITIONAL, on the `layers_off` idiom: a body whose altimetry reached its poles records + # nothing and keeps the recipe it has always had. It belongs HERE rather than in the light block + # — unlike `ground_scale`, this changes the heightfield itself, so the elevation texture reads it + # too and must restage with it. `cap_heights` is the one place that applies it, for that reason. + if smooth is not None: + fields["pole_smooth"] = asdict(smooth) return fields -def cap_recipe(grid: CapGrid) -> str: +@dataclass(frozen=True) +class PoleSmooth: + """How far a body's altimetry failed to reach its own pole, and how hard to smooth inside that. + + NOT A CARTOGRAPHIC PREFERENCE — `interpolated_lat` is an orbit. Read the module note. + """ + + interpolated_lat: float + angle_degrees: float + isotropic_km: float + taper_km: float + + +#: MARS ONLY, and it is a fact about one spacecraft rather than about poles in general. +#: +#: MGS flew at 92.9 degrees inclination, so MOLA's nadir tracks reached no higher than 87.1. Measured +#: in the MOLA team's own COUNTS_PER_BIN raster, which interpolates nothing: inside that circle +#: **96.5% of bins hold no observation at all**, against 63.7% outside. Every pixel the blend shows +#: there is a spline's opinion about the space between a few off-nadir tracks, and at 20x +#: exaggeration it rendered as a starburst of ridges radiating from the pole. +#: +#: The strength is not "as much as looked good". Smoothing lands the polar interior's directional +#: anisotropy on **0.83**, which is the value undisturbed terrain shows further out on this same +#: cap — so the interior stops being distinguishable from real ground rather than merely looking +#: calmer. `scripts/` has no reproducer for this one; the measurement lives in HISTORY. +#: +#: A body absent from here is smoothed not at all and records nothing in its recipe. Earth belongs +#: absent: its poles are not reconstructed from an altimeter that missed them. +POLE_SMOOTH_BY_BODY: dict[str, PoleSmooth] = { + "mars": PoleSmooth(interpolated_lat=87.1, angle_degrees=30.0, isotropic_km=4.0, + taper_km=40.0), +} + + +def cap_heights(grid: CapGrid, raw: np.ndarray) -> np.ndarray: + """The warped heightfield as every cap consumer must see it: nodata flattened, pole smoothed. + + ONE OWNER FOR BOTH CONSUMERS. The composite shades this and the elevation texture encodes it, + and they are required to describe ONE surface — a starburst suppressed in the shading but left + in the displacement mesh is the same defect wearing a different hat. + """ + heights = np.where(raw < -1e4, 0.0, raw).astype(np.float32) + return smooth_interpolated_pole(grid, heights) + + +def _pole_weight(grid: CapGrid, smooth: PoleSmooth, scale: float) -> "tuple[np.ndarray, float]": + """(strength per pixel, knee radius in px) — full inside the gap, zero outside it. + + The knee comes from the grid's OWN geometry rather than a radius constant: an azimuthal + equidistant cap is linear in colatitude, so the interpolated boundary sits at a fixed fraction of + the disc and follows `edge_lat` automatically if it ever moves again. + + A PLATEAU, NOT A CONE. Ramping from the pole outward puts almost no strength at the boundary, + which is exactly where the artifact is strongest — measured, after building that version first. + """ + knee_px = (90.0 - smooth.interpolated_lat) / (90.0 - abs(grid.edge_lat)) * (grid.px / 2.0) + taper_px = smooth.taper_km * 1000.0 / scale + centre = (grid.px - 1) / 2.0 + axis = np.arange(grid.px, dtype=np.float32) - centre + radius = np.hypot(axis[:, None], axis[None, :]) + t = np.clip((knee_px + taper_px / 2.0 - radius) / taper_px, 0.0, 1.0) + return (t * t * (3.0 - 2.0 * t)).astype(np.float32), knee_px + + +def smooth_interpolated_pole(grid: CapGrid, heights: np.ndarray) -> np.ndarray: + """Replace the interpolator's invented detail with the shape the data actually supports. + + SMOOTHED BY A CONSTANT ANGLE, because the artifact is radial: a spoke holds its angular width + while its arc width GROWS with radius, so a filter of fixed arc length kills it near the pole and + misses it further out. That was measured too — a constant-arc version left a visible ring. + + The isotropic pass afterwards is what takes the residual the angular pass cannot reach, since a + boxcar in angle leaves structure aligned with its own window. + """ + smooth = POLE_SMOOTH_BY_BODY.get(grid.body.name) + if smooth is None: + return heights + + scale = cap_ground_metres_per_px(grid) + weight, knee_px = _pole_weight(grid, smooth, scale) + outer = knee_px * 1.6 # cover the taper's outer half, which reaches past the knee + centre = (grid.px - 1) / 2.0 + + radii = np.arange(1.0, outer + 2.0, 1.0) + n_theta = max(64, round(2.0 * np.pi * outer)) + theta = np.linspace(0.0, 2.0 * np.pi, n_theta, endpoint=False) + rings = map_coordinates( + heights, [centre - np.outer(radii, np.cos(theta)), centre + np.outer(radii, np.sin(theta))], + order=1, mode="nearest") + window = max(3, round(smooth.angle_degrees / 360.0 * n_theta)) + rings = uniform_filter1d(rings, window, axis=1, mode="wrap") + + axis = np.arange(grid.px, dtype=np.float32) - centre + row, column = axis[:, None] * np.ones_like(axis)[None, :], np.ones_like(axis)[:, None] * axis + radius = np.hypot(row, column) + inside = radius <= outer + angle = np.mod(np.arctan2(column[inside], -row[inside]), 2.0 * np.pi) + flat = map_coordinates( + rings, [np.clip(radius[inside] - radii[0], 0.0, len(radii) - 1.0), + angle / (2.0 * np.pi) * n_theta], order=1, mode="wrap") + + angular = heights.copy() + angular[inside] = flat + blended = heights * (1.0 - weight) + angular * weight + isotropic = gaussian_filter(blended, smooth.isotropic_km * 1000.0 / scale) + return (blended * (1.0 - weight) + isotropic * weight).astype(np.float32) + + +def cap_recipe(grid: CapGrid, rasters: frozenset[str]) -> str: """Everything a cap's pixels depend on besides the source rasters, serialised for the freshness sidecar. Reuses shade_planet.composite_params — ONE recipe home — so any look change - that restages the tile composite also restages the caps: exactly the coupling whose absence let - once let both caps sit stale against the tiles they feather into, the north −6.7 DN - adrift. `fill_strength` is listed explicitly because - composite_params filters it out as hillshade-stage — for the tiles it rides in hs_params.json, - but the caps have no hillshade sidecar, so it must ride here.""" - return json.dumps({"grid": grid_recipe_fields(grid), - "light": {"az": AZ, "alt": ALT, "exag": EXAG, + that restages the tile composite also restages the caps: exactly the coupling whose absence once + let both caps sit stale against the tiles they feather into, the north −6.7 DN adrift. + `fill_strength` is listed explicitly because composite_params filters it out as hillshade-stage — + for the tiles it rides in hs_params.json, but the caps have no hillshade sidecar, so it must + ride here. + + `ground_scale` is recorded UNCONDITIONALLY, where `hs_params` records the Mercator one only when + it is not 1.0. Not an inconsistency: that idiom exists to keep a body whose value IS the identity + byte-identical, and no body's cap ratio is the identity (`bodies.ground_metres_per_aeqd_unit` + holds why), so a conditional here would never fire and would only read as though it might. + + It sits in the light block rather than in `grid_recipe_fields` deliberately. That helper is + shared with `cap_elev_recipe`, and the elevation texture encodes true metres with no slope in it + at all — a ground scale there would drag both displacement textures through a re-encode for a + number they never read. + + `layers_off` IS conditional, and for the reason the tile composite's is: Earth declares every cap + layer, so nothing is written and its recipe keeps the shape it has always had. It cannot be left + to `cap_sources` mtimes, because turning a layer off REMOVES its source from that list — the + dependency disappears along with the layer, and the cap would sit fresh wearing the old one.""" + absent = layers.layers_off(grid.body, layers.CAP_LAYERS) + missing = {"layers_off": absent} if absent else {} + return json.dumps({**missing, + "grid": grid_recipe_fields(grid), + "light": {"az": AZ, "alt": ALT, "exag": grid.body.exaggeration, + "ground_scale": bodies.ground_metres_per_aeqd_unit(grid.body), "fill_azimuth": hillshade.FILL_AZIMUTH, "fill_altitude": hillshade.FILL_ALTITUDE, "fill_strength": KNOBS["fill_strength"]}, "coast_rgb": list(COAST_RGB), "asset": {"format": "webp", "quality": CAP_WEBP_QUALITY, "rungs": list(CAP_RUNGS)}, - "composite": json.loads(composite_params({}))}, + "composite": json.loads(composite_params({}, grid.body, rasters))}, sort_keys=True, indent=2) @@ -341,7 +514,7 @@ def served_url(asset: Path) -> str: while a nesting body writes to `caps//` and would have had its whole texture set advertised one directory up — every rung a 404, discovered only by loading the globe. """ - return "/" + asset.relative_to(bodies.PUBLIC_ROOT).as_posix() + return "/" + asset.relative_to(bodies.public_root()).as_posix() def caps_manifest(body: bodies.Body) -> str: @@ -369,15 +542,27 @@ def caps_manifest(body: bodies.Body) -> str: }, sort_keys=True, indent=2) -def cap_sources(grid: CapGrid) -> list[Path]: +def cap_sources(grid: CapGrid, rasters: frozenset[str]) -> list[Path]: """The source files whose change must re-render this cap — composite_deps' sibling. Constants - ride in cap_recipe; these are the mtime dependencies.""" - planet = planet_work_dir(grid.body) - sources = [planet / "planet_heightfield.vrt", planet / "planet_oceanmask.vrt", - planet / "planet_watermask.vrt", Path(seaice.SEAICE_SRC)] - if grid.name == "north": - sources.append(Path(snow.SP_NC)) # the south's snow is FORCED, not read from a dataset - if grid.coast_opacity > 0.0: + ride in cap_recipe; these are the mtime dependencies. + + ONLY THE FILES THIS BODY'S CAP ACTUALLY OPENS. A source listed for a layer the body does not have + is not merely noise: `cap_is_fresh` requires every source to EXIST and to be older than the + oldest rung, so listing Earth's sea-ice climatology for a body that paints no sea ice ties that + body's caps to the mtime of a file whose contents can never reach a pixel of them. The layer's + own absence is tracked in `cap_recipe` instead, where turning it off restages exactly once. + """ + sources = [planet_seam.vrt_path(grid.body, raster) + for raster in planet_seam.PLANET_RASTERS if raster in rasters] + if layers.SEA_ICE.name in grid.body.surface_layers: + sources.append(Path(seaice.SEAICE_SRC)) + if layers.PERENNIAL_ICE.name in grid.body.surface_layers: + # ASKED OF THE PRODUCER, NOT SPELLED OUT HERE. This was `grid.name == "north"` plus Earth's + # NetCDF, which is two of Earth's facts written down as though they were the layer's: that + # only the north reads a file, and which file. Both are the producer's to state — Earth's + # south genuinely reads none, and a body grading its ice off its own rasters reads several. + sources.extend(perennial_ice.cap_ice(grid.body, grid.name).sources()) + if bakes_coastline(grid): sources.append(COAST_SHP) return sources @@ -416,6 +601,38 @@ def _warp(grid: CapGrid, src, out: Path, resampling: str, dtype: str, srcnodata= return dataset.read(1) +def cap_ground_metres_per_px(grid: CapGrid) -> float: + """Ground metres one cap pixel spans — NOT the AEQD map metres `edge_m` is measured in. + + THE TWO DIFFER BY A FACTOR OF TWO ON MARS and by a thousandth on Earth, which is exactly what + made getting it wrong survive review: every projection here is Earth-sphered, so a cap map-metre + is `ground_metres_per_aeqd_unit` ground metres — 0.533 for Mars. A ground distance converted to + pixels with the map figure draws at half the width its own constant claims, and the prototype + this replaces did precisely that with the ice feather. + + Public because it is the number a producer must be handed rather than derive, and because the + guard on it has to be able to name it. + """ + return (2.0 * grid.edge_m / grid.px) * bodies.ground_metres_per_aeqd_unit(grid.body) + + +def _burn(grid: CapGrid, source: Path, name: str, must_draw: "str | None") -> np.ndarray: + """Rasterize one vector source onto this cap's AEQD grid; return it as a boolean mask. + + The reproject-then-burn is `vector_raster.burn_onto_grid`, whose module note holds why those are + one act. Same shape as `_bake_coastline`'s call, which is the reason this is an exposure of an + existing capability rather than a new one. + """ + edge = grid.edge_m + work = cap_work_dir(grid.body) + burnt = vector_raster.burn_onto_grid( + source, grid.aeqd, (-edge, -edge, edge, edge), grid.px, grid.px, + projected=work / f"cap_{grid.name}_{name}_aeqd.gpkg", + out=work / f"cap_{grid.name}_{name}.tif", must_draw=must_draw) + with rasterio.open(burnt) as dataset: + return dataset.read(1) != 0 + + def _lonlat_grid(grid: CapGrid) -> tuple[np.ndarray, np.ndarray]: """True (longitude, latitude) in degrees at each AEQD pixel centre, via an exact AEQD->4326 transform. row 0 is the +y (pole-up) top of the image. lon drives the per-pixel light azimuth; @@ -431,13 +648,26 @@ def _lonlat_grid(grid: CapGrid) -> tuple[np.ndarray, np.ndarray]: def _shade(grid: CapGrid, heights: np.ndarray, longitude: np.ndarray) -> np.ndarray: """Combined light (main + fill) with the per-pixel longitude-rotated azimuth. heights get a 1-row edge halo top+bottom (hillshade_array wraps columns itself; the wrapped seam sits in the - unused corners past the edge latitude).""" + unused corners past the edge latitude). + + The exaggeration is the GRID's body, not this module's: it was imported from shade_planet, so + the caps drew every planet at Earth's relief however `--body` was set, and feathered that into + tiles shaded at the right one. A seam nothing would have flagged as a parameterisation bug. + + AND THE Z-FACTOR CARRIES THE CAP'S OWN UNIT CONVERSION, which is not the tiles'. `cell` is in + AEQD map units -- metres on `aeqd_radius_m`, the sphere PROJ forces every body onto -- while + `heights` are ground metres on the body itself, so the rise and the run are measured with + different rulers and the quotient is a slope in neither. Dividing the exaggeration by + `ground_metres_per_aeqd_unit` makes the units cancel. There is no latitude term to go with it, + unlike the Mercator path: AEQD tangential distortion inside the edge latitude is small, which is + the same reason a scalar z-factor is admissible here at all.""" cell = 2 * grid.edge_m / grid.px + zfactor = grid.body.exaggeration / bodies.ground_metres_per_aeqd_unit(grid.body) haloed = np.pad(heights, ((1, 1), (0, 0)), mode="edge") main_az = (AZ + grid.az_sign * longitude).astype(np.float32) fill_az = (hillshade.FILL_AZIMUTH + grid.az_sign * longitude).astype(np.float32) - shaded = hillshade.hillshade_array(haloed, cell, EXAG, ALT, main_az) - fill = hillshade.hillshade_array(haloed, cell, EXAG, hillshade.FILL_ALTITUDE, fill_az) + shaded = hillshade.hillshade_array(haloed, cell, zfactor, ALT, main_az) + fill = hillshade.hillshade_array(haloed, cell, zfactor, hillshade.FILL_ALTITUDE, fill_az) # No pole special-case: the rotating azimuth's pinwheel wash at the exact pole is quenched by # `shade.KNOBS["ice_relief_damp"]` (the pack conceals the shading that fed the wash). The # colat-3 flat taper that used to sit here was measured retirable at damp 0.75 and deleted: @@ -449,19 +679,20 @@ def _bake_coastline(grid: CapGrid, rgb: np.ndarray) -> None: """Blend the coastline as a subtle dark line over the cap RGB, in place. No-op when the cap opts out (`coast_opacity <= 0`, e.g. the south, where white ice on teal ocean self-separates). - ne_10m_coastline is 4326, so reproject to AEQD (gdal_rasterize does not reproject) before burning - it onto the cap grid; a small dilation makes it a ~3 px line rather than a 1 px thread at 4096. + The reproject-then-burn is `vector_raster.burn_onto_grid`, whose module note holds why those are + one act; a small dilation makes it a ~3 px line rather than a 1 px thread at 4096. """ - if grid.coast_opacity <= 0.0: + if not bakes_coastline(grid): return edge = grid.edge_m work = cap_work_dir(grid.body) - coast_aeqd = work / f"cap_{grid.name}_coast_aeqd.gpkg" - coast_tif = work / f"cap_{grid.name}_coast.tif" - _run(["ogr2ogr", "-overwrite", "-t_srs", grid.aeqd, str(coast_aeqd), str(COAST_SHP)]) - _run(["gdal_rasterize", "-q", "-burn", "1", "-init", "0", "-ot", "Byte", - "-te", str(-edge), str(-edge), str(edge), str(edge), - "-ts", str(grid.px), str(grid.px), str(coast_aeqd), str(coast_tif)]) + coast_tif = vector_raster.burn_onto_grid( + COAST_SHP, grid.aeqd, (-edge, -edge, edge, edge), grid.px, grid.px, + projected=work / f"cap_{grid.name}_coast_aeqd.gpkg", + out=work / f"cap_{grid.name}_coast.tif", + # A cap that bakes the line has already passed `bakes_coastline`, so the body declares the + # layer and the shapefile is there. An empty burn past that gate is the projection. + must_draw=f"{grid.body.name}'s {grid.name} cap coastline") with rasterio.open(coast_tif) as dataset: line = dataset.read(1) != 0 if grid.coast_dilate: @@ -472,13 +703,18 @@ def _bake_coastline(grid: CapGrid, rgb: np.ndarray) -> None: def _write_cap(grid: CapGrid, heights: np.ndarray, ocean: np.ndarray, water: np.ndarray, - snow_a: np.ndarray, ice_a: np.ndarray, hillshade_dn: np.ndarray) -> Path: + snow_a: np.ndarray, ice_a: "np.ndarray | None", hillshade_dn: np.ndarray, + snow_paint: "tuple[Any, Any] | None") -> Path: """Shared composite + coastline bake + WebP write for either pole. SVF off, measured: the tiles' ocean SVF is thresholded out over flat seafloor, so a cap SVF pass changes the ocean sub-perceptibly and does not close the cap<->tile seam -- the seam is projection/DEM, not SVF).""" occ = np.zeros((grid.px, grid.px), dtype=np.float32) # occ below threshold -> no SVF burn + # The sea-ice pair follows the alpha's own presence: `_cap_sea_ice` returns None where the body + # paints no pack, and one home answers both tiers for what that pack is painted in. rgb = shade.composite(heights, ocean, water, snow_a, hillshade_dn, occ, occ.shape, - (grid.px, grid.px), depth=None, ice_a=ice_a) + (grid.px, grid.px), depth=None, ice_a=ice_a, + look=palette.look_for(grid.body.name), snow_paint=snow_paint, + ice_paint=None if ice_a is None else seaice.ice_white()) _bake_coastline(grid, rgb) # the land/sea line, so ice sheet reads distinct from sea ice at the pole tif = cap_work_dir(grid.body) / f"cap_{grid.name}.tif" @@ -496,66 +732,157 @@ def _write_cap(grid: CapGrid, heights: np.ndarray, ocean: np.ndarray, water: np. return cap_asset(grid, grid.px) -def render_cap_north(grid: CapGrid) -> Path: - """North cap from the fused planet VRTs + snow persistence + sea ice.""" - planet = planet_work_dir(grid.body) - cap_work_dir(grid.body).mkdir(parents=True, exist_ok=True) - height = _warp(grid, planet / "planet_heightfield.vrt", cap_height_warp(grid), "bilinear", "Float32") - ocean_raw = _warp(grid, planet / "planet_oceanmask.vrt", cap_warp(grid, "ocean"), "near", "Byte") - watercode = _warp(grid, planet / "planet_watermask.vrt", cap_warp(grid, "water"), "near", "Byte") - sp_raw = _warp(grid, f'NETCDF:"{snow.SP_NC}":{snow.SP_VAR}', cap_warp(grid, "sp"), - "bilinear", "Float32", srcnodata=snow.SP_FILL) +def _cap_sea_ice(grid: CapGrid, consequence: str) -> "np.ndarray | None": + """This cap's sea-ice alpha, or None when the body has no sea ice — shared by both poles. + + ONE HOME BECAUSE BOTH POLES WARP THE SAME CLIMATOLOGY. The two renderers differ only in their + `ice_lo` / `ice_max_alpha` overrides, which ride on the grid; writing the gate out twice is how a + fix reaches one pole and not the other, which this module has already done once with GEBCO. + + None rather than a zero array: `shade.composite` takes `ice_a=None` and skips the blend entirely, + where zeros would run it and multiply the whole disc by nothing. + """ + if not layers.layer_is_buildable(grid.body, layers.SEA_ICE, Path(seaice.SEAICE_SRC), + consequence): + return None ice_raw = _warp(grid, seaice.SEAICE_SRC, cap_warp(grid, "seaice"), "bilinear", "Float32", srcnodata=seaice.ICE_FILL) + # No latitude term in ice_alpha -> valid on an AEQD grid. The south's fainter, pulled-in fringe + # comes from the grid's seaice.SH_ICE_* overrides, not from a second code path. + return seaice.ice_alpha(seaice.unpack_seaice(ice_raw), + ice_lo=grid.ice_lo, ice_max_alpha=grid.ice_max_alpha) + + +def _cap_perennial_ice(grid: CapGrid, ocean: np.ndarray, water: np.ndarray, latitude: np.ndarray, + consequence: str) -> "tuple[np.ndarray, tuple[Any, Any] | None]": + """This cap's perennial-ice alpha, from the producer this body registered for this pole. + + ONE HOME BECAUSE BOTH POLES ASK THE SAME QUESTION, exactly as `_cap_sea_ice` does — and here the + two answers are a NetCDF warp and a latitude rule, so writing the gate out twice is how a fix + reaches one pole and not the other. + + THE DISK HALF ASKS THE PRODUCER FOR ITS SOURCES. It used to name `snow.SP_NC` at the north call + site, which is correct for exactly one body at exactly one pole: a second planet declaring this + layer would have been gated on whether EARTH's climatology had been downloaded, and then warped + it onto its own pole. `all(...)` over an empty tuple is True, which is what lets Earth's south — + latitude and land, no file — pass on the body's declaration alone, with no special case. + + Zeros rather than None on the way out, unlike the sea-ice twin: `shade.composite` takes + `snow_a` as a required array and blends it, so an all-zero alpha is the arithmetic saying no + pixel is ice, where None would be a different function signature.""" + if not (layers.body_declares_layer(grid.body, layers.PERENNIAL_ICE, consequence) + and all(layers.layer_is_buildable(grid.body, layers.PERENNIAL_ICE, source, consequence) + for source in perennial_ice.cap_ice(grid.body, grid.name).sources())): + return np.zeros((grid.px, grid.px), dtype=np.float32), None + inputs = perennial_ice.CapIceInputs( + land=~(ocean | water), # the tile composite's land definition + latitude=latitude, + warp=lambda source, name, resampling, dtype, srcnodata=None: _warp( + grid, source, cap_warp(grid, name), resampling, dtype, srcnodata), + burn=lambda source, name, must_draw: _burn(grid, source, name, must_draw), + ground_metres_per_px=cap_ground_metres_per_px(grid), + ) + producer = perennial_ice.cap_ice(grid.body, grid.name) + return producer.alpha(inputs), producer.paint() + + +def _announce(grid: CapGrid, raster: str, consequence: str) -> None: + """Say what was skipped AND what follows from it, the way the layer gates do. + + Both masks announce separately even though one body switches both off together: a pass that goes + quiet about a skipped input is a pass whose output cannot be read back, and the case where they + diverge is the next one — a sea at a chosen contour, with no inland water behind it. + """ + print(f"{grid.body.name}'s planet stage emitted no {raster} -> {grid.name} cap: {consequence}", + flush=True) - heights = np.where(height < -1e4, 0.0, height).astype(np.float32) # DEM nodata -> flat, as hillshade does - ocean = ocean_raw != 0 - water = lake_depth.inland_water(watercode) - longitude, _lat = _lonlat_grid(grid) - hillshade_dn = _shade(grid, heights, longitude) - # Snow alpha: the whole cap is >CAP_EDGE_LAT (78) > snow.RAMP_LAT_HI (63), so snow_alpha's - # latitude ramp is CONSTANT here -- reproduce it with the fixed high-latitude thresholds rather - # than snow_alpha, whose per-row latitude is Mercator-specific and wrong on an AEQD grid. - persistence = snow.unpack_persistence(sp_raw) - low = snow.RAMP_LOW_MAX - high = low + snow.RAMP_BAND - fraction = np.clip((persistence - low) / (high - low), 0.0, 1.0) - snow_a = fraction * fraction * (3.0 - 2.0 * fraction) # float64, as before the N/S refactor +def _cap_masks(grid: CapGrid, rasters: frozenset[str], + shape: tuple[int, int]) -> tuple[np.ndarray, np.ndarray]: + """This cap's land/sea and inland-water selectors, warped when the planet has them. - ice_a = seaice.ice_alpha(seaice.unpack_seaice(ice_raw), # no latitude term -> valid on AEQD - ice_lo=grid.ice_lo, ice_max_alpha=grid.ice_max_alpha) - return _write_cap(grid, heights, ocean, water, snow_a, ice_a, hillshade_dn) + ALL-FALSE RATHER THAN AN ALL-ZERO RASTER ON DISK, which is the decision the whole planet seam + turns on. A sea-less body could have been handed a synthesised mask and nothing here would have + changed; but a file of zeros cannot be told apart from one produced by measuring the planet's + oceans and finding none, and it would be the only body fact in this project written as a + fabricated dataset. Derived here from the seam's declaration, it is arithmetic with a premise. + Gated per raster, not as a pair: Phase 2's chosen shoreline contour gives Mars an ocean mask + while it still has no inland water, and that combination must not need a code change. -def render_cap_south(grid: CapGrid) -> Path: + `shade.composite` takes both as plain boolean selectors, so all-False means every pixel takes + the land ramp — which is the answer, not a degraded stand-in for one. + """ + if "oceanmask" in rasters: + ocean = _warp(grid, planet_seam.vrt_path(grid.body, "oceanmask"), + cap_warp(grid, "ocean"), "near", "Byte") != 0 + else: + _announce(grid, "oceanmask", "every pixel takes the land ramp") + ocean = np.zeros(shape, dtype=bool) + if "watermask" in rasters: + water = lake_depth.inland_water( + _warp(grid, planet_seam.vrt_path(grid.body, "watermask"), + cap_warp(grid, "water"), "near", "Byte")) + else: + _announce(grid, "watermask", "no lake or river is drawn") + water = np.zeros(shape, dtype=bool) + return ocean, water + + +def render_cap_north(grid: CapGrid, rasters: frozenset[str]) -> Path: + """North cap from the fused planet VRTs + this body's north perennial ice + sea ice. + + The two cryosphere layers are gated on the BODY, not on their files: every source is a single + global path to an Earth dataset, so an existence check passes for every planet and would have + painted an Arctic onto whatever was rendered. Off, the cap is bare relief and bathymetry, which + is a complete picture rather than a degraded one. + + WHICH ice, and read from WHAT, is `perennial_ice.cap_ice(body, "north")`'s answer rather than + this function's. Earth's is NSIDC-0791 persistence; the north pole of another world is another + mechanism, not another path, and the two must not be spelled out here or the renderer grows a + body branch per planet. + """ + cap_work_dir(grid.body).mkdir(parents=True, exist_ok=True) + height = _warp(grid, planet_seam.vrt_path(grid.body, "heightfield"), cap_height_warp(grid), + "bilinear", "Float32") + + heights = cap_heights(grid, height) + ocean, water = _cap_masks(grid, rasters, heights.shape) + longitude, latitude = _lonlat_grid(grid) + hillshade_dn = _shade(grid, heights, longitude) + + snow_a, snow_paint = _cap_perennial_ice(grid, ocean, water, latitude, "the north cap paints no ice") + ice_a = _cap_sea_ice(grid, "the north cap paints no pack ice") + return _write_cap(grid, heights, ocean, water, snow_a, ice_a, hillshade_dn, snow_paint) + + +def render_cap_south(grid: CapGrid, rasters: frozenset[str]) -> Path: """South (Antarctica) cap from the fused planet VRTs + the SH half of the sea-ice climatology. Re-sourced from GEBCO-direct when the Antarctica fill pushed the planet VRTs to -90: the cap now shades the SAME fused heightfield and masks as the tiles, so the tone across the -84 cap<->tile crossfade agrees by construction (the GEBCO cap measured ~2.5 DN darker than - the tiles it feathered into -- the visible interior ring). Snow is FORCED over Antarctic land - (no SH snow dataset, no RGI region 19), exactly as the tile composite does. + the tiles it feathered into -- the visible interior ring). Earth's ice here is FORCED over + Antarctic land (no SH snow dataset, no RGI region 19), exactly as the tile composite does. + + THAT FORCED PATCH IS THE ONE PRODUCER WITH NO FILE BEHIND IT, which is why a producer's + `sources` may honestly be empty. It is latitude and land and nothing else, so no missing dataset + could ever switch it off; it rides the `perennial_ice` layer via `body_declares_layer`. Left + ungated on a body with no sea, it whitens every piece of land below 60 degrees south -- a polar + ice cap invented out of arithmetic. """ - planet = planet_work_dir(grid.body) cap_work_dir(grid.body).mkdir(parents=True, exist_ok=True) - height = _warp(grid, planet / "planet_heightfield.vrt", cap_height_warp(grid), "bilinear", "Float32") - ocean_raw = _warp(grid, planet / "planet_oceanmask.vrt", cap_warp(grid, "ocean"), "near", "Byte") - watercode = _warp(grid, planet / "planet_watermask.vrt", cap_warp(grid, "water"), "near", "Byte") - ice_raw = _warp(grid, seaice.SEAICE_SRC, cap_warp(grid, "seaice"), - "bilinear", "Float32", srcnodata=seaice.ICE_FILL) + height = _warp(grid, planet_seam.vrt_path(grid.body, "heightfield"), cap_height_warp(grid), + "bilinear", "Float32") - heights = np.where(height < -1e4, 0.0, height).astype(np.float32) # DEM nodata -> flat, as hillshade does - ocean = ocean_raw != 0 - water = lake_depth.inland_water(watercode) + heights = cap_heights(grid, height) + ocean, water = _cap_masks(grid, rasters, heights.shape) longitude, latitude = _lonlat_grid(grid) hillshade_dn = _shade(grid, heights, longitude) - land = ~(ocean | water) # the tile composite's land definition - snow_a = snow.antarctic_snow_mask(land, latitude) # Antarctica = permanent ice -> forced white - ice_a = seaice.ice_alpha(seaice.unpack_seaice(ice_raw), # fainter, pulled-in fringe (seaice.SH_ICE_*) - ice_lo=grid.ice_lo, ice_max_alpha=grid.ice_max_alpha) - return _write_cap(grid, heights, ocean, water, snow_a, ice_a, hillshade_dn) + snow_a, snow_paint = _cap_perennial_ice(grid, ocean, water, latitude, "polar land stays on the relief ramp") + ice_a = _cap_sea_ice(grid, "the south cap paints no pack ice") + return _write_cap(grid, heights, ocean, water, snow_a, ice_a, hillshade_dn, snow_paint) def build_parser() -> argparse.ArgumentParser: @@ -582,6 +909,19 @@ def build_parser() -> argparse.ArgumentParser: def main() -> int: args = build_parser().parse_args() body = bodies.get(args.body) # raises on an unknown name; never falls back to Earth + # THE BODY BEFORE ANYTHING ELSE, and a refusal rather than a quiet exit 0. The shade pass already + # declines to invoke this for such a body, so reaching here means an operator asked directly — + # and the honest answer to "render Mars's caps" is that this body publishes none, not a pair of + # discs in a palette it has never been given. Same rule the layer gates follow: ask the body, then + # the disk, because the disk cannot tell "publishes none" from "the render died". + if not body.renders_polar_caps: + sys.exit(f"{body.name} publishes no polar caps — nothing to render. Its relief would shade " + f"from the same ramps as the tiles, so turning this on is a look decision: set " + f"renders_polar_caps on the body in pipeline/bodies.py once they are ratified.") + # Read once and threaded, exactly as the shade pass does it. This raises when the planet stage + # never finished, so a cap can never be rendered from half a fusion — which used to be + # indistinguishable from a planet that genuinely has no masks. + rasters = planet_seam.declared(body) for wanted, grid, render in ((not args.south, north_grid(body), render_cap_north), (not args.north, south_grid(body), render_cap_south)): @@ -589,13 +929,13 @@ def main() -> int: continue work = cap_work_dir(grid.body) if not args.elev_only: - recipe = cap_recipe(grid) + recipe = cap_recipe(grid, rasters) sidecar = work / f"cap_{grid.name}_params.json" if not args.force and cap_is_fresh(recipe, cap_assets(grid), sidecar, - cap_sources(grid)): + cap_sources(grid, rasters)): print(f"cap {grid.name} fresh -> skip", flush=True) else: - print(f"wrote {render(grid)}", flush=True) + print(f"wrote {render(grid, rasters)}", flush=True) sidecar.write_text(recipe) # AFTER the render, so a crash leaves the cap stale # Gated separately, and NOT behind the colour stage's `continue`: the displacement texture @@ -605,7 +945,7 @@ def main() -> int: elev_sidecar = work / f"cap_{grid.name}_elev_params.json" if not args.force and cap_is_fresh( elev_recipe, [cap_elev_asset(grid)], elev_sidecar, - [planet_work_dir(grid.body) / "planet_heightfield.vrt"]): + [planet_seam.vrt_path(grid.body, "heightfield")]): print(f"cap {grid.name} elevation fresh -> skip", flush=True) else: print(f"wrote {write_cap_elevation(grid)}", flush=True) diff --git a/pipeline/tile/pack_pmtiles.py b/pipeline/tile/pack_pmtiles.py index db0f1cf..c946389 100644 --- a/pipeline/tile/pack_pmtiles.py +++ b/pipeline/tile/pack_pmtiles.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Pack the XYZ tile pyramid into an MBTiles file — the bridge to `pmtiles convert`. The vendored go-pmtiles CLI reads only MBTiles (its GDAL driver counterpart is @@ -17,8 +16,7 @@ `tms_row` is the single home of that conversion, pinned by tests either way (a silent flip error would serve a vertically mirrored planet). -Usage: python -m pipeline.tile.pack_pmtiles \ - [--tiles data/work/planet_tiles/tiles] [--out data/work/planet_tiles/planet.mbtiles] +Usage: python -m pipeline.tile.pack_pmtiles --body earth [--tiles DIR] [--out FILE] [--name NAME] """ import argparse import os @@ -27,11 +25,13 @@ import time from pathlib import Path -ROOT = Path(__file__).resolve().parents[2] -DEFAULT_TILES = ROOT / "data/work/planet_tiles/tiles" -DEFAULT_OUT = ROOT / "data/work/planet_tiles/planet.mbtiles" -# The pyramid's geographic extent since the Antarctica fill: the full -# Web-Mercator square. Metadata only — pmtiles carries it through to the archive header. +from pipeline import bodies + +#: The pyramid's geographic extent: the full Web-Mercator square, since the Antarctica fill. Shared +#: by every body rather than derived from one, and that is not an oversight — PROJ refuses to +#: project a Mars-radius sphere into EPSG:3857 at all, so every planet is cut on this same grid and +#: the square is a fact about the PROJECTION, not about the planet in it. Metadata only; pmtiles +#: carries it through to the archive header. BOUNDS = "-180.0,-85.0511,180.0,85.0511" INSERT_BATCH = 2048 # File suffix -> the MBTiles `format` string for it. Also the filter that decides what counts as a @@ -39,6 +39,22 @@ TILE_SUFFIXES = {".png": "png", ".webp": "webp", ".jpg": "jpg"} +def default_tiles(body: bodies.Body) -> Path: + """The cut pyramid this body's pack reads, unless an operator names another. + + Derived at CALL time from `bodies.work_dir`, which is the one home for "where does this body's + stage live" — so a relocated `MAPS_DATA` moves the pack along with the cut that filled it. The + literal this replaced was joined onto the checkout, which made a relocated store pack from a + directory nothing had written: not an error, just an empty tree or a stale one. + """ + return bodies.work_dir(body, "planet_tiles") / "tiles" + + +def default_out(body: bodies.Body) -> Path: + """Where this body's MBTiles bridge is written. Beside its own pyramid, never beside another's.""" + return bodies.work_dir(body, "planet_tiles") / "planet.mbtiles" + + def tms_row(zoom: int, xyz_row: int) -> int: """XYZ row (origin top-left) -> TMS row (origin bottom-left) at this zoom.""" return (1 << zoom) - 1 - xyz_row @@ -113,11 +129,26 @@ def pack_directory(tiles_dir: Path, out_mbtiles: Path, name: str) -> int: def main() -> int: parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) - parser.add_argument("--tiles", type=Path, default=DEFAULT_TILES) - parser.add_argument("--out", type=Path, default=DEFAULT_OUT) + # REQUIRED, WITH NO DEFAULT, on the same argument the rest of the tile chain makes: packing the + # wrong planet's directory is not a loud failure. It finds tiles, packs them, and writes a + # complete, valid archive under the other body's name — discovered only when a globe draws Earth. + parser.add_argument("--body", required=True, + help=f"which planet's pyramid to pack " + f"({', '.join(sorted(bodies.BODIES))})") + parser.add_argument("--tiles", type=Path, default=None, + help="override the pyramid directory (this is how the terrain pyramid, " + "which is a sibling of the relief one, is packed)") + parser.add_argument("--out", type=Path, default=None, + help="override the MBTiles path") + # NOT body-derived, deliberately — see `TestTheArchiveNameIsNotTheBodys`. It reads + # {site}-{layer}, the body rides in the path, and this string reaches the archive header, which + # is inside the SHA that becomes the tile token in every served URL. parser.add_argument("--name", default="terrella-relief") args = parser.parse_args() - pack_directory(args.tiles, args.out, name=args.name) + body = bodies.get(args.body) + tiles = args.tiles if args.tiles is not None else default_tiles(body) + out = args.out if args.out is not None else default_out(body) + pack_directory(tiles, out, name=args.name) return 0 diff --git a/pipeline/tile/shade.py b/pipeline/tile/shade.py index fadc5e2..b289ef4 100644 --- a/pipeline/tile/shade.py +++ b/pipeline/tile/shade.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Shade planet chunks into one seamless Web Mercator RGB raster, ready to tile. Reproject each chunk's height + masks @@ -25,7 +24,7 @@ from rasterio.enums import Resampling from scipy.ndimage import zoom -from pipeline import paths +from pipeline import bodies, paths from pipeline.raster_io import GTIFF_CREATE from pipeline.render import hillshade, lake_depth, palette, relief, snow from pipeline.render.sky_view import ( @@ -37,7 +36,11 @@ DATA = paths.DATA CHUNKS = DATA / "work/planet/chunks" Z8_MERC_RES = 305.7483 # metres/pixel of a 512px WebMercatorQuad tile at zoom 8 -EXAG = 15.0 +EXAG = palette.EXAGGERATION # the region path exists to PREDICT the planet, so it cannot hold its + # own copy of a look value. This was a third literal 15.0, uncovered by + # the guard that calls the hero/planet pair "the last copy-pair". It + # stays Earth-shaped: this path takes --cells, not --body, and a + # Copernicus cell name is not a thing another planet has. MERCATOR = "EPSG:3857" class Knobs(TypedDict): @@ -164,9 +167,9 @@ def reproject_cell(name: str, merc_dir: Path): with rasterio.open(height) as dataset: te = [repr(value) for value in dataset.bounds] ts = [str(dataset.width), str(dataset.height)] - for layer in ("oceanmask", "watermask"): + for raster in ("oceanmask", "watermask"): run(["gdalwarp", "-overwrite", "-q", "-t_srs", MERCATOR, "-te", *te, "-ts", *ts, - "-r", "near", chunk / f"{layer}_10s.tif", merc_dir / f"{name}_{layer}.tif"]) + "-r", "near", chunk / f"{raster}_10s.tif", merc_dir / f"{name}_{raster}.tif"]) def build_vrt(vrt_path, sources): @@ -273,6 +276,11 @@ def main(): print(f"hillshade: per-row z-factor (EXAG={EXAG}/cos(lat)), custom seamless shader" f"{shadow_note}", flush=True) hillshade.per_row_zfactor_hillshade(height_vrt, hs_tif, EXAG, KNOBS["alt"], 315.0, + # Spelled through the registry rather than as a bare + # 1.0: this path takes Copernicus cells, so Earth is + # not a default here, it is the subject. + ground_scale=bodies.ground_metres_per_mercator_unit( + bodies.EARTH), fill_strength=KNOBS["fill_strength"], shadow_strength=KNOBS["shadow_strength"], shadow_reach_px=int(KNOBS["shadow_reach"])) @@ -334,8 +342,13 @@ def main(): low = np.nan_to_num(np.where(low < -500, np.nan, low), nan=0.0) occ = normalised_occlusion(low, m_per_px) + # Earth's look for the same reason EXAG above is Earth's exaggeration: this path predicts the + # planet and takes --cells, and a Copernicus cell name is not a thing another planet has. + # Earth's whites for the reason the look above is Earth's: this path predicts the planet and + # takes --cells, and a Copernicus cell name is not a thing another planet has. rgb = composite(heights, ocean, water, snow_a, hs, occ, (sh, sw), (grid_h, grid_w), - depth=depth) + depth=depth, look=palette.EARTH_LOOK, + snow_paint=(palette.SNOW_RGB, palette.SNOW_SHADOW_RGB), ice_paint=None) out_tif = args.out / "region_rgb.tif" with rasterio.open(height_vrt) as src: @@ -479,9 +492,44 @@ def snow_position(light, curve): raise ValueError(f"unknown snow_curve {curve!r} (linear | gamma4 | gamma8 | knee)") -def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=None, ice_a=None): +def paint_end(value) -> np.ndarray: + """One end of a light-keyed white as float32 broadcastable against `(3, H, W)`. + + A bare `(R, G, B)` becomes `(3, 1, 1)`; anything already shaped per row or per pixel is passed + through, so a producer chooses its own granularity and this function never has to know which. + Public because the callers that assemble a paint want the same normalisation the blend applies. + """ + array = np.asarray(value, dtype=np.float32) + return array.reshape(3, 1, 1) if array.ndim == 1 else array + + +def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=None, ice_a=None, + *, look: palette.Look, snow_paint, ice_paint): """Composite one window of the planet/region from ELEVATION, not pre-coloured rasters. + `look` is the body's ramp pair and is REQUIRED — this is the one function that turns elevation + into colour, so a default here would be a whole planet's palette chosen by omission. + + THE WHITES ARE REQUIRED FOR THE SAME REASON AND THIS FUNCTION NO LONGER KNOWS ANY OF THEM. + `snow_paint` and `ice_paint` are each `(sunlit, shadowed)` or None, handed over by whoever + computed the alpha beside them, because the code that knows what material a pixel is, is the + code that decided to paint it at all. This module used to read `palette.SNOW_RGB` directly, + which is Earth's white, and a second body then painted its ice in it — silently, since a global + read cannot show a reader that another planet exists. Mars's two poles measure as different + colours from each other, so there is not one white even within a body. + + BOTH MOVED TOGETHER even though only one had been falsified, because leaving the sea-ice pair + behind would make this function half-ignorant of colour: a reader would have to discover that + one white arrives and one is fetched, and the rule dividing them is a fact about the past. + + Each end broadcasts against `(3, H, W)`: `(3, 1, 1)` for a constant, `(3, H, 1)` to vary by row. + The blend already materialises the full `(3, H, W)` colour array either way, so a constant or a + per-row paint costs no memory at all and only a genuinely per-pixel one costs more. + + None means this window paints no such white and the blend is SKIPPED rather than run against a + zero alpha — the idiom `ice_a=None` already sets here, and bit-identical to running it, since + `base * 1.0 + anything * 0.0` is exactly `base` in float32. + `heights` is metres on the fused heightfield; the land and sea ramps are applied here via `palette.relief_lut`, which replaced two `gdaldem color-relief` passes. Those cost **28:19 and 24.4% of all pass CPU**, single-threaded, each reading the full 31 GB @@ -499,8 +547,8 @@ def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=Non # float32 throughout — the output is 8-bit, and on the full-width planet windows float64 # doubled peak RAM (~18 GB) and OOM-killed the box. asarray is a no-op when already float32. heights = np.asarray(heights, dtype=np.float32) - land = palette.lut_lookup(palette.relief_lut("land"), "land", heights).astype(np.float32) - sea = palette.lut_lookup(palette.relief_lut("sea"), "sea", heights).astype(np.float32) + land = palette.lut_lookup(palette.relief_lut("land", look=look), "land", heights, + look=look).astype(np.float32) hs = np.asarray(hs, dtype=np.float32) snow_a = np.asarray(snow_a, dtype=np.float32) occ = np.asarray(occ, dtype=np.float32) @@ -509,9 +557,25 @@ def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=Non * np.array([1.0, 1.0 - 0.5 * KNOBS["warmth"], 1.0 - KNOBS["warmth"]], dtype=np.float32).reshape(3, 1, 1), 0, 255) - sea_lum = 0.299 * sea[0] + 0.587 * sea[1] + 0.114 * sea[2] - sea = np.clip(sea_lum[None] + (sea - sea_lum[None]) * KNOBS["sea_saturation"], 0, 255) - color = np.where(ocean[None], sea, land) + if look.sea is None: + # A body that draws no sea. The caller's ocean mask comes from the planet seam's + # DECLARATION, so on such a body it is all-False and `np.where` would select the sea ramp + # nowhere — skipping it states that, where building a second ramp would mean inventing a + # colour no pixel is painted in. The check is not defensive: an ocean mask arriving with + # pixels set means the look and the declaration disagree about the planet, and every one of + # those pixels would silently render as land. + if bool(np.any(ocean)): + raise ValueError( + "look draws no sea but the ocean mask has pixels set — the body's planet seam " + "declares an oceanmask its look has no ramp for, so sea would render as land." + ) + color = land + else: + sea = palette.lut_lookup(palette.relief_lut("sea", look=look), "sea", heights, + look=look).astype(np.float32) + sea_lum = 0.299 * sea[0] + 0.587 * sea[1] + 0.114 * sea[2] + sea = np.clip(sea_lum[None] + (sea - sea_lum[None]) * KNOBS["sea_saturation"], 0, 255) + color = np.where(ocean[None], sea, land) # Inland water: flat WATER_RGB by default. Where a lake carries GLOBathy depth, ramp it # instead -- on ABSOLUTE depth, never normalised per lake, since a per-lake normalisation # is the artificial gradient the prototype was rejected for (a pond would read @@ -555,11 +619,13 @@ def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=Non # in sun (a two-colour ramp, not a neutral multiply), so snow keeps relief form instead of # muddying to grey on rugged terrain the way SNOW_RGB*light did. alpha = np.where(ocean | water, 0.0, snow_a) - snow_t = snow_position(light, KNOBS["snow_curve"]) - snow_shadow = np.array(palette.SNOW_SHADOW_RGB, dtype=np.float32).reshape(3, 1, 1) - snow_lit = np.array(palette.SNOW_RGB, dtype=np.float32).reshape(3, 1, 1) - snow_rgb = snow_shadow + (snow_lit - snow_shadow) * snow_t[None] - final = base_rgb * (1.0 - alpha)[None] + snow_rgb * alpha[None] + snow_t = snow_position(light, KNOBS["snow_curve"]) # also the sea-ice light key below + if snow_paint is None: + final = base_rgb # no layer here paints a land white; see the docstring on why not zeros + else: + snow_shadow, snow_lit = paint_end(snow_paint[1]), paint_end(snow_paint[0]) + snow_rgb = snow_shadow + (snow_lit - snow_shadow) * snow_t[None] + final = base_rgb * (1.0 - alpha)[None] + snow_rgb * alpha[None] # soft-alpha sea ice: the sea-side mirror of the snow blend above. Gated on `ocean` (the mirror # of snow's ~(ocean|water) land gate) so ice paints ONLY over open sea -- never land, never the @@ -571,8 +637,7 @@ def composite(heights, ocean, water, snow_a, hs, occ, occ_shape, grid, depth=Non # Sea ice is a cooler/dimmer white than snow (palette.ICE_*), light-keyed by the same snow_t # so it still takes the hillshade on pressure ridges / shelf edges. Distinct from land snow # without a hard colour split -- the coastline and relief carry the rest. - ice_shadow = np.array(palette.ICE_SHADOW_RGB, dtype=np.float32).reshape(3, 1, 1) - ice_lit = np.array(palette.ICE_RGB, dtype=np.float32).reshape(3, 1, 1) + ice_shadow, ice_lit = paint_end(ice_paint[1]), paint_end(ice_paint[0]) gated_ice = np.where(ocean, np.asarray(ice_a, dtype=np.float32), 0.0) ice_light_key = snow_t if KNOBS["ice_relief_damp"] > 0.0: diff --git a/pipeline/tile/shade_planet.py b/pipeline/tile/shade_planet.py index fc5252a..bea2f66 100644 --- a/pipeline/tile/shade_planet.py +++ b/pipeline/tile/shade_planet.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Shade the whole (non-Antarctic) planet into ONE seamless Web-Mercator RGB raster. Supersedes the 194-strip `tile_planet.py`, whose seam-avoidance hacks (a single global @@ -19,13 +18,15 @@ 5. composite each full-width horizontal window (reusing tile/shade.py::composite) with the latitude-ramped snow (blue-white shadows) and RGI glaciers, and cap both polar edges (>84N, <-59.5S -> flat pale sea-ice) so MapLibre's globe shows clean polar discs; - 6. cut z0-8 512px tiles (no overview step -- `gdal raster tile` never reads them; see build_tiles). + 6. cut 512px tiles from z0 to THIS BODY's ceiling -- z8 for Earth, and the body says so rather + than this module (no overview step: `gdal raster tile` never reads them; see build_tiles). Every stage skips if its output is FRESH -- present, completed, and newer than everything it derives from (`is_stale`). An exists()-only guard cannot tell "built" from "still correct": the Caspian re-fuse rewrote 4 of the 540 chunks, and a plain re-run would have skipped every stage and silently re-cut tiles from the pre-Caspian, pre-sea-rework rasters. -Grid matches the existing tile pyramid exactly (131072 x 93009). +Grid matches the existing tile pyramid exactly (131072 x 131072 — square since Antarctica was +fused in; it was 131072 x 93009 while the pyramid stopped at -60). python -m pipeline.tile.shade_planet --body earth # shade only python -m pipeline.tile.shade_planet --body earth --tiles # + cut tiles @@ -34,7 +35,6 @@ import argparse import gc import json -import math import subprocess import sys import time @@ -50,17 +50,42 @@ from rasterio.enums import Resampling from rasterio.windows import Window -from pipeline import bodies, paths +from pipeline import bodies, layers, planet_seam, wrap_seam +from pipeline.freshness import ( + done_marker, + is_stale, + mark_done, + reference_needs_rebuild, + warp_needs_rebuild, + write_if_changed, +) from pipeline.raster_io import GTIFF_CREATE, band_window -from pipeline.render import cast_shadow, hillshade, lake_depth, palette, seaice, snow -from pipeline.render import sky_view +from pipeline.render import ( + cast_shadow, + hillshade, + lake_depth, + layer_producers, + palette, + sky_view, + snow, +) from pipeline.render.sky_view import normalised_occlusion, occlusion_shape from pipeline.tile import shade from pipeline.tile.shade import KNOBS -ROOT = paths.ROOT -Z8_RES = 305.7483 # metres/pixel of a 512px WebMercatorQuad tile at zoom 8 -EXAG = palette.EXAGGERATION +# The grid resolution used to live here as a module constant named for the one zoom Earth cuts to. +# It is `Body.map_units_per_pixel` now, because a planet with a different ceiling needs a different +# pixel and a module constant cannot have one — and because a constant with no field to be bridged +# to is exactly how this one survived the body parameterisation with every gate green. +# +# The value is deliberately NOT written out here. `tests/test_bodies.py` scans this file for it, and +# a comment quoting a deleted number re-creates the needle the scan exists to find. +# +# The vertical exaggeration left the same way and for a sharper reason: it is a LOOK decision, and +# two bodies whose relief is a different fraction of their radius cannot read right at one value. +# It is `Body.exaggeration`, threaded to the two places that shade — the hillshade here, and the +# caps, which used to import it from this module and therefore drew every planet at Earth's. +# Same rule as above: the number is not written out, because the same scan looks for its name. ALT, AZ = KNOBS["alt"], 315.0 WINDOW_ROWS = 256 # the snow-persistence banded-warp height (Phase A) AND composite_planet's # DEFAULT window. Must stay 256: the persistence raster is banded at this @@ -85,6 +110,18 @@ # now that Antarctica is fused into the pyramid: the flat fill covers only the last # smeared Mercator sliver past -84, not real Antarctica (which is shaded down to the -85.06 grid edge). # It was -59.5 while the pyramid stopped at -60 and the AEQD cap supplied everything south of it. +# +# ON A BODY THAT RENDERS CAPS THIS FILL IS DEAD PIXELS, AND THAT IS THE POINT OF IT. polarCaps.ts +# feathers with `smoothstep(FEATHER_LO 81, feather_hi)` where `feather_hi` IS CAP_NORTH, so the cap +# is fully opaque from exactly this latitude poleward and nothing under it is ever seen. The fill +# exists because the raster must hold SOMETHING between here and the 85.05 grid edge, and a smeared +# Mercator sliver is uglier than a flat plug in the one case the plug shows. +# +# It shows on a body with `renders_polar_caps = False`, where it becomes the whole pole — MapLibre +# extends the top tile row over the projection's hole as well, so a flat disc replaces the ice cap. +# NO REGISTERED BODY IS IN THAT STATE, which is why this stays one constant rather than a per-body +# field: Mars was the only capless body and its caps went on with its ramps. Pricing a colour per +# planet would be pricing pixels that are covered on every planet that exists. CAP_NORTH, CAP_SOUTH = 84.0, -84.0 CAP_RGB = (216, 226, 233) # pale sea-ice fill for the poles (web-mercator has no data past ~85 deg) INFLIGHT_BUFFER = 2 # windows read AHEAD of the workers (optimisation #5): the main thread @@ -97,93 +134,6 @@ def _run(cmd): subprocess.run([str(part) for part in cmd], check=True) -def done_marker(output: Path) -> Path: - """The completion stamp beside `output` (height_3857.tif -> height_3857.done).""" - return output.with_suffix(".done") - - -def mark_done(output: Path) -> None: - """Stamp `output` complete. Call ONLY after its stage has returned successfully.""" - done_marker(output).touch() - - -def newest_mtime(*inputs: Path) -> float: - """Newest mtime among `inputs`, recursing into directories. Missing paths score 0.0. - - Directories are walked rather than stat'ed because a VRT's own mtime does NOT move when - the chunks it points at are re-fused -- which is exactly how the Caspian re-fuse stayed - invisible to the old guard. The planet is 540 cells x 3 rasters, so this is ~1.6k stats. - """ - newest = 0.0 - for path in inputs: - if not path.exists(): - continue - if path.is_dir(): - for child in path.rglob("*"): - if child.is_file(): - newest = max(newest, child.stat().st_mtime) - else: - newest = max(newest, path.stat().st_mtime) - return newest - - -def is_stale(output: Path, *inputs: Path) -> bool: - """True if `output` must be rebuilt: never completed, or older than any of `inputs`. - - Freshness is read from the .done marker, never from `output` itself: GDAL creates its - target at the START of a run, so a crashed pass leaves a full-sized, freshly-stamped, - half-written raster that an mtime test on the raster would happily accept as current. - """ - if not output.exists() or not done_marker(output).exists(): - return True - return newest_mtime(*inputs) > done_marker(output).stat().st_mtime - - -def grid_matches(path: Path, width: int, height: int, bounds) -> bool: - """True if `path` exists on exactly the reference grid (`width` x `height`, same `bounds`). - - Every 3857 raster below `height_3857` is warped to height's grid (via -te/-ts), but each one's - freshness is gated on its own SOURCE, not on height. A re-fuse that GROWS the grid -- un-skipping - Antarctica takes the planet from 93009 to 131072 rows -- re-warps height while these sit falsely - fresh at the old dimensions, and the composite then reads window slices past their bottom (silent - corruption). A dimension/bounds comparison catches exactly that, and is deliberately NOT an mtime - dependency on height: that would re-warp all of them on a SAME-grid re-fuse (the Caspian - rewrote 4 chunks without moving the grid), which is 30+ min of needless work. - - Bounds are compared with a 1 m tolerance -- far below the 305 m pixel, so a real grid shift always - trips it, while the float noise of a -te repr round-trip never does. -> PLAN Antarctica precondition. - """ - if not path.exists(): - return False - with rasterio.open(path) as dataset: - return (dataset.width == width and dataset.height == height - and all(math.isclose(actual, expected, abs_tol=1.0) - for actual, expected in zip(tuple(dataset.bounds), tuple(bounds)))) - - -def warp_needs_rebuild(out: Path, grid, *inputs: Path) -> bool: - """Whether a 3857 warp target must be rebuilt: `is_stale` (a source moved) OR off `grid` - (a re-fuse resized the planet under it). `grid` is (width, height, bounds). - - Split out so the composed condition is testable on its own. The load-bearing case is the one - `is_stale` alone cannot see: a raster whose SOURCE is unchanged but whose grid shrank beneath it. - """ - return is_stale(out, *inputs) or not grid_matches(out, *grid) - - -def write_if_changed(path: Path, text: str) -> Path: - """Write `text` to `path` only when it differs, and return `path`. - - The only-when-different part is load-bearing, not an optimisation: it lets a generated - file stand in as a dependency for `is_stale`. Tunables like KNOBS and the ramp colours - live in source, whose mtime moves on any `git checkout` and would force a full planet - rebuild; materialised here, their mtime moves if and only if a VALUE actually changed. - """ - if not path.exists() or path.read_text() != text: - path.write_text(text) - return path - - # KNOBS entries consumed by the HILLSHADE stage rather than by composite(). Excluded from # composite_params below. # @@ -200,9 +150,14 @@ def write_if_changed(path: Path, text: str) -> Path: HILLSHADE_ONLY_KNOBS = frozenset({"fill_strength", "shadow_strength", "shadow_reach"}) -def hs_params() -> str: +def hs_params(body: bodies.Body) -> str: """The hillshade's tunables, recorded as hs_3857's dependency — composite_params' sibling. + Takes the body because the exaggeration is one of those tunables and belongs to the planet, not + to this module. Recording it was already right; sourcing it from a module constant was not, and + the two were indistinguishable while Earth was the only body. Earth's sidecar is unmoved — its + field holds the value the constant did — so this cannot restage the live pyramid. + Split out of build_hillshade so BOTH halves of the freshness contract are testable from the outside. The asymmetry was itself the hazard: composite_params had tests pinning what it must and must not record, and this side had none, while every freshness bug so @@ -218,7 +173,15 @@ def hs_params() -> str: Composite-stage knobs must NOT appear here: this raster cannot see them, so recording one restages an 11:48 hillshade that would produce identical bytes. """ - params: dict[str, Any] = {"exag": EXAG, "alt": ALT, "az": AZ} + params: dict[str, Any] = {"exag": body.exaggeration, "alt": ALT, "az": AZ} + # Recorded only when it is not the identity, the same rule the fill and the shadow follow below + # and for the same reason: on Earth the scale is exactly 1.0, so writing it would restage an + # 8:28 hillshade, a 53.8 min composite and a 3:44 cut to reproduce identical bytes. On any other + # body it is a genuine input to every slope in the raster, and an untracked one would leave a + # re-shaded planet reporting fresh. + ground_scale = bodies.ground_metres_per_mercator_unit(body) + if ground_scale != 1.0: + params["ground_scale"] = ground_scale if KNOBS["fill_strength"] != 0.0: params["fill"] = {"strength": KNOBS["fill_strength"], "alt": hillshade.FILL_ALTITUDE, "az": hillshade.FILL_AZIMUTH} @@ -232,7 +195,40 @@ def hs_params() -> str: return json.dumps(params, sort_keys=True, indent=2) -def composite_params(variants, window_rows=WINDOW_ROWS) -> str: +def _ramp_origin(kind: str, ramp: palette.Surface) -> dict[str, float]: + """`{kind}_origin_m`, recorded ONLY when the ramp does not start at the datum. + + The conditional-record idiom, for the fourth time in this module (`fill`, `shadow`, + `ground_scale`) and for the identical reason. Earth's two ramps both hinge on 0 m, so an + unconditional record would add a key to a 2,672-byte sidecar that has been stable across the + whole shipped pyramid — restaging a 46 GB planet, a 21:37 composite and a 4:19 cut to reproduce + byte-identical output, and reporting the LIVE pyramid stale on the way. + + Conditional is not the same as untracked, which is the trap this idiom always has to answer. + The origin cannot move WITHIN a body without changing this dict, because the only two states + are absent (0 m) and present (some other number) — and a body moving off the datum flips it + from absent to present, which is a change. What it deliberately cannot do is distinguish two + bodies, and it does not have to: sidecars are per-body files. + """ + return {} if ramp.origin_m == 0.0 else {f"{kind}_origin_m": ramp.origin_m} + + +def _when(evaluated: bool, values: dict[str, Any]) -> dict[str, Any]: + """`values` when this body's composite evaluates them, `{}` when it cannot reach them. + + A body records only what can move its own pixels, so one body's re-tune cannot restage + another's composite for output that could not have changed. + + Conditional is not untracked, and every caller has to earn that. A gate is correct only when + the values behind it are unreachable while it is false, AND the gate itself rides in the + recipe, so that opening it is a change: `layers_off` and `rasters_off` carry the layer and + raster gates, `knobs` carries the knob gates. + """ + return values if evaluated else {} + + +def composite_params(variants, body: bodies.Body, rasters: frozenset[str], + window_rows=WINDOW_ROWS) -> str: """The composite's tunables, recorded as planet_rgb's dependency. KNOBS and the palette colours never reach a file of their own, so without this a knob or @@ -251,7 +247,58 @@ def composite_params(variants, window_rows=WINDOW_ROWS) -> str: arrive here through `hs`, so repeating them would force composites that change nothing. """ knobs = {key: value for key, value in KNOBS.items() if key not in HILLSHADE_ONLY_KNOBS} - return json.dumps({"knobs": knobs, "water_rgb": palette.WATER_RGB, + # THE LAYERS THAT ARE OFF, never the ones that are on — the conditional-record idiom that `fill`, + # `shadow` and `ground_scale` already follow. Earth has every layer, so its list is empty and + # nothing is written: the live 46 GB composite stays fresh. A body missing one records it, and + # turning a layer off on a body that had it correctly restages, which file mtimes cannot do — + # `newest_mtime` scores an absent path 0.0, so an unbuilt raster is silently not a dependency. + # + # `layers.COMPOSITE_LAYERS`, not the whole vocabulary: the caps read a coastline and this stage + # does not, so enumerating every layer here would make a cap-only decision restage the planet. + # + # `rasters_off` is the same idiom one tier up, and it tracks the OTHER direction of the same + # trap. When a mask APPEARS, `warp_inputs` builds it and `composite_deps` sees a new mtime, so + # the composite restages on its own. When one goes AWAY, nothing moves at all: the old + # `ocean_3857.tif` is still sitting on disk from the last run, and the composite painted with it + # reads perfectly fresh. That is exactly the loop Phase 2 runs on Mars — a shoreline contour, + # then a different one, then none — so the transition that has no mtime behind it needs a record. + absent_layers = layers.layers_off(body, layers.COMPOSITE_LAYERS) + absent_rasters = planet_seam.rasters_off(rasters) + missing: dict[str, list[str]] = {} + if absent_layers: + missing["layers_off"] = absent_layers + if absent_rasters: + missing["rasters_off"] = absent_rasters + look = palette.look_for(body.name) + # The sea ramp is recorded only when the body draws one — the conditional-record idiom this + # function already uses for `layers_off` and `rasters_off`, and `hs_params` for `ground_scale`. + # A body with no sea has no sea values to track, and writing nulls would put entries in the + # recipe that no edit could ever move, which reads as tracked while tracking nothing. + sea_recipe: dict[str, Any] = ( + {} if look.sea is None + else {"sea_stops": look.sea.stops, "sea_min_m": look.sea.extreme_m, + **_ramp_origin("sea", look.sea)} + ) + declared = body.surface_layers + # `snow_position` keys BOTH the snow and the sea-ice whites, so its curve constants are live + # when either layer is painted. + keys_white = bool({layers.PERENNIAL_ICE.name, layers.SEA_ICE.name} & declared) + # WHAT EACH DECLARED LAYER'S OWN PRODUCER READS, asked of the producer rather than spelled out + # here — `LayerProducer.recipe` carries the argument. This function gates on whether a body + # paints a layer, which stays right for every body; what it cannot know is HOW that body grades + # it, and a second planet painting the same layer by different arithmetic is exactly where a + # gate holding one body's constants starts recording the wrong ones. Merging keeps the answer + # beside the code that computes it and keeps a body's name out of this stage entirely. + # + # Order-free by construction: `sort_keys` below normalises the output, so where a key enters + # this dict cannot move a byte of a live sidecar. `layers.LAYERS` all the same, for its stated + # contract rather than `declared`'s arbitrary set iteration order. + produced: dict[str, Any] = {} + for layer in layers.LAYERS: + if layer.in_composite and layer.name in declared: + produced.update(layer_producers.producer_for(body, layer).recipe()) + return json.dumps({**missing, **sea_recipe, **produced, + "knobs": knobs, "composite_window_rows": window_rows, # The occlusion resolution reached NO freshness record at all -- # it was a module constant (`SVF_LONG_EDGE`, now OCCLUSION_TARGET_M_PER_PX) @@ -265,24 +312,36 @@ def composite_params(variants, window_rows=WINDOW_ROWS) -> str: # whole purpose was to gate the gdaldem stages. With those gone, nothing # else would notice a ramp re-tune and planet_rgb would sit falsely fresh # -- the exact failure this function exists to prevent. - "land_stops": palette.LAND_STOPS, "sea_stops": palette.SEA_STOPS, - "land_max_m": palette.LAND_MAX_M, "sea_min_m": palette.SEA_MIN_M, + # Read off the BODY'S look, not the module globals. Those globals are + # Earth's, so every planet's recipe used to record Earth's ramp — and this + # dict is precisely what decides whether a look change restages. A Mars + # re-tune would have left the Mars composite reading fresh. + "land_stops": look.land.stops, + "land_max_m": look.land.extreme_m, + **_ramp_origin("land", look.land), "lut_step_m": palette.LUT_STEP_M, - "snow_rgb": palette.SNOW_RGB, - "snow_shadow_rgb": palette.SNOW_SHADOW_RGB, - "ice_rgb": palette.ICE_RGB, - "ice_shadow_rgb": palette.ICE_SHADOW_RGB, - # sea-ice alpha knobs run at composite time inside seaice.ice_alpha, so they - # ride here (not in composite_deps) -- the untracked-input trap that let snow's - # RAMP_* constants slip freshness; do not repeat it. - "ice_lo": seaice.ICE_LO, "ice_band": seaice.ICE_BAND, - "ice_max_alpha": seaice.ICE_MAX_ALPHA, - # The toned SH pack (seaice.SH_ICE_*) runs at composite time for southern - # windows, so it rides here too -- the same untracked-input trap the globals - # above avoid. A re-tune must restage the composite. - "sh_ice_lo": seaice.SH_ICE_LO, "sh_ice_max_alpha": seaice.SH_ICE_MAX_ALPHA, - "lake_stops": palette.LAKE_STOPS, - "lake_max_m": palette.LAKE_MAX_M, + # Inland water is filled flat with this colour, so it is live only where a + # watermask exists to select it. + **_when("watermask" in rasters, {"water_rgb": palette.WATER_RGB}), + **_when(layers.LAKE_DEPTH.name in declared, + {"lake_stops": palette.LAKE_STOPS, + "lake_max_m": palette.LAKE_MAX_M}), + # BOTH WHITE PAIRS MOVED INTO `produced` ABOVE, where the producer that + # paints with them declares them. This gate said "one white family for every + # body's perennial ice", which held while Earth was the only body painting + # any and fails twice over now: a second body arrived, and its two poles + # measure as different colours from each other. A layer gate can only hold + # one value per layer, so it had no way to be right for both. + # + # Earth's keys and values are unchanged — `produced` merges flat into this + # dict and `sort_keys` normalises it — so this move restages nothing. + # `shadow_tint` returns exactly 1.0 at warmth 0, bit-identical to not being + # called, so the tint vector is live only above it. + **_when(KNOBS["shadow_warmth"] != 0.0, + {"shadow_tint": list(shade.SHADOW_TINT)}), + # One branch of `snow_position`; the other curves never read them. + **_when(KNOBS["snow_curve"] == "knee" and keys_white, + {"knee_x": shade.KNEE_X, "knee_share": shade.KNEE_SHARE}), "cap": [CAP_NORTH, CAP_SOUTH, list(CAP_RGB)], "variants": {str(name): knobs for name, knobs in variants.items()}}, sort_keys=True, indent=2) @@ -295,38 +354,95 @@ def composite_deps(work, hs, params) -> tuple: ramps itself from elevation, so the height raster IS the colour input. The ramp constants ride in `params` (composite_params) rather than in ramp_*.txt, which no longer exists. - `snow_persistence_3857.tif` + `glacier_3857.tif` joined (optimisation #4): the - composite reads pre-warped snow slices per window instead of forking gdalwarp/gdal_rasterize in - the loop, so a re-warp (new NSIDC/RGI, or a re-fuse to a new grid) must restage it. `glacier` - may be absent (RGI not downloaded) -- `newest_mtime` scores a missing path 0.0, so listing it + THE BUILT LAYERS COME FROM `layers.WARPED_LAYERS`, so this list, the warp that fills it and the + per-window reads cannot disagree about the set or the order. Each is joined here because the + composite reads a pre-warped slice per window instead of forking gdalwarp/gdal_rasterize in the + loop (optimisation #4), so a re-warp -- new NSIDC/RGI, or a re-fuse to a new grid -- must restage + it. Any of them may be absent, and `newest_mtime` scores a missing path 0.0, so naming them all unconditionally is safe. The ramp TUNABLES (`RAMP_*`) run at composite time inside `snow_alpha`, - so they ride in `composite_params`, NOT here -- this pair tracks the warp SOURCES only. - - `seaice_3857.tif` joined, the sea-side twin of snow persistence: its warp SOURCE is - tracked here, its ICE_LO/ICE_BAND alpha knobs in `composite_params`. Optional -- a missing path - scores `newest_mtime` 0.0, so listing it unconditionally is safe when the source isn't built. + so they do not belong here either -- this tracks the warp SOURCES only. + + THEY RIDE IN `composite_params` INSTEAD, and the split is the point: a warp source moves an + mtime, a constant moves nothing at all. A look constant that reaches a pixel and reaches + neither record leaves a stale composite reading fresh forever, so a new one goes into the + recipe beside the layer that reads it — never into this list, which cannot see it. + + THAT SAFETY CUTS THE OTHER WAY AND IS WHY `composite_params` RECORDS THE ABSENT LAYERS. A path + that scores 0.0 is not merely harmless, it is INVISIBLE: switching a layer off leaves the old + composite — painted with that layer — looking perfectly fresh against a dependency list that can + no longer see it. The mtimes here track a layer that is ON; the recipe is what tracks one going + OFF. + + SO THIS LIST IS DELIBERATELY OVER-INCLUSIVE, AND ITS SIBLING `cap_render.cap_sources` IS + DELIBERATELY EXACT. That reads like an inconsistency and is not: the two feed different + predicates. `is_stale` merely takes the newest mtime, so naming an input this planet does not + have costs nothing. `cap_is_fresh` requires every source to EXIST, so naming one there pins the + cap to a file that will never appear and leaves it permanently stale. Unifying them would mean + making this one exact, which trades a harmless imprecision for the chance to under-track — and + under-tracking is the direction that is silent. `test_the_two_freshness_predicates_disagree_on_a + _missing_input` is the executable form of this paragraph; read it before changing either. """ return (work / "height_3857.tif", hs, work / "ocean_3857.tif", work / "water_3857.tif", - work / "lakedepth_3857.tif", work / "snow_persistence_3857.tif", - work / "glacier_3857.tif", work / "seaice_3857.tif", params) + *(layer.warped_in(work) for layer in layers.WARPED_LAYERS), params) + +#: What the finished pixels lose when a layer is skipped at the WARP stage, by layer name. +#: +#: A CONSEQUENCE IS PER (LAYER, STAGE), which is why this is here and not a column on `Layer`: +#: `cap_render` says something different about the same layers because it paints a different +#: picture. Each states what the reader will see rather than what was missing, so a partial build +#: can be read back off its own output. +#: +#: Keyed by name and looked up unconditionally, so a layer added to `WARPED_LAYERS` and forgotten +#: here raises on the next pass of any body rather than being quietly skipped forever. +WARP_CONSEQUENCE: dict[str, str] = { + layers.LAKE_DEPTH.name: "lakes stay flat; run pipeline.acquire.extract_globathy", + layers.PERENNIAL_ICE.name: "no ice painted; the composite reads None and skips it", + layers.GLACIERS.name: "persistence-only snow", + layers.SEA_ICE.name: "bathymetry bare at the poles", +} -def warp_inputs(work: Path, planet: Path): - """Warp height + ocean/water masks to the shared WMQ-aligned 3857 grid (skip if fresh). + +def warp_inputs(work: Path, planet: Path, body: bodies.Body, rasters: frozenset[str]): + """Warp height + whichever masks this planet HAS to the shared WMQ-aligned 3857 grid. Each warp depends on the chunk DIRECTORY, not just its VRT -- re-fusing a cell leaves the VRT untouched, so the directory walk is the only thing that sees the change. + + `rasters` is the planet stage's own declaration of what it emitted (`planet_seam`), and it is + passed in rather than read off the disk here for the reason the layer gates already follow: a + mask's presence and a body's answer are different questions, and the file system can only answer + the first. The two masks are gated SEPARATELY because the known next case needs it — a Mars that + gains a sea at a chosen contour has an ocean mask and still no inland water. """ chunks = planet / "chunks" height = work / "height_3857.tif" - if is_stale(height, planet / "planet_heightfield.vrt", chunks): + resolution = body.map_units_per_pixel + # NOT `is_stale` ALONE: this raster's inputs are a VRT and a chunk directory, and neither moves + # when the body's ceiling does. `reference_needs_rebuild` asks the raster its own pixel size. + if reference_needs_rebuild(height, resolution, planet / "planet_heightfield.vrt", chunks): print("warp height -> 3857 ...", flush=True) height.unlink(missing_ok=True) # gdalwarp UPDATES an existing target; it must be gone - _run(["gdalwarp", "-q", "-t_srs", "EPSG:3857", "-tr", Z8_RES, Z8_RES, "-tap", + _run(["gdalwarp", "-q", "-t_srs", "EPSG:3857", "-tr", resolution, resolution, "-tap", "-r", "bilinear", "-ot", "Float32", "-co", "TILED=YES", "-co", "COMPRESS=DEFLATE", "-co", "BIGTIFF=YES", "-co", "NUM_THREADS=ALL_CPUS", planet / "planet_heightfield.vrt", height]) mark_done(height) + # OUTSIDE THE FRESHNESS GATE ON PURPOSE, and this is the half that is easy to get wrong: the + # raster this has to reach was written by a warp that already ran, so gating the fill on a + # re-warp would leave every existing planet exactly as broken while reading as fixed. It is free + # to re-ask — a closed raster declares no nodata, so `close_wrap_seam` returns without touching + # a pixel — which is what lets it sit here rather than behind a condition. + # + # Height only, and deliberately not the mask warps below: those are class codes resampled with + # `near`, where the midpoint between two categories is not a category. + filled = wrap_seam.close_wrap_seam(height) + if filled: + # The bytes moved, so everything keyed on this marker has to rebuild — the hillshade reads + # the filled column as ground instead of a cliff, and the composite ramps it as terrain + # instead of clamping it to the darkest stop. Re-stamping IS that instruction. + print(f"wrap seam: filled {filled} px at the antimeridian -> height restaged", flush=True) + mark_done(height) with rasterio.open(height) as dataset: bounds = [repr(value) for value in dataset.bounds] size = [str(dataset.width), str(dataset.height)] @@ -337,7 +453,12 @@ def warp_inputs(work: Path, planet: Path): # The reference grid every raster below is warped onto. warp_needs_rebuild re-warps a target when # its source moved OR when this grid grew under it (the Antarctica re-fuse; see grid_matches). grid = (grid_width, grid_height, grid_bounds) - for name, src in (("ocean", "planet_oceanmask.vrt"), ("water", "planet_watermask.vrt")): + for name, raster in (("ocean", "oceanmask"), ("water", "watermask")): + if raster not in rasters: + print(f"{body.name}'s planet stage emitted no {raster} -> {name}_3857 skipped " + f"(the composite reads None and treats every pixel as land)", flush=True) + continue + src = f"planet_{raster}.vrt" out = work / f"{name}_3857.tif" if warp_needs_rebuild(out, grid, planet / src, chunks): print(f"warp {name} -> 3857 ...", flush=True) @@ -347,74 +468,50 @@ def warp_inputs(work: Path, planet: Path): "-co", "BIGTIFF=YES", planet / src, out]) mark_done(out) - # GLOBathy lake depth, warped ONCE here rather than per window: it is an 83k-source VRT, - # and a many-source VRT re-reads every source on each touch (the same reason the tiler - # materialises before cutting). Deliberately NOT in the loop above -- depth is continuous, - # so it needs bilinear/Float32, while `near`/Byte is right for the class codes and would - # quantise every lake to whole metres and hard-step its gradient. - # Its dependency is the VRT alone, unlike the chunk directory above: extract_globathy - # rebuilds the VRT whenever the raster set changes, so its mtime really does move. - depth_out = work / "lakedepth_3857.tif" - if not lake_depth.LAKE_VRT.exists(): - print(f"no {lake_depth.LAKE_VRT.name} -> lakes stay flat " - f"(run pipeline.acquire.extract_globathy)", flush=True) - elif warp_needs_rebuild(depth_out, grid, lake_depth.LAKE_VRT): - print("warp lake depth -> 3857 ...", flush=True) - depth_out.unlink(missing_ok=True) - _run(["gdalwarp", "-q", "-t_srs", "EPSG:3857", "-te", *bounds, "-ts", *size, - "-srcnodata", str(lake_depth.GLOBATHY_NODATA), "-dstnodata", "0", - "-r", "bilinear", "-ot", "Float32", "-co", "TILED=YES", - "-co", "COMPRESS=DEFLATE", "-co", "BIGTIFF=YES", - "-co", "NUM_THREADS=ALL_CPUS", lake_depth.LAKE_VRT, depth_out]) - mark_done(depth_out) - - # Snow persistence + RGI glaciers, warped ONCE here rather than per window (optimisation #4). - # The old composite loop forked gdalwarp + gdal_rasterize for every window (~728 subprocesses) - # into two fixed-path temps -- the shared paths that blocked threading the composite. Same - # precedent as lakedepth above. Persistence stores the RAW PACKED Float32 (snow.warp_persistence_ - # raster's docstring: the composite unpacks per window in float64, so a slice is bit-identical to - # the old per-window warp). Glacier is a 0/1 Byte mask; absent RGI leaves it unbuilt and the snow - # is persistence-only, exactly as before. - persistence_out = work / "snow_persistence_3857.tif" - if not snow.SP_NC.exists(): - print(f"no {snow.SP_NC.name} -> snow layer unavailable (composite would fail)", flush=True) - elif warp_needs_rebuild(persistence_out, grid, snow.SP_NC): - print("warp snow persistence -> 3857 (banded) ...", flush=True) - persistence_out.unlink(missing_ok=True) - # band_rows == the composite window height, aligned to it: each band is exactly the - # per-window warp it replaces, so the mosaic is byte-identical to the old per-window path. - # A single whole-grid warp would DECIMATE this coarse source (snow.warp_persistence_raster). - snow.warp_persistence_raster(grid_bounds, grid_width, grid_height, persistence_out, - band_rows=WINDOW_ROWS) - mark_done(persistence_out) - - glacier_out = work / "glacier_3857.tif" - if not snow.RGI_GPKG.exists(): - print(f"no {snow.RGI_GPKG.name} -> glaciers skipped (persistence-only snow)", flush=True) - elif warp_needs_rebuild(glacier_out, grid, snow.RGI_GPKG): - print("rasterize RGI glaciers -> 3857 ...", flush=True) - snow.rasterize_glaciers_raster(grid_bounds, grid_width, grid_height, glacier_out) - mark_done(glacier_out) - - # Sea-ice frequency climatology, warped ONCE here like snow persistence (same banded warp: a - # single whole-grid warp of the coarse 0.1deg source would decimate the ice edge). Optional, - # like glacier/depth -- an absent source just skips it and the composite paints no ice, leaving - # the bathymetry bare at the poles. - seaice_out = work / "seaice_3857.tif" - if not seaice.SEAICE_SRC.exists(): - print(f"no {seaice.SEAICE_SRC.name} -> sea ice skipped (bathymetry bare at the poles)", - flush=True) - elif warp_needs_rebuild(seaice_out, grid, seaice.SEAICE_SRC): - print("warp sea-ice frequency -> 3857 (banded) ...", flush=True) - seaice_out.unlink(missing_ok=True) - seaice.warp_seaice_raster(grid_bounds, grid_width, grid_height, seaice_out, - band_rows=WINDOW_ROWS) - mark_done(seaice_out) + # Every optional surface layer, warped ONCE here rather than per window (optimisation #4). The + # old composite loop forked gdalwarp + gdal_rasterize for every window (~728 subprocesses) into + # fixed-path temps -- the shared paths that blocked threading the composite. Deliberately NOT in + # the mask loop above: those are class codes wanting `near`/Byte, and every one of these is a + # continuous or vector source with its own resampling. + for layer in layers.WARPED_LAYERS: + consequence = WARP_CONSEQUENCE[layer.name] + out = layer.warped_in(work) + # THE BODY IS ASKED FIRST AND THE PRODUCER SECOND, and the order is what makes the disk + # question honest: `producer.sources()` are that body's files, where the constants they + # replaced were Earth's at a fixed global path on every planet alike. + if not layers.body_declares_layer(body, layer, consequence): + continue + producer = layer_producers.producer_for(body, layer) + sources = producer.sources() + if not all(layers.layer_is_buildable(body, layer, source, consequence) + for source in sources): + continue + # A BUILD-TIME CONSTANT IS MATERIALISED INTO A SOURCE, because `warp_needs_rebuild` is closed + # over PATHS and no Python value can reach it. A producer that grades before it writes has + # its tunables frozen into the file; recorded only in `composite_params`, changing one would + # restage the whole composite and then repaint from the unchanged raster — the same wrong + # pixels behind a restage that looks like it worked. `write_if_changed` moves an mtime if and + # only if a value moved, and is written BEFORE the question is asked, per its own docstring. + # + # Empty for every Earth producer, which writes no file and leaves this list exactly as it + # was — the reason adopting this restages nothing. + tunables = producer.build_recipe() + if tunables: + sources = (*sources, write_if_changed( + out.with_name(f"{out.stem}_build.json"), + json.dumps(tunables, indent=2, sort_keys=True) + "\n")) + # `warp_needs_rebuild` re-warps when a source moved OR when the grid grew under the target; + # the sources are the producer's because they are what it will actually read. + if warp_needs_rebuild(out, grid, *sources): + producer.build(layer_producers.LayerBuild( + bounds=grid_bounds, width=grid_width, height=grid_height, out=out, + band_rows=WINDOW_ROWS)) + mark_done(out) return height -def build_hillshade(work: Path, height: Path): - """The seamless per-row-z hillshade (skip if fresh). +def build_hillshade(work: Path, height: Path, body: bodies.Body): + """The seamless per-row-z hillshade (skip if fresh), at THIS body's vertical exaggeration. Was `color_and_hillshade`: the two `gdaldem color-relief` passes it also ran were deleted (28:19 and 24.4% of all pass CPU, single-threaded; profile said `libgdal` 19.37% @@ -423,42 +520,59 @@ def build_hillshade(work: Path, height: Path): over all 12.19 G px, 6/6 bands, zero pixels beyond 1 DN. """ hs = work / "hs_3857.tif" - hs_params_path = write_if_changed(work / "hs_params.json", hs_params()) + hs_params_path = write_if_changed(work / "hs_params.json", hs_params(body)) if is_stale(hs, height, hs_params_path): fill_note = (f", fill {KNOBS['fill_strength']:.2f}" if KNOBS["fill_strength"] else "") shadow_note = (f", shadow {KNOBS['shadow_strength']:.2f}" if KNOBS["shadow_strength"] else "") - print(f"per-row-z hillshade (EXAG={EXAG}{fill_note}{shadow_note}) ...", flush=True) - hillshade.per_row_zfactor_hillshade(height, hs, EXAG, ALT, AZ, - fill_strength=KNOBS["fill_strength"], - shadow_strength=KNOBS["shadow_strength"], - shadow_reach_px=int(KNOBS["shadow_reach"])) + print(f"per-row-z hillshade (exag={body.exaggeration}{fill_note}{shadow_note}) ...", + flush=True) + hillshade.per_row_zfactor_hillshade( + height, hs, body.exaggeration, ALT, AZ, + fill_strength=KNOBS["fill_strength"], + shadow_strength=KNOBS["shadow_strength"], + shadow_reach_px=int(KNOBS["shadow_reach"]), + ground_scale=bodies.ground_metres_per_mercator_unit(body)) mark_done(hs) return hs -def global_occlusion(height: Path): +def global_occlusion(height: Path, body: bodies.Body): """Sky-view occlusion (1 = valley, 0 = open) on a global downsample, normalised globally. Sized from `sky_view.OCCLUSION_TARGET_M_PER_PX` — the same constant the region path uses — so a region preview and the planet it predicts can no longer drift apart. `SVF_LONG_EDGE` was the old planet-only spelling of this and is now derived, not chosen. - KNOWN INCORRECT, deliberately unchanged here: `Z8_RES` is a MAP-unit scale, and - ground metres in Web Mercator are `Z8_RES * cos(lat)`. Using map units understates the horizon - run by `1/cos(lat)` — 1.22x at 35N, 2.00x at 60N, 3.86x at 75N — so high latitudes are - systematically under-occluded, and the global affine renormalisation provably cannot absorb a - latitude-varying error. Fixing it needs a per-ROW ground scale (the hillshade's z-factor trick) - and changes production pixels, so it rides with the resolution change rather than sneaking in - under a refactor. + `occlusion_shape` and `normalised_occlusion` both document that they want a GROUND scale, and + this function used to hand them a map-unit one. Two independent errors hid behind that, and only + one of them is fixed here. + + FIXED: THE BODY TERM. Every projection in this pipeline is Earth-sphered, so a map unit is a + ground metre only on Earth; elsewhere it is worth `ground_metres_per_mercator_unit(body)`. On + Mars the horizon run would be overstated by 1.878x, flattening the sky-view exactly where the + relief is most dramatic. The factor is EXACTLY 1.0 for Earth by construction of EPSG:3857, so + adopting it here moves no existing pixel. + + NOT FIXED: THE LATITUDE TERM, which is the older half and is Earth's. Ground metres in Web + Mercator are also `cos(lat)` times map units, so the run is understated by `1/cos(lat)` — + 1.22x at 35N, 2.00x at 60N, 3.86x at 75N — and high latitudes come out systematically + under-occluded. The global affine renormalisation provably cannot absorb a latitude-varying + error. It stays out because it is a different KIND of change: it needs a per-ROW scale (the + hillshade's z-factor trick) plus a decision about which latitude sizes the downsample, and it + moves Earth's production pixels — a re-shade and a re-cut, judged on the sphere rather than + accepted from a number. The region path already applies its own `cos(mid_lat)`, so the two + shading paths disagree on this term today; that is the shape of the outstanding work. """ + ground = bodies.ground_metres_per_mercator_unit(body) + ground_res = body.map_units_per_pixel * ground with rasterio.open(height) as dataset: full_w, full_h = dataset.width, dataset.height - small_h, small_w = occlusion_shape(full_w, full_h, Z8_RES) + small_h, small_w = occlusion_shape(full_w, full_h, ground_res) low = dataset.read(1, out_shape=(small_h, small_w), resampling=Resampling.average).astype(float) low = np.nan_to_num(np.where(low < -500, np.nan, low), nan=0.0) - m_per_px = Z8_RES * (full_w / small_w) + m_per_px = ground_res * (full_w / small_w) return normalised_occlusion(low, m_per_px) # shape (small_h, small_w) @@ -480,7 +594,13 @@ class _WindowInputs: read lives here and the pure-numpy compute (`_compute_shared`/`_compose`) takes this bundle. The fields are the untransformed slices (`ocean_raw`, not `ocean != 0`); the cheap numpy that derives masks/alpha from them runs on the worker, which is where optimisation #5 wants - the CPU. `depth_raw`/`glacier_raw`/`sea_ice_raw` are None when that optional input was never built. + the CPU. + + `ocean_raw`/`watercode` are None on a planet whose seam emitted no masks, and that is a + DIFFERENT kind of None from a `layer_raw` entry's: that one says a dataset was not downloaded, + this one says the planet has no sea. `_compute_shared` turns it into an all-False selector, + which is the true answer rather than a stand-in — no pixel is ocean, none is inland water, and + every pixel is land. """ win: Window @@ -488,14 +608,19 @@ class _WindowInputs: win_top: float win_bottom: float height_win: np.ndarray - ocean_raw: np.ndarray - watercode: np.ndarray + ocean_raw: "np.ndarray | None" + watercode: "np.ndarray | None" hs_raw: np.ndarray - depth_raw: np.ndarray | None - persistence_raw: np.ndarray - glacier_raw: np.ndarray | None - sea_ice_raw: np.ndarray | None + #: One slice per `layers.WARPED_LAYERS` row, keyed by layer name, None where that layer's + #: raster was never built. ONE DICT AND NOT A FIELD PER LAYER: the reads are gated identically, + #: so a field each is four chances for the fourth to be written without the guard — which is + #: exactly what happened to snow persistence before this was one expression. + layer_raw: dict[str, "np.ndarray | None"] occ_win: np.ndarray + #: Which planet this window belongs to, and therefore which producers `_compute_shared` asks + #: for its layers. Deliberately not carrying paths: this struct crosses onto worker threads, + #: and a stage that read the filesystem there would leave `_compute_shared` impure. + body: bodies.Body @dataclass @@ -512,9 +637,44 @@ class _WindowShared: depth_win: np.ndarray | None snow_a: np.ndarray ice_a: np.ndarray | None + #: The whites their producers declared for this window, or None where nothing paints one. + #: Variant-independent for the reason everything else here is: a sea knob cannot move them. + snow_paint: "tuple[Any, Any] | None" + ice_paint: "tuple[Any, Any] | None" cap: np.ndarray +def _merge_paint(current: "tuple[Any, Any] | None", current_alpha: np.ndarray, + incoming: "tuple[Any, Any] | None", + incoming_alpha: np.ndarray) -> "tuple[Any, Any] | None": + """Fold one layer's white into the union's: the layer that WINS a pixel's alpha paints it. + + The union is a `np.maximum` over several layers' alphas, so without this the first-registered + layer's colour would be painted over another layer's pixels — invisible on a body whose layers + agree and wrong on one whose layers do not. + + EQUAL WHITES SHORT-CIRCUIT, and that is the only path any shipping body takes today: Earth's + perennial ice and glaciers both declare `palette.SNOW_RGB`, and Mars declares one ice layer with + nothing to merge against. The general branch below is therefore reached by NO live data, so its + guard has to construct two disagreeing paints rather than borrow a pair — a test that took its + negative case from the registry would go quiet the moment a body changed, not red. + + The short circuit is also what keeps the cost at zero. Merging materialises `(3, H, W)` per end, + which on a planet window is ~400 MB apiece; agreeing whites never pay it. + """ + if incoming is None: + return current + if current is None: + return incoming + if all(np.array_equal(np.asarray(old), np.asarray(new)) + for old, new in zip(current, incoming, strict=True)): + return current + wins = (incoming_alpha > current_alpha)[None] + merged = tuple(np.where(wins, shade.paint_end(new), shade.paint_end(old)) + for old, new in zip(current, incoming, strict=True)) + return merged[0], merged[1] + + def _compute_shared(inputs: _WindowInputs) -> _WindowShared: """Derive the variant-independent per-window arrays from the raw reads (pure numpy). @@ -523,43 +683,60 @@ def _compute_shared(inputs: _WindowInputs) -> _WindowShared: Runs on a worker thread in the threaded path; numpy releases the GIL, which is the whole basis of optimisation #5. """ - ocean_win = inputs.ocean_raw != 0 + # ALL-FALSE, NOT A SYNTHESISED RASTER, and the difference is the whole of Gap A. A planet with + # no sea could have been given an all-zero ocean mask on disk, and nothing downstream would have + # needed a line changed — but that file is indistinguishable from one produced by measuring the + # planet's oceans and finding none, and it would be the only body fact in this project expressed + # as a fabricated dataset. Built here instead, from the seam's declaration, it is a computation + # with a stated premise. `shade.composite` needs no branch either way: its eight selectors are + # boolean, and all-False means land everywhere, which is the answer. + shape = inputs.height_win.shape + ocean_win = inputs.ocean_raw != 0 if inputs.ocean_raw is not None else np.zeros(shape, bool) watercode = inputs.watercode - water_win = lake_depth.inland_water(watercode) + water_win = (lake_depth.inland_water(watercode) if watercode is not None + else np.zeros(shape, bool)) hs_win = inputs.hs_raw.astype(float) - # Lake depth, zeroed off class 2 so rivers stay flat and the (class 1) Caspian keeps GEBCO's - # measured bathymetry instead of GLOBathy's cone. - depth_win = (lake_depth.lakes_only(inputs.depth_raw, watercode) - if inputs.depth_raw is not None else None) - # unpack_persistence runs the float64 unpack per window (as the old per-window path did), so - # snow_alpha sees bit-identical input. Glacier is optional (persistence-only when RGI absent). - persistence_win = snow.unpack_persistence(inputs.persistence_raw) - snow_a = snow.snow_alpha(persistence_win, inputs.win_top, inputs.win_bottom) - if inputs.glacier_raw is not None: - snow_a = np.maximum(snow_a, inputs.glacier_raw.astype(float)) - latitude = snow.latitude_per_row(inputs.win_top, inputs.win_bottom, inputs.win_h) - # Force Antarctic land white: NSIDC-0791 is NH-only and RGI region 19 is excluded, so snow_a is 0 - # over the continent and it would render on the tan LAND ramp. The same shared rule the south cap - # uses, so the two agree across the -84 seam (snow.antarctic_snow_mask). land_win = ~(ocean_win | water_win) - snow_a = np.maximum(snow_a, snow.antarctic_snow_mask(land_win, latitude)) - # Sea-ice alpha: frequency -> smoothstep, the sea-side twin of snow_a (no latitude ramp needed). - # Optional (None when the seaice source was never warped); shade.composite gates it on ocean. South - # of the equator the SH pack is toned to the cap's fainter, pulled-in fringe (seaice.SH_ICE_*), else - # the full-strength Antarctic belt reads as a bright halo -- proven on the cap. No window straddles - # both hemispheres' ice, and the equator is ice-free, so the per-row split is exact. - if inputs.sea_ice_raw is not None: - frequency = seaice.unpack_seaice(inputs.sea_ice_raw) - ice_a = seaice.ice_alpha(frequency) - southern = latitude < 0.0 - if southern.any(): - toned = seaice.ice_alpha(frequency, ice_lo=seaice.SH_ICE_LO, - ice_max_alpha=seaice.SH_ICE_MAX_ALPHA) - ice_a = np.where(southern[:, None], toned, ice_a) - else: - ice_a = None + latitude = snow.latitude_per_row(inputs.win_top, inputs.win_bottom, inputs.win_h) + # ASKED OF THE BODY, never of the raster on disk. A producer runs because the planet DECLARED + # the layer, which is what lets Earth's perennial ice carry the forced Antarctic patch — a + # latitude-and-land rule with no file behind it, so no missing raster could ever switch it off. + # Reading the declaration off `layer_raw` instead would drop it the day NSIDC went absent. + contributions: dict[str, np.ndarray] = {} + paints: dict[str, tuple[Any, Any]] = {} + for layer in layers.WARPED_LAYERS: + if layer.name not in inputs.body.surface_layers: + continue + producer = layer_producers.producer_for(inputs.body, layer) + window = layer_producers.LayerWindow( + raw=inputs.layer_raw[layer.name], watercode=watercode, land=land_win, + latitude=latitude, top=inputs.win_top, bottom=inputs.win_bottom) + value = producer.contribution(window) + if value is not None: + contributions[layer.name] = value + # Asked ONLY of a layer that contributed, so a producer that paints nothing this window + # never has to answer what colour it would have used. + paint = producer.paint(window) + if paint is not None: + paints[layer.name] = paint + depth_win = contributions.get(layers.LAKE_DEPTH.name) + # The white union, in the table's order. float64 base because that is what `snow_alpha` returns + # and what the maxima promote to; a float32 base would narrow every pixel `composite` blends. + # `np.maximum` reorders freely and every contribution is non-negative, so which layer lands + # first cannot move a bit. + snow_a = np.zeros(inputs.height_win.shape, dtype=float) + snow_paint = None + for layer in (layers.PERENNIAL_ICE, layers.GLACIERS): + contribution = contributions.get(layer.name) + if contribution is not None: + snow_paint = _merge_paint(snow_paint, snow_a, paints.get(layer.name), contribution) + snow_a = np.maximum(snow_a, contribution) + # The sea-side twin, kept OUT of that union on purpose: `shade.composite` gates it on the ocean + # selector where snow_a paints land, and None there means the layer is absent rather than zero. + ice_a = contributions.get(layers.SEA_ICE.name) cap = (latitude > CAP_NORTH) | (latitude < CAP_SOUTH) - return _WindowShared(ocean_win, water_win, hs_win, depth_win, snow_a, ice_a, cap) + return _WindowShared(ocean_win, water_win, hs_win, depth_win, snow_a, ice_a, + snow_paint, paints.get(layers.SEA_ICE.name), cap) def _compose(inputs: _WindowInputs, shared: _WindowShared) -> np.ndarray: @@ -572,7 +749,8 @@ def _compose(inputs: _WindowInputs, shared: _WindowShared) -> np.ndarray: rgb = shade.composite(inputs.height_win, shared.ocean_win, shared.water_win, shared.snow_a, shared.hs_win, inputs.occ_win, inputs.occ_win.shape, (inputs.win_h, inputs.height_win.shape[1]), depth=shared.depth_win, - ice_a=shared.ice_a) + ice_a=shared.ice_a, look=palette.look_for(inputs.body.name), + snow_paint=shared.snow_paint, ice_paint=shared.ice_paint) if shared.cap.any(): # force the smeared polar edges to a flat deep-sea disc for band in range(3): rgb[band][shared.cap] = CAP_RGB[band] @@ -585,7 +763,8 @@ def _compute_window_rgb(inputs: _WindowInputs) -> tuple[Window, np.ndarray]: return inputs.win, _compose(inputs, _compute_shared(inputs)) -def composite_planet(work: Path, hs, compute_occlusion: Callable[[], np.ndarray], variants=None, +def composite_planet(work: Path, hs, compute_occlusion: Callable[[], np.ndarray], + body: bodies.Body, rasters: frozenset[str], variants=None, window_rows=WINDOW_ROWS, max_windows=None, max_workers=1, row_start=0): """Composite the whole planet window-by-window into seamless RGB GeoTIFF(s). @@ -616,7 +795,8 @@ def composite_planet(work: Path, hs, compute_occlusion: Callable[[], np.ndarray] if variants is None: variants = {None: None} outs = {name: work / f"planet_rgb{f'_{name}' if name else ''}.tif" for name in variants} - params = write_if_changed(work / "composite_params.json", composite_params(variants, window_rows)) + params = write_if_changed(work / "composite_params.json", + composite_params(variants, body, rasters, window_rows)) deps = composite_deps(work, hs, params) # One shared params file is sound because every variant is composited in a single pass: # the guard rebuilds all of them or none. @@ -627,7 +807,7 @@ def composite_planet(work: Path, hs, compute_occlusion: Callable[[], np.ndarray] occ = compute_occlusion() with rasterio.open(work / "height_3857.tif") as h: width, height, transform = h.width, h.height, h.transform - small_h, small_w = occ.shape + small_h, _small_w = occ.shape # dict[str, Any]: GDAL creation options are a heterogeneous bag, and `**profile` otherwise # hands rasterio.open's bool-typed `sharing`/`thread_safe` an inferred `str | int`. profile: dict[str, Any] = dict( @@ -635,10 +815,7 @@ def composite_planet(work: Path, hs, compute_occlusion: Callable[[], np.ndarray] crs="EPSG:3857", transform=transform, photometric="RGB", BIGTIFF="YES", num_threads="ALL_CPUS", **GTIFF_CREATE) ocean_p, water_p = work / "ocean_3857.tif", work / "water_3857.tif" - depth_p = work / "lakedepth_3857.tif" - persistence_p = work / "snow_persistence_3857.tif" - glacier_p = work / "glacier_3857.tif" - seaice_p = work / "seaice_3857.tif" + layer_paths = {layer.name: layer.warped_in(work) for layer in layers.WARPED_LAYERS} def read_window(row0: int) -> _WindowInputs: """Gather one window's raw reads + geometry — MAIN thread only (GDAL is not thread-safe).""" @@ -646,20 +823,26 @@ def read_window(row0: int) -> _WindowInputs: win = band_window(width, row0, row1) # sky-view occlusion slice for this window (smooth -> nearest rows are fine) sr0 = int(row0 / height * small_h) - sr1 = max(sr0 + 1, int(round(row1 / height * small_h))) + sr1 = max(sr0 + 1, round(row1 / height * small_h)) return _WindowInputs( win=win, win_h=row1 - row0, win_top=transform.f + row0 * transform.e, win_bottom=transform.f + row1 * transform.e, height_win=read1_window(work / "height_3857.tif", win), - ocean_raw=read1_window(ocean_p, win), - watercode=read1_window(water_p, win), + # Gated on the SEAM'S DECLARATION, never on `ocean_p.exists()`. A declared mask that is + # missing from disk must crash here — the planet said it has one — where an existence + # check would quietly composite a sea-less Earth after a half-finished warp. + ocean_raw=read1_window(ocean_p, win) if "oceanmask" in rasters else None, + watercode=read1_window(water_p, win) if "watermask" in rasters else None, hs_raw=read1_window(hs, win), - depth_raw=read1_window(depth_p, win) if depth_p.exists() else None, - persistence_raw=read1_window(persistence_p, win), - glacier_raw=read1_window(glacier_p, win) if glacier_p.exists() else None, - sea_ice_raw=read1_window(seaice_p, win) if seaice_p.exists() else None, - occ_win=occ[sr0:sr1]) + # ONE EXPRESSION FOR EVERY BUILT LAYER, which is what retires a bug rather than + # guarding it: written a field each, three of the four had this check and snow + # persistence did not, so a body whose ice layer was off crashed on a raster nothing + # had built. Four separate reads is four chances to write the fourth without it. + layer_raw={name: read1_window(path, win) if path.exists() else None + for name, path in layer_paths.items()}, + occ_win=occ[sr0:sr1], + body=body) rows = list(range(row_start, height, window_rows)) if max_windows is not None: # smoke test: only the first N windows @@ -743,21 +926,39 @@ class TileCut(TypedDict): # zooms: q95 is 20.0% of PNG byte-weighted, and z8 -- three quarters of the pyramid -- is the # cheapest at 14.8%, so the aggregate is conservative. The archive goes ~16 GB -> ~3.2 GB and the # Worker's single R2 read per cold tile drops with it (~380 ms -> ~80 ms, it is bandwidth-bound). -TILE_CUT = TileCut(format="WEBP", quality=95, tile_size=512, min_zoom=0, max_zoom=8, +def tile_cut(body: bodies.Body) -> TileCut: + """This body's cut settings — eight encoder facts that are the same everywhere, and its ceiling. + + A FUNCTION RATHER THAN A CONSTANT because exactly one of these keys belongs to the planet. + `max_zoom` was a literal 8 here, which is Earth's ceiling and nobody else's, and it is the + second of the two constants that survived the body parameterisation by having no field to be + bridged to. The other seven are properties of the encoder and the tile scheme, so they stay + written once here rather than being copied onto every body — a body answers for what differs + about it, not for what does not. + + Earth's result is the same dict the constant held, so `tile_params` still serialises the exact + bytes beside the live pyramid and the cut does not restage. + """ + return TileCut(format="WEBP", quality=95, tile_size=512, min_zoom=0, + max_zoom=body.tile_max_zoom, resampling="cubic", overview_resampling="cubic", convention="xyz", skip_blank=True) -def tile_params() -> str: +def tile_params(body: bodies.Body) -> str: """The tile cut's own settings, recorded as the live pyramid's dependency — hs_params' sibling. This stage used to key freshness off `planet_rgb` ALONE, which meant the cut was the one stage that could not see its own recipe: changing the output format left `tiles_are_fresh` true, so the PNG->WebP switch would have silently shipped the old pyramid. Everything in - TILE_CUT alters the emitted bytes, and nothing outside it does — the input raster and the output + the cut alters the emitted bytes, and nothing outside it does — the input raster and the output directory are `is_stale`'s own arguments, not settings. + + The BODY is not recorded here and must not be: each body writes this file into its own work + tree, so the recipe is already body-specific by location, and adding the name would restage + Earth's entire pyramid the day a second planet existed for no pixel change at all. """ - return json.dumps(TILE_CUT, sort_keys=True, indent=2) + return json.dumps(tile_cut(body), sort_keys=True, indent=2) def tile_params_path(out: Path) -> Path: @@ -765,10 +966,10 @@ def tile_params_path(out: Path) -> Path: return out / "tile_params.json" -def _tile_cmd(planet_tif: Path, staging: Path) -> list[str]: - """The `gdal raster tile` invocation that cuts z0-8 512px tiles into `staging`. +def _tile_cmd(planet_tif: Path, staging: Path, body: bodies.Body) -> list[str]: + """The `gdal raster tile` invocation that cuts this body's 512px tiles into `staging`. - Built FROM `TILE_CUT` rather than from literals, so the command and the freshness record cannot + Built FROM `tile_cut` rather than from literals, so the command and the freshness record cannot disagree about what was cut — the same one-fact-one-spelling rule pack_pmtiles now follows for the tile encoding. @@ -784,14 +985,15 @@ def _tile_cmd(planet_tif: Path, staging: Path) -> list[str]: so a truncated tile from a mid-write kill would survive a resume. build_tiles instead removes any partial staging dir and cuts clean every time -- see its docstring. """ + cut = tile_cut(body) cmd = ["gdal", "raster", "tile", - f"--min-zoom={TILE_CUT['min_zoom']}", f"--max-zoom={TILE_CUT['max_zoom']}", - f"--tile-size={TILE_CUT['tile_size']}", - f"--resampling={TILE_CUT['resampling']}", - f"--overview-resampling={TILE_CUT['overview_resampling']}", - f"--convention={TILE_CUT['convention']}", - f"--format={TILE_CUT['format']}", "--co", f"QUALITY={TILE_CUT['quality']}"] - if TILE_CUT["skip_blank"]: + f"--min-zoom={cut['min_zoom']}", f"--max-zoom={cut['max_zoom']}", + f"--tile-size={cut['tile_size']}", + f"--resampling={cut['resampling']}", + f"--overview-resampling={cut['overview_resampling']}", + f"--convention={cut['convention']}", + f"--format={cut['format']}", "--co", f"QUALITY={cut['quality']}"] + if cut["skip_blank"]: cmd.append("--skip-blank") return [*cmd, "--webviewer=none", str(planet_tif), str(staging)] @@ -815,8 +1017,8 @@ def tiles_are_fresh(planet_tif: Path, out: Path) -> bool: and not is_stale(live, done_marker(planet_tif), tile_params_path(out))) -def build_tiles(planet_tif: Path, out: Path): - """Cut z0-8 512px tiles into a staging dir, then swap over the live tiles. +def build_tiles(planet_tif: Path, out: Path, body: bodies.Body): + """Cut this body's 512px tiles into a staging dir, then swap over the live tiles. Fresh-guarded like every other stage (`tiles_are_fresh`): a re-run whose `planet_rgb` AND `tile_params.json` are unchanged skips the ~4:19 cut entirely. This used to be the one @@ -824,9 +1026,7 @@ def build_tiles(planet_tif: Path, out: Path): empty and the cut re-ran in full every time. The completion stamp is `tiles.done`, touched only after the swap. - The recipe is written BEFORE the freshness question is asked, so changing TILE_CUT is what - triggers its own re-cut; `write_if_changed` means an unchanged recipe never moves an mtime and - never restages a pyramid that is still correct. + Recipe-gated in the usual order — see `freshness.write_if_changed`. EVERY CUT IS A CLEAN FULL CUT: the staging dir is removed first and `--resume` is not passed (see `_tile_cmd`). GDAL writes each tile in place, so a worker killed mid-write leaves a @@ -841,15 +1041,17 @@ def build_tiles(planet_tif: Path, out: Path): note that justified them credited a confounded fix: materialising the 194-source VRT to a GTiff was the real speed-up; the overviews rode along on the same commit untested. """ - write_if_changed(tile_params_path(out), tile_params()) + cut = tile_cut(body) + write_if_changed(tile_params_path(out), tile_params(body)) if tiles_are_fresh(planet_tif, out): print("tiles fresh -> skip cut", flush=True) return staging = out / "tiles_new" if staging.exists(): _run(["rm", "-rf", str(staging)]) # a partial from a prior mid-cut crash: never resume over it - print(f"cutting z0-8 512px tiles -> {staging} ...", flush=True) - _run(_tile_cmd(planet_tif, staging)) + print(f"cutting z{cut['min_zoom']}-{cut['max_zoom']} {cut['tile_size']}px tiles " + f"-> {staging} ...", flush=True) + _run(_tile_cmd(planet_tif, staging, body)) live = out / "tiles" if live.exists(): old = out / "tiles_old" @@ -869,10 +1071,13 @@ def build_parser() -> argparse.ArgumentParser: # pyramid, and the cost of discovering that late is a planet. Naming it costs one word. ap.add_argument("--body", required=True, help=f"which planet this pass is for ({', '.join(sorted(bodies.BODIES))})") - # Optional override. Left unset it follows the body, which also honours the MAPS_DATA seam that - # the old `ROOT / "data/..."` default bypassed; set, it is how a look A/B is pointed elsewhere. + # Optional override. Left unset it follows the body, which also honours the MAPS_DATA seam its + # checkout-rooted default used to bypass; set, it is how a look A/B is pointed elsewhere. The + # old spelling is described rather than quoted — `tests/test_paths.py` scans for it, and a + # comment reproducing it re-creates the needle the scan exists to find. ap.add_argument("--out", type=Path, default=None) - ap.add_argument("--tiles", action="store_true", help="also cut z0-8 tiles from the mosaic") + ap.add_argument("--tiles", action="store_true", + help="also cut tiles from the mosaic, z0 to the body's own ceiling") ap.add_argument("--knob", action="append", default=[], metavar="KEY=VALUE", help="override a locked KNOBS entry (repeatable), as tile/shade.py does. " "Look changes used to be made by EDITING the constant, which meant an " @@ -888,6 +1093,17 @@ def resolve_body(args: argparse.Namespace) -> bodies.Body: return bodies.get(args.body) +def runs_cap_pass(body: bodies.Body) -> bool: + """Whether a shade pass for this body ends by rendering polar caps. + + A named predicate rather than the field read inline, so the DECISION is testable without + spawning a subprocess. Inline, the only way to prove the pass respects the registry is to run it + and watch what it shells out to — which needs a composited planet on disk, so in practice it + would be proven by nothing. + """ + return body.renders_polar_caps + + def cap_pass_command(body: bodies.Body) -> list[str]: """The command that renders this body's polar caps at the tail of a shade pass. @@ -919,15 +1135,20 @@ def main(): work = resolve_out(args) work.mkdir(parents=True, exist_ok=True) - height = warp_inputs(work, bodies.work_dir(body, "planet")) - hs = build_hillshade(work, height) + # Read ONCE, at the top, and threaded down. The planet stage declares what it emitted and this + # raises if it never finished, so a half-built planet stops here rather than being shaded into a + # plausible-looking pyramid. Threading it (rather than each stage reading the file) keeps + # `_compute_shared` a pure function of its arguments, which is what lets it run on workers. + rasters = planet_seam.declared(body) + height = warp_inputs(work, planet_seam.planet_dir(body), body, rasters) + hs = build_hillshade(work, height, body) # Passed unevaluated: composite_planet runs it only if the composite is actually stale. # Production composite is threaded at COMPOSITE_ROWS/N_WORKERS (optimisation #5); the snow # persistence stays banded at WINDOW_ROWS (256), sliced 128 rows at a time. - planet_tif = composite_planet(work, hs, lambda: global_occlusion(height), + planet_tif = composite_planet(work, hs, lambda: global_occlusion(height, body), body, rasters, window_rows=COMPOSITE_ROWS, max_workers=N_WORKERS)[None] if args.tiles: - build_tiles(planet_tif, work) + build_tiles(planet_tif, work, body) # The polar caps are shade-stage outputs too: they run the same composite over the same # sources, so a look change that restages planet_rgb must restage them. Both caps once sat # stale against the tiles they feather into (the north −6.7 DN adrift) because nothing @@ -935,8 +1156,16 @@ def main(): # itself (cap_is_fresh), so a fresh pass pays only the ~2 s import here. Subprocess, not # import: cap_render imports FROM this module, and the caps' pyproj/scipy stack stays out # of the tile pass. - print("polar caps ...", flush=True) - subprocess.run(cap_pass_command(body), check=True) + if runs_cap_pass(body): + print("polar caps ...", flush=True) + subprocess.run(cap_pass_command(body), check=True) + else: + # SAID OUT LOUD, because the alternative is a pass that silently does less than the last one + # did. The cap pass would otherwise run and SUCCEED here — it needs only the heightfield once + # a body declares no surface layers — spending ~14 GB per pole to publish discs shaded by + # ramps this body has not ratified. + print(f"polar caps: {body.name} publishes none — skipped " + f"(the globe carries a hole above the Mercator limit)", flush=True) print("DONE", flush=True) diff --git a/pipeline/tile/terrain_rgb.py b/pipeline/tile/terrain_rgb.py index f43c878..c43a39e 100644 --- a/pipeline/tile/terrain_rgb.py +++ b/pipeline/tile/terrain_rgb.py @@ -2,8 +2,7 @@ Sibling of `shade_planet.py`, and deliberately NOT part of it: that module cuts *colour*, this one cuts *elevation*. They share one input (`height_3857.tif`), one tiling scheme, and the -stage-freshness primitives imported below — `cap_render.py` already treats `shade_planet` as -their one home, so a third spelling of `is_stale` would be the thing to avoid, not the import. +stage-freshness primitives, which live in `pipeline/freshness.py` and belong to neither stage. Nothing about the LOOK crosses over. MapLibre consumes this as a second `raster-dem` source with its own `maxzoom`, so the two pyramids need not be the same depth. @@ -12,7 +11,7 @@ Elevation packed into RGB is not an image. The value is `R*256 + G - 32768`, so the green byte WRAPS every 256 metres — and interpolating across a wrap invents a 256 m cliff. That makes every smooth resampler wrong on this data: `average`, `cubic`, `bilinear`, `lanczos` all mix bytes. -`shade_planet.TILE_CUT` uses `cubic` for both the cut and its overviews, which is correct for +`shade_planet.tile_cut` uses `cubic` for both the cut and its overviews, which is correct for colour and catastrophic here. So the pyramid is built **per zoom, from elevation downsampled in elevation space**, and each @@ -32,21 +31,20 @@ --------- Two guards, because this stage has two kinds of output and they fail differently. -The elevation chain is stamped with `.done` markers rather than tested with `exists()`. That is -not tidiness: rasterio creates its target at the START of a write, so the BigTIFF crash of -2026-07-28 left a full-sized, freshly-stamped, half-written `elev_z8.tif` on disk — and an -existence test would have built the entire pyramid on top of it, silently, since a truncated -float32 raster reads as a very flat planet rather than as an error. +The elevation chain is stamped with `.done` markers rather than tested with `exists()`, per +`freshness.is_stale`. This stage is where that rule was paid for: a BigTIFF crash left a +full-sized, freshly-stamped, half-written `elev_z8.tif` on disk, and an existence test would have +built the entire pyramid on top of it, silently, since a truncated float32 raster reads as a very +flat planet rather than as an error. The pyramid itself is cut into a staging dir and swapped, and keyed on the master's marker plus `terrain_params.json`. The recipe is the load-bearing half: the variant DIRECTORY name carries -only sea treatment, step and feather, so without a sidecar a `--format` or `--max-zoom` change is -invisible to every guard and to anyone reading the store. +only sea treatment and step, so without a sidecar a `--format` or `--max-zoom` change is invisible +to every guard and to anyone reading the store. """ import argparse import json -import math import shutil import subprocess from pathlib import Path @@ -54,14 +52,15 @@ import numpy as np import rasterio -from pipeline import paths +from pipeline import bodies +from pipeline.freshness import ( + done_marker, + is_stale, + mark_done, + write_if_changed, +) from pipeline.raster_io import GTIFF_CREATE, band_window, row_bands -from pipeline.tile.shade_planet import done_marker, is_stale, mark_done, write_if_changed -ROOT = paths.ROOT - -#: Native grid of `height_3857.tif` — 512 px x 2^8, i.e. the colour pyramid's z8. -MASTER_ZOOM = 8 TILE_SIZE = 512 #: Terrarium's zero point. Elevation `e` at quantisation `step` stores as `(e + 32768) / step`. @@ -72,13 +71,21 @@ #: caps carry their own elevation texture and MUST encode identically, or cap and tiles displace to #: different heights across the alpha crossfade and ghost against each other. Two spellings of one #: number is the copy-drift that bit the hero/tile colour constants four times. +#: +#: AND AGREEING HERE IS NOT ENOUGH ON ITS OWN — an agreement about BYTES is not one about METRES. +#: These two producers shared every constant on this line and still displaced to heights kilometres +#: apart, because only one of them multiplied its metres afterwards. `encode_array` is a pure +#: function of elevation for that reason; its docstring holds the argument. #: 8 m is the ratified knee (0.49x the archive; the error lands in mountains, not on plains) and #: bathymetry is ratified over sea-clamping, so the sea displaces rather than reading as a wall. #: -#: CAUTION, and it is not hypothetical: `--sea` still DEFAULTS to "clamp" while the shipped -#: `terrain_params.json` records `sea_clamp: false`, so the live archive was cut with an explicit -#: `--sea bathy` and the bare command would rebuild a different pyramid. SHIPPED_SEA_CLAMP names -#: what is on the wire, which is what the caps must match — not what argparse hands you. +#: SHIPPED_SEA_CLAMP NAMES WHAT IS ON THE WIRE, and `--sea` now derives its default from it rather +#: than restating one. It restated the opposite for a while, so the bare command rebuilt a pyramid +#: the live one had not been cut with — the trap this constant existed to warn about, closed by +#: making the two the same fact. Do not spell the default out again. +#: +#: The clamp stops being a look question entirely on a body with no sea: every point below zero +#: there is real ground, and flattening it deletes the deepest basin on the planet. QUANTISATION_M = 8.0 SHIPPED_SEA_CLAMP = False @@ -92,18 +99,6 @@ "webp": ("WEBP", ["LOSSLESS=YES"]), } -#: Latitude band over which encoded elevation ramps to zero, so the tiles flatten into the polar -#: caps. The caps are a CUSTOM layer and MapLibre does not drape custom layers onto the terrain -#: mesh (`LAYERS_TO_TEXTURES` in render_to_texture.ts), so displaced tiles under an undisplaced -#: cap would open a geometric seam — worst in the south, where this band is 2-3 km of Antarctic -#: ice. `polarCaps.ts` feathers its alpha over the same latitudes; this is the geometric twin. -FEATHER_LAT_LO = 78.0 -FEATHER_LAT_HI = 85.0 - -#: Web Mercator's sphere radius — the same constant the projection itself is defined on. -MERCATOR_RADIUS = 6378137.0 - - def _run(cmd) -> None: subprocess.run([str(part) for part in cmd], check=True) @@ -113,27 +108,67 @@ def grid_size(zoom: int) -> int: return TILE_SIZE * 2**zoom -def row_latitudes(row0: int, row1: int, height: int, north: float, south: float) -> np.ndarray: - """Latitude (degrees) at the centre of each raster row in [row0, row1). +def master_zoom_for(body: bodies.Body) -> int: + """Native grid zoom of this body's `height_3857.tif` — 512 px x 2^n — and so the deepest + terrain zoom there is to cut. - Rows are inverse-Mercator projected rather than linearly interpolated: latitude is not linear - in y, and the whole point of the feather is that it lands on the right parallels. + DERIVED RATHER THAN STORED, because the chain that makes deriving safe already has two owners + and a third copy of the number is exactly the drift they exist to prevent. `shade_planet` warps + the master at `body.map_units_per_pixel` and rebuilds it whenever the raster's own pixel size + disagrees, so the file matches the field; `test_bodies` then pins that field against + `tile_max_zoom` relationally, at a tolerance that admits Earth's rounded value and cannot admit + a factor of two. + + IT USED TO BE A MODULE CONSTANT HOLDING EARTH'S CEILING, and the reason that was dangerous + rather than merely wrong is that `build` does ARITHMETIC with it: the master is halved by + `2 ** (master_zoom - max_zoom)`. Against a body whose master is one level shallower, Earth's + number asks for a factor of two where the answer is one — which raises nothing, reads as a + normal run, and emits a pyramid at half the resolution its own recipe claims. """ - rows = np.arange(row0, row1, dtype=np.float64) + 0.5 - y = north - rows * (north - south) / height - return np.degrees(np.arctan(np.sinh(y / MERCATOR_RADIUS))) + return body.tile_max_zoom + +def out_under_body(body: bodies.Body, out: Path) -> Path: + """`out` resolved, having checked it lands in THIS body's terrain stage. Raises if it does not. -def feather_factor(latitudes: np.ndarray) -> np.ndarray: - """1.0 equatorward of FEATHER_LAT_LO, 0.0 poleward of FEATHER_LAT_HI, smoothstep between. + The variant directory below the stage is operator-named, so the destination cannot be derived — + but it can be bounded, and the bound is the failure worth catching. Pointing a Mars cut at + Earth's tree writes Martian elevation into the directory the packer reads for Earth, and the + tiles look exactly like tiles. - Smoothstep rather than linear because this multiplies GEOMETRY: a linear ramp leaves a slope - discontinuity at each end of the band, and a crease in a displacement mesh is visible in a way - a crease in an alpha ramp is not. + Earth's empty `path_prefix` is why the bound is the STAGE directory rather than the body's + root: `data/work` contains every planet's tree, so a containment test one level up would pass + for both bodies and guard nothing. `data/work/planet_terrain` and `data/work/mars/planet_terrain` + contain neither the other. """ - span = np.clip( - (FEATHER_LAT_HI - np.abs(latitudes)) / (FEATHER_LAT_HI - FEATHER_LAT_LO), 0.0, 1.0) - return span * span * (3.0 - 2.0 * span) + stage = bodies.work_dir(body, "planet_terrain").resolve() + resolved = out.resolve() + if resolved != stage and stage not in resolved.parents: + raise SystemExit(f"--out {out} is not under {body.name}'s terrain stage ({stage}) — a cut " + f"written outside it lands in another planet's tree, or somewhere the " + f"packer will never look") + return resolved + + +def master_grid_mismatch(master: Path, master_zoom: int) -> str | None: + """Why `master` is not the 512 x 2^`master_zoom` grid it was declared to be, or None. + + The declaration comes from the body and this is where it meets the artifact, which is the one + check that can catch the failure above. Everything downstream of a wrong `master_zoom` succeeds: + the downsample runs, every zoom encodes, every tile writes, and the recipe beside them records + the ceiling that was asked for rather than the one that was cut. + + A STRING RATHER THAN AN ASSERT so the caller decides what to do with it, and so a test can read + the sentence instead of matching an exception's text. + """ + with rasterio.open(master) as raster: + width, height = raster.width, raster.height + expected = grid_size(master_zoom) + if width == expected and height == expected: + return None + return (f"{master} is {width}x{height}, not the {expected}x{expected} grid of a " + f"{TILE_SIZE}px tile at z{master_zoom} — the descent would resample by the wrong " + f"factor and emit a pyramid at the wrong resolution without failing") #: Source rows held in memory at once by `downsample_elevation`. Budgeted on the SOURCE side, not @@ -173,20 +208,30 @@ def downsample_elevation(src: Path, dst: Path, factor: int, band_rows: int | Non 1, window=band_window(out_width, out0, out1)) -def encode_array(elevation: np.ndarray, step: float, sea_clamp: bool, - latitudes: np.ndarray | None = None) -> np.ndarray: +def encode_array(elevation: np.ndarray, step: float, sea_clamp: bool) -> np.ndarray: """Pack metres into the (3, h, w) uint8 terrarium-with-zero-blue form described in the header. `sea_clamp` raises everything below zero to zero — land rises out of a smooth sphere, which is what a physical relief globe does, and what stops a continental shelf reading as a cliff wall. It is also most of the archive: an abyssal tile measured 162 KiB carrying real bathymetry and 1.5 KiB flat. + + NOTHING HERE MAY DEPEND ON WHERE A PIXEL IS, and that is the anti-redo guard rather than a + style note. `cap_render.write_cap_elevation` calls this same function for the polar caps, whose + grid is azimuthal-equidistant and has no rows to speak of, and the two surfaces are drawn + across each other through `polarCaps.ts`'s alpha crossfade — so any per-row or per-latitude term + added here separates two surfaces the viewer sees at once. + + A latitude ramp did live here, to flatten the tiles toward datum from 78 to 85 degrees. It was + written when the cap could not displace at all, and the seam it closed reopened the moment + `polarCaps.vertexSrc` started lifting cap vertices: the tiles then flattened away from a cap + holding true elevation, and the 84-85 degree plug rim stood proud of it by the full local + relief. The fix was deleting the ramp, not tuning its band, so the shape to refuse is any + argument of the form "the poles need special treatment in the encode". """ metres = np.nan_to_num(elevation.astype(np.float64), nan=0.0) if sea_clamp: metres = np.maximum(metres, 0.0) - if latitudes is not None: - metres = metres * feather_factor(latitudes)[:, None] packed = np.clip(np.round((metres + BASE_SHIFT) / step), 0, 65535).astype(np.uint16) return np.stack([(packed >> 8).astype(np.uint8), (packed & 0xFF).astype(np.uint8), np.zeros(packed.shape, np.uint8)]) @@ -203,19 +248,20 @@ def decode_array(encoded: np.ndarray, step: float) -> np.ndarray: def encode_raster(elev_tif: Path, dst: Path, step: float, sea_clamp: bool, - feather: bool = True, band_rows: int = 512) -> None: - """Encode a whole elevation raster to a 3-band Byte GTiff, streaming by row band.""" + band_rows: int = 512) -> None: + """Encode a whole elevation raster to a 3-band Byte GTiff, streaming by row band. + + The band split is a MEMORY decision and must stay one: every band goes through `encode_array` + with nothing said about where it sits, so the seams between bands cannot carry a value. + """ with rasterio.open(elev_tif) as source: - north, south = source.bounds.top, source.bounds.bottom # Same 4 GB ceiling as the elevation sink above: three uint8 bands at z8 is 51.5 GB raw. profile = source.profile | GTIFF_CREATE | { "dtype": "uint8", "count": 3, "nodata": None, "bigtiff": "IF_SAFER"} with rasterio.open(dst, "w", **profile) as sink: for row0, row1 in row_bands(source.height, band_rows): window = band_window(source.width, row0, row1) - latitudes = (row_latitudes(row0, row1, source.height, north, south) - if feather else None) - sink.write(encode_array(source.read(1, window=window), step, sea_clamp, latitudes), + sink.write(encode_array(source.read(1, window=window), step, sea_clamp), window=window) @@ -234,27 +280,23 @@ def cut_zoom(src: Path, staging: Path, zoom: int, tile_format: str = "png") -> N str(src), str(staging)]) -def terrain_params(max_zoom: int, step: float, sea_clamp: bool, feather: bool, - tile_format: str) -> str: +def terrain_params(max_zoom: int, step: float, sea_clamp: bool, tile_format: str) -> str: """The cut's own settings, recorded beside the pyramid as its freshness dependency. Everything listed alters the emitted bytes and nothing outside it does — the master and the - output dir are `is_stale`'s own arguments, not settings. The polar feather latitudes and the - terrarium zero point are in here because they are module CONSTANTS: they have no other file to - move an mtime, so editing one would otherwise restage nothing while changing every tile. + output dir are `is_stale`'s own arguments, not settings. The terrarium zero point is in here + because it is a module CONSTANT: it has no other file to move an mtime, so editing it would + otherwise restage nothing while changing every tile. """ driver, creation_options = TILE_FORMATS[tile_format] return json.dumps({ "max_zoom": max_zoom, "step": step, "sea_clamp": sea_clamp, - "feather": feather, "tile_size": TILE_SIZE, "format": driver, "creation_options": creation_options, "base_shift": BASE_SHIFT, - "feather_lat_lo": FEATHER_LAT_LO, - "feather_lat_hi": FEATHER_LAT_HI, }, sort_keys=True, indent=2) @@ -278,22 +320,22 @@ def tiles_are_fresh(out: Path, master: Path) -> bool: and not is_stale(live, done_marker(master), terrain_params_path(out))) -def elevation_source(work: Path, zoom: int, master: Path) -> Path: +def elevation_source(work: Path, zoom: int, master: Path, master_zoom: int) -> Path: """The elevation raster for `zoom` — THE MASTER ITSELF at its native zoom. - `height_3857.tif` is already 512 x 2^MASTER_ZOOM, so "downsampling" to that zoom is a box-mean + `height_3857.tif` is already 512 x 2^`master_zoom`, so "downsampling" to that zoom is a box-mean by a factor of 1: the identity. Materialising it wrote a 47 GB byte-for-value copy of a file already on disk, on every build, and the only transform it applied was NaN -> 0, which `encode_array` applies again regardless. Verified over six windows from Everest to the Pacific abyss at max |master - copy| of exactly 0.0000 m, with shifted-window controls differing by 240-1660 m so the comparison could actually fail. """ - return master if zoom == MASTER_ZOOM else work / f"elev_z{zoom}.tif" + return master if zoom == master_zoom else work / f"elev_z{zoom}.tif" -def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, - master: Path, work: Path | None = None, keep_intermediates: bool = False, - tile_format: str = "png") -> Path: +def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, + master: Path, master_zoom: int, work: Path | None = None, + keep_intermediates: bool = False, tile_format: str = "png") -> Path: """Build a complete z0..max_zoom terrain-RGB pyramid under `out`, returning the tile dir. The elevation chain is built once at `max_zoom` and halved from there, so the expensive read of @@ -305,9 +347,8 @@ def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, identical whatever codec the tiles are written in, so re-cutting a built variant into another lossless format costs one encode pass, not another descent from the master. - Guarded per the module header's FRESHNESS section: the recipe is written BEFORE freshness is - asked, so changing a setting is what triggers its own re-cut, while `write_if_changed` means an - unchanged recipe never moves an mtime and never restages a pyramid that is still correct. + Recipe-gated in the usual order — see `freshness.write_if_changed`, and the module header's + FRESHNESS section for what each of this stage's two guards catches. EVERY CUT IS A CLEAN FULL CUT into `tiles_new`, swapped over `tiles` only on success, with one generation of rollback at `tiles_old`. GDAL writes each tile in place, so a run killed mid-cut @@ -315,7 +356,7 @@ def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, """ out.mkdir(parents=True, exist_ok=True) write_if_changed(terrain_params_path(out), - terrain_params(max_zoom, step, sea_clamp, feather, tile_format)) + terrain_params(max_zoom, step, sea_clamp, tile_format)) tiles = out / "tiles" if tiles_are_fresh(out, master): print(f"terrain tiles fresh -> skip cut ({tiles})", flush=True) @@ -323,11 +364,11 @@ def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, work = work or out / "work" work.mkdir(parents=True, exist_ok=True) - variant = f"{'sea0' if sea_clamp else 'bathy'}_s{step:g}{'' if feather else '_nofeather'}" + variant = f"{'sea0' if sea_clamp else 'bathy'}_s{step:g}" - top = elevation_source(work, max_zoom, master) + top = elevation_source(work, max_zoom, master, master_zoom) if top != master and is_stale(top, done_marker(master)): - factor = 2 ** (MASTER_ZOOM - max_zoom) + factor = 2 ** (master_zoom - max_zoom) print(f"downsample master /{factor} -> {grid_size(max_zoom)}^2 ...", flush=True) downsample_elevation(master, top, factor) mark_done(top) @@ -338,14 +379,14 @@ def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, staging.mkdir(parents=True) for zoom in range(max_zoom, -1, -1): - level = elevation_source(work, zoom, master) + level = elevation_source(work, zoom, master, master_zoom) if zoom < max_zoom: - parent = elevation_source(work, zoom + 1, master) + parent = elevation_source(work, zoom + 1, master, master_zoom) if is_stale(level, done_marker(parent)): downsample_elevation(parent, level, 2) mark_done(level) encoded = work / f"rgb_{variant}_z{zoom}.tif" - encode_raster(level, encoded, step, sea_clamp, feather) + encode_raster(level, encoded, step, sea_clamp) print(f"z{zoom}: encoded {grid_size(zoom)}^2 -> cutting ...", flush=True) cut_zoom(encoded, staging, zoom, tile_format) if not keep_intermediates: @@ -362,28 +403,65 @@ def build(out: Path, max_zoom: int, step: float, sea_clamp: bool, feather: bool, return tiles -def main() -> None: +def build_parser() -> argparse.ArgumentParser: + """The CLI, separable from `main` so its defaults are assertable without running a cut. + + `cap_ladder` and `pack_pmtiles` split theirs for the same reason: the interesting properties + here are what happens when a flag is OMITTED, and that is unreachable through a function whose + next statement reads a 46 GB raster. + """ ap = argparse.ArgumentParser(description=__doc__) - ap.add_argument("--out", type=Path, required=True, help="pyramid root (tiles/ inside)") + # REQUIRED, WITH NO DEFAULT, exactly as shade_planet's is and for the same reason: a pass that + # assumes Earth because nobody said otherwise does not fail, it emits a complete and plausible + # pyramid for the wrong planet. Here it decides four things at once — the master, where the + # output lands, the ceiling cut to, and the descent's factor. + ap.add_argument("--body", required=True, + help=f"which planet this cut is for ({', '.join(sorted(bodies.BODIES))})") + # STILL REQUIRED, AND DELIBERATELY NOT DEFAULTED TO THE STAGE DIRECTORY. The live pyramid sits + # one level further down, in an operator-named variant directory (``), and + # that name is a choice rather than a derivation — the module header records that the name + # carries only part of the recipe, which is why the sidecar exists. A default would therefore + # not have pointed at the live pyramid: it would have built a second one beside it, found the + # first "missing", and left the packer reading whichever the operator remembered. + # It is checked against the body instead, below, which is the half that was actually unsafe. + ap.add_argument("--out", type=Path, required=True, + help="pyramid root, under this body's planet_terrain dir (tiles/ inside)") ap.add_argument("--work", type=Path, default=None, help="elevation-chain dir; share it across variants to read the master once") - ap.add_argument("--master", type=Path, - default=paths.DATA / "work/planet_tiles/height_3857.tif") - ap.add_argument("--max-zoom", type=int, default=8) + ap.add_argument("--master", type=Path, default=None) + ap.add_argument("--max-zoom", type=int, default=None) ap.add_argument("--step", type=float, default=QUANTISATION_M, help="metres per encoded level") - ap.add_argument("--sea", choices=["clamp", "bathy"], default="clamp", + # DEFAULTS TO WHAT IS ON THE WIRE, which it did not: the choice defaulted to "clamp" while the + # shipped recipe records the opposite, so the bare command rebuilt a pyramid that was not the + # live one. Deriving it from SHIPPED_SEA_CLAMP is what makes the two agree by construction — + # and the clamp is not merely a look question on a body with no sea, where every point below + # zero is real ground and flattening it deletes the deepest basin on the planet. + ap.add_argument("--sea", choices=["clamp", "bathy"], + default="clamp" if SHIPPED_SEA_CLAMP else "bathy", help="clamp: sea flattened to 0; bathy: seafloor displaced too") - ap.add_argument("--no-feather", action="store_true", - help="skip the polar ramp (only for isolating the cap seam)") ap.add_argument("--format", choices=sorted(TILE_FORMATS), default="webp", help="delivery codec; both lossless, webp is ~0.67x png") ap.add_argument("--keep-intermediates", action="store_true") - args = ap.parse_args() + return ap - tiles = build(args.out, args.max_zoom, args.step, args.sea == "clamp", - not args.no_feather, args.master, args.work, args.keep_intermediates, - args.format) + +def main() -> None: + args = build_parser().parse_args() + body = bodies.BODIES[args.body] + master = args.master or bodies.work_dir(body, "planet_tiles") / "height_3857.tif" + out = out_under_body(body, args.out) + native = master_zoom_for(body) + # THE ONE PLACE THE DECLARATION MEETS THE ARTIFACT. Every failure a wrong `native` causes is + # silent downstream — see `master_grid_mismatch` — so it is checked here, before the descent + # that would encode it into every tile and into the recipe beside them. + mismatch = master_grid_mismatch(master, native) + if mismatch: + raise SystemExit(mismatch) + + tiles = build(out, args.max_zoom if args.max_zoom is not None else native, + args.step, args.sea == "clamp", + master, native, args.work, args.keep_intermediates, args.format) count = sum(1 for _ in tiles.rglob(f"*.{args.format}")) size = sum(path.stat().st_size for path in tiles.rglob(f"*.{args.format}")) print(f"{count} tiles, {size / 1e9:.2f} GB -> {tiles}", flush=True) diff --git a/pipeline/vector_raster.py b/pipeline/vector_raster.py new file mode 100644 index 0000000..47e8843 --- /dev/null +++ b/pipeline/vector_raster.py @@ -0,0 +1,121 @@ +"""Reproject a vector onto a target grid and burn it — one owner for a two-step nobody may split. + +`gdal_rasterize` DOES NOT REPROJECT, AND ITS FAILURE IS A FILE FULL OF ZEROS. Handed a vector in one +CRS and a `-te` extent in another, it finds every vertex outside the extent, burns nothing, exits 0, +and writes a raster whose header is exactly what was asked for. Downstream that output cannot be told +from "this body has nothing there" by any type, any test, or any eye — so the reprojection and the +burn are one function here rather than two lines a caller is trusted to keep in order. + +`-a_srs` IS NOT THE REPROJECTION, which is the confusion this module exists to make unrepresentable. +`ogr2ogr -a_srs` ASSIGNS a label and moves no coordinate; used in place of `-t_srs` it produces +exactly the all-zero raster above. It is a cure in one situation only — stripping a celestial-body +label PROJ refuses to operate across — and that situation belongs to sources this pipeline does not +take: Natural Earth is 4326 outright, and the SIM 3292 GeoJSON already reads as 4326 unaided. + +THE EMPTINESS GUARD IS THE CALLER'S CLAIM AND NOT THIS MODULE'S. Only the caller knows whether +nothing is a legitimate answer, so `must_draw` carries a sentence about what should have appeared and +None means an empty burn is fine. A windowed scan stops at the first non-zero pixel, so the whole +raster is read only in the case that is about to raise anyway. + +THE TWO STEPS ANSWER DIFFERENTLY ON A RE-RUN, AND THIS NOTE ONCE COVERED ONLY THE SECOND. Measured: +given `-te` and `-ts`, `gdal_rasterize` RECREATES an existing target at the new size rather than +opening it in update mode, creation options included — so the BURN needs no unlink, and +`snow.rasterize_glaciers_raster`'s unlink guards the call shape that omits them, which this one +cannot express. That measurement is true and it says nothing about `ogr2ogr`, which the sentence +above it used to be read as covering. + +`ogr2ogr` CANNOT WRITE OVER AN EXISTING GEOJSON BY ANY FLAG, so the reprojection unlinks first. +`-overwrite` asks the driver to DeleteLayer and the GeoJSON driver has none ("DeleteLayer() not +supported by this dataset"); dropping the flag only trades that for "The GeoJSON driver does not +overwrite existing files". Both exit 1, so nothing was ever silently stale — but the stage succeeded +exactly once and failed on every re-run, against a repo rule that stages are resumable. It went +unseen because the projected intermediate outlives a run and no caller had ever produced one twice: +the first thing to ask for it was a grid change, which is a body's ceiling moving. +""" + +import subprocess +from pathlib import Path + +import rasterio + + +class NothingBurnt(RuntimeError): + """A burn the caller declared non-empty produced no pixels. + + Its own class rather than a bare `RuntimeError` so a caller can tell this apart from a GDAL + failure: the subprocesses raise `CalledProcessError`, and the whole point of this exception is + that both commands SUCCEEDED and the answer is still wrong. + """ + + +def reproject_argv(source: Path, target_srs: str, out: Path) -> list[str]: + """`ogr2ogr` into the target CRS. Pure, so a test can pin the flags without a GDAL run. + + `-t_srs` and never `-a_srs`; the module note holds why that is the whole subject here. The output + driver comes from `out`'s extension, which is `ogr2ogr`'s own convention rather than ours. + + NO `-overwrite`, and its absence is the fix rather than an omission: the flag asks for a + DeleteLayer the GeoJSON driver does not implement, so it turned every re-run into an exit 1. + `burn_onto_grid` removes `out` first, which is what overwriting a single-layer file actually is. + """ + return ["ogr2ogr", "-t_srs", target_srs, str(out), str(source)] + + +def rasterize_argv(vector: Path, bounds: tuple[float, float, float, float], width: int, height: int, + out: Path, creation_options: tuple[str, ...] = ()) -> list[str]: + """`gdal_rasterize` of an ALREADY-PROJECTED vector onto this grid, as a 0/1 Byte mask. + + `creation_options` is empty for a cap-sized target and carries TILED/DEFLATE/BIGTIFF for a + planet-sized one — the axis two shipping callers actually differ on, rather than a knob invented + for a caller that does not exist. Each entry is one `-co` argument, e.g. `"TILED=YES"`. + """ + left, bottom, right, top = bounds + options: list[str] = [] + for option in creation_options: + options += ["-co", option] + return ["gdal_rasterize", "-q", "-burn", "1", "-init", "0", "-ot", "Byte", *options, + "-te", str(left), str(bottom), str(right), str(top), + "-ts", str(width), str(height), str(vector), str(out)] + + +def drew_nothing(raster: Path) -> bool: + """Whether a burnt raster is entirely zero, read a block at a time and short-circuiting. + + Windowed rather than `read(1).any()` because the planet-grid caller's mask is 32768² Byte: a + whole read is a gigabyte to answer a yes/no question, and the yes case exits on the first block + holding anything. + """ + with rasterio.open(raster) as dataset: + for _index, window in dataset.block_windows(1): + if dataset.read(1, window=window).any(): + return False + return True + + +def burn_onto_grid(source: Path, target_srs: str, bounds: tuple[float, float, float, float], + width: int, height: int, projected: Path, out: Path, + creation_options: tuple[str, ...] = (), + must_draw: "str | None" = None) -> Path: + """Reproject `source` into `target_srs`, burn it onto this grid, and return the raster. + + `projected` is the intermediate the caller names, on `perennial_ice.WarpToCap`'s rule: a helper + inventing its own filename spells out a convention the caller already owns, and the caller is + what has a work directory and a pole to name it after. + + `must_draw` names what the caller expects to see and raises `NothingBurnt` when nothing appears. + Pass it wherever an empty answer would be a broken projection rather than an honest fact about the + body — which is every caller whose geometry is known to intersect the grid. + """ + # The overwrite, done the one way that works for a single-layer file — see the module note. It + # must be here rather than in `reproject_argv`, which is pure so the flags stay checkable. + projected.unlink(missing_ok=True) + subprocess.run(reproject_argv(source, target_srs, projected), check=True, capture_output=True) + subprocess.run(rasterize_argv(projected, bounds, width, height, out, creation_options), + check=True, capture_output=True) + if must_draw is not None and drew_nothing(out): + raise NothingBurnt( + f"{must_draw} rasterised to nothing. Both commands succeeded, so this is geometry that " + f"missed the grid rather than a GDAL failure: check that {target_srs} is the CRS " + f"{bounds} is measured in, and that the reprojection ran at all." + ) + return out diff --git a/pipeline/verify.py b/pipeline/verify.py index c1da8eb..0ed36e4 100644 --- a/pipeline/verify.py +++ b/pipeline/verify.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Raster comparison that cannot quietly lie to you. This module exists because of a measured pattern, not a hunch. Across two days, SEVEN @@ -69,8 +68,8 @@ def beyond_tolerance(self) -> int: return int(self.histogram[self.tolerance + 1:].sum()) def report(self) -> str: - lines = [f"compared {self.pixels_compared:,} of {self.pixels_total:,} px " - f"({self.pixels_compared / self.pixels_total * 100:.1f}% coverage)"] + lines = [(f"compared {self.pixels_compared:,} of {self.pixels_total:,} px " + f"({self.pixels_compared / self.pixels_total * 100:.1f}% coverage)")] if self.pixels_compared < self.pixels_total: lines.append(" !! PARTIAL SCAN -- this is a sample, not a proof") lines.append("") diff --git a/pipeline/wrap_seam.py b/pipeline/wrap_seam.py new file mode 100644 index 0000000..22b63e3 --- /dev/null +++ b/pipeline/wrap_seam.py @@ -0,0 +1,110 @@ +"""Close the one column a global warp cannot fill — where the raster meets itself at ±180. + +Warping a global source into EPSG:3857 leaves the easternmost pixel column with nothing to sample. +The source ends at +180 and the resampling kernel wants ground beyond it, which exists only on the +far side of the seam; GDAL has no notion that the two edges are neighbours, so it writes nodata. + +THE REASON THAT IS NOT A COSMETIC EDGE CASE is that downstream nothing can tell "missing" from "very +low ground" — they are the same float. Measured on Mars: the elevation ramp painted the column at +the bottom of its scale, `render/hillshade` read it as the lip of a 30 km cliff and shadowed the +column BESIDE it, and `tile/terrain_rgb.encode_array` packed it as a real height. One warp artifact, +two adjacent dark columns, a straight line from pole to pole, shipped. + +THE FILL IS THE MIDPOINT OF THE TWO NEIGHBOURS, AND THAT IS EXACT RATHER THAN TOLERABLE. The seam +column sits one pixel east of the second-to-last column and one pixel west of column 0, so their +mean is a linear interpolation evaluated exactly where the missing sample belongs — not an +approximation chosen for being cheap. Validated against the 13,594 rows of Mars's seam the warp DID +fill, i.e. against ground truth rather than against taste: median error 5.8 m, p95 52 m, worst +232 m, mean +3.2 m, on terrain spanning 29 km. Filling with zero instead measures 3,008 m median. + +WHY THIS REFUSES EVERYWHERE ELSE RATHER THAN FILLING. A general "interpolate every hole" rule +invents ground silently on a body whose source genuinely has gaps, and Earth's land DEM is exactly +such a source — a missing Copernicus tile fuses as ocean with nothing raised. The seam is the one +hole whose neighbours are known to BE its neighbours, because the projection says so. Any other +hole is a data problem to be seen, not smoothed, so this raises instead. +""" + +from pathlib import Path + +import numpy as np +import rasterio + +from pipeline.mercator import MERCATOR_HALF_M +from pipeline.raster_io import band_window, column_window, row_bands + +#: Source rows held at once while scanning for holes. At the widest grid this pipeline warps +#: (65536 px) a band of this many float32 rows is ~268 MB, which leaves the one-heavy-job cap room +#: for GDAL's own block cache beside it. +SCAN_ROW_BUDGET = 1024 + + +def spans_the_world(dataset) -> bool: + """Does this raster cover the whole Mercator plane in longitude, to within one pixel? + + TO WITHIN A PIXEL, NOT TO THE DECIMAL, and the tolerance is the load-bearing part. Earth's + heightfield overshoots `MERCATOR_HALF_M` by 12.25 m where Mars lands on it exactly, and both are + global; a test written against the digits would accept one body, reject the other, and describe + itself as a check on the projection while actually checking whose warp wrote the file. + """ + span = dataset.bounds.right - dataset.bounds.left + return abs(span - 2.0 * MERCATOR_HALF_M) <= dataset.res[0] + + +def close_wrap_seam(raster: Path, band_rows: int = SCAN_ROW_BUDGET) -> int: + """Fill `raster`'s antimeridian column in place from its two neighbours; return pixels filled. + + IDEMPOTENT BY WAY OF THE DECLARATION, which is what makes it safe in a resumable pipeline: a + closed raster declares no nodata, so a second call returns 0 without reading a pixel. That also + makes the postcondition a thing a reader can check — after this, the file SAYS it has no missing + data, and `terrain_rgb.encode_array`'s NaN-only guard is correct for this body rather than + accidentally correct because nobody declared a sentinel. + + Raises rather than returns on the two failures that must not pass quietly: a raster that is not + global (its edges are not neighbours, so there is nothing to interpolate ACROSS) and a hole + anywhere off the seam (see the module docstring — that is data to look at, not to smooth). + """ + with rasterio.open(raster, "r+") as dataset: + missing = dataset.nodata + if missing is None: + return 0 + if not spans_the_world(dataset): + raise SystemExit( + f"{raster} declares nodata {missing} but spans " + f"{dataset.bounds.right - dataset.bounds.left:.1f} map units against a world of " + f"{2 * MERCATOR_HALF_M:.1f} — its east and west edges are not neighbours, so a wrap " + f"fill would interpolate across ground that is not there") + + width, height = dataset.width, dataset.height + seam = width - 1 + holes_per_column = np.zeros(width, dtype=np.int64) + for row0, row1 in row_bands(height, band_rows): + block = dataset.read(1, window=band_window(width, row0, row1)) + holes_per_column += (block == missing).sum(axis=0) + + off_seam = np.flatnonzero(holes_per_column) + off_seam = off_seam[off_seam != seam] + if off_seam.size: + raise SystemExit( + f"{raster} has nodata in {off_seam.size} column(s) other than its wrap seam — " + f"first at column {int(off_seam[0])} ({int(holes_per_column[off_seam[0]])} px), " + f"{int(holes_per_column.sum() - holes_per_column[seam])} px in total. Only the seam " + f"has known neighbours; a hole anywhere else is a gap in the source and filling it " + f"would invent ground. Fix the fusion, do not widen this.") + + filled = int(holes_per_column[seam]) + if filled: + # Read as float64 before averaging: the raster is float32, and the midpoint of two + # float32s rounded back to float32 is the value we want written, not an accumulation. + west = dataset.read(1, window=column_window(height, seam - 1, seam))[:, 0] + east = dataset.read(1, window=column_window(height, 0, 1))[:, 0] + column = dataset.read(1, window=column_window(height, seam, width))[:, 0] + midpoint = 0.5 * (west.astype(np.float64) + east.astype(np.float64)) + column = np.where(column == missing, midpoint, column).astype(dataset.dtypes[0]) + dataset.write(column.reshape(height, 1), 1, + window=column_window(height, seam, width)) + + # LAST, AND ONLY AFTER THE WRITE. The declaration is the file's claim about its own + # contents, so clearing it before the fill would leave a window in which a crash produced a + # raster that reads as complete and is not — the `.done` marker rule, one level down. + dataset.nodata = None + return filled diff --git a/pyproject.toml b/pyproject.toml index 786b8c6..aa4d7dc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ dev = [ "pyright>=1.1.411", "pytest>=9.1.1", "pytest-cov>=7.1.0", + "ruff>=0.16.1", ] # Same engine as Pylance, so `uv run pyright` reproduces the IDE's findings. @@ -46,6 +47,37 @@ venvPath = "." venv = ".venv" typeCheckingMode = "standard" +# The second static gate, beside pyright and asking a different question: pyright asks whether the +# types line up, ruff asks whether the code says what it means. Both stay at ZERO — there is no +# "pre-existing finding" allowance here any more than there is for the type checker. +# +# NO `select`, deliberately. This takes ruff's own default rule set rather than a hand-written list +# of families, because the default is a CURATED SUBSET and not a union: bare `E` is 5,034 findings +# in this repo (line length alone), `PL` is 235, and a hand-written list would have to be +# re-derived at every upgrade to stay equal to it. The bargain is that `uv lock --upgrade` is where +# new rules arrive and where the gate can go red on an otherwise unrelated change — the same +# bargain already struck with pyright's `standard` mode, and a linter that never learns anything +# new is not worth running. +# +# `target-version` is inferred from `requires-python` above, so ruff targets 3.12 and cannot +# propose syntax that Blender's bundled 3.13 lacks. That is load-bearing rather than incidental: +# palette.py and the two bpy scripts are shared with an interpreter this venv never runs, and +# `scripts/check_blender_drift.sh` is the only other thing that looks. +[tool.ruff.lint] +# E402 is NOT in the default set, and it is turned on here because the repo had already asked for +# it: seven `# noqa: E402` directives were written against it. Six of them suppress nothing — they +# are leftovers from a refactor that moved the imports to the top, and RUF100 is what says so. The +# seventh, in gen_spotlight.py, is a real mid-file import with a paragraph explaining itself. +# Enabling the rule costs zero findings and is what keeps that one comment honest. +extend-select = ["E402"] + +ignore = [ + # `dict(driver="GTiff", width=…)` — 40 sites, nearly all GDAL creation profiles, every one of + # them keyword-argument form rather than an empty `dict()`. The literal rewrite is a wash at + # best and reads worse wherever the keys line up as identifiers. + "C408", +] + # Tests cover the pure geometry/config resolution layer (framing math, config # validation, scope + frame resolution) — the deterministic core where a silent # regression would render a wrong hero. Rendering/fusion/downloads are not unit diff --git a/scripts/measure_cap_tile_agreement.py b/scripts/measure_cap_tile_agreement.py new file mode 100644 index 0000000..88b1172 --- /dev/null +++ b/scripts/measure_cap_tile_agreement.py @@ -0,0 +1,100 @@ +"""Do a body's shipped terrain tiles and its polar cap texture report the same height? + +The unit test for this property (`test_the_encode_is_a_pure_function_of_metres_at_every_latitude`) +runs a constructed raster through the encoder, deliberately touching no archive: it has to stay +green between a recipe change and the re-cut that answers it, so it cannot read what is on disk. +This is the other half — it reads the ARTIFACTS, and it is the check to run after a cut. + +The two producers overlap between `cap_render.CAP_EDGE_LAT` and the Mercator limit, and +`polarCaps.ts` crossfades between them there, so a disagreement is two surfaces the viewer sees at +once. A polar elevation ramp in the tile path once put them 4.7 km apart on Mars and 3.5 km on +Earth while every encoding test passed, because those tests compare bytes and this compares ground. + +Usage: `python -m scripts.measure_cap_tile_agreement --body mars` (or `earth`). +""" +import argparse +import math +from pathlib import Path + +import rasterio + +from pipeline import bodies +from pipeline.tile import cap_render, terrain_rgb + +#: Latitudes to compare — the crossfade band, from the cap disc's own edge to the Mercator limit. +#: Nothing below `CAP_EDGE_LAT` belongs here: the cap does not exist there, and a sampler that +#: clamped to the texture's edge instead of refusing reported a 1.2 km disagreement that was its +#: own doing. +SAMPLE_LATITUDES = (80.0, 81.0, 82.0, 83.0, 84.0, 84.8) + +#: A disagreement this large cannot be the cap texture's own coarseness — it samples ~2.3 km/px on +#: Mars against a tile's ~325 m, so a few hundred metres of terrain roughness is expected and a +#: kilometre is a systematic term. +TOLERANCE_M = 500.0 + + +def tile_metres(tiles: Path, zoom: int, latitude: float, longitude: float) -> float: + """Decode one shipped tile's pixel at a lat/lon, through the same arithmetic MapLibre applies.""" + world = terrain_rgb.TILE_SIZE * 2**zoom + sin_lat = math.sin(math.radians(latitude)) + pixel_x = (longitude + 180.0) / 360.0 * world + pixel_y = (0.5 - math.log((1 + sin_lat) / (1 - sin_lat)) / (4 * math.pi)) * world + tile_x, tile_y = int(pixel_x // terrain_rgb.TILE_SIZE), int(pixel_y // terrain_rgb.TILE_SIZE) + path = tiles / str(zoom) / str(tile_x) / f"{tile_y}.webp" + if not path.exists(): + raise SystemExit(f"no tile at {path} — cut the pyramid first") + with rasterio.open(path) as dataset: + encoded = dataset.read() + column = int(pixel_x) % terrain_rgb.TILE_SIZE + row = int(pixel_y) % terrain_rgb.TILE_SIZE + return float(terrain_rgb.decode_array(encoded, terrain_rgb.QUANTISATION_M)[row, column]) + + +def cap_metres(body: bodies.Body, latitude: float, longitude: float) -> float: + """Decode the north cap's AEQD elevation texture at a lat/lon. Radius is LINEAR in colatitude.""" + grid = cap_render.north_grid(body) + with rasterio.open(cap_render.cap_elev_asset(grid)) as dataset: + encoded = dataset.read() + side = encoded.shape[1] + radius = (90.0 - abs(latitude)) / (90.0 - abs(grid.edge_lat)) + if radius > 1.0: + raise SystemExit(f"lat {latitude} is outside the cap disc (edge {grid.edge_lat})") + angle = math.radians(longitude) + column = min(side - 1, max(0, int((0.5 + 0.5 * radius * math.sin(angle)) * side))) + row = min(side - 1, max(0, int((0.5 + 0.5 * radius * math.cos(angle)) * side))) + return float(terrain_rgb.decode_array(encoded, terrain_rgb.QUANTISATION_M)[row, column]) + + +def main() -> int: + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--body", required=True, choices=sorted(bodies.BODIES)) + parser.add_argument("--tiles", type=Path, default=None, + help="override the pyramid directory (default: the shipped variant)") + parser.add_argument("--lon", type=float, default=0.0, help="meridian to sample") + arguments = parser.parse_args() + body = bodies.get(arguments.body) + tiles = arguments.tiles or (bodies.work_dir(body, "planet_terrain") + / f"bathy_s{terrain_rgb.QUANTISATION_M:g}_webp" / "tiles") + zoom = terrain_rgb.master_zoom_for(body) + + print(f"{body.name}: z{zoom} tiles under {tiles}, north cap texture, meridian {arguments.lon}\n") + print(f"{'lat':>6} {'cap m':>10} {'tile m':>10} {'gap m':>10}") + print("-" * 40) + worst = 0.0 + for latitude in SAMPLE_LATITUDES: + cap = cap_metres(body, latitude, arguments.lon) + tile = tile_metres(tiles, zoom, latitude, arguments.lon) + gap = cap - tile + worst = max(worst, abs(gap)) + print(f"{latitude:6.1f} {cap:10.1f} {tile:10.1f} {gap:10.1f}") + + print(f"\nworst |gap| {worst:.1f} m against a {TOLERANCE_M:.0f} m tolerance") + if worst > TOLERANCE_M: + print("DISAGREE — the two surfaces the crossfade blends are not at the same height") + return 1 + print("AGREE") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/measure_mars_ice_white.py b/scripts/measure_mars_ice_white.py new file mode 100644 index 0000000..796e880 --- /dev/null +++ b/scripts/measure_mars_ice_white.py @@ -0,0 +1,302 @@ +"""Re-measure the colour of Mars's polar ice from the Viking mosaic, on the LIVE cap grids. + +THE ONLY REPRODUCER OF `palette.MARS_ICE_WHITE`, which is why it is tracked. Those four values decide +what colour each pole renders and no unit test can re-derive them: the target is an alpha-weighted +mean over the shipped ice, so producing one means building the cap grids, burning the mapped units +and grading the field exactly as the renderer does. + +A TARGET IS VOID WHEN ANYTHING UNDER IT MOVES, and two things already have: `ALPHA_LEVELS` was +re-pinned once when the original pair turned out to be percentiles of a scaffold rather than of the +shipped field, and `edge_lat` went 78 to 80. Either changes which pixels the mean runs over. Nothing +else in the repo can notice — the hex values keep reading as measured, because they were, against a +subject that has since moved. + +RED AND VIOLET ARE EVIDENCE; GREEN IS NOT AND CANNOT BE. The mosaic's green band is a fixed linear +combination of its own red and violet, so it carries no independent information and this script +measures a RATIO, never a colour. Turning that ratio into a colour is the second half below, and it +is a stated design rule rather than a measurement. + +THE RATIO AND THE WEIGHT COME OFF THE SAME PIXELS, which is the reason for one warp rather than two. +The alpha is graded from luma and the luma is collapsed from the very RGB being averaged, so no +resampling difference can creep between the weight and the thing it weights. + +WHAT IS DELIBERATELY NOT RESPELLED HERE. The extent rule, the levels, the grading curve, the feather +and the luma weights each have an owner in `render/mars_ice`, and this script is the INSTRUMENT that +produces the target — so it is exactly the copy whose drift would be invisible, agreeing with itself +while measuring something the renderer no longer paints. + +Usage, from the repo root: + systemd-run --user --scope -q -p MemoryMax=16G -p MemorySwapMax=0 \ + --working-directory="$PWD" -E PYTHONPATH="$PWD" -E GDAL_CACHEMAX=512 \ + .venv/bin/python scripts/measure_mars_ice_white.py [--compare] + + --compare is the standing oracle: it re-measures, re-derives, and exits non-zero if the pinned + constants stop reproducing. +""" + +import subprocess +import sys +from pathlib import Path + +import numpy as np +import rasterio + +from pipeline import bodies +from pipeline.acquire import download_viking_mosaic +from pipeline.render import mars_ice, palette, viking_luma +from pipeline.tile import cap_render + +#: Degrees of latitude kept either side of the pole, matching the levels script's band for the same +#: reason: the square cap FRAME reaches sqrt(2) * 10 degrees from the pole at its corners. +BAND_DEGREES = 20.0 + +#: Rec709, the same weights the luminance of a white is judged by everywhere else on this site. +REC709 = (0.2126, 0.7152, 0.0722) + +#: EARTH'S OWN WHITE IS THE UNIT OF MEASURE, not an arbitrary brightness. Holding these makes the +#: change from Earth's pair to a body's own a pure hue rotation, so nothing but colour moves. +LIT_LUMINANCE = 239.4 +SHADOW_LUMINANCE = 195.6 + +#: Earth's sunlit white sits this far above its own (R+B)/2. Green is unmeasurable, so it follows +#: Earth's rule rather than the data; this offset IS that rule, read off the reference body. +LOCUS_OFFSET = 2.0 + +#: HOW EACH POLE'S RATIO IS SPENT, and the south's is not a preference. sRGB cannot hold Earth's +#: luminance at the south's warmth — red pins at 255 and the locus arm falls to 231.6 — so the south +#: buys the brightness back by lifting green off the locus. That was ratified by eye, and it is the +#: one step here that a measurement did not decide. +STYLE_BY_POLE = {"north": "locus", "south": "cream"} + +#: How far a pinned pair's own red:violet may sit from the measured target before this is drift. +#: +#: THE TOLERANCE IS ON THE RATIO BECAUSE THE RATIO IS WHAT IS MEASURED. Comparing the four rounded +#: bytes instead makes the oracle fire on the last bit of an 8-bit channel: the first run of this +#: script reproduced the pinned north sunlit exactly and missed three other values by one DN, on +#: ratios agreeing to 0.004. An oracle that is red on arithmetic noise gets ignored, which is worse +#: than not having one. +#: +#: 0.02 is set against both ends of the range it must separate. Two measurement paths over the same +#: ice agree to about 0.004, and the two poles differ from each other by 0.242 — so this sits five +#: times above the noise and twelve times below the smallest difference that means anything. +RATIO_TOLERANCE = 0.02 + + +def out_dir() -> Path: + """Scratch for the warps and masks, resolved at call time so a redirected data root moves it.""" + return bodies.work_dir(bodies.get("mars"), "_ice_white") + + +def run(command: list[str]) -> None: + subprocess.run(command, check=True) + + +def luminance(rgb) -> float: + return sum(weight * channel for weight, channel in zip(REC709, rgb, strict=True)) + + +def on_locus(ratio: float, target: float, offset: float) -> tuple[float, float, float]: + """The (R, G, B) with this red:violet ratio, this Rec709 luminance, and green on the locus. + + CLIPS BY PINNING RED AT 255, which spends luminance rather than hue. Past a certain warmth the + sRGB cube simply cannot hold Earth's brightness, and that is a fact about the colour space rather + than a decision — the caller sees it as a luminance that came out below `target`. + """ + red_weight, green_weight, blue_weight = REC709 + blue = ((target - green_weight * offset) + / (red_weight * ratio + green_weight * (ratio + 1) / 2 + blue_weight)) + red = ratio * blue + if red > 255.0: + red, blue = 255.0, 255.0 / ratio + return red, (red + blue) / 2 + offset, blue + + +def cream(ratio: float, target: float) -> tuple[float, float, float]: + """The same hue at full red, buying back the luminance the locus loses by lifting green.""" + red, blue = 255.0, 255.0 / ratio + green = (target - REC709[0] * red - REC709[2] * blue) / REC709[1] + return red, green, blue + + +def arm(ratio: float, style: str) -> tuple[tuple[int, ...], tuple[int, ...]]: + """One pole's `(sunlit, shadowed)` pair. + + THE SHADOW INHERITS THE LIT END'S REALISED GREEN OFFSET rather than the nominal one, so both ends + sit on a single locus and the pair reads as one white under a light. Deriving the shadow + independently gives two whites that happen to share a ratio, which is visible as a hue shift + across the terminator. + """ + lit = (cream(ratio, LIT_LUMINANCE) if style == "cream" + else on_locus(ratio, LIT_LUMINANCE, LOCUS_OFFSET)) + realised_offset = lit[1] - (lit[0] + lit[2]) / 2 + shadow = on_locus(ratio, SHADOW_LUMINANCE, realised_offset) + return tuple(round(channel) for channel in lit), tuple(round(channel) for channel in shadow) + + +def earth_control() -> bool: + """Derive Earth's shipped pair from Earth's own ratio and luminance, and say whether it lands. + + THE RUN IS REFUSED IF THIS FAILS. Every number below comes out of the same two functions, so a + derivation that cannot reproduce a pair we have looked at for months has no standing on a pair + nobody has seen. The shadow's offset is Earth's own realised one, for the reason `arm` gives. + """ + lit = tuple(round(channel) for channel in + on_locus(palette.SNOW_RGB[0] / palette.SNOW_RGB[2], LIT_LUMINANCE, LOCUS_OFFSET)) + shadow_offset = (palette.SNOW_SHADOW_RGB[1] + - (palette.SNOW_SHADOW_RGB[0] + palette.SNOW_SHADOW_RGB[2]) / 2) + shadow = tuple(round(channel) for channel in + on_locus(palette.SNOW_SHADOW_RGB[0] / palette.SNOW_SHADOW_RGB[2], + SHADOW_LUMINANCE, shadow_offset)) + ok = lit == palette.SNOW_RGB and shadow == palette.SNOW_SHADOW_RGB + print("CONTROL Earth's pair, re-derived from its own ratio and luminance") + print(f" lit {lit} vs shipped {tuple(palette.SNOW_RGB)}" + f" {'OK' if lit == palette.SNOW_RGB else 'MISMATCH'}") + print(f" shadow {shadow} vs shipped {tuple(palette.SNOW_SHADOW_RGB)}" + f" {'OK' if shadow == palette.SNOW_SHADOW_RGB else 'MISMATCH'}") + return ok + + +def viking_rgb_on_cap(grid: cap_render.CapGrid) -> np.ndarray: + """The mosaic's three colours on this cap's AEQD grid, as float32 `(3, px, px)`. + + CROPPED TO A POLAR BAND BEFORE THE CAP WARP, and that is correctness rather than speed. Warping + the global mosaic straight onto an AEQD disc makes GDAL read the source at the disc's + pole-inflated average scale and silently decimate it — no error, a plausible raster, and an + ice edge made of exactly the structure that decimation erases. + + `viking_luma.degrees_vrt` owns the two-step relabel out of SimpleCylindrical metres and carries + why it cannot be one step; only the window is local to this script. + """ + warped = out_dir() / f"viking_rgb_{grid.name}_cap.tif" + if not warped.exists(): + whole = viking_luma.degrees_vrt(download_viking_mosaic.mosaic_path(), + out_dir() / "viking_degrees.vrt") + northern = grid.name == "north" + lat_lo, lat_hi = ((90.0 - BAND_DEGREES, 90.0) if northern + else (-90.0, BAND_DEGREES - 90.0)) + band = out_dir() / f"viking_rgb_{grid.name}_band.tif" + run(["gdal_translate", "-q", "-projwin", "-180", str(lat_hi), "180", str(lat_lo), + "-co", "TILED=YES", str(whole), str(band)]) + run(["gdalwarp", "-q", "-overwrite", "-t_srs", grid.aeqd, + "-te", str(-grid.edge_m), str(-grid.edge_m), str(grid.edge_m), str(grid.edge_m), + "-ts", str(grid.px), str(grid.px), "-r", "bilinear", + "-co", "TILED=YES", str(band), str(warped)]) + band.unlink() + + with rasterio.open(warped) as dataset: + return dataset.read().astype(np.float32) + + +def shipped_alpha(grid: cap_render.CapGrid, rgb: np.ndarray) -> np.ndarray: + """This pole's ice alpha, composed exactly as `perennial_ice._mars_cap_ice` composes it. + + Graded field times feathered extent, every term through its owner in `mars_ice`. The field is the + luma of the RGB passed in rather than the shipped raster, so the weight and the colour it weights + are the same pixels; `measure_viking_levels.py --compare` is what separately holds that luma to + the one the renderer reads. + """ + field = mars_ice.luma(rgb) + graded = mars_ice.albedo_alpha(field, mars_ice.ALPHA_LEVELS[grid.name], viking_luma.NODATA) + bounds = (-grid.edge_m, -grid.edge_m, grid.edge_m, grid.edge_m) + masks: dict[str, np.ndarray] = {} + for unit in mars_ice.NORTH_UNITS: + burnt = out_dir() / f"{unit.lower()}_{grid.name}.tif" + if not burnt.exists(): + mars_ice.burn_unit( + unit, grid.aeqd, bounds, grid.px, grid.px, + projected=out_dir() / f"{unit.lower()}_{grid.name}_aeqd.geojson", out=burnt, + must_draw=f"{unit} must reach the {grid.name} cap disc at edge_lat {grid.edge_lat}") + with rasterio.open(burnt) as dataset: + masks[unit] = dataset.read(1).astype(bool) + + extent = mars_ice.extent_for(masks, grid.name == "north") + feather = mars_ice.feather_alpha(extent, cap_render.cap_ground_metres_per_px(grid)) + return graded * feather + + +def paint_target(grid: cap_render.CapGrid) -> float: + """This pole's alpha-weighted red:violet, over the ice as it is actually painted. + + WEIGHTED, NOT MASKED, because the ice does not have an edge — it has a grading. A threshold would + make the answer a function of where the threshold was put, and the dim tail is a large part of + the north's extent. + """ + rgb = viking_rgb_on_cap(grid) + alpha = shipped_alpha(grid, rgb) + weight = alpha.sum() + red, blue = float((alpha * rgb[0]).sum()), float((alpha * rgb[2]).sum()) + ratio = red / blue + + print(f"\n{grid.name.upper()} (edge_lat {grid.edge_lat}, {grid.px} px, " + f"{cap_render.cap_ground_metres_per_px(grid):.2f} ground m/px)") + print(f" painted ice: {100 * (alpha > 0).mean():5.2f}% of the disc, " + f"mean alpha where painted {alpha[alpha > 0].mean():.3f}") + print(f" alpha-weighted red {red / weight:7.2f} violet {blue / weight:7.2f}" + f" ratio {ratio:.3f}") + return ratio + + +def report(): + """Measure and derive both poles as `(ratios, pairs)`, or None if the control refused.""" + if not earth_control(): + print("\nREFUSING: the derivation does not reproduce Earth's shipped pair", flush=True) + return None + + mars = bodies.get("mars") + measured, derived = {}, {} + for grid in (cap_render.north_grid(mars), cap_render.south_grid(mars)): + ratio = paint_target(grid) + style = STYLE_BY_POLE[grid.name] + lit, shadow = arm(ratio, style) + measured[grid.name], derived[grid.name] = ratio, (lit, shadow) + print(f" {style:>6} arm -> #{lit[0]:02X}{lit[1]:02X}{lit[2]:02X} {lit}" + f" / #{shadow[0]:02X}{shadow[1]:02X}{shadow[2]:02X} {shadow}" + f" luminance {luminance(lit):.1f}") + return measured, derived + + +def compare(measured, derived) -> int: + """Hold the SHIPPED whites to the hue the ice now measures. + + THE TEST IS THAT EACH PINNED PAIR STILL DESCRIBES ITS OWN ICE, which is a question about hue and + not about bytes. Both ends of a pair are checked because a pair is one white under two lights: if + the shadow's ratio drifted alone, the terminator would shift hue and nothing about the sunlit end + would say so. + + The derived pair is printed rather than asserted. It is what a re-pin would copy, but a re-pin is + a look decision — the eye ratified these, and a ratio has no standing to overrule it. + """ + print("\n--- do the shipped whites still describe this ice? ---") + worst = 0.0 + for pole, target in sorted(measured.items()): + for end, pinned in (("sunlit", palette.MARS_ICE_WHITE[pole][0]), + ("shadow", palette.MARS_ICE_WHITE[pole][1])): + ratio = pinned[0] / pinned[2] + deviation = abs(ratio - target) + worst = max(worst, deviation) + print(f" {pole:>5} {end}: pinned {tuple(pinned)} is red:violet {ratio:.3f} " + f"against a measured {target:.3f} off by {deviation:.3f}" + f" {'OK' if deviation <= RATIO_TOLERANCE else 'DRIFT'}") + + print(f"\nworst deviation of a shipped white from its measured hue: {worst:.3f} " + f"(tolerance {RATIO_TOLERANCE})") + if worst > RATIO_TOLERANCE: + print("\nThe subject moved. What this run derives from the current data:") + for pole, (lit, shadow) in sorted(derived.items()): + print(f' "{pole}": ({tuple(lit)}, {tuple(shadow)}),') + print("Re-judge on the globe before re-pinning.") + return 0 if worst <= RATIO_TOLERANCE else 1 + + +def main() -> int: + out_dir().mkdir(parents=True, exist_ok=True) + result = report() + if result is None: + return 1 + if "--compare" in sys.argv: + return compare(*result) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/measure_viking_levels.py b/scripts/measure_viking_levels.py new file mode 100644 index 0000000..a870c8a --- /dev/null +++ b/scripts/measure_viking_levels.py @@ -0,0 +1,270 @@ +"""Re-measure Mars's ice alpha levels from the Viking brightness field, on the LIVE cap grids. + +THE ONLY REPRODUCER OF `mars_ice.ALPHA_LEVELS`, which is why it is tracked. Those four numbers +decide how white Mars's poles render and no unit test can re-derive them: a level is a percentile +over a pixel set, so producing one means building the polar cap grids and burning the mapped units +onto them. It is tracked so the constants in shipped code have a reachable owner — a pointer a +reader cannot follow asserts that an explanation exists somewhere, which is worse than none. + +A LEVEL IS VOID WHEN EITHER THE FIELD OR THE GRID MOVES, and both have moved before. Re-run this +after any change to `render/viking_luma`'s output or to `cap_render`'s `edge_lat` — the last such +change moved the north's cap level by 11.4% while leaving the south's at 0.03 DN, so neither +direction of "it probably did not matter" is safe. + +THE TWO DEFINITIONS BELOW ARE THE RATIFIED ARM'S AND MUST NOT BE IMPROVED HERE: + + ground = valid & ~lApc & ~Apu median -> alpha 0 + cap = valid & lApc median -> alpha 1 + +Two subtleties that a re-derivation gets wrong by being reasonable. `ground` excludes BOTH units at +BOTH poles, including the south where `Apu` is not part of the extent — so it is not `~extent`. And +`cap` is `lApc` ALONE at both poles, including the north where the extent is `lApc | Apu` — so it is +not `extent` either. Using the extent for either one changes the levels and therefore the look. +`mars_ice.albedo_alpha` owns what is done with them; these two lines are the part that lives here. + +Grade on LUMA, never the blue/red ratio: that discriminant inverts at the north pole. + +Usage, from the repo root: + systemd-run --user --scope -q -p MemoryMax=12G -p MemorySwapMax=0 \ + --working-directory="$PWD" -E PYTHONPATH="$PWD" -E GDAL_CACHEMAX=512 \ + .venv/bin/python scripts/measure_viking_levels.py [--pipeline|--compare] + + --compare is the standing oracle: it measures both ways and exits non-zero if the shipped field + stops reproducing the pinned levels. +""" + +import subprocess +import sys +from pathlib import Path + +import numpy as np +import rasterio + +from pipeline import bodies +from pipeline.acquire import download_viking_mosaic +from pipeline.render import mars_ice, viking_luma +from pipeline.tile import cap_render + +#: Degrees of latitude kept either side of the pole. The square cap FRAME reaches sqrt(2) * 10 +#: degrees from the pole at its corners, i.e. ~75.9, so 20 clears it with room for the ground ring +#: the `ground` percentile is taken over. +BAND_DEGREES = 20.0 + +#: Neither the mosaic path, the sphere, nor the luma weights are respelled here. Each has an owner +#: — `download_viking_mosaic`, `viking_luma`, `mars_ice` — and this script is the INSTRUMENT that +#: produces the levels, so it is exactly the copy whose drift would be invisible: it would keep +#: agreeing with itself while measuring a field the renderer no longer reads. + + +def out_dir() -> Path: + """Scratch for the warps and masks, resolved at call time so a redirected data root moves it.""" + return bodies.work_dir(bodies.get("mars"), "_ice_levels") + + +def run(command: list[str]) -> None: + subprocess.run(command, check=True) + + +def viking_band(northern: bool) -> Path: + """Viking's polar band as EPSG:4326-labelled degrees, rebuilt from the raw mosaic. + + THE INFORMATIONAL ARM, kept because `--compare` needs a second opinion that does not come from + the stage under test. It rebuilds the band on its own square-degree grid, which is NOT the grid + the pipeline ships; the difference between the two is the measurement `--compare` reports. + + REBUILT RATHER THAN REUSED. Equivalent bands were left behind by an earlier pass with no record + of how they were made, which is the provenance gap the acquirers exist to close. + + THE RELABEL IS AN IDENTITY ON THE ANGLES, not a reprojection. Once out of SimpleCylindrical + metres the grid is already degrees on an unflattened sphere; PROJ refuses to warp between + celestial bodies, so declaring EPSG:4326 is what lets the cap's Earth-sphered AEQD read it. + `fuse/relabel_mars.py` carries the full argument. + """ + mosaic = download_viking_mosaic.mosaic_path() + name = "north" if northern else "south" + out = out_dir() / f"viking_{name}_4326.tif" + if out.exists(): + return out + + with rasterio.open(mosaic) as dataset: + degrees_per_px = 360.0 / dataset.width + + lat_lo, lat_hi = (90.0 - BAND_DEGREES, 90.0) if northern else (-90.0, BAND_DEGREES - 90.0) + degrees = out_dir() / f"viking_{name}_marsdeg.tif" + run(["gdalwarp", "-q", "-overwrite", "-t_srs", viking_luma.MARS_LONGLAT, + "-te", "-180", str(lat_lo), "180", str(lat_hi), + "-tr", str(degrees_per_px), str(degrees_per_px), "-r", "bilinear", + "-co", "TILED=YES", str(mosaic), str(degrees)]) + run(["gdal_translate", "-q", "-a_srs", "EPSG:4326", str(degrees), str(out)]) + degrees.unlink() + return out + + +def viking_on_cap(grid: cap_render.CapGrid, + from_pipeline: bool = False) -> tuple[np.ndarray, np.ndarray]: + """Viking luma on this cap's AEQD grid, and the validity mask, as `(luma, valid)`. + + `from_pipeline` READS `render/viking_luma`'s SHIPPED RASTER INSTEAD OF REBUILDING THE BAND, and + that is what makes this script an oracle over the pipeline rather than a parallel implementation + of it. The levels are percentiles of a particular field; if the stage that ships that field + resamples it even slightly differently from the chain the levels were taken through, the four + pinned numbers stop describing what gets rendered. Running both modes and comparing is the only + thing that can say so. + """ + if from_pipeline: + band = viking_luma.luma_path() + warped = out_dir() / f"viking_{grid.name}_cap_pipeline.tif" + else: + band = viking_band(grid.name == "north") + warped = out_dir() / f"viking_{grid.name}_cap.tif" + if not warped.exists(): + run(["gdalwarp", "-q", "-overwrite", "-t_srs", grid.aeqd, + "-te", str(-grid.edge_m), str(-grid.edge_m), str(grid.edge_m), str(grid.edge_m), + "-ts", str(grid.px), str(grid.px), "-r", "bilinear", + "-co", "TILED=YES", str(band), str(warped)]) + + with rasterio.open(warped) as dataset: + raw = dataset.read().astype(np.float32) + # The pipeline raster is ALREADY the collapse; the band is still three colours. Both arrive here + # as the same quantity, which is the comparison's whole premise. + values = raw[0] if from_pipeline else mars_ice.luma(raw) + # `values > 0` IS the all-bands-nonzero test the earlier spelling wrote out by hand, and it is + # identical rather than merely close: the weights are positive and the channels non-negative, so + # the sum vanishes exactly where all three do. `mars_ice.luma` carries the argument. + return values, values > 0 + + +def unit_masks(grid: cap_render.CapGrid) -> dict[str, np.ndarray]: + """`lApc` and `Apu` burnt onto this cap grid, both poles, both units. + + BOTH UNITS AT BOTH POLES even though the south's extent is `lApc` alone, because the `ground` + percentile excludes `Apu` everywhere and would otherwise take dusty layered deposits as bare + ground in the one hemisphere where they cover most of the disc. + """ + bounds = (-grid.edge_m, -grid.edge_m, grid.edge_m, grid.edge_m) + masks: dict[str, np.ndarray] = {} + for unit in ("lApc", "Apu"): + out = out_dir() / f"{unit.lower()}_{grid.name}.tif" + if not out.exists(): + mars_ice.burn_unit( + unit, grid.aeqd, bounds, grid.px, grid.px, + projected=out_dir() / f"{unit.lower()}_{grid.name}_aeqd.geojson", out=out, + must_draw=f"{unit} must reach the {grid.name} cap disc at edge_lat " + f"{grid.edge_lat}") + with rasterio.open(out) as dataset: + masks[unit] = dataset.read(1).astype(bool) + return masks + + +def measure(grid: cap_render.CapGrid, from_pipeline: bool = False) -> tuple[float, float]: + """This pole's `(ground, cap)` levels, plus the numbers that say whether they are sane.""" + luma, valid = viking_on_cap(grid, from_pipeline) + masks = unit_masks(grid) + lapc, apu = masks["lApc"], masks["Apu"] + + ground_pixels = valid & ~lapc & ~apu + cap_pixels = valid & lapc + ground_level = float(np.percentile(luma[ground_pixels], 50)) + cap_level = float(np.percentile(luma[cap_pixels], 50)) + + print(f"\n{grid.name.upper()} (edge_lat {grid.edge_lat}, {grid.px} px, " + f"{cap_render.cap_ground_metres_per_px(grid):.2f} ground m/px)") + print(f" valid {100 * valid.mean():5.2f}% lApc {100 * lapc.mean():5.2f}% " + f"Apu {100 * apu.mean():5.2f}% ground {100 * ground_pixels.mean():5.2f}%") + print(f" ground p50 {ground_level:8.2f} -> alpha 0") + print(f" cap p50 {cap_level:8.2f} -> alpha 1 (separation " + f"{cap_level - ground_level:.2f} DN)") + + alpha = mars_ice.albedo_alpha(luma, (ground_level, cap_level), nodata=-1.0) + alpha[~valid] = 0.0 + extent = (lapc | apu) if grid.name == "north" else lapc + print(f" ground above half alpha: {100 * (alpha[ground_pixels] > 0.5).mean():5.2f}%" + f" mean alpha inside extent: {alpha[extent].mean():.3f}") + for label, region in (("lApc", lapc), ("Apu only", apu & ~lapc)): + if region.any(): + print(f" {label:>9}: mean alpha {alpha[region].mean():.3f}") + return ground_level, cap_level + + +def sweep(grid: cap_render.CapGrid, from_pipeline: bool = False) -> None: + """Mean alpha over `lApc` against the percentile chosen for the alpha-1 level, WITH ITS COST. + + THE TWO COLUMNS MOVE TOGETHER AND THAT IS THE WHOLE POINT. A lower percentile shrinks the + denominator, so the cap saturates — and the same change pushes bare ground up the ramp. + Reporting saturation without the spill beside it would make the choice look free, which is how a + look gets picked on one number. + + `ground` stays the p50 throughout: it is the level read from ice-free ground and nothing about + the alpha-1 choice argues with it. + """ + luma, valid = viking_on_cap(grid, from_pipeline) + masks = unit_masks(grid) + lapc, apu = masks["lApc"], masks["Apu"] + ground_pixels = valid & ~lapc & ~apu + cap_pixels = valid & lapc + ground_level = float(np.percentile(luma[ground_pixels], 50)) + extent = (lapc | apu) if grid.name == "north" else lapc + + print(f"\n{grid.name.upper()} — alpha-1 level against saturation and spill " + f"(ground p50 {ground_level:.2f} fixed)") + print(" pct cap DN mean a(lApc) mean a(extent) ground>half") + for percentile in (5, 10, 15, 20, 25, 30, 40, 50): + cap_level = float(np.percentile(luma[cap_pixels], percentile)) + if cap_level <= ground_level: + print(f" {percentile:3d} {cap_level:6.2f} (below ground level — degenerate)") + continue + alpha = mars_ice.albedo_alpha(luma, (ground_level, cap_level), nodata=-1.0) + alpha[~valid] = 0.0 + print(f" {percentile:3d} {cap_level:6.2f} {alpha[lapc].mean():10.3f} " + f"{alpha[extent].mean():12.3f} {100 * (alpha[ground_pixels] > 0.5).mean():8.2f}%") + + +def compare_against_the_pipeline() -> int: + """Measure the levels BOTH ways and refuse any disagreement at the pinned precision. + + THE ONE CHECK THAT SAYS THE SHIPPED FIELD IS THE MEASURED FIELD. `ALPHA_LEVELS` is four + percentiles of a particular set of numbers; `render/viking_luma` builds those numbers on a grid + stated rather than inherited, so nothing but running both and subtracting can establish that the + stage ships what the instrument measured. + """ + mars = bodies.get("mars") + worst = 0.0 + for grid in (cap_render.north_grid(mars), cap_render.south_grid(mars)): + from_band = measure(grid, from_pipeline=False) + from_stage = measure(grid, from_pipeline=True) + pinned = mars_ice.ALPHA_LEVELS[grid.name] + print(f"\n{grid.name.upper()} pinned {pinned}") + print(f" viking_luma {tuple(round(value, 2) for value in from_stage)} <- the authority") + print(f" rebuilt band {tuple(round(value, 2) for value in from_band)} " + f"(prototype grid, informational)") + for index, name in enumerate(("ground", "cap")): + worst = max(worst, abs(from_stage[index] - pinned[index])) + if round(from_stage[index], 2) != pinned[index]: + print(f" MISMATCH {name}: stage {from_stage[index]:.4f} vs pinned {pinned[index]}") + print(f" band-vs-stage {name}: {from_band[index] - from_stage[index]:+.4f} DN") + print(f"\nworst deviation of the SHIPPED field from the pinned levels: {worst:.6f} DN") + return 0 if worst < 0.005 else 1 + + +def main() -> int: + out_dir().mkdir(parents=True, exist_ok=True) + if "--compare" in sys.argv: + return compare_against_the_pipeline() + + mars = bodies.get("mars") + from_pipeline = "--pipeline" in sys.argv + levels = {} + for grid in (cap_render.north_grid(mars), cap_render.south_grid(mars)): + levels[grid.name] = measure(grid, from_pipeline) + for grid in (cap_render.north_grid(mars), cap_render.south_grid(mars)): + sweep(grid, from_pipeline) + + print("\nALPHA_LEVELS: dict[str, tuple[float, float]] = {") + for pole, (ground_level, cap_level) in levels.items(): + print(f' "{pole}": ({ground_level:.2f}, {cap_level:.2f}),') + print("}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/prose_report.py b/scripts/prose_report.py new file mode 100644 index 0000000..83fea69 --- /dev/null +++ b/scripts/prose_report.py @@ -0,0 +1,94 @@ +"""Where the prose sits, per module and per documented thing. + +AN INSTRUMENT, NOT A GATE, and that distinction is the whole design. A ratio threshold would fail on +modules that are legitimately almost all prose — a constants file is one value and the paragraph that +makes it safe to change — and a guard that cries wolf gets ignored and then deleted, which +`scripts/sabotage.py` already records as this repo's own experience. So this prints and returns 0. + +WHAT IT IS ACTUALLY FOR. A per-comment rule cannot see a cross-comment property: the same concept +re-established at every field that touches it leaves each individual comment correct and the file +unreadable. `--by-target` is the view that shows it, because one concept explained four times reads +as four separate well-documented fields until you line them up. + + python -m scripts.prose_report # per module, worst first + python -m scripts.prose_report --by-target pipeline/bodies.py +""" + +import argparse +import io +import tokenize +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def split_prose(source: str) -> tuple[int, int]: + """(prose lines, code lines) for one module. Docstrings count as prose, blanks count as neither.""" + prose = 0 + for token in tokenize.generate_tokens(io.StringIO(source).readline): + if token.type == tokenize.COMMENT: + prose += 1 + elif token.type == tokenize.STRING and token.line.strip().startswith(('"""', "'''", 'r"""')): + prose += token.string.count("\n") + 1 + live = len([line for line in source.splitlines() if line.strip()]) + return prose, max(live - prose, 1) + + +def by_target(source: str) -> list[tuple[int, str]]: + """Prose lines attributed to the code line each block precedes — the cross-comment view. + + Attribution is positional rather than semantic: a comment block belongs to the next line of real + code under it, which is how Python's own `#:` convention reads. Good enough to show one concept + told at four sites, which is the only question this view is asked. + """ + blocks: list[tuple[int, str]] = [] + pending = 0 + for line in source.splitlines(): + stripped = line.strip() + if not stripped: + continue + if stripped.startswith(("#", '"""', "'''")): + pending += 1 + continue + blocks.append((pending, stripped[:70])) + pending = 0 + return sorted(blocks, reverse=True) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--by-target", metavar="MODULE", + help="attribute one module's prose to what it documents") + parser.add_argument("--package", default="pipeline", help="package to sweep (default: pipeline)") + parser.add_argument("--top", type=int, default=12, help="rows to print (default: 12)") + args = parser.parse_args() + + if args.by_target: + path = (ROOT / args.by_target).resolve() + rows = by_target(path.read_text(encoding="utf-8")) + print(f"{path.relative_to(ROOT)} — prose lines per documented thing\n") + for count, code in rows[:args.top]: + if count: + print(f" {count:4d} {code}") + print("\n Same concept appearing against several targets is one concept without a home.") + return 0 + + total_prose = total_code = 0 + rows = [] + for path in sorted((ROOT / args.package).rglob("*.py")): + prose, code = split_prose(path.read_text(encoding="utf-8")) + total_prose += prose + total_code += code + rows.append((prose / code, prose, code, path.relative_to(ROOT))) + + print(f"{args.package}/: {total_prose} prose / {total_code} code " + f"= {total_prose / total_code:.2f} lines of English per line of Python\n") + for ratio, prose, code, relative in sorted(rows, reverse=True)[:args.top]: + print(f" {ratio:6.2f}x {prose:5d} prose /{code:5d} code {relative}") + print("\n A high ratio is a QUESTION, not a verdict: ask whether one concept is being" + "\n re-established at each site, or whether the file is genuinely that dense.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/sabotage.py b/scripts/sabotage.py index b74709f..43c3f9f 100644 --- a/scripts/sabotage.py +++ b/scripts/sabotage.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Mutation harness — break each guard's subject and confirm the guard fails. A guard that passes whether or not its subject is present is decoration, and this repo has shipped @@ -53,14 +52,13 @@ needle that a refactor moved is a 0.1 s pytest failure rather than a shrugged-off SKIP 5 min in. """ -from __future__ import annotations - import argparse import os import re import shutil import subprocess import sys +from collections.abc import Sequence from pathlib import Path from typing import NamedTuple @@ -75,6 +73,11 @@ MUTABLE_ROOTS = ( "web/src", "web/worker", + # Joined when the deploy preflight started ENUMERATING the archive registry rather than naming + # two keys out of the Worker's config. That check is the only thing between a re-cut and a site + # whose every tile 404s, and it cannot run in CI (it needs R2) — so its shape is what gets + # mutation-tested, and the mutations have to be able to reach it. + "web/scripts", "scripts", "PROCESS.md", "web/vitest.config.ts", @@ -89,16 +92,101 @@ # carries an exemption list, and a skip-list nobody can mutate is a skip-list nobody can prove # is still doing anything — which is the failure mode it exists to prevent. "tests/test_hero_variants.py", + # The bulk-edit guard, which is a PARSER and therefore both guard and subject. Its checks read + # every tracked text file, so it is the one place where a wrong answer is spread across the whole + # repo and blamed on whichever file happens to expose it — the reason it earns mutation coverage + # is that its failures name the wrong line by construction. + "tests/test_repo_integrity.py", # Joined for the body registry. Its whole safety story is a set of bridge tests holding the # duplicated constants (`EARTH_RADIUS` twice, `EXAGGERATION` once) to the registry's copy until # each original is deleted — and a bridge nobody can mutate is a bridge nobody can prove is # load-bearing. The look package as a whole, because the parameterisation touches all of it. "pipeline/bodies.py", "pipeline/render", + # Joined with the layer table, which took the body-half gate and the stage split out of the + # planet shader. Both of its guards are invisible while Earth is the only body that declares a + # layer: "ask the body before the disk" passes either way on a box holding Earth's files, and a + # wrong stage column just moves a key in a recipe nobody re-reads. Neither has an output to + # inspect, so mutation is the only proof they still fire. + "pipeline/layers.py", + # Joined with the reproject-then-burn owner, whose whole subject is a GDAL command that succeeds + # while producing nothing. Earth's one caller draws a coastline that is obviously there, so every + # guard over it passes on this box whether it fires or not; the body it protects is the one that + # burns a mapped unit, and that body has no output to inspect yet. + "pipeline/vector_raster.py", + # Joined when a stale `.done` marker was found vouching for bytes it never saw. Every stage in + # the pipeline asks this module whether to run, so a weakened answer here is silent everywhere + # at once and shows up as a rebuilt planet that quietly kept one empty layer. There is no output + # to inspect for a stage that DIDN'T run, which is the whole reason mutation is the only proof. + "pipeline/freshness.py", + # Joined with the antimeridian fill, whose every wrong answer is a PLAUSIBLE one. Copying a + # neighbour instead of interpolating across the seam is within noise of correct on real terrain + # (the two differ by a median of 2.2 m), and the refusal that keeps it from smoothing genuine + # source gaps produces no output at all while it is working. Neither has an artifact to inspect. + "pipeline/wrap_seam.py", # Joined with the required `--body`. The planet entry points are where a silent Earth assumption # would be reintroduced, and it is invisible while Earth is the only body — so the guards against # it are worth exactly as much as the proof that they still fire. "pipeline/tile", + # Joined with the Mars DEM recipe, and for the sharpest version of the same argument: an + # acquisition guard runs ONCE, against a server, before ~10.6 GiB lands. It cannot be exercised + # by any pipeline run, it has no output to inspect, and by the time it would have mattered the + # wrong edition is already on disk. Mutation is the only proof available that it still fires. + "pipeline/acquire", + # Joined with the HTTP identity, whose failure mode is the least visible in the package: every + # acquisition test mocks the network, so a missing header is invisible to the suite, and every + # host WITHOUT bot protection serves us anyway, so it is invisible to a run. It surfaced as a + # 403 on a 10.6 GiB download and could only have surfaced that way. + "pipeline/fetch.py", + # Joined with the planet seam, the one contract two different tiers write and two more read. Its + # whole job is to keep three situations apart — no mask, no producer, a crashed producer — and + # every way of collapsing them leaves a module that imports and answers. There is no output to + # inspect either: the failure is a planet that shades from half a fusion and reports DONE. + "pipeline/planet_seam.py", + "pipeline/fuse", + # Joined with the one home for Natural Earth, and for a reason particular to this seam: every + # way of breaking it is invisible on a developer box, where `MAPS_DATA` is unset and the two + # roots resolve to the same directory. Its guards therefore never fire during ordinary work, + # and a guard that never fires is one nobody can tell apart from a guard that cannot. + "pipeline/naturalearth.py", + # Joined with the hero pipeline's paths. This is the one tier whose stages talk to each other in + # SHELL STRINGS rather than in Python values, so its wiring is invisible to both the type + # checker and the import probe — the only proof that anything watches it is breaking it. + "pipeline/frame", + "pipeline/batch.py", + # A second single test file, on `tests/test_hero_variants.py`'s principle. The store probe's + # PREDICATE lives in the test module, so the predicate is both guard and subject, and there is + # nowhere else to break it. It has now been wrong in both directions — blind to a spelling + # three times, then reporting sixteen correct constants as offenders because CI checks out + # under a directory named `work` — and neither direction is visible from this machine. + "tests/test_paths.py", + # Same principle as the two test files above: the import scan IS the guard, so the + # only way to prove it still sees anything is to narrow it and watch something fail. + "tests/test_fetch.py", + # Joined with the look seam, on the principle the test files above share: the ramp-bypass sweep + # IS the guard, so the only way to prove it still reads the shading path is to narrow its walk + # and watch something fail. Narrowing is also the realistic mistake — the scan sits in a + # palette test, and scoping it to the render package reads as tidying rather than gutting. + "tests/test_palette.py", + # Joined with the output licence, which is stated in four files and was checked in none. Two + # roots, for two different reasons. `LICENSE` because it is a mutation TARGET no other root + # reaches — it has no extension, sits outside every package, and is the copy a reader treats as + # authoritative. `tests/test_attributions.py` because its sweep is guard and subject both: the + # suffix set decides which files are read at all, so narrowing it silences the check from the + # inside while every assertion still passes. + "LICENSE", + "tests/test_attributions.py", + # The fourth on that principle, and the sharpest: the cross-language parity guard has to PARSE + # `web/src/lib/bodies.ts` to compare it, so its brace counter is both guard and subject. A + # counter that stops counting still returns blocks, still finds a body, and reads the wrong + # planet's answer — a failure with no error and no output to inspect. + "tests/test_bodies.py", + # Joined when the pass's memory cap stopped being one number. The harness is a SHELL SCRIPT, so + # neither pyright nor ruff reads it, and every way of reverting it leaves a script that runs and + # prints a plausible preflight line — the cap is simply the wrong planet's. Its two failure + # directions are also both expensive and neither is a crash at the edit: too high refuses a pass + # the box could have run, too low OOM-kills hours in. + "pipeline/profile", ) # Set for the duration of one case, so the backup THIS run is holding does not trip the leftover @@ -179,8 +267,11 @@ class Sabotage(NamedTuple): suite='python', label='delete a closing */ so a doc comment swallows an export', path='web/src/lib/terrainSource.ts', - needle=' * suggests, which is worth knowing before anyone cuts a z9 that could never load. */', - replacement=' * suggests, which is worth knowing before anyone cuts a z9 that could never load.', + # Re-anchored: this block gained a closing paragraph, so the `*/` moved off the line the + # needle named. The case is about the LAST line of the comment above an export, which is a + # position rather than a sentence — so it re-anchors whenever that block is edited. + needle=' * made to fail. Threading the archive is what makes it checkable before a second body has one. */', + replacement=' * made to fail. Threading the archive is what makes it checkable before a second body has one.', guard='test_no_block_comment_swallows_a_declaration', ), Sabotage( @@ -215,6 +306,168 @@ class Sabotage(NamedTuple): replacement='// See ' + 'HISTORY' + ' \u00a7 something.\n// The tile base is the one', guard='test_no_reference_to_a_file_a_clone_will_not_have', ), + # The scratch-directory half of the same guard, and it needs its own case because it is a + # separate alternation with a separate way of being wrong: the pattern requires a TRAILING SLASH + # so a bare directory name can still be passed as an argument, and a needle without one would + # pass while the real citation form went uncaught. + Sabotage( + suite='python', + label='cite a prototype script from the module whose constants it owns', + path='pipeline/render/mars_ice.py', + needle='FEATHER_KM = 5.0', + replacement='#: Reproduced by ' + '_ice_ab' + '/scripts/feather.py\nFEATHER_KM = 5.0', + guard='test_no_reference_to_a_file_a_clone_will_not_have', + ), + # --- The ice white becomes the layer's own ------------------------------------------------------ + # Every one of these leaves a cap that opens, a recipe that parses and a pass that exits 0. Two + # of them are only wrong on a planet nobody looks at, and one is only wrong in a branch no + # shipping body reaches — which is precisely why they are mutations rather than review comments. + Sabotage( + suite='python', + label="the union stops following the winner, so one layer's colour paints another's pixels", + path='pipeline/tile/shade_planet.py', + needle=' wins = (incoming_alpha > current_alpha)[None]', + replacement=' wins = np.ones(incoming_alpha.shape, dtype=bool)[None]', + guard='test_the_layer_with_the_HIGHER_alpha_supplies_each_pixels_colour', + ), + Sabotage( + suite='python', + label='Mars paints both poles in its northern white, losing the measurement entirely', + path='pipeline/render/layer_producers.py', + needle=' northern = np.asarray(window.latitude) >= 0.0', + replacement=' northern = np.ones(np.asarray(window.latitude).shape, dtype=bool)', + guard='test_mars_paints_its_two_poles_in_DIFFERENT_whites', + ), + Sabotage( + suite='python', + label="a body's whites paint pixels and reach no recipe, so a re-tune looks fresh", + path='pipeline/render/layer_producers.py', + needle='def _mars_ice_recipe() -> dict[str, Any]:\n', + replacement='def _mars_ice_recipe() -> dict[str, Any]:\n return {}\n', + guard='test_every_declared_white_reaches_that_bodys_recipe', + ), + Sabotage( + suite='python', + label='the cap and the tiles disagree about one ice colour across the crossfade', + path='pipeline/render/perennial_ice.py', + needle=' paint=lambda: palette.MARS_ICE_WHITE["north"]),', + replacement=' paint=lambda: palette.MARS_ICE_WHITE["south"]),', + guard='test_each_body_paints_one_pole_the_same_in_both_tiers', + ), + Sabotage( + suite='python', + label="Earth's two union layers declare different whites, so one wins by table order", + path='pipeline/render/layer_producers.py', + needle=' build=_build_glaciers, contribution=_earth_glaciers, paint=_earth_white,', + replacement=' build=_build_glaciers, contribution=_earth_glaciers,\n' + ' paint=lambda _window: ((9, 9, 9), (1, 1, 1)),', + guard='test_earths_two_union_layers_declare_the_SAME_white', + ), + # --- The interpolated pole: corrections that look fine in every artifact ------------------------- + # Each of these still renders a cap that opens, feathers and ships. The failure is either an edit + # to ground the altimeter DID measure, or a correction quietly not applied — neither of which + # announces itself in a WebP. + Sabotage( + suite='python', + label='Earth is smoothed too, as though every pole had a data gap', + path='pipeline/tile/cap_render.py', + needle=' smooth = POLE_SMOOTH_BY_BODY.get(grid.body.name)\n if smooth is None:\n' + ' return heights', + replacement=' smooth = POLE_SMOOTH_BY_BODY.get(grid.body.name,\n' + ' PoleSmooth(87.1, 30.0, 4.0, 40.0))', + guard='test_a_body_whose_altimeter_reached_its_pole_is_left_alone', + ), + Sabotage( + suite='python', + label='the boundary is pinned to the disc instead of to the parallel', + path='pipeline/tile/cap_render.py', + needle=' knee_px = (90.0 - smooth.interpolated_lat) / (90.0 - abs(grid.edge_lat)) ' + '* (grid.px / 2.0)', + replacement=' knee_px = 0.29 * (grid.px / 2.0)', + guard='test_the_boundary_follows_the_edge_latitude', + ), + Sabotage( + suite='python', + label='the correction runs everywhere rather than only over the gap', + path='pipeline/tile/cap_render.py', + needle=' t = np.clip((knee_px + taper_px / 2.0 - radius) / taper_px, 0.0, 1.0)', + replacement=' t = np.ones_like(radius)', + guard='test_nothing_beyond_the_boundary_is_touched', + ), + Sabotage( + suite='python', + label='the smoothing stops reaching the freshness recipe', + path='pipeline/tile/cap_render.py', + needle=' fields["pole_smooth"] = asdict(smooth)', + replacement=' pass', + guard='test_only_a_body_with_a_gap_records_one', + ), + Sabotage( + suite='python', + label='the elevation texture re-spells the nodata rule and skips the correction', + path='pipeline/tile/cap_render.py', + needle=' heights = cap_heights(grid, raw)\n\n factor = CAP_PX // CAP_ELEV_PX', + replacement=' heights = np.where(raw < -1e4, 0.0, raw).astype(np.float32)\n\n' + ' factor = CAP_PX // CAP_ELEV_PX', + guard='test_the_nodata_convention_has_exactly_one_owner', + ), + + # --- Mars's ice registration: the guards with no output to inspect ------------------------------- + # None of these five has an artifact a reader could check. Mars's ice is a band of a few degrees + # at one pole, and every one of these mutations leaves a raster that opens, a recipe that parses + # and a pass that exits 0 — which is exactly the shape mutation exists for. + Sabotage( + suite='python', + label='a build-time constant stops reaching the freshness gate', + path='pipeline/tile/shade_planet.py', + needle=' tunables = producer.build_recipe()', + replacement=' tunables = {}', + guard='test_a_changed_build_constant_rebuilds_the_raster', + ), + Sabotage( + suite='python', + label='the two poles are graded against each other\'s levels', + path='pipeline/render/mars_ice.py', + needle=' albedo_alpha(field, ALPHA_LEVELS["north"], nodata),\n' + ' albedo_alpha(field, ALPHA_LEVELS["south"], nodata))', + replacement=' albedo_alpha(field, ALPHA_LEVELS["south"], nodata),\n' + ' albedo_alpha(field, ALPHA_LEVELS["north"], nodata))', + guard='test_each_pole_is_graded_against_its_own_levels', + ), + Sabotage( + suite='python', + label='a unit span stops being taken per hemisphere, so one band swallows the planet', + path='pipeline/render/mars_ice.py', + needle=' if (value >= 0.0) == northern]', + replacement=' if True]', + guard='test_a_span_is_taken_PER_HEMISPHERE', + ), + Sabotage( + suite='python', + label='the ice band loses its pad, clipping the feather at the band edge', + path='pipeline/render/mars_ice.py', + needle=' row0, row1 = max(0, min(rows) - pad_rows), min(height, max(rows) + pad_rows)', + replacement=' row0, row1 = max(0, min(rows)), min(height, max(rows))', + guard='test_the_pad_widens_the_band_on_both_sides', + ), + Sabotage( + suite='python', + label="Mars's cap grades both poles against the north's levels", + path='pipeline/render/perennial_ice.py', + needle=' graded = mars_ice.albedo_alpha(field, mars_ice.ALPHA_LEVELS[pole], ' + 'viking_luma.NODATA)', + replacement=' graded = mars_ice.albedo_alpha(field, mars_ice.ALPHA_LEVELS["north"], ' + 'viking_luma.NODATA)', + guard='test_each_pole_grades_against_its_OWN_levels', + ), + Sabotage( + suite='python', + label='the alpha levels drop out of the build recipe, so a re-tune leaves a stale raster', + path='pipeline/render/layer_producers.py', + needle=' "mars_alpha_levels": {pole: list(levels)', + replacement=' "mars_alpha_levels_unread": {pole: list(levels)', + guard='test_mars_declares_the_two_constants_its_build_bakes_in', + ), # --- span attribution: the three ways it could quietly start lying ------------------------------- # All three mutations leave a report that still RENDERS and still reads plausible, which is the # only reason they are worth a case: a broken attribution does not throw, it just blames the @@ -248,7 +501,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='cap ordering: put applyCacheCap back BEFORE setTerrain', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' map.setTerrain({ source: TERRAIN_SOURCE, exaggeration: exaggerationFor(map.getZoom()) });\n applyCacheCap();', replacement=' applyCacheCap();\n map.setTerrain({ source: TERRAIN_SOURCE, exaggeration: exaggerationFor(map.getZoom()) });', guard='caps the DEM cache AFTER setTerrain, which is what builds the manager it lands on', @@ -256,7 +509,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='recovery watch no longer re-adds the polar caps', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' reassertPolarCaps();\n', replacement='', guard='puts back what a restore silently drops, once the map reads healthy', @@ -264,7 +517,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='recovery watch no longer re-asserts the DEM bound', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' reassertTerrainBound();\n', replacement='', guard='puts back what a restore silently drops, once the map reads healthy', @@ -272,7 +525,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='loss handler stops starting the watch (back to event-driven recovery)', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' startRecoveryWatch(performance.now() + GL_RESTORE_GRACE_MS);', replacement='', guard='starts the recovery watch from the LOSS, because the restore event may never fire', @@ -280,7 +533,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='loss handler stops charging the recurrence budget', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' if (recoveryVerdict(chargedLosses) === "give-up") {', replacement=' if (false) {', guard='bounds recovery by recurrence rather than trying to read a cause that does not exist', @@ -288,7 +541,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='cap re-assertion reports without repairing', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' applyCacheCap();\n console.info(`[terrain] DEM cache cap was not in force', replacement=' console.info(`[terrain] DEM cache cap was not in force', guard='REPAIRS a dropped cap before reporting it, and lets the next idle be the judge', @@ -296,7 +549,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='cap re-assertion verifies its own write synchronously (the stale-oracle bug)', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' applyCacheCap();\n console.info(', replacement=' applyCacheCap();\n demCacheCapFault(map.style?.tileManagers?.[TERRAIN_SOURCE], intendedCacheSlots);\n console.info(', guard='REPAIRS a dropped cap before reporting it, and lets the next idle be the judge', @@ -304,7 +557,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='polar cap re-add stops clearing the dead layers first', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' if (map.getLayer(layerId)) map.removeLayer(layerId);', replacement=' void layerId;', guard='re-adds the caps on recovery, from OUTSIDE style.load, because that ordering is too early', @@ -312,7 +565,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='restore handler touches the notice again (the original bug)', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' window.clearTimeout(restoreWatchdog);\n startRecoveryWatch(performance.now());', replacement=' window.clearTimeout(restoreWatchdog);\n glLostNotice?.setAttribute("hidden", "");\n startRecoveryWatch(performance.now());', guard='NEVER hides the notice on the restore event alone — this is the whole bug', @@ -336,11 +589,27 @@ class Sabotage(NamedTuple): # --- the z0 relief base pin (2026-07-29) --------------------------------------------------------- Sabotage( suite='web', + # Re-anchored when both source specs left the page for a module that a test can import. The + # mutation is the same one and it got MORE plausible in the move: the body's ceiling is now + # two lines above, so writing it here reads as removing an inconsistency. label='base source uncapped — maxzoom follows relief, losing the one-tile guarantee', - path='web/src/pages/earth.astro', + path='web/src/lib/reliefSources.ts', needle=' maxzoom: RELIEF_BASE_MAX_ZOOM,', - replacement=' maxzoom: RELIEF_MAX_ZOOM,', - guard='caps the base source at z0, because that is what makes it unmissable', + replacement=' maxzoom: archive.maxZoom,', + guard='caps the base source at z0 for every body, because that is what makes it unmissable', + ), + Sabotage( + suite='web', + # The defect this whole split exists to make impossible. The two bodies are cut to different + # ceilings, so Earth's numbers written out here — which is what the page held until the + # registry became the source of truth — make the shallower globe request levels that were + # never cut. Nothing errors: the address is refused without a storage read, so the tiles + # simply never arrive and the globe looks slow rather than wrong. + label='the relief source takes Earth\'s zoom range instead of the body\'s', + path='web/src/lib/reliefSources.ts', + needle=' minzoom: archive.minZoom,\n maxzoom: archive.maxZoom,', + replacement=' minzoom: 0,\n maxzoom: 8,', + guard='takes each body\'s own zoom range from the registry, never Earth\'s constants', ), Sabotage( suite='web', @@ -348,12 +617,12 @@ class Sabotage(NamedTuple): path='web/src/lib/reliefTiles.ts', needle='export const RELIEF_BASE_MAX_ZOOM = 0;', replacement='export const RELIEF_BASE_MAX_ZOOM = 1;', - guard='caps the base source at z0, because that is what makes it unmissable', + guard='caps the base source at z0 for every body, because that is what makes it unmissable', ), Sabotage( suite='web', label='base layer drawn OVER relief, hiding the real tiles', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' { id: "relief", type: "raster", source: "relief", paint: { "raster-fade-duration": 0 } },\n ],', replacement=' ],', guard='draws the base UNDER relief and OVER the background, or it is pointless', @@ -361,7 +630,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='base source registered but never added to the style', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle='sources: { relief: reliefSource, "relief-base": reliefBaseSource },', replacement='sources: { relief: reliefSource },', guard='draws the base UNDER relief and OVER the background, or it is pointless', @@ -369,16 +638,19 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='base source grows a second attribution, doubling the credit', - path='web/src/pages/earth.astro', - needle=' maxzoom: RELIEF_BASE_MAX_ZOOM,\n tileSize: 256,\n };', - replacement=' maxzoom: RELIEF_BASE_MAX_ZOOM,\n tileSize: 256,\n attribution: CREDITS,\n };', - guard='caps the base source at z0, because that is what makes it unmissable', + path='web/src/lib/reliefSources.ts', + needle=' maxzoom: RELIEF_BASE_MAX_ZOOM,\n tileSize: DECLARED_TILE_SIZE,\n };', + replacement=( + ' maxzoom: RELIEF_BASE_MAX_ZOOM,\n tileSize: DECLARED_TILE_SIZE,\n' + ' attribution: "one archive, credited twice",\n };' + ), + guard='carries no attribution, so one archive does not credit itself twice', ), # --- the terrain-retirement flag (2026-07-29) ---------------------------------------------------- Sabotage( suite='web', label='re-assertion stops honouring the retirement flag', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' reassertTerrainBound = () => {\n if (terrainRetired) return;', replacement=' reassertTerrainBound = () => {\n if (false) return;', guard='goes quiet when the FPS ladder retires terrain, instead of crying wolf about the cap', @@ -386,7 +658,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='applyCacheCap stops honouring the retirement flag', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' const applyCacheCap = () => {\n if (terrainRetired) return;', replacement=' const applyCacheCap = () => {\n if (false) return;', guard='goes quiet when the FPS ladder retires terrain, instead of crying wolf about the cap', @@ -394,7 +666,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='flag raised AFTER the teardown, so an idle inside it still false-alarms', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' terrainRetired = true;\n map.setTerrain(null);', replacement=' map.setTerrain(null);\n terrainRetired = true;', guard='goes quiet when the FPS ladder retires terrain, instead of crying wolf about the cap', @@ -440,6 +712,64 @@ class Sabotage(NamedTuple): replacement='return { mobileClass: false, via: "pointer-coarse" };', guard='reports NO SIGNAL as its own state, instead of a desktop verdict from no evidence', ), + Sabotage( + suite='web', + # The fallback only runs where `createImageBitmap` rejects, which is no machine we own, so + # for its whole life the branch was covered by nothing. It is also the shape a tidy-up + # reverses without a thought — the assignment form is two lines shorter and looks equivalent. + label='the cap Image fallback goes back to assigning onload', + path='web/src/lib/polarCaps.ts', + needle=' image.addEventListener("load", () => resolve(image), { once: true });', + replacement=' image.onload = () => resolve(image);', + guard='still uploads when the engine has no createImageBitmap and the Image path takes over', + ), + # --- a body fetches its OWN polar-cap manifest, and Earth's URL is not the universal one --- + Sabotage( + suite='web', + # The literal this restores is not a typo — it is what shipped, correctly, for as long as + # one planet had caps. It is also the tidy a reader makes when a derived URL looks like + # ceremony around a constant. What makes it the worst case in this file is that it does not + # 404: Earth's prefix is empty, so the wrong body gets a 200, a valid manifest, and Earth's + # Arctic textures drawn over its pole at the right size. + label='the cap manifest goes back to the literal that is really Earth-only', + path='web/src/lib/polarCaps.ts', + needle='const response = await fetch(manifestUrl, { cache: "no-cache" });', + replacement='const response = await fetch("/caps/caps.json", { cache: "no-cache" });', + guard="fetches Mars's manifest for Mars, not the one Earth has always used", + ), + Sabotage( + suite='web', + # The prefix and the slug are the same word on every body that nests, so `slug` reads as + # the obvious simplification — and it is wrong on exactly one body, the one whose prefix is + # deliberately empty. Every Mars URL keeps working; every Earth URL moves. + label='the served prefix is replaced by the slug it happens to equal', + path='web/src/lib/assetBase.ts', + needle='["caps", BODIES[body].pathPrefix, "caps.json"]', + replacement='["caps", body, "caps.json"]', + guard="keeps Earth's URL byte-for-byte the one every warm browser cache already holds", + ), + Sabotage( + suite='web', + # Reverses the collapse, which is the half a reader trims when the filter looks redundant. + # Only Earth can see it: `/caps//caps.json` is a path no server was told to write. + label='the empty prefix stops collapsing and doubles the separator', + path='web/src/lib/assetBase.ts', + needle='.filter(Boolean).join("/")}`;', + replacement='.join("/")}`;', + guard="keeps Earth's URL byte-for-byte the one every warm browser cache already holds", + ), + Sabotage( + suite='python', + # The cross-language half, and a PYTHON case over a web file for the reason the cap-flag + # case above gives: the pipeline is what WRITES the files, so the browser's prefix is only + # ever the second half of that fact. A drift here cannot be seen by any type, any linter, or + # any page that only ever loads Earth. + label="a body's served prefix drifts to Earth's empty one", + path='web/src/lib/bodies.ts', + needle=' pathPrefix: "mars",', + replacement=' pathPrefix: "",', + guard='test_the_two_registries_agree_on_where_a_body_nests_its_served_assets', + ), Sabotage( suite='web', label='the collapsed view grows past a phone corner', @@ -480,6 +810,124 @@ class Sabotage(NamedTuple): replacement='return "the context came back but MapLibre never rebuilt the style";', guard='reports the most fundamental failure first', ), + + # --- the arm a capture belongs to, and the seam that drives it ---------------------------------- + # Every case here defends the same property: a capture must be able to say which ARM produced it. + # The instrument was already strict about provenance and a measurement went around it anyway, so + # what these protect is the path that makes going around it unnecessary. + Sabotage( + suite='web', + label='the origin records flag keys again, so two arms of one sweep read identically', + path='web/src/lib/perf/perfSnapshot.ts', + needle=' else for (const value of values) described.push(`${key}=${value}`);', + replacement=' else described.push(key);', + guard='separates two arms that differ only in what a flag is SET TO', + ), + Sabotage( + suite='web', + label='a blank arm becomes a name, so two unnamed captures collide', + path='web/src/lib/perf/perfOverlay.ts', + needle=' return arm === null || arm.trim() === "" ? undefined : arm;', + replacement=' return arm === null ? undefined : arm;', + guard='treats a blank arm as unnamed rather than as a name', + ), + Sabotage( + suite='web', + label='the export stops carrying the arm, so every capture is a bare timestamp', + path='web/src/lib/perf/perfOverlay.ts', + needle=' options.arm === undefined ? path : `${path}?arm=${encodeURIComponent(options.arm)}`;', + replacement=' path;', + guard='names the capture when an arm is given', + ), + Sabotage( + suite='web', + label='the arm slug admits a path separator, so a label can escape the capture directory', + path='web/src/lib/perfCaptureName.ts', + needle=' .replace(/[^a-z0-9]+/g, "-")', + replacement=' .replace(/[^a-z0-9./]+/g, "-")', + guard='cannot emit a path separator or a dot, so traversal has nothing to work with', + ), + Sabotage( + suite='web', + label='the capture is named arm-first, splitting the one run whose arms get compared', + path='web/src/lib/perfCaptureName.ts', + needle=' return slug === null ? `${stamp}.json` : `${stamp}-${slug}.json`;', + replacement=' return slug === null ? `${stamp}.json` : `${slug}-${stamp}.json`;', + guard="puts the timestamp first, so one run's arms sort adjacent", + ), + # The seam duplicate that actually shipped, replayed under a name the guard has never seen. The + # previous version of that guard named ONE handle and so could not have caught this at all. + Sabotage( + suite='web', + label='the page hands the live map to a global again, under a brand-new name', + path='web/src/components/Globe.astro', + needle=' // The scripted-diagnosis seam is NOT here.', + replacement=' window.debugMap = map;\n // The scripted-diagnosis seam is NOT here.', + guard='is not also written from the page, where nothing structural would gate it', + ), + + # --- the arm flags, whose whole failure mode is being ignored quietly --------------------------- + Sabotage( + suite='web', + label='arm flags stop needing ?perf, so a pasted link reconfigures a stranger', + path='web/src/lib/perfArms.ts', + needle=' return params.has("perf");', + replacement=' return true;', + guard='changes nothing on a production URL, so a pasted link cannot reconfigure a stranger', + ), + Sabotage( + suite='web', + label='?lod falls back to a default instead of refusing, so a run measures the wrong arm', + path='web/src/lib/perfArms.ts', + needle=' if (!Number.isFinite(value)) return null;', + replacement=' if (!Number.isFinite(value)) return 9.314;', + guard='is null on anything doubtful rather than falling back to the default', + ), + Sabotage( + suite='web', + label='?refresh takes a number, so refresh=2 rounds into a silent arm', + path='web/src/lib/perfArms.ts', + needle=' return mode === "on" ? true : mode === "off" ? false : null;', + replacement=' return mode === "on" || mode === "1" ? true : mode === "off" || mode === "0" ? false : null;', + guard='takes named modes only, so a number cannot round into a silent arm', + ), + Sabotage( + suite='web', + label='an ignored arm flag stops complaining, so a typo reads as the default', + path='web/src/lib/perfArms.ts', + needle=' if (!params.has(flag) || honoured) return null;', + replacement=' if (true) return null;', + guard='says so rather than ignoring the flag in silence', + ), + # The defect that actually shipped: the complaint fired on PRESENCE, so a working ?lod=11 warned + # "not a value this flag takes" next to the line saying it applied. Every test covered a failure + # path and none covered success, so nothing was red. + Sabotage( + suite='web', + label='the complaint fires on presence again, crying wolf over every valid arm', + path='web/src/lib/perfArms.ts', + needle=' if (!params.has(flag) || honoured) return null;', + replacement=' if (!params.has(flag)) return null;', + guard='STAYS QUIET when the flag was honoured, which is the case that shipped broken', + ), + # The renderable count is the only term in the census that is a CAUSE, and both of its failure + # modes are silent: a missing count reads as "no terrain", and a present one crowds the row. + Sabotage( + suite='web', + label='an absent renderable count reads as terrain drawing nothing', + path='web/src/lib/rttPoolTrim.ts', + needle=' return Array.isArray(keys) ? keys.length : null;', + replacement=' return Array.isArray(keys) ? keys.length : 0;', + guard='is null, not 0, when there is nothing to read', + ), + Sabotage( + suite='web', + label='the renderable count is put back on the panel row, over the phone budget', + path='web/src/lib/rttPoolTrim.ts', + needle=' return `rtt ${stats.pooled} idle · ${stats.held} held · peak ${stats.peakTotal}`;', + replacement=' return `rtt ${stats.pooled} idle · ${stats.held} held · peak ${stats.peakTotal} · drawn ${stats.renderable}`;', + guard='leaves the renderable count off the row, whatever it reads', + ), Sabotage( suite='web', label='the loss line drops when the timestamp is missing', @@ -499,7 +947,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label="the ladder reads the DISPLAY ratio again, not the map's", - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' pixelRatioLowered,\n devicePixelRatio: map.getPixelRatio(),', replacement=' pixelRatioLowered,\n devicePixelRatio: window.devicePixelRatio || 1,', guard="feeds the ladder the MAP's ratio, never the display's", @@ -507,7 +955,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='the dead-globe notice sinks back under the perf panel', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' z-index: 50;', replacement=' z-index: 20;', guard='keeps the dead-globe notice above the ?perf panel', @@ -515,7 +963,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label="the perf report reads the DISPLAY ratio instead of the map's", - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' devicePixelRatio: map.getPixelRatio(),', replacement=' devicePixelRatio: window.devicePixelRatio || 1,', guard="reports the MAP's ratio in the perf snapshot too, not the display's", @@ -523,7 +971,7 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='the report probes capabilities per tick again — 13.3 WebGL contexts/second', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' signals: probedSignals,', replacement=' signals: probeSignals(),', guard="is never called from the ?perf overlay's per-tick path", @@ -531,37 +979,126 @@ class Sabotage(NamedTuple): Sabotage( suite='web', label='the tier is cached, so a mid-session quality change goes unreported', - path='web/src/pages/earth.astro', + path='web/src/components/Globe.astro', needle=' tier: decideGlobeTier(probedSignals, getQuality()),', replacement=' tier: bootTier,', guard='still tracks a quality change the user makes mid-session', ), Sabotage( suite='web', - # The chord comes back. `+` matches DOM order, so hiding fullscreen leaves its divider on - # the quiet button below, and the group's 999px radius clips it into a dark arc. - label='quiet mode keeps the divider of the button it hid', + # The chord comes back, by the route that replaced the old `border-top-width: 0` cancel: + # `+` matches DOM order, so a quiet button placed AFTER fullscreen keeps the divider of the + # button quiet mode hid, and the group's 999px radius clips it into a dark arc. + label='the quiet toggle stops leading its pill', + path='web/src/components/Globe.astro', + needle='joinRailGroup(map.getContainer(), ".maplibregl-ctrl-fullscreen", quietToggle.button, "start");', + replacement='joinRailGroup(map.getContainer(), ".maplibregl-ctrl-fullscreen", quietToggle.button);', + # The SOURCE scan, not the rendered one — and the split is structural rather than an + # attribution slip. `railIcons.browser.test` mounts its own markup, so no edit to the page + # can ever reach it; what proves that rendered assertion non-vacuous is its own positive + # control, which the divider-deletion case below exercises. Both halves of the page's + # ordering land on the one guard that reads the page. + guard='keeps the page building that order, which no stylesheet can state', + ), + Sabotage( + suite='web', + # The other half of the same defect, and the one no stylesheet can see. `addControl` appends, + # so adding the camera group first puts the frame group under a five-button pill that quiet + # mode hides with `visibility` — leaving the eye floating partway down an empty right edge. + label='the camera group goes back above the frame group', + path='web/src/components/Globe.astro', + needle=( + ' map.addControl(new maplibregl.FullscreenControl({ container: document.body }), "top-right");\n' + ' map.addControl(new maplibregl.NavigationControl({ visualizePitch: true }), "top-right");' + ), + replacement=( + ' map.addControl(new maplibregl.NavigationControl({ visualizePitch: true }), "top-right");\n' + ' map.addControl(new maplibregl.FullscreenControl({ container: document.body }), "top-right");' + ), + guard='keeps the page building that order, which no stylesheet can state', + ), + Sabotage( + suite='web', + # The cure that flattens the rail. Deleting the divider outright also removes the chord, so + # the no-hairline assertion alone would pass — its positive control is what refuses this. + label='the rail loses the divider between its buttons', + path='web/src/styles/globe.css', + needle=' border-top: 1px solid var(--line);', + replacement=' border-top: 0;', + guard='grows the chord straight back if the group is reordered — the positive control', + ), + Sabotage( + suite='web', + # The iPhone-Safari half. `FullscreenControl` renders nothing where the Fullscreen API is + # absent, so "start" has to decide where the FALLBACK pill goes as well; appending it parks + # the eye below the camera group on exactly the devices this reorder was reported from. + label='a "start" placement stops reaching the group it had to create', + path='web/src/lib/railControls.ts', + needle=' if (placement === "start") container.prepend(group);\n else container.append(group);', + replacement=' container.append(group);', + guard='carries "start" to the fallback GROUP too, not just to the button', + ), + Sabotage( + suite='web', + # The rail goes back to MapLibre's own hardcoded margin, 9.2px above and outside the + # top-left row it is meant to line up with. Their rule is injected at RUNTIME, so an + # equal-specificity override loses on source order and nothing anywhere reports it. + label="the rail's inset loses to MapLibre's own control margin", path='web/src/styles/globe.css', - needle=' border-top-width: 0;', - replacement=' border-top-width: 1px;', - guard='cancels the hairline on the button after the hidden fullscreen control', + needle='.maplibregl-ctrl-top-right .maplibregl-ctrl.maplibregl-ctrl {', + replacement='.maplibregl-ctrl-top-right .maplibregl-ctrl {', + # The RENDERED assertion, not its positive control — the control cannot catch this by + # construction. It weakens the selector itself, so against an already-weak source its + # `replace` matches nothing and it goes on measuring MapLibre's 10px and passing. + guard='takes both offsets from the one token the top-left row uses', + ), + Sabotage( + suite='web', + # The band state read from one occupant instead of both. Every open runs the other's close + # first, so this leaves the class stuck off after a search hit opens a card — a phone whose + # gallery link is gone with nothing on screen to explain it. + label='the open-panel class is written from the card alone', + path='web/src/components/Globe.astro', + needle='const occupied = !panel.hidden || (searchPanel?.isOpen() ?? false);', + replacement='const occupied = !panel.hidden;', + guard="writes the class from BOTH occupants' state, not from whoever moved last", ), Sabotage( suite='web', - # The specificity, tidied away. `body.is-quiet .rg-ctrl-quiet` is the obvious way to write - # this cancel and it is (0,3,1) against a (0,4,2) divider — it loses, silently. - label="the divider cancel is rewritten without the specificity that makes it win", + # Renamed on the side that can be renamed. A stylesheet cannot import a constant, so the + # class exists twice — and a selector matching nothing is valid CSS that cascades quietly. + label='the open-panel class is renamed in the page but not the stylesheet', + path='web/src/components/Globe.astro', + needle='const PANEL_OPEN_CLASS = "panel-open";', + replacement='const PANEL_OPEN_CLASS = "band-occupied";', + guard='spells the class the same on both sides of a seam nothing can close', + ), + Sabotage( + suite='web', + # The credit "restored" while a panel covers its corner — the tidy-up that looks like a + # licence fix and leaves a 326px panel sitting on top of the ⓘ instead. + label='the credit is exempted from the row that yields to an open panel', path='web/src/styles/globe.css', - needle=' .maplibregl-ctrl-group.maplibregl-ctrl-group\n .maplibregl-ctrl-fullscreen\n + button {', - replacement=' .maplibregl-ctrl-group\n .maplibregl-ctrl-fullscreen\n + button {', - guard='keeps the cancel more specific than the divider it has to beat', + needle=' body.panel-open .globe-source,\n body.panel-open .chrome-credit.chrome-credit.maplibregl-ctrl {', + replacement=' body.panel-open .globe-source {', + guard='hides the credit with the links it sits beside, and says why in the same breath', + ), + Sabotage( + suite='web', + # An edge offset written as its own literal again, which is how the rail and the row drifted + # 9.2px apart with every rule individually correct. + label='a floating element goes back to its own copy of the inset', + path='web/src/components/Globe.astro', + needle=' .globe-chrome {\n position: fixed;\n top: var(--page-inset);', + replacement=' .globe-chrome {\n position: fixed;\n top: 1.2rem;', + guard='leaves no edge offset written as its own literal', ), Sabotage( suite='web', # The original defect restored: a side-effect import makes Vite hoist MapLibre's 70 KB # widget sheet into a render-blocking , in front of a paint that needs none of it. label="MapLibre's stylesheet goes back to blocking first paint", - path='web/src/pages/earth.astro', + path='web/src/components/MapStylesheet.astro', needle='import maplibreStylesheet from "maplibre-gl/dist/maplibre-gl.css?url";', replacement='import "maplibre-gl/dist/maplibre-gl.css";\nconst maplibreStylesheet = "";', guard='imports it for its URL, never for its side effect', @@ -571,11 +1108,53 @@ class Sabotage(NamedTuple): # The scripts-off hole. `onload` is an inline handler, so without the noscript twin a # visitor with JS disabled keeps media="print" forever and the controls render unstyled. label='the deferred stylesheet loses its noscript fallback', - path='web/src/pages/earth.astro', - needle='