Reconcile catalog API + validate grid redesign: Catalog fetch/shard-map split, odc.geo grids, signature-enforced runs

[espg]

Status — 2026-06-10 (updated)

This PR's scope grew into the full catalog-API reconcile (#24): a fetch/shard-map
split (Catalog/CMRSource/ShardMap), an odc.geo.GeoBox-backed RectilinearGrid,
grid.signature() enforcement at run time, a --config-driven catalog CLI, the spherely
S2 SpatialIndex backend (fork wheel hosted as a GitHub release asset), a consolidated
single build_layer.sh (pyproj/odc-geo in, earthaccess out) with x86 bumped to py3.12.

Done: the catalog hard-break (Phase 5.4 — build_catalog + the EPSG:3031/grid-driven
legacy paths removed) and the CHANGELOG (5.6). Spherely-vs-mortie equivalence (orig.
Phase 2) was validated via bench/drift_catalog.py: the new ShardMap.build reproduces
the pre-refactor catalogs 100% (0 omission / 0 commission) for both backends on
cycle-22 ATL06. CI: Lint / Tests (3.12 + 3.13) / Docs green; the Lambda layer builds at
~125 MB unzipped.

Remaining (the soft-deprecation culls + post-merge ops): process_morton_cell (5.1),
xdggs_zarr_template(n_parent_cells=) (5.2), the HEALPix dense layout (5.3 — biggest,
spans 5 files), and the cryocloud notebook ARN (5.5); plus live deploy (Phases 3–4) and
CFN teardown docs (Phase 6). Per the plan, merging first eases CloudFormation testing.

---

Plan: Validate the grid/catalog redesign, then cull deprecated code

Checklist

Phase 0 — integrate spherely

• Land general_grids (spherely) on main
• Rebase release-cleanup-testing on updated main
• Full env installed (uv sync + .[catalog,analysis,test])

Phase 1 — offline validation

• offline python -m pytest -q green (202 passed)
• python -m pytest -m slow green (2 passed)
• ruff check src/ tests/ clean
• dense/fullsphere equivalence + rectilinear + catalog + deprecation-warning tests pass

Phase 2 — catalog (spherely vs mortie)

• cat_spherely.json + cat_mortie.json built
• coverage diff within tolerance (per Spherical geometry backend and dispatch for build_catalog #19)
• --polygon / --bbox spot-checked

Phase 3 — local end-to-end (real ATL06)

• morton ./test_healpix.zarr (group 12) verified
• rectilinear ./test_rect.zarr (group rectilinear) verified
• HTTPS driver exercised

Phase 4 — deployment + Lambda

• backend stood up (stand_up.sh / deploy.sh)
• Lambda dispatch writes to S3; cost output sane
• teardown clean

Phase 5 — cull deprecated code (gated on Phases 1–4)

• remove process_morton_cell
• remove xdggs_zarr_template(n_parent_cells=...)
• remove HEALPix dense layout (default → fullsphere)
• remove catalog legacy paths (_build_catalog_healpix/_grid_driven + shapely dispatch)
• update cryocloud_example.ipynb (ARN + process_shard)
• update CHANGELOG.md [Unreleased]
• re-run Phase 1 green after each cull

Phase 6 — CFN teardown docs

• teardown section added (docs/deployment/lambda.md + README.md)

---

Context

This week's merged PRs reshaped zagg's core: #18 generalized "morton cells" →
arbitrary grids/shards (process_shard, OutputGrid protocol, HealpixGrid

• new RectilinearGrid), Spherical geometry backend and dispatch for build_catalog #19 added a spherely (S2) catalog backend with
  mortie fallback, and Enable user side infrastructure standup #22 added user-side standup. Several deprecated paths
  were left in place deliberately, to be removed after the replacements are
  verified. The goal here is a runbook you execute that (1) validates
  deployment, the new grid class (rectilinear / non-morton), the still-working
  morton workflow, and the new catalog module — combining steps where possible —
  and then (2) culls the deprecated code once validation is green.

Key state facts (verified this session)

• Spherely (Spherical geometry backend and dispatch for build_catalog #19) is on origin/general_grids, NOT on main. Current
  catalog.py on your branch is mortie-only. Per your decision we land
  spherely on main first, then rebase the working branch on it — so the
  "new catalog module" (spherely+mortie) is part of main and its legacy paths
  become cull-eligible on a clean base.
• HealpixGrid still defaults to layout="dense" (src/zagg/grids/healpix.py:48)
  while from_config defaults to fullsphere (src/zagg/grids/__init__.py:42)
  — an inconsistency to fix in the dense cull.
• The process-morton-cell → process-shard function-name rename is staged
  (uncommitted) on release-cleanup-testing.
• Tests: pytest (testpaths=tests, only marker slow, pytest-cov present).
• Decisions: full test tiers incl. live Lambda; cull all four deprecated areas
  plus the cryocloud notebook; CFN teardown = docs only.

---

Phase 0 — Land spherely on main, then rebase the working branch

1. Put spherely on main: land origin/general_grids onto main
   (preferred: open a PR general_grids → main and merge it; or merge
   locally) and push main (you've authorized pushing to main). Resolve any
   merge conflicts — likely confined to src/zagg/catalog.py and friends, since
   general_grids is the branch that changed them.
2. Commit the staged process-shard rename on release-cleanup-testing.
3. Rebase release-cleanup-testing onto the updated origin/main (now
   containing spherely). Expect conflicts where the rename meets the redesign
   edits — resolve favoring the rename + spherely:
   src/zagg/{catalog,processing,runner,__init__,schema,config}.py,
   src/zagg/grids/*, deployment/aws/lambda_handler.py, pyproject.toml,
   tests/*.
4. Install the full env: uv sync then uv pip install -e '.[catalog,analysis,test]'
   (the catalog extra brings spherely).
5. (Optional "mark" step, per your ask) add # TODO(cull #NN): remove after validation comments at each deprecated definition (process_morton_cell,
   dense branches, xdggs_zarr_template(n_parent_cells=...), _build_catalog_healpix/
   _build_catalog_grid_driven/shapely dispatch + legacy parent_order) and a
   top-cell note in notebooks/cryocloud_example.ipynb, so the cull targets are
   self-documenting before removal.

---

Phase 1 — Offline automated validation (no network)

This proves the deprecated paths still work and their replacements are
equivalent — the gate for culling.

uv run pytest -q                 # full suite
uv run pytest -q -m slow         # lambda build/size/version-lockstep
uv run ruff check src/ tests/

Confirm specifically:

• morton equivalence: tests/test_integration.py::test_dense_fullsphere_equivalence passes (dense removal is safe).
• rectilinear: tests/test_rectilinear.py + the rectilinear case in test_integration.py pass (non-morton grid emits/writes correctly).
• catalog (mocked): tests/test_catalog.py passes incl. the spherely path now present from general_grids.
• deprecations fire: test_grids.py::test_healpix_explicit_dense_warns, test_runner.py::test_dense_layout_emits_warning.

---

Phase 2 — Catalog module: spherely vs mortie (needs network: CMR)

Tests the new catalog and produces the catalogs reused downstream (combined
per your note).

# auto backend -> spherely (catalog extra installed)
uv run python -m zagg.catalog --cycle 22 --parent-order 6 --output cat_spherely.json
# force mortie fallback
uv run python -m zagg.catalog --cycle 22 --parent-order 6 --geometry-backend mortie --output cat_mortie.json

• Diff shard_keys + per-shard granule coverage between the two (reuse the
  existing bench/verify_spherely_cycle22.py; target the ~100% shard /
  <0.01% pair disagreement already reported in Spherical geometry backend and dispatch for build_catalog #19).
• Spot-check --polygon <geojson> and --bbox paths produce a valid catalog.

---

Phase 3 — Local end-to-end: morton + rectilinear (needs NASA Earthdata)

Validates process_shard against real ATL06 reads, both grid types.

# MORTON still works (fullsphere):
uv run python -m zagg --config atl06.yaml --catalog cat_spherely.json --store ./test_healpix.zarr --max-cells 3 --dry-run
uv run python -m zagg --config atl06.yaml --catalog cat_spherely.json --store ./test_healpix.zarr --max-cells 3

# RECTILINEAR (non-morton), polar config:
uv run python -m zagg --config atl06_polar.yaml --catalog cat_spherely.json --store ./test_rect.zarr --max-cells 3

• Open each Zarr (xarray/zarr) and confirm group path (12 vs rectilinear),
  array shapes, populated cells, and aggregation variables match config.
• Exercise the HTTPS driver once (--driver https) to validate s3→https
  URL rewriting from catalog metadata.

---

Phase 4 — Deployment + Lambda backend (needs AWS creds; small $ cost)

# Stand up backend (CFN work) OR maintainer deploy.sh:
OUTPUT_BUCKET=<your-bucket> bash deployment/aws/stand_up.sh
# Dispatch a few cells:
uv run python -m zagg --config atl06.yaml --catalog cat_spherely.json \
    --store s3://<your-bucket>/test.zarr --backend lambda --max-cells 5

• Confirm objects land in S3, check the printed gb_seconds/estimated_cost_usd.
• Teardown: aws cloudformation delete-stack --stack-name zagg-backend
  (after emptying the staged zips + bucket — see Phase 6).

---

Phase 5 — Cull deprecated code (gated on Phases 1–4 green)

One focused commit per item; re-run Phase 1 after each. Inventory with
exact anchors:

1. process_morton_cell — remove def src/zagg/processing.py:432-472;
   drop from src/zagg/__init__.py:29,52; update docs/index.md:72,80,
   docs/api/processing.md:7, docs/design/{architecture.md:117,schema.md:64};
   fix tests/test_lambda_build.py:39 import + tests/conftest.py:17 docstring.
2. xdggs_zarr_template(n_parent_cells=...) — remove the param/branch
   src/zagg/schema.py:43-97 (warn at :82); update tests/test_schema.py
   (e.g. test_array_shape_n_parent_cells) and tests/test_grids.py:239-247.
3. HEALPix dense layout — change default to fullsphere
   (src/zagg/grids/healpix.py:48); remove dense branches in healpix.py,
   grids/__init__.py:34-41, config.py:154-157,386, runner.py:30-39,105,259-269,
   schema.py; update deployment/aws/lambda_handler.py:87-98 (drop
   n_parent_cells/dense setup); retire dense tests + the now-trivial
   test_dense_fullsphere_equivalence; update test_grids.py::test_default_layout_is_dense.
4. Catalog legacy paths — remove _build_catalog_healpix +
   _build_catalog_grid_driven + geometry_backend in {"shapely","shapely-3031"}
   dispatch + the legacy parent_order= kwarg path in src/zagg/catalog.py
   (keep spherely + mortie); update tests/test_catalog.py accordingly.
5. cryocloud notebook — notebooks/cryocloud_example.ipynb: update the
   hardcoded ARN ...:function:process-morton-cell → process-shard and switch
   any process_morton_cell usage to process_shard.
6. Update CHANGELOG.md [Unreleased] with the removals.

---

Phase 6 — CFN teardown docs (docs only)

Add a Teardown section to docs/deployment/lambda.md (and a line in
README.md Step 2) documenting: (a) staged layer/function zips uploaded by
stand_up.sh are not removed by delete-stack — aws s3 rm them; (b) a
stack-created output bucket must be emptied before delete-stack succeeds; then
(c) aws cloudformation delete-stack --stack-name zagg-backend.

---

Critical files

• Grids: src/zagg/grids/{base,healpix,rectilinear,__init__}.py
• Catalog: src/zagg/catalog.py (spherely arrives via general_grids merge)
• Processing/dispatch: src/zagg/processing.py, src/zagg/runner.py, src/zagg/__main__.py
• Schema/config: src/zagg/schema.py, src/zagg/config.py
• Configs: src/zagg/configs/{atl06.yaml,atl06_polar.yaml}
• Deploy: deployment/aws/{template.yaml,stand_up.sh,deploy.sh,lambda_handler.py,invoke_lambda.py}
• Tests: tests/test_{grids,rectilinear,catalog,processing,runner,integration,schema,lambda_build}.py, tests/conftest.py
• Notebook: notebooks/cryocloud_example.ipynb
• Verification helper: bench/verify_spherely_cycle22.py

Verification (how we know it worked)

• Phases 1 green (full pytest + pytest -m slow + ruff) before any cull, and
  re-green after each cull commit.
• Phase 2: spherely vs mortie catalogs agree within Spherical geometry backend and dispatch for build_catalog #19's tolerance.
• Phase 3: both ./test_healpix.zarr (group 12) and ./test_rect.zarr (group
  rectilinear) open and contain expected shapes/vars from real reads.
• Phase 4: S3 objects written via Lambda; clean teardown.
• Post-cull: git grep -nE "process_morton_cell|layout.*dense|n_parent_cells|_build_catalog_healpix|_build_catalog_grid_driven|shapely-3031" returns only intended/none; no DeprecationWarning left for removed items.

Out of scope — track as follow-up issues (from PR loose ends)

H3Grid (Phase 1 design), odc.geo.GeoBox for RectilinearGrid, zagg.fast_read(),
spherely aarch64 wheel, PolygonZoneGrid/S2Grid (Phases 2/3), backend refactor
(#20), Python-API status (#13), temporal (#12). Not part of this validate+cull pass.

[espg]

Catalog backend validation — findings so far

Investigated spherely vs mortie for build_catalog, treating spherely (exact S2 geodesic intersection) as the geometric reference. Full cycle-22 ATL06: 14,710 granules → 1,330 order-6 shards, 76,575 (shard, granule) pairs.

The regression was a stale mortie, not the method

• mortie 0.6.3 (our pinned/locked version) + flat morton_coverage @ order 8: 1,872 omitted pairs (2.45%), all near the South Pole; 0 commission; 4 files dropped; ~4 s.
• mortie 0.7.0 + morton_coverage_moc (mixed-order MOC): omissions collapse to 9 (0.012%).

The CLI was also still calling the deprecated EPSG:3031 path (it never used spherely/mortie); fixed to route through the grid path with --geometry-backend.

Backend sweep (vs spherely = 76,575 pairs)

backend	order	omission	commission	files dropped	time
spherely (ref)	—	—	—	—	~43 s
flat morton_coverage	10	9	0	0	41 s
morton_coverage_moc	6	9	15	0	6.0 s
morton_coverage_moc	8	9	2	0	12.1 s
morton_coverage_moc	10	9	0	0	49.4 s

Sweet spot: MOC @ order 8 ≈ 12 s — spherely-equivalent, ~3.5× faster, no extra dependency. Commission shrinks with order and is the safe direction (extra files get read then filtered downstream; cost is just parsing).

The 9 residual omissions are REAL, not artifacts

Geodesic adjudication (pyproj Geod.npts densification → EPSG:3031 intersection area) shows all 9 spherely-only pairs are genuine overlaps (0.7–46 M m²) that mortie misses even at order 10 — a mortie polygon edge-interpretation effect near the pole (rhumb vs geodesic, espg/mortie#32). So spherely is fully correct; mortie-MOC carries a tiny 0.012% polar omission (data loss, not commission).

Pending

• Wire _build_catalog_mortie → morton_coverage_moc; bump mortie>=0.7.0.
• Decide default backend: spherely (exact, 43 s) vs mortie-MOC (12 s, 0.012% polar omission).
• Remove EPSG:3031 path (_build_catalog_healpix + shapely-3031) — now redundant.
• Review notebook with polar plots of the mismatches.

[espg]

General notes on clean-up / packaging--

1. We need to integrate the optimized spherely code into the catalog module.
2. We need to integrate spherely itself into the library build pipeline-- right now it's an optional dep, but against the main pip installable version; our spatial index hasn't been upstreamed yet, so we need to pin to https://github.com/espg/spherely/tree/feature/spatial-index
3. We need to address and close Reconcile Catalog API #24
4. We should also sweep deps that aren't needed anymore-- do we really still use healpy??
5. What's up with the rest of the deps in our toml -- mortie should be at least 0.7.2 given the fixes that are there... does shapely need to be in core, or should it go in catalog?

Realistically, all of this needs to be addressed before we can validate anything, since we'll be validating the arbitrary grid case as well as morton, and arbitrary won't work without these items being resolved.

[espg]

Re (1): integrating the optimized spherely (SpatialIndex) path into the catalog module

Where it stands. The catalog already has a geometry_backend="spherely" path (_build_catalog_spherely), but it's the brute-force one — per shard it broadcasts spherely.intersects(g_polys, s_poly) over all granule polygons, i.e. O(N_shards × N_granules). Its own docstring calls this out: "Spherely 0.1.x has no STRtree, so per-shard query is O(N_granules)." On cycle-22 ATL06 (14,710 granules → 1,330 order-6 shards, 76,575 pairs) that's ~56 s — correct, but quadratic.

The "optimized spherely code" is the SpatialIndex from the open PR (espg/spherely feature/spatial-index, benbovy/spherely#118). Integrating it is a small, contained change: build the index once over the granule polygons, then issue one exact query(..., predicate="intersects") per shard. Broad-phase (S2 cell overlap) + narrow-phase (exact intersects refinement) both happen in C++, so we go from O(N_shards × N_granules) to roughly O(N_shards · log N_granules).

The integration (the only change to _build_catalog_spherely — the granule-polygon build above it is unchanged):

    import spherely

    # ... g_polys (np.ndarray of granule polygons) and granule_urls built as before ...

    # Build the spatial index ONCE over all granule polygons.
    t0 = time.perf_counter()
    tree = spherely.SpatialIndex(g_polys)
    timings["index_build"] = time.perf_counter() - t0

    # Per-shard index query: broad-phase cell overlap + exact "intersects"
    # refinement, all in C++. Replaces the O(N_shards x N_granules) broadcast.
    t0 = time.perf_counter()
    catalog: Dict[int, List[str]] = {}
    for shard_key in all_shards:
        footprint = grid.shard_footprint(shard_key)
        sx, sy = footprint.exterior.coords.xy
        s_poly = _to_spherely_polygon(np.asarray(sy), np.asarray(sx))
        if s_poly is None or len(g_polys) == 0:
            continue
        # query() returns the granule indices directly (sorted np.intp);
        # predicate="intersects" makes it exact — no commission, no np.where.
        hits = tree.query(s_poly, predicate="intersects")
        if len(hits) > 0:
            catalog[int(shard_key)] = [granule_urls[int(i)] for i in hits]
    timings["index_queries"] = time.perf_counter() - t0

Same (catalog, timings) contract, same geometry_backend="spherely" entry point — callers don't change. This does depend on the spatial-index build of spherely being installed, which is exactly item (2): pin to spherely @ git+https://github.com/espg/spherely@feature/spatial-index until it's upstreamed.

The benchmark test we ran to re-build the cycle-22 ATL06 catalog (bench/benchmark_spherely_index.py). It reuses the cached CMR pull from verify_spherely_cycle22.py, rebuilds the catalog both ways on the same granule/shard polygons, asserts the two catalogs are pair-for-pair identical (proving the index is exact, not approximate — 0 omission / 0 commission vs the brute reference), and reports wall time:

"""Benchmark spherely brute-force vs SpatialIndex on the cycle-22 ATL06 catalog.

Requires the spatial-index build of spherely:
    pip install "spherely @ git+https://github.com/espg/spherely@feature/spatial-index"
"""
from __future__ import annotations

import time
import numpy as np
import spherely

from zagg.catalog import (
    _to_spherely_polygon,
    extract_granule_info,
    load_antarctic_basins,
)
from zagg.grids import HealpixGrid
from verify_spherely_cycle22 import get_granules  # cached cycle-22 CMR pull


def granule_polys(granules):
    polys, urls = [], []
    for g in granules:
        info = extract_granule_info(g)
        if not info["s3_url"] or len(info["points"]) < 3:
            continue
        lats = np.array([p[0] for p in info["points"]])
        lons = np.array([p[1] for p in info["points"]])
        poly = _to_spherely_polygon(lats, lons)
        if poly is not None:
            polys.append(poly)
            urls.append(info["s3_url"])
    return np.array(polys), urls


def shard_polys(grid, parts):
    out = []
    for key in grid.coverage(parts):
        fp = grid.shard_footprint(key)
        sx, sy = fp.exterior.coords.xy
        s = _to_spherely_polygon(np.asarray(sy), np.asarray(sx))
        if s is not None:
            out.append((int(key), s))
    return out


def build_brute(g_polys, urls, shards):
    cat = {}
    for key, s_poly in shards:
        hits = np.where(spherely.intersects(g_polys, s_poly))[0]
        if len(hits):
            cat[key] = [urls[i] for i in hits]
    return cat


def build_index(g_polys, urls, shards):
    tree = spherely.SpatialIndex(g_polys)              # built once, reused
    cat = {}
    for key, s_poly in shards:
        hits = tree.query(s_poly, predicate="intersects")
        if len(hits):
            cat[key] = [urls[int(i)] for i in hits]
    return cat


def main():
    granules = get_granules()
    parts = load_antarctic_basins()
    grid = HealpixGrid(parent_order=6, child_order=12, layout="fullsphere")

    g_polys, urls = granule_polys(granules)
    shards = shard_polys(grid, parts)
    print(f"granules={len(g_polys)}  shards={len(shards)}")

    t0 = time.perf_counter(); brute = build_brute(g_polys, urls, shards)
    t_brute = time.perf_counter() - t0
    t0 = time.perf_counter(); index = build_index(g_polys, urls, shards)
    t_index = time.perf_counter() - t0

    # The index path must be exact — identical to the brute reference.
    bp = {(s, u) for s, us in brute.items() for u in us}
    ip = {(s, u) for s, us in index.items() for u in us}
    assert bp == ip, f"mismatch: brute-only={len(bp - ip)} index-only={len(ip - bp)}"

    print(f"pairs            : {len(bp):,}")
    print(f"brute  intersect : {t_brute:6.1f} s")
    print(f"index  query     : {t_index:6.1f} s   ({t_brute / t_index:.0f}x faster)")


if __name__ == "__main__":
    main()

Result (the numbers reported on spherely#118): identical catalogs (76,575 pairs, 0 omission / 0 commission), brute ≈ 56 s → SpatialIndex ≈ 2 s (~25–30×) — and unlike mortie it needs no order/cell-size parameter. So integrating it is just (a) the _build_catalog_spherely swap above and (b) the spherely pin in item (2); I'll fold the benchmark into bench/ so we can re-run it as a regression check once the dep is pinned.

[espg]

Phase 0 results

🤖 Claude Code — executing the #24 refactor plan; Phase 0 (baseline & drift harness) is done.

Harness: bench/drift_catalog.py — format-agnostic (reads both legacy catalog-dict and new shard_keys/granules JSON), normalizes granule identity to basename, with build (snapshot from the cached pull) and compare (omission/commission) subcommands.

Baselines captured (all from the identical 4,103-granule cached cycle-22 pull, order-6 fullsphere):

role	backend	shards	pairs
old	buggy mortie + EPSG:3031 (Dec 2025)	1742	60,786
recent	mortie 0.7.2 MOC @8	1330	76,577
recent (oracle)	spherely brute (exact S2)	1330	76,575

The spherely build reproduced this issue's reference exactly (76,575 pairs), which validates the harness.

Drift findings (recorded in bench/drift/BASELINE_REPORT.md):

• recent mortie vs spherely-oracle: 99.997% overlap — 2 commission, 0 omission. The current default is effectively exact.
• old vs oracle: 69.5% overlap — old had 412 spurious shards, 7,565 commission, and 23,354 omission (missed ~30% of true overlaps). That's the "stale mortie" regression documented here, now measured end-to-end.
• Expectation for new: should match recent to ~100% (the SpatialIndex swap is the same S2 predicate as brute; mortie math unchanged). The improvement already happened at old→recent; new-vs-recent is a regression check, not an improvement target.

[espg]

Phases 1 & 2 — status

🤖 Claude Code — continuing the #24 execution.

Phase 1 (deps/packaging) — committed; plus the spherely fork is built and working:

• SpatialIndex: True, smoke-test query returns [0].
• It lives in a dedicated zagg-spherely conda env, not the uv .venv. The fork compiles against conda-forge s2geometry/s2geography (mainline ships a self-contained wheel); cross-linking those C++ libs into the venv risks an abseil/libstdc++ ABI mismatch, so a conda toolchain is the clean route. (Looking at whether the fork's CI can publish prebuilt wheels to make this .venv-installable.)

Phase 2 (fetch layer) — done:

• catalog.py → catalog/ package (clean rename; python -m zagg.catalog preserved via __main__.py).
• catalog/sources.py: Query, CMRSource, Catalog.
• Validated live against CMR-STAC (ATL03_007 over SERC): fetch → geoparquet → reload → granule_records(). Both s3 + https data hrefs preserved per granule (endpoint-neutral); thumbnails dropped by default with a preserve_thumbnails opt-in; zagg provenance and the GeoParquet geo key both round-trip.
• Full suite: 202 passed.

odc.geo GeoBox building blocks confirmed (SERC 10 km/10 m → GeoBox 1000×1000, GeoboxTiles(250,250) → 4×4 tiles). Proceeding to Phase 3 (GridSpec protocol + odc.geo RectilinearGrid + signature()/nests_with()).

[espg]

Phase 3 — GridSpec protocol + odc.geo

🤖 Claude Code

• RectilinearGrid rewritten on odc.geo.GeoBox + GeoboxTiles: coverage now uses GeoboxTiles.tiles(geom) with odc.geo reprojection (replacing the hand-rolled STRtree); footprint uses tile.extent.to_crs(...); the GeoBox affine is the single source of truth. The tested integer surface (width/height/_pack/shards_of/assign) is preserved — south-pole→center, OOB→sentinel, unchanged.
  • Note: a pole-spanning tile's footprint collapses to area-0 under odc.geo's default reprojection, so shard_footprint densifies (~32 pts/edge), matching the old code's behavior. Tests confirm parity.
• signature() + nests_with() on both grids and the protocol. Validated: 10 m nests 20 m (aligned, ratio 2); rejects ratio 2.5, cross-CRS, and cross-family (rect↔healpix → False). This is the future cross-aggregator-compatibility primitive, and signature() is what Phase 5 enforces at run time.

Full suite: 211 passed (9 new tests). Next: Phase 4 (ShardMap + spherely SpatialIndex integration + pruning the EPSG:3031/grid-driven dead paths).

[espg]

Phase 4 — ShardMap + backends + spherely wheel

🤖 Claude Code

catalog/shardmap.py: ShardMap (option-C self-contained JSON — each granule carries both s3 + https hrefs, endpoint-neutral; records grid.signature()) and ShardMap.build(catalog, grid, backend) over Catalog.granule_records(). Three backends, all validated in .venv:

• spherely — exact S2 via SpatialIndex (the swap from this PR: build the index once, query per shard). Identical results to shapely on the test grid.
• mortie — HEALPix MOC.
• shapely — WGS84 STRtree fallback.

auto is grid-aware: spherely if available, else mortie for HEALPix / shapely for rectilinear. This fixes a real trap — mortie's global MOC order (~25 km cells) is far coarser than a 2.5 km UTM tile, so on fine projected grids it collapses every granule into every shard. Auto now avoids that.

Spherely fork wheel is now installable (resolves the conda-toolchain hassle): the fork's release.yml built manylinux x86_64 wheels (cp310–cp314), released as v0.1.1-spatialindex, pinned in zagg's catalog extra and verified uv pip install-able into .venv. No merge to the fork's main was needed — workflow_dispatch against feature/spatial-index + a one-line version bump (0.1.1+spatialindex) was enough.

Full suite: 222 passed (11 new). Legacy build_catalog + dead paths stay until Phase 5 (expand-contract).

---

Deferred followup — Option B: gh-pages find-links index for spherely

The current catalog-extra pin is a direct-URL, cp312/linux-x86_64-only asset (Option A). It works for the dev .venv and py3.12-linux CI, but a single direct URL can't resolve across python versions or platforms — so zagg[catalog] on py3.13 or non-linux would miss it.

Fix (do after Phases 5–7): publish the fork's built wheels (cp310–cp314 + mac/windows, all already on the v0.1.1-spatialindex release) to a PEP 503 "simple" index on a gh-pages branch of espg/spherely, then pin spherely==0.1.1+spatialindex normally with --find-links https://espg.github.io/spherely/simple/ (or --extra-index-url). pip/uv then auto-selects the right wheel per platform/pyver, just like PyPI.

Needs: a small workflow job that lays built wheels into gh-pages /simple/spherely/index.html, and keeping the +spatialindex local-version label so resolvers treat it as distinct from mainline 0.1.1.

[espg]

Phase 5 (in progress) — 5a + 5b: single source of truth + signature enforcement

🤖 Claude Code

The catalog/run grid-mismatch footgun (a shard map built for one grid silently mis-run against another) is now closed.

5a — single source of truth: parent_order moved into output.grid (config), so from_config(config) fully defines the grid. No more split where parent_order came from catalog metadata and the rest from the YAML.

5b — runner rewired:

• Builds the grid from the run config and enforces grid.signature() == shardmap.grid_signature, e.g.:

  ValueError: ShardMap was built for a different grid than this run config.
    shard map : {'type': 'healpix', ..., 'parent_order': 99, 'child_order': 12, ...}
    run config: {'type': 'healpix', ..., 'parent_order': 6,  'child_order': 12, ...}
  

• Loads the ShardMap format (grid_signature + {id, s3, https} granules); pre-Phase-5 catalogs are rejected with regeneration instructions.
• Endpoint resolved at dispatch per data_source.driver (picks s3/https per granule), and the fragile s3_base/https_base URL-rewriter hack is deleted from processing.py.
• Lambda path updated symmetrically.

Full suite: 223 passed. Remaining in Phase 5: 5c (the --config-driven CMRSource+ShardMap CLI — rectilinear gets a CLI path for the first time) and 5d (hard-remove build_catalog + the EPSG:3031/grid-driven dead paths + legacy query_cmr/extract_granule_info; migrate the drift harness; rewrite test_catalog).

[espg]

Phase 5 complete — the hard-break (5a–5d)

🤖 Claude Code

sub	what
5a	parent_order moved into output.grid → from_config(config) fully defines the grid (single source of truth)
5b	runner builds grid from config, enforces grid.signature(), loads the ShardMap format, resolves s3/https per data_source.driver at dispatch; the s3_base/https_base URL-rewriter hack is deleted
5c	new CLI python -m zagg.catalog --config X.yaml … (CMRSource + ShardMap, grid from config) + make_shardmap — rectilinear gets a CLI path for the first time
5d	hard-removed build_catalog, _build_catalog_healpix (EPSG:3031), _build_catalog_grid_driven, query_cmr, extract_granule_info, discover_cells, _extract_base_urls; migrated the drift harness; rewrote test_catalog

Net −1,344 / +211 lines (catalog/__init__.py: ~1,014 → ~290).

Behavior preserved (drift check): the new ShardMap.build path reproduces the pre-refactor baselines 100% — 0 omission, 0 commission — for both mortie (76,577 pairs) and spherely (76,575) on the same cycle-22 granules. SpatialIndex == brute spherely exactly; the hard-break changed the code, not the output.

Original request works end-to-end: --config serc_atl03.yaml --short-name ATL03 --bbox=… fetched ATL03 over SERC via CMR-STAC, built the UTM grid from config, and wrote a ShardMap (option-C {id,s3,https} records) — the "UTM ATL03 has no CLI path" gap is closed.

Full suite: 198 passed. Next: Phase 7 (validate_compatible stub + docs/CHANGELOG sweep to the new CLI/API).

[espg]

Phase 7 complete — plan done (Phases 0–7)

🤖 Claude Code

• validate_compatible(grids) added (grids package): the grid-compatibility core of Reconcile Catalog API #24's future multi-product validator — every pair must nests_with the other; cross-family rejected. Spatial-overlap check noted as deferred until the multi-product API lands. (nests_with/signature already landed in Phase 3.)
• Docs sweep: docs/api/catalog.md rewritten to the new fetch/shard-map API (the old mkdocstrings refs to removed query_cmr/build_catalog would have broken the docs build); CLI examples in README / index / quickstart / lambda / rectilinear updated to --config; the obsolete healpy build instructions removed from the deployment docs.
• CHANGELOG entry under Unreleased.

Full suite: 201 passed. No stale build_catalog/query_cmr/--parent-order/healpy references remain in tracked code or docs.

Plan status

phase
0 baseline + drift harness	✅
1 deps/packaging	✅
2 fetch (Catalog/CMRSource)	✅
3 GridSpec + odc.geo + signature/nests_with	✅
4 ShardMap + backends (incl. SpatialIndex)	✅
5 runner wiring + --config CLI + hard-break	✅
6 validation (new == recent, 100%)	✅
7 validate_compatible stub + docs/CHANGELOG	✅

#24 can be closed once this is reviewed.

Operational follow-ups (out of plan scope)

• Lambda arm64 layer rebuild + size check (script updated in Phase 1, not yet rebuilt; pyproj's PROJ data is the line item — pandas→pyarrow is the reclaim lever if needed).
• Spherely Option B: gh-pages find-links index so zagg[catalog] resolves spherely across python versions/platforms (current pin is cp312/linux-x86_64 only).
• Remove the obsolete zagg-spherely conda env.

[espg]

arm64 Lambda layer/function build + deploy (for Apple Silicon)

🤖 Claude Code — context + runnable steps for building and deploying the arm64 Lambda from an Apple Silicon Mac with podman. This box (x86_64, no QEMU) can't build arm64, so the production arm64 artifact is built here.

What changed (why a rebuild is needed)

The #24 catalog refactor changed the Lambda layer:

• Added pyproj + odc-geo (+ affine, cachetools) — zagg.grids imports odc.geo at module load, and rectilinear/UTM grids reproject at processing time. Without them the Lambda fails on import.
• Removed earthaccess — it's orchestrator-only (auth); zagg/__init__.py is now lazy so the worker never imports it. Do not re-add it to the layer.
• zarr/obstore/pydantic-zarr are function-level deps (build_function.sh), not in the layer.
• Build scripts consolidated to a single deployment/aws/build_layer.sh [x86_64|arm64] (the old build_arm64_layer.sh and build_layer_v14.sh are gone). Both arches now target py3.12 / manylinux_2_28.

Reference: the x86_64 layer built clean at 125 MB unzipped / 76 MB zipped (well under the 250 MB limit), so arm64 should be comfortable too. pyproj's PROJ data is the main weight.

Build (run on Apple Silicon, arm64-native — podman)

Both builds must run inside a linux/arm64 manylinux_2_28 container so every wheel is linux-arm64 (running build_function.sh natively on macOS would grab macOS wheels — wrong). Pull latest main first.

cd <repo root>

# 1) Layer (numpy is built from source with 64KB page alignment for arm64 — a few min)
podman run --rm -v "$PWD":/workspace:Z -w /workspace/deployment/aws \
  quay.io/pypa/manylinux_2_28_aarch64 \
  bash -c "yum install -y zip && chmod +x build_layer.sh && ./build_layer.sh arm64"
# -> deployment/layers/lambda_layer_arm64.zip  (script enforces <250MB unzipped)

# 2) Function code (needs py3.12 on PATH inside the container)
podman run --rm -v "$PWD":/workspace:Z -w /workspace \
  quay.io/pypa/manylinux_2_28_aarch64 \
  bash -c "export PATH=/opt/python/cp312-cp312/bin:\$PATH && yum install -y zip && bash deployment/aws/build_function.sh"
# -> deployment/builds/lambda_function_arm64_py312.zip

If the layer is over budget, the strip block in build_layer.sh (removes pyarrow/xarray/etc, strips .so, drops tests/__pycache__) is where to tune; the pandas→pyarrow swap in processing.py is the heavy lever but was not needed for x86.

Verify the layer imports (no earthaccess, geo present)

podman run --rm -v "$PWD":/workspace:Z quay.io/pypa/manylinux_2_28_aarch64 bash -c '
  cd /tmp && unzip -q /workspace/deployment/layers/lambda_layer_arm64.zip
  /opt/python/cp312-cp312/bin/python -c "
import sys; sys.path.insert(0, \"python\")
import pyproj, odc.geo, shapely, numpy, pandas, h5coro, mortie
print(\"layer imports OK:\", pyproj.__version__)"'

Deploy (CloudFormation; both arches are py3.12 now)

template.yaml defaults to Architecture: arm64, Runtime: python3.12. Standup helper:

# Uploads layer+function zips to the artifact bucket and creates/updates the stack.
bash deployment/aws/stand_up.sh arm64
# (sets LAYER_ZIP=lambda_layer_arm64.zip, FUNC_ZIP=lambda_function_arm64_py312.zip)

Or deploy.sh arm64 for the layer-publish + function-update path (RUNTIME=python3.12). Confirm the CloudFormation params LayerS3Key/FunctionS3Key match the produced zip names.

Gotchas

• Must be arm64-native (Apple Silicon) — no QEMU.
• py3.12 / manylinux_2_28 (glibc 2.28) — compatible with the py3.12 Lambda runtime (AL2023, glibc 2.34). This is exactly why x86 was bumped 3.11→3.12: pyproj only ships manylinux_2_28 wheels, which the old AL2/py3.11 (glibc 2.26) runtime couldn't load.
• Don't re-add earthaccess; don't move zarr/obstore/pydantic-zarr into the layer.
• After deploy, smoke-test one shard (a HEALPix cell) to confirm the worker imports + writes zarr.

[espg]

arm64 Lambda build — done (Apple Silicon, native, podman)

🤖 Claude Code — executed the arm64 build/handoff from the previous comment against HEAD 1375940.

Built arm64-native on Apple Silicon (podman machine = aarch64, container uname -m = aarch64, no QEMU). Installed podman 5.8.2 (machine init/start) since it wasn't present.

Results

Artifact	Zipped	Unzipped	Limit
lambda_layer_arm64.zip	36 MB	128 MB	<250 MB ✅
lambda_function_arm64_py312.zip	8.8 MB	25 MB	(30 MB code budget) ✅

Comparable to the x86 reference (125 MB unzipped); the numpy-from-source (64 KB page align) + pyproj/odc-geo additions land us at 128 MB with plenty of headroom.

Build commands (ran exactly as in the handoff, podman)

# layer
podman run --rm -v "$PWD":/workspace:Z -w /workspace/deployment/aws \
  quay.io/pypa/manylinux_2_28_aarch64 \
  bash -c "yum install -y zip && chmod +x build_layer.sh && ./build_layer.sh arm64"
# function (cp312 on PATH)
podman run --rm -v "$PWD":/workspace:Z -w /workspace \
  quay.io/pypa/manylinux_2_28_aarch64 \
  bash -c "export PATH=/opt/python/cp312-cp312/bin:\$PATH && yum install -y zip && bash deployment/aws/build_function.sh"

Layer import verification

layer imports OK: 3.7.2        # pyproj, odc.geo, shapely, numpy, pandas, h5coro, mortie all import
earthaccess absent (correct)

numpy resolved to 2.2.6 (built from source, <2.3 guard satisfied); mortie 0.7.2, odc-geo 0.5.1, shapely 2.1.2.

Artifacts

• deployment/layers/lambda_layer_arm64.zip
• deployment/builds/lambda_function_arm64_py312.zip

Not yet done (next steps)

• stand_up.sh arm64 / deploy.sh arm64 — CloudFormation deploy not run
• smoke-test one HEALPix shard end-to-end (worker imports + writes zarr)

Build scripts at HEAD now match the handoff spec (py3.12, pyproj+odc-geo present, no earthaccess). Ready to deploy on your go.