Coverage MOCs: worker-owned leaf tier + end-of-run root (issue 200)

deployment/aws/lambda_handler.py

@@ -441,6 +441,7 @@ def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
 
     Default ``mode`` (or no mode) runs per-cell processing. ``mode="setup"``
     creates the zarr template; ``mode="finalize"`` consolidates metadata;
+    ``mode="coverage"`` writes the store-root ``coverage.moc`` (issue #200);
     ``mode="extract"`` extracts chunk-boundary geometry parquets (issue #148);
     ``mode="process_event"`` runs the temporal/event worker (issue #12).
     """
@@ -455,6 +456,8 @@ def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
         return _handle_setup(event)
     if mode == "finalize":
         return _handle_finalize(event)
+    if mode == "coverage":
+        return _handle_coverage(event)
     # Extract mode returns directly: the result_url mirror below is for the
     # per-unit fan-out handlers (spatial process, temporal process_event) only.
     if mode == "extract":
@@ -673,6 +676,36 @@ def _handle_finalize(event: Dict[str, Any]) -> Dict[str, Any]:
         return {"statusCode": 500, "body": json.dumps({"error": str(e), "mode": "finalize"})}
 
 
+def _handle_coverage(event: Dict[str, Any]) -> Dict[str, Any]:
+    """Write/union the store-root ``coverage.moc`` (issue #200 phase 3).
+
+    Posted fire-and-forget (``InvocationType="Event"``) by the dispatcher at
+    end of run: the orchestrator can compute the shard-order MOC but cannot
+    PUT to S3, so the SERIALIZED envelope rides in the event (bounded by
+    construction — see the dispatch-site comment in ``zagg.runner``) and the
+    worker GET-unions-PUTs one root object. Nobody reads this response on
+    the Event invoke; errors are logged and fail open — the root MOC is a
+    regenerable cache (D9): readers degrade to the sweep MOC or the walk,
+    never to wrong answers.
+    """
+    from zagg.hive import write_root_coverage
+
+    logger.info(f"Coverage mode: writing root coverage.moc at {event.get('store_path')}")
+    try:
+        merged = write_root_coverage(
+            event["store_path"], event["coverage"], **_output_store_kwargs(event)
+        )
+        return {
+            "statusCode": 200,
+            "body": json.dumps(
+                {"ok": True, "mode": "coverage", "ranges": len(merged.get("ranges", []))}
+            ),
+        }
+    except Exception as e:
+        logger.exception(e)
+        return {"statusCode": 500, "body": json.dumps({"error": str(e), "mode": "coverage"})}
+
+
 def _json_scalar(v: Any) -> Any:
     """Coerce one result value to a JSON-safe scalar (numpy float -> float)."""
     if v is None:

docs/design/sparse_coverage.md

@@ -127,19 +127,21 @@ the data itself). Tracks [#200](https://github.com/englacial/zagg/issues/200).
   is frozen with the mortie-side spec (O8). Future *store-level* covers that cross base
   cells generalize to **≤ 12 members** (the 12 base cells). Readers
   AOI-reject on the box without parsing anything larger.
-- **Tier 1 — the budgeted shard MOC**: cell-level coverage, exact if it fits
-  the budget, else **coarsened until it fits**. Coarsening only over-covers
-  (conservative superset): false positives cost one wasted leaf read; false
-  negatives are impossible. The achieved depth is recorded with the payload.
-  Budget scale is open (O8) and is matched to **S3/cloud conventions, not
-  database-page conventions** — candidates span KB to low MB (the PostgreSQL
-  ~2 KB TOAST inline threshold is retained only as a portability footnote).
-  The budget also picks the carrier: a KB-scale payload rides the commit
-  stamp attrs (zero extra objects); a hundreds-of-KB tier becomes its own
-  object inside the leaf, with only the box + achieved depth + pointer in
-  attrs (one extra GET, paid only by readers that want it). If the in-leaf
-  object survives O8, it is a *noted exception* to §2's "vanilla zarr v3"
-  leaf: one foreign key inside the leaf, invisible to zarr readers.
+- **Tier 1 — the exact shard bitmap** *(as O8 resolved it — the originally
+  drafted "budgeted, coarsen-to-fit MOC" was superseded by the
+  [#202 item (6) measurement](https://github.com/englacial/zagg/issues/200#issuecomment-4939264286):
+  for linear-track occupancy, coarsen-to-fit ranges at KB budgets deliver
+  only box-level filtering while a compressed bitmap reaches exact at
+  ~25 KB)*: the shard's cell-order occupancy as a **zstd-compressed bit
+  field**, one bit per subtree cell in ascending packed-word order, stored
+  as the in-leaf `coverage.moc` sidecar. Raw size is deterministic
+  (`ceil(4^depth/8)`) regardless of fragmentation; no coarsened variant is
+  built (one code path, sidecar-only). The stamp attrs carry only the box +
+  the bitmap's order + pointer + byte sizes (one extra GET, paid only by
+  readers that pass the box test). The sidecar is the *noted exception* to
+  §2's "vanilla zarr v3" leaf: one foreign key inside the leaf, ignored by
+  zarr readers (data reads unaffected; member enumeration warns and skips
+  it).
 - **Tier 2 — exact**: the `morton` coordinate array in the leaf *is* the
   exact cell list. The MOC tiers are indexes, never truth (D9 discipline,
   applied one level down).
@@ -171,10 +173,13 @@ the data itself). Tracks [#200](https://github.com/englacial/zagg/issues/200).
   cost effectively nothing and answer the actual question readers ask
   ("where is there data?") in one GET.
 
-Open items: budget scale / serialization / carrier / pad sentinel for the
-shard MOC tiers (O8, measured on
-[#202](https://github.com/englacial/zagg/issues/202) item (6)); the
-end-of-run root MOC default (O9); root-object serialization format (O1).
+O8 and O9 are **resolved** (espg-ratified on the
+[#200 thread](https://github.com/englacial/zagg/issues/200#issuecomment-4939477871),
+implemented on PR #208): the shard tier is an **exact cell-order occupancy
+bitmap, zstd-compressed, as an in-leaf sidecar object** — attrs carry only
+the tier-0 box + order + pointer + sizes, with a null pad sentinel — and the
+end-of-run root MOC **defaults on** for hive stores. The root object
+serializes per O1 as JSON ranges with decimal-string endpoints.
 
 ## 5. Layer 3: reader architecture
 
@@ -323,17 +328,31 @@ write path (§2) is load-bearing; this phase is optimization.
   doesn't list), regenerate explicitly (`refresh=True` / the sweep); no
   wall-clock staleness horizon. The incremental-run half is settled by the
   §4 lifecycle (leaves are durable truth; the sweep rebuilds correctly).
-- **O8 — shard-MOC budget, serialization, carrier, pad sentinel**: byte
-  scale matched to S3/cloud conventions (candidates KB → low MB); ranges
-  vs. bitmap serialization; commit-stamp attrs payload vs. in-leaf object
-  carrier; box pad sentinel (base-0/null vs. repetition). Decided using
-  the #202 item (6) measurement (SERC + 88S shards; the tier-0 box is the
-  baseline each budget must beat; over-coverage reported in STAC
-  simplification-error terms). Frozen alongside the mortie spec.
-- **O9 — end-of-run root MOC default**: on (the fire-and-forget Event
-  invoke is fail-open and run-size independent) vs. off until #202's
-  full-AOI runs record the invoke round trip. Either is a one-line flag;
-  espg leaning cautious pending the benchmark-impact numbers.
+  Implemented in this shape by PR #208's reader primitives
+  (`zagg.coverage`: `warn_if_stale` — once per store, never auto-walk —
+  and `refresh_root_coverage`, the explicit walk); the concurrent-run
+  GET-union-PUT race (last writer wins until re-union/sweep) is recorded
+  there as accepted under this same lean.
+- **O8 — shard-MOC budget, serialization, carrier, pad sentinel**:
+  **RESOLVED** ([espg-ratified](https://github.com/englacial/zagg/issues/200#issuecomment-4939477871),
+  from the [#202 item (6) measurement](https://github.com/englacial/zagg/issues/200#issuecomment-4939264286)):
+  no budgeted/coarsened tier — the leaf encoding is an **exact cell-order
+  bitmap, zstd-compressed, as the in-leaf `coverage.moc` sidecar** (raw size
+  deterministic at `ceil(4^depth/8)`, immune to the ragged worst case where
+  exact ranges hit MB-scale on linear-track occupancy at ~1.1 cells/range).
+  Stamp attrs carry only the tier-0 box + the bitmap's order + pointer +
+  byte sizes; pad sentinel is JSON null; the envelope gains an
+  `encoding: "ranges" | "bitmap"` discriminator. Bit convention (frozen
+  with the mortie spec, golden-vector-pinned on PR #208): bit i = the i-th
+  shard-subtree cell in ascending packed-word order (base-4 D1 digit tail,
+  digits 1..4 → 0..3), MSB-first per byte.
+- **O9 — end-of-run root MOC default**: **RESOLVED — on** for hive stores
+  ([espg](https://github.com/englacial/zagg/issues/200#issuecomment-4938859764):
+  coverage MOCs are the default for healpix templates; PR #208 implements
+  `output.coverage_moc`, default true under `store_layout: hive`, explicit
+  true rejected elsewhere). The write is fail-open on both backends; the
+  Lambda leg is one fire-and-forget `mode: "coverage"` Event invoke with the
+  pre-serialized ranges envelope.
 
 ## 9. References
 

docs/hive_layout.md

@@ -27,11 +27,16 @@ round — wired to the **local backend only** (see [Status](#status)).
   naturally: every order is a legal node.
 - **Full id at the leaf** (D3): `.../-5/1/1/2/3/3/3/-5112333.zarr` is
   self-describing without parsing its directory chain — greppable in
-  inventories, unambiguous if moved. Each leaf is a completely vanilla zarr v3
+  inventories, unambiguous if moved. Each leaf is a vanilla zarr v3
   store: the same group/array template as the flat layout, sized to one shard
   (dense arrays hold `cells_per_shard` cells; `resolution: chunk` companions
   hold the shard's K inner chunks; CSR ragged subgroups sit under the same
-  group path, named by the shard label at K==1).
+  group path, named by the shard label at K==1). One recorded exception
+  ([issue #200](https://github.com/englacial/zagg/issues/200), O8): the
+  `coverage.moc` occupancy-bitmap sidecar inside the leaf — a single foreign
+  key that zarr readers ignore (data reads are unaffected; member
+  enumeration like `members()`/`tree()` emits a `ZarrUserWarning` and skips
+  it).
 - **Node invariant** (D5): below the root, a node contains *only* digit
   children (`[1-4]/`) and `*.zarr` objects — zero zarr metadata above the
   leaf, no shared mutable state across workers. The root alone also carries
@@ -120,21 +125,137 @@ template creates anyway. A shard that errors, or streams no chunks (no data),
 leaves no stamp; a fully empty shard leaves no `.zarr/` prefix at all (the
 leaf is created lazily on the first chunk write).
 
+The stamp also carries the shard's **coverage envelope** — see
+[Coverage](#coverage) below. The sidecar it points to is written before the
+stamp, so coverage shares the debris semantics: no stamp, no visible coverage.
+
+## Coverage
+
+Where the data is, declared hierarchically
+([issue #200](https://github.com/englacial/zagg/issues/200), design §4 as
+amended by PR #206; O8/O9 resolved on the issue thread). Three tiers per
+shard plus one store-root object:
+
+| tier | what | where | cost to read |
+|---|---|---|---|
+| 0 — morton box | canonical ≤ 4-member cover of the occupied cells (DCA children, each tightened) | `coverage` payload on the commit stamp | free — rides the stamp GET readers already make |
+| 1 — exact bitmap | zstd-compressed bit field over the shard subtree at `cell_order` | `{full_id}.zarr/coverage.moc` sidecar | one opt-in GET |
+| 2 — exact truth | the leaf's `morton` coordinate array | the leaf's data plane | array read; the tiers above are indexes, never truth (D9) |
+| root | shard-order ranges MOC over all completed shards | `{store_root}/coverage.moc` | one GET — the discovery bootstrap |
+
+**Leaf envelope** (on the stamp, `zagg.hive.read_coverage`; strict
+`spec: morton-moc/1` gate — unknown specs read as absent):
+
+```json
+"coverage": {
+  "spec": "morton-moc/1",
+  "box": ["-42113221", "-42113224", null, null],
+  "cell_order": 12,
+  "source": "worker",
+  "encoding": "bitmap",
+  "sidecar": "coverage.moc",
+  "nbytes": 213,
+  "raw_nbytes": 512
+}
+```
+
+`box` is always exactly 4 slots, nulls trailing; members are D1 decimal
+strings. `encoding`/`sidecar`/sizes appear only when the bitmap exists — a
+box-only envelope (phase-1-era leaf, or a depth-0 `child_order ==
+parent_order` config) is read as "box only". No `generated_at`: the stamp's
+`written_at` is the one clock and one writer. Bit convention (frozen with
+the mortie-side spec): bit i = the i-th shard-subtree cell in ascending
+packed-word order (base-4 value of the D1 digit tail, digits 1..4 → 0..3),
+MSB-first per byte. A corrupt sidecar (bad zstd, wrong size) **raises**; a
+missing one degrades to `None` — a truncated bitmap must never read as a
+plausible partial cell set. The sidecar is the one foreign key inside the
+otherwise-vanilla leaf: zarr data reads are unaffected, but member
+enumeration (`members()`/`tree()`) emits a `ZarrUserWarning` and skips it.
+
+**Root envelope** (`{store_root}/coverage.moc`, `zagg.coverage.load_coverage`):
+
+```json
+{
+  "spec": "morton-moc/1",
+  "encoding": "ranges",
+  "order": 6,
+  "source": "dispatcher",
+  "generated_at": "2026-07-10T22:59:35+00:00",
+  "ranges": [["5112333", "5112333"], ["-4211321", "-4211324"]]
+}
+```
+
+The example above is `zagg.hive.build_root_coverage` output for the shards
+`-4211321..-4211324` plus `5112333` (all order 6) and round-trips through
+`root_coverage_words`; the test suite parses it straight out of this file so
+the reference example can never drift from the implementation.
+
+A range is an inclusive run of same-order cells within one base cell,
+consecutive in digit-tail rank; endpoints are decimal **strings** (packed
+u64 words exceed 2^53 and raw JSON numbers get mangled by float-based
+parsers). `source` is `"dispatcher"` (end-of-run write) or `"refresh"` (the
+explicit walk rebuild); the sweep will add its own.
+
+**Reader flow** (`zagg.coverage`): `load_coverage` → `root_coverage_and`
+against the AOI to pick candidate shards (one GET, no walk); per leaf,
+`box_and` on the stamp payload for the cheap reject, then `bitmap_and` for
+exact cell-level filtering (falls back to the box verdict with `None` when
+the leaf is box-only), then the `morton` coordinate as truth. The box is a
+conservative superset — false positives cost one wasted read, false
+negatives are impossible; the bitmap and the root MOC are exact for what
+they list.
+
+**Staleness (O7)**: readers trust silently on the hot path. The root object
+is written fail-open at **end of run** while leaves stamp continuously, so
+the most common gap is benign — a run still in progress. Beyond that, a
+crashed run, an out-of-band write, or the benign concurrent-run union race
+(GET-union-PUT is not atomic; last writer wins until the next re-union)
+leaves it missing shards, which degrades to "reader doesn't see the newest
+run", never a wrong answer. `zagg.coverage.warn_if_stale` implements the
+lazy detection lean: when a reader opens a commit-stamped leaf the root MOC
+doesn't list, it warns once per store and suggests
+`zagg.coverage.refresh_root_coverage` — the explicit delimiter-LIST walk
+that rebuilds the root MOC from the stamped leaves (debris excluded) and
+writes it with `source: "refresh"`. No reader ever auto-walks (D10).
+
+**Deploy note** (mirrors the PR #205 setup-echo note): the Lambda leg posts
+one fire-and-forget `mode: "coverage"` invoke, which requires the
+redeployed function. An **older deployment 400s the event in its process
+handler** — a logged error line in CloudWatch, but no writes, no result
+mirror, and no async redelivery — so the failure is fail-open by
+construction; the root object simply doesn't appear until the sweep or a
+refresh builds it.
+
 ## Reading a hive store
 
 There is no store-root `zarr.open()` (deliberately — D12; a root hierarchy can
 be added later by the sweep as a derived artifact). Readers:
 
 1. GET `morton_hive.json` (once, cacheable) → `shard_order`, `cell_order`.
-2. Compute a shard's leaf path by string arithmetic on its decimal id
+2. GET `coverage.moc` (`zagg.coverage.load_coverage`) → the covered shard
+   set, intersected with the AOI (`root_coverage_and`) — see
+   [Coverage](#coverage).
+3. Compute a shard's leaf path by string arithmetic on its decimal id
    (`zagg.hive.shard_leaf_path`), open the leaf zarr, and **check the commit
-   stamp** (`zagg.hive.read_commit`) before trusting the contents.
-3. Discovery without a shard list is a delimiter-LIST walk: recurse on
-   `[1-4]/` children; a `*.zarr` entry is data at that node; no digit children
-   ⇒ nothing finer. Never LIST per observation in a join loop (D10).
+   stamp** (`zagg.hive.read_commit`) before trusting the contents; the
+   stamp's coverage payload pre-filters the AOI (`box_and`/`bitmap_and`).
+4. Discovery without a root MOC falls back to the delimiter-LIST walk:
+   recurse on `[1-4]/` children; a `*.zarr` entry is data at that node; no
+   digit children ⇒ nothing finer. Never LIST per observation in a join
+   loop (D10).
 
-The `coverage.moc` domain declaration (§4 of the design record) is a follow-on
-issue and will remove the walk from the discovery path too.
+The store-root `coverage.moc` ([issue #200](https://github.com/englacial/zagg/issues/200)
+phase 3, default-on for hive) removes the walk from the bootstrap path: one
+GET of the root object yields the shard-order coverage MOC (JSON ranges,
+decimal-string endpoints). It is written fail-open at end of run — by the
+dispatcher directly (local) or one fire-and-forget `mode: "coverage"` worker
+invoke (Lambda; an older deployment has no coverage mode and 400s the event
+in its process handler — logged, no writes, no async retry — which is safe:
+the object is a regenerable cache under D9, and readers degrade to the walk).
+Incremental runs union with the existing object; concurrent runs race
+benignly (GET-union-PUT is not atomic: last writer wins, and its union may
+miss the loser's shards until the sweep or the next run re-unions — accepted
+under D9/O7). The §7 sweep remains the authoritative rebuilder.
 
 ## Status
 
@@ -146,7 +267,12 @@ issue and will remove the walk from the discovery path too.
   the event config's orders, emits its own leaf template, and stamps
   completion as its final PUT. The async status channel stays at the flat
   sibling prefix (`{store_root}.status/<run_id>/…`), outside the digit tree.
-- The store-level `coverage.moc` (and any per-shard MOC stamping) waits on
-  the design in [issue #200](https://github.com/englacial/zagg/issues/200).
+- **Coverage ships** ([issue #200](https://github.com/englacial/zagg/issues/200)
+  phases 1–4): the tier-0 morton box on the commit stamp, the exact
+  zstd-bitmap `coverage.moc` sidecar inside each leaf, the end-of-run
+  store-root `coverage.moc` (shard-order ranges MOC, `output.coverage_moc`,
+  default on for hive) for the one-GET bootstrap, plus the `zagg.coverage`
+  reader primitives (per-tier AOI intersection, O7 staleness lean, explicit
+  refresh).
 - Write-throughput validation at fleet scale is tracked with the benchmark
   machinery in [issue #202](https://github.com/englacial/zagg/issues/202).