Generic hierarchical filtering & targeted reads for multi-level point datasets

[espg]

Motivation

zagg aggregates sparse point observations onto grids, but data_source is today
flat and single-level with exactly one quality_filter that is
equality-only on a 1-D dataset (flag == value, keep matches — see
processing.py::_read_group). That is too limited for real products and is
leaving large performance on the table. Four related gaps:

1. Memory floor of spatial filtering. To assign points to grid cells we must
   read the full coordinate arrays before we can subset. Measured on a real
   ATL03 v007 granule via h5coro: lat_ph+lon_ph full read is up to 245 MB
   per strong beam, ~660 MB peak per granule — paid before any filter or
   aggregation, and multiplied by granules-per-worker. This is the dominant
   driver of the Lambda OOMs on the ATL03 SERC benchmark.
2. Filtering is underpowered. ATL03 noise removal is signal_conf_ph != 0 on
   a 2-D array (per-surface-type column) — the current equality-only / 1-D
   filter cannot express it. We also need multiple independent filters, and
   filters phrased as "keep these flag values" / "drop these".
3. Cross-level (multi-rate) filters. Some quality flags live at a coarser
   rate than the observations. Clearest case: ATL03 podppd_flag (geolocation,
   20 m segment rate) marks geolocation failures that should drop the
   corresponding photons. We need to filter photon-rate data using segment-rate
   flags.
4. Targeted/hierarchical reads. The same coarse level that carries those
   flags also lets us bound the read: geolocation is ~30× smaller — read it
   fully, find the segments intersecting the AOI, and read only the corresponding
   heights chunks.

These are facets of one structural idea: datasets have multiple linked levels
(resolutions), and both filtering and IO should be expressible across them. We
want one clean, generic abstraction — built for ATL now, extending to other
missions/datasets later.

Evidence: hierarchical reads work (prototype)

benchmarks/geoloc_prefilter_proto.py (branch feature/claude-geoloc-prefilter,
off current main) implements box → segment range → ph_index_beg slice → read
only the overlapping heights chunks, and verifies it against the full-read path.

Verified index translation (over all 71,607 populated segments of a beam,
exact lat and lon match): the absolute heights index of a segment's reference
photon is ph_index_beg - 1 + reference_photon_index - 1 (both 1-based;
reference_photon_index is segment-relative, not an absolute heights
index). For bounding a read, only ph_index_beg + segment_ph_cnt are needed;
they tile heights exactly (sum(segment_ph_cnt) == n_photons).

Prototype result (one granule, ~10 km AOI, all 6 beams; in-box photon counts
match the full-read path exactly):

beam	in-box photons	base coord MB	hierarchical MB	heights chunks read
gt1l	35,834	59.3	4.27	2 / 38
gt1r	145,089	241.2	6.02	2 / 151
gt2l–gt3r (no AOI data)	0	57–245	3.69 (geoloc only)	0
total		856.8	25.0 (2.9%)

The empty-beam case is the common one for small AOIs: the geolocation index alone
(~3.7 MB) proves there is nothing to read, vs a 57–245 MB full coord read.

Proposed design direction

Generalize data_source from a flat single level to named levels linked by
index maps, with filters scoped to a level, and a spatial pre-read
driven by a chosen level. Aggregation still happens at the finest ("base") level.

Levels + linkage

data_source:
  reader: h5coro
  driver: s3
  groups: [gt1l, gt1r, gt2l, gt2r, gt3l, gt3r]   # repeated sub-structures ("beams")
  base_level: photon            # where aggregation happens (finest)
  levels:
    photon:
      path: "/{group}/heights"
      coordinates: { latitude: lat_ph, longitude: lon_ph }
      variables:  { h_ph: h_ph }
    segment:
      path: "/{group}/geolocation"
      coordinates: { latitude: reference_photon_lat, longitude: reference_photon_lon }
      variables:  { podppd_flag: podppd_flag }
      link:                              # how this coarse level maps to a finer one
        to: photon
        index_beg: ph_index_beg          # start of each parent's children in `to`
        count: segment_ph_cnt            # children per parent
        index_base: 1                    # ph_index_beg is 1-based
        reference_index: reference_photon_index   # optional, 1-based within-parent

The link is the one piece of mission-specific knowledge, declared, not
hardcoded. It powers both cross-level filtering and targeted reads. Generic
over index base (0/1) and start+count vs explicit start/end conventions. Other
missions declare their own (or a single level with no link).

Filters: a list, each scoped to a level, with real predicates

  filters:
    - name: drop_noise
      level: photon
      dataset: signal_conf_ph
      column: 0                          # N-D column selector (ATL03 surface-type: land)
      keep: { op: ne, value: 0 }         # or { op: in, values: [2, 3, 4] }
    - name: good_geolocation
      level: segment                     # coarse flag -> expanded to photons via link
      dataset: podppd_flag
      keep: { op: eq, value: 0 }

• Multiple filters AND together.
• keep operators: eq, ne, in, not_in, ge, le, lt, gt (single value, multi-value
  sets, ranges). column (or a named index) handles N-D flag arrays.
• A coarse-level filter yields a coarse mask, expanded to the base level via the
  level's link (repeat each parent's verdict count times from index_beg),
  then AND-ed with base-level filters.
• Today's quality_filter: {dataset, value} becomes sugar for one base-level
  {op: eq} filter (back-compat; migrate the atl06 configs).

Targeted (hierarchical) reads

  read_plan:
    spatial_index: segment    # bound the read using this level's coordinates
    pad: 1                    # pad the matched parent range by N elements per side

Per group: read the spatial_index level's coords (small) → AOI-match parents →
contiguous padded parent range → translate to a base-level slice via link →
read only the overlapping base-level chunks (coords + needed variables + any
coarse-level flag ranges for cross-level filters) → apply combined filters →
aggregate. Falls back to a full read when no level/link is declared (today's
behavior).

Why this generalizes

Nothing above names ATL fields except inside the config. The engine only knows
levels, links (index_beg/count/base), filters (level, dataset,
column, predicate), and read via a spatial-index level. Single-level datasets
omit levels/link and use today's flat form; other instruments map their own
hierarchy (swath scan-line → pixel, beam → bin, …) onto the same primitives.

Open questions (for iteration)

1. Schema surface vs back-compat. Keep flat coordinates/variables/
   quality_filter as sugar over a synthesized single photon level, or migrate
   all configs to levels? (Lean: support both; synthesize a base level from the
   flat form.)
2. Column selectors by name. ATL03 signal_conf_ph columns are surface types
   (land/ocean/sea-ice/land-ice/inland-water). Allow column: land via a
   per-dataset enum map, or integer-only?
3. Filter combination across levels. AND-only, or named groups / OR?
   (Lean: AND-only first.)
4. Multi-run AOIs. A track can clip an AOI twice → non-contiguous parent runs.
   v1: min..max (conservative, reads the gap); later: split into runs.
5. Where the read planner lives. Extend _read_group, or a new read_plan
   module the grid/runner call? (Affects the Lambda path; coordinate with the
   arrow refactor that just landed.)
6. Decode the spatial-index coords once per granule and reuse across beams
   where shared (ATL: per-beam; other missions may share).

Phasing / dependencies

• The grouping + arrow reducer (Refactor the per-cell aggregation handoff: sort/hash grouping + Arrow path (additive, benchmarked) #30 phases 1/2/2b) is now on main. First
  measure whether arrow alone gets the ATL03 SERC run under the 2 GB Lambda
  threshold — that calibrates urgency. The read floor is independent of arrow, so
  hierarchical reads help regardless.
• Phase A: multiple filters + predicate operators + N-D column selector
  (base-level only). Unblocks signal_conf_ph != 0. Aggregation-time only; no
  read-path restructure.
• Phase B: levels + link abstraction + cross-level (coarse→fine) filter
  expansion (podppd_flag).
• Phase C: targeted hierarchical reads driven by the spatial-index level (the
  prototype), folded into the read path on top of B.
• Each phase is independently testable; B and C share the link machinery.

Refs: prototype benchmarks/geoloc_prefilter_proto.py; memory-floor +
index-translation findings above; builds on #30 (grouping/arrow).

[espg]

claude, critique the proposed design above. Are there other competing approaches we should think about or consider?

[espg]

🤖 from Claude

Critique of the levels + link + scoped-filters + spatial-pre-read design, then competing approaches.

What's strong (keep)

• The link (index_beg / count, base, reference_index) is the right primitive. Declaring the parent→child tiling instead of hardcoding it is what makes this generic, and your verified index translation (ph_index_beg-1 + reference_photon_index-1, sum(segment_ph_cnt)==n_photons) is the load-bearing fact — it's solid.
• The A → B → C phasing is correct and independently valuable. Phase A alone unblocks the real signal_conf_ph != 0 (2-D, ne) noise filter and "multiple ANDed filters" with no read-path restructure — that's the highest value-to-risk step and should ship first regardless of B/C.
• Flat-form back-compat by synthesizing a single base level (your open-Q1 lean) is right; do that.

Risks / things I'd push on

1. link bakes in contiguous one-parent→one-finer tiling. Perfect for ATL03 photon/segment, but it's a real constraint: many-to-many maps, non-contiguous children, or flags keyed by time/value rather than index won't fit index_beg/count. Don't over-generalize now — just name the contiguity assumption explicitly and leave room for an explicit-index-array link variant later. Otherwise the abstraction will quietly mislead the next mission.
2. Don't grow a second query DSL. The {op, value/values} predicate schema overlaps the expression mechanism agg fields already use. Suggestion: keep structured predicates for anything that must be pushed down to bound a read (those need to be machine-inspectable), but allow an expression escape hatch for base-level-only filters so we're not reinventing numexpr. Be deliberate about which set each filter can use.
3. Cross-level expansion vs. the read floor. Expanding a coarse mask to photon rate (repeat verdict count times) is an O(n_photons) materialization — fine at aggregation time, but it fights the very memory win Phase C exists for if done before the read. The planner should let a coarse flag bound/skip the read in the same pass it filters, not materialize a full photon mask first.
4. Targeted reads are only as good as chunk alignment. Your 2.9% is real but depends on HDF5 chunking; a scattered AOI can touch many chunks and the win collapses. You need a selectivity cost model with a fall-back to full read, plus a story for the Lambda fan-out: if one worker = one cell, multiple workers will each re-read the same granule's spatial-index level. Your open-Q6 (decode the index once per granule) is exactly this — I'd elevate it from "nice" to "required for the Lambda economics."
5. The empty-beam skip is most of the prize. Your table shows the dominant win is proving there's nothing to read (3.7 MB geoloc vs 57–245 MB coords) for small AOIs. That's achievable with far less machinery than full cross-level filtering.

Competing / alternative approaches worth weighing

• (C-lite MVP) Ship just the bounding-box → "is this granule/beam empty?" skip first, using only the spatial-index level — captures most of the measured benefit before building the full filter/link engine. Strong cost/benefit.
• Build-vs-buy the ATL path: SlideRule. It already does server-side ATL03 photon subsetting + signal_conf/podppd classification — i.e. exactly Phases A–C for ICESat-2. Even if zagg keeps its generic engine for other instruments, it's worth a paragraph on why we're rebuilding ATL subsetting rather than calling SlideRule for that one mission.
• kerchunk / VirtualiZarr for the read planner. Instead of a bespoke read_plan over h5coro byte-ranges, expose the HDF5 as a virtual Zarr and let Zarr/dask do chunk-aligned lazy slicing (+ array-level predicate masking). Moves "targeted reads" onto a maintained layer; tradeoff is less control and a dependency, but a lot less bespoke IO code. Given the catalog is already Arrow/stac-geoparquet, this fits the ecosystem you're already in.
• One-time repacking to chunked Zarr/Parquet. If a dataset is read many times, converting once to properly-chunked Zarr makes targeted reads + pushdown trivial and reusable (and helps Backend support (i.e., local processing support) #20/Refactor the per-cell aggregation handoff: sort/hash grouping + Arrow path (additive, benchmarked) #30). Doesn't fit the "read native granules on demand" model, but worth flagging as the alternative axis (preprocess vs. read-optimize).
• N-D flag column selector: start integer-index only (column: 0); add the named-enum map (column: land) as pure sugar later (your open-Q2). Don't block Phase A on the enum map.

Recommendation

1. Phase A now (multi-filter + eq/ne/in/not_in/ge/le/lt/gt + N-D integer column), base-level only — it's self-contained and unblocks the noise filter.
2. Prototype B's link narrowly for ATL03 with the contiguity assumption documented.
3. Before committing to C's bespoke planner, do a short spike: (a) the C-lite empty-skip MVP, and (b) a build-vs-buy read of SlideRule + kerchunk/VirtualiZarr for the read floor. Decide bespoke-vs-ecosystem with that in hand.
4. First, though — per your own phasing note — measure whether the just-landed Arrow reducer (Refactor the per-cell aggregation handoff: sort/hash grouping + Arrow path (additive, benchmarked) #30) alone gets the ATL03 SERC run under the 2 GB Lambda budget. That calibrates how urgently C is needed vs. A.

Happy to turn the agreed scope into a phased plan when you relabel. Marking waiting.

---

Generated by Claude Code

[espg]

Where the read planner lives. Extend _read_group, or a new read_plan
module the grid/runner call? (Affects the Lambda path; coordinate with the
arrow refactor that just landed.)

My lean is probably new read_plan , but open. Would like to hear pros / cons of each approach

The grouping + arrow reducer (#30 phases 1/2/2b) is now on main. First
measure whether arrow alone gets the ATL03 SERC run under the 2 GB Lambda
threshold — that calibrates urgency. The read floor is independent of arrow, so
hierarchical reads help regardless.

Didn't drop us below 2 GB lambda when implemented, so this issue is the path forward

Schema surface vs back-compat. Keep flat coordinates/variables/
quality_filter as sugar over a synthesized single photon level, or migrate
all configs to levels? (Lean: support both; synthesize a base level from the
flat form.)

Show me a sample config for what we have now, vs what we would migrate to if we migrated all configs to levels. What's the verbosity trade off, and how complicated is the mental model? One thing to flag here-- ATL06 doesn't have the same hierarchy, so what does that look like if it's config is migrated to levels?

link bakes in contiguous one-parent→one-finer tiling. Perfect for ATL03 photon/segment, but it's a real constraint: many-to-many maps, non-contiguous children, or flags keyed by time/value rather than index won't fit index_beg/count. Don't over-generalize now — just name the contiguity assumption explicitly and leave room for an explicit-index-array link variant later. Otherwise the abstraction will quietly mislead the next mission.

Not sure how this lands as we move beyond ATL03 to other missions. We should investigate (and prototype) with GEDI waveforms. Different mission context-- space station mounted, no repeat track. That should provide an example to push on the level of generality needed. Investigate and post some some notes on this point.

Don't grow a second query DSL. The {op, value/values} predicate schema overlaps the expression mechanism agg fields already use. Suggestion: keep structured predicates for anything that must be pushed down to bound a read (those need to be machine-inspectable), but allow an expression escape hatch for base-level-only filters so we're not reinventing numexpr. Be deliberate about which set each filter can use.

"Don't grow a second query DSL" seems reasonable. Also like the "Suggestion: keep structured predicates for anything that must be pushed down to bound a read", which we should keep. Can you explain "allow an expression escape hatch for base-level-only filters so we're not reinventing numexpr. Be deliberate about which set each filter can use." in more detail?

The planner should let a coarse flag bound/skip the read in the same pass it filters, not materialize a full photon mask first.

Makes sense, I agree.

You need a selectivity cost model with a fall-back to full read, plus a story for the Lambda fan-out: if one worker = one cell, multiple workers will each re-read the same granule's spatial-index level.

yup. don't mind multiple workers re-reading spatial-index level, especially at the coarse level (i.e., the geolocation group for the ATL03 example).

Your open-Q6 (decode the index once per granule) is exactly this — I'd elevate it from "nice" to "required for the Lambda economics."

I don't understand this. explain

Decode the spatial-index coords once per granule and reuse across beams
where shared (ATL: per-beam; other missions may share).

or this. explain. the geolocation group is nested per beam, so what are you proposing to reuse here?

The empty-beam skip is most of the prize. Your table shows the dominant win is proving there's nothing to read (3.7 MB geoloc vs 57–245 MB coords) for small AOIs. That's achievable with far less machinery than full cross-level filtering.

What's your point? What do you mean by " full cross-level filtering"?

(C-lite MVP) Ship just the bounding-box → "is this granule/beam empty?" skip first, using only the spatial-index level — captures most of the measured benefit before building the full filter/link engine. Strong cost/benefit.

We already do this-- not for empty beams, but for is the granule empty. That's what the nasa cmr to shard map does; tells us which granules are expected to be non-empty. I fail to see what "Ship just the bounding-box → "is this granule/beam empty?" skip first" gets us that isn't already done upstream in the catalog / shard map step.

Build-vs-buy the ATL path: SlideRule. It already does server-side ATL03 photon subsetting + signal_conf/podppd classification — i.e. exactly Phases A–C for ICESat-2. Even if zagg keeps its generic engine for other instruments, it's worth a paragraph on why we're rebuilding ATL subsetting rather than calling SlideRule for that one mission.

zagg is being funded by nasa sliderule -- so this is us. We are prototyping a more efficient pipeline that will become part of sliderule. So no, we can differ this to hitting nasa sliderule. Any we don't need any prose addressing why we're rebuilding ATL subsetting

kerchunk / VirtualiZarr for the read planner. Instead of a bespoke read_plan over h5coro byte-ranges, expose the HDF5 as a virtual Zarr and let Zarr/dask do chunk-aligned lazy slicing (+ array-level predicate masking). Moves "targeted reads" onto a maintained layer; tradeoff is less control and a dependency, but a lot less bespoke IO code. Given the catalog is already Arrow/stac-geoparquet, this fits the ecosystem you're already in.

This is in discussion. Doing VirtualiZarr over the ATL03 catalog would mean building a VirtualiZarr over that dataset. We may go down this path... the thing that would determine that is how expensive zagg is in the lambda space (i.e., how much $ and gb/seconds do typical operations cost). To know that answer, we need to finish this prototype so we run the experiment and know what the zagg pricing is. If we did do virtualization, it would still nest with zagg -- we'd just add it as another reader backend like h5coro. So valid point, but out of scope until this issue is completed, merged, tested, and benchmarked.

One-time repacking to chunked Zarr/Parquet. If a dataset is read many times, converting once to properly-chunked Zarr makes targeted reads + pushdown trivial and reusable (and helps #20). Doesn't fit the "read native granules on demand" model, but worth flagging as the alternative axis (preprocess vs. read-optimize).

...what do you mean? For the entire ATL03 data, this is petabytes. Possibly a discussion years from now, with a strong user / use case"

N-D flag column selector: start integer-index only (column: 0); add the named-enum map (column: land) as pure sugar later (your open-Q2). Don't block Phase A on the enum map.

Explain this.

[espg]

🤖 from Claude

Answering your batch in order. Where I overstated something earlier I say so plainly.

Confirmed scope

Arrow alone didn't get ATL03 SERC under the 2 GB Lambda ceiling → this issue (the read floor) is the path forward, independent of the reducer. Agreed, and it sets the urgency for Phases B/C.

---

1. Where the read planner lives — read_plan module vs extend _read_group

	extend _read_group	new read_plan module (my lean, = yours)
Cohesion	read+filter stay in one place	"what to read" (plan) split from "how to read" (executor)
Testability	planner only testable through an S3 read	planner is a pure function — unit-testable offline with no creds (your prototype's index translation is exactly this)
Backend seam	adding VirtualiZarr later means surgery inside _read_group	clean seam: emit one plan, swap executors (h5coro now, VirtualiZarr later as another reader backend)
Lambda path	the OOM-critical path grows planning logic inline	_read_group stays thin: it executes a plan
Cost	none	one small ReadPlan dataclass threaded through runner/grid; one extra module

Recommendation: new read_plan module that PRODUCES an inspectable plan (parent runs → base-level slices → chunk lists → any coarse-flag ranges), consumed by a thin executor kept next to _read_group so the actual h5coro byte-range calls stay in one place. The deciding factor is exactly the offline-testability + the backend seam you already flagged for the possible VirtualiZarr path. So: planning is new and pure; execution stays where reads already live.

9. N-D flag column selector — why integer-only first

signal_conf_ph is 2-D: [n_photons, 5], the 5 columns being surface types (0 land, 1 ocean, 2 sea-ice, 3 land-ice, 4 inland-water). A filter must pick a column then compare, e.g. land = column 0, != 0. Two ways to name the column:

• (a) integer column: 0 — direct, no lookup table.
• (b) named column: land — needs a per-dataset enum map (land→0, sea_ice→2, …) maintained in the engine.

"Integer-only first" = ship Phase A accepting just column: 0. That fully unblocks signal_conf_ph != 0 with zero extra machinery; the named-enum map is pure sugar layered on top later. Don't make Phase A wait on building/maintaining per-mission enum tables it doesn't need to function.

5. "Expression escape hatch for base-level-only filters" — in detail

There are two filter languages, and the distinction is can the planner inspect it to bound a read?

• Structured predicates {op, value/values, column} are machine-inspectable. The planner can read them and decide what to skip/bound (this filter touches podppd_flag, so I can drop those segments before reading their photons). Pushdown-eligible filters MUST be structured.
• Expressions (the expression mechanism agg fields already use) handle arbitrary boolean logic over multiple base arrays — e.g. (h_ph > 0) & (signal_conf_ph[:,0] != 0) & (dem_h - h_ph < 50) — trivial to write, but opaque to the planner.

The "escape hatch" = allow a filter phrased as an expression only when it is base-level (it runs at aggregation time on arrays already read, so it never needs to be pushed down or inspected). That gives you full numexpr-style power for complex base-level masks without reinventing a second query DSL for the structured predicates. "Be deliberate about which set each filter can use" = make it explicit in the schema/engine that an expression filter forfeits pushdown (aggregation-time only), while structured predicates are the only kind eligible to bound a read. One language for inspectable pushdown, one escape hatch for rich base-level masks — never both jobs in one grammar.

6 & "decode once per granule" — you're right, I overstated this

You caught a real imprecision. The geolocation group is nested per-beam, so within an ATL03 granule there is nothing shared across beams — each beam carries its own geolocation. So my "reuse across beams" claim does not apply to ATL03; withdraw it. And since you're fine with workers re-reading the (small, ~3.7 MB) coarse level, the "elevate Q6 to required for Lambda economics" framing was too strong — retract that too. The only real, modest win that remains for ATL03 is within a single worker: decode that beam's spatial-index coords once and reuse them for both the empty-check and the read-bounding, rather than reading/decoding twice in the same invocation. That's a nice-to-have, not a requirement. "Reuse across beams" only ever mattered for a hypothetical mission with one granule-level coarse index shared by all sub-structures — which, per #4 below, ATL doesn't have and GEDI doesn't either.

7 & 8 — "full cross-level filtering" and the empty-beam skip (and why it's not separate machinery)

• "Full cross-level filtering" = the complete Phase B machinery: read a coarse-rate flag (podppd_flag at 20 m segment rate), build a coarse mask, and expand it down to photon rate via the link (repeat each segment's verdict count times) to AND with photon-rate filters. That's the heavyweight part.
• My point in 8 was: the dominant measured memory win — proving a beam has nothing to read (3.7 MB geoloc vs 57–245 MB coords) — needs far less than that. But you're right that granule-level emptiness is already handled upstream by the CMR→shard-map step. So the only thing beam-level adds is: a granule that passes the granule filter still has 6 beams, and for a 10 km AOI typically only 1–2 actually cross it (your table: gt1l/gt1r had photons, gt2l–gt3r had zero). The catalog can't see that; without it you'd still read the full 57–245 MB coord array per empty beam just to discover it's empty.
• The honest conclusion: this is not a separate phase or separate machinery. Phase C reads the small per-beam spatial-index level anyway to bound the read — the empty-beam case falls out for free (matched-parent-range is empty → read nothing). So I'm retracting the "C-lite empty-skip MVP" as a standalone step: it isn't one. It's a byproduct of Phase C, on top of the granule skip you already have. No standalone work to scope there.

3. Flat config now vs. migrated-to-levels — verbosity & ATL06

Today (flat, e.g. an ATL06-style source):

data_source:
  reader: h5coro
  groups: [gt1l, gt1r, gt2l, gt2r, gt3l, gt3r]
  coordinates: { latitude: latitude, longitude: longitude }
  variables:  { h_li: h_li }
  quality_filter: { dataset: atl06_quality_summary, value: 0 }

Same source migrated to levels (single level, no hierarchy — the ATL06 case):

data_source:
  reader: h5coro
  groups: [gt1l, gt1r, gt2l, gt2r, gt3l, gt3r]
  base_level: seg
  levels:
    seg:
      path: "/{group}/land_ice_segments"
      coordinates: { latitude: latitude, longitude: longitude }
      variables:  { h_li: h_li }
  filters:
    - { level: seg, dataset: atl06_quality_summary, keep: { op: eq, value: 0 } }

For a non-hierarchical product like ATL06, levels buys nothing and costs ~3 extra lines + a level name you have to invent — strictly more verbose, same mental model plus an indirection. The hierarchy only earns its keep on ATL03 (photon ← segment), where link is doing real work.

So: keep the flat form as first-class sugar; synthesize a single base level from it internally; do NOT force-migrate ATL06. The engine treats coordinates/variables/quality_filter as exactly "one base level, no link, one op: eq filter." Mental model stays "flat datasets are flat; declare levels only when you actually have linked resolutions." That's the lowest-surprise path and it makes the ATL03 hierarchy opt-in rather than a tax on every config.

4. GEDI waveforms — generality notes (investigated)

GEDI (ISS-mounted full-waveform lidar) is a good stress test precisely because it differs from ATL03. What I found:

• The contiguous index_beg/count link generalizes cleanly to GEDI L1B. L1B rxwaveform is a single flattened 1-D array of all return-waveform samples across all shots, tiled by rx_sample_start_index (1-based) + rx_sample_count per shot — the exact same parent(shot)→children(samples) contiguous pattern as ATL03 ph_index_beg/segment_ph_cnt, with sum(count) == n_samples. So the primitive isn't ATL-specific; that's reassuring.
• But GEDI has no intermediate coarse-rate geolocation level. Geolocation is per-shot (per footprint), not at a coarser segment rate. So the ATL03 podppd_flag case — a coarse-rate flag expanded down to fine rate — has no GEDI analog: GEDI quality (quality_flag, degrade_flag, sensitivity) is already at observation (shot) rate, i.e. base-level. The cross-level filter expansion (Phase B) is ATL03-specific, absent in GEDI.
• The targeted-read win is narrower for GEDI. For the common aggregation targets (L2A height metrics / RH percentiles, L4A footprint biomass) the data is already per-shot scalar with no sub-level — so GEDI-L2A is effectively single-level, like ATL06, declared as one base level with no link. Only L1B waveform aggregation (shot → sample) benefits from the shot-bounded read, and that's the same shape as ATL03 photon bounding.
• ISS orbit specifics: precessing, non-repeat ground track, ±51.6° latitude only (no polar — no GEDI analog to the 88°S hard case). 8 beams, each its own HDF5 group (like ATL03's 6) — same per-group repeat structure, and (per Update lambda function to write directly to zarr #6 above) no cross-beam shared index either.

What this says about the level of generality to build now: the engine needs exactly three optional facets, and GEDI exercises a different subset than ATL03 — which is the argument for making them independent and optional:

1. levels + contiguous link (index_beg/count/base) — shared by ATL03 photon/segment and GEDI L1B shot/sample. Build it, but name the contiguity assumption explicitly and leave a slot for an explicit-index-array link variant later.
2. coarse→fine filter expansion (Phase B) — ATL03-only; GEDI doesn't need it. Make it an optional facet keyed on a coarse-rate flag level, not a baked-in assumption.
3. single base level, no link — GEDI-L2A and ATL06 both live here, identical to the flat form.

Net: don't over-generalize the link (contiguous-only is fine for both missions we can see), but keep Phase B optional, because the very next mission (GEDI) skips it entirely.

On the deferred items — acknowledged

SlideRule = this is us (zagg is NASA-SlideRule-funded and will fold back in), so no build-vs-buy prose and we defer hitting SlideRule. VirtualiZarr stays out of scope until this prototype is finished/benchmarked and we know the Lambda $/GB-s — and if we do it, it nests as just another reader backend (which is exactly why the read_plan/executor split in #1 is worth it now). One-time repacking is a years-out, strong-use-case conversation given ATL03 is petabyte-scale. Coarse flags should bound/skip in the same pass (agreed); workers re-reading the coarse level is fine.

Suggested next step

The scope feels converged: Phase A (multi-filter + eq/ne/in/not_in/ge/le/lt/gt + integer N-D column, base-level only — unblocks signal_conf_ph != 0), then Phase B (levels + contiguous link + optional coarse→fine expansion), then Phase C (targeted reads via a pure read_plan module + thin executor; empty-beam skip falls out for free). Relabel discuss → plan and I'll turn this into a phased implementation plan (files touched, acceptance criteria, the ReadPlan dataclass shape). Marking waiting on your read.

---

Generated by Claude Code

[espg]

Recommendation: new read_plan module that PRODUCES an inspectable plan (parent runs → base-level slices → chunk lists → any coarse-flag ranges), consumed by a thin executor kept next to _read_group so the actual h5coro byte-range calls stay in one place.

Approved.

The honest conclusion: this is not a separate phase or separate machinery. Phase C reads the small per-beam spatial-index level anyway to bound the read — the empty-beam case falls out for free (matched-parent-range is empty → read nothing). So I'm retracting the "C-lite empty-skip MVP" as a standalone step: it isn't one. It's a byproduct of Phase C, on top of the granule skip you already have. No standalone work to scope there.

great.

So: keep the flat form as first-class sugar; synthesize a single base level from it internally; do NOT force-migrate ATL06. The engine treats coordinates/variables/quality_filter as exactly "one base level, no link, one op: eq filter." Mental model stays "flat datasets are flat; declare levels only when you actually have linked resolutions." That's the lowest-surprise path and it makes the ATL03 hierarchy opt-in rather than a tax on every config.

agree-- right now atl03 is the only case that requires link. The common case is that we don't have a hierarchy to unwrap (i.e., are single level).

[espg]

🤖 from Claude

Scope has converged on the prior thread (you approved: flat form stays first-class sugar with a synthesized single base level; structured predicates for pushdown + an expression escape hatch for base-level-only masks; a pure read_plan module producing an inspectable plan consumed by a thin executor next to _read_group; contiguous link is enough for ATL03 and GEDI L1B, and the coarse→fine expansion stays optional/ATL03-only). Here is the phased implementation plan.

Phase A — base-level multi-filter + N-D integer column (self-contained; unblocks signal_conf_ph noise filter)

• config.py: add an ordered filters: list of structured predicates {level?, dataset, column?, op, value|values, keep|drop?}, ops eq / ne / in / not_in / ge / le / lt / gt. Keep flat quality_filter as sugar that synthesizes one base-level op: eq filter (ATL06 unchanged). Allow an expression filter variant that is base-level only, aggregation-time, and forfeits pushdown (the escape hatch — opaque to the planner, so never read-bounding). Validation: reject unknown ops; require column for 2-D flags, reject column on 1-D; type-check value/values.
• processing.py: generalize _quality_mask → apply the ordered predicate list as ANDed masks, with N-D integer column slicing before compare. Keep the ATL06 single-eq path byte-for-byte identical.
• Acceptance: signal_conf_ph 2-D filter (e.g. TEP != -2); multiple ANDed filters; integer column: N; flat configs unchanged; ATL06 byte-identical.
• Files: src/zagg/config.py, src/zagg/processing.py, tests/test_config.py, tests/test_processing.py.

Phase B — levels + contiguous link

• config.py: optional levels: {name: {path, coordinates, variables}} + base_level + link: {base, index_beg, count, reference_index?} declaring contiguous parent→child tiling. Document the contiguity assumption explicitly and leave a slot for a future explicit-index-array link variant. Flat form ⇒ single base level, no link (the common case: ATL06, GEDI-L2A). Optional coarse→fine filter expansion (ATL03-only; GEDI skips it): expand a coarse-rate flag mask to base rate via the link (repeat each parent verdict count times) and AND with base-rate filters — done in the same pass that bounds the read, never as a pre-materialized full photon mask.
• Acceptance: ATL03 photon←segment link with the verified index translation (ph_index_beg-1 + reference_photon_index-1, sum(segment_ph_cnt) == n_photons); podppd_flag coarse expansion; single-level configs identical to the flat form.
• Files: src/zagg/config.py, src/zagg/processing.py (+ reader), tests.

Phase C — targeted reads: pure read_plan module + thin executor (approved split)

• New src/zagg/read_plan.py: a pure function producing an inspectable ReadPlan (parent runs → base-level slices → chunk lists → coarse-flag ranges), unit-testable offline with no creds. Only structured predicates are pushdown-eligible; expression filters forfeit pushdown. Include a selectivity cost model with fall-back to full read (a scattered AOI that touches many chunks shouldn't lose to a full read). The empty-beam skip falls out for free (matched parent-range empty ⇒ read nothing) — not separate machinery, layered on top of the existing CMR→shard-map granule skip. Within a single worker, decode the small (~3.7 MB) per-beam spatial-index level once and reuse for both the empty-check and the read-bounding (nice-to-have, not required; workers re-reading the coarse level across cells is fine).
• Thin executor kept next to _read_group so the actual h5coro byte-range calls stay in one place — it just executes a plan. This is the seam that lets a future VirtualiZarr reader drop in as another backend (out of scope here).
• Proposed ReadPlan shape (for your sign-off):

  @dataclass
  class GroupPlan:
      parent_runs:   list[tuple[int, int]]   # (start, count) over the coarse/parent level
      base_slices:   list[tuple[int, int]]   # (start, count) over the base level
      chunk_ids:     list[int]               # HDF5 chunks to fetch
      coarse_ranges: list[tuple[int, int]]   # coarse-flag ranges to read for expansion
      selectivity:   float                   # est. fraction of the granule touched
      fall_back_full: bool                   # true ⇒ ignore the slices, read whole array
  
  @dataclass
  class ReadPlan:
      groups: dict[str, GroupPlan]           # per beam/group

• Acceptance: pure planner unit tests (offline); ATL03 SERC run under the 2 GB Lambda ceiling (the calibration target — Arrow alone didn't get under it, which is why this issue is the path forward).
• Files: new src/zagg/read_plan.py, src/zagg/processing.py (executor near _read_group), runner/grid threading, tests/test_read_plan.py.

Explicitly out of scope (recorded from the thread)

• SlideRule build-vs-buy prose — zagg is SlideRule (NASA-funded, folds back in); no prose, defer hitting SlideRule.
• VirtualiZarr / kerchunk — revisit only after this prototype is benchmarked and we know Lambda $/GB-s; if adopted it nests as another reader backend (which is exactly why the read_plan/executor split is worth it now).
• One-time repacking to chunked Zarr/Parquet — years out, petabyte-scale, needs a strong use case.
• Named-enum column map (column: land) — pure sugar over integer column: 0, layered later; Phase A is integer-only.
• GEDI generality — the contiguous link generalizes to GEDI L1B shot/sample; Phase B's coarse→fine expansion is ATL03-only and stays optional (GEDI quality is already base-rate).

Open questions

1. PR shape — ship Phase A as its own PR first (self-contained, unblocks the noise filter), then B, then C (one issue → one PR per §5)? Or phase all three within a single PR?
2. expression escape hatch — confirm we allow base-level-only expression filters that explicitly forfeit pushdown (one inspectable structured-predicate language for read-bounding, one escape hatch for rich base-level masks — never both jobs in one grammar).
3. ReadPlan field set above — good as-is, or adjust?

Marking waiting on your read. No code written (this is plan).

---

Generated by Claude Code