Gaps: selection discards segment information and the spool index cannot represent it

[d-chambers]

Summary

CoordSegmented is the type that records discontinuities in a coordinate, but nothing in the selection path produces it and nothing in the spool index can store it. The result is that gaps are representable on a Patch and invisible to a Spool, and the two disagree about what a selection returns.

Three related problems, in increasing order of consequence.

Background

CoordSegmented (dascore/core/coords.py:2148) exists precisely for this:

Segment boundaries record discontinuities (e.g. data gaps) without altering any value, which makes this the natural coordinate for data merged across nearly-contiguous blocks.

It appears to have been built for the merge direction — time chunks reassembled across gaps, which chunk can preserve. The selection direction produces structurally identical data and never reaches for it.

1. Patch.select on scattered values does not produce a segmented coord

import dascore as dc, numpy as np
from dascore.core.coords import concat_coords, get_coord

p = dc.get_example_patch()
d = p.coords.get_array("distance")

gappy = p.select(distance=np.r_[d[0:10], d[40:50]])   # two disjoint runs
type(gappy.coords.coord_map["distance"]).__name__
# -> 'CoordMonotonicArray'

seg = concat_coords(get_coord(values=d[0:10]), get_coord(values=d[40:50]))
type(seg).__name__, seg.segment_count
# -> ('CoordSegmented', 2)

The selection knows exactly where the discontinuity is and discards that knowledge. CoordMonotonicArray is not wrong, but it is less than what was known, and the type that holds the missing information already exists.

2. The spool index has nowhere to record segments

Even given a genuine CoordSegmented on the patch, the index keeps only an envelope:

sp = dc.spool([gappy.update_coords(distance=seg)])
df = sp.get_contents()
[c for c in df.columns if c.startswith("distance")]
# -> ['distance_min', 'distance_max', 'distance_step', 'distance_units']

df.iloc[0].distance_min, df.iloc[0].distance_max
# -> (0.0, 49.0)

len(sp.select(distance=(11, 39)))    # a range lying entirely inside the gap
# -> 1

The row claims channels the patch does not have, and a selection over the hole matches it. The gap survives on the patch and is lost on the way into the index, because a flat min/max/step schema has no place to put it.

3. Patch.select and Spool.select therefore disagree in arity

Same method name, same arguments, different number of results. Self-contained, using only dascore.examples:

import dascore as dc
from dascore.core.inventory import OpticalPathLabel
from dascore.examples import inventory_patch_pair

patch, inv = inventory_patch_pair()

# give the path a label whose values interleave along the fiber
array = inv.networks[0].fiber_arrays[0]
path = array.optical_paths[0]
leg = tuple(
    OpticalPathLabel(start_distance=s, end_distance=s + 50, group="leg",
                     value="down" if i % 2 == 0 else "up")
    for i, s in enumerate(range(100, 400, 50))
)
inv2 = inv.new(networks=(inv.networks[0].new(fiber_arrays=(
    array.new(optical_paths=(path.new(labels=path.labels + leg),)),)),)).check()

p = patch.enrich(inv2)
p.select(leg="down").shape
# -> (150, 2000)      ONE patch, three disjoint runs

sp = dc.spool(patch).attach_inventory(inv2).enrich()
len(sp.select(leg="down"))
# -> 3                THREE patches

The spool is not being fussy: splitting into contiguous runs is the only way to keep the index honest given problem 2. But the divergence is undocumented at the point of use — neither Spool.select nor Patch.select mentions that a non-contiguous match splits one patch into several.

This propagates. On a real archive, expand_by("leg") over 31 files produced 868 patches — 31 files x 14 boreholes x 2 legs — where a value-wise grouping would give 2. The factor is entirely the contiguity requirement.

What to decide

• Should selection emit CoordSegmented when the result is discontiguous? That is the smallest fix and makes problem 1 go away on its own.
• Should the index learn about segments — a segment count, a side table, anything that lets a row describe a gap? That is the larger change, and it is what would let Patch.select and Spool.select agree.
• Failing that, both select docstrings should say plainly that a non-contiguous match splits a patch at spool level and does not at patch level.

The third is worth doing regardless of whether the first two happen.

Environment

dascore 0.1.22.dev204+g13a43d334 (dev), python 3.13