_carried_columns resolves each attribute once per partition, but a segmented chunk turns one partition into many outputs. Every output of that partition is then stamped with the same carried value, while loading an output returns the attrs of the members that actually went into it. Under conflict="keep_first" (or "drop") the two disagree.
Reproduction
import numpy as np
import dascore as dc
t0 = np.datetime64("2020-01-01", "ns")
p1 = dc.get_example_patch(time_min=t0).update_attrs(foo="a")
time = p1.get_coord("time")
p2 = dc.get_example_patch(time_min=time.max() + time.step).update_attrs(foo="b")
spool = dc.spool([p1, p2])
duration = (time.max() - time.min() + time.step) / np.timedelta64(1, "s")
out = spool.chunk(time=duration, conflict="keep_first")
print(list(out.get_contents()["foo"])) # ['a', 'a']
print([p.attrs.foo for p in out]) # ['a', 'b']
The second output's row says foo="a" and the patch says foo="b". The consequence is worse than a cosmetic mismatch, because selection reads the rows:
print([p.attrs.foo for p in out.select(foo="a")]) # ['a', 'b']
select(foo="a") yields a patch whose foo is "b".
Cause
_carried_columns (dascore/utils/chunk_plan.py) aggregates by codes, which label partitions, and the carried value rides onto every output the partition produces. In merge mode a partition is one output, so the two coincide; in segment mode they do not. conflict="raise" hides it, since a partition with differing values never gets that far.
Scope
Pre-existing — reproduced unchanged on dev at 4ce599f3, and before that too. PlanResolver._stamp never covered it: it only ever filled attributes the assembled patch was missing, and here the patch states its own value, so the fill was skipped.
Found by a Codex review of #984, which touches _carried_columns for an unrelated reason (missing-value semantics) and does not change this behaviour either way.
Fix sketch
Resolve carried columns per output rather than per partition, or — if a partition-level value is deliberate — leave the attribute out of the rows of any partition whose members disagree, so the catalog never claims something the patch contradicts. Either way select and get_contents must not describe a patch by a value it does not hold.
_carried_columnsresolves each attribute once per partition, but a segmented chunk turns one partition into many outputs. Every output of that partition is then stamped with the same carried value, while loading an output returns the attrs of the members that actually went into it. Underconflict="keep_first"(or"drop") the two disagree.Reproduction
The second output's row says
foo="a"and the patch saysfoo="b". The consequence is worse than a cosmetic mismatch, because selection reads the rows:select(foo="a")yields a patch whosefoois"b".Cause
_carried_columns(dascore/utils/chunk_plan.py) aggregates bycodes, which label partitions, and the carried value rides onto every output the partition produces. In merge mode a partition is one output, so the two coincide; in segment mode they do not.conflict="raise"hides it, since a partition with differing values never gets that far.Scope
Pre-existing — reproduced unchanged on
devat4ce599f3, and before that too.PlanResolver._stampnever covered it: it only ever filled attributes the assembled patch was missing, and here the patch states its own value, so the fill was skipped.Found by a Codex review of #984, which touches
_carried_columnsfor an unrelated reason (missing-value semantics) and does not change this behaviour either way.Fix sketch
Resolve carried columns per output rather than per partition, or — if a partition-level value is deliberate — leave the attribute out of the rows of any partition whose members disagree, so the catalog never claims something the patch contradicts. Either way
selectandget_contentsmust not describe a patch by a value it does not hold.