Skip to content

Segmented chunk rows can disagree with their patches under conflict='keep_first' #986

Description

@d-chambers

_carried_columns resolves each attribute once per partition, but a segmented chunk turns one partition into many outputs. Every output of that partition is then stamped with the same carried value, while loading an output returns the attrs of the members that actually went into it. Under conflict="keep_first" (or "drop") the two disagree.

Reproduction

import numpy as np
import dascore as dc

t0 = np.datetime64("2020-01-01", "ns")
p1 = dc.get_example_patch(time_min=t0).update_attrs(foo="a")
time = p1.get_coord("time")
p2 = dc.get_example_patch(time_min=time.max() + time.step).update_attrs(foo="b")

spool = dc.spool([p1, p2])
duration = (time.max() - time.min() + time.step) / np.timedelta64(1, "s")
out = spool.chunk(time=duration, conflict="keep_first")

print(list(out.get_contents()["foo"]))   # ['a', 'a']
print([p.attrs.foo for p in out])        # ['a', 'b']

The second output's row says foo="a" and the patch says foo="b". The consequence is worse than a cosmetic mismatch, because selection reads the rows:

print([p.attrs.foo for p in out.select(foo="a")])   # ['a', 'b']

select(foo="a") yields a patch whose foo is "b".

Cause

_carried_columns (dascore/utils/chunk_plan.py) aggregates by codes, which label partitions, and the carried value rides onto every output the partition produces. In merge mode a partition is one output, so the two coincide; in segment mode they do not. conflict="raise" hides it, since a partition with differing values never gets that far.

Scope

Pre-existing — reproduced unchanged on dev at 4ce599f3, and before that too. PlanResolver._stamp never covered it: it only ever filled attributes the assembled patch was missing, and here the patch states its own value, so the fill was skipped.

Found by a Codex review of #984, which touches _carried_columns for an unrelated reason (missing-value semantics) and does not change this behaviour either way.

Fix sketch

Resolve carried columns per output rather than per partition, or — if a partition-level value is deliberate — leave the attribute out of the rows of any partition whose members disagree, so the catalog never claims something the patch contradicts. Either way select and get_contents must not describe a patch by a value it does not hold.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions