Skip to content

Promote differing non-kind attrs to a coordinate when concatenating #987

Description

@d-chambers

When patches are placed side by side, an attribute the members disagree on is currently resolved by conflict: "raise" refuses, "keep_first" keeps one value, "drop" throws them all away. All three lose information that the output could carry, because concatenation preserves the distinction between its members — each member still occupies a known range of the concatenated dimension.

The proposal is a fourth policy, conflict="promote": an attribute whose members differ becomes a non-dimensional coordinate along the concatenated dimension, holding each member's value over that member's samples.

a = patch.update_attrs(quality="good")
b = other.update_attrs(quality="poor")

out = dc.spool([a, b]).concatenate(time=None, conflict="promote")

out[0].get_coord("quality")     # "good" over a's samples, "poor" over b's
out[0].attrs.get("quality")     # None — it is a coordinate now, not an attr

Why

keep_first fabricates provenance: the half of the array that came from b comes out claiming a's value. drop deletes the answer entirely. Promotion is the only resolution that keeps the output honest about all of its data, and it is exactly the information a concatenation has and a merge does not.

Round trip

The acceptance test, from the design note:

Collapse a constant coordinate to an attr on each of N patches, concatenate them, and the original coordinate comes back exactly — including empty values.

patches = [p.update_attrs(quality=q) for p, q in zip(patches, values)]
out = dc.spool(patches).concatenate(time=None, conflict="promote")
assert np.all(out[0].get_coord("quality").values == expected)

Empty values must survive as empty rather than being filled from a neighbour — see #983, which established that a missing value is a value where patches are partitioned.

Scope decisions

Implementation notes

The planner is the harder half. build_concat_plan/build_chunk_plan describe an output's attrs as row values in the catalog, and a promoted attribute is no longer an attr of the output at all — the catalog has to describe it as a coordinate whose values it knows only per member. derived_catalog already records per-member envelope columns, so the member rows carry what is needed; what is missing is a way for a plan row to say "this name is a coordinate here, not an attr" so that get_contents() and select() do not keep offering it as a scalar.

Assembly is the easier half: concatenate_planned knows each member's value and each member's extent along the dimension, so it can build the coordinate directly alongside the existing rider-joining logic in _concatenate_group.

Worth deciding during implementation: whether a promoted coordinate is worth the memory (one entry per sample, so a string array the length of the concatenated dimension), or whether it should use a run-length/segmented coordinate — CoordSegmented already exists for the non-uniform case.

Follow-up to #983 and #984; the design note is .scratch/kind-empty-rule.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions