Skip to content

A coordinate's fingerprint depends on how its scalars were spelled #978

Description

@d-chambers

Problem

A coordinate's fingerprint depends on how its scalars were spelled, not only on the values it holds. Two coordinates which compare equal, hold identical values and share a dtype can therefore carry different identities.

import numpy as np
from dascore.core.coords import get_coord

# 1. python type of the scalar
a = get_coord(start=0,   stop=10,   step=1.0)   # int start, float64 dtype
b = get_coord(start=0.0, stop=10.0, step=1.0)
a == b                          # True
a.dtype == b.dtype              # True
np.array_equal(a.values, b.values)  # True
a.fingerprint() == b.fingerprint()  # False   <-- different identity

# 2. precision of a time step
t0 = np.datetime64("2020-01-01", "ns")
c = get_coord(start=t0, stop=t0 + np.timedelta64(400, "ms"),
              step=np.timedelta64(4, "ms"))
d = get_coord(start=t0, stop=t0 + np.timedelta64(400_000_000, "ns"),
              step=np.timedelta64(4_000_000, "ns"))
c == d                          # True
np.array_equal(c.values, d.values)  # True
c.fingerprint() == d.fingerprint()  # False   <-- different identity

Why it matters

CoordRange.validate_start_stop_step_len (dascore/core/coords.py:1516) derives dtype from start + step but leaves start, stop and step as whatever objects the caller passed. _fingerprint_components (coords.py:1591) then hashes those scalars through _hash_scalar, which encodes the scalar's own type and precision. _get_fingerprintable_coord (coords.py:615) already normalizes units before hashing, precisely so metres and centimetres do not produce different identities — scalar type and time precision are the same class of difference and are not normalized.

Consequences:

  • CoordSummary conforms its values with ensure_consistent_dtype (coords.py:97), so a round-trip changes the identity: c.fingerprint() != c.to_summary().to_coord().fingerprint().
  • The index stores that fingerprint as a coordinate's def_key (dascore/io/index/ingest.py:109), so equal coordinates can be stored as two definitions rather than deduplicating, and an identity match can miss.

Real patch coordinates from the readers and get_example_patch are unaffected — they arrive as datetime64 ns and float arrays — so this is latent rather than actively breaking anything today. It surfaced while building the summary-level join for #972, where a prediction's identity must equal what re-ingesting the assembled patch records.

Fix, and why it is not a one-liner

Conforming the scalars in the validator, exactly as CoordSummary does, unifies both cases:

dtype = np.asarray(start + step).dtype
values["dtype"] = dtype
for name in ("start", "stop", "step"):
    values[name] = ensure_consistent_dtype(values[name], name, dtype)

Tried on dev: it makes the fingerprints agree, but

  • 4 tests fail — tests/test_core/test_coords.py::TestCoordRange::test_len_one_array_like_start_no_deprecation and three in tests/test_io/test_prodml/test_prodml_write.py::TestProdMLWriteTimePrecision, which depend on sub-nanosecond and out-of-range datetimes surviving unconverted;
  • it changes stored fingerprints, so def_keys written by an older version no longer match the ones a new version computes for the same coordinate. Existing indexes would need rebuilding, or the change needs an index version bump.

An alternative worth weighing is normalizing inside _hash_scalar instead of at construction: the stored scalars keep their spelling, only the hash canonicalizes. That avoids the prodml fallout but still changes fingerprints, so the migration question is the same.

Either way this deserves its own decision about index compatibility rather than riding along with #972.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions