Repository navigation
Conversation
ewoodbury
left a comment
There was a problem hiding this comment.
Contributing my review here, changes look solid. Agree with the approach of promoting the bound where it is compared with a checked multiply, and it follows #3483. I confirmed the new evaluator tests fail with the date arms reverted, so they pin the fix.
Main asks from me before merge:
- Handle the spec's 4-byte bound case under a timestamp/timestamp_ns schema. Those manifests still fail to load.
- Settle whether the partition-value half belongs here. My reading of the spec is that it rules out this promotion for identity partitions.
- Pin the pruned side of <, > and IN.
Date columns promoted to timestamp or timestamp_ns keep day-count lower/upper bounds in older manifests. Convert those bounds to midnight of each day (including 4-byte date payloads under a timestamp schema) so bound pruning keeps working across the promotion. Dates that overflow the target range return DataInvalid. Partition date→timestamp is not promoted: the spec rules that out for identity transforms. Closes apache#3503
84dd5ba to
8fc8a81
Compare
ewoodbury
left a comment
There was a problem hiding this comment.
LGTM! Thanks for the updates, all addressed and I reran the mutation checks on the new tests. Leaving to a maintainer for the final round
| pub(crate) const NANOSECONDS_PER_DAY: i64 = 86_400_000_000_000; | ||
|
|
||
| /// Converts `days` from the epoch to a timestamp literal in units of `units_per_day`. | ||
| pub(crate) fn date_to_timestamp( |
There was a problem hiding this comment.
we should move this to temporal.rs, there are already conversions like this
There was a problem hiding this comment.
Moved in 4adddcc. The day constants and the conversion now live in spec/values/temporal.rs under date as days_to_microseconds / days_to_nanoseconds (both return None on i64 overflow, next to date_to_days and friends). datum.rs keeps only a small private date_to_timestamp(days, target) that picks the unit from target and turns the overflow into DataInvalid, which both the 4-byte bound decode and Datum::promote_to use.
(AI-assisted reply.)
| /// # Errors | ||
| /// Returns `DataInvalid` if a date is outside the range of the target timestamp | ||
| /// type. | ||
| pub(crate) fn promote_to(&self, target: &PrimitiveType) -> Result<Option<Datum>> { |
There was a problem hiding this comment.
PrimitiveLiteral::promote_to and Datum::promote_to now cover different promotion sets under the same name. we could add some doc to PrimitiveLiteral::promote_to to help differentiate them
There was a problem hiding this comment.
Added a paragraph to PrimitiveLiteral::promote_to in 4adddcc: a bare literal does not know its source type, so it only covers int -> long and float -> double, and it points to Datum::promote_to, which matches on the source type and adds date -> timestamp / timestamp_ns.
(AI-assisted reply.)
Address review: the day-count conversion helpers now live next to the other date conversions in spec/values/temporal.rs, PrimitiveLiteral::promote_to documents that it does not cover date -> timestamp (use Datum::promote_to), and the column-stats bound promotion in scan/task.rs handles the new Result return of Datum::promote_to.
8fc8a81 to
f306c2f
Compare
|
Rebased onto current main (
Local on this head: (AI-assisted change and comment.) |
There was a problem hiding this comment.
🟡 Changes recommended
The changed return type breaks type-checking in the scan module because bound_as_column_type still expects an Option; the caller and statistics-building path must handle the Result with an explicit promotion-overflow policy.
1 open issue
What changed in this PR
This PR addresses incorrect file pruning after a date column is promoted to timestamp or timestamp_ns, extending the earlier numeric-promotion fix.
Changes:
- Adds checked conversion of date bounds to timestamp units, reporting overflow as
DataInvalid. - Handles legacy four-byte date bounds during decoding and promotes date-typed bounds during metrics evaluation.
- Adds conversion, manifest decoding, pruning, and overflow tests.
The PR description could be shortened by removing the per-function and per-test walkthroughs.
| File | Description |
|---|---|
crates/iceberg/src/spec/values/tests.rs |
Tests date promotion, overflow, and legacy bound decoding. |
crates/iceberg/src/spec/values/primitive.rs |
Adds shared checked date-to-timestamp conversion. |
crates/iceberg/src/spec/values/datum.rs |
Extends bound decoding and makes promotion fallible. |
crates/iceberg/src/spec/manifest/mod.rs |
Tests date bounds decoded under timestamp schemas. |
crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs |
Promotes date bounds before comparisons and propagates errors. |
Review effort: Balanced
| /// # Errors | ||
| /// Returns `DataInvalid` if a date is outside the range of the target timestamp | ||
| /// type. | ||
| pub(crate) fn promote_to(&self, target: &PrimitiveType) -> Result<Option<Datum>> { |
|
The caller is already updated at |
JeonDaehong
left a comment
There was a problem hiding this comment.
Ran the iceberg lib tests on 4adddcc, all green. With promote_bound returning the bound unchanged, eval_promotes_date_bounds and eval_errors_on_date_bound_out_of_timestamp_ns_range fail along with the int and float promotion tests, so the date arms are pinned down. LGTM.

Which issue does this PR close?
What changes are included in this PR?
Follow-up to #3483. With v3 type promotion a
datecolumn can becometimestamportimestamp_ns, but older manifests keep day-count lower/upper bounds. Compared as raw numbers those day counts sit far below any timestamp in micros/nanos, so bound pruning wrongly drops files.Datum::promote_toconverts a date (days from epoch) totimestamp(micros) ortimestamp_ns(nanos) via checked multiplication; a date outside the target range returnsDataInvalid. Matching is on(self.data_type(), target)so a bareIntis not treated as a date.Datum::try_from_bytesaccepts 4-byte payloads under atimestamp/timestamp_nsschema (manifests that carried an older date bound forward) and reuses the same conversion.InclusiveMetricsEvaluatorpromotes date lower/upper bounds before comparing them to a timestamp predicate (field name attached on error) and propagates overflow asDataInvalid.Partition
date→timestamp/timestamp_nsis not promoted: the spec rules that out for identity transforms.timestamptztargets are left alone (date → timestamptz is not an allowed promotion).Are these changes tested?
Yes, new / folded unit tests:
spec::values::tests::test_datum_promote_date: table-driven date → timestamp / timestamp_ns (including negative),timestamp_nsedges day 106751 fits / 106752 doesn't, overflow →DataInvalid, bareIntdoes not promote.spec::values::tests::avro_bytes_timestamp_from_dateandtest_parse_manifest_promotes_date_bounds_under_timestamp_schema: 4-byte date bounds under a timestamp schema.inclusive_metrics_evaluator::eval_promotes_date_bounds: table-driven</>/=/INwith inside (kept) and outside (pruned) literals for both targets; overflow assertsErrorKind::DataInvalid.AI Disclosure
I used an AI coding assistant to help write the code and tests. I reviewed the changes and ran the tests locally.