You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg scan returns NULL for an INT96 column read through a name mapping when other columns are selected #7212
The native Iceberg scan returns NULL for an INT96 timestamp column when the data file has no Iceberg field ids, the table reads it through a name mapping (schema.name-mapping.default), and the query also selects other columns. Spark returns the values, and nothing fails, so the wrong result is silent.
SELECT ts on its own returns the right values.
The same file read without a name mapping returns the right values.
With micros (TIMESTAMP_MICROS) timestamps, the name-mapping path is correct.
Comet 1.1.0 returns the same NULLs (release jar, Spark 3.5.7, Iceberg 1.8.1), so this is not new in 1.2. Files like this come from add_files, migrate or snapshot over Parquet that Spark wrote with spark.sql.parquet.outputTimestampType=INT96.
Steps to reproduce
// Hadoop catalog `c`
spark.sql("CREATE TABLE c.db.t (id BIGINT, ts TIMESTAMP, s STRING) USING iceberg")
valtable=Spark3Util.loadIcebergTable(spark, "c.db.t")
table.updateProperties()
.set("schema.name-mapping.default", NameMappingParser.toJson(MappingUtil.create(table.schema())))
.commit()
spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
spark.range(5).selectExpr("id", "timestamp_micros(id * 1000003) AS ts", "CAST(id AS STRING) AS s")
.coalesce(1).write.parquet(dataPath)
valfile=/* the single part file under dataPath */
table.newAppend().appendFile(
DataFiles.builder(table.spec()).withPath(file.getPath.toString)
.withFileSizeInBytes(file.getLen).withRecordCount(5)
.withFormat(FileFormat.PARQUET).build()).commit()
spark.sql("REFRESH TABLE c.db.t")
spark.sql("SELECT id, ts, s FROM c.db.t").show()
Spark: ts holds the five timestamps.
Comet: ts is NULL in every row. SELECT ts FROM c.db.t is correct.
Expected behavior
The same values as Spark.
Additional context
Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression. Related: #6790 (NULL nested fields through a name mapping) and #6794 (nested fields with no ids and no mapping). Those cover nested columns; this one is a top-level INT96 column.
I reproduced directly in iceberg-rust (b154603a, Arrow/parquet 59.3.2), without Comet. Projecting only ts works; adding other columns returns NULLs for ts. Projection rejects the INT96 timestamp against timestamptz, drops ts, and the transformer fills NULLs. Direct parquet-rs decoding is correct, pointing to iceberg-rust's projection/type reconciliation. So I opened an upstream issue: apache/iceberg-rust#3611.
Update: the upstream fix is apache/iceberg-rust#3632, which closes iceberg-rust#3611 and also iceberg-rust#3616 (INT64 nanosecond timestamps read 1000x too large). It makes the INT96 coercion set the UTC timezone for timestamptz columns, so projection no longer drops them. It assigns field ids from the name mapping before it coerces, so the no-ids plus name-mapping case here is covered. mbutrovich confirmed in the review that a Comet-style filtered scan on that branch returns correct values. The PR is approved by one reviewer and the review rounds are done, and it is waiting on a re-review.
Until the iceberg-rust pin includes #3632, this stays open. The pin in #7249 is based on b154603a, the commit where this was reproduced, so the DF 56 build will have the same bug. Once the pin moves, this issue needs a regression test for the repro above.
Activity
sam-1112 commentedon Nov 3, 2026
I reproduced directly in iceberg-rust (
b154603a, Arrow/parquet 59.3.2), without Comet. Projecting onlytsworks; adding other columns returns NULLs forts. Projection rejects the INT96 timestamp againsttimestamptz, dropsts, and the transformer fills NULLs. Direct parquet-rs decoding is correct, pointing to iceberg-rust's projection/type reconciliation. So I opened an upstream issue: apache/iceberg-rust#3611.andygrove commentedon Nov 20, 2026
Update: the upstream fix is apache/iceberg-rust#3632, which closes iceberg-rust#3611 and also iceberg-rust#3616 (INT64 nanosecond timestamps read 1000x too large). It makes the INT96 coercion set the UTC timezone for
timestamptzcolumns, so projection no longer drops them. It assigns field ids from the name mapping before it coerces, so the no-ids plus name-mapping case here is covered. mbutrovich confirmed in the review that a Comet-style filtered scan on that branch returns correct values. The PR is approved by one reviewer and the review rounds are done, and it is waiting on a re-review.Until the iceberg-rust pin includes #3632, this stays open. The pin in #7249 is based on
b154603a, the commit where this was reproduced, so the DF 56 build will have the same bug. Once the pin moves, this issue needs a regression test for the repro above.