Skip to content

Native Iceberg scan returns NULL for an INT96 column read through a name mapping when other columns are selected #7212

Open

Description

@andygrove

Describe the bug

The native Iceberg scan returns NULL for an INT96 timestamp column when the data file has no Iceberg field ids, the table reads it through a name mapping (schema.name-mapping.default), and the query also selects other columns. Spark returns the values, and nothing fails, so the wrong result is silent.

  • SELECT ts on its own returns the right values.
  • The same file read without a name mapping returns the right values.
  • With micros (TIMESTAMP_MICROS) timestamps, the name-mapping path is correct.

Comet 1.1.0 returns the same NULLs (release jar, Spark 3.5.7, Iceberg 1.8.1), so this is not new in 1.2. Files like this come from add_files, migrate or snapshot over Parquet that Spark wrote with spark.sql.parquet.outputTimestampType=INT96.

Steps to reproduce

// Hadoop catalog `c`
spark.sql("CREATE TABLE c.db.t (id BIGINT, ts TIMESTAMP, s STRING) USING iceberg")
val table = Spark3Util.loadIcebergTable(spark, "c.db.t")
table.updateProperties()
  .set("schema.name-mapping.default", NameMappingParser.toJson(MappingUtil.create(table.schema())))
  .commit()

spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
spark.range(5).selectExpr("id", "timestamp_micros(id * 1000003) AS ts", "CAST(id AS STRING) AS s")
  .coalesce(1).write.parquet(dataPath)
val file = /* the single part file under dataPath */
table.newAppend().appendFile(
  DataFiles.builder(table.spec()).withPath(file.getPath.toString)
    .withFileSizeInBytes(file.getLen).withRecordCount(5)
    .withFormat(FileFormat.PARQUET).build()).commit()

spark.sql("REFRESH TABLE c.db.t")
spark.sql("SELECT id, ts, s FROM c.db.t").show()
  • Spark: ts holds the five timestamps.
  • Comet: ts is NULL in every row. SELECT ts FROM c.db.t is correct.

Expected behavior

The same values as Spark.

Additional context

Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression. Related: #6790 (NULL nested fields through a name mapping) and #6794 (nested fields with no ids and no mapping). Those cover nested columns; this one is a top-level INT96 column.

Activity

  1. added
    priority:criticalData corruption, silent wrong results, security issues
    area:scanParquet scan / data reading
    and removed on Oct 26, 2026
  2. added this to the 1.3.0 milestone on Oct 26, 2026
  3. sam-1112 commented on Nov 3, 2026

    @sam-1112
    Contributor

    I reproduced directly in iceberg-rust (b154603a, Arrow/parquet 59.3.2), without Comet. Projecting only ts works; adding other columns returns NULLs for ts. Projection rejects the INT96 timestamp against timestamptz, drops ts, and the transformer fills NULLs. Direct parquet-rs decoding is correct, pointing to iceberg-rust's projection/type reconciliation. So I opened an upstream issue: apache/iceberg-rust#3611.

  4. andygrove commented on Nov 20, 2026

    @andygrove
    MemberAuthor

    Update: the upstream fix is apache/iceberg-rust#3632, which closes iceberg-rust#3611 and also iceberg-rust#3616 (INT64 nanosecond timestamps read 1000x too large). It makes the INT96 coercion set the UTC timezone for timestamptz columns, so projection no longer drops them. It assigns field ids from the name mapping before it coerces, so the no-ids plus name-mapping case here is covered. mbutrovich confirmed in the review that a Comet-style filtered scan on that branch returns correct values. The PR is approved by one reviewer and the review rounds are done, and it is waiting on a re-review.

    Until the iceberg-rust pin includes #3632, this stays open. The pin in #7249 is based on b154603a, the commit where this was reproduced, so the DF 56 build will have the same bug. Once the pin moves, this issue needs a regression test for the repro above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Icebergarea:scanParquet scan / data readingbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions