Skip to content

fix: read partition source columns through PartitionExpr children - #92

Draft
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/partition-expr-children
Draft

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/partition-expr-children

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

PartitionExpr read its source columns by position in the input batch, but reported no children. When ProjectionPushdown merged the partition projection into the projection below it, it had no column references to rewrite, so the positions kept pointing at the old layout:

INSERT INTO t SELECT column2 AS id, column1 AS val FROM (VALUES (1, 100), (2, 200))
-- t is partitioned by identity(id)

partitioned the rows by column1 instead of column2. Files landed in the wrong partitions, and partition pruning then skipped matching rows: SELECT * FROM t WHERE id = 100 returns nothing.

What changes are included in this PR?

  • The table's columns are children. PartitionExpr has one child per top-level table column, in table order, and reads the batch only through them. The optimizer rewrites them like any other column reference, and EXPLAIN shows the column each partition field reads:

    iceberg_partition_values(id: identity(column2@1))
    
  • Iceberg still computes the values. evaluate lays the children out as the table and passes them to iceberg's PartitionValueCalculator, so the transforms and nested sources are handled as before.

  • try_new takes the columns. A codec passes the decoded children; project_with_partition passes its input's columns. with_new_children rejects the wrong number of children.

Are these changes tested?

Yes.

  • insert_into::test_insert_into_partitioned_through_reordering_projection runs the INSERT above and snapshots the plan, the row count, and a read with a filter on the partition column. On main the read returns no rows.
  • Unit tests cover reading through remapped children, the child-count check, equality depending on the children, the column shown for a nested source, and the new try_new error.

Are there any user-facing changes?

  • Breaking: PartitionExpr::try_new takes a third argument, columns: Vec<Arc<dyn PhysicalExpr>>, one per top-level table column. A codec that rebuilds the expression should pass the decoded children.
  • EXPLAIN output for partitioned inserts uses the format above.

#19 relaxes the input schema check in project_with_partition; whichever lands second resolves a small conflict there.

AI Disclosure

  • Used Claude Code to find the cause, write the fix and tests and draft this description. I reviewed the change and reproduced the failure on main.

@NoahKusaba
NoahKusaba force-pushed the fix/partition-expr-children branch from 4e98604 to 2552390 Compare October 29, 2026 19:33
PartitionExpr read its source columns by position in the input batch but
reported no children. When ProjectionPushdown merged the partition
projection into the projection below it, nothing rewrote those positions,
so partition values were computed from the wrong columns. Files landed in
the wrong partitions and partition pruning skipped them.

PartitionExpr's children are now the table's top-level columns, in table
order, and it reads the batch only through them. The optimizer rewrites
them like any other column reference. Evaluation lays the children out as
the table and still uses iceberg's PartitionValueCalculator.

The public `try_new` takes the columns, so a codec passes the decoded
children.

Closes apache#35
@NoahKusaba
NoahKusaba force-pushed the fix/partition-expr-children branch from 0eae2ff to 813e370 Compare November 21, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partitioned INSERT computes partition values from the wrong columns when the input plan ends in a projection

1 participant