Skip to content

fix: skip the partition sort for unpartitioned tables without fanout - #135

Closed
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/unpartitioned-fanout-insert
Closed

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/unpartitioned-fanout-insert

Conversation

@NoahKusaba

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

With write.datafusion.fanout.enabled=false, an INSERT into an unpartitioned table fails at planning:

Error during planning: Partition column '_partition' not found in schema. Ensure the plan has been extended with partition values using project_with_partition.

insert_into adds the _partition column only for partitioned tables, but sorted by it whenever fanout was disabled. The sort is there for the clustered writer, which partitioned tables use when fanout is disabled. Unpartitioned tables always use UnpartitionedWriter, which needs no sort.

What changes are included in this PR?

insert_into sorts by _partition only when fanout is disabled and the table is partitioned. A comment says why.

Are these changes tested?

Yes. insert_into::inserts_into_an_unpartitioned_table_without_fanout inserts two rows into an unpartitioned table with fanout disabled, and snapshots the plan (no sort), the row count and the rows read back. On main it fails with the error above. It's the first test that sets write.datafusion.fanout.enabled.

Partitioned tables with fanout disabled are left to #38, where the optimizer removes this sort.

Are there any user-facing changes?

INSERT into an unpartitioned table with write.datafusion.fanout.enabled=false now works.

With write.datafusion.fanout.enabled=false, insert_into sorted its input by
_partition, a column only partitioned tables have, so INSERT into an
unpartitioned table failed at planning. Only the clustered writer used for
partitioned tables needs the sort.

Closes apache#132.
@NoahKusaba NoahKusaba closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INSERT into an unpartitioned table fails when write.datafusion.fanout.enabled is false

1 participant