Skip to content

refactor: tidy sort_by_partition - #133

Closed
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/sort-by-partition
Closed

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/sort-by-partition

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A. Refactor; no behavior change.

Rationale for this change

When write.datafusion.fanout.enabled is false, insert_into calls sort_by_partition so the clustered writer gets its input grouped by partition. The function only looks up the _partition column and wraps the input in a SortExec that keeps its partitions. The code around that is noisier than it needs to be:

  • Docs restate the signature. 14 lines, with # Arguments and # Returns sections, for a pub(crate) function with one argument.
  • Comments restate the code, and one is wrong. Nearly every statement has a comment. SortOptions::default(), // Ascending, nulls last is incorrect: Arrow's default puts nulls first.
  • An error branch can't run. LexOrdering::new returns None only for an empty list, and it's always given one expression.
  • The tests check DataFusion, not this function. They run DataFusion's sort on in-memory data and check the rows come out in order. They never check the plan the function builds, including preserve_partitioning.

What changes are included in this PR?

  • The docs say in three lines what the function does and why.
  • The LexOrdering is built with LexOrdering::from([PhysicalSortExpr::new_default(..)]), removing the unreachable error branch. These are the same calls as before: new_default is PhysicalSortExpr::new(expr, SortOptions::default()), and From<[PhysicalSortExpr; N]> calls LexOrdering::new. It takes a fixed-size array, which can't be empty, so it has no None case to handle.
  • Two tests replace three:
    • test_sort_by_partition_sorts_each_partition snapshots the plan: SortExec: expr=[_partition@1 ASC], preserve_partitioning=[true]. The old tests ran on a single partition, so they never checked preserve_partitioning, which the clustered writer depends on.
    • test_sort_by_partition_requires_partition_column snapshots the full error, instead of checking that it contains a substring.

The column lookup and the error message are unchanged. The new plan snapshot prints the sort options (ASC here; other options would show as DESC or NULLS LAST), and both new tests pass unchanged against the implementation on main.

Are these changes tested?

Yes, by the two tests above. The removed tests checked that rows come out sorted, including how Arrow orders struct values. That's DataFusion's and Arrow's behavior, and the clustered writer only needs equal partition values next to each other.

What this crate does need to check is that a real fanout-disabled INSERT keeps the sort through the optimizer and writes correctly. That needs end-to-end tests, and the old unit tests couldn't catch it: they passed while #38 broke every such INSERT into a partitioned table. Those tests belong with the fixes: #132 (unpartitioned tables) and #38 (partitioned tables).

Are there any user-facing changes?

No.

AI Disclosure

  • Used Claude Code to review the write path, make the change and draft this description. I reviewed the change.

@NoahKusaba
NoahKusaba force-pushed the refactor/sort-by-partition branch from c2cb36f to 6375a90 Compare November 21, 2026 13:31
@NoahKusaba

Copy link
Copy Markdown
Contributor Author

Starting to cleanup the insert related code, as all other read related work has an open PR.

@NoahKusaba
NoahKusaba marked this pull request as ready for review November 21, 2026 13:42
Shorten its docs, build the LexOrdering with From instead of handling a
LexOrdering::new error that can't happen, and replace tests that executed
DataFusion's sort with snapshots of the plan and the error.
@NoahKusaba
NoahKusaba force-pushed the refactor/sort-by-partition branch from 6375a90 to ea2b4d5 Compare November 21, 2026 13:53
@NoahKusaba NoahKusaba closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INSERT into an unpartitioned table fails when write.datafusion.fanout.enabled is false

1 participant