Skip to content

fix(arrow): keep first-occurrence order in RecordBatchPartitionSplitter::split - #3754

Open
CaptainAni187 wants to merge 1 commit into
apache:mainfrom
CaptainAni187:fix/partition-splitter-first-occurrence-order
Open

CaptainAni187 wants to merge 1 commit into
apache:mainfrom
CaptainAni187:fix/partition-splitter-first-occurrence-order

Conversation

@CaptainAni187

@CaptainAni187 CaptainAni187 commented Nov 21, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

split grouped rows in a HashMap and returned the pieces in hash order. Now each partition value remembers its position the first time it shows up, so the pieces come back in first-occurrence order and sorted input stays sorted for ClusteredWriter. The order is noted in the split docs since callers can rely on it now.

Related: apache/datafusion-iceberg#81 works around this downstream until it picks up a release with this.

Are these changes tested?

Yes, test_record_batch_partition_split_keeps_first_occurrence_order splits 50 partitions given in descending order and checks the keys come back in that order. It fails on main.

AI Disclosure

Used Claude Code to help write the fix and the test. I reviewed the change myself and made the design calls independently (fixing it in the splitter rather than downstream, and keeping it dependency-free), and it was checked against the iceberg lib tests, fmt and clippy locally. Worth a look: whether stating the order in the split docs is fine as a guarantee, and the one extra Struct clone per partition for the index map.

…er::split

split grouped rows in a HashMap, so the pieces came back in hash order. ClusteredWriter needs sorted input to stay sorted: when a partition spans two batches, writing a later piece first closes it and the next batch fails with "Cannot write to partition that was previously closed".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RecordBatchPartitionSplitter::split returns partitions in hash order

1 participant