Repository navigation
refactor: tidy write repartitioning - #134
Draft
NoahKusaba wants to merge 1 commit into
Draft
NoahKusaba wants to merge 1 commit into
NoahKusaba wants to merge 1 commit into
Conversation
repartition takes the partition spec, the only part of the table metadata it reads. Replace its ~100 lines of docs, including a claim that it skips repartitioning when the input already matches, with a short summary, use early returns, and replace 13 tests with snapshots of the partitioning chosen for each transform and of the missing _partition error. No behaviour change.
NoahKusaba
force-pushed
the
refactor/write-repartition
branch
from
November 21, 2026 14:04
4b4cef6 to
76cbb7c
Compare
NoahKusaba
marked this pull request as ready for review
November 21, 2026 14:52
NoahKusaba
marked this pull request as draft
November 21, 2026 15:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A. Refactor; no behavior change.
Rationale for this change
insert_intocallsrepartitionto spread its input across the session's target partitions before writing. It hashes rows on_partitionwhen a partition field uses an identity or bucket transform, and otherwise distributes batches round-robin. The code around that is harder to read than it needs to be:RepartitionExec.pub(crate)functions. The strategy is explained on bothrepartitionanddetermine_partitioning_strategy, and the example is anignoreblock that is never compiled.TableMetadataRefbut only reads the default partition spec, so its tests build whole tables.matches!,containsand hand-written panics. Several don't test this function:test_repartition_zero_partitions_failsonly checks thatNonZeroUsize::new(0)isNone.test_schema_ref_convenience_methodchecks iceberg'sTable::current_schema_ref.test_none_distribution_mode_fallbacksetswrite.distribution-mode, which the code never reads.test_bucket_aware_partitioningputs the bucket transform in the sort order of an unpartitioned table and accepts either partitioning, so it can't fail.What changes are included in this PR?
repartitiontakes&PartitionSpecinstead ofTableMetadataRef. Its one caller passestable.metadata().default_partition_spec().repartitionreplaces the docs on both functions.partitioning, renamed fromdetermine_partitioning_strategy, takes the input schema and uses early returns instead of matching on(bool, Result).test_partitioning_by_transformsnapshots the partitioning chosen for each transform and for a spec mixing them:The old tests didn't cover year, hour or truncate. Round-robin for truncate and the year/month/day/hour transforms can leave each writer with every partition value, and so more, smaller files. That's a behavior question for a separate issue.
test_partition_column_is_required_only_for_partitioned_tablessnapshots that an unpartitioned table needs no_partitioncolumn and that a partitioned one fails without it. The old tests didn't cover the error.The decisions and the error message are unchanged. Both new tests also pass against the old function body, with only its parameter changed from
TableMetadatato the partition spec it read.Are these changes tested?
Yes, by the two tests above. Partitioned inserts in
tests/insert_into.rsrun the repartition end to end.Are there any user-facing changes?
No.