Skip to content

refactor: tidy write repartitioning - #134

Draft
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/write-repartition
Draft

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/write-repartition

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A. Refactor; no behavior change.

Rationale for this change

insert_into calls repartition to spread its input across the session's target partitions before writing. It hashes rows on _partition when a partition field uses an identity or bucket transform, and otherwise distributes batches round-robin. The code around that is harder to read than it needs to be:

  • The docs make a false claim. They say it "only adds repartitioning when the input partitioning differs from the target", but it always wraps its input in a RepartitionExec.
  • The docs are long and repeated. 82 lines of docs cover two pub(crate) functions. The strategy is explained on both repartition and determine_partitioning_strategy, and the example is an ignore block that is never compiled.
  • The function takes more than it uses. It takes TableMetadataRef but only reads the default partition spec, so its tests build whole tables.
  • The tests are long and check little. 13 tests and about 700 lines, checked with matches!, contains and hand-written panics. Several don't test this function:
    • test_repartition_zero_partitions_fails only checks that NonZeroUsize::new(0) is None.
    • test_schema_ref_convenience_method checks iceberg's Table::current_schema_ref.
    • test_none_distribution_mode_fallback sets write.distribution-mode, which the code never reads.
    • test_bucket_aware_partitioning puts the bucket transform in the sort order of an unpartitioned table and accepts either partitioning, so it can't fail.

What changes are included in this PR?

  • repartition takes &PartitionSpec instead of TableMetadataRef. Its one caller passes table.metadata().default_partition_spec().
  • A short doc comment on repartition replaces the docs on both functions.
  • partitioning, renamed from determine_partitioning_strategy, takes the input schema and uses early returns instead of matching on (bool, Result).
  • Two tests replace 13:
    • test_partitioning_by_transform snapshots the partitioning chosen for each transform and for a spec mixing them:

      [] => RoundRobinBatch(4)
      [identity(id)] => Hash([_partition@1], 4)
      [bucket[8](id)] => Hash([_partition@1], 4)
      [truncate[2](name)] => RoundRobinBatch(4)
      [year(ts)] => RoundRobinBatch(4)
      [month(ts)] => RoundRobinBatch(4)
      [day(ts)] => RoundRobinBatch(4)
      [hour(ts)] => RoundRobinBatch(4)
      [day(ts), identity(id)] => Hash([_partition@1], 4)
      

      The old tests didn't cover year, hour or truncate. Round-robin for truncate and the year/month/day/hour transforms can leave each writer with every partition value, and so more, smaller files. That's a behavior question for a separate issue.

    • test_partition_column_is_required_only_for_partitioned_tables snapshots that an unpartitioned table needs no _partition column and that a partitioned one fails without it. The old tests didn't cover the error.

The decisions and the error message are unchanged. Both new tests also pass against the old function body, with only its parameter changed from TableMetadata to the partition spec it read.

Are these changes tested?

Yes, by the two tests above. Partitioned inserts in tests/insert_into.rs run the repartition end to end.

Are there any user-facing changes?

No.

repartition takes the partition spec, the only part of the table metadata it
reads. Replace its ~100 lines of docs, including a claim that it skips
repartitioning when the input already matches, with a short summary, use
early returns, and replace 13 tests with snapshots of the partitioning chosen
for each transform and of the missing _partition error. No behaviour change.
@NoahKusaba
NoahKusaba force-pushed the refactor/write-repartition branch from 4b4cef6 to 76cbb7c Compare November 21, 2026 14:04
@NoahKusaba
NoahKusaba marked this pull request as ready for review November 21, 2026 14:52
@NoahKusaba
NoahKusaba marked this pull request as draft November 21, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant