Skip to content

Optimize mixed DuckLake LEFT JOIN refreshes - #14

Merged
ila merged 4 commits into
mainfrom
codex/mixed-mutation-performance
Sep 1, 2026
Merged

Optimize mixed DuckLake LEFT JOIN refreshes#14
ila merged 4 commits into
mainfrom
codex/mixed-mutation-performance

Conversation

@ila

@ila ila commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Summary

  • replace broad affected-key recompute for eligible mixed DuckLake LEFT JOIN projections with tuple-level signed-delta consolidation
  • inline a single-use leaf MV delta (or use a temporary delta table) to avoid persistent DuckLake delta write/cleanup amplification
  • attempt an exhaustive full-tuple delete and verify DuckLake changed-row count; rollback and retry with ranked rowid deletion when duplicate-bag multiplicity is partial
  • batch repeated source-metadata lookups during qualification and snapshot assembly
  • persist conservative LEFT JOIN transition-safety and nullable-side lineage; legacy metadata retains the existing fallback

Scope and safety

  • native DuckLake only; Spark/external compilation is unchanged
  • mixed LEFT JOIN simple projections only
  • insert-only refreshes keep the existing append fast path
  • views with downstream consumers keep persistent cascade deltas
  • composite, multiple, and non-comparison outer joins retain affected-key recompute

Correctness

  • extended test/sql/ducklake_left_join.test with batched INSERT + DELETE + UPDATE cases
  • verifies full bag equality using EXCEPT ALL in both directions after every refresh
  • deterministically covers exhaustive-delete success, rollback/ranked-delete retry, composite joins, and a LogicalAnyJoin combined with another outer join
  • focused test passes: 157 assertions
  • native SF10 mixed-workload validation passes all 49 models in all three batches
  • Spark SF100 validation passes all 49 models in all three batches

Performance evidence

  • full native SF100 mixed run (batch 2: 1% deletes; batch 3: 1% updates): 272.4 / 132.9 / 176.5s
  • prior main control with the same mutation rates: 247.7 / 861.8 / 1594.0s
  • SF50 fact_market_history batch 2: 380.8s before -> 11.2s candidate; full DuckDB model 13.0s
  • Spark is unaffected because external compilation uses compile_only=true; the matching SF100 validation run showed no regression

@ila
ila marked this pull request as ready for review September 1, 2026 13:18
@ila
ila merged commit c3d230a into main Sep 1, 2026
24 checks passed
@ila
ila deleted the codex/mixed-mutation-performance branch September 1, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant