feat(deltalite): per-phase timings in UpsertStats - #75966
Merged
Conversation
delta-rs's merge stats gave a scan/rewrite/execute time breakdown; deltalite only measured the total. Add plan_ms (listing + pruning partition files), rewrite_ms (reading + rewriting the touched partitions) and commit_ms (committing to the Delta log) to UpsertStats, exposed through the pyo3 getters. Bumps 0.1.2 -> 0.1.3.
|
😎 This pull request was merged. |
Contributor
🤖 CI report❌ Hobby preview — Health check did not pass within 35 minutesFailing fast because: Health check did not pass within 35 minutes Run 30637659718 | Consecutive failures: 2
|
estefaniarabadan
approved these changes
Jul 31, 2026
Generated-By: PostHog Code Task-Id: 20ee634e-79db-4267-b2df-9a336f96eb93
Member
Author
|
Status update after bringing this branch up to date with
🦉 via talyn.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
delta-rs's
merge().execute()returned a timing breakdown (scan / rewrite / execute). deltalite only measured the total upsert duration (for itsdeltalite_upsert_duration_secondshistogram) and returned no timing inUpsertStats. Follow-up to #75959, which restored structured stats logging on the warehouse side but has no phase timings to log.Changes
Add three per-phase wall-clock timings (ms) to
UpsertStats, measured inupsert_innerand exposed through the pyo3 getters:plan_ms— listing each touched partition's files + pruning which to rewrite.rewrite_ms— the parallel partition rewrites (reading + probing targets, writing new files) — the bulk.commit_ms— committing the new file set to the Delta log.Also added to the
"upsert committed"info!line. Bumps 0.1.2 → 0.1.3 (Cargo.toml + pyproject.toml + Cargo.lock); the README'sUpsertStatsfield list is updated.Testing
cargo check -p deltalite-core -p deltalite-pythonis clean (0 errors). Theci-deltalite-pythonbuild + differential parity suite runs on this PR.After merge
build-deltalitepublishes the 0.1.3 wheel and auto-pins it in the app. Because #75959's write log enumerates theUpsertStatsgetters generically,plan_ms/rewrite_ms/commit_mswill start appearing in thedeltalite write: committedlogs automatically once the pin lands — no further app change.