Skip to content

feat(deltalite): per-phase timings in UpsertStats - #75966

Merged
talyn-app[bot] merged 3 commits into
masterfrom
tom/deltalite-upsert-phase-timings
Jul 31, 2026
Merged

feat(deltalite): per-phase timings in UpsertStats#75966
talyn-app[bot] merged 3 commits into
masterfrom
tom/deltalite-upsert-phase-timings

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

delta-rs's merge().execute() returned a timing breakdown (scan / rewrite / execute). deltalite only measured the total upsert duration (for its deltalite_upsert_duration_seconds histogram) and returned no timing in UpsertStats. Follow-up to #75959, which restored structured stats logging on the warehouse side but has no phase timings to log.

Changes

Add three per-phase wall-clock timings (ms) to UpsertStats, measured in upsert_inner and exposed through the pyo3 getters:

  • plan_ms — listing each touched partition's files + pruning which to rewrite.
  • rewrite_ms — the parallel partition rewrites (reading + probing targets, writing new files) — the bulk.
  • commit_ms — committing the new file set to the Delta log.

Also added to the "upsert committed" info! line. Bumps 0.1.2 → 0.1.3 (Cargo.toml + pyproject.toml + Cargo.lock); the README's UpsertStats field list is updated.

Testing

cargo check -p deltalite-core -p deltalite-python is clean (0 errors). The ci-deltalite-python build + differential parity suite runs on this PR.

After merge

build-deltalite publishes the 0.1.3 wheel and auto-pins it in the app. Because #75959's write log enumerates the UpsertStats getters generically, plan_ms / rewrite_ms / commit_ms will start appearing in the deltalite write: committed logs automatically once the pin lands — no further app change.

delta-rs's merge stats gave a scan/rewrite/execute time breakdown; deltalite
only measured the total. Add plan_ms (listing + pruning partition files),
rewrite_ms (reading + rewriting the touched partitions) and commit_ms
(committing to the Delta log) to UpsertStats, exposed through the pyo3 getters.
Bumps 0.1.2 -> 0.1.3.
@Gilbert09 Gilbert09 self-assigned this Jul 31, 2026
@trunk-io

trunk-io Bot commented Jul 31, 2026

Copy link
Copy Markdown

😎 This pull request was merged.

@Gilbert09
Gilbert09 requested review from a team July 31, 2026 13:17
@Gilbert09
Gilbert09 deployed to pypi-deltalite July 31, 2026 13:33 — with GitHub Actions Active
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

Playwright — all passed

All tests passed.

View test results →

Hobby preview — Health check did not pass within 35 minutes

Failing fast because: Health check did not pass within 35 minutes


Run 30637659718 | Consecutive failures: 2

⚠️ deltalite — version bump needed

It looks like the code of deltalite changed in this PR, but its version stayed the same at 0.1.3. 👀
Bump the version in rust/deltalite/python/Cargo.toml AND rust/deltalite/python/pyproject.toml (the two must match) before merging.

@talyn-app
talyn-app Bot enabled auto-merge (squash) July 31, 2026 13:48
Generated-By: PostHog Code
Task-Id: 20ee634e-79db-4267-b2df-9a336f96eb93
@talyn-app
talyn-app Bot merged commit ac7bde6 into master Jul 31, 2026
292 of 293 checks passed
@talyn-app
talyn-app Bot deleted the tom/deltalite-upsert-phase-timings branch July 31, 2026 14:35
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 31, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-31 15:16 UTC Run
prod-us ✅ Deployed 2026-07-31 15:35 UTC Run
prod-eu ✅ Deployed 2026-07-31 15:35 UTC Run

Copy link
Copy Markdown
Member Author

Status update after bringing this branch up to date with master:

  • All required status checks are green (Rust, Django, Node.js, Frontend, Playwright, Docker build, Semgrep, shellcheck, etc.).
  • The one red check, Setup DO Hobby Instance and test, is not a required check and is failing on a pre-existing infra flake, not on anything in this PR. The droplet replays the full migration history from an empty DB on boot; that now runs past the job's 35-minute health-check window (Postgres migrations were still applying at ~34 min in), so the smoke test times out before the app serves /_health. This PR contains no migrations or app code — only the Rust deltalite change plus the deltalite dependency pin, which is what pulls uv.lock into the hobby path filter and triggers the run. Re-ran it once; it timed out the same way. The deltalite parity suite and the Rust build/tests all pass.
  • The deltalite version-bump reminder in the CI report is a benign false positive: 0.1.3 is already published to PyPI and pinned here (0.1.20.1.3), and the required "Check version legitimacy" job passes. The reminder re-appears only because the workflow self-publishes during PR CI and then re-observes its own published version on the next push.

🦉 via talyn.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants