Skip to content

feat(warehouse): structured deltalite write stats + duration - #75959

Merged
talyn-app[bot] merged 1 commit into
masterfrom
tom/deltalite-write-structured-stats
Jul 31, 2026
Merged

feat(warehouse): structured deltalite write stats + duration#75959
talyn-app[bot] merged 1 commit into
masterfrom
tom/deltalite-write-structured-stats

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

delta-rs's merge().execute() returned a rich stats dict that we logged as structured, parseable output (Delta Merge Stats: {json.dumps(...)}). The deltalite write path (#75884) instead logged a formatted string (deltalite write: committed v5 (+3 inserted / ~2 updated / …)) — human-readable but not parseable, and with no timing. deltalite's UpsertStats actually carries as much detail as delta-rs (and more: files_carried_over, files_probed, null_pk_rows) — we were just throwing the structure away.

Changes

  • The commit log is now structured: ainfo("deltalite write: committed", duration_ms=…, version=…, rows_inserted=…, …) — every UpsertStats field becomes its own log key, so it's parseable again (parity with the old Delta Merge Stats line). The deltalite write: committed prefix is unchanged, so existing LogQL still matches.
  • Total timing: the upsert is wall-clock timed and emitted as duration_ms on the log and a new warehouse_load_deltalite_write_duration_seconds histogram. (The log field is the cross-worker signal — the v2 worker's Prometheus isn't scraped.)
  • _deltalite_write_stats enumerates the pyo3 getters rather than a fixed list, so fields added crate-side later flow through automatically — which is exactly what the follow-up deltalite PR (per-phase timings in UpsertStats) will add.

How did you test this code?

I'm an agent. TestDeltaliteWritePath — 7 pass — now also asserts the commit is logged with the UpsertStats fields as structured keys + a duration_ms, plus a unit test for _deltalite_write_stats (scalar getters only, methods/privates dropped). ruff clean.

Follow-up

A deltalite crate PR adds per-phase timers (probe / write / commit) to UpsertStats for full delta-rs-style timing breakdown; those fields will appear in this log automatically once the wheel is bumped.

Restores the parseable structured output we had from delta-rs's merge stats.
The deltalite write now logs `deltalite write: committed` with every UpsertStats
field as its own structlog key (version, partitions_touched, files_*, rows_*,
source_rows, null_pk_rows, ...) plus a wall-clock `duration_ms`, instead of a
formatted string. _deltalite_write_stats enumerates the pyo3 getters, so fields
added crate-side later (e.g. per-phase timings) flow through automatically. Also
adds a warehouse_load_deltalite_write_duration_seconds histogram.
@Gilbert09 Gilbert09 self-assigned this Jul 31, 2026
@trunk-io

trunk-io Bot commented Jul 31, 2026

Copy link
Copy Markdown

😎 This pull request was merged.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Backend coverage — 93.0% of changed backend lines covered — 2 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 93.0% (29 / 31)

File Patch Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta_table_helper.py 87.5% 321–322

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30633053487 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
demo ███████████░░░░░░░░░ 56.3% 1,497 / 2,661
tasks ██████████████░░░░░░ 70.1% 32,867 / 46,884
signals ████████████████░░░░ 81.3% 24,949 / 30,693
cdp ████████████████░░░░ 82.1% 3,285 / 3,999
data_modeling █████████████████░░░ 85.7% 7,716 / 9,008
notebooks █████████████████░░░ 86.0% 7,794 / 9,060
actions █████████████████░░░ 86.6% 717 / 828
cohorts ██████████████████░░ 87.8% 6,181 / 7,040
product_tours ██████████████████░░ 87.9% 1,303 / 1,482
exports ██████████████████░░ 88.2% 7,046 / 7,986
data_warehouse ██████████████████░░ 88.4% 12,192 / 13,798
engineering_analytics ██████████████████░░ 89.3% 6,529 / 7,309
conversations ██████████████████░░ 89.3% 17,600 / 19,701
dashboards ██████████████████░░ 89.5% 5,989 / 6,693
alerts ██████████████████░░ 90.2% 4,458 / 4,942
streamlit_apps ██████████████████░░ 90.7% 2,630 / 2,901
mcp_analytics ██████████████████░░ 90.9% 3,191 / 3,511
error_tracking ██████████████████░░ 91.0% 10,926 / 12,008
slack_app ██████████████████░░ 91.1% 9,664 / 10,610
marketing_analytics ██████████████████░░ 91.2% 12,092 / 13,265
stamphog ██████████████████░░ 91.3% 4,505 / 4,936
product_analytics ███████████████████░ 92.5% 5,849 / 6,321
early_access_features ███████████████████░ 92.6% 1,287 / 1,390
ai_observability ███████████████████░ 92.8% 15,618 / 16,821
surveys ███████████████████░ 93.1% 5,771 / 6,197
posthog_ai ███████████████████░ 93.2% 1,326 / 1,422
web_analytics ███████████████████░ 93.3% 14,911 / 15,976
approvals ███████████████████░ 93.3% 3,437 / 3,682
reminders ███████████████████░ 93.4% 468 / 501
workflows ███████████████████░ 94.1% 7,241 / 7,698
endpoints ███████████████████░ 94.2% 8,655 / 9,192
review_hog ███████████████████░ 94.6% 8,101 / 8,563
skills ███████████████████░ 94.6% 3,158 / 3,337
experiments ███████████████████░ 95.5% 26,047 / 27,287
logs ███████████████████░ 95.5% 10,447 / 10,944
annotations ███████████████████░ 96.2% 732 / 761
revenue_analytics ███████████████████░ 96.3% 1,887 / 1,960
feature_flags ███████████████████░ 96.4% 17,488 / 18,144
replay_vision ███████████████████░ 96.4% 16,081 / 16,679
user_interviews ███████████████████░ 96.5% 2,638 / 2,734
customer_analytics ███████████████████░ 97.1% 10,444 / 10,758
warehouse_sources ███████████████████░ 97.3% 359,337 / 369,337
data_catalog ████████████████████ 97.7% 2,588 / 2,648
pulse ████████████████████ 98.4% 2,017 / 2,049

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@talyn-app
talyn-app Bot merged commit 3825910 into master Jul 31, 2026
243 checks passed
@talyn-app
talyn-app Bot deleted the tom/deltalite-write-structured-stats branch July 31, 2026 13:45
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 31, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-31 14:14 UTC Run
prod-us ✅ Deployed 2026-07-31 14:30 UTC Run
prod-eu ✅ Deployed 2026-07-31 14:29 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants