fix(data-imports): classify a concurrent reset losing a delta log commit file as transient

[Gilbert09]

Problem

Error tracking issue surfaced a DeltaError during a Google Sheets full-refresh sync:

Generic error: Kernel error: File not found: .../_delta_log/00000000000000000001.json

The call path: _post_run_operations -> compact_table() -> vacuum_table() -> delta-rs's table.vacuum(), which needs to read a _delta_log commit file to reconstruct the table's file history.

Neither vacuum() nor optimize.compact() ever delete a _delta_log/*.json commit file themselves, so something else removed it mid-read. The prime suspect is reset_table(), which purges the whole table prefix (including _delta_log) at the start of a full-refresh sync — exactly the sync type on this event. If a still-running maintenance pass from an earlier, heartbeat-timed-out Temporal activity attempt (a "zombie", per the existing comments in this file) opened the table before a fresh attempt's reset_table() purge landed, its vacuum() call loses the commit file out from under it.

This is the same category of race TRANSIENT_DELTA_MAINTENANCE_ERRORS/is_transient_delta_maintenance_error already exists to classify (currently only for optimize.compact's "Optimize selected-file scan failed" signature) — it's a caller losing a file to a concurrent maintenance/reset pass, not a corrupt table. The call sites that swallow compact_table()/vacuum_table() failures (_post_run_operations in pipeline.py, _run_delta_maintenance in common/load.py) already skip reporting for is_transient_object_store_error, but neither checked is_transient_delta_maintenance_error at all — so this DeltaError (and the existing "Optimize selected-file scan failed" one, at these specific call sites) fell through to capture_exception as if it were a real bug.

This is best-effort maintenance code in both cases: the exception is already caught and never re-raised, so the sync itself was never at risk. This only affects whether a self-healing race gets reported as a defect.

Changes

• Extended is_transient_delta_maintenance_error in delta_table_helper.py to also match a DeltaError whose message contains both "File not found" and "_delta_log/" — the signature of a transaction-log commit file disappearing mid-read, distinct from a missing data file or a genuinely corrupt table (which still get captured).
• Added the missing is_transient_delta_maintenance_error check alongside the existing is_transient_object_store_error check in _post_run_operations (pipeline.py) and _run_delta_maintenance (common/load.py), matching the pattern run_pre_write_defensive_compact already uses in common/extract.py.

How did you test this code?

• Extended TestIsTransientDeltaMaintenanceError with the missing-log-file signature (expects True) and a "File not found" case outside _delta_log (expects False, guarding against over-broadening the match) — this is the regression an unguarded revert of the new check would reintroduce.
• Ran uv run pytest products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test/test_delta_table_helper.py — 79 passed (5 pre-existing failures in TestGetDeltaTableUnrecoverableErrors need a running object-storage service not available in this sandbox; they fail identically on master).
• ruff check / ruff format --check clean on all changed files.
• Repo-scoped uv run mypy --cache-fine-grained on the three changed source files — clean.

Docs update

N/A — internal error-classification change, no user-facing or API surface change.

🤖 Agent context

Autonomy: Fully autonomous

Triaged from a live PostHog error-tracking issue (webhook-delivered) using Claude Code. Pulled the stack trace and warehouse_sources_* event properties via the PostHog MCP tools, traced the call path into delta_table_helper.py, and confirmed via is_deltatable/vacuum/optimize.compact semantics that a missing log commit file mid-vacuum can only come from something else deleting it concurrently.

Searched open PRs (by exception type, message phrase, module path, and the maintainer's own open PRs) before starting. Found three related-but-distinct open PRs touching the same file: #73454 (broadens the object-store classifier to DeltaError, but only for S3-throttling substrings, not this signature), #74541 (fixes a different bug — a redundant get_delta_table() re-fetch racing a different table's cache eviction, raising a plain Exception, not this DeltaError), and #74594 (disables delta-rs's automatic 30-day log-retention cleanup on writes/merges — a different trigger than a concurrent reset_table purge, and doesn't touch the classifier or the maintenance call sites this PR fixes). None overlap this fix.

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[github-actions]

🤖 CI report

⚠️ Playwright — 1 flaky

🎭 Playwright report · View test results →

⚠️ 1 flaky test:

• Add insight to new dashboard and view it there (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

⚠️ Backend snapshots — 13 updated (13 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 13 snapshots (13 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

✅ Backend coverage — all changed backend lines covered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 100.0% (11 / 11)

All changed backend lines are covered ✅

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.3%	1,497 / 2,661
tasks	██████████████░░░░░░ 70.0%	32,688 / 46,675
signals	████████████████░░░░ 81.1%	24,318 / 29,985
cdp	████████████████░░░░ 82.1%	3,285 / 3,999
data_modeling	█████████████████░░░ 85.6%	7,668 / 8,959
notebooks	█████████████████░░░ 86.0%	7,794 / 9,060
actions	█████████████████░░░ 86.6%	717 / 828
cohorts	██████████████████░░ 87.8%	6,181 / 7,040
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
exports	██████████████████░░ 88.2%	7,046 / 7,986
data_warehouse	██████████████████░░ 88.4%	12,192 / 13,798
engineering_analytics	██████████████████░░ 89.3%	6,529 / 7,309
conversations	██████████████████░░ 89.3%	17,600 / 19,701
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
alerts	██████████████████░░ 90.2%	4,458 / 4,942
mcp_analytics	██████████████████░░ 90.5%	3,041 / 3,361
streamlit_apps	██████████████████░░ 90.7%	2,630 / 2,901
error_tracking	██████████████████░░ 91.0%	10,925 / 12,004
slack_app	██████████████████░░ 91.1%	9,632 / 10,576
stamphog	██████████████████░░ 91.1%	4,056 / 4,450
marketing_analytics	██████████████████░░ 91.2%	12,092 / 13,265
product_analytics	███████████████████░ 92.5%	5,849 / 6,321
early_access_features	███████████████████░ 92.6%	1,287 / 1,390
ai_observability	███████████████████░ 92.8%	15,618 / 16,821
surveys	███████████████████░ 93.1%	5,771 / 6,197
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
web_analytics	███████████████████░ 93.3%	14,911 / 15,976
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
workflows	███████████████████░ 93.9%	6,922 / 7,375
endpoints	███████████████████░ 94.2%	8,655 / 9,192
review_hog	███████████████████░ 94.6%	7,979 / 8,437
skills	███████████████████░ 94.6%	3,158 / 3,337
experiments	███████████████████░ 95.4%	25,938 / 27,177
logs	███████████████████░ 95.5%	10,435 / 10,928
annotations	███████████████████░ 96.2%	732 / 761
revenue_analytics	███████████████████░ 96.3%	1,887 / 1,960
feature_flags	███████████████████░ 96.4%	17,474 / 18,130
replay_vision	███████████████████░ 96.4%	16,081 / 16,679
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
customer_analytics	███████████████████░ 97.1%	10,444 / 10,758
warehouse_sources	███████████████████░ 97.3%	358,420 / 368,487
data_catalog	████████████████████ 97.8%	2,589 / 2,648
pulse	████████████████████ 98.4%	2,017 / 2,049

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[Gilbert09]

CI note: Django tests – Core … (9/19) failure is a pre-existing, intermittent master-side flake — unrelated to this PR

The only red check is the backend shard Django tests – Core (persons-on-events off, events schema legacy) … (9/19). It is not caused by anything in this PR:

• This PR's diff touches only products/warehouse_sources/… (4 files). It changes nothing under posthog/hogql/. All of the warehouse-import tests it adds/modifies pass.
• The shard failure is a Postgres-connection crash cascade in the HogQL parser tests. The first failure is posthog/hogql/test/test_parser_cpp_json.py::TestParserCppJson::test_deeply_nested_input_does_not_stack_overflow, after which every subsequent test in the shard fails with django.db.utils.OperationalError: the connection is closed (~1900 occurrences, 764 tests). A worker crash drops the shared DB connection and the rest of the shard cascades.
• It reproduces identically on master's own CI, and it's intermittent there — same shard, same first failing test, same connection is closed cascade, same 764 failures. Recent master ci-backend runs: one shard-9/19 success immediately followed by two failures in the same window, which is the signature of an environmental flake rather than a code regression.

Because the root cause is a flaky worker/DB crash in unrelated HogQL parser tests on master, it can't be fixed from within this warehouse-imports change, and re-running the shard is the correct remedy. I've re-triggered it; it will go green once the worker doesn't crash on that run.

_{🦉 via talyn.dev}

[Gilbert09]

Update: root cause of the … (9/19) failure pinned to a master regression (#73958)

Following up on the note above with the exact cause. The blocking shard's first failure is a real assertion, not the connection error:

posthog/hogql/test/_test_parser.py:2935: in test_deeply_nested_input_does_not_stack_overflow
    with self.assertRaises(SyntaxError, msg=rule) as cm:
E   AssertionError: SyntaxError not raised : expr

TestParserCppJson.test_deeply_nested_input_does_not_stack_overflow feeds a 1500-deep prefix-operator chain ("- " * 1500 + "1") and expects the C++ parser to reject it with a SyntaxError. The parser doesn't reject it, so the deep native recursion destabilizes the worker and drops its Postgres connection — after which every remaining test in the shard fails with django.db.utils.OperationalError: the connection is closed (~1900 occurrences, 764 tests). The connection-closed cascade is a symptom; this assertion is the trigger.

This was introduced by #73958 (fix(hogql): cap recursion depth in the C++ parser), which landed on master at 12:20 UTC today and un-skipped this case for the C++ parser. master's own ci-backend runs confirm it: shard 9/19 was green on every run before that commit and has failed on every run after it. It is not yet reverted or fixed on master.

This is unrelated to this PR. The diff here touches only products/warehouse_sources/… (4 files) and nothing under posthog/hogql/; all of this PR's own tests pass. The branch is up to date with master, mergeable, and every other check is green. It will go green once the #73958 regression is fixed on master and merged in — retrying the shard won't help, since the failure is deterministic post-#73958.

_{🦉 via talyn.dev}

[stamphog]

Narrow, well-tested error-classification fix in already-swallowed best-effort Delta maintenance code (never re-raised, so sync correctness is unaffected); author owns this area with STRONG familiarity and added regression tests for both the positive and negative match cases.

• Author wrote 100% of the modified lines and has 37 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	22L, 2F substantive, 52L/4F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (52L, 4F, single-area, fix)
stamphog 2.0.0b4		.stamphog/policy.yml @ c551d59 · reviewed head 1ac8c1e