fix(data-imports): classify DB connection blips as transient in pre-write maintenance

[Gilbert09]

Problem

Error tracking surfaced an OperationalError: [Errno -2] Name or service not known from products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py, with the top in-app frame in ExternalDataSchema.folder_path.

The call chain: run_pre_write_defensive_compact -> DeltaTableHelper.run_maintenance -> compact_if_fragmented -> get_delta_table -> _get_delta_table_uri, which resolves the Delta table's S3 URI via job.folder_path() — a Django ORM lookup that traverses job.schema and schema.source FKs on a pooled app-DB connection. A DNS/pooler blip while opening that connection raised the OperationalError.

This is a transient, self-healing infra blip, not a maintenance bug: run_pre_write_defensive_compact already catches every exception and skips a matching transient one with a quiet warning (the next sync's maintenance pass just retries from scratch). But the existing classifier (is_transient_object_store_error) only recognized S3/object-store OSErrors from delta-rs, so this DB-connection error fell through to capture_exception instead — noise in error tracking for something that isn't a code defect.

The same failure class (Postgres OperationalError/InterfaceError from a transient connection blip) is already classified as transient infra elsewhere in this tree, in repartition_table.py's _is_transient_infra_error.

Changes

• Added is_transient_maintenance_error in delta_table_helper.py, which treats django.db.OperationalError/InterfaceError as transient on top of the existing S3/object-store substring matches. It delegates to is_transient_object_store_error, which stays unchanged so its other object-store-specific callers (get_delta_table, reset_table, CDC load) keep their narrower classification.
• Updated extract.py's run_pre_write_defensive_compact to use the broadened is_transient_maintenance_error classifier.

I found an open PR (#71934) that reduces how often this call path issues the underlying lazy DB query in the first place (via select_related + memoization). That's a complementary root-cause fix for the redundant query; it doesn't touch this classifier, so this change still closes the gap for connection blips on the remaining DB round-trip. I also found two open PRs (#69787, #69309) that suppress transient DNS/DB errors at the Temporal activity-interceptor level — those cover uncaught activity failures. This error never reaches that interceptor: it's caught and manually reported inside run_pre_write_defensive_compact itself, so those PRs don't cover this code path.

How did you test this code?

Extended TestRunPreWriteDefensiveCompact in test_extract.py with test_logs_transient_db_connection_error_without_capturing, parameterized over OperationalError (DNS failure) and InterfaceError (dropped pooler connection) — asserts capture_exception is skipped and a warning is logged instead, mirroring the existing S3/object-store transient-error test right above it. This locks in the exact regression this PR fixes: without the classifier change, both cases would fall through to capture_exception again.

Ran:

• uv run pytest products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py -k TestRunPreWriteDefensiveCompact — 9 passed
• uv run mypy --cache-fine-grained on the three changed files — clean
• ruff check --fix / ruff format — clean

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

N/A — internal error-classification change, no user-facing behavior change.

🤖 Agent context

Autonomy: Fully autonomous

Triaging a webhook-delivered error tracking issue. Used the PostHog MCP error-tracking tools (query-error-tracking-issue, query-error-tracking-issue-events) to pull the stack trace, then an Explore subagent to read posthog/sync.py, the folder_path property chain, delta_table_helper.py, and existing NonRetryableErrors/transient-error precedent across the pipeline. Confirmed the DB round-trip and precedent (repartition_table.py's _is_transient_infra_error) directly before writing the fix. Invoked /writing-tests before adding the regression test. Checked for duplicate open PRs by exception type, message phrase, module path, and the maintainer's own open PRs — found related-but-non-overlapping work (see Changes section) rather than a duplicate.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[stamphog]

Narrow widening of an already-existing transient-error classifier in a warehouse-sync maintenance path, backed by regression tests; author owns this area with strong familiarity and no risky territory (no migration, API, auth, billing, or CI changes) is touched.

• Author wrote 100% of the modified lines and has 40 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	24L, 2F substantive, 50L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (50L, 3F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ 4ac650f · reviewed head cc2d38e

[trunk-io]

😎 Merged manually by talyn-app[bot] - details.

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

[stamphog]

Small, well-tested error-classification change in an internal warehouse-sync maintenance path (already wrapped in try/except, doesn't affect the actual sync); not risky territory, and author has strong familiarity with these exact files.

• Author wrote 100% of the modified lines and has 15 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	25L, 2F substantive, 51L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (51L, 3F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ 7c4151e · reviewed head abb0751

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

[talyn-app]

/trunk merge

[stamphog]

Small, well-tested broadening of an existing transient-error classifier to also cover DB connection blips, consistent with established precedent elsewhere in the same pipeline; no risky-territory surface touched and author has strong ownership/familiarity.

• Author wrote 100% of the modified lines and has 22 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	29L, 2F substantive, 55L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (55L, 3F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ c033211 · reviewed head 0801005

[Gilbert09]

Heads-up on the red Django Tests Pass check — it is not caused by this PR.

The posthog/hogql/test/test_parser_cpp_json.py::TestParserCppJson suite is failing on master itself, starting exactly at #73958 (fix(hogql): cap recursion depth in the C++ parser). That change updated the C++ parser source and the test to expect a too deeply nested SyntaxError, but the published hogql-parser wheel used by CI (when common/hogql_parser/ is unchanged relative to master, as it is here) has not been rebuilt yet — so test_deeply_nested_input_does_not_stack_overflow runs the old parser, spins for ~8.5 min without raising, and drops the idle Postgres connection, cascading every remaining test in the shard to OperationalError: the connection is closed.

The wheel republish is already in flight in #75455 (fix(hogql): publish hogql-parser 1.3.83 with recursion depth limit). Once that lands on master, I'll merge master into this branch and re-run — the shard should go green.

This PR itself only touches warehouse_sources transient-error classification; its own tests pass locally and in the warehouse-sources product shards.

_{🦉 via talyn.dev}

[talyn-app]

/trunk merge

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

[stamphog]

Small, well-tested broadening of a transient-error classifier in a background maintenance retry path; no data model, API, auth, billing, CI, or dependency surface touched, and the author (owning team, STRONG familiarity) covers it with a regression test matching the described bug.

• Author wrote 100% of the modified lines and has 25 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	29L, 2F substantive, 55L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (55L, 3F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ ec35a5a · reviewed head dcf80ed

[Gilbert09]

Status update — the red CI here is a master-side regression, not this PR.

This PR only touches warehouse_sources transient-error classification (3 files). Its own tests pass locally (TestRunPreWriteDefensiveCompact, 11/11), and the branch is rebased onto master, conflict-free, and MERGEABLE.

What's actually red and why:

• The required Django Tests Pass is red because the non-required Repo checks job fails deterministically on current master. Repo checks runs hogli product:lint --all, which fails the isolation-chain / facade-wiring-couplings doctrine for two unrelated products (customer_analytics and error_tracking) — their facades re-export classes from outside the declared wiring locations. This reproduces on a clean origin/master checkout with no changes from this PR, so it isn't introduced here.
• On the pull_request event, that deterministic Repo checks failure trips cancel-backend-on-repo-check-failure, which cancels the whole Backend CI matrix — so the Django test jobs never run and Django Tests Pass reports red by cascade.
• In the Trunk merge queue (merge_group event) that cancel job doesn't run (it's pull_request-gated) and Repo checks is optional ([optional, does not block merge]), which is why backend PRs continue to merge past this on master.

Plan: the branch is already kept current with master. Once the customer_analytics / error_tracking isolation-chain violation is resolved on master, the next run's Backend CI will no longer be cancelled and this shard will go green. Fixing those products' facade structure is out of scope for a warehouse_sources fix, so I'm not changing them here.

(This supersedes my earlier note about the hogql-parser wheel — that specific failure is no longer the blocker.)

_{🦉 via talyn.dev}

[Gilbert09]

CI update — the branch is now rebased onto latest master (cleanly resolving the overlap with #73454 and #74541), and the earlier hogql-parser breakage is gone (1.3.83 is in the tree). This PR's own suites pass.

The current red Repo checks is a separate, master-wide issue, not from this PR. Its Lint product structure step (./bin/hogli product:lint --all) fails with:

✗ 2 product(s) failed: customer_analytics, error_tracking

Both report that they are sealed and carry backend:contract-check but their turbo.json doesn't narrow contract-check inputs — a rule tightened by #75140 (chore(ci): narrow merge-queue lanes for isolated product changes, merged earlier today). Neither product is touched by this PR (the diff is exactly three warehouse_sources files), and product:lint --all reproduces the same two failures locally on a clean checkout — so it's a master-state problem those product owners (or a revert of #75140) need to resolve. When Repo checks goes green on master again, I'll merge master in and the gate should pass here too.

_{🦉 via talyn.dev}

[Gilbert09]

CI status: current red checks are a fresh master-wide regression, not this PR

The earlier test_parser_cpp_json failure is gone. The red checks now are Repo checks (the product:lint --all "Lint product structure" step) and the downstream Django Tests Pass gate that keys off it.

Root cause — #75140 (merged to master today, ~1h before this run). It broadened the backend:contract-check inputs in two products:

• products/customer_analytics/turbo.json: backend/models/team_customer_analytics_config.py → backend/models/** (plus backend/test/factories.py)
• products/error_tracking/turbo.json: added backend/models.py

IsolationChainCheck / has_narrowed_turbo_inputs only treats a product as "narrowed" when every contract-check input is confined to an accepted surface (facade/presentation, routes, garages, permanent, carve-out). A broad backend/models/** glob isn't confined, so has_narrowed_turbo flips to false and the "fully sealed and eligible ... but turbo.json does not narrow" error fires for both products:

✗ 2 product(s) failed: customer_analytics, error_tracking

Proof it's master-wide, not this branch: unrelated PR #75701 (fix(signup), touches neither warehouse_sources nor those two products) fails the identical lint on the same two products. This PR only touches 3 warehouse_sources files and cannot fix another product's isolation config.

Fix belongs on master — the two turbo.json files need to go back to specific carve-out paths (or the accepted-surface set extended). Reported to the devex team. Once master is green, this branch just needs a re-merge; no change is needed here.

_{🦉 via talyn.dev}

[Gilbert09]

Fixed a second, pre-existing master breakage that blocked this product's tests

After the master merge above turned Repo checks green, the Product tests (warehouse-sources …) job surfaced a separate pre-existing breakage: pytest collection was interrupted by 6 ModuleNotFoundErrors —

ModuleNotFoundError: No module named
'products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline'

Five source modules in aws_cost_explorer and app_store_connect imported SourceInputs/SourceResponse from …data_imports.pipelines.pipeline.typings, a path that no longer exists — the canonical location is …data_imports.sources.common.typings (used by every other source). This is present on master itself (unrelated to this PR's 3 classifier files), so it interrupts the whole warehouse_sources suite for any PR that touches the product.

Rather than block on it, I corrected the five imports to the canonical path (commit cc792c8) — a zero-behavior path fix. The PR now owns 8 files: the 3 original classifier changes plus these 5 import corrections. Also reported the breakage to the devex team.

_{🦉 via talyn.dev}

New commits pushed (delta classified non_trivial_delta) — stamphog approval dismissed; re-review running automatically.

[stamphog]

Contained error-classification change in the warehouse-sources import pipeline (widens transient-error handling to include known DB connection blips, matching existing precedent elsewhere), plus a verified no-op import-path fix for a pre-existing breakage on 5 files. Author is on the owning team with strong recent familiarity, tests were added, and the behavior change only affects error-tracking noise, not sync correctness.

• Author wrote 100% of the modified lines and has 30 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	43L, 6F substantive, 71L/8F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1c-medium (71L, 8F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ b258ee3 · reviewed head 148078c

[stamphog]

Retaining stamphog approval — delta since last review classified as trivial_paths.

[stamphog]

Retaining stamphog approval — delta since last review classified as mixed_trivial.

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

[stamphog]

Small, focused fix that broadens an existing transient-error classifier in a data-warehouse maintenance path to also treat DB connection blips as non-fatal, matching an established pattern elsewhere in the same codebase; author owns this code (owning team + STRONG familiarity, 100% of touched lines), added regression tests, and no reviewer concerns or holds are present.

• Author wrote 100% of the modified lines and has 28 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	29L, 2F substantive, 55L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (55L, 3F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ e370063 · reviewed head 1a9f8ea

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-30 20:54 UTC	Run
prod-us	✅ Deployed	2026-07-30 21:15 UTC	Run
prod-eu	✅ Deployed	2026-07-30 21:14 UTC	Run