fix(data-imports): stop reporting duckgres sink maintenance timeouts as errors - #73565
Merged
Gilbert09 merged 1 commit intoJul 28, 2026
Merged
Conversation
…as errors The eligibility-CTE maintenance queries (`supersede_replaced_runs`, `get_backlog_stats`, `count_orgs_at_budget`) are bounded by a 30s statement timeout specifically so a slow/loaded queue DB fails fast and the poll loop retries on the next tick — the surrounding code already documents this as expected. But the catch-all in `_run_maintenance` reported every exception, including that expected timeout, to error tracking as a defect. Added `is_eligibility_query_timeout` next to the timeout constant in `jobs_db.py` and branched on it in `consumer.py`: a `QueryCanceled` from these queries now logs a warning and skips the tick as before, without `capture_exception`; any other, genuinely unexpected error still gets captured. Generated-By: PostHog Code Task-Id: 74c8471e-3e63-4af7-850f-5b69730bc85b
Contributor
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
There was a problem hiding this comment.
Small, well-tested change that only reclassifies an already-swallowed, expected timeout so it stops polluting error tracking — no control-flow, data, API, or infra impact, and author is on the owning team.
- 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 18L, 2F substantive, 62L/3F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1b-small (62L, 3F, single-area, fix) |
| stamphog 2.0.0b3 | .stamphog/policy.yml @ 1920144 · reviewed head 3a9e27e |
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Error tracking surfaced a
QueryCanceled: canceling statement due to statement timeoutfromproducts/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/consumer.py, raised inside_run_maintenance's call toDuckgresBatchQueue.supersede_replaced_runsinjobs_db.py._run_maintenanceruns every ~30s to sweep obsolete runs and refresh backlog gauges. Its three queries (supersede_replaced_runs,get_backlog_stats,count_orgs_at_budget) are already wrapped in a 30sstatement_timeout(ELIGIBILITY_QUERY_STATEMENT_TIMEOUT_MS) specifically so a slow/loaded queue DB fails fast instead of stacking up long-running scans — the code comment right above the call already says a timeout here "must not wedge or crash the poll loop — skip this tick and retry on the next one."The poll loop does correctly skip and retry: the
except Exceptionaround the whole maintenance block swallows the error so nothing crashes. But it also unconditionally callscapture_exception(e)for every exception in that block, including this expected, self-healing timeout. So the very safeguard designed to make timeouts benign was reporting each one to error tracking as if it were a defect.Changes
is_eligibility_query_timeoutnext toELIGIBILITY_QUERY_STATEMENT_TIMEOUT_MSinjobs_db.py— a small classifier forpsycopg.errors.QueryCanceled, the expected shape of that bound tripping.consumer.py's_run_maintenancenow branches on it: aQueryCanceledlogs a warning and skips the tick as before, withoutcapture_exception. Any other, genuinely unexpected error in that block still gets captured, same as today.This only changes what gets reported to error tracking — the swallow-and-retry behavior for the whole block is unchanged.
How did you test this code?
Extended the existing
test_fetch_swallows_maintenance_query_timeout_and_skips_planner(which already asserted the timeout is swallowed and the planner/fetch are skipped) to also assertcapture_exceptionis not called. Added a siblingtest_fetch_reports_unexpected_maintenance_query_errorasserting a genuinely unexpected error (e.g. aRuntimeError) still gets captured — locking in that the fix is scoped to the specific timeout case, not a blanket suppression.Ran:
uv run pytest products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/test_consumer.py -k "test_fetch_swallows_maintenance_query_timeout_and_skips_planner or test_fetch_reports_unexpected_maintenance_query_error"— 2 passeduv run mypy --cache-fine-grainedon the two changed source files — cleanruff check/ruff format— cleanhogli ci:preflight --fix— 0 failuresThe rest of the duckgres test suite (DB-backed integration tests) requires a running Postgres instance not available in this sandbox; those were not run, but this change doesn't touch any DB-facing query logic.
Docs update
N/A — internal error-classification change, no user-facing behavior change.
🤖 Agent context
Autonomy: Fully autonomous
Triaged a webhook-delivered PostHog error tracking issue using the PostHog MCP error-tracking tools (
query-error-tracking-issue,query-error-tracking-issue-events) to pull the stack trace and sync context, then readconsumer.pyandjobs_db.pydirectly to confirm the timeout is already an expected, bounded condition per the existing code comments and a pre-existing test (test_fetch_swallows_maintenance_query_timeout_and_skips_planner) that only asserted the swallow, not the error-tracking report. Used an Explore-style subagent to check for a shared "transient DB error" classifier elsewhere in the pipeline (repartition_table.py's_is_transient_infra_error,delta_table_helper.py's object-store classifier,sources/postgres/postgres.py'sQueryCanceledhandling) — none covered this specific queue-DB maintenance path, so I added a small local classifier next to the timeout constant it's paired with. Invoked/writing-testsbefore extending the test. Checked for duplicate open PRs (exception type, message phrase, module path, and the maintainer's own open PRs) — found related-but-non-overlapping prior work (e.g. #73386, which classifies a different error type in a different file) rather than a duplicate.Created with PostHog Code