Skip to content

fix(data-imports): keep source-classified retryable errors out of error tracking - #74744

Merged
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-retryable-error-tracking-noise
Jul 29, 2026
Merged

fix(data-imports): keep source-classified retryable errors out of error tracking#74744
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-retryable-error-tracking-noise

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

Error tracking surfaced an OperationalError: (2013, 'Lost connection to MySQL server during query (The read operation timed out)') from the MySQL data-warehouse import source, raised from the streaming row-fetch loop in mysql.py's _stream_with_optional_force_index.

This is exactly the class of error MySQL's get_retryable_errors() already lists ("Lost connection to MySQL server during query"), specifically so it stays out of error tracking as benign, self-recovering noise — Temporal retries the whole activity and the connection drop resolves itself. But it still got reported.

The gap is in the shared _handle_import_error (import_data_sync.py): when an error matches a source's get_retryable_errors(), it logs at warning (not exception) and re-raises the original exception. Logging at warning only affects our own structured logs — the Temporal activity interceptor (posthog/temporal/common/posthog_client.py) reports whatever exception type escapes the activity to error tracking regardless of log level. It only skips reporting for NonReportableError (and a couple of other known-benign marker types). Since the message-matched retryable path re-raised the bare exception instead of a NonReportableError, it was captured anyway — silently defeating the intent already documented in this function's docstring and in MySQLSource.get_non_retryable_errors's sibling get_retryable_errors.

This affects every source that implements get_retryable_errors() (MySQL, Postgres, Snowflake, Salesforce, Mixpanel, HubSpot, and others), not just MySQL.

Nothing here needed adding to NonRetryableErrors — the failure is transient infrastructure (a read timeout mid-stream) and Temporal's retry already handles it correctly; the only problem was the error-tracking noise.

Changes

  • _handle_import_error now re-raises a source-classified retryable error as NonReportableError(error_msg) from error instead of the bare exception, so Temporal still retries the activity as usual but the interceptor no longer reports it. Mirrors how RESTClientRetryableError (already a NonReportableError subclass) achieves this for REST-based sources.
  • Updated the docstring to describe the actual mechanism instead of the previously-aspirational claim that logging at warning alone kept these out of error tracking.

How did you test this code?

  • Extended test_source_classified_retryable_error_logged_as_warning_not_exception in test_import_data_sync.py to assert the re-raised exception is a NonReportableError (not just any Exception) with the original error chained via __cause__ — this is the regression guard: reverting to raise error would fail this assertion even though the log-level assertions would still pass.
  • Ran pytest products/warehouse_sources/backend/temporal/data_imports/workflow_activities/tests/test_import_data_sync.py (12 passed) and the MySQL source's retryable/non-retryable classification tests (35 passed).
  • ruff check / ruff format --check on the changed files: clean.
  • mypy --cache-fine-grained targeted at the changed file: clean. A repo-wide mypy run (as CI runs it) was still in progress when this PR was opened; will follow up if it surfaces anything.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

N/A — internal error-classification/reporting change, no user-facing or API behavior change.

🤖 Agent context

Autonomy: Fully autonomous

Triaged directly from the linked error-tracking issue via PostHog's error-tracking MCP tools (query-error-tracking-issue, query-error-tracking-issue-events), which surfaced the full stack trace confirming the failure genuinely originated in mysql.py. Traced the call path from the MySQL streaming cursor through get_rows/_handle_import_error/import_data_activity_sync up to the Temporal activity interceptor, and read posthog/temporal/common/posthog_client.py to confirm it captures by exception type, not log level.

Checked for duplicate open PRs by exception type, message phrase, and module path, and scanned the maintainer's own open PR queue. Found one closely related PR, #74388 ("keep transient S3 credential blips out of error tracking"), which fixes the same class of interceptor-reporting gap but for a different, non-overlapping branch of _handle_import_error (a new is_transient_object_store_error check) — it doesn't touch the get_retryable_errors() branch this PR fixes.

Invoked /writing-tests before extending the regression test.

…or tracking

A source's `get_retryable_errors()` (e.g. MySQL's "Lost connection to MySQL server during query") is meant to keep a transient, self-recovering error out of error tracking once it escapes to `_handle_import_error`. The handler logged it at `warning` instead of `exception`, but that only affects our own logs — the Temporal activity interceptor reports whatever exception type escapes the activity, regardless of log level, unless it's a `NonReportableError`. Re-raising the bare exception meant these errors landed in error tracking anyway.

Re-raise as `NonReportableError` (chained via `from error`) instead, mirroring how `RESTClientRetryableError` already does this for REST sources.

Generated-By: PostHog Code
Task-Id: d79e0d52-59ff-4e1f-9f4c-4dedc17c7faa
@trunk-io

trunk-io Bot commented Jul 29, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 29, 2026 14:18
@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Jul 29, 2026 — with PostHog

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, contained fix in warehouse-sources error handling (not risky territory) by an owning-team author with strong familiarity; the diff matches the description and the NonReportableError mechanism it relies on checks out in the codebase.

  • Author wrote 83% of the modified lines and has 24 merged PRs in these paths (familiarity STRONG).
  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 12L, 1F substantive, 23L/2F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1b-small (23L, 2F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ 30ad54b · reviewed head b7d0c5d

@talyn-app

talyn-app Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@talyn-app
talyn-app Bot enabled auto-merge (squash) July 29, 2026 16:41
@trunk-io
trunk-io Bot merged commit 381282b into master Jul 29, 2026
365 checks passed
@trunk-io
trunk-io Bot deleted the posthog-code/fix-retryable-error-tracking-noise branch July 29, 2026 17:49
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 29, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-29 18:37 UTC Run
prod-us ✅ Deployed 2026-07-29 18:59 UTC Run
prod-eu ✅ Deployed 2026-07-29 19:01 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant