Skip to content

fix(data-imports): keep exhausted REST retries out of error tracking - #74640

Merged
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-rest-retryable-error-tracking-noise
Jul 29, 2026
Merged

fix(data-imports): keep exhausted REST retries out of error tracking#74640
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-rest-retryable-error-tracking-noise

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

An error-tracking issue fired for a data warehouse import sync: RESTClientRetryableError: HTTP 524 for https://api.cal.com/v2/bookings (error tracking issue). HTTP 524 is Cloudflare's "origin didn't respond in time" timeout - a transient upstream blip, not a bug.

The shared REST engine already retries this internally (tenacity, 5 attempts with backoff) before giving up, and import_data_sync.py's _handle_import_error already classifies RESTClientRetryableError as "benign, self-recovering" and logs it at warning instead of exception specifically so it wouldn't show up as error-tracking noise. That intent doesn't hold: the Temporal activity interceptor that decides whether to call capture_exception (posthog/temporal/common/posthog_client.py) only skips reporting for exceptions that subclass NonReportableError - the application log level has no effect on it. So every exhausted-retry transient error from any REST-based source (429s, 5xx, connection resets/timeouts) still mints a fresh error-tracking issue, once its retry budget runs out, on every occurrence.

RESTClientNonRetryableError (the sibling "retrying can never help" error) already solves this correctly by subclassing NonReportableError. RESTClientRetryableError didn't.

Changes

  • RESTClientRetryableError now subclasses NonReportableError, the same pattern RESTClientNonRetryableError already uses, so the activity interceptor actually keeps it out of error tracking once it escapes the REST client's own retry loop.
  • Temporal's own activity-level retry is unaffected - this only changes whether the escaped exception is reported, not whether (or how many times) the sync retries.

How did you test this code?

  • Extended test_send_request_raises_retryable_after_persistent_transient_error in test_rest_client.py to assert the raised RESTClientRetryableError is also a NonReportableError, mirroring the existing assertion for RESTClientNonRetryableError in test_non_json_response_body_is_non_retryable.
  • uv run pytest products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_rest_client.py - 34 passed.
  • uv run pytest products/warehouse_sources/backend/temporal/data_imports/workflow_activities/tests/test_import_data_sync.py - 12 passed (unaffected, confirms _handle_import_error's type-based RESTClientRetryableError branch still behaves as before).
  • uv run mypy --cache-fine-grained . - no issues in 17074 source files.
  • uv run ruff check / ruff format --check on changed files - clean.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

N/A - internal error-tracking classification, no user-facing behavior or API change.

🤖 Agent context

Autonomy: Fully autonomous

  • Triggered by an error-tracking webhook for the issue linked above. I (Claude) triaged the stack trace, traced the exception path from the Cal.com sync's REST call up through the shared pipeline and Temporal activity interceptor, and found the retry-vs-report classification gap.
  • Checked existing open PRs for overlap first: #74388 fixes the same underlying gap (log level doesn't suppress capture_exception) for a different error class (transient S3/object-store errors), confirming this is an established pattern rather than a novel fix - it doesn't touch RESTClientRetryableError or this file, so no duplicate.
  • No skills invoked; this was a direct code read + targeted fix following the existing RESTClientNonRetryableError precedent in the same file.

RESTClientRetryableError already carries `NonReportableError`'s sibling type contract in spirit (`_handle_import_error` logs it at warning, not exception), but the activity interceptor that decides whether to call `capture_exception` only skips reporting for exception types that actually subclass `NonReportableError`. Log level never reached that check, so a transient 429/5xx/timeout that survives all 5 tenacity attempts still minted an error-tracking issue on every occurrence - e.g. a Cloudflare HTTP 524 timeout on a Cal.com sync.

Make `RESTClientRetryableError` subclass `NonReportableError`, the same way `RESTClientNonRetryableError` already does, so the activity interceptor actually honors the "benign, self-recovering" classification these errors were always meant to have. Temporal still retries the whole activity as before - this only changes whether the escaped exception is reported.

branch: posthog-code/fix-rest-retryable-error-tracking-noise

Generated-By: PostHog Code
Task-Id: 3eff308d-cef3-4ca5-8c9e-9b42407c2459
@trunk-io

trunk-io Bot commented Jul 29, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 29, 2026 11:47
@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Jul 29, 2026 — with PostHog

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial, well-tested reclassification of an already-existing exception to suppress error-tracking noise for transient retryable failures; author is on the owning team and diff matches the description exactly.

  • Author wrote 0% of the modified lines and has 15 merged PRs in these paths (familiarity MODERATE).
  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 9L, 1F substantive, 15L/2F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1a-trivial (15L, 2F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ 545937f · reviewed head 1893fb5

@talyn-app

talyn-app Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@trunk-io
trunk-io Bot merged commit b7e0cc6 into master Jul 29, 2026
367 checks passed
@trunk-io
trunk-io Bot deleted the posthog-code/fix-rest-retryable-error-tracking-noise branch July 29, 2026 13:02
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 29, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-29 14:33 UTC Run
prod-us ✅ Deployed 2026-07-29 14:46 UTC Run
prod-eu ✅ Deployed 2026-07-29 14:49 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant