fix(data-imports): back off full interval after duckgres enablement refresh failure

[Gilbert09]

Problem

Error tracking surfaced a RuntimeError ("duckgres control plane unreachable; keeping the previous sink enablement") from products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/enablement.py, raised through DuckgresBatchConsumerAdapter._enabled_team_ids in consumer.py.

_enabled_team_ids caches the duckgres-enabled team set and only re-resolves it every ENABLEMENT_REFRESH_SECONDS (60s) — the poll loop itself runs every ~2s. On a refresh failure, the except block correctly keeps the previous team set (so a transient blip doesn't blind the sink), but it only advanced _team_ids_fetched_at when this was the very first resolution ever attempted:

except Exception as e:
    ...
    if self._team_ids_fetched_at is None:
        self._team_ids = []
        self._team_ids_fetched_at = now

Once a refresh had already succeeded at least once, a later failure left _team_ids_fetched_at stuck at its last successful timestamp. Since real time keeps advancing past it, the cache guard (now - self._team_ids_fetched_at < ENABLEMENT_REFRESH_SECONDS) never holds again, so every subsequent poll tick re-attempted the refresh instead of waiting for the intended 60s window — hammering the control plane during an outage and reporting the same error to error tracking roughly every 2 seconds instead of once per minute.

The sampled events for this issue all lacked any warehouse_sources_* sync context, consistent with the failure coming from this background cache refresh rather than any specific source/schema/sync.

Changes

• consumer.py: _enabled_team_ids now always advances _team_ids_fetched_at on a refresh failure, not just on the first-ever resolution. The previous team set is still retained on failure (unchanged), but the next retry now waits the full ENABLEMENT_REFRESH_SECONDS interval instead of firing on every poll tick.

How did you test this code?

Added test_enablement_refresh_failure_after_success_waits_full_interval_before_retrying in test_consumer.py, covering a refresh failure that happens after a prior successful resolution (the case the old code mishandled). It asserts the previous team set is retained and that a second immediate poll does not re-trigger the refresh. Verified it fails against the pre-fix code (2 refresh attempts instead of 1) and passes with the fix.

Ran:

• uv run pytest products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/test_consumer.py -q — 26 passed (3 pre-existing errors in TestLiveApplyStamp need a live Postgres connection and are unaffected by this change)
• uv run mypy --cache-fine-grained . — clean, 16964 files
• ruff check / ruff format --check — clean
• hogli ci:preflight --fix — 0 failures

Docs update

N/A — internal reliability fix to a background cache refresh, no user-facing or documented behavior change.

🤖 Agent context

Autonomy: Fully autonomous

I (Claude) triaged this from a PostHog error-tracking webhook alert with no human steering the investigation. I pulled the issue's stack trace, impact, and sampled $exception events via the PostHog MCP error-tracking tools, confirmed all sampled events lacked warehouse_sources_* sync context (pointing at background/shared code rather than a specific source), then read enablement.py and consumer.py to trace the raise site back through _enabled_team_ids's caching logic. Comparing the refresh-guard condition against the except-block's timestamp update revealed the bug: the timestamp only advances on the very first failure, so a later failure re-triggers on every ~2s poll tick instead of backing off for the full 60s window — which also matches the observed ~2s-apart event cadence in error tracking.

Checked for duplicates: searched open PRs by exception message, module path, and function name, and reviewed the maintainer's (Gilbert09) own open PRs. Found two related-but-distinct prior fixes in this same file/class (#73565, #73649) that reclassify timeout exceptions so they're not double-reported to error tracking — neither touches the enablement cache or this retry-cadence bug, so this isn't a duplicate.

Invoked /writing-tests before adding the regression test.

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here