Skip to content

fix(data-imports): map DeltaLake object-store access errors to an actionable message - #73490

Merged
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-delta-kernel-access-error-message
Jul 30, 2026
Merged

fix(data-imports): map DeltaLake object-store access errors to an actionable message#73490
trunk-io[bot] merged 1 commit into
masterfrom
posthog-code/fix-delta-kernel-access-error-message

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

Error tracking surfaced a generic exception during a MySQL sync's schema-validation step: issue.

Exception: Could not read the files from your storage bucket. Check that the files URL pattern, file format, and credentials are correct, then try again.

The call path: import_data_activity_sync -> pipeline.run -> validate_schema_and_update_table -> DataWarehouseTable.get_columns() (products/warehouse_sources/backend/models/table.py), which issues a ClickHouse DESCRIBE TABLE against the synced Delta table. That query failed with a genuine object-store access error from ClickHouse's DeltaLake kernel:

DB::Exception: Received DeltaLake kernel error ObjectStoreError: Error interacting with object store:
The operation lacked the necessary privileges to complete for path .../dw_sku_dim/_delta_log/_last_checkpoint:
... Server returned non-2xx status code: 403 Forbidden: AccessDenied

get_columns()'s fallback error handler (_safe_expose_ch_error) matches known ClickHouse error substrings in ExtractErrors to surface a specific, actionable message, falling back to a generic one otherwise. ExtractErrors already has an entry for this exact situation from ClickHouse's native S3 client ("Access Denied: while reading key:"), but every warehouse_sources synced table reads its schema through ClickHouse's DeltaLake kernel instead (format="DeltaS3Wrapper"), which uses different wording from the underlying Rust object_store crate. That vocabulary wasn't represented in ExtractErrors at all, so this whole error class — not just this one occurrence — fell through to the generic fallback regardless of the actual cause.

This occurred 3 times in a ~36 second window during a single sync attempt and never recurred, consistent with a transient storage blip rather than a persistent credential problem — retries (5 attempts with backoff) are already correct behavior here and I didn't change the retry policy. The gap is purely in the message a user sees when this class of error does surface.

Changes

  • Added an ExtractErrors entry matching the DeltaLake-kernel object_store crate's permission-error wording ("The operation lacked the necessary privileges to complete"), mapping it to the same actionable message already used for the equivalent native-ClickHouse-S3 case.

How did you test this code?

Added test_delta_kernel_permission_error_gets_actionable_message to the existing TestSafeExposeChError class in test_table.py, using the real captured error text (with identifiers redacted) — this is exactly the shape of the bug this PR fixes: without the ExtractErrors entry, this case falls through to the generic message like every other unmatched error.

Ran:

  • uv run pytest products/warehouse_sources/backend/tests/test_table.py -k SafeExposeChError — 3 passed
  • uv run mypy --cache-fine-grained products/warehouse_sources/backend/models/table.py products/warehouse_sources/backend/tests/test_table.py — clean
  • ./bin/hogli ci:preflight --fix — 0 failures

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

N/A — internal error-message mapping, no user-facing API/config change.

🤖 Agent context

Autonomy: Fully autonomous

Triaged directly from the linked error-tracking issue via PostHog's error-tracking MCP tools (query-error-tracking-issue, query-error-tracking-issue-events), which surfaced the full stack/exception-chain and confirmed the failure originates in shared schema-introspection code (table.py), not the MySQL source connector. Traced the ClickHouse error-wrapping path (posthog/errors.py's wrap_clickhouse_query_error, dynamically-built CHQueryErrorDeltaKernelError for ClickHouse error code 742) to confirm the DeltaLake-kernel error vocabulary is structurally different from the native-S3 vocabulary ExtractErrors already covers.

Checked for duplicate open PRs by exception type, message phrase, and module path, and scanned the maintainer's own open PR queue. Found #73454 ("recognize DeltaError as a transient object-store error"), which fixes a related-but-distinct gap in delta_table_helper.py's best-effort pre-write maintenance classifier (a different code path, using the Python deltalake package rather than ClickHouse's SQL-level DeltaLake kernel) — it doesn't touch the ExtractErrors/get_columns gap this PR closes.

Invoked /writing-tests before adding the regression test.


Created with PostHog Code

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 24, 2026 11:25
@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Jul 24, 2026 — with PostHog
stamphog[bot]
stamphog Bot previously approved these changes Jul 24, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, contained fix adding one error-string mapping plus a test, authored by the owning team; no risky territory touched.

  • Author wrote 0% of the modified lines and has 28 merged PRs in these paths (familiarity MODERATE).
  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 3L, 1F substantive, 20L/2F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1a-trivial (20L, 2F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ 591ed2d · reviewed head 4348ef2

@trunk-io

trunk-io Bot commented Jul 28, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

@talyn-app

talyn-app Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@talyn-app

talyn-app Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

1 similar comment
@talyn-app

talyn-app Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@talyn-app
talyn-app Bot disabled auto-merge July 29, 2026 17:34
…ionable message

Error tracking surfaced a generic "Could not read the files from your storage bucket" exception during a MySQL sync's schema-validation step, with an underlying DeltaLake-kernel object-store access error that `ExtractErrors` had no entry for.

`get_columns()`'s fallback error handler (`_safe_expose_ch_error`) matches known ClickHouse error substrings against `ExtractErrors` to give a specific, actionable message, falling back to a generic one otherwise. `ExtractErrors` only covered ClickHouse's native S3 client error vocabulary, not the different wording used by ClickHouse's DeltaLake kernel (the `object_store` Rust crate) — the format used by every warehouse_sources synced table. Added a matching entry so this error class surfaces the same actionable message as the equivalent native-S3 case.

Generated-By: PostHog Code
Task-Id: 5407d15d-430b-4648-b6ee-483880dcbfae
@Gilbert09
Gilbert09 force-pushed the posthog-code/fix-delta-kernel-access-error-message branch from 0f412be to 00a4c16 Compare July 30, 2026 09:30
@stamphog
stamphog Bot dismissed their stale review July 30, 2026 09:32

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

@talyn-app

talyn-app Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial, well-tested error-message mapping fix by an owning-team author; no risky territory touched.

  • Author wrote 0% of the modified lines and has 31 merged PRs in these paths (familiarity MODERATE).
  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 3L, 1F substantive, 20L/2F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1a-trivial (20L, 2F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ 8bba179 · reviewed head 00a4c16

@trunk-io
trunk-io Bot merged commit 84c9db6 into master Jul 30, 2026
240 checks passed
@trunk-io
trunk-io Bot deleted the posthog-code/fix-delta-kernel-access-error-message branch July 30, 2026 10:22
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 30, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-30 11:02 UTC Run
prod-us ✅ Deployed 2026-07-30 11:18 UTC Run
prod-eu ✅ Deployed 2026-07-30 11:20 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant