fix(data-imports): disable hive-partitioning inference on warehouse table introspection queries

[Gilbert09]

Problem

Error tracking surfaced a generic Could not read the files from your storage bucket exception raised from DataWarehouseTable.get_count, wrapping an underlying ClickHouse error: DB::Exception: Cannot convert string '2017-06-30T05' to type Date, thrown while reading a warehouse table's Delta/S3 files (ReadFromObjectStorage / HivePartitioningUtils::addPartitionColumnsToChunk).

ClickHouse's Hive-style partitioning support infers a type for each partition-folder value it samples (here, our internal _ph_partition_key column) independently of the physical column type actually written to the table. Our automatic repartitioning controller steps an oversized datetime-partitioned table through progressively finer tiers over time (month → week → day → hour), so a long-lived table's partition folders can mix value shapes across tiers, e.g. an hour-tier value like 2017-06-30T05 alongside older week-tier folders. ClickHouse misclassifies the column as Date from a sample and then fails outright when it hits a value that doesn't fit.

HogQLGlobalSettings.use_hive_partitioning already disables this inference by default for every normal HogQL query, clearly to avoid exactly this class of failure. But get_columns, get_max_value_for_column, get_count, and _validate_csv_double_quotes_setting on DataWarehouseTable issue raw ClickHouse queries directly against the table's own S3 table function, bypassing the HogQL query path entirely, and never applied the same setting. Any warehouse table whose partition scheme has stepped through more than one granularity tier can hit this on introspection.

Changes

• Added a shared DISABLE_HIVE_PARTITIONING_SETTINGS constant in products/warehouse_sources/backend/models/table.py.
• Applied it to every raw ClickHouse query issued directly against a table's Delta/S3 data: the chdb and cluster-fallback branches of get_columns and get_count, get_max_value_for_column, and _validate_csv_double_quotes_setting.
• This is shared pipeline code, not source-specific — the fix applies regardless of which connector wrote the table.

How did you test this code?

• ruff check / ruff format --check on the touched files: pass.
• mypy on the touched files: pass, no new issues.
• Added TestWarehouseQueryDisablesHivePartitioning in products/warehouse_sources/backend/tests/test_table.py, covering get_count and get_max_value_for_column: mocks sync_execute and asserts use_hive_partitioning: 0 is present in the settings passed on the ClickHouse-cluster fallback path (the path this incident actually hit, since TEST=True always short-circuits the chdb attempt in this environment). Regression this catches: a future change dropping the settings kwarg from either call would silently reintroduce the crash.
• I could not run the Django test suite live in this environment (no Postgres/ClickHouse/docker available), so I verified the new tests via static/manual code tracing against the existing test patterns in the same file rather than a live pytest run — flagging this explicitly since I can't claim to have executed them.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Fully autonomous

• Investigated via PostHog error tracking MCP tools (issue details, stack trace, and event properties) to identify the failing frame and scope which sync/schema reproduced it, then read the shared pipeline code (table.py, partitioning.py, repartition.py) to trace the root cause to ClickHouse's Hive-partition type inference against a repartitioned table's _ph_partition_key folder values.
• Skills invoked: /writing-tests before adding the regression test.
• Considered whether this was upstream/user error (bad credentials, deleted data) vs. our bug — ruled that out once the DeltaLake stack frame and the HogQLGlobalSettings.use_hive_partitioning=0 precedent made clear this is our own raw-query code path not applying a setting the rest of the codebase already treats as mandatory.
• Checked for duplicate open PRs (by exception text, module path, and the maintainer's own open PRs) — found two open PRs touching the same file (fix(data-imports): map DeltaLake object-store access errors to an actionable message #73490, fix(data-warehouse): fail syncs loudly on schema-drift read errors #74001) but addressing unrelated bugs (error-message mapping, DeltaS3Wrapper schema drift), not this Hive-partitioning issue.

---

Created with PostHog Code

[trunk-io]

😎 Merged successfully - details.

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[stamphog]

Small, well-tested fix adding a ClickHouse setting to disable Hive-partition type inference on raw introspection queries, authored by a member of the owning team; diff matches description and carries no risk to schemas, APIs, auth, or CI.

• Author wrote 0% of the modified lines and has 30 merged PRs in these paths (familiarity MODERATE).
• 👍 on the PR from hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	19L, 1F substantive, 48L/2F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (48L, 2F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ 7673382 · reviewed head fc938a6

[github-actions]

🤖 CI report

⚠️ Backend coverage — 88.0% of changed backend lines covered — 2 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ██████████████████░░ 88.0% (15 / 17)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/models/table.py	50.0%	453, 577

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30473115907 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	██████████████░░░░░░ 69.7%	31,949 / 45,830
cdp	████████████████░░░░ 81.0%	3,146 / 3,883
signals	████████████████░░░░ 81.2%	23,969 / 29,526
data_modeling	█████████████████░░░ 85.3%	7,345 / 8,612
notebooks	█████████████████░░░ 86.0%	7,794 / 9,060
actions	█████████████████░░░ 86.6%	717 / 828
cohorts	█████████████████░░░ 86.9%	5,648 / 6,496
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
exports	██████████████████░░ 88.2%	7,046 / 7,986
data_warehouse	██████████████████░░ 88.3%	12,036 / 13,635
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
engineering_analytics	██████████████████░░ 89.4%	6,441 / 7,202
conversations	██████████████████░░ 89.6%	17,277 / 19,284
alerts	██████████████████░░ 90.0%	4,342 / 4,827
mcp_analytics	██████████████████░░ 90.2%	2,883 / 3,198
streamlit_apps	██████████████████░░ 90.7%	2,630 / 2,901
error_tracking	██████████████████░░ 91.0%	10,925 / 12,004
slack_app	██████████████████░░ 91.1%	9,554 / 10,492
stamphog	██████████████████░░ 91.1%	4,056 / 4,450
marketing_analytics	██████████████████░░ 91.2%	12,058 / 13,222
product_analytics	███████████████████░ 92.5%	5,849 / 6,321
early_access_features	███████████████████░ 92.6%	1,287 / 1,390
ai_observability	███████████████████░ 92.8%	15,328 / 16,517
surveys	███████████████████░ 93.1%	5,771 / 6,197
web_analytics	███████████████████░ 93.2%	14,826 / 15,906
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
workflows	███████████████████░ 93.8%	6,943 / 7,399
endpoints	███████████████████░ 94.2%	8,655 / 9,192
skills	███████████████████░ 94.6%	3,158 / 3,337
review_hog	███████████████████░ 94.6%	6,912 / 7,303
experiments	███████████████████░ 95.4%	25,800 / 27,031
logs	███████████████████░ 95.5%	10,435 / 10,928
annotations	███████████████████░ 96.2%	732 / 761
revenue_analytics	███████████████████░ 96.3%	1,887 / 1,960
feature_flags	███████████████████░ 96.4%	17,371 / 18,023
replay_vision	███████████████████░ 96.4%	15,861 / 16,455
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
customer_analytics	███████████████████░ 97.1%	9,866 / 10,157
warehouse_sources	███████████████████░ 97.2%	347,160 / 357,057
data_catalog	████████████████████ 97.7%	2,555 / 2,615
pulse	████████████████████ 98.4%	2,017 / 2,049

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[talyn-app]

/trunk merge

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-30 11:02 UTC	Run
prod-us	✅ Deployed	2026-07-30 11:18 UTC	Run
prod-eu	✅ Deployed	2026-07-30 11:20 UTC	Run