fix(data-imports): detect a hollow delta table on the sync path

[posthog]

Problem

• A customer's Postgres sync 404s on parquet files in 2017-2019 partitions, fails, and fails again on every retry. Re-enabling the schema heals nothing. Their only exit is a full resync from source and a multi-day gap in warehouse data.
• The table is hollow: the delta log still references files vacuum already deleted from S3. We have a self-heal for exactly this, but detection lives only in the repartition scan (_missing_live_object_path / _live_missing_data_file). handle_corrupted_delta_log fires on an unreadable _delta_log or a delta_revive_required marker only the repartition path ever sets. A sync that hits the 404 itself has no equivalent check.
• Vacuum is the likely mechanism. It passed enforce_retention_duration=False, which deliberately turns delta-rs's retention guard off, so a file can be physically deleted while a concurrent snapshot, an interleaved repartition swap, or a zombie Temporal activity attempt still references it. Vacuum has been rolling out to a lot more tables, and revives have been clustering on a small number of schemas that get torn down and rebuilt repeatedly.

Changes

Arm the revive from the sync's failure path. Both pipelines now call schedule_revive_if_table_is_hollow when the run raises. It reuses the two repartition helpers unchanged, so there is one definition of "hollow" rather than two:

Check	Purpose
_missing_live_object_path	Does the error name a missing object, and is it inside the live table (not a __repartitioned temp)? Returns early for everything else, so a normal failure costs nothing.
_live_missing_data_file	Does the current log still reference it, and is it really gone from S3? Guards against a stale snapshot racing a legitimate rewrite, which would reset a healthy table.

Both true, and it sets the delta_revive_required marker. The error still propagates - this run is already past the point where the revive would have run, so the next run's handle_corrupted_delta_log does the reset and non-billable rebuild.

Stop vacuum making hollow tables. Retention now follows the table's own deletedFileRetentionDuration (a week, by default) with the guard enforced:

-table.vacuum(retention_hours=24, enforce_retention_duration=False, dry_run=False)
+table.vacuum(dry_run=False)

Note

The trade-off is slower reclamation: dead files now linger up to the configured retention instead of 24 hours. Worth it against a failure mode whose only recovery is a full rebuild from source. If we want the space back sooner, the right lever is lowering deletedFileRetentionDuration on the table so the guard stays honest, not disabling the guard.

Complementary to #75240 and #75242, which classify racy file-not-found errors on the maintenance paths as transient. Those retry past the symptom; a genuinely hollow table stays undetected. This PR adds the detection and removes the cause. No file overlap beyond delta_table_helper.py, and different functions within it.

How did you test this code?

Automated tests I added, in test_extract.py::TestScheduleReviveIfTableIsHollow:

• A missing file the live log still references arms the marker with reason="sync_missing_data_file". This is the customer's failure, and no existing test covers detection outside the repartition path.
• An unrelated error and a file under a __repartitioned temp sibling both return early without touching the schema, and without paying for a live-log scan.
• A stale-snapshot race (log no longer references the file) does not arm the marker. Without this, the fix would reset healthy tables.

I could not run the pytest suite in this environment: it has no Docker and only the Postgres client libraries, so django_db tests error on connect. What I did run:

What actually ran

• uv run mypy --cache-fine-grained . repo-wide: Success: no issues found in 17251 source files.
• ruff check / ruff format and hogli ci:preflight --fix: clean.
• pytest .../test_delta_table_helper.py -k vacuum: 8 passed (these do not need the DB).
• Exercised schedule_revive_if_table_is_hollow directly in a Django shell against mocks for all four branches above, confirming the marker payload and that the verification call is skipped on the early-return paths.
• Confirmed against the installed deltalake that vacuum's defaults are retention_hours=None, enforce_retention_duration=True, so dropping the arguments is exactly "use the table's window, enforce the guard".

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

No user-facing surface changes, so nothing to update.

🤖 Agent context

Autonomy: Fully autonomous

Written by Claude (Opus) via PostHog Code from a Signals inbox report. Skills invoked: /writing-tests, /writing-code-comments.

Decisions along the way:

• I first considered healing in the same run rather than the next one. Rejected: the revive resets the table and rebuilds from source, and by the time the sync trips the 404 it has already streamed rows. Restarting extraction mid-run is a much bigger change for a state that is already rare, and the marker gets the customer to a healed table on the next scheduled sync instead of never.
• For vacuum I considered widening retention to a fixed constant with the guard still off (the import activity's start_to_close_timeout reaches a week on some paths, so no constant is obviously safe). Deferring to the table's configured window with the guard on is simpler and keeps the safety property rather than re-picking a number that can go stale.
• I hooked the detection into both pipeline_v2 and pipeline_v3 rather than into the delta write helper, so it also catches a 404 raised from a read or from post-load work, not just from the merge.

---

Created with PostHog Desktop from this inbox report.

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here