Optimize PostgreSQL metadata passthrough

[benben]

Summary

• improve the PostHog fork's existing PostgreSQL passthrough manager directly; no parallel adapter, attach option, or alternate transaction design
• execute the high-cardinality snapshot-row deletion as a set-based PostgreSQL operation while keeping the shared DuckLakeMetadataManager::DeleteSnapshots coordinator as the sole owner of cleanup policy
• aggregate table/file sizes in PostgreSQL and restrict commit-time global stats loading to the transaction write set through the canonical shared query builder
• preserve shared cleanup behavior for relative paths, inline data/delete tables, dependent metadata, stats-cache invalidation, and PREPARE/autocommit lifecycle semantics
• add reproducible in-cluster benchmarks with actual PostgreSQL fixture row counts, including 1,000,000 snapshots

This is based on posthog/v1.5.3 (49ec0dc8). The implementation uses the fork's existing PassthroughQuery/PassthroughExecute architecture and v1.5 transaction state. It does not add a PostgresFast manager or restore a second commit architecture.

Following review, PostgreSQL now overrides only the narrow DeleteSnapshotRows primitive. The generic cleanup coordinator handles everything else, so future upstream cleanup-policy changes apply automatically. This also incorporates upstream c812f6e6's cleanup of physical inlined-delete tables and adds parity regressions for both inline table types and relocated relative paths.

Xlarge benchmark results

Rerun in-cluster against PostgreSQL 18.3 after the review fixes at d20f1d38. Every trial records actual per-table fixture counts before timing and logical result cardinality. The committed artifact contains 384 unique, alternating trials: 192 pristine v1.5.3 and 192 feature trials.

operation	fixture metadata rows	result rows	v1.5.3	feature	speedup
expire snapshots	2,000,000	500,000	14.616s	7.155s	2.04x
stats lookup	1,010,000	100	1.151s	0.405s	2.84x
table information	1,101,000	1,000	0.812s	0.557s	1.46x
list snapshots	2,000,000	1,000,000	4.943s	4.881s	1.01x
current snapshot	1,000,000	1	0.360s	0.360s	1.00x
cleanup candidates	1,000,000	1,000,000	1.896s	1.892s	1.00x
expiration dry run	2,000,000	500,000	2.213s	2.177s	1.02x
file discovery	1,100,001	1,000,000	4.368s	4.541s	0.96x

Unchanged operations remain within approximately 4% and are treated as noise. Full methodology and per-table row-count breakdowns are in benchmark/postgres/RESULTS.md and benchmark/postgres/results/release.jsonl.

Validation

• pristine posthog/v1.5.3: 95,310 assertions in 468 test cases passed (11 skipped)
• reviewed feature release suite: 95,321 assertions in 469 test cases passed (12 expected environment/extension skips)
• PostgreSQL passthrough regression against the DEV PostgreSQL catalog: 49 assertions passed, including PREPARE/autocommit expiration, relocated cleanup paths, and physical inline data/delete table cleanup
• ported upstream inline-delete cleanup regression: 11 assertions passed
• full release build passed
• diff and artifact-integrity checks passed; repository format check is also exercised by CI (the benchmark pod does not contain Black)
• benchmark validation: 384 rows, 384 unique (variant, operation, scale, trial) keys, zero errors/duplicates; every aggregate fixture count equals its stored per-table breakdown

The PostgreSQL test config sets the regression's environment gate so PostHog catalog CI runs it with the official v1.5.3 PostgreSQL extension.

[bill-ph]

The redesign is substantially better: this now targets posthog/v1.5.3, uses the existing passthrough manager, and removes the parallel adapter/factory/version matrix. The table-size override also looks appropriately contained.

I am still requesting changes for two reasons:

1. PostgreSQL expiration is currently broken for dead inlined tables; the catalog CI failures reproduce the invalid DROP TABLE IF EXISTS %I.%I command.
2. The full DeleteSnapshots override still duplicates cleanup policy. A merge with current upstream/v1.5-variegata is textually clean, but it already misses the new inlined-file-deletion-table cleanup from upstream commit c812f6e6. This is exactly the silent semantic drift periodic syncing needs to avoid.

Textual sync cost is much improved: the PR adds no current v1.5 conflicts and only two conflict paths beyond the existing fork when compared with upstream main. The remaining sync risk is concentrated in expiration policy. Please fix the concrete failure and either derive the PostgreSQL batch from shared cleanup builders/policy or add a comparably robust parity mechanism before merging.

[benben]

Addressed the review in d20f1d3: PostgreSQL now overrides only the narrow snapshot-row deletion primitive, while the shared DeleteSnapshots coordinator owns all cleanup policy, paths, inline tables, dependencies, and cache invalidation. This removes the invalid %I batch entirely and incorporates the upstream c812f6e inlined-delete-table cleanup in the shared path. Added PostgreSQL parity coverage for relocated relative paths and physical inline data/delete tables, plus the upstream regression. Fresh post-fix benchmark artifacts are in e6848cd and the PR description has been updated. @bill-ph could you please re-review?

[benben]

CI follow-up: the two red catalog jobs were caused by the test gate declaring DUCKLAKE_POSTGRES, which postgres.json already defines. ecf471c switches the test to the established DUCKLAKE_CI workflow gate used by the neighboring PostgreSQL tests. The exact parser path now gets past the former duplicate-variable error, and the focused DEV PostgreSQL run passes all 49 assertions. Fresh CI is running on ecf471c. All four outdated review threads are resolved.