feat(duckdbservice): export ducklake commit-loop stats from ducklake_commit_stats()

[fuziontech]

Summary

Workers now poll the ducklake extension's (upcoming) ducklake_commit_stats() table function every 30s and re-export its process-global counters as Prometheus metrics. This gives visibility INSIDE the extension's commit retry loop — the existing duckgres_worker_ducklake_conflict_* counters only see conflicts that escape it, which is why they read ~0 while megaduck's metadata store shows a 12.4% transaction rollback rate.

New metrics (all labelled by catalog; conflicts additionally by cause ∈ {primary_key, unique, conflict, concurrent}):

• duckgres_worker_ducklake_commit_attempts_total
• duckgres_worker_ducklake_commit_successes_total
• duckgres_worker_ducklake_commit_retries_exhausted_total
• duckgres_worker_ducklake_commit_nonretryable_errors_total
• duckgres_worker_ducklake_commit_backoff_ms_total
• duckgres_worker_ducklake_commit_duration_ms_total
• duckgres_worker_ducklake_commit_conflicts_total

Design

• Contract (shared with the extension-side PR in PostHog/ducklake): ducklake_commit_stats() → rows of (catalog VARCHAR, stat VARCHAR, value BIGINT), cumulative and monotonic per process. Cumulative→counter conversion via last-seen delta tracking (decrease → treated as fresh baseline; negative values → dropped, never stored as baseline).
• Graceful degradation: capability probe (duckdb_functions()) once per engine; no released extension binary has the function yet, so on current pins this logs once at info and stays dormant. Ships safely before/independently of the extension PR.
• Lifecycle: commitStatsLoop mirrors metadataMetricsLoop — started in NewDuckDBService, 30s ticker, stops via the pool's stopCh, scrapes over controlDB to avoid queueing behind pinned user-query connections.

Test plan

• 6 new unit tests: delta table (incl. decrease/negative), stat→metric mapping (incl. unknown-cause passthrough + unknown-stat rejection), end-to-end record via prometheus/testutil, probe against real in-memory DuckDB (macro stand-in), probe-once/disabled path, nil-DB skip
• just test-unit (incl. repository hygiene), just lint, go vet: clean
• e2e positive path: not exercisable until an extension binary ships ducklake_commit_stats() (noted per harness rule); degradation path is what current pins exercise

Follow-ups: millpond needs the same poller (separate repo — that's the writer where megaduck's numbers will appear); Grafana panels once metrics flow.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LUnQHp46rmLKdupmDUr7kG

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	1	0	0
E2E/journey files	0	0	0
Workflow files	0	0	0

Signals

• Test cases: +7 / -0
• Assertions: +20 / -0
• Skips or known failures added: 0
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

[fuziontech]

Adversarial self-review (requested by James). One real bug found and fixed in be2e015; the rest verified clean.

Bug found & fixed: the delta tracker's defensive path for negative values reset the baseline to 0 (lastSeen[key] = 0), which would make the next valid cumulative value count in full — double-exporting everything already recorded (100 exported, glitch, 102 arrives → +102 instead of +2). The inline comment claimed this prevented inflated deltas; it caused them. Fix: keep the baseline untouched and emit nothing. The test now regression-covers the exact double-count scenario ("post-glitch value counts only the increase over the kept baseline"). Contract-wise negatives can't occur (extension counters are monotonic int64 atomics), so this was latent, but a defense that amplifies is worse than none.

Verified correct:

• Probe caching is by *sql.DB identity and a probe error records no verdict (retries next tick) — no risk of permanently wedging into "unavailable" from a transient failure; engine replacement triggers exactly one re-probe.
• Exactly one poller goroutine per process: NewDuckDBService is invoked once in the service path, and the exporter state is confined to that goroutine (no locking needed, as documented).
• scrapeCommitStats takes p.mu.RLock only to read the controlDB pointer, never across a query; 5s scrape timeout; falls back to activation DB only when controlDB is unwired (test paths).
• Counter-reset semantics: decrease → current value as delta is the standard restart heuristic; correct here since exporter state and extension counters share process lifetime.
• Zero-delta touches materialize all known series on first poll (good for rate() continuity).

Notes for reviewers (accepted trade-offs):

1. *_ms_total deviates from the Prometheus base-unit (seconds) convention — kept deliberately for 1:1 symmetry with the extension contract and integer delta math. rate() dashboards just divide by 1e3.
2. Unknown conflicts.<cause> values export under their own label (forward-compatible with new extension causes; cardinality is extension-controlled — ours). Unknown non-conflict stats are dropped.
3. The positive e2e path can't run until an extension binary ships ducklake_commit_stats() (PostHog/ducklake#35 → next release tag); until then prod exercises exactly the probe-absent path, which is unit-tested against real DuckDB.

Post-fix: go build, go vet, unit tests, just lint all clean.