fix(OUT-3937): harden DB connection handling against transient econnrefused - #216
Conversation
…efused The root layout SSRs settings on every /client request; a momentary Supabase pooler connection refusal (econnrefused) crashed the entire page render. Root cause is DB connection management under serverless fan-out, not a query bug. - cap postgres connections per instance (max: 1) + add idle/connect timeouts so concurrent Fluid instances don't exhaust the pooler - drop debug logging from the prod client - retry the root-layout settings load on transient connection errors (rethrows real failures) as a seatbelt - split DIRECT_URL for migrations (transaction pooler can't run DDL) Fixes CLIENT-HOME-V3-1D Fixes CLIENT-HOME-V3-1E Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR hardens the root-layout DB path against transient Supabase pooler
Confidence Score: 4/5Safe to merge; the retry wrapper correctly isolates transient errors from real failures and all non-transient errors still propagate to Sentry unchanged. The root cause analysis is sound and the implementation is well-structured: cause-chain walking handles Drizzle's error wrapping, the transient error set is comprehensive, and max:1 correctly caps per-instance fan-out against the transaction pooler. The one gap worth addressing is the missing jitter on the backoff — if a pooler blip drops many instances at once, deterministic retry intervals could produce a synchronized reconnect burst that re-triggers the same overload. src/lib/core/db-retry.ts — the backoff jitter suggestion is the only item worth revisiting before a high-traffic event. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Client as Browser
participant Layout as RootLayout (SSR)
participant Retry as retryOnTransientDbError
participant DB as postgres.js (max:1)
participant Pooler as Supabase Pooler
Client->>Layout: GET /client
Layout->>Retry: settingsService.getForClient()
loop Up to 3 attempts
Retry->>DB: execute query
DB->>Pooler: acquire connection
alt Transient error (ECONNREFUSED etc.)
Pooler-->>DB: connection refused
DB-->>Retry: throws transient error
Retry->>Retry: "sleep(baseDelay * 2^attempt)"
else Success
Pooler-->>DB: query result
DB-->>Retry: settings data
Retry-->>Layout: settings data
end
end
alt All retries exhausted
Retry-->>Layout: rethrows last error
end
Layout-->>Client: rendered HTML
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Client as Browser
participant Layout as RootLayout (SSR)
participant Retry as retryOnTransientDbError
participant DB as postgres.js (max:1)
participant Pooler as Supabase Pooler
Client->>Layout: GET /client
Layout->>Retry: settingsService.getForClient()
loop Up to 3 attempts
Retry->>DB: execute query
DB->>Pooler: acquire connection
alt Transient error (ECONNREFUSED etc.)
Pooler-->>DB: connection refused
DB-->>Retry: throws transient error
Retry->>Retry: "sleep(baseDelay * 2^attempt)"
else Success
Pooler-->>DB: query result
DB-->>Retry: settings data
Retry-->>Layout: settings data
end
end
alt All retries exhausted
Retry-->>Layout: rethrows last error
end
Layout-->>Client: rendered HTML
Reviews (1): Last reviewed commit: "fix(OUT-3937): harden DB connection hand..." | Re-trigger Greptile |
| if (!isTransientDbError(error) || attempt === retries) { | ||
| throw error | ||
| } | ||
| await sleep(baseDelayMs * 2 ** attempt) |
There was a problem hiding this comment.
Missing jitter in exponential backoff — when many serverless instances all hit the same transient pooler blip simultaneously (the exact failure mode this PR targets), they'll all wake up and retry at the same intervals, creating a synchronized burst against the same pooler that just recovered. Adding full jitter (random delay between 0 and the calculated ceiling) spreads retries across the window and avoids re-triggering the problem.
| await sleep(baseDelayMs * 2 ** attempt) | |
| await sleep(Math.random() * baseDelayMs * 2 ** attempt) |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…saturation Retrying econnrefused can worsen a saturated pooler (thundering herd). Rely on capped per-instance connections (max: 1) + correct pooler URL instead. Removes db-retry helper and reverts the root-layout wrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Changes
Resolves the
Failed query ... Failed to connect to database: {:error, :econnrefused}errors crashing the/clientpage (OUT-3937 & OUT-3936).Root cause: not a query bug. The root
layout.tsxSSRs settings on every/clientrequest (getForClient()→ 3 parallel DB queries). A momentary Supabase pooler connection refusal under serverless connection fan-out crashed the entire render. Both Sentry issues fired in the same 16:55–16:56 window from the same blip — they only differ by which of the 3 parallel queries lost the race.src/db/db.ts— cap postgres connections per instance (max: 1) + addidle_timeout/connect_timeoutso many concurrent Fluid/serverless instances don't exhaust the pooler and triggereconnrefused. Also dropdebug: truefrom the prod client. This is the fix.drizzle.config.ts/src/config/env.ts/.env.example— introduce optionalDIRECT_URLfor migrations (transaction pooler can't run DDL); falls back toDATABASE_URLwhen unset.Note: a transient-error retry was considered and deliberately dropped — retrying
econnrefusedrisks amplifying a saturated pooler (thundering herd). Capped per-instance connections address the cause directly.Testing Criteria
pnpm typecheck+pnpm lintpass (changed files clean)./clientrenders normally with a healthy DB (settings load unchanged).pnpm drizzle-kit migrate) with and withoutDIRECT_URLset.Notes
DATABASE_URLis the transaction-mode pooler (...pooler.supabase.com:6543,?pgbouncer=true).max: 1assumes the pooler. If it's the direct:5432connection, that's the core problem.DIRECT_URL(session/direct:5432) in Vercel for migrations. Optional — falls back toDATABASE_URL.Impact & Surface Area of Change
db.ts—max: 1+ timeouts change connection behavior app-wide. Watch for latency under load (single connection per instance) — expected fine on the transaction pooler.Fixes CLIENT-HOME-V3-1D
Fixes CLIENT-HOME-V3-1E
🤖 Generated with Claude Code