Skip to content

fix(OUT-3937): harden DB connection handling against transient econnrefused - #216

Open
priosshrsth wants to merge 2 commits into
mainfrom
anit/out-3937-error-failed-query-select-id-workspace_id-custom_field
Open

fix(OUT-3937): harden DB connection handling against transient econnrefused#216
priosshrsth wants to merge 2 commits into
mainfrom
anit/out-3937-error-failed-query-select-id-workspace_id-custom_field

Conversation

@priosshrsth

@priosshrsth priosshrsth commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Changes

Resolves the Failed query ... Failed to connect to database: {:error, :econnrefused} errors crashing the /client page (OUT-3937 & OUT-3936).

Root cause: not a query bug. The root layout.tsx SSRs settings on every /client request (getForClient() → 3 parallel DB queries). A momentary Supabase pooler connection refusal under serverless connection fan-out crashed the entire render. Both Sentry issues fired in the same 16:55–16:56 window from the same blip — they only differ by which of the 3 parallel queries lost the race.

  • src/db/db.ts — cap postgres connections per instance (max: 1) + add idle_timeout/connect_timeout so many concurrent Fluid/serverless instances don't exhaust the pooler and trigger econnrefused. Also drop debug: true from the prod client. This is the fix.
  • drizzle.config.ts / src/config/env.ts / .env.example — introduce optional DIRECT_URL for migrations (transaction pooler can't run DDL); falls back to DATABASE_URL when unset.

Note: a transient-error retry was considered and deliberately dropped — retrying econnrefused risks amplifying a saturated pooler (thundering herd). Capped per-instance connections address the cause directly.

Testing Criteria

  • pnpm typecheck + pnpm lint pass (changed files clean).
  • /client renders normally with a healthy DB (settings load unchanged).
  • Migrations still run (pnpm drizzle-kit migrate) with and without DIRECT_URL set.
  • Loom: TODO

Notes

⚠️ Env / infra actions required (not in this PR):

  1. Verify DATABASE_URL is the transaction-mode pooler (...pooler.supabase.com:6543, ?pgbouncer=true). max: 1 assumes the pooler. If it's the direct :5432 connection, that's the core problem.
  2. Add DIRECT_URL (session/direct :5432) in Vercel for migrations. Optional — falls back to DATABASE_URL.
  3. Supabase infra: consider a tier bump + pooler pool-size tuning for durable headroom.

Impact & Surface Area of Change

  • Every DB-backed request shares the pooled client in db.tsmax: 1 + timeouts change connection behavior app-wide. Watch for latency under load (single connection per instance) — expected fine on the transaction pooler.

Fixes CLIENT-HOME-V3-1D
Fixes CLIENT-HOME-V3-1E

🤖 Generated with Claude Code

…efused

The root layout SSRs settings on every /client request; a momentary
Supabase pooler connection refusal (econnrefused) crashed the entire
page render. Root cause is DB connection management under serverless
fan-out, not a query bug.

- cap postgres connections per instance (max: 1) + add idle/connect
  timeouts so concurrent Fluid instances don't exhaust the pooler
- drop debug logging from the prod client
- retry the root-layout settings load on transient connection errors
  (rethrows real failures) as a seatbelt
- split DIRECT_URL for migrations (transaction pooler can't run DDL)

Fixes CLIENT-HOME-V3-1D
Fixes CLIENT-HOME-V3-1E

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Jul 2, 2026

Copy link
Copy Markdown

OUT-3937

@vercel

vercel Bot commented Jul 2, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
client-home-v3 Ready Ready Preview, Comment Jul 2, 2026 8:54am

Request Review

@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown

Greptile Summary

This PR hardens the root-layout DB path against transient Supabase pooler econnrefused errors that were crashing the /client page. It caps postgres connections per serverless instance (max: 1 + timeouts), adds a retryOnTransientDbError wrapper with exponential backoff applied to the root-layout settings load, and introduces an optional DIRECT_URL so migrations can bypass the transaction-mode pooler.

  • src/db/db.ts: sets max: 1, idle_timeout: 20, connect_timeout: 10, and removes debug: true — reduces connection fan-out from concurrent serverless instances against the pooler.
  • src/lib/core/db-retry.ts: new utility that walks the Drizzle/postgres.js cause chain to classify transient errors, retries up to 2 times with exponential backoff, and rethrows non-transient errors unchanged.
  • drizzle.config.ts / src/config/env.ts / .env.example: adds optional DIRECT_URL so drizzle-kit migrate uses a session/direct connection for DDL while the app runtime continues using the pooler URL.

Confidence Score: 4/5

Safe to merge; the retry wrapper correctly isolates transient errors from real failures and all non-transient errors still propagate to Sentry unchanged.

The root cause analysis is sound and the implementation is well-structured: cause-chain walking handles Drizzle's error wrapping, the transient error set is comprehensive, and max:1 correctly caps per-instance fan-out against the transaction pooler. The one gap worth addressing is the missing jitter on the backoff — if a pooler blip drops many instances at once, deterministic retry intervals could produce a synchronized reconnect burst that re-triggers the same overload.

src/lib/core/db-retry.ts — the backoff jitter suggestion is the only item worth revisiting before a high-traffic event.

Important Files Changed

Filename Overview
src/lib/core/db-retry.ts New retry utility with cause-chain walking, TRANSIENT_ERROR_CODES set, and exponential backoff — solid logic overall; missing jitter in the backoff delay could recreate burst load on recovery.
src/db/db.ts Adds max:1 + idle/connect timeouts, removes debug:true — appropriate settings for serverless + transaction-mode pooler; intentionally serializes concurrent queries from the same instance.
src/app/layout.tsx Wraps root-layout settings fetch with retryOnTransientDbError; non-transient errors still propagate to Sentry as before.
drizzle.config.ts Adds DIRECT_URL fallback for migrations so DDL can run over session/direct connection instead of transaction pooler; falls back to DATABASE_URL when unset.
src/config/env.ts Adds DIRECT_URL as an optional validated URL in the env schema — correct and non-breaking.
.env.example Documents the new DIRECT_URL variable with inline comments explaining its role and port.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Client as Browser
    participant Layout as RootLayout (SSR)
    participant Retry as retryOnTransientDbError
    participant DB as postgres.js (max:1)
    participant Pooler as Supabase Pooler

    Client->>Layout: GET /client
    Layout->>Retry: settingsService.getForClient()
    loop Up to 3 attempts
        Retry->>DB: execute query
        DB->>Pooler: acquire connection
        alt Transient error (ECONNREFUSED etc.)
            Pooler-->>DB: connection refused
            DB-->>Retry: throws transient error
            Retry->>Retry: "sleep(baseDelay * 2^attempt)"
        else Success
            Pooler-->>DB: query result
            DB-->>Retry: settings data
            Retry-->>Layout: settings data
        end
    end
    alt All retries exhausted
        Retry-->>Layout: rethrows last error
    end
    Layout-->>Client: rendered HTML
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Client as Browser
    participant Layout as RootLayout (SSR)
    participant Retry as retryOnTransientDbError
    participant DB as postgres.js (max:1)
    participant Pooler as Supabase Pooler

    Client->>Layout: GET /client
    Layout->>Retry: settingsService.getForClient()
    loop Up to 3 attempts
        Retry->>DB: execute query
        DB->>Pooler: acquire connection
        alt Transient error (ECONNREFUSED etc.)
            Pooler-->>DB: connection refused
            DB-->>Retry: throws transient error
            Retry->>Retry: "sleep(baseDelay * 2^attempt)"
        else Success
            Pooler-->>DB: query result
            DB-->>Retry: settings data
            Retry-->>Layout: settings data
        end
    end
    alt All retries exhausted
        Retry-->>Layout: rethrows last error
    end
    Layout-->>Client: rendered HTML
Loading

Reviews (1): Last reviewed commit: "fix(OUT-3937): harden DB connection hand..." | Re-trigger Greptile

Comment thread src/lib/core/db-retry.ts Outdated
if (!isTransientDbError(error) || attempt === retries) {
throw error
}
await sleep(baseDelayMs * 2 ** attempt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing jitter in exponential backoff — when many serverless instances all hit the same transient pooler blip simultaneously (the exact failure mode this PR targets), they'll all wake up and retry at the same intervals, creating a synchronized burst against the same pooler that just recovered. Adding full jitter (random delay between 0 and the calculated ceiling) spreads retries across the window and avoids re-triggering the problem.

Suggested change
await sleep(baseDelayMs * 2 ** attempt)
await sleep(Math.random() * baseDelayMs * 2 ** attempt)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

…saturation

Retrying econnrefused can worsen a saturated pooler (thundering herd).
Rely on capped per-instance connections (max: 1) + correct pooler URL
instead. Removes db-retry helper and reverts the root-layout wrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant