Raise the DB connection pool off its development default - #193
Merged
Conversation
The pool was ten connections. That is survivable for a single editor and fatal under public traffic: Scalene calls /v1/settings/site on every page render, and with ten connections those requests serialize behind each other. Measured on the public cutover, /v1/settings/site took 25-125 seconds and the site timed out behind Cloudflare 524s. What makes this hard to see is that nothing looks busy. The database was at two running threads and 14 of 200 connections, the CMS host at 0.02 load, the database host and MaxScale both nearly idle. Every machine reports plenty of headroom, because the contention is in the queue in front of the database rather than in the database itself -- the pool is a limit the metrics do not mention. Default is now 50. DB1 allows 200, so two blue/green slots at 50 leaves room for MaxScale's monitor, the replica, and an operator session. DB_MAX_OPEN_CONNS overrides it, because this failure mode only appears under real traffic and resizing the pool should not require a rebuild. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production is currently down behind this. Merging deploys the fix.
What happened
database.gocapped the pool atSetMaxOpenConns(10). Scalene calls/v1/settings/siteon every page render, so under public traffic those requests serialize behind ten connections. On the cutover that endpoint was taking 25–125 seconds, Scalene's fetches returned Cloudflare 524, and www timed out entirely.Why it was hard to spot
Every host reported idle:
The database is idle because only ten requests can reach it at once. The contention is in the queue in front of it, and no host metric names that queue — which is what makes this look like a resource shortage on some other machine.
The inactive blue/green slot answered
/v1/health/dbin 0.5 ms throughout, purely because it was serving no traffic. That contrast is the tell.The change
Default 50, overridable with
DB_MAX_OPEN_CONNS. DB1 allows 200 connections, so two slots at 50 leaves headroom for MaxScale's monitor, the replica, and an operator session. The env override exists because this only reproduces under real traffic.Follow-up worth doing separately
/v1/settings/siteis a per-render call on a value that changes rarely; a short TTL cache would cut the query volume by orders of magnitude regardless of pool size. Not in this PR — this one should stay minimal enough to merge while the site is down.Verified:
go build ./...,go vet,go test ./internal/database/all clean.🤖 Generated with Claude Code