Repository navigation
Size workers by the CPUs the container may use - #384
Open
namespaceMarcello wants to merge 1 commit into
Open
namespaceMarcello wants to merge 1 commit into
namespaceMarcello wants to merge 1 commit into
Conversation
Puma and the Resque pool count their workers from Concurrent.processor_count, which sees every CPU of the host. A container started with a CPU limit (ONCE's CPU setting, docker run --cpus) still gets a worker for every host core: on a 16-CPU host with a 2-CPU limit, 11 web workers and 8 job workers share two CPUs' worth of time, and the memory of all of them. Concurrent.available_processor_count is the container's CPU quota when it has one and processor_count otherwise. It is never more than processor_count, so a host without a limit keeps the counts it has today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approved
Both changes use the supported dependency API, preserve existing formulas and overrides, and have no identified blocking issues.
No open issues 🎉
What changed in this PR
This PR makes Puma and Resque worker counts respect container CPU quotas, reducing excess processes while preserving existing sizing formulas and behavior on hosts without quotas.
Changes:
- Use
Concurrent.available_processor_countfor web and job worker sizing. - Document quota-aware sizing in Puma while retaining
WEB_CONCURRENCYoverrides andJOB_CONCURRENCYpropagation.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
config/resque-pool.yml |
Size job workers using the quota-aware CPU count. |
config/puma.rb |
Size web workers using the quota-aware CPU count and explain the behavior. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Puma and the Resque pool size themselves from
Concurrent.processor_count, which counts every CPU of the host. A CPU limit doesn't change that number: ONCE's CPU setting gives the container a CPU quota (NanoCPUs, whatdocker run --cpussets), as Kubernetes limits do, so a Campfire limited to 2 CPUs on a 16-CPU host starts 11 web workers and 8 job workers, which share two CPUs' worth of time and each hold their own memory. ONCE also passesNUM_CPUSfor this case ("An application can use this to vary the number of worker processes it spawns to an appropriate number for that quota"), and Campfire doesn't read it.This PR counts with
Concurrent.available_processor_countinconfig/puma.rbandconfig/resque-pool.yml. It returns the container's CPU quota when it has one andprocessor_countotherwise, never more thanprocessor_count, so a host without a limit keeps the counts it has today.Impact, on our machine (laptop, Docker Desktop), production images built from
main(eccb2d6) and from this branch, each with--cpus 2on 16 visible CPUs and 50 seeded people:main(11 web, 8 jobs), 9 here (2 web, 1 job).main, 652 MB here.main, 12 here. Most of it is the job workers polling Redis every 0.1 s, eight of them onmain.main, 803 here (medians; every round here was above every round onmain). p99 81 ms onmain, 51 here. p50 2.7 ms onmain, 5.7 here: 8 connections on 2 processes now take turns on each process's GVL, where onmaineach request had a process of its own and waited out the quota's throttling instead (475 s throttled over the container's run onmain, 69 s here).Affects anyone running Campfire under a CPU quota: ONCE with a CPU limit set,
docker run --cpus, Kubernetes limits. Hosts without one are unchanged: a 4-CPU cpuset starts 3 web and 2 job workers on both, 1 CPU starts 1 and 1. The benchmarks inbench/and in once-campfire-verification pin CPUs with--cpuset-cpus, so their counts don't change either.Rails dropped
available_processor_countas the default worker count of itspuma.rbtemplate (rails/rails#52522) because some platforms expose no quota, and there it counts every CPU of the host. That is today's behavior here: where no quota is visible,available_processor_countgives whatprocessor_countgives.Risk: under a CPU limit there are fewer processes, the count a host with that many cores gets. Jobs run on
ceil(limit × 0.5)workers, one at a 2-CPU limit, so a bot webhook that takes its full 7 s (Webhook::ENDPOINT_TIMEOUT) holds the only worker meanwhile, as it does on a 2-core host today; web push deliveries use their own thread pool.WEB_CONCURRENCYstill sets the web count. Reverting the commit restores today's counts.Changes
config/puma.rb:worker_countfromConcurrent.available_processor_count, with a comment on why.JOB_CONCURRENCYkeeps following it; Replace Resque with Solid Queue #346 sizes Solid Queue from that variable, so it would pick this up too.config/resque-pool.yml: the same for the job workers.Checks
mainand from this branch, started with--cpus 2on 16 visible CPUs, without a limit on 4 CPUs, and on 1 CPU, counting Puma's workers and the pool's workers once/upanswers: 11 + 8, 3 + 2, 1 + 1 onmain; 2 + 1, 3 + 2, 1 + 1 here.config/puma.rbalone put back as onmainbrings back 11 web workers under the limit;config/resque-pool.ymlalone brings back 8 job workers.cpu.max(cgroup v2) orcpu.cfs_quota_us(v1) and falls back toprocessor_countwhen there is none. In the image under a 1.5-CPU limit it returns 1.5, which the two formulas turn into 1 web and 1 job worker.Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.