Skip to content

Size workers by the CPUs the container may use - #384

Open
namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:size-workers-by-cpu-quota
Open

namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:size-workers-by-cpu-quota

Conversation

@namespaceMarcello

Copy link
Copy Markdown
Contributor

Summary

Puma and the Resque pool size themselves from Concurrent.processor_count, which counts every CPU of the host. A CPU limit doesn't change that number: ONCE's CPU setting gives the container a CPU quota (NanoCPUs, what docker run --cpus sets), as Kubernetes limits do, so a Campfire limited to 2 CPUs on a 16-CPU host starts 11 web workers and 8 job workers, which share two CPUs' worth of time and each hold their own memory. ONCE also passes NUM_CPUS for this case ("An application can use this to vary the number of worker processes it spawns to an appropriate number for that quota"), and Campfire doesn't read it.

This PR counts with Concurrent.available_processor_count in config/puma.rb and config/resque-pool.yml. It returns the container's CPU quota when it has one and processor_count otherwise, never more than processor_count, so a host without a limit keeps the counts it has today.

Impact, on our machine (laptop, Docker Desktop), production images built from main (eccb2d6) and from this branch, each with --cpus 2 on 16 visible CPUs and 50 seeded people:

  • Processes: 25 on main (11 web, 8 jobs), 9 here (2 web, 1 job).
  • Memory, sum of PSS of every process after ~25 s of pages and posts: 2,197 MB on main, 652 MB here.
  • CPU with nobody connected: 62 ms per second on main, 12 here. Most of it is the job workers polling Redis every 0.1 s, eight of them on main.
  • The room page from 8 connections, warmed up, five alternated rounds of 15 s: 707 requests per second on main, 803 here (medians; every round here was above every round on main). p99 81 ms on main, 51 here. p50 2.7 ms on main, 5.7 here: 8 connections on 2 processes now take turns on each process's GVL, where on main each request had a process of its own and waited out the quota's throttling instead (475 s throttled over the container's run on main, 69 s here).

Affects anyone running Campfire under a CPU quota: ONCE with a CPU limit set, docker run --cpus, Kubernetes limits. Hosts without one are unchanged: a 4-CPU cpuset starts 3 web and 2 job workers on both, 1 CPU starts 1 and 1. The benchmarks in bench/ and in once-campfire-verification pin CPUs with --cpuset-cpus, so their counts don't change either.

Rails dropped available_processor_count as the default worker count of its puma.rb template (rails/rails#52522) because some platforms expose no quota, and there it counts every CPU of the host. That is today's behavior here: where no quota is visible, available_processor_count gives what processor_count gives.

Risk: under a CPU limit there are fewer processes, the count a host with that many cores gets. Jobs run on ceil(limit × 0.5) workers, one at a 2-CPU limit, so a bot webhook that takes its full 7 s (Webhook::ENDPOINT_TIMEOUT) holds the only worker meanwhile, as it does on a 2-core host today; web push deliveries use their own thread pool. WEB_CONCURRENCY still sets the web count. Reverting the commit restores today's counts.

Changes

  • config/puma.rb: worker_count from Concurrent.available_processor_count, with a comment on why. JOB_CONCURRENCY keeps following it; Replace Resque with Solid Queue #346 sizes Solid Queue from that variable, so it would pick this up too.
  • config/resque-pool.yml: the same for the job workers.

Checks

  • Images built from main and from this branch, started with --cpus 2 on 16 visible CPUs, without a limit on 4 CPUs, and on 1 CPU, counting Puma's workers and the pool's workers once /up answers: 11 + 8, 3 + 2, 1 + 1 on main; 2 + 1, 3 + 2, 1 + 1 here.
  • On this branch, config/puma.rb alone put back as on main brings back 11 web workers under the limit; config/resque-pool.yml alone brings back 8 job workers.
  • concurrent-ruby 1.3.8 reads the quota from cpu.max (cgroup v2) or cpu.cfs_quota_us (v1) and falls back to processor_count when there is none. In the image under a 1.5-CPU limit it returns 1.5, which the two formulas turn into 1 web and 1 job worker.
  • Unit suite (649 tests), rubocop, herb, brakeman and the 30 system tests, locally (Docker, headless Chrome).

Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.

Puma and the Resque pool count their workers from
Concurrent.processor_count, which sees every CPU of the host. A container
started with a CPU limit (ONCE's CPU setting, docker run --cpus) still
gets a worker for every host core: on a 16-CPU host with a 2-CPU limit,
11 web workers and 8 job workers share two CPUs' worth of time, and the
memory of all of them.

Concurrent.available_processor_count is the container's CPU quota when
it has one and processor_count otherwise. It is never more than
processor_count, so a host without a limit keeps the counts it has today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 23:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

Both changes use the supported dependency API, preserve existing formulas and overrides, and have no identified blocking issues.

No open issues 🎉

What changed in this PR

This PR makes Puma and Resque worker counts respect container CPU quotas, reducing excess processes while preserving existing sizing formulas and behavior on hosts without quotas.

Changes:

  • Use Concurrent.available_processor_count for web and job worker sizing.
  • Document quota-aware sizing in Puma while retaining WEB_CONCURRENCY overrides and JOB_CONCURRENCY propagation.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
config/​resque-pool.yml Size job workers using the quota-aware CPU count.
config/​puma.rb Size web workers using the quota-aware CPU count and explain the behavior.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants