Description
The worker service (from chatbotx-docker-compose, docker compose -p chatbotx up -d) exhibits rapid, uniform memory growth across ALL 13 sub-workers (chat, ai-agent, integration, heavy, webhook, notification, schedule, default, events, trigger, sequence-scheduler/consumer/producer) immediately after startup, leading to an OOM crash-loop within 10-60 seconds.
Environment
- Deployment:
chatbotx-docker-compose (github.com/chatbotxio/chatbotx-docker-compose), fresh clone
- Server: 4 vCPU / 8 GB RAM, Ubuntu 24.04.5 LTS, kernel 6.8.0-139-generic
- Docker: 29.8.1 (Engine + CLI), containerd 2.3.5, cgroup v2/systemd, overlayfs
- Images tested:
ghcr.io/chatbotxio/chatbotx-worker:latest and :1.6.0 — both reproduce the issue
- All other services (postgres/timescaledb, redis, rustfs, builder, realtime, javascript-executor) are healthy and stable
Steps to reproduce
git clone https://github.com/chatbotxio/chatbotx-docker-compose.git && cd chatbotx-docker-compose- Set real secrets via
.env + docker-compose.override.yml (all defaults from docker-compose.yml replaced with random values — same variable names/shapes)
docker compose -p chatbotx up -d- Watch
docker stats chatbotx-worker-1
Observed behavior
- At startup, 13 separate Node.js processes are spawned (one per queue), each already at 150-350 MB RSS baseline (~3.1 GB total just to boot).
- Within 10-40 seconds, ALL 13 processes grow roughly proportionally (not one outlier) to 450-600+ MB each, pushing total container memory to 5.4-6.5 GB.
redis-cli DBSIZE on the redis container shows only 10 keys (bull:<queue>:meta for each queue) — zero actual jobs queued (bull:*:wait lists are empty). So this is not a job-processing backlog.- Container is eventually killed (exit 137) and restarts (
Worker process <pid> exited with status 137, stopping others...), repeating indefinitely.
- During the crash-loop, host load average spiked to 50-78 (on a 4-vCPU box) and intermittently made SSH to the host time out — this affects the whole server, not just the container.
Expected behavior
Idle workers with an empty queue should hold roughly constant, bounded memory — not grow several hundred MB per process per few seconds with nothing to process.
Additional notes
- No relevant errors found in
docker logs chatbotx-worker-1 (only Node deprecation warnings about url.parse()), and no dmesg "Killed process" entries — the kill appears to come from system-wide memory pressure (possibly systemd-oomd) rather than a classic cgroup OOM.
- Reproduced twice independently (fresh
docker compose up -d worker each time), on both latest and 1.6.0, so this isn't a recent regression.
- Happy to provide
docker stats snapshots / heap snapshots if useful for triage — let me know what would help.
Description
The
workerservice (fromchatbotx-docker-compose,docker compose -p chatbotx up -d) exhibits rapid, uniform memory growth across ALL 13 sub-workers (chat, ai-agent, integration, heavy, webhook, notification, schedule, default, events, trigger, sequence-scheduler/consumer/producer) immediately after startup, leading to an OOM crash-loop within 10-60 seconds.Environment
chatbotx-docker-compose(github.com/chatbotxio/chatbotx-docker-compose), fresh cloneghcr.io/chatbotxio/chatbotx-worker:latestand:1.6.0— both reproduce the issueSteps to reproduce
git clone https://github.com/chatbotxio/chatbotx-docker-compose.git && cd chatbotx-docker-compose.env+docker-compose.override.yml(all defaults fromdocker-compose.ymlreplaced with random values — same variable names/shapes)docker compose -p chatbotx up -ddocker stats chatbotx-worker-1Observed behavior
redis-cli DBSIZEon therediscontainer shows only 10 keys (bull:<queue>:metafor each queue) — zero actual jobs queued (bull:*:waitlists are empty). So this is not a job-processing backlog.Worker process <pid> exited with status 137, stopping others...), repeating indefinitely.Expected behavior
Idle workers with an empty queue should hold roughly constant, bounded memory — not grow several hundred MB per process per few seconds with nothing to process.
Additional notes
docker logs chatbotx-worker-1(only Node deprecation warnings abouturl.parse()), and nodmesg"Killed process" entries — the kill appears to come from system-wide memory pressure (possiblysystemd-oomd) rather than a classic cgroup OOM.docker compose up -d workereach time), on bothlatestand1.6.0, so this isn't a recent regression.docker statssnapshots / heap snapshots if useful for triage — let me know what would help.