Skip to content

perf(worker): harden chat queue dispatch (concurrency, dedup, retries) - #1195

Open
realcodesiman wants to merge 4 commits into
mainfrom
perf/chat-queue-dispatch-hardening
Open

realcodesiman wants to merge 4 commits into
mainfrom
perf/chat-queue-dispatch-hardening

Conversation

@realcodesiman

@realcodesiman realcodesiman commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Implements P0 items B1, B2, B5 from the system-design-review audit.

B1 — chat queue concurrency + rate limit

apps/worker/src/chat/worker.ts hardcoded concurrency: 5 with no limiter — the first bottleneck for broadcast/flow/template sends. Added CHAT_WORKER_CONCURRENCY (default 20), CHAT_WORKER_RATE_LIMIT_MAX/_DURATION_MS (default 80/1000ms) env vars in apps/worker/src/env.ts, matching the existing per-queue concurrency-var convention (WEBHOOK_WORKER_CONCURRENCY etc.). Note: BullMQ OSS's limiter is queue-wide, not per-channel/page — a coarse throughput cap, not a per-inbox token bucket (that would be a separate, bigger feature, out of scope here).

B2 — deterministic jobId for incomingMessage

Messenger, Instagram, WhatsApp, Telegram, TikTok, and Zalo webhook handlers enqueued incomingMessage with no jobId, so a provider retry storm created duplicate jobs (dedup only happened later, in the DB). Added toBullMqSafeIdSegment to @chatbotx.io/utils (removed the duplicate private copy in the whatsapp package) and added a deterministic jobId per channel (incoming-<channel>-<sourceId>), built only when a source id is available. TikTok's existing delay: 2000 echo-race option is preserved (merged into the same options object, not replaced).

B5 — stronger retry policy for chat/integration queues

Added resilientJobOptions (attempts: 5, exponential 10s backoff) in packages/worker-config/src/lib/connection.ts, used only by the chat and integration queues (every other queue keeps defaultJobOptions, attempts: 2). No custom jitter — BullMQ's built-in exponential backoff has none; documenting rather than fabricating one.

Verification

  • pnpm --filter @chatbotx.io/worker-config check-types && test — 46/46 pass
  • pnpm --filter worker check-types && test — 2339/2339 pass
  • Touched integration packages (messenger, whatsapp, instagram, telegram, tiktok, zalo, utils) check-types + test green

@github-actions github-actions Bot added the improvement Refactor or performance improvement label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Refactor or performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant