feat(worker): move coexist media backfill jobs to a dedicated low-priority queue - #1253
Merged
Merged
Conversation
…ority queue Move coexistAttachmentDownload and updateContactAvatar off the latency-sensitive integration queue onto a new workload-class `low` queue + worker, so a historical import burst no longer starves the customer-reply path. - add the `low` queue (queueNames + queues/low) and LOW_WORKER_CONCURRENCY (default 30) - new src/low/worker.ts consumer wrapped in withBlockedOwnerGuard, high concurrency - extract a shared, channel-agnostic enqueueAttachmentDownloadJobs helper (DRY); each caller keeps its original error policy (WhatsApp/Messenger best-effort, Instagram propagates so the run re-drives before the watermark advances) - point coexist / customer-scan enqueues at the low queue; the integration worker keeps the old cases for a two-phase cutover so in-flight jobs are not dropped - tests: low worker routing + blocked-owner guard, attachment + avatar enqueue helpers
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Move
coexistAttachmentDownloadandupdateContactAvataroff the latency-sensitiveintegrationqueue onto a new workload-classlowqueue + worker, so a historical-import burst no longer starves the customer-reply path.Changes
lowqueue (queueNames+queues/low) and workersrc/low/worker.ts(wrapped inwithBlockedOwnerGuard), tunable viaLOW_WORKER_CONCURRENCY(default 30).enqueueAttachmentDownloadJobshelper; each caller keeps its original error policy (WhatsApp/Messenger best-effort, Instagram propagates).lowqueue.Two-phase cutover (safety)
Release 1 (this PR): producers enqueue to
low, but theintegrationworker still handles the two actions so in-flight jobs already queued are not dropped. Release 2 (later, once theintegrationqueue is drained) removes the old cases.Deploy note
Deploy the batch stack so a
worker lowconsumer is running, otherwise the jobs queue inlowunprocessed.Test plan
pnpm --filter worker check-types,pnpm --filter @chatbotx.io/worker-config check-typespnpm --filter worker test(242 files, 2644 tests)pnpm --filter worker buildemitsdist/low/worker.mjs