Skip to content

fix(health): report degraded when the queue is backed up with nothing active - #72

Open
valentinpanizza wants to merge 1 commit into
diwenne:mainfrom
valentinpanizza:fix/health-detects-stuck-queue
Open

valentinpanizza wants to merge 1 commit into
diwenne:mainfrom
valentinpanizza:fix/health-detects-stuck-queue

Conversation

@valentinpanizza

Copy link
Copy Markdown

What happens

/api/health answers 200 ok while the worker has stopped consuming and the
queue piles up. Every documented use of the endpoint — docs/setup.md,
docs/deploy-dokploy.md, docs/zernio.md — points people at it to confirm the
worker is healthy, and in this failure mode it tells them it is.

On my instance the BullMQ consumer stopped taking jobs while the process stayed
up. The container was running, the heartbeat kept ticking, /api/health kept
returning 200, and the uptime monitor pointed at it never fired. 385 jobs sat
waiting for hours
; every one of them was somebody who had commented and was
waiting for their link. I only found it because a user messaged the account
owner to ask what happened.

Why the current check misses it

The worker check reads getWorkerHealth(), which is the heartbeat written by an
interval in worker/dm-worker.ts:

const heartbeatTimer = setInterval(() => void heartbeat(), HEARTBEAT_INTERVAL_MS);

That timer is independent of the BullMQ Worker. If the consumer dies — dropped
queue connection, crashed consumer loop — the timer keeps running and the
heartbeat stays fresh, so worker.healthy stays true.

checkQueue() already fetches the counts that would reveal it, but only uses
them as reporting: it returns status: "ok" as long as the call itself did not
throw.

The fix

Use the counts that are already being fetched. A backlog with nothing in flight
is an unambiguous signal — a healthy worker with concurrency: 5 never leaves
jobs waiting while active is 0:

if (waiting >= STUCK_QUEUE_MIN_WAITING && active === 0) {
  return { status: "error", detail: `${waiting} jobs waiting with none active …`, counts };
}

HEALTH_STUCK_QUEUE_WAITING defaults to 25, purely to ride out the moment
between a job being enqueued and the worker picking it up. Anyone whose workload
is burstier can raise it.

No new queries, no new dependency — the counts were already in the response.

Why it matters beyond one deployment

This is the failure mode the endpoint is least able to survive being wrong
about. A worker that is down takes the heartbeat with it and is already
caught. A worker that is up but idle is invisible, and it is the one that
silently stops serving users while every dashboard stays green.

… active

A fresh heartbeat does not mean the worker is still consuming. The heartbeat
runs on its own interval in worker/dm-worker.ts, independent of the BullMQ
Worker, so the consumer can stop taking jobs — dropped queue connection,
crashed consumer loop — while the process and its heartbeat stay alive.

/api/health then keeps answering 200 while the backlog grows. On my instance
385 jobs sat waiting for hours behind an uptime monitor that never alerted;
every one was someone who had commented and was waiting for their link. All
three docs that mention the endpoint point people at it to confirm the worker
is healthy, and it reported healthy throughout.

checkQueue() already fetched the counts that reveal this and used them only as
reporting. Use them: a backlog with nothing in flight is unambiguous, since a
healthy worker with concurrency 5 never leaves jobs waiting while active is 0.
HEALTH_STUCK_QUEUE_WAITING (default 25) only exists to ride out the moment
between enqueue and pickup.
@vercel

vercel Bot commented Sep 22, 2026

Copy link
Copy Markdown

@valentinpanizza is attempting to deploy a commit to the diwenne's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant