Skip to content

fix(realtime): a committed doc mention must not 500 on a Redis outage - #217

Merged
KIDA-MNESIA merged 1 commit into
mainfrom
fix/perf-h1-redis-failopen
Sep 5, 2026
Merged

fix(realtime): a committed doc mention must not 500 on a Redis outage#217
KIDA-MNESIA merged 1 commit into
mainfrom
fix/perf-h1-redis-failopen

Conversation

@KIDA-MNESIA

Copy link
Copy Markdown
Collaborator

Closes the last two gaps in PERF-H1 from the security/performance review (Redis outages cause unbounded request waits and offline-queue growth). The structural half of that finding has already landed: redis.ts:13-22 bounds the command client (maxRetriesPerRequest: 1 / enableOfflineQueue: false / commandTimeout: 2s) while leaving the subscriber free to reconnect, and realtime-outbox.ts moved durable mutations onto a transactional outbox. Two call sites never got the treatment.

1. The doc-mention toast was published after COMMIT, unguarded

processDocMention() (ws.ts) writes document_mentions — and an agent_log row for mentioned agents — then committed, then did:

await publish(CH_DOC_MENTION, event)

document_mentions is not just a record, it is the dedup ledger: the loop above it skips any (document_id, mentioner_id, mentioned_id) tuple written in the last 60 seconds. So on a Redis outage:

  1. the mention rows commit,
  2. publish throws, the caller sees a failure for work that succeeded,
  3. the retry lands inside its own 60s window, freshRows comes back empty, and it returns early.

The notice isn't late — it's gone. This is exactly the "after PostgreSQL commits, await publish() can hang / throw" shape the review flagged, on the one durable path that hadn't been converted.

Fixed by enqueueing on the same transaction. mentionedIds is settled before COMMIT, so enqueueBroadcast(client, CH_DOC_MENTION, event) fits in-band, followed by nudgeRealtimeOutbox() after commit — the pattern onboardCompany.ts:354-365, polls.ts, membership.ts and tools.ts already use. A degraded Redis now only delays the toast; the outbox worker retries with backoff and the dedup window is never spent on a delivery that didn't happen.

2. The human typing indicator 500'd on a Redis outage

POST /conversations/:id/typing published bare. Typing is pure ephemera the renderer expires on its own, and the composer fires one of these every few seconds while a user types — turning a Redis blip into a steady stream of 500s teaches the client to hammer the error path for something nobody needs delivered. Now .catch()-logged and still { ok: true }, matching how the agent-side emitters on the very same channel already behave (inproc-client.ts:557-566, scheduler.ts:450).

What is deliberately unchanged

Every other publish() in the tree is either already .catch()-guarded, already inside a try, or genuinely ephemeral (doc awareness, presence, wake/control bus — all of which have a documented durable fallback). sub keeps maxRetriesPerRequest: null on purpose: a subscriber must reconnect indefinitely, which is the separation the review asked for.

Verification

processDocMention is now exported so the integration suite can drive it directly — the only production caller is a doc.mention.notify WS frame, and standing up a socket to assert a durable/outbox contract would test the transport instead. The new case in realtime-outbox.test.ts reuses the file's existing publishFn fault-injection:

  • after the call, document_mentions has 1 row and the outbox holds exactly 1 pending cumora:doc.mention event — proving the enqueue is in-band, not post-COMMIT
  • a drain with a throwing publisher leaves { claimed: 1, published: 0, failed: 1 } and both rows intact
  • replay delivers the event once, with mentionedIds preserved, and the ledger still holds exactly one mention

Local (this machine has no Postgres/Redis and no usable Docker daemon, so npm test / npm run test:integration are covered by PR CI's service containers — same situation as #212):

npm run lint                    ✅ 502 files
npm run typecheck               ✅
npm run server:typecheck        ✅
npm run guard:big-brain         ✅
npm run guard:llm-tracked       ✅
npm run guard:engine-registry   ✅

`document_mentions` is the dedup ledger — processDocMention() skips any
(doc, mentioner, mentioned) tuple written in the last 60 seconds. The
toast was published AFTER that transaction committed, with no guard, so
a Redis outage threw for work that had already succeeded. The retry then
found its own rows inside the dedup window and did nothing: the notice
was gone for good, not merely late.

Move the event onto the transactional outbox that every other durable
mutation already uses. mentionedIds is settled before COMMIT, so the
enqueue fits in-band and a degraded Redis only delays delivery.

Also fail-open the human typing indicator. It is pure ephemera the
renderer expires on its own, and a composer that POSTs one every few
seconds should not be taught to hammer the error path when Redis is
down — the agent-side emitters on the same channel already swallow it.
@KIDA-MNESIA
KIDA-MNESIA merged commit 862fa83 into main Sep 5, 2026
13 of 14 checks passed
@KIDA-MNESIA
KIDA-MNESIA deleted the fix/perf-h1-redis-failopen branch September 5, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant