Skip to content

Commit 68aed04

Browse files
committed
fix(plugin-email): sweep sys_email rows stranded at queued, and report a failed drain at error (#5161)
`status:'queued'` had exactly one consumer — the afterInsert outbox drain that fires during the insert itself (plus, since #5160, the email.send.async job send() publishes). A process that died between the insert and the delivery, or a drain whose delivery threw, left the row at `queued` forever: a state named after a queue with no reader, with the caller already told the message was accepted. - `sweepStrandedOutbox` runs once per boot at kernel:ready, after the queue subscriber and the #5160 boot gate. Queue mode publishes `{ rowId }` through EmailService.enqueuePersistedRow (send()'s own producer, options and `sys_email:<id>` idempotency key, so a row with a pending job collapses onto it); inline mode finalizes the row in place via deliverPersistedRow. - Only rows older than OUTBOX_SWEEP_MIN_AGE_MS (5m) are eligible — a young row is somebody's in-flight work, on this instance or a sibling, and age is the only property that means the same thing on every instance. Service-managed rows and rows carrying a message_id are skipped. Batch bounded at 500, oldest first, truncation reported. - Boot does not await the sweep; it self-catches and reports at error, since a throwing kernel:ready handler is swallowed on LiteKernel (#5170). - Both drain-hook catches now log at error with the consequence (the message was NOT sent, the row stays at `queued`) and the fix, per the AGENTS.md degradation-log-level rule, and deliverPersistedRow joins DURABILITY_CRITICAL_CALLEES so the level cannot regress. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017MCKJaEomEqg4tvz4SzdNd
1 parent 9c4f174 commit 68aed04

8 files changed

Lines changed: 1048 additions & 2 deletions

File tree

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
"@objectstack/plugin-email": minor
3+
---
4+
5+
fix(plugin-email): `sys_email` rows stranded at `queued` are swept at boot, and a failed drain says so at `error` (#5161)
6+
7+
`status: 'queued'` had exactly one consumer: the `afterInsert` outbox drain that
8+
fires during the insert itself (plus, since #5160, the `email.send.async` job
9+
`send()` publishes). Nothing ever looked at such a row again. A process that
10+
died between the insert and the delivery — or a drain whose delivery threw —
11+
left the row at `queued` **forever**: a state named after a queue that had no
12+
reader, while the caller had already been told the message was accepted.
13+
14+
**A once-per-boot sweep is now that reader.** At `kernel:ready`, after the
15+
registries are settled and the `email.send.async` subscriber is attached,
16+
`sweepStrandedOutbox` picks up `sys_email` rows still at `queued` and advances
17+
them:
18+
19+
- **durable queue delivery on** → the row is published as an `{ rowId }` job to
20+
`email.send.async` through the same producer, options and
21+
`sys_email:<id>` idempotency key `send()` uses, so a row that still has a
22+
pending job collapses onto it instead of putting a second worker on it;
23+
- **inline delivery** → the row is delivered and finalized in place (`sent` /
24+
`failed`), which is what the drain hook would have done had the process lived.
25+
26+
Only rows **older than five minutes** are eligible. A row inserted seconds ago
27+
is not stranded, it is someone's in-flight work — this process's `send()`, its
28+
deferred drain hook, or the same on another instance — and sweeping it would
29+
send that message twice. (Age, not "created before this boot": one instance's
30+
boot time says nothing about a sibling's row inserted a second ago.) Rows this
31+
process is delivering right now, and rows that already carry a `message_id`, are
32+
skipped. The batch is bounded at 500 rows per boot, oldest first, and says so
33+
when it truncates. One `info` line reports the counts; boot does **not** wait on
34+
the sweep, and a sweep that cannot run reports at `error` rather than relying on
35+
`kernel:ready` error propagation.
36+
37+
**Drain-hook failures are now `error`, not `warn`.** A drain that throws means
38+
the mail was not sent while the insert reported success and the row still reads
39+
`queued` — the durability class the degradation-log-level rule pins at `error`.
40+
Both lines now name the consequence (this message was NOT sent, the row stays at
41+
`queued`) and the fix (the boot sweep picks it up on the next restart; turn on
42+
durable queue delivery to have failures retried and dead-lettered instead).
43+
`deliverPersistedRow` joins `DURABILITY_CRITICAL_CALLEES`, so a future `catch`
44+
that quietly downgrades it fails `pnpm check:durability-log-level`.
45+
46+
New exports: `sweepStrandedOutbox`, `OUTBOX_OBJECT`, `OUTBOX_SWEEP_MIN_AGE_MS`,
47+
`OUTBOX_SWEEP_LIMIT`, `EmailService.enqueuePersistedRow`, and
48+
`EmailServicePlugin.outboxSweepSettled` (the sweep's promise, for callers that
49+
need determinism). The normal `send()` → deliver path is byte-for-byte
50+
unchanged.

0 commit comments

Comments
 (0)