fix(api): retain channel final-delivery jobs after ambiguous Queue sends

[AsperforMias]

Area

API

Summary

Channel final-delivery jobs are written to D1 before CHANNEL_FINAL_DELIVERY_QUEUE.send(), but the producer deletes the job whenever send() rejects. A producer-side timeout or connection reset does not prove that Cloudflare Queue discarded the message: Queue may already have accepted it.

If that happens, the retained Queue message reaches the consumer after its D1 ledger row has been deleted. The consumer logs channel-final-delivery.ledger_missing and acknowledges the message without sending the final reply.

This is the same distributed-systems failure mode addressed for api_command in #284 / #299, but it remains in the separate channel_final_delivery_job pipeline.

Steps to reproduce

1. Use the local SQLite D1 fixture and the channel final-delivery Queue stub.
2. Create a completed Telegram final-delivery job.
3. Replace CHANNEL_FINAL_DELIVERY_QUEUE.send() with a fault-injection implementation that first records the message as accepted, then throws "producer timeout after Queue acceptance".
4. Call enqueueChannelFinalDeliveryJob() for a second external event.
5. Observe that the producer rejects and the Queue still contains the target jobId.
6. Query channel_final_delivery_job for that jobId: the row has been deleted by the producer error path.
7. Deliver the retained Queue message to processChannelFinalDeliveryMessage().

Expected behavior

A successfully persisted final-delivery job must remain durable when the immediate Queue producer outcome is ambiguous. It should be marked pending/failed for a safe redrive, and duplicate Queue messages must remain harmless through the existing job claim/idempotency logic.

The consumer must not silently discard a message solely because the producer previously removed its ledger after an ambiguous send response.

Actual behavior

The local fault injection reproduces the loss sequence:

• the Queue stub recorded the target message;
• enqueueChannelFinalDeliveryJob() rejected after the simulated timeout;
• the target ledger row was absent before consumption;
• the consumer logged channel-final-delivery.ledger_missing and acknowledged the retained message.

No provider final-reply request was made.

Evidence

Relevant code:

• Producer deletes the durable job in its Queue.send() catch path: apps/api/src/modules/channels/application/channel-final-delivery-jobs.ts:271-277.
• Consumer acknowledges a missing ledger: apps/api/src/modules/channels/application/channel-final-delivery.service.ts:129-137.
• fix(api): retain commands after ambiguous Queue sends #299 fixes the analogous API-command path but its diff only changes api-command files and scheduled redrive, not channel final delivery.

Local fault-injection result:

{
  "ledgerBeforeConsumption": false,
  "consumerActions": [{ "type": "ack" }]
}

Production impact

Cloudflare Queue producer outcomes are not an atomic transaction with D1. A timeout after Queue acceptance is a normal distributed-systems ambiguity, not evidence that the message was rejected.

In production, this can permanently suppress a completed Agent's final reply in Slack, Telegram, Discord, Lark, or WeChat. The originating run may complete successfully while the user sees only the working response or no terminal response. The current consumer acknowledgement removes the last retriable signal, making the loss silent and difficult to reconcile. External webhook retries may also create duplicate upstream handling depending on the provider.

Environment

Mosoo commit: e640185.

Local Bun 1.3.14, SQLite D1 fixture, and injected Queue stub. No remote Cloudflare Queue, provider credential, model-provider call, or billable Cloudflare resource was used. This demonstrates the local D1/Queue consistency boundary; it is not a claim that a remote production incident was observed.

Contribution

I can submit a PR.

Checklist

• I searched existing issues before opening this report.
• I included a reproducible path or explained why I cannot reproduce it locally.
• I removed or redacted secrets from logs, screenshots, and config.

[AsperforMias]

状态更新：当前暂无 channel 相关开发计划，且现有入口处于隐藏、未完成状态，因此暂时搁置该 issue。此前的复现分析和对应修复 PR 会保留，待该功能恢复规划时再继续推进

[AsperforMias]

Current-main revalidation (2026-07-27): this remains reproducible on
b4a5f674bcc05ad5ba5221ea1d82c95dbc20e6b9.

The focused D1/Queue fault injection makes CHANNEL_FINAL_DELIVERY_QUEUE.send()
reject after the channel_final_delivery_job row is inserted. Current code
deletes that only durable row; querying by external_event_id returns no job,
so there is no scheduled redrive source. This is the same failure mode
described in the original report, not a new duplicate.

This issue was closed as deferred while Channels were hidden, not because the
code was fixed. The corresponding draft PR #308 is still open and behind
main; it must be rebased and reviewed before merge. Reopening this issue so
its state matches the current implementation and the still-open fix.

[AsperforMias]

Re-deferred: Channel is currently deliberately unavailable as an end-to-end production feature (no reachable setup UI and no recorded provider live-account smoke test). The queue-send ambiguity is still reproducible in dormant code, but is not planned until Channel is explicitly revived. Draft PR #308 is retained for that future work.