Area
API
Summary
Channel final delivery has terminal failure states with no recovery path. A transient provider outage can consume the application-level delivery budget, after which the job is marked failed and the Queue message is acknowledged. The scheduled Worker does not scan failed channel_final_delivery_job rows or requeue them.
There is a second terminal path for unexpected consumer failures: Wrangler routes channel-final-delivery messages to channel-final-delivery-dlq after five broker retries, but no consumer or replay workflow is configured for that DLQ.
The result is durable diagnostic data without a durable recovery protocol: completed Agent work can permanently lose its user-facing final reply after a temporary channel or Worker incident.
Steps to reproduce
- Use the local SQLite D1 fixture, channel final-delivery Queue stub, and Telegram provider fixture.
- Create a completed Telegram final-delivery job.
- Set its attemptCount to 7 to model seven earlier transient provider failures.
- Make the next provider send fail with "provider unavailable".
- Process the queued message through processChannelFinalDeliveryMessage().
- Invoke the real createApiWorker().scheduled() handler with the same bindings.
- Read the final-delivery job and captured Queue messages.
Expected behavior
A temporary provider or worker/Queue incident must leave a recoverable delivery record. The platform should have an explicit policy to redrive failed final-delivery jobs, consume/replay the channel DLQ, or expose an operator-safe retry mechanism that preserves the idempotency claim contract.
The scheduled recovery path should eventually return transiently failed final replies to delivery after the provider recovers.
Actual behavior
The local injected reproduction produced:
{
"job": { "attemptCount": 8, "status": "failed" },
"queueActions": [{ "type": "ack" }],
"channelQueueMessages": 1
}
The only message was the original delivery message. Calling the actual scheduled Worker handler did not enqueue a replacement channel message and did not change the job state.
The implementation marks the job failed at the eighth application-level delivery attempt, then acknowledges it. The Worker scheduled handler only redrives API-command ledger rows; it does not inspect final-delivery jobs. Separately, wrangler configures channel-final-delivery-dlq as a dead-letter queue but has no consumer/replay binding for it.
Evidence
Relevant code:
- Terminal application-level failure and acknowledgement: apps/api/src/modules/channels/application/channel-final-delivery.service.ts:307-331.
- Scheduled Worker only invokes redriveFailedApiCommandEnqueues() and scheduled maintenance: apps/api/src/platform/cloudflare/create-api-worker.ts:33-37.
- channel-final-delivery has a dead_letter_queue: apps/api/wrangler.toml:216-221.
- There is no consumer configuration for channel-final-delivery-dlq in the repository.
Related history: #206 explicitly listed a channel-final-delivery-dlq consumer as a known deferral; it did not implement one.
Production impact
A provider outage, temporary credential-decryption fault, Worker exception, D1 transient failure, or Queue delivery failure can turn a completed Agent Run into a permanently missing final reply. Users may see a working message with no completion, while the Agent run and associated model cost are already complete.
Because no automatic redrive, DLQ consumer, or exposed retry path exists, restoration requires direct database/Queue intervention. This creates an operational consistency gap: platform state records a failed final delivery, but the user-facing channel never converges to a terminal response. At volume, a short incident can leave many completed conversations permanently unresolved.
Environment
Mosoo commit: e640185.
Local Bun 1.3.14, SQLite D1 fixture, injected Telegram provider failure, Queue stub, and the real local Worker scheduled handler. No remote Cloudflare Queue, channel provider credential, model-provider call, or billable Cloudflare resource was used. This validates the local failure/recovery boundary; it is not a claim that a remote production incident was observed.
Contribution
I can submit a PR.
Checklist
Area
API
Summary
Channel final delivery has terminal failure states with no recovery path. A transient provider outage can consume the application-level delivery budget, after which the job is marked failed and the Queue message is acknowledged. The scheduled Worker does not scan failed channel_final_delivery_job rows or requeue them.
There is a second terminal path for unexpected consumer failures: Wrangler routes channel-final-delivery messages to channel-final-delivery-dlq after five broker retries, but no consumer or replay workflow is configured for that DLQ.
The result is durable diagnostic data without a durable recovery protocol: completed Agent work can permanently lose its user-facing final reply after a temporary channel or Worker incident.
Steps to reproduce
Expected behavior
A temporary provider or worker/Queue incident must leave a recoverable delivery record. The platform should have an explicit policy to redrive failed final-delivery jobs, consume/replay the channel DLQ, or expose an operator-safe retry mechanism that preserves the idempotency claim contract.
The scheduled recovery path should eventually return transiently failed final replies to delivery after the provider recovers.
Actual behavior
The local injected reproduction produced:
The only message was the original delivery message. Calling the actual scheduled Worker handler did not enqueue a replacement channel message and did not change the job state.
The implementation marks the job failed at the eighth application-level delivery attempt, then acknowledges it. The Worker scheduled handler only redrives API-command ledger rows; it does not inspect final-delivery jobs. Separately, wrangler configures channel-final-delivery-dlq as a dead-letter queue but has no consumer/replay binding for it.
Evidence
Relevant code:
Related history: #206 explicitly listed a channel-final-delivery-dlq consumer as a known deferral; it did not implement one.
Production impact
A provider outage, temporary credential-decryption fault, Worker exception, D1 transient failure, or Queue delivery failure can turn a completed Agent Run into a permanently missing final reply. Users may see a working message with no completion, while the Agent run and associated model cost are already complete.
Because no automatic redrive, DLQ consumer, or exposed retry path exists, restoration requires direct database/Queue intervention. This creates an operational consistency gap: platform state records a failed final delivery, but the user-facing channel never converges to a terminal response. At volume, a short incident can leave many completed conversations permanently unresolved.
Environment
Mosoo commit: e640185.
Local Bun 1.3.14, SQLite D1 fixture, injected Telegram provider failure, Queue stub, and the real local Worker scheduled handler. No remote Cloudflare Queue, channel provider credential, model-provider call, or billable Cloudflare resource was used. This validates the local failure/recovery boundary; it is not a claim that a remote production incident was observed.
Contribution
I can submit a PR.
Checklist