Skip to content

fix(channels): prevent duplicate final replies after post-send D1 failure #417

Description

@AsperforMias

Area

API

Summary

Severity: P1 when Channels are enabled. A channel final reply can be
accepted by a provider while the subsequent markJobDelivered() D1 update
fails. The queue consumer records a retry and requeues the same job. The retry
calls the provider again, producing a duplicate user-visible final reply.

The final-delivery ledger stores only Mosoo's post-send state. It has no
provider receipt/message ID, stable provider idempotency key, or unknown
state for the post-send, pre-ledger-commit interval, so it cannot distinguish a
lost send from an accepted send.

Steps to reproduce

  1. Check out main at b4a5f674bcc05ad5ba5221ea1d82c95dbc20e6b9.
  2. Use the local SQLite D1 fixture, final-delivery Queue stub, and Telegram
    provider fixture to create a completed final-delivery job.
  3. Process the Queue message and let the provider send succeed, recording one
    received final reply.
  4. Inject one failure only for the immediately following
    markJobDelivered() D1 update.
  5. Observe that the job remains dispatched, the original message is ACKed,
    and a 60-second retry is queued.
  6. Remove the D1 failure and process the scheduled retry.

Expected behavior

After an ambiguous post-send persistence failure, Mosoo must not blindly repeat
a non-idempotent provider send. The delivery protocol needs an explicit
unknown/reconciliation state and a provider-specific idempotency key, receipt,
or queryable external message identifier. Providers without a verification
contract require operator-visible resolution rather than automatic duplication.

Actual behavior

The reproduction sent one successful Telegram reply, then failed only the
markJobDelivered() write. Mosoo recorded a retry. The retry sent Telegram a
second final reply, then finally marked the job delivered:

{
  "firstAttempt": { "providerSends": 1, "attemptCount": 1, "status": "dispatched" },
  "retry": { "providerSends": 2, "attemptCount": 2, "status": "delivered" }
}

Evidence

  • sendProviderFinalReply() precedes markJobDelivered() in
    apps/api/src/modules/channels/application/channel-final-delivery.service.ts.

  • The catch path records an attempt and requeues the job after a non-terminal
    error in the same service.

  • channel_final_delivery_job stores a Mosoo lease/claim but no provider
    receipt, external message ID, or unknown post-send state.

  • Focused local reproduction:

    bun test /private/tmp/mosoo-channel-p1-repro.test.ts
    # 2 pass, 0 fail
    

Existing #303 / #308 cover producer-side D1-to-Queue ambiguity. Existing #304
/ #309 cover retry exhaustion. Neither changes the provider-success to
ledger-write-failure boundary.

Environment

  • mosoo commit: b4a5f674bcc05ad5ba5221ea1d82c95dbc20e6b9
  • Local Bun 1.4.0-canary.1 and SQLite D1 fixture
  • Real final-delivery service path with a fake Telegram provider and one
    injected D1 failure after provider acceptance
  • No remote Cloudflare account, provider credential, external provider request,
    or billable resource

Contribution

I can test a fix.

Checklist

  • I searched existing issues before opening this report.
  • I included a reproducible path.
  • I removed or redacted secrets from logs, screenshots, and config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions