Skip to content

Webhook client retries: exponential backoff, DLQ, and idempotent receiver handling #133

@greatest0fallt1me

Description

@greatest0fallt1me

Outbound webhooks: retry with jitter, signature per attempt, and delivery id

    ## Description

    When **StreamPay** calls customer **webhooks** (stream settled, payment tick), implement

durable retry with exponential backoff, jitter, max attempts, and
idempotent delivery ids. Customers must dedupe; we must not lose events in
our DLQ without visibility.
Pair with HMAC from the inbound verification issue for full story.

    ## Requirements and context

    - **At-least-once** delivery; document minimum **2xx** and retry-on-5xx/timeout.
  • Jittered schedule; per-endpoint circuit if they always 500.

  • Table of attempts with next_attempt_at for observability.

  • Tests: flaky receiver in integration tests; ensure eventual success or DLQ.

  • PII in payload minimized; doc retention.

      ## Suggested execution
    
      1. `git checkout -b feature/webhook-delivery-retry`
    
  1. Extract sender from ad-hoc code; add integration test with httpbin or mock server.

  2. PR with table of status codes; security note on not signing different bodies across retries.

  3. SLO for delivery latency in PR description.

  4. Timeframe: 96h; split DLQ admin UI in follow-up.

     ## Test and commit
    
  • Run the full test suite; add or update tests until the agreed coverage bar is met.
  • Cover edge cases listed in this issue; document any intentional exclusions with brief rationale in the PR.
  • Include relevant test output (e.g. test runner summary) or a link to a passing CI run in the pull request.
  • Add security notes for auth, keys, PII, chain settlement, or money movement (assumptions verified, out-of-scope items).

Example commit message

feat(webhooks): outbound retry with jitter, idempotent delivery id, and DLQ on failure

Guidelines

  • Target: at least 95% coverage on new or meaningfully changed code (per the repo’s standard tooling).
  • Documentation: update contributor-facing or API documentation where a reviewer would be blocked without it.
  • Timeframe: 96 hours to ready-for-review (surface blockers early).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-observabilityStreamPay ghit: area-observabilitydomain-integrationsStreamPay ghit: domain-integrationspriority-p2StreamPay ghit: priority-p2type-reliabilityStreamPay ghit: type-reliability

    Type

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions