[Hussman256]

Closes #49

Problem

After the anchor was paid, forwardPayout did three more writes outside the error handling: the sent update, then a transaction with exit_fill and the journal pair. If any of them failed, the API answered 502. The caller treated the payout as failed, a retry got 409, and the ledger never got its usdc_cash / operator:payout_hop rows.

Change

• Stable refId. The journal ref_id (the quoteId when given, otherwise a fresh uuid) and the quote_id are written onto the payout_forward claim before sending. Migration 0010_payout_forward_booking adds ref_id uuid NOT NULL DEFAULT gen_random_uuid() (which backfills existing rows) and quote_id uuid. The original attempt and any repair book under the same id.
• sent means booked. Setting the status to sent, inserting exit_fill and writing the journal pair now happen in one transaction (bookPayout).
• Failure after sending still returns 200.
  • If that transaction throws, the row is set to bookkeeping_pending with the tx hash, a payout_unbooked alert fires (new AlertKind), and the response is 200 { hash, status, booked: false }.
  • If the database is gone and even that flag can't be written, the response is still 200 with the hash, and the alert says the row is still claimed and carries the tx hash.
• Repair. bookPendingPayouts(db, alerter) in src/repair.ts runs at startup and every PAYOUT_HOP_REPAIR_INTERVAL_S (default 300).
  • Each row is taken with UPDATE … SET status='sent' WHERE from_tx_hash=? AND status='bookkeeping_pending' RETURNING inside its own transaction. That locks the row, so repairs running in parallel can't book it twice.
  • If a payout_hop journal row already exists for that ref_id, the original commit landed and only its acknowledgement was lost. The row is just marked sent.
  • Otherwise it writes exit_fill (if there's a quote) and the pair. A row that still fails stays pending and alerts again.
• README: response shape, the "What it writes" section, a new repair section, environment variables, tests and known limits.

Tests (25 passing)

The hop test's fake database was rewritten to keep real rows, honour the eq/and filters, apply the payout_forward primary key, and roll back a transaction that throws. New cases:

• Booking transaction throws after sending: returns { hash, booked: false }, the row is bookkeeping_pending with the hash and ref_id, nothing is half-written, payout_unbooked fires with the reason, and a retry still gets DuplicatePayoutError (the anchor is not paid twice).
• Database gone entirely: still returns the hash, the row stays claimed, and the alert has flagged: false.
• Repair:
  • books under the claim's ref_id, both the quote case and the generated case, and writes exit_fill;
  • is idempotent;
  • only marks sent a row whose pair already exists;
  • leaves a failed repair pending with an alert and books it on the next run;
  • ignores claimed, failed and sent rows.
• I checked that the tests can fail: removing the "already booked" check makes the matching test fail.

pnpm --filter @ballast/service-payout-hop test, typecheck and build pass. The @ballast/observability tests also pass.

Note: this overlaps forwardPayout with #1101 (the first-leg verification for #50). Whichever merges second will need a small rebase.

[drips-wave]

@Hussman256 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits