fix(money): every terminal state a money row reaches now has a way out

README.md

@@ -45,7 +45,11 @@ first probe rather than a week of unexplained 404s.
 locked total, so two watcher ticks cannot mint two credits for the same funds.
 The comparison is `balance + swept` against a high-water mark; an unexplained
 drop is recorded as a `regression` that pauses crediting for that address rather
-than silently forgiving it.
+than silently forgiving it. The txid also names the *basis* the total was
+computed under, because a payment identity made of the total alone is unique only
+while the mark rises — and the one-time XRP reserve restatement lowers it, after
+which the address climbs back onto totals it has already recorded and the next
+deposit is refused as a duplicate instead of being credited.
 
 **Conversion runs in both shapes.** `POST /coins/convert` turns a coin balance
 into Shards. `POST /coins/convert-to-ember` turns any of BTC/ETH/SOL/XRP into
@@ -129,10 +133,20 @@ Sequence, and one of them could never land.
 
 **Money only goes back when the chain says it did not move.** A refund happens
 when nothing was signed at all, or when a node reports the payment as
-applied-and-failed. Anything else that does not resolve becomes `stuck` and waits
+applied-and-failed. "Nothing was signed at all" has a deadline on it: a build
+that keeps failing for a reason this service cannot classify — a treasury
+account that does not exist, a keyvault it cannot reach — is retried only until
+`PAY_WITHDRAWAL_STUCK_MINUTES` and then refunded, because a withdrawal left
+queued forever is both the user's money withheld and, payments being serial per
+chain, every other withdrawal on that chain withheld behind it. Anything else
+that does not resolve becomes `stuck` and waits
 for an operator, who can abandon it through `POST /admin/withdrawals/:id/abandon`
 — a route that asks the chain first and refuses to refund a payment the network
-can still see.
+can still see. It refuses more than that: once bytes have been signed, a refund
+needs POSITIVE proof that those exact bytes can never be applied — a nonce taken
+by another transaction on EVM, an unconsumed Sequence past its
+`LastLedgerSequence` on XRP. A missing receipt is a fact about the node, not
+about the signature, and refunding on one pays the user twice.
 
 ## The treasury, and the sweeper
 
@@ -160,6 +174,27 @@ refuses with `no_treasury_pinned`, which sets that one chain aside and no others
 > crediting that address. That route takes the transaction id and **refuses until
 > the movement is as deep as the sweeper's own accounting requires**; recording it
 > at the moment of the POST is the same double credit described below.
+>
+> **Its `:id` is the deposit address row's uuid, and two things now hand it over**:
+> the `regression` alarm itself carries `addressId`, so the line that names the
+> remedy also names its argument, and `GET /admin/deposit-addresses` answers the
+> other direction — from an address off an explorer, a coin, or a user — with
+> `?scan=true` to say which addresses are frozen right now. Before that the only
+> place the id existed for anyone but the address's own owner was a `psql` shell
+> on the production database, which is not a remedy anybody reaches for at 3am.
+> The sweeper's own alarms carry it too, which matters because the commonest way
+> an address freezes is a sweep of *ours* going `stuck` rather than an operator
+> moving anything.
+>
+> **The amount is checked against the transaction, and so is where it went.** The
+> posted figure must equal `value + fee` exactly — the refusal states the number
+> the chain gives, so the second attempt is a paste rather than arithmetic — and
+> the transaction must have left this deposit address and arrived somewhere that
+> is not this platform's custody. A transaction back to the same address moves
+> nothing out of it; one paid to another customer's deposit address is credited
+> to that customer by their own watcher. Either recorded as a sweep advances the
+> swept total against coins that are still ours, and the watcher's next tick
+> hands the difference to a depositor as a deposit nobody made.
 
 Three things about the sweeper are worth knowing before it is ever switched on.
 
@@ -206,6 +241,29 @@ sign a transfer and a `treasury` cannot sign a creation.
 > deletes a treasury row on its own, because that row is the only record of where
 > the platform's money is.
 
+## When money is waiting on a human
+
+Both loops that move funds on chain — the withdrawal worker and the sweeper —
+have a state they cannot get themselves out of. A withdrawal whose signed bytes
+the network will neither confirm nor reject becomes `stuck` and is *never*
+auto-refunded, because refunding it while a valid signature exists is how a user
+gets paid twice; a sweep becomes `stuck` and freezes the deposit address it came
+out of. Both are, in the workers' own comments, a request for a human.
+
+Nothing carried that request to one. `GET /internal/money-loops` is the reading
+that can: `{ state: 'ok' | 'degraded', reasons, counts }`, three aggregate
+queries, no chain traffic, always 200 so a monitor can tell "reporting a
+problem" from "unreachable". It is behind the service token — the shape of the
+withdrawal queue is not a public reading. Alongside it, a five-minute watch
+restates the same verdict into the log at error level for as long as it is true,
+because a single line at the moment a row got stuck is exactly what already
+existed and exactly what nobody sees.
+
+> **This is still only half an alert.** Nothing yet *calls* that route: the
+> Beacon target, a journey that exercises a withdrawal, and any notification path
+> at all live in the `stack` repo, and Lantern has no notify module. Until one of
+> them lands, the log line and the route are for whoever is already looking.
+
 ## Running it
 
 ```bash

docs/money-review.md

@@ -41,7 +41,37 @@ reports one or more `regression` ticks. That is unavoidable for an out-of-band m
 is no sweep row to explain it — and it self-heals on the POST. The gate permits recording from
 exactly the block the probe stops seeing the funds, so a prompt operator sees none of it.
 
-Findings 3, 5, 6 and 8 are untouched and still live as written.
+**Finding 1's last paragraph — "no comparison against the chain" — no longer holds either.**
+The route now asks the node what the transaction actually moved (`outbound.OutboundMovement`)
+and refuses unless three things are true: it was sent BY this deposit address, it moved
+exactly the posted `value + fee`, and it went somewhere that is not this platform's own
+custody. The third is not decoration: verifying only the sender and the size accepts a
+transaction from the deposit address straight back to itself, which moves nothing out of
+custody while advancing `swept` — the same mint as an overstated amount, reached by a pasted
+destination (CF-08). A coin whose transactions this service cannot look up is refused outright
+rather than recorded on trust, which closed the SOL hole the same finding did not see: that
+path skipped the whole verification block and still added the operator's number to `swept`.
+
+Findings 3, 6 and 8 are untouched and still live as written.
+
+**Finding 5 is fixed, and this document was wrong about its scope.** `POST /admin/withdrawals/
+:id/abandon` now refuses to refund a row with `raw_tx` unless the chain proves those exact
+bytes can never be applied: for EVM, no receipt for the hash derived from the bytes AND
+`eth_getTransactionCount(treasury,'latest')` past the nonce read out of the RLP; for XRP, not
+in the ledger, the treasury's `Sequence` still short of the one signed, and the ledger past the
+blob's `LastLedgerSequence`. The decision is a pure function (`outbound.judgeAbandon`) and
+every branch of it is a test.
+
+The correction is the sentence "XRP is protected by `LastLedgerSequence`". It is not. The
+route asked the chain only `if (withdrawal.txid)`, and `buildAndSignXrp` returns `txid: null` —
+so an XRP payment whose `submit` reached rippled and WAS APPLIED, but whose HTTP response was
+lost, was refunded with no chain contact of any kind. `LastLedgerSequence` bounds when a blob
+can still be applied; it says nothing about one that already was, and the double pay in that
+case is permanent rather than bounded by ~70 minutes. The remedy needed two things this
+service did not have: a transaction id derivable from an XRP blob (`outbound.xrpTxHash`, which
+is rippled's own SHA-512Half over the `TXN\0` prefix) and the two numbers the blob commits to,
+which are now written down at signing time (`withdrawals.signed_sequence` / `signed_expiry`)
+because there is no XRPL deserializer here to read them back. See CF-06.
 
 ---
 
@@ -276,9 +306,10 @@ The question posed was whether the new `settling` tolerance can absorb a genuine
 `observeDeposit`'s test is `inFlight >= shortfall` (`store.ts:847`) and `inFlight` counts
 sweeps in `signed | broadcast | confirmed` with `recorded_at IS NULL` (`store.ts:862-884`).
 In the normal case the tolerance is bounded — a sweep that never matures is marked `stuck`
-after `PAY_WITHDRAWAL_STUCK_MINUTES` (`sweeper.ts:287-295`), `stuck` leaves the set, and the
-address correctly flips to `regression`. So a theft alongside an in-flight sweep is hidden for
-at most one hour, not permanently. That is sound.
+after `chains.sweepStuckMinutes` (61 minutes for EMBER; it was `PAY_WITHDRAWAL_STUCK_MINUTES`
+when this was written, see finding 8), `stuck` leaves the set, and the address correctly flips
+to `regression`. So a theft alongside an in-flight sweep is hidden for about an hour, not
+permanently. That is sound.
 
 **The unbounded case is when nothing advances the sweep at all.** Three ways to get there:
 
@@ -306,16 +337,20 @@ settling any more.
 
 ### 8. Lower severity, recorded rather than argued
 
-- **A sweep that goes `stuck` below maturity freezes its address.** `advanceSweep`
-  (`sweeper.ts:287-295`) marks a sweep `stuck` if maturity is not reached within
-  `PAY_WITHDRAWAL_STUCK_MINUTES` (60). `stuck` can never mature (`matureSweep` guards on
-  `broadcast|confirmed`, `store.ts:1560`) and stops explaining, so the funds have left, `swept`
-  was never recorded, and the address latches `regression`. The code names the trigger
-  ("reachable for EMBER on a stalled chain") but not the consequence. Recoverable via the admin
-  route — and by then the movement is >60 min deep, so finding 1 does not bite. EMBER maturity
-  is 61 blocks ≈ 15 min against a 60-minute deadline; the margin disappears if block time
-  triples, which `findings.md` P1-4 says is a live concern for Hearth. The sweep deadline should
-  be derived from the coin's maturity, not borrowed from the withdrawal one (`sweeper.ts:89`).
+- **A sweep that goes `stuck` below maturity freezes its address.** FIXED, and this entry was
+  wrong about the one thing that mattered. `advanceSweep` marks a sweep `stuck` if maturity is
+  not reached inside the deadline. `stuck` could never mature (`matureSweep` guarded on
+  `broadcast|confirmed`) and stops explaining, so the funds have left, `swept` was never
+  recorded, and the address latches `regression`. "Recoverable via the admin route" was not
+  true: that route inserts a row with the same `(address_id, txid)` as the stuck sweep, hits
+  `sweeps_address_txid_idx`, records nothing, and answers 200 `replayed: true` — the documented
+  remedy was a silent no-op against exactly this state, and only a hand-written `UPDATE sweeps`
+  cleared it (CF-07). Now: `matureSweep` accepts `stuck`, the sweeper re-checks stuck sweeps the
+  chain gave a `mined_height` for and records them when the depth arrives, and the admin route
+  reconciles the conflicting row in place. The deadline is also derived from the coin's maturity
+  rather than borrowed from the withdrawal worker — `chains.sweepStuckMinutes` — which is what
+  this entry asked for and what stops the margin disappearing if Hearth's block time triples
+  (`findings.md` P1-4).
 - **The reorg controls that `@cloudsforge/shared` says live here do not exist here.**
   `shared/src/deposits.ts` (EMBER entry) states plainly that depth "is explicitly NOT the real
   control … the controls that matter are economic — per-user caps, a halt on any reorg past ~5

services/pay/src/chains.ts

@@ -125,6 +125,70 @@ export function sweepIsMature(coin: DepositCoin, confirmations: number): boolean
   return confirmations >= sweepMaturityConfirmations(coin)
 }
 
+/**
+ * Nominal seconds per unit of the thing each family counts confirmations in — a block for
+ * the EVM chains and Bitcoin, a validated ledger for XRP, and for Solana the time its
+ * `finalized` commitment takes to catch up with a slot.
+ *
+ * Nothing here measures a chain and nothing depends on these being exact. They exist to
+ * answer one question — how long ought WAITING for a given depth to take? — and the answer
+ * is multiplied by a wide slack before anything is decided on it.
+ */
+const BLOCK_SECONDS: Record<DepositFamily, number> = {
+  // Hearth mines every 15 seconds (hearth/docs/evm-spec.md), which is the cadence EMBER's
+  // 60-block deposit depth was argued against in the first place.
+  hearth: 15,
+  // Ethereum's slot time. A missed slot is inside the slack below.
+  evm: 12,
+  bitcoin: 600,
+  // `finalized` is roughly 32 slots behind the tip at ~400 ms a slot.
+  solana: 13,
+  xrp: 4,
+}
+
+/**
+ * How much slower than nominal a chain may run before a sweep that has not reached its depth
+ * is somebody's problem rather than the weather.
+ *
+ * Generous on purpose, and the asymmetry is the argument: declaring a sweep stuck too early
+ * costs a deposit address that stops crediting anybody (a `stuck` sweep leaves
+ * `store.UNMATURED_SWEEP_STATUSES` and stops explaining the address's shortfall), while
+ * declaring one too late costs nothing but a later alarm on a chain that has already stopped.
+ */
+export const SWEEP_STUCK_SLACK = 4
+
+/**
+ * No chain's deadline is shorter than this, however fast its blocks are.
+ *
+ * XRP validates a ledger every four seconds and needs exactly one of them, so its derived
+ * deadline is sixteen seconds — which would mark a sweep stuck for one slow answer from
+ * rippled, or for one restart of this process. The floor is what keeps the deadline a
+ * statement about the chain rather than about a single RPC call. Note also that the deadline
+ * is only ever evaluated inside a sweeper tick, so a deployment that polls more slowly than
+ * the default five minutes stretches every deadline with it automatically and needs nothing
+ * configured here.
+ */
+export const SWEEP_STUCK_FLOOR_MINUTES = 15
+
+/**
+ * How long a sweep may sit below the depth its accounting needs before it is `stuck`.
+ *
+ * DERIVED FROM THE DEPTH IT IS WAITING FOR. It used to be `PAY_WITHDRAWAL_STUCK_MINUTES`, a
+ * flat hour borrowed from a different worker waiting for a different thing. A withdrawal is
+ * late when a destination has not been paid, which is a question about one transaction being
+ * mined; a sweep is late when the HEAD has not moved `sweepMaturityConfirmations` past it,
+ * which is a question about the rate of the whole chain. Nothing ties the two together, and
+ * the borrowed number is wrong in both directions: EMBER's 61 blocks are fifteen minutes of a
+ * healthy chain, so an hour is four times the honest deadline; and a deposit depth this
+ * estate raises past ~59 blocks (hearth/docs/exchange-integration.md §4 already argues for
+ * depth) would put nominal maturity BEYOND the hour, marking every sweep on a perfectly
+ * healthy chain stuck — which freezes the deposit address of every depositor swept.
+ */
+export function sweepStuckMinutes(coin: DepositCoin): number {
+  const nominalSeconds = sweepMaturityConfirmations(coin) * BLOCK_SECONDS[depositChainInfo(coin).family]
+  return Math.max(SWEEP_STUCK_FLOOR_MINUTES, Math.ceil((nominalSeconds * SWEEP_STUCK_SLACK) / 60))
+}
+
 /** A single address probe at both confirmation depths. */
 export interface FundedProbe {
   /** Funded total at the required confirmation depth (smallest units). Creditable. */

services/pay/src/db/migrate.ts

@@ -89,6 +89,20 @@ export async function migrate() {
   // every already-credited XRP address would report `regression` forever from the moment that
   // change shipped. See store.MARK_BASIS for the fence around the restatement.
   await client`ALTER TABLE deposit_addresses ADD COLUMN IF NOT EXISTS mark_basis TEXT`
+  // Look one up BY ADDRESS, which three money decisions do and none of them had an index for.
+  // `store.isPlatformAddress` refuses a withdrawal that would pay this platform's own custody
+  // and runs on every withdrawal request; `store.custodialDepositAddress` refuses a manual
+  // sweep into another customer's deposit address (CF-08); `store.depositAddressOverview`
+  // answers an operator holding an address off an explorer (CF-22). Two indexes because there
+  // are two comparisons and one cannot serve the other: EVM addresses are matched with the
+  // case folded (a node and an explorer both render them lowercase, this table stores the
+  // EIP-55 form), and base58/bech32 exactly, where folding would match an address nobody
+  // asked about.
+  await client`CREATE INDEX IF NOT EXISTS deposit_addresses_address_idx ON deposit_addresses (address)`
+  await client`
+    CREATE INDEX IF NOT EXISTS deposit_addresses_address_lower_idx
+      ON deposit_addresses (lower(address))
+  `
   await client`
     CREATE TABLE IF NOT EXISTS deposit_payments (
       id            TEXT PRIMARY KEY,
@@ -185,6 +199,8 @@ export async function migrate() {
       status         TEXT NOT NULL DEFAULT 'pending',
       txid           TEXT,
       raw_tx         TEXT,
+      signed_sequence TEXT,
+      signed_expiry   TEXT,
       confirmations  INTEGER NOT NULL DEFAULT 0,
       failure_reason TEXT,
       created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
@@ -193,6 +209,12 @@ export async function migrate() {
       confirmed_at   TIMESTAMPTZ
     )
   `
+  // Added after the table: what an XRP blob committed to, which is the only evidence that can
+  // ever show an abandoned withdrawal's signature is dead (CF-06, schema.ts). Nullable and
+  // left null on every existing row on purpose — a row signed before this existed carries a
+  // blob nothing here can read, and it is refused rather than refunded on an assumption.
+  await client`ALTER TABLE withdrawals ADD COLUMN IF NOT EXISTS signed_sequence TEXT`
+  await client`ALTER TABLE withdrawals ADD COLUMN IF NOT EXISTS signed_expiry TEXT`
   await client`CREATE INDEX IF NOT EXISTS withdrawals_user_idx ON withdrawals (user_id, created_at DESC)`
   // The outbound worker's whole working set is "everything not finished yet", and on a table
   // that only grows that is a shrinking fraction of it. Partial, so the index stays small.

services/pay/src/db/schema.ts

@@ -170,6 +170,22 @@ export const withdrawals = pgTable('withdrawals', {
   status: text('status').notNull().default('pending'),
   txid: text('txid'),
   rawTx: text('raw_tx'),
+  /**
+   * The account sequence `raw_tx` consumes, and the chain height past which it can never be
+   * applied — both as decimal strings, both written in the same statement as the bytes.
+   *
+   * XRP ONLY, and null everywhere else. They exist because abandoning a withdrawal refunds a
+   * user while a valid signature for the same money is still in `raw_tx`, so it may only
+   * happen against POSITIVE proof that those bytes can never apply (CF-06). On an EVM chain
+   * that proof is derivable from the bytes — `outbound.legacyNonce` reads the nonce back out
+   * of the RLP — but XRPL binary serialization is a format this service does not speak, so
+   * the two numbers `buildAndSignXrp` signs are written down here instead. A row that lacks
+   * them cannot be abandoned at all, which is the fail-closed answer and not an oversight:
+   * every withdrawal signed before this column existed is one whose blob nothing here can
+   * read.
+   */
+  signedSequence: text('signed_sequence'),
+  signedExpiry: text('signed_expiry'),
   confirmations: integer('confirmations').notNull().default(0),
   /** Why it failed, or why it is stuck. Shown to the user, so it is written for one. */
   failureReason: text('failure_reason'),