Regression: swap-CLI soak fails at proposal delivery on master (post-#534/#536); daemon-vs-CLI take divergent paths on token validation

[vrogojin]

Summary

manual-test-swap-roundtrip.sh (Scenario A) fails on master with ASSERT FAIL (proposal-ingest-timeout) — bob does not receive alice's swap proposal within the 90s budget. The same soak script, against the same testnet+escrow, passes green end-to-end when sphere-cli loads a pre-#534/#536 sphere-sdk (commit 8c7d93d).

Surfaced while validating trader-service#29 (PR vrogojin/trader-service#32). Independent of #29; my code changes are confined to trader-service.

A/B test result

Two consecutive runs of bash manual-test-swap-roundtrip.sh SCENARIO=A KEEP=1 differing only in which sphere-sdk checkout sphere-cli/node_modules/@unicitylabs/sphere-sdk symlinks to:

sphere-sdk state	Result
commit 8c7d93d (pre-#534, pre-#536)	PASS — proposal delivered, both sides reached completed, poison-pill scan clean
current master (post-#534/#536) with freshly-built dist	FAIL — proposal-ingest-timeout; bob saw No swaps found for the full 90s window; reproduced twice

The diff between the two checkouts is exclusively #534 (fix(oracle,soak): payments.validate() always reported invalid + stale swap-soak escrow) + #536 (fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout) — see git diff 8c7d93d HEAD -- '*.ts' (12 files, +1074/-127).

This contradicts the assertion in vrogojin/trader-service#29 that the swap-CLI soak validated #536 end-to-end. Either that validation was incomplete (e.g. did not include the cross-process Nostr-DM ingest step of manual-test-swap-roundtrip.sh), or something has regressed since.

Failure mode is divergent between daemon and CLI consumers

Same SDK delta, different observed symptom depending on consumer:

• sphere-cli (CLI, short-lived processes) with post-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536: dies at PROPOSAL DELIVERY. Bob's sphere swap list --role acceptor returns empty for 90s. The swap never reaches verifyPayout because bob never sees the proposal.
• trader-service (daemon, long-lived process) with pre-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536: dies at PAYOUT VERIFICATION. The deal accepts, the swap executes, the escrow pays out, but verifyPayout loops on "L3 validation found N invalid token(s) covering this payout invoice — retry after wallet sync" for the full 900s budget. (Observed when I rebuilt trader image against the pre-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 sibling sphere-sdk; pre-existing failure mode fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 was meant to eliminate.)
• sphere-cli (CLI) with pre-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536: passes green — the verifyPayout retry loop documented in payout-verifier.ts's file header does NOT fire here.

So pre-#534/#536 the CLI's sphere swap wait --state completed reaches completed successfully, while the daemon's auto-verify loop hangs. The two consumers exercise the validation gate differently:

• daemon = long-running, drives verifyPayout directly via its auto-retry loop
• CLI = short-lived, drives swap progression event-by-event through sphere swap wait

The wait command may bypass or short-circuit the validation gate that the daemon's polling loop hits. Worth understanding both paths before locking in either #534 or #536's behavior.

Repro

# Working tree assumed: sphere-cli/node_modules/@unicitylabs/sphere-sdk is a symlink
# to whichever sphere-sdk checkout you want to test.

# Pin to pre-#534/#536:
ln -sfn /path/to/sphere-sdk-at-8c7d93d sphere-cli/node_modules/@unicitylabs/sphere-sdk
( cd /path/to/sphere-sdk-at-8c7d93d && npm run build )
rm -rf /tmp/swap-soak && SWAP_TEST_DIR=/tmp/swap-soak SCENARIO=A KEEP=1 \
  bash manual-test-swap-roundtrip.sh
# → ALL GREEN — swap round-trip succeeded (Scenario A)

# Pin to master (post-#534/#536):
ln -sfn /path/to/sphere-sdk-at-master sphere-cli/node_modules/@unicitylabs/sphere-sdk
( cd /path/to/sphere-sdk-at-master && npm run build )
rm -rf /tmp/swap-soak && SWAP_TEST_DIR=/tmp/swap-soak SCENARIO=A KEEP=1 \
  bash manual-test-swap-roundtrip.sh
# → ASSERT FAIL (proposal-ingest-timeout) after 90s

Workspace preserved at /tmp/swap-soak for log inspection.

Investigation candidates

1. Bisect fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534 vs fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536. Apply fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534 alone (a063d4b7) and re-run; apply fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 alone (8b1955a9) and re-run. Quickly narrows which one introduced the proposal-delivery regression.
2. Where does the CLI's sphere swap wait skip validation that the daemon hits? Compare the verify-trigger sequence in both consumers. The CLI's pre-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 green-completion suggests the wait command isn't running the same payments.validate() invariant the daemon's auto-verify loop is. If true, fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534's validate() fix has different blast radius across consumers.
3. Is the proposal-ingest failure really about the proposer or about the acceptor? Capture alice's transport state (her Nostr publish ack) and bob's transport subscription with a probe to determine whether alice's CLI exits before the publish flushes, or whether bob's subscription is misconfigured under post-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 SDK.

Workaround (until root cause lands)

Pin sphere-cli's @unicitylabs/sphere-sdk symlink to commit 8c7d93d (pre-#534/#536). Trader-service's daemon is not currently green against any sphere-sdk version, so the workaround restores the swap-CLI soak only.

Related

• trader-service#29 / fix(intent-engine)(#29): pre-flight balance check at create_intent vrogojin/trader-service#32 (closed) — the trader pre-flight balance check that surfaced this regression while validating Phase B.
• sphere-sdk#534 — the regression candidate.
• sphere-sdk#536 / SwapModule.verifyPayout: stop calling payments.validate(); use state-transition-sdk primitives directly #535 — the other regression candidate.

[vrogojin]

Bisect plan — narrow #534 vs #536

Two commits in the regression window, both on main 2026-06-14:

SHA	PR	Subject
a063d4b7	#534	fix(oracle,soak): payments.validate() always reported invalid + stale swap-soak escrow
8b1955a9	#536	fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout

#534 merged at 13:31 +0300, #536 at 13:47 +0300 — 16 minutes apart. Linear history (#536 builds on #534), so we can binary-bisect by testing the midpoint.

Test states

State	Tree	Status
A	8c7d93d (#482, pre-#534)	✅ verified GREEN — swap-CLI soak passes Scenario A end-to-end
B	a063d4b7 (#534 only)	⏳ TO TEST
C	8b1955a9 (#534 + #536)	❌ verified FAIL — proposal-ingest-timeout (×2 fresh dist)

If B is GREEN → regression introduced by #536. If B is FAIL → regression introduced by #534.

Repro for State B

Run from a sphere-sdk checkout that sphere-cli's node_modules/@unicitylabs/sphere-sdk symlinks at:

# 1. Pin the SDK to #534-only
cd /path/to/sphere-sdk-checkout
git fetch --all
git checkout a063d4b7

# 2. Rebuild dist (the swap-CLI imports dist/index.cjs)
npm install
npm run build

# 3. Make sure sphere-cli loads this checkout
cd /path/to/sphere-cli
ln -sfn /path/to/sphere-sdk-checkout node_modules/@unicitylabs/sphere-sdk

# 4. Run swap soak, Scenario A only
rm -rf /tmp/swap-soak
SWAP_TEST_DIR=/tmp/swap-soak SCENARIO=A KEEP=1 \
  bash /path/to/sphere-sdk-checkout/manual-test-swap-roundtrip.sh

Expected per state:

• B = GREEN → `ALL GREEN — swap round-trip succeeded (Scenario A)`. Then file the follow-up issue against fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 specifically.
• B = FAIL → `ASSERT FAIL (proposal-ingest-timeout): bob did NOT receive proposal`. File against fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534.

If B is also a tie (both states tested but root cause spans both)

Cherry-pick #536's tree onto 8c7d93d directly so #534's changes are absent but #536's are present:

git checkout 8c7d93d
git checkout -b probe/536-without-534
git cherry-pick 8b1955a9   # will conflict on files #534 also touched
# resolve conflicts taking #536's intent against the pre-#534 baseline
npm run build
# re-run the soak

This isolates #536's behavior in the absence of #534. Conflicts are expected in oracle/UnicityAggregatorProvider.ts and the swap module's deps; resolve by keeping the pre-#534 oracle surface and grafting #536's payout-verifier on top.

Diagnostic to add regardless of which commit is named

The CLI-side failure surfaces at PROPOSAL DELIVERY (a Nostr DM publish/subscribe step), but #534/#536 only touch oracle + swap modules. So the SDK delta must be reaching the transport indirectly — either through:

1. Shutdown timing: alice's CLI exits a tick earlier post-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 (e.g. a removed await payments.validate() no longer holds the loop open while Nostr publishes flush), so the propose-DM doesn't complete its WS send before the process dies. Probe: add a --keep-running debug flag or await new Promise(r => setTimeout(r, 5000)) after swap.propose() and re-run.
2. Subscription filter: bob's CLI subscribes with a since cursor that post-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 SDK initializes differently (e.g. oracle init time changes the wallet-state load order). Probe: compare bob's transport state file (/tmp/swap-soak/.../bob-peer/.sphere-cli/.../transport.json or equivalent) between B and C runs.
3. Identity binding event order: alice's transport pubkey binding event publishes later post-fix(oracle,soak): payments.validate() always reported invalid; stale swap-soak escrow #534/fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536, so bob can't decrypt the NIP-17 DM by the time he polls. Probe: tail Nostr relay logs (docker logs unicity-relay) during both B and C runs and look for the binding event timing relative to the propose DM.

Daemon-side (trader-service) behavior on the same delta requires a separate trader-image rebuild — captured in vrogojin/trader-service#32's Phase B notes. The daemon's auto-verify loop will hit a different code path (it polls verifyPayout directly rather than awaiting swap:completed events), so a #534-only image may behave differently from a #534+#536 image on the daemon side.

[vrogojin]

Bisect result — State B (#534 only) is GREEN

Ran the bisect plan against the same sphere-cli workspace + testnet, two consecutive scenario-A runs:

State	Tree	Workspace	Result
B	a063d4b7 (#534 only)	/tmp/swap-soak-B	✅ ALL GREEN — swap 0918a947, alice −50 UCT +5 ETH, bob +50 UCT −5 ETH, both reached completed, poison-pill clean
C	8b1955a9 (#534 + #536)	/tmp/swap-soak-C	✅ ALL GREEN — swap 3a3553a1, same deltas, both reached completed, poison-pill clean

Per the bisect plan, B-GREEN would point to #536. But C also ran GREEN for me 1/1 — so the failure does not appear to be a deterministic regression in #536. The reporter's two-fail run series is not reproducing under the same setup ~2 h later.

Evidence the SDK delta actually took effect on State C

State C's verifyPayout log shape comes from the new payout-verifier, confirming #536's code path executed:

# State B (#534 only) — wallet-wide validate() path
[Swap] verifyPayout for 0918a9474e71: L3 validation found 1 invalid token(s) covering this payout invoice — retry after wallet sync

# State C (#534 + #536) — scoped payout-verifier path
[Swap] verifyPayout for 3a3553a1d327: transient (payout-token-not-finalized:c75589f9bdf6150f…) — will retry

So #536's verifyPayout is running on State C and resolving cleanly — actually a tick faster than B (1 transient log vs B's 2), consistent with #535's stated goal.

Implications

1. The CLI's sphere swap wait/list path on State C is not unconditionally broken by fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536. Whatever happened in the reporter's two-fail series is intermittent in this code, not deterministic.
2. The bisect protocol's binary verdict (B-GREEN ⇒ blame fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536) is inconclusive without a deterministic State C failure. Until one is captured, "revert fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536" is premature.
3. The actual proposal-ingest path that the reporter's failures hit is swap.proposeSwap() → communications.sendDM() → NIP-17 gift-wrap publish via NostrTransportProvider. The full fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 diff (95 lines in SwapModule.ts, all inside verifyPayout; helper extraction in PaymentsModule; deps wiring in Sphere.ts) does not touch propose, sendDM, transport, identity binding, or shutdown. There is no obvious mechanism in fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536 that would intermittently lose a swap-propose gift-wrap.

Suggested next steps for the original reporter

• Re-run State C 3–5 times in fresh /tmp/swap-soak-* workspaces. If failure rate is ≥40%, this is a real regression; if ≤20%, it's likely intermittent and the bisect signal is noise.
• Capture relay-side state during a failing run: wscat -c wss://nostr-relay.testnet.unicity.network and subscribe to the gift-wrap kind to see whether alice's propose-DM landed at all, vs. landed but bob's CLI didn't process it. This bisects "publish lost" vs. "subscribe lost".
• Probe shutdown timing as the issue's own diagnostic Sphere SDK #1 suggests: insert await new Promise(r => setTimeout(r, 5000)) after swap.propose() in legacy-cli.ts:5337 and re-run. If State C goes green with the delay, alice's process is exiting before the NIP-17 publish flushes — the fix is in Sphere.destroy()/NostrTransportProvider.destroy(), not in fix(swap)(sphere-sdk#535): use state-transition-sdk primitives for verifyPayout #536.

Workspaces preserved

• /tmp/swap-soak-B (State B GREEN)
• /tmp/swap-soak-C (State C GREEN)

Both runs used the same sphere-cli (HEAD c1ff2c1) with the only delta being the symlinked sphere-sdk checkout SHA.