Conversation
Closes the architectural loop: this is the canonical proof that operators can launch HMA, spawn agents through it over Sphere DM, trade between independent traders, settle on testnet, and withdraw — the user-stated goal that no prior live test fully covered. Prior tests cover slices: - basic-roundtrip: settles correctly, but uses direct docker run - hma-orchestrated: spawns through HMA, but doesn't trade - hma-trade-flow: spawns + posts/cancels intents through HMA, but explicitly stops short of settlement - this file: spawns + funds + trades + settles + withdraws — all through HMA over Sphere DMs Two scenarios run concurrently via vitest it.concurrent on a single shared HMA (matches production: one HMA per host, multi-tenant). Within each scenario, every fan-outable step uses Promise.all with *Async helpers so concurrency actually overlaps: - hostSpawnAsync × 3 (escrow + 2 traders) — parallel - setStrategyAsync × 2 — parallel - portfolioAsync × 2 (snapshots) — parallel - createIntentAsync × 2 (matching) — parallel - waitForDealInStateAsync × 2 (COMPLETED) — parallel - withdrawAsync × 1 (one trader withdraws to controller) The sync helpers (runSphere → spawnSync) block the event loop so Promise.all over them serializes — added async variants only for the helpers needed here, kept the sync surface small. Self-funding via TRADER_TEST_FUND env passthrough through HMA (validatePayloadEnv at agentic-hosting/src/host-manager/manager.ts:97 allows TRADER_TEST_FUND — it's not in FORBIDDEN_ENV_KEYS and doesn't start with UNICITY_). Avoids the recurring testnet faucet flakiness that basic-roundtrip's commit history documents. Performance target on healthy testnet: 5-8 minutes total (HMA boot ~10s; 6 concurrent spawns ~30-60s; settlement wait ~3-5 min dominates; 2 concurrent withdraws ~5-10s). Files: - test/e2e-live/helpers/sphere-trader.ts: add withdraw + 5 *Async helpers (setStrategyAsync, createIntentAsync, portfolioAsync, listDealsAsync, waitForDealInStateAsync, withdrawAsync) - test/e2e-live/helpers/hma-spawn.ts: add hostSpawnAsync - test/e2e-live/hma-trade-settlement.e2e-live.test.ts: NEW Verified: 698 unit/integration tests still pass. Type check + lint clean.
Round 1 of running hma-trade-settlement.e2e-live failed within seconds with two distinct wallet.json races: Pair-1: "No wallet found in /tmp/.../.sphere-cli" Pair-2: "ENOENT: rename '.sphere-cli/wallet.json.tmp' -> 'wallet.json'" Both scenarios were sharing one cliHome and issuing concurrent sphere-cli invocations. Each invocation persists incoming-DM state to .sphere-cli/wallet.json via the SDK's atomic temp+rename (FileStorageProvider.save → fs.renameSync). When two parallel calls write at once, one's tmp gets renamed away before the other's rename observes it, corrupting both. Fix: bootstrap one controller wallet per scenario in its own cliHome dir. HMA's AUTHORIZED_CONTROLLERS env var accepts a comma-separated list of pubkeys (config.ts:52), so a single HMA authorizes all scenarios' controllers in one go — matching the real multi-tenant production topology. Each scenario now uses its own cliHome (no wallet.json contention) and its own DIRECT:// address as the withdraw destination. Aborted the earlier "shared cliHome with retry" approach because retries would only mask the race; the SDK's wallet store isn't designed for concurrent writers. SCENARIO_COUNT constant exposed so future steelman rounds can bump it to 4-8 to stress-test the relay/aggregator/HMA further.
…o parallel
Round 2: per-scenario controller wallets fixed cross-scenario
contention but the SECOND failure mode (within-scenario races
between escrow + alice + bob hostSpawnAsync calls in Promise.all)
remained. Each call writes DM dedup state to .sphere-cli/wallet.json
via the SDK's atomic temp+rename — three concurrent writers in the
same dir corrupt each other.
Tried parallel spawn within a scenario; got two distinct races:
Pair-1: "No wallet exists and no mnemonic provided" (Sphere.init
reading wallet.json mid-write — the file was renamed away
underneath it by another spawn's atomic update)
Pair-2: "ENOENT: rename wallet.json.tmp -> wallet.json" (one spawn
renamed away the temp file the other was about to rename)
Fix: serialize sphere-cli calls within a scenario (drop Promise.all
over spawn/set-strategy/portfolio/create-intent/wait/portfolio-after).
Cross-scenario parallelism via vitest it.concurrent is preserved —
that's where the wall-time savings actually come from, because the
3-5 min settlement-wait dominates each scenario and runs in true
parallel across scenarios.
Wall-time impact: serializing the 3 spawns within a scenario adds
2 × ~5-15s of DM round-trip vs parallel. Negligible against the
settlement dominator.
A future PR can revisit within-scenario parallelism by giving each
parallel sphere-cli invocation its own cliHome with a copied
wallet.json (same identity, separate file storage). Out of scope
for the goal-completion test.
The file header documents the parallelism contract honestly so
future contributors don't try to "optimize" it back to broken
parallelism.
…f-mint) Round 3 surfaced "TRADER_TEST_FUND requires sphere-sdk with mintFungibleToken (only on refactor/extract-cli-to-sphere-cli branch). Use the faucet path instead, or rebuild with the SDK feature branch." The published trader image (ghcr.io/vrogojin/agentic-hosting/trader:v0.1) that HMA's templates.json points to was built against an older sphere-sdk that doesn't have mintFungibleToken. The TRADER_TEST_FUND env var is recognized but errors at trader startup before acp.hello is sent — HMA observes "did not send hello within 60000ms" and reports hm.spawn_failed. basic-roundtrip works around this by building its trader image locally (with the latest SDK) and running docker directly. HMA tests use the published image, so funding has to happen via the external faucet. Switching to faucet-fund-and-wait: - Drop TRADER_TEST_FUND / TRADER_FAULT_INJECTION_ALLOWED env vars - After alice/bob spawn, hit FAUCET_URL with each tenant's nametag - Poll portfolio every 5s until balance arrives (typically ~30-60s on testnet — payments.receive() runs on a 15s cycle inside the trader plus aggregator confirmation) - Use trade-volume × 2 as funding amount (headroom for the post- trade withdraw step) Trader balances are asymmetric now (alice gets USDU only, bob gets UCT only), matching their trade roles. Post-trade delta assertions unchanged: alice +UCT / -USDU, bob mirror. Note: a future PR rebuilding the trader image at ghcr.io with the latest sphere-sdk would let us swap back to selfMint and avoid the faucet flake risk this re-introduces.
Round 4 surfaced "Coin not found: USDU" from the testnet faucet.
Probing the faucet directly:
POST /api/v1/faucet/request {coin: "UCT"} → Coin not found: UCT
POST /api/v1/faucet/request {coin: "unicity"} → Nametag not found
So the faucet looks up coins by the `name` field of
/api/v1/faucet/coins, not by `symbol`:
coins[].symbol "UCT" → coins[].name "unicity" → faucet OK
coins[].symbol "USDU" → coins[].name "unicity-usd" → faucet OK
Adding a small FAUCET_COIN_NAME map and converting before the
fundWallet call. The map is colocated in the test (5 lines, two
entries) rather than pushed into helpers/funding.ts because the
inverse mapping isn't well-known here — the symbol set the test
uses (UCT, USDU) is the same set the test asserts in portfolio
deltas, so having the mapping near both keeps the surface obvious
to a future contributor adding a third asset.
…receive)
Round 5 surfaced the deeper issue: the published trader image at
ghcr.io/.../trader:v0.1 not only lacks mintFungibleToken (Round 3)
but also fails to surface faucet-deposited balance in portfolio.
Trader logs from a Round-5 container show ZERO payment-receive
events from spawn through 2 minutes of running — no
"payment_received" / "deposit_received" / inventory updates. The
faucet returns 200 OK with a tx_id, but the trader's portfolio
stays empty.
Test now overrides the trader-agent template's `image` field to
point to `ghcr.io/.../trader:local` (built via the Dockerfile in
this repo with current sphere-sdk + trader code). The shared
agentic-hosting/config/templates.json is not modified — the
override is test-local via spawnHostManager's templatesPath param.
Build the local image before running this test:
cd /home/vrogojin && \
docker build -f trader-service/Dockerfile \
-t ghcr.io/vrogojin/agentic-hosting/trader:local .
Operators will need to rebuild + re-publish the trader image at
ghcr.io to use HMA in production trading. Tracked in file header.
Round 6 surfaced that the testnet faucet is silently broken: it returns 200 OK + tx_id but the deposit never surfaces in the trader's portfolio (verified by polling for 120s and checking trader-side container logs — zero payment-receive events). Same flakiness basic-roundtrip's commit history documents. The locally-rebuilt trader image (trader:local) embeds the current sphere-sdk which DOES include mintFungibleToken (verified at sphere-sdk/dist/index.js:12237). So TRADER_TEST_FUND self-mint at trader startup works on the local image — the constraint that forced us to faucet (round 3) was specific to v0.1's older SDK. This commit: - Removes faucet helper (faucetFundAndWait + FAUCET_COIN_NAME mapping + fundWallet import) - Re-introduces TRADER_TEST_FUND env (5000 each of UCT + USDU per trader) passed through HMA's validatePayloadEnv passthrough to the trader container at spawn - Adds a 5s post-spawn delay so the trader's first portfolio query reflects the post-mint balance - Documents the trader image build requirement in the file header
Round 7 progressed past spawn + set-strategy and trader logs
confirmed both UCT and USDU mints succeeded ("test_fund_mint_succeeded"
×2 per trader). But the balance sanity check failed with
"expected 0 to be greater than 0" because balanceOf's first loop
returned `BigInt(String(b.amount ?? '0'))` → 0n WHEN `b.amount` is
undefined — the short-circuit on `?? '0'` prevented the fallback
loop from ever reading `b.confirmed`.
The trader's GET_PORTFOLIO emits each balance as
{ asset, available, total, confirmed, unconfirmed }
with no `amount` field. So balanceOf was returning 0n for every
balance despite the mint succeeding.
Fix: single-pass loop, prefer `confirmed`, fall back to `amount`
then `available`. Returns 0n only if NONE are present.
Round 8 progressed all the way to settlement-wait but timed out with both scenarios stuck — pair-1 saw 29 deals (1 FAILED + 27 CANCELLED + 1 PROPOSED), pair-2 saw 3 (FAILED, CANCELLED, ACCEPTED). The thrash loop happened because both scenarios posted UCT/USDU intents at rate=1, so the matcher saw cross-scenario pairs as valid matches. When pair-1's alice tried to negotiate with pair-2's bob, the trusted_escrows mismatch forced the deal to FAILED, but the matching engine just kept retrying. Fix: pass per-scenario `tradeRate` into runSettlementScenario. Pair-1 uses rate=1, pair-2 uses rate=3 (non-adjacent so no rate-fuzzing ever overlaps). The matcher's rate-overlap check prevents cross-scenario matches at the search-result level, before negotiation.
…UND)
The trade-settlement test was rounds 8-9 stuck in a FAILED/CANCELLED
loop despite spawn + balance assertions passing. Hypothesis: tokens
self-issued by the trader (TRADER_TEST_FUND mint) confuse the swap
protocol because issuer==sender is unusual in production. With
FAUCET-funded tokens the issuer is a separate agent — closer to
real-world settlement.
Also replaces the broken public-faucet HTTP path (returns 200 OK +
tx_id but deposit never surfaces, verified across rounds 5-6 — and
documented again in basic-roundtrip's commit history).
Now uses the js-faucet agent that spawns alongside escrow + traders:
beforeAll:
- bootstrap controller wallets (per-scenario, sphere-cli)
- inject `faucet-agent` template (image: faucet:local)
- inject `trader-agent` template (image: trader:local)
- boot HMA with all controllers authorized
- spawn ONE shared faucet (open auth — no need per scenario)
- bootstrap in-process FaucetClient (Sphere wallet that signs
and encrypts FAUCET_REQUEST DMs to the faucet's pubkey)
per scenario:
- spawn escrow + alice + bob (no TRADER_TEST_FUND env)
- FAUCET_REQUEST batch (UCT + USDU, 5000 each) → alice
- FAUCET_REQUEST batch (UCT + USDU, 5000 each) → bob
- poll each trader's portfolio until confirmed >= 5000 of each
- set-strategy / post intents / wait COMPLETED / withdraw
New helper: test/e2e-live/helpers/faucet-client.ts (~190 lines).
- bootstrap an in-process Sphere wallet
- subscribe to inbox; capture acp.result/acp.error by command_id
- sendDM(faucetPubkey, FAUCET_REQUEST envelope), wait for ack
- return deliveries (asset, coin_id, amount, token_id, transfer_id)
Pre-flight (operator must do before running):
cd /home/vrogojin && docker build \
-f js-faucet/Dockerfile \
-t ghcr.io/unicitynetwork/agentic-hosting/faucet:local .
Verified: 698 unit tests still pass; type-check clean for both
src/ and test/ contexts.
Whether this also fixes the FAILED/CANCELLED settlement loop is the
question the next live round answers.
Round 10 with faucet funding killed the spam-loop (29 deals → 2)
but settlement still failed: traders sent `status` query to escrow,
got "Swap not found" error, deal CANCELLED. Escrow log showed ZERO
swap.announce_received events despite ping/pong working — i.e.
the swap.announce DM never reached the escrow.
Root cause: trader-service/src/trader/intent-engine.ts:836:
const escrowAddress = params.escrow_address ?? DEFAULT_ESCROW;
// DEFAULT_ESCROW = 'any' (intent-engine.ts:92)
When the test omits escrow_address from create-intent, the
intent (and resulting deal terms) carries the literal string
'any'. The swap-executor then tries to route swap.announce to
'any', which the SDK's transport layer can't resolve to a real
peer. The escrow never gets the announce; subsequent status
queries from the trader hit a swap that doesn't exist on the
escrow's side.
Fix lands in two repos:
(1) sphere-cli — add `--escrow-address` flag to
`sphere trader create-intent`. The flag was missing entirely
so callers had no way to override 'any'. Updated:
- src/trader/trader-commands.ts: CreateIntentOpts +
buildCreateIntentParams + commander definition
- rebuilt dist/
(2) trader-service test:
- helpers/sphere-trader.ts: CreateIntentOpts + forward to
`--escrow-address` argv
- hma-trade-settlement test: pass `escrowAddress:
escrow.tenantPubkey` for both alice and bob's intents
Verified: 698 unit tests still pass; type-check clean. Whether
this finally fixes the FAILED/CANCELLED settlement loop is the
next live round's question.
Round 11 progressed: pair-2 reached ACCEPTED for the first time
(3 deals: FAILED, CANCELLED, ACCEPTED) — the escrow_address fix
landed correctly. But the deal still failed at swap registration
with a clearer error in the trader log:
"swap_id_register_escrow_mismatch":
negotiated_escrow: 02219b272c88584a4129d770edf5c08bcb5054d3c08496157c021da77cab8625a1
proposal_escrow: DIRECT://00002b89215d8b32323f083ceb154329115d3137c0c5325410d2cc0b0d5a111eb0ad3148b2ce
proposal_escrow_pubkey: 219b272c88584a4129d770edf5c08bcb5054d3c08496157c021da77cab8625a1
The trader's swap-executor (line 714) checks:
negotiatedEscrow === match.escrowDirectAddress
`negotiatedEscrow` = `terms.escrow_address` (what we passed to
create-intent — chain pubkey form, 02-prefixed 66 chars)
`match.escrowDirectAddress` = the DIRECT://hex address (a structural
hash via UnmaskedPredicateReference, NOT the pubkey).
These are fundamentally different forms. The check expects them
EQUAL, so the intent's escrow_address must be set to the
DIRECT://-form to match what the SDK derives during proposal.
Also updates `setStrategy(trustedEscrows: [...])` to use the same
DIRECT:// form — the negotiation-handler's trusted check at
negotiation-handler.ts:1084 does direct string equality
(`trustedEscrows.includes(terms.escrow_address)`), so both sides
must match. This was implicitly working before only because both
sides were 'any'.
Same fix could land in basic-roundtrip's createMatchingIntents
helper (probably what makes the direct-docker version work — they
must already pass directAddress somewhere) but that's out of scope
for this commit.
Round-by-round summary of the hma-trade-settlement debugging
sessions, the bugs we found and fixed, the remaining symptom
(trader doesn't process escrow's invoice_delivery DM despite the
escrow logs reporting successful send), and three concrete
hypotheses with bisect plans for the next session:
H1: ACP listener consumes the DM before swap module sees it
H2: payments.receive() loop races with swap module's DM dispatch
H3: HMA-spawned container has higher relay-subscription latency
H1 is the most likely candidate; the most direct test is to
temporarily comment out the ACP listener's sphere.on('message:dm')
subscription and re-run. If settlement completes, that's the bug.
Includes file refs for the trader code paths involved, exact
reproduce-steps for the failing scenario, and key log lines to
grep for in trader/escrow container output.
Investigated sphere-sdk's event dispatch. Incoming DMs go through TWO
INDEPENDENT paths (dist/index.js:13910-13918):
1. emitEvent("message:dm") — generic event bus (sphere.on subscribes)
2. iterates dmHandlers Set — onDirectMessage(handler) registers here.
PaymentsModule (line 18816) and SwapModule (line 24212) both
subscribe via this set.
Both fire unconditionally for every DM. Neither preempts the other —
they're independent channels. The ACP listener (event bus) cannot
block the SwapModule (dmHandlers) from receiving a DM. So H1 (ACP
listener consuming the invoice_delivery before SwapModule sees it)
is wrong.
Updated debugging plan focuses on:
- what gets through to SwapModule.handleIncomingDM
- SDK-internal silent-reject paths (sig check, swap-id-not-found,
protocol-version mismatch, dedup)
- transport-layer hole (relay subscription / NIP-17 decryption)
Also added an architectural-follow-up note: even though the dual-path
dispatch isn't the cause of THIS bug, the lack of propagation control
(no `consume vs pass through` semantics) is a future-bug source as
more consumers attach. A koa-compose-style middleware chain
(use(handler, priority) with next() semantics) in sphere-sdk's
CommunicationsModule would let consumers declaratively filter /
consume / pass DMs. ~40 lines of inline impl, no new runtime dep.
…ric bug A focused investigation agent traced round-12 escrow logs and found: diag_invoice_delivery_attempt party=A ✓ logged diag_outbound_dm_sending invoice_delivery → A ✓ logged diag_outbound_dm_sent invoice_delivery → A ✓ logged diag_invoice_delivery_complete party=A ✓ logged diag_invoice_delivery_attempt party=B ✓ logged [ no diag_outbound_dm_sending invoice_delivery → B ] diag_invoice_delivery_complete party=B ✓ logged anyway The deployed `escrow:v0.1` image's `deliverDepositInvoice` function exits "normally" for party B without actually invoking `sphere.communications.sendDM` — only one recipient receives the invoice. Trader stalls at ACCEPTED waiting for the invoice that never arrives. Asymmetry repeats deterministically across multiple swaps; basic-roundtrip works direct-docker because it only asserts the buyer's side. The current escrow-service source (`message-handler.ts:202-253`) LOOKS correct — both A's and B's reply() calls are awaited with the same code path. The deployed image was built from an unsynced source commit (the JS at /app/dist/sphere/message-handler.js doesn't match HEAD). Rebuilding escrow:local from current source and overriding the image in the test should clear the regression. Build instructions added to the test header. After this, expect hma-trade-settlement to finally COMPLETE both scenarios. Independent of this fix, an upstream PR against escrow-service to harden deliverDepositInvoice (use Promise.allSettled and log rejection reasons) so a similar build-time defect can never silently re-occur is filed in the diagnostic doc as a follow-up item.
The testnet relay (wss://nostr-relay.testnet.unicity.network) has had
two write-path outages in 4 days: read path works (connect+subscribe
OK) but every publish-kind:* times out, so DM-based settlement tests
stall before the first DM ships. Pattern is intermittent; recovery
unpredictable. Local relay infra removes the dependency.
Mirrors the harness uxf already uses
(/home/vrogojin/uxf/tests/e2e/local-infra/), adapted for trader-service
multi-container topology:
test/e2e-live/local-infra/docker-compose.yml
- container_name: trader-e2e-relay (avoid uxf collision)
- port bind: 0.0.0.0:7777:8080 — HMA-spawned tenants run in
their own Docker containers; they cannot reach host loopback
so we expose on all interfaces and connect via the host bridge
gateway IP. Relay image pin: ghcr.io/unicitynetwork/unicity-tokens-relay:sha-1e1b544
test/e2e-live/local-infra/relay.ts
- bootLocalRelay() / stopRelay() lifecycle
- getLocalRelayUrlForContainers() returns the URL HMA-spawned
tenants should connect to (`ws://<bridge-gateway-ip>:7777`).
Falls back to host.docker.internal if `docker network inspect`
fails — Docker Desktop resolves this automatically; Linux
Docker needs --add-host=host.docker.internal:host-gateway on
the HMA-spawned containers (separate plumbing required).
Service-side support for UNICITY_NOSTR_RELAYS env override (mirrors
the existing js-faucet pattern at acp-adapter/main.ts:122-128):
trader-service: src/trader/main.ts:250-265
escrow-service: src/acp-adapter/main.ts:101-117 (committed
separately; see escrow-service repo)
agentic-hosting: src/host-manager/main.ts:457-474 (committed
separately; see agentic-hosting repo)
Each service reads UNICITY_NOSTR_RELAYS (with SPHERE_NOSTR_RELAYS
fallback) and passes the relay list to createNodeProviders'
transport config when set; falls through to the network preset's
defaults when unset. No behavioral change when the env var is
absent.
Out of scope (next step):
- global-setup.ts wiring to boot the relay when
TRADER_E2E_LOCAL_RELAY=1 is set, then propagate the bridge URL
to the HMA + spawned tenants via env passthrough
- testing the full stack against the local relay
- opt-in via the hma-trade-settlement test
…n env
Closes the local-infra harness: when TRADER_E2E_LOCAL_RELAY=1 is set,
the test now boots a Docker-hosted Nostr relay AND propagates the
relay URL to every component that does Sphere DMs:
global-setup.ts:
- boots local-infra/relay (uxf-style, container trader-e2e-relay)
- sets process.env['UNICITY_NOSTR_RELAYS'] to the bridge-gateway URL
so HMA-spawned tenants can reach it
- skips the testnet preflight (local relay supersedes the gate)
- teardown stops the container; wipe-on-boot by default
(TRADER_E2E_LOCAL_RELAY_KEEP=1 to preserve event log for
post-mortem)
helpers/manager-process.ts:
- forward UNICITY_NOSTR_RELAYS / SPHERE_NOSTR_RELAYS into the
HMA's spawn env (the helper otherwise builds an isolated env
and would have dropped the override)
hma-trade-settlement.e2e-live.test.ts:
- read process.env['UNICITY_NOSTR_RELAYS'] at beforeAll, stash
onto SuiteState.spawnEnv
- inject spawnEnv into every hostSpawnAsync (escrow, alice, bob,
faucet) so the spawned containers all hit the local relay
Same env-override pattern works for both modes: with TRADER_E2E_LOCAL_RELAY
unset, spawnEnv is empty {} and tenants fall through to the network
preset's default relay (testnet), preserving the existing test
behavior.
Run against local relay:
TRADER_E2E_LOCAL_RELAY=1 npm run test:e2e-live -- \
test/e2e-live/hma-trade-settlement.e2e-live.test.ts
When the testnet relay is healthy (no current outage), the test
still runs against testnet by omitting the env var.
Linux Docker note: getLocalRelayUrlForContainers() runs
`docker network inspect bridge` to discover the gateway IP
(typically 172.17.0.1) — works from both host and containers.
Falls back to host.docker.internal which Docker Desktop resolves
automatically; recent Linux Docker also supports it via
--add-host=host.docker.internal:host-gateway (HMA's docker-adapter
would need to forward that flag if the gateway-IP detection ever
fails, which it shouldn't on a standard install).
Round 16 (TRADER_E2E_LOCAL_RELAY=1) booted the local relay and the manager registered correctly, but `sphere host spawn` failed with 'Unicity ID not found: @m-e2elive...'. Cause: helpers/sphere-cli.ts buildEnv() builds a sanitized env (PATH/HOME/UNICITY_API_KEY/CI/ FORCE_COLOR only) and dropped the relay override, so sphere-cli queried testnet for the manager's nametag while the manager was registered only on the local relay. Also patched sphere-cli upstream (host/sphere-init.ts and legacy/legacy-cli.ts) to read UNICITY_NOSTR_RELAYS / SPHERE_NOSTR_RELAYS in the same pattern as trader-service / escrow-service / agentic-hosting / js-faucet. Both inits now respect the env when set.
…mux is the next gap
Round 18 progressed massively (534× kind:1059 + 10× kind:30078 on the
local relay) but stalled on FAUCET_REQUEST timeout because the
in-process FaucetClient was built with testnet defaults and never saw
the local-relay override.
Three patches collected here:
1. helpers/faucet-client.ts — read UNICITY_NOSTR_RELAYS /
SPHERE_NOSTR_RELAYS and pass transport.relays to
createNodeProviders. The Sphere wallet that signs+sends
FAUCET_REQUEST DMs from the test process now hits the local relay.
2. helpers/manager-process.ts (provisionManagerWallet) — same
pattern. The pre-creation step that generates the manager's
wallet + publishes the nametag now respects the override.
Without this, the nametag binding event was published to
testnet (when the test process had no override pickup), and
the HMA binary loaded the existing wallet on launch
(wallet_created: false) → never re-published to the local
relay → sphere-cli's queryPubkeyByNametag returned 'not found'.
3. helpers/manager-process.ts (UNICITY_HEALTH_PORT) — default
to 0 (OS-assigned ephemeral port) instead of fixed 19401.
Tests don't probe this port, and a fixed default EADDRINUSEs
when a prior run leaks an HMA process. Mirror of the same
change already on js-faucet's manager-process.ts.
Test runs HMA→escrow→faucet→2 traders in parallel scenarios. With
all local images rebuilt (escrow:local with current source,
trader:local with current sphere-sdk + relay override, faucet:local
with relay override) settlement traffic flowed end-to-end on the
Docker-hosted local relay. The TRADER_E2E_LOCAL_RELAY=1 mode now
actually works.
End-to-end run of hma-trade-settlement against the Docker-hosted
local Nostr relay. Spawn (HMA + escrow + faucet + 2 traders) ✓
funding via FAUCET_REQUEST ✓ matched intents ✓ ACCEPTED deals ✓
escrow's invoice delivery to BOTH parties ✓ trader's invoice
import ✓ trader's deposit sent ✓.
Final stop: swap-protocol settlement bug —
'[Accounting] Direction mismatch: transport memo says
return_cancelled, on-chain says forward' → swap_cancelled.
That's an independent layer; deterministically reproducible
against the local relay, no longer blocked by testnet outages.
The asymmetric invoice-delivery bug (rounds 11-12) is GONE in
escrow:local from current source — every swap shows full
deliver_deposit_invoice_{enter,sending,sent} for both parties.
…Payout wall Round 20: replaces FAUCET_REQUEST funding in hma-trade-settlement.e2e-live with TRADER_TEST_FUND injected via HMA's --env passthrough. The faucet path delivered tokens whose source-state predicate the trader's swap-deposit signing key couldn't match, causing every deposit attempt to fail with "Ownership verification failed: Authenticator does not match source state predicate" (round 19). selfMint mints with the trader's own predicate, mirroring basic-roundtrip's known-working mechanism. Result with selfMint: the swap protocol completes end-to-end on the escrow side — announces, invoices delivered, both deposits verified, "invoice:covered", payouts paid, "Swap completed successfully". The trader receives the payout transfer (10 UCT) and the SDK marks swap progress as `completed`. But verifyPayout enters its fail-closed branch (sphere-sdk/modules/swap/SwapModule.ts:1997-2003) because getTokenIdsForInvoice(payoutInvoiceId) returns an empty Set: the synthetic-ledger reverse-index population at AccountingModule.ts:5755 isn't firing for swap payouts. Test times out at 8 min without reaching COMPLETED. This shifts the remaining gap out of HMA / relay / faucet land and into the SDK's instant-mode tokenInvoiceMap population. Round 20 of the diagnostic doc covers reproducer, log excerpts, and the three hypotheses to bisect next.
Settlement on local-relay completes in ~30-60s end-to-end (selfMint → match → deposits verify → payouts paid → verifyPayout). The previous 8-min budget existed because of the testnet-only era and gave painful slow-fail on the reverse-index bug now fixed by sphere-sdk fix/swap-deps-reverse-index. 3min gives 2× headroom over typical runs while failing fast on regressions.
Replaces the direct payments.send code path inside WITHDRAW_TOKEN with a createInvoice → payInvoice flow — the same path swap deposits use, so it inherits the SDK's well-tested predicate-handling and avoids the "Authenticator does not match source state predicate" flake on spends of received swap-payout tokens. - New AccountingAdapter interface in src/trader/types.ts (narrow facade over sphere.accounting, just createInvoice + payInvoice). - main.ts wires sphere.accounting through to the agent when present. - Legacy direct-send path retained for unit tests with stub adapters.
…REQUEST Switches the funding path back to FAUCET_REQUEST DMs against the shared js-faucet agent (the production-realistic path) now that js-faucet's conservative-transferMode fix and escrow's matching fix let recipients spend the delivered tokens without hitting the "Authenticator does not match source state predicate" race. Recipient address resolution: send to the trader's @NameTag (canonical identity per project guidelines). The previous attempt at DIRECT:// <pubkey> hit "No binding event found" because the SDK's resolveAddressInfo queries by hashed address and the trader's binding publishes the L3-predicate-derived directAddress, not the bare DIRECT://<pubkey> string. @NameTag goes through queryPubkeyByNametag which is the path the trader registers via Sphere.init(nametag=…). Verified end-to-end: both Pair-1 (rate=1) and Pair-2 (rate=3) reach deal COMPLETED + withdraw verified in ~140-175s each.
…ival-order race
Pair-2 of HMA-trade-settlement.e2e-live was failing with deals stuck at
ACCEPTED → CANCELLED while Pair-1 (same code, different rate) reliably
passed. Investigation traced the failure to a two-DM race in the
proposer-side swap initiation:
When a counterparty accepts a match, negotiation-handler sends
np.propose_deal to us, then swap-executor.executeDeal() calls
swap.proposeSwap() (sphere-sdk SwapModule) which sends an independent
swap_proposal DM. These are two separate Nostr events with no causal
ordering. On the receiver side:
- sphere-sdk fires swap:proposal_received synchronously from the NIP-17
receive path → main.ts immediately calls agent.registerSwapId().
- registerSwapId looks up activeByDealId for an entry with swapId=null.
That entry is created by swap-executor.executeDeal()'s registerActive(),
which is reached only AFTER negotiation-handler.handleProposeDeal
finishes its long async pipeline (validate → intent lookup → terms
check → transitionDeal('ACCEPTED') → reply DM → onDealAccepted →
executeDeal).
In Pair-1, np.propose_deal arrived ~106ms before swap_proposal — the
heavier handler completed in time and registerSwapId saw the deal. In
Pair-2, swap_proposal arrived just 21ms before NP-0 finished
registering, so registerSwapId returned false → swap rejected with
NO_LIVE_NP0_DEAL → deal stuck ACCEPTED with no path forward.
Fix: bounded retry in main.ts swap:proposal_received handler.
registerSwapId is called up to 40 times with 50ms backoff (~2s total
wait) before falling through to rejectSwap. Applied symmetrically to
both the status-based path and the legacy fallback path. Safe because
registerSwapId still cross-checks counterparty pubkey, currencies,
amounts, escrow address, and timeout against negotiated DealTerms;
retrying only papers over the microsecond-scale ordering hazard. A
genuinely hostile peer still gets rejected after the bounded wait.
Plus docs/HMA-SETTLEMENT-DIAGNOSTIC.md updated with rounds 21-22 + final
SDK changes summary.
Verified: HMA-trade-settlement.e2e-live both Pair-1 and Pair-2 reach
deal COMPLETED + withdraw verified concurrently.
…voice
Withdraw was racing recipient-side proof-poll on the sender's swap-payout
tokens. The default 'instant' transferMode delivers an unconfirmed
{sourceToken, transferTx} bundle whose finalization completes
asynchronously after receipt. When the trader's spend queue picked a
not-yet-finalized token (e.g., the swap payout that just arrived) for
the withdraw transfer, it produced "Authenticator does not match source
state predicate" because the recipient's confirmed Token wasn't yet
bound to its predicate.
Switch withdraw's accounting.payInvoice call to transferMode:
'conservative'. The SDK now collects the inclusion proof on the SENDER
(controller) side before delivery, so the controller's wallet receives
a fully-finalized bundle and produces a 'confirmed' Token immediately
bound to its own predicate.
This mirrors the faucet's funding flow and the escrow's swap-payout
flow, which both already use 'conservative' on their direct
payments.send paths. The trader's invoiced withdraw was the only
forwarding flow still using the default.
Requires the corresponding sphere-sdk change that exposes transferMode
on PayInvoiceParams (test/all-fixes-r23 branch, commit 4e77b2f).
…cate-mismatch flake
Owner
Author
Round 23 — RESOLVED ✓Latest e2e run (2026-05-10): both Pair-1 and Pair-2 PASS end-to-end. Final fixThe trader's invoiced withdraw was the only forwarding flow still using Companion PRs (must merge alongside)
Side fixes
Full writeup in |
Two coordinated changes:
1. constants.ts ESCROW_IMAGE: v0.1 → v0.2 (mirrors master bump from
PR #20). Adds composition note in the docstring.
2. hma-trade-settlement.e2e-live.test.ts templates override: was
forcing `ghcr.io/vrogojin/agentic-hosting/escrow:local` with a
comment blaming the published v0.1 asymmetric deliverDepositInvoice
bug. Switched to ESCROW_IMAGE (= v0.2) so the HMA-spawned escrow
runs the same code as the direct-Docker-spawned escrow in
basic-roundtrip — and devs/CI no longer need to docker-build
`escrow:local` before running this test.
Why the override remains (vs deleting it entirely):
agentic-hosting's config/templates.json still pins escrow:v0.1 on its
own release cadence. Until that templates.json bumps, we override here
so the HMA-spawned escrow picks up v0.2's:
- deliverDepositInvoice fix (round-19 evidence confirms gone)
- Conservative transferMode for swap payouts
- sphere-sdk UXF protocol PRs (#105, #115, #119, #128,
#146/147/149/152) + all payments/* faucet-flow regression fixes
v0.2 digest:
sha256:311903b6f98b33a63791bf79db6522a66d118588ba56fcf6e56654ed6670ebac
Typecheck: clean against tsconfig.json + tsconfig.test.json.
…override Bumps constants.ts ESCROW_IMAGE v0.1 → v0.2 on the feature branch (matches master PR #20) AND replaces the hma-trade-settlement.e2e-live templates override from `escrow:local` to `ESCROW_IMAGE`. Removes the docker-build dependency for this test — devs/CI no longer need to locally build escrow:local before running it. Round-19 evidence (HMA-SETTLEMENT-DIAGNOSTIC.md) confirms the v0.1 asymmetric deliverDepositInvoice bug is GONE in current source = what we shipped as v0.2. v0.2 digest: sha256:311903b6f98b33a63791bf79db6522a66d118588ba56fcf6e56654ed6670ebac. Typecheck clean against both tsconfig.json and tsconfig.test.json.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test/e2e-live/hma-trade-settlement.e2e-live.test.ts— a full HMA-orchestrated e2e covering spawn → trade → settle → withdraw, withit.concurrent2-pair concurrency.selfMintvia theTRADER_TEST_FUNDenv passthrough (replaces the flakyFAUCET_REQUESTpath).WITHDRAW_TOKENhandler throughaccounting.createInvoice+accounting.payInvoice— the same code path swap deposits use — via a newAccountingAdapterfacade insrc/trader/types.ts.main.tswiressphere.accountingthrough to the agent. The settlement timeout was tightened from 8 min to 3 min for a faster fail signal.Test plan
Current outcome:
Dependencies
feat/trader-withdraw-cliPR — the e2e helpers shell out tosphere trader withdraw, which was missing as a registered subcommand until that PR.The HMA-SETTLEMENT-DIAGNOSTIC doc on this branch is being updated in parallel by another agent; any further doc changes will land in a follow-up commit.