test(e2e-live): HMA-orchestrated tenant lifecycle (foundation for PR-C trade ops) - #14
Merged
Merged
Conversation
…e host)
Adds the architectural foundation for live e2e tests that spawn every
tenant THROUGH the host-manager agent (HMA) instead of via direct
docker run. This is the production architecture: HMA owns lifecycle,
controllers reach the HMA via HMCP DMs, and trade ops (subsequent PR-C)
go controller→tenant directly via ACP DMs.
What's new in this PR:
test/e2e-live/helpers/manager-process.ts
Pre-provisions a Sphere wallet for the manager, then spawns
dist/host-manager.js from the agentic-hosting checkout (default
/home/vrogojin/agentic_hosting, override AGENTIC_HOSTING_PATH).
Resolves once `host_manager_started` lands in stdout. Drift-guard
env vars (MANAGER_PUBKEY/MANAGER_DIRECT_ADDRESS) are wired from
the pre-created wallet so the manager loads it cleanly.
test/e2e-live/helpers/sphere-cli.ts
Locates the sphere binary (default
/home/vrogojin/sphere-cli-work/sphere-cli/bin/sphere.mjs, override
SPHERE_CLI_BIN), probes via `sphere --help`, returns ok/skip
reason. Includes createSphereCliEnv() for an isolated CWD with
.sphere-cli/config.json, and bootstrapControllerWallet() that
runs `sphere wallet init` and parses chainPubkey from output.
test/e2e-live/helpers/hma-spawn.ts
hostSpawn / hostStop / hostList wrappers around sphere-cli's
`sphere host …` subcommands. Parses --json output, returns
typed payloads (SpawnedTenant with tenantPubkey,
tenantDirectAddress, tenantNametag). Throws on hm.error /
hm.spawn_failed so callers don't need defensive parsing.
test/e2e-live/hma-orchestrated.e2e-live.test.ts
Foundation test: bootstraps controller, boots manager, spawns
1 escrow + 2 traders via `sphere host spawn`, asserts each
returns hm.spawn_ready RUNNING with a valid tenant pubkey,
verifies all three appear in `sphere host list`. Cleans up
each tenant via `sphere host stop` then stops the manager.
test/e2e-live/{preflight.ts,global-setup.ts,infra-probe.d.ts}
@unicitylabs/infra-probe preflight (mirrors agentic-hosting's
pattern). Aborts the run up-front if testnet Nostr/aggregator/
IPFS/Fulcrum/Market is unreachable. Bypass via
TRADER_E2E_SKIP_PREFLIGHT=1.
vitest.e2e-live.config.ts
Wires globalSetup; refreshes the stale comment block (the prior
"tests are NOT runnable in trader-service standalone" note no
longer applied since direct-docker tests already work). Tightens
include glob to `*.e2e-live.test.ts` (helper unit tests stay in
the default suite).
Skip semantics:
describe.skipIf gates the new test file when sphere-cli isn't
runnable OR the agentic-hosting binary is missing. Both repos move
independently; an upstream regression in either should not red-CI
this branch. The skip message names which prerequisite is missing
so the operator can fix it.
Verified live (2026-05-03 testnet, all 5 services HEALTHY):
TRADER_E2E_SKIP_PREFLIGHT=1 npx vitest run --config vitest.e2e-live.config.ts \
test/e2e-live/hma-orchestrated.e2e-live.test.ts
→ 1/1 PASS in 38s.
Manager booted in ~5s (Sphere.init + nametag mint).
Three real Docker spawns × ~10s each (HMCP request → docker create
→ tenant Sphere.init in container → acp.hello → spawn_ready).
Default `vitest run`: 651 tests still pass. Existing direct-docker
e2e-live tests remain untouched — the migration from direct-docker
to HMA-orchestrated is incremental (this PR lands the foundation;
subsequent PRs migrate scenarios one at a time).
Adds @unicitylabs/infra-probe@^0.3.0 as devDependency, plus npm
scripts: `preflight`, `preflight:json`.
Depends on:
- agentic-hosting PR #22 (Phase 5 DM transport) — already merged.
- sphere-cli PR #6 (encrypt/decrypt L1 namespace fix) — open.
Aggregated fixes from parallel reviews of PR #14 (refactor/e2e-live-via-hma) by code-reviewer × 2, security-auditor, architect-review agents. CRITICAL — code-reviewer (process lifecycle): manager-process.ts - stop(): wrap c.kill('SIGTERM') in try/catch matching the SIGKILL fallback. The asymmetric handling was a latent ESRCH time-bomb if the process exited between the exitCode check and the kill (Node sets exitCode on the tick AFTER the OS-level exit). - ready promise: setInterval watcher.unref() so a missed cleanup path can never keep the event loop alive past hookTimeout. All clearInterval calls are still present; this is a defense-in-depth safety net. WARNING — security-auditor: manager-process.ts / sphere-cli.ts - mkdir(..., { recursive: true }) without explicit mode honours umask (typically 0o022 → 0o755), exposing testnet wallet material to other local users on shared CI runners. Pass `mode: 0o700` to every mkdir/mkdirSync call that creates a directory under which mnemonics or chain keys land. Add chmodSync(home, 0o700) in createSphereCliEnv as defense-in-depth for platforms where mkdtempSync may deviate from posix-default. - bootstrapControllerWallet's failure paths previously echoed init.stderr and init.stdout into thrown Error messages. The upstream `sphere wallet init` writes the freshly-minted mnemonic to stdout (suppressed only when isTTY=false; behaviour depends on the upstream version). Vitest captures error messages to log files, persisting potential mnemonics on disk. Redact the subprocess output entirely; tell the operator to re-run with stdout connected to debug. - ensureTrustbase: the URL pins to a mutable refs/heads/main ref, a real attack surface for any future mainnet-targeted use. Add a SHA-256 log line and a TODO to pin to a commit SHA. The hash check is advisory (we don't fail on mismatch since pinning a known hash here would create churn on legitimate upstream updates) — but an unexpected hash will surface in test output. WARNING — code-reviewer (parsing consistency): hma-spawn.ts - hostList silently returned [] when payload.instances was missing or malformed. Now throws like hostSpawn does — a future protocol rename to `tenants` would surface as a clear "missing payload.instances" error rather than as a misleading "expected 3 RUNNING, got 0". WARNING — architect-review (developer-default footgun): manager-process.ts / sphere-cli.ts - Hard-coded /home/vrogojin/ defaults are kept as developer fallbacks but rejected when CI=1. A missing SPHERE_CLI_BIN or AGENTIC_HOSTING_PATH on a CI runner is now a hard fail, not a silent skip — eliminates the false-confidence "tests passed but were skipped" signal flagged by the architectural review. - New checkAgenticHostingPath() helper returns a structured {ok,reason} so the test's describe.skipIf can surface a precise diagnostic without the test re-implementing the env-var logic. WARNING — architect-review (deprecation signaling): helpers/contracts.ts - The architectural prelude described the direct-docker pattern as the target architecture. Rewritten to document BOTH flavors (Architecture A: direct-docker, scheduled for removal; Architecture B: HMA-orchestrated, target). Future contributors reading this file see the migration plan, not a stale declaration of intent. helpers/tenant-fixture.ts - Added @deprecated JSDoc pointing to hma-spawn.ts as the successor. Existing tests using provisionTrader() continue to work; new tests should follow Architecture B. Verified live: hma-orchestrated.e2e-live.test.ts still passes in 40s on testnet (was 38s pre-fix; the +2s is the 250ms unref'd interval plus the trustbase hash log). 651 default tests still pass.
Round 1 of steelman loop on PR #14. Six findings addressed: WARNING — CI gate accepted only `CI=1`, missing every mainstream CI provider (GitHub Actions, GitLab, CircleCI all set `CI=true`). The fix's stated goal (fail loud on misconfigured runners) silently failed on the most common targets. New `isCi()` helper in `sphere-cli.ts` matches `CI=true|True|1|<any-truthy>` and Azure's `TF_BUILD`. Both `probeSphereCli()` and `resolveAgenticHostingPath()` now use it. WARNING — `stop()` deadlocked for 12s if the manager exited before stop() was called. The `c.once('exit')` listener never fires for an already-emitted exit event, so the await hung until the (unref'd) killTimer expired on the event loop. With cascading test failures this could pile up to (12s × N) afterAll wait. Fix: check `exitCode` AND `signalCode` BEFORE attaching the listener; also re-check inside the Promise constructor to close the listener-attach race window. NOTE — Trustbase SHA-256 was truncated to 16 hex chars in the log line, defeating the verification utility (16 chars is marginal collision resistance and not enough for an operator to paste into a Slack message and compare). Log full 64 chars. NOTE — `chmodSync` comment misattributed the threat. Real risk is ordering: any future writeFileSync between mkdtempSync and chmodSync becomes a TOCTOU disclosure on non-POSIX platforms. Comment rewritten to flag the ordering dependency. NOTE — `hostList` error echoed full payload via `JSON.stringify(obj)`, which could include peer pubkeys / instance IDs in vitest log files. Now logs only top-level + payload key names. NOTE — Architecture-A symbols in `contracts.ts` (DockerRunOptions, DockerContainer, RunContainer, StopContainer, RemoveContainer, GetContainerLogs, WaitForContainerRunning, ProvisionTraderOptions, ProvisionedTenant, ProvisionTrader) had no per-symbol @deprecated JSDoc. The prelude alone is not surfaced at call sites by TypeScript's language server. Added @deprecated tags everywhere. Verified: hma-orchestrated.e2e-live.test.ts passes in 38s, default 651-test suite green, lint+typecheck clean.
Round 2 of steelman loop on PR #14. Four findings addressed: NOTE — isCi() accepted `CI=no` and `TF_BUILD=False` as truthy. Apply the same falsy normalization to TF_BUILD that CI already had; add `no` (case-insensitive) to the falsy set. Hedge the JSDoc to "common providers" rather than over-claiming "every truthy value." WARNING — hostStop was synchronous (spawnSync), so `Promise.allSettled(spawned.map(hostStop))` ran SEQUENTIALLY despite the parallel-looking shape. Three sequential 75s budgets approach the 240s afterAll hookTimeout under any tenant slowdown — a tail test failure could trip the hook timeout. Fix: add `runSphereAsync` (uses spawn instead of spawnSync) and convert hostStop to return a Promise. The afterAll teardown now genuinely fans out three DM round-trips concurrently; total budget is bounded by the SLOWEST tenant, not the SUM. Drop the cosmetic Promise.resolve() wrap in the test now that hostStop is genuinely async. WARNING — TraderCtlOptions, TraderCtlResult, RunTraderCtl in contracts.ts had no @deprecated tags. Round 1 added tags on the Architecture-A docker types but missed the Architecture-A trader-ctl driver types. Added now — the IDE/lint signal at call sites is restored. NOTE — Trustbase hash log has no baseline to compare against. Document this with a TODO: until upstream unicity-ids publishes a canonical hash per release tag, the log is decorative. Either pin the URL or store an expected hash in this repo and assert on it. Verified: live test passes in 35s (down from 38s — the parallel teardown shaved a few seconds), lint+typecheck clean.
… cap) Round 3 of steelman loop on PR #14. Three findings addressed (PR-A returned ROUND CLEAN with no new findings). NOTE — runSphereAsync timer callback unconditionally set `timedOut = true` and called child.kill, even if the process already exited cleanly at the timer boundary. Result: a successful hostStop could log status=null (false-timeout) — confusing in post-mortem debugging though not a test-failure (Promise.allSettled swallowed). Fix: guard the timer callback with `child.exitCode !== null || child.signalCode !== null` short-circuit, matching the same pattern stop() already uses. NOTE — isCi() edge cases: • `CI=off` was treated as truthy. Added 'off' to the falsy set (matches npm/shell-script convention; alongside '0', 'false', 'no'). • `CI=' '` (whitespace-only) was treated as truthy. Added .trim() before the empty-string check. NOTE — runSphereAsync had no maxBuffer guard. spawnSync defaults to maxBuffer=1MB and throws on overflow; child_process.spawn has no such limit. A misbehaving subprocess flooding stdout could OOM the test process. Fix: track per-stream byte count, kill the child and reject the promise with a descriptive error if either stream exceeds 10 MiB. Bound memory deterministically. Verified: live test passes in 37s, lint+typecheck clean. PR-A (sphere-cli #6) returned ROUND CLEAN this round — every attack either failed against the current code or surfaced pre-existing process notes (undated TODO, uncommitted trader-commands files) already acknowledged in earlier rounds. Ready for merge.
Round 4 of steelman loop on PR #14. Three findings addressed: WARNING — runSphereAsync's `error` handler called `reject(err)` without checking the `overflowed` flag, while the parallel `close` handler had `if (overflowed) return;`. The asymmetry was a documented reliance on Promise idempotency (Node silently ignores double-settle) — not a runtime bug, but inconsistent. Add the same overflow guard to the error handler so both settle paths follow the identical pattern. NOTE — Timer comment said "Race-safe" but the fix only NARROWS the race between OS-level exit and Node processing SIGCHLD. Both exitCode and signalCode are null in the few-microseconds window between those events, so a timer firing in that window can still misreport a clean exit as a timeout. Updated comment to "narrows the race window" and explicitly notes the residual edge case. NOTE — `MAX_BUFFER_BYTES` doc said "10 MiB per stream" but didn't make the worst-case clear. Clarified: stdout + stderr each get their own 10 MiB cap, so worst-case memory before overflow fires is 20 MiB. Acceptable for a test helper but worth documenting. Verified: live test passes in 27s (down from 37s — testnet was faster this run, not a code change), lint+typecheck clean.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the architectural foundation for live e2e tests that spawn every tenant through the host-manager agent (HMA) instead of via direct
docker run. This is the production architecture: HMA owns lifecycle, controllers reach the HMA via HMCP DMs, and trade ops (subsequent PR-C) go controller→tenant directly via ACP DMs.This is part 1 of 2 — this PR proves lifecycle works end-to-end on real testnet; PR-C will add the trade-ops layer (
sphere trader create-intent, portfolio, swap settlement).Verified live (2026-05-03 testnet)
Three real Docker spawns through the HMA in 38s:
acp.hello→hm.spawn_ready: ~10ssphere host listverifies all three RUNNING with valid tenant pubkeyssphere host stop× 3, manager.stop()Architecture
What's new
test/e2e-live/helpers/manager-process.tshost_manager_startedlog line. Drift-guard env vars wired from the pre-created wallet.test/e2e-live/helpers/sphere-cli.tsspherebinary (SPHERE_CLI_BINoverride), probes via--help,bootstrapControllerWallet()runssphere wallet initand parses chainPubkey.test/e2e-live/helpers/hma-spawn.tshostSpawn/hostStop/hostListtyped wrappers aroundsphere host …subcommands. Throws onhm.error/hm.spawn_failed.test/e2e-live/hma-orchestrated.e2e-live.test.tssphere host list.test/e2e-live/{preflight.ts,global-setup.ts,infra-probe.d.ts}@unicitylabs/infra-probepreflight gate. Bypass:TRADER_E2E_SKIP_PREFLIGHT=1.vitest.e2e-live.config.ts*.e2e-live.test.ts; refreshes the stale "tests not runnable standalone" comment.Skip semantics
describe.skipIfgates the new test when sphere-cli isn't runnable OR the agentic-hosting binary is missing — both repos move independently; an upstream regression in either should not red-CI this branch. The skip message names which prerequisite is missing.Default
vitest run651 tests still pass. Existing direct-docker e2e-live tests remain untouched — the migration is incremental (this PR lands the foundation; subsequent PRs migrate scenarios one at a time).
Dependencies
--helpexits non-zero, so this PR is mergeable independently.Test plan
npm run preflightto confirm testnet reachable.npm run test:e2e-live -- test/e2e-live/hma-orchestrated.e2e-live.test.tsagainst live testnet — should complete in <1 min.vitest.e2e-live.config.tscomment block and confirms it accurately describes the two-flavor (direct-docker + HMA-orchestrated) state.Out of scope (PR-C)
sphere trader create-intent | portfolio | list-deals | set-strategy | cancel-intentinvocations