Skip to content

fleet spawn reports failure (unconfirmed dispatch) for agents that did launch - #1942

Open
agent-relay-code[bot] wants to merge 7 commits into
mainfrom
relayflow/relay-software-garden-f6614fab
Open

agent-relay-code[bot] wants to merge 7 commits into
mainfrom
relayflow/relay-software-garden-f6614fab

Conversation

@agent-relay-code

@agent-relay-code agent-relay-code Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Fix late fleet spawn confirmation and provide dispatch status reads

fleet spawn previously exhausted its two-minute confirmation budget while agents were still launching, then suggested redispatching without confirmation. Launches taking 1–5 minutes now have a six-minute default budget in the CLI and SDK. Explicit timeout overrides and the existing verified-spawn minimum remain supported.

Add agent-relay fleet spawn-status <invocation-id> to read the original spawn invocation without dispatching another worker. Timeout errors retain their structured invocation ID and direct callers to this command. It uses existing lifecycle receipts to report confirmed readiness, terminal failure, or an outcome still awaiting confirmation. A successful unverified launch retains spawned: true, ready: false with state accepted. Workspace-only callers mint and clean up a temporary reader using the established launcher pattern. Output excludes task payloads and raw handler errors.

Silence cannot establish that a process never started. A node that never returns a result remains explicitly unconfirmed; a terminal node failure is reported as failed. No roster-only inference of readiness is introduced. These tests use deterministic fixtures, not the affected physical fleet nodes.

The new leaf is carried onto the surfaces that enumerate commands: the bootstrap leaf-command inventory, the trusted Fleet CLI inventory snapshot (tests/relayflows/cleanroom/fleet-cli-inventory.json) and its matrix pin, the fleet README, and the feature manifest. The Daytona board defers fleet spawn-status rather than mapping it to an operation: a read-only poll of a recorded invocation id proves nothing about placement that the spawn it reads has not already proven.

Validation:

  • sh .relayflow/check.sh — all checks passed (npm ci, codegen check, build, typecheck, lint, format, PR-proof and subscription guards, vitest run: 200 test files, 3754 tests).
  • Regression coverage: a five-minute launch confirmed with exactly one dispatch; expiration of the extended budget returning a pollable ID whose message names the poll command; pending/ready/failed/unverified status reads; launch evidence (output.spawned/ready, dispatch and handler node ids) surviving the sanitized read; safe receipt output; temporary reader cleanup; fleet spawn-status present as a public leaf in both command inventories.

Two check failures were environmental, not code: node-claim.test.ts needs lsof/ps and broker-process-identity.test.ts compiles a C fixture with cc. GitHub's ubuntu runners ship these (node-compat.yml installs lsof procps; test-install.yml installs build-essential), so the install step was added to the uncommitted .relayflow/check.sh rather than changing any test.

Pre-existing, left alone: the committed Fleet CLI inventory snapshot is stale for six node commands (--state-dir, --broker-url, --api-key were never snapshotted after addBrokerOptions was applied to them), so the qualification job's inventory comparison fails on main independently of this change. The snapshot regenerated here restores those six records verbatim to keep that drift out of this change.

No workflow files changed. The existing unrelated active Trail trajectory prevented starting a new one; it was left intact.

Checks

Relayflow ran this repository's checks (.relayflow/check.sh) and they passed.

What ran (.relayflow/check.sh)
```text
#!/bin/sh
# Fresh-machine equivalent of this repo's CI checks.
#
# Mirrors .github/workflows/test.yml (jobs: test, lint) and the Node-side
# pre-test steps, plus .github/workflows/codegen-models.yml and the
# toolchain-gated Rust jobs in .github/workflows/rust-ci.yml.
#
# Deliberately omitted, because they need secrets, deployments or services
# this machine does not have:
#   - coverage upload (test.yml "coverage" job): needs the Codecov token.
#   - Windows credential ACL job (test.yml) and the Windows/macOS matrix
#     legs: need other runner OSes.
#   - Swift SDK job (test.yml "swift-test"): needs a macOS runner + Swift.
#   - cross-compile legs (rust-ci.yml): need cross linkers/targets.
#   - e2e-tests, fleet-e2e, prod-smoke, publish, verify-publish*, relay-evals,
#     deploy-relayflows, *-qualification, stress-tests: these drive live
#     brokers, cloud/Daytona nodes, real agent CLIs, registries or releases.
#   - npm test:integration* and npm run eval*: require real agent CLIs
#     (RELAY_INTEGRATION_REAL_CLI) and provider API keys.
set -e

cd "$(dirname "$0")/.."

# CI sets this on every job; keeps the checks from emitting telemetry.
export AGENT_RELAY_TELEMETRY_DISABLED=1

# The Node suites shell out to these: node-claim.test.ts probes hold descriptors
# with `lsof`/`ps` (node-compat.yml installs `lsof procps` the same way), and
# broker-process-identity.test.ts compiles a C fixture with `cc`
# (test-install.yml installs `build-essential`). GitHub's ubuntu runners ship
# all of them; a bare container does not.
echo "==> Installing test OS dependencies (lsof, procps, cc)"
if ! command -v lsof >/dev/null 2>&1 || ! command -v ps >/dev/null 2>&1 || ! command -v cc >/dev/null 2>&1; then
  sudo apt-get update
  sudo apt-get install -y --no-install-recommends lsof procps gcc libc6-dev
fi

echo "==> Installing dependencies (npm ci)"
npm ci

# test.yml does this on every Node job: npm's optional-dependency handling can
# leave the platform rollup binary out of a lockfile install.
echo "==> Ensuring rollup optional dependencies are installed"
npm install --no-save rollup || true

# codegen-models.yml regenerates these from packages/utils/cli-registry.yaml and
# fails the PR if the result differs from what is committed.
echo "==> Code generation (models)"
npm run codegen:models
if ! git diff --quiet -- packages/config/src/cli-registry.generated.ts packages/sdk-py/src/agent_relay/models.py; then
  echo "Generated model files are out of date; commit the result of 'npm run codegen:models'." >&2
  git --no-pager diff -- packages/config/src/cli-registry.generated.ts packages/sdk-py/src/agent_relay/models.py >&2
  exit 1
fi

# `npm test` runs this via its pretest hook; run it up front so a build break
# fails before the suite starts. build:rust self-skips when cargo is absent.
echo "==> Build (clean + rust broker if cargo present + all packages)"
npm run build

echo "==> Typecheck"
npm run typecheck

echo "==> Lint"
npm run lint

# Formatting is auto-remediated on push/PR by prettier-fmt-fix.yml, so it is
# reported but not treated as a failure here.
echo "==> Format check (advisory)"
npm run format:check || echo "note: prettier reported unformatted files; prettier-fmt-fix.yml fixes these in CI"

# Rust jobs (rust-ci.yml) only run when crates/ changes, and only if a
# toolchain is present on this machine.
if command -v cargo >/dev/null 2>&1 || [ -x "$HOME/.cargo/bin/cargo" ]; then
```

_…truncated to fit GitHub's limit: 4171 bytes were cut to under 4000. The full text is .relayflow/check.sh in the run workspace._

Fixes #1935

View guided diff


Note

Medium Risk
Changes fleet spawn confirmation timing and adds a new operational path for pending dispatches; mis-timed timeouts or misread poll states could still lead to duplicate spawns if operators ignore guidance, but behavior is read-only and backward-compatible for explicit timeouts.

Overview
Fixes late fleet spawn confirmation by extending the default launch-confirmation window from two to six minutes in the CLI (fleet spawn --confirm-timeout) and SDK placement polling, so workers that register minutes after dispatch are confirmed instead of surfacing spawn_unconfirmed.

Adds agent-relay fleet spawn-status <invocation-id>, a read-only poll of the original spawn invocation (no second dispatch). JSON output uses sanitized lifecycle receipts with placement.state (ready, failed, accepted, unconfirmed_may_be_running) and preserves dispatch/handler node evidence; workspace-only callers get a short-lived reader identity. Timeout errors now point at this command rather than suggesting a blind retry.

The SDK RelayActionInvocation type and normalizeActionInvocation now surface handlerNodeId / dispatchedNodeId when the server reports them. Command inventories, README, changelog, and feature manifest are updated accordingly.

Reviewed by Cursor Bugbot for commit 3a6f0c0. Bugbot is set up for automated code reviews on this repo. Configure here.

Relayflow and others added 3 commits October 8, 2026 23:50
`fleet spawn-status` was registered without updating the surfaces that
enumerate commands, so the bootstrap leaf inventory and the Fleet CLI
inventory test failed.

- Add the leaf to `expectedLeafCommands` and to the current-main Fleet
  inventory counts, with an assertion that it is a public leaf: the spawn
  timeout error names the command, so a hidden or missing one makes that
  guidance unactionable.
- Refresh the trusted Fleet CLI inventory snapshot and the matrix pin for
  the new leaf and the 360000ms `--confirm-timeout` default, and declare
  its option coverage. The Daytona board defers the leaf rather than
  mapping an operation: a read-only poll of a recorded invocation id
  proves nothing about placement that the spawn it reads has not already
  proven. Six `node` records with pre-existing snapshot drift
  (`--state-dir`, `--broker-url`, `--api-key`) are restored verbatim to
  keep that drift out of this change.
- Cover the evidence the fix exists for: launch proof and the answering
  node id survive the sanitized read, and a dispatch that has not
  reported acquires no launch claim.
- Document the command in the CLI README and the feature manifest, and
  split the changelog entry (new command is Added, hence Minor).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3278f15f-d029-4537-9fa9-655cb9733ff0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@agent-relay-code
agent-relay-code Bot changed the base branch from main to trunk October 9, 2026 00:11

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread packages/cli/src/cli/commands/fleet.ts Outdated
Comment thread packages/cli/src/cli/commands/fleet.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 99c51b3. Configure here.

Comment thread packages/cli/src/cli/commands/fleet.ts Outdated
Comment thread packages/cli/src/cli/commands/fleet.ts
@khaliqgant
khaliqgant changed the base branch from trunk to main October 10, 2026 07:23
@khaliqgant khaliqgant closed this Oct 10, 2026
@khaliqgant khaliqgant reopened this Oct 10, 2026
khaliqgant and others added 3 commits October 10, 2026 00:27
…hangelog sections)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…cation node ids

spawn-status passed the workspace key alongside its minted launcher token,
which createAgentRelay rejects, and sent that token to the default gateway
rather than the one that minted it. Read with the token and the resolved
workspace origin only, as the sandbox spawn path does.

normalizeActionInvocation dropped handler_node_id and dispatched_node_id, so
spawn-status could never report the node that answered a dispatch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
khaliqgant
khaliqgant previously approved these changes Oct 10, 2026
…are-garden-f6614fab

# Conflicts:
#	packages/sdk/src/__tests__/relaycast-translate.test.ts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fleet spawn reports failure (unconfirmed dispatch) for agents that did launch

1 participant