Skip to content

AGENT-RUNTIME.md

Latest commit

 

History

History
1244 lines (1085 loc) · 72.8 KB

AGENT-RUNTIME.md

File metadata and controls

1244 lines (1085 loc) · 72.8 KB
Raw

The fleet agent runtime

fleet runs one native agent loop, in the fleet process. Model-authored local execution — bash, run_python, and file I/O — runs inside a rootless-Podman sandbox. MCP and the explicitly inventoried native broker/control-plane tools execute fixed host code so their credentials remain host-side and never enter the sandbox. There is no flavor picker and no external-agent delegation: the loop, the sandbox, and the credential broker are the whole story.

This guide documents the runtime mechanics an operator needs: the per-turn sandbox seal, the cost/token ceilings, context-window compaction, the per-task MCP credential allowlist, the scheduled end-of-run verifier, the optional "phone a friend" super-LLM review, and git-worktree isolation for scheduled tasks.

The per-turn execution sandbox

Every bash / run_python call — and every view_file / write_file / edit_file call, via the sandbox FileOp seam (#784, Sandbox.RunFileOp) — runs inside an ephemeral rootless-Podman container over a persistent per-conversation workspace. The file tools do host-side path validation as defense-in-depth input, then execute the actual read/write/edit in the sandbox (a one-shot podman exec python3), so they inherit the same runtime, seccomp, caps, cgroups, disk/PID limits, and lockdown network posture as bash; the helper is confined to the narrow conversation/worktree root with descriptor-relative no-follow traversal, and cancellation kills + retires the container before returning. With no sandbox they fail closed (no host fallback). The agent loop itself runs in the fleet process, but it holds no privileged local executor: local data-plane calls are handed to the sandbox under full host policy (cost ceilings, repeat detection, critical-tool approval staging, the email send gate, …), and fleet records the real, executed tool calls as the audit trail, not a self-report. A small set of native tools are deliberately host-side control-plane/broker operations (host network fetch, brokered credentials, governed datastore writes) — enumerated in ADR-0036.

MCP credentials never enter the sandbox or model context. MCP tools are advertised to the model, but every mcp_* call is executed by fixed host-side code against a credentialed client. The mcpbroker package and fleet mcp-broker subprocess implement the intended out-of-process boundary, including value-free, incident-correlated responses when broker call or discovery code panics. Its internal scoped-session protocol can also open a per-run client from non-secret account/task/workspace identifiers and route calls through an opaque scope ID, which is required before scheduled runs with different account selections can share the subprocess safely. The child backend owns those scoped clients and reaps them on close. Production starts fleet mcp-broker before serving, verifies it with liveness and public catalog/account discovery, and then removes connector env keys plus resolved MCP and inline-HTTP definitions from the parent. Config hot-reload permanently excludes those names, including account-suffixed variants, so it cannot rehydrate them from the env file. If a connector tries to reuse an env name the parent still needs (a provider/webhook secret, Fleet runtime setting, or core process variable), startup fails instead of silently breaking either owner.

Every interactive turn and scheduled task opens a child-owned scope and closes it with a fresh bounded context. Manager discovery, picker metadata, account names, approvals, and scheduled empty-selection/workspace decisions consume only public broker results. MCP hot-reload is also child-owned: the parent sends no resolved definitions and publishes the returned public view atomically for future runs. Connector env values are a boot snapshot; rotate or add one by restarting Fleet. Go cannot guarantee cryptographic zeroization of immutable strings that existed during boot, so “scrub” here means environment removal and dropping/overwriting reachable runtime definitions, not a claim about forensic recovery from old heap pages.

Per-user remote hosted MCP overlays use the same child-owned scope boundary. The parent sends identity and public selection names; the child decrypts or refreshes credentials and builds the short-lived SSRF-guarded client (ADR-0040). Explicit OAuth/connectors HTTP control-plane endpoints remain parent-side and are not model-callable.

Lockdown / network-sealed sandboxes

Lockdown bounds tool execution, not the model call. A lockdown turn hands the agent a no-network (--network=none) per-turn sandbox, so the tool calls cannot reach the network while the model loop continues normally. Scheduled runs default to this sealed posture — see below.

Hypervisor-isolated runtimes (Kata / libkrun, #217)

By default the per-turn container runs under Podman's shared-kernel OCI runtime (crun/runc). A deployment handling untrusted prompts or sensitive data can raise the isolation posture so that every sandbox container is a dedicated KVM VM by setting the bundle manifest's sandbox.runtime (or FLEET_SANDBOX_RUNTIME) to kata or libkrun — an escape then requires a hypervisor CVE, not just a container-escape. The VM boundary has exactly the granularity of the container it replaces: one per interactive turn, one per scheduled run, one per conversation in persistent-REPL mode — every tool call in that turn shares it. fleet emits the value as podman run --runtime=<value>, fail-closed preflights /dev/kvm + the runtime binary at boot (a missing KVM aborts startup rather than degrading to a shared-kernel container), and adds the Kata guest-memory overhead so the --memory cap still reflects usable guest RAM. Everything else — credentials staying host-side, seccomp, dropped caps, network sealing, per-task limits — is unchanged. See SANDBOX-RUNTIMES.md and ADR-0010.

run_python kernel lifetime (per-turn vs persistent, #213)

run_python executes in a long-lived IPython kernel inside the sandbox, so multiple calls within one turn already share state. What FLEET_PYTHON_REPL_MODE controls is whether that kernel survives between turns:

  • per-turn (default): the kernel dies with the per-turn sandbox at turn end. Variables/imports do not carry into the next turn — write to the workspace to persist anything. Unchanged legacy behaviour.
  • persistent: one sandbox + kernel is kept alive per conversation (keyed by conversation ID) and reused across that conversation's turns, so a DataFrame built in turn 1 is still in scope in turn 3 with no re-read. It is never shared across conversations — that is the real isolation invariant (see ADR-0008). Lockdown turns and scheduled runs always stay per-turn. Pass reset_kernel=true to wipe a persistent kernel back to a clean slate mid-conversation.

A persistent sandbox is reclaimed on conversation delete, on process shutdown, and by an idle reaper (FLEET_PYTHON_REPL_IDLE_TTL, default 1800s). FLEET_PYTHON_REPL_MAX (default 32; 0 disables it) bounds how many are kept: each time a persistent sandbox is created, least-recently-used idle sessions are evicted until the count is back under the cap. It is a soft cap — checked only on that create path, and never against a session with a turn in flight — so the live count can exceed it and stay there until the next create or the idle reaper reclaims the excess. Size host memory with that overshoot in mind. Each one carries the same cgroup memory/CPU/PID/disk caps as a per-turn sandbox.

Process shutdown is a hard lifecycle boundary for both modes. Once Pool.Close marks the sandbox pool closed, new per-turn, persistent, lockdown, and allowlisted takes fail with ErrClosed; a cold start already in progress is reaped if it finishes after that boundary. The pool is marked closed before its host-side egress proxy is stopped, so no allowlisted container can be returned with a proxy that was already shut down. This is fail-closed resource lifecycle behavior, not a host execution fallback.

Independent of mode, FLEET_PYTHON_CELL_TIMEOUT (default 0 = disabled) is a host-operator ceiling on a single cell; the effective per-cell timeout is min(the call's timeout_seconds, this).

Inline figures. When the kernel emits an image/png (e.g. plt.show() / display(fig)), the bridge writes it to a figures/ subdir of the conversation workspace under a server-generated filename and returns only the small relative path in the result's image_files. The chat UI renders it inline via the same authenticated workspace-file proxy that serves ![](chart.png) — so the agent needs no plt.savefig(), and the (large) base64 bytes never enter the model's tool result. Bounded to 20 figures / 10 MiB each per cell.

Cost and token ceilings

Each turn runs against configurable per-task cost and token ceilings, an iteration cap, and a per-turn timeout. They are enforced, not advisory: a model that won't stop calling tools is stopped by the ceiling. Usage and cost are accumulated as the agent works and checked against the spec ceilings in-loop, so a runaway loop costs a capped turn, not an open-ended invoice. The observer events persist as a per-turn audit trail answering "what did this agent do, and what did it cost?".

A ceiling stop is a clean abort of the run, but not a completed task. An interactive turn renders it as "budget reached" with the partial transcript; a scheduled run terminates as a failure with class cost_ceiling (#1105) — failure notification, no email reply-back, partial transcript preserved — non-retryable by default (a re-run would just burn the budget again) unless the task's retry policy opts cost_ceiling into retry_on. It previously fell through to success, skipping the finish gates entirely. A sub-agent child stopping at its sliced ceiling is the exception by design: the slice is the parent's leash, so the child returns its partial answer and spend to the parent rather than failing the run.

Budget wind-down (#990). The hard stop gets a soft leading edge, borrowed from Prime Agent's goal budget wind-down: once a run's spend crosses FLEET_BUDGET_WINDDOWN_FRACTION (default 0.8, clamped to (0,1]; the usual prefix aliases apply) of a configured cost or token ceiling, every subsequent provider call carries a request-local BUDGET WIND-DOWN notice — stop starting substantive work, wrap up with progress made, remaining work, blockers, and a concrete next step. Request-local means it is appended at the PrepareStep boundary and never enters the persisted history, the carried round transcript, or compaction input. A one-shot fleet.budget_winddown SSE event marks the crossing. Unlimited (zero) ceilings never wind down, and setting the fraction to 1 disables the notice in practice (the hard ceiling fires first). The point: a run that would otherwise be cut off mid-thought at the ceiling gets a budgeted chance to finish cleanly and report a useful partial result.

Auxiliary model calls are metered too (#1118). Model calls fleet makes on a run's behalf but outside the main step loop follow one rule — visible or counted, never invisible: the compaction summarizer and the model-invocable suggest_* git-metadata tools meter into the same run accounting the ceilings read (the summarizer additionally pre-checks the ceiling and degrades to a deterministic truncation summary once it is met), while the host-side extras (end-of-run verifier, phone-a-friend review, loop exit-condition verifier) stay off-ceiling by their documented semantics but record labeled aux_usage entries in the session log. See docs/AUX-MODEL-CALL-METERING.md.

Scheduled runs additionally carry a per-task wall-clock timeout (#724): FLEET_TASK_WALL_TIMEOUT (a Go duration; default 4h, 0 disables) bounds one run's total elapsed time, enforced by the worker pool around the run invocation. The iteration cap and cost/token ceilings bound loop progress, but a single hung tool call inside an iteration observes neither — the wall-clock deadline cancels the run's context so the pool slot is always reclaimed. Expiry is a deterministic terminal failure with a clear timeout error: it never consumes the task's transient-retry budget (a run that hit the ceiling once would hit it again). It is a global operator knob, not a per-task column — per-task expectations stay in expected_duration_minutes (SLA monitoring, advisory) and a loop task's time_budget_seconds.

Approval timeouts: default-deny on no answer (#225)

High-risk tool calls (outbound email, risky shell commands, the advanced-model nudge) are staged for human approval rather than executed directly. A staged approval carries an expires_at deadline; if no one answers in time it is auto-denied — a default-DENY-on-timeout contract so a card a user walks away from never silently lingers as an executable action. A background sweep (every 30s) flips expired pending approvals to rejected and writes the outcome into the conversation so the next turn knows the action was not taken. A human who clicks Send in the brief grace window before the sweep still wins the race (the atomic claim decides), so a late decision is honored rather than lost. A click that lands after the deadline resolves the row as timed out right then (the same claim primitive the sweep uses), so the card settles deterministically instead of echoing a still-pending state back at the user. A timed-out card offers a one-click Ask again, which submits a user turn asking the agent to re-stage the action.

Two exceptions by design: preview_email stages with no deadline — the card is display-only (Dismiss is its only action), so there is nothing for default-deny to protect against — and notify-mode records (#1153, below) are created already resolved.

The wait window resolves highest-priority-first:

  1. Per-tool — agent_policy.critical_tool_timeouts in the client bundle manifest (keyed by the same bare tool-name suffix as critical_tools).
  2. Per-conversation — POST /conversations/{id}/approval-timeout with {"approval_timeout_seconds": N} (or null to clear).
  3. Admin override — the approval_timeout_seconds row in Settings → Admin → Features (60–86400s, applies live to the next staged card; ADMIN-SETTINGS.md).
  4. Global — FLEET_APPROVAL_TIMEOUT_SECONDS (default 3600). A non-positive value is treated as "use the 3600s default", never as "deny instantly". The default was 300s until the cards rework: the card lands whenever the agent reaches it — often minutes into a run the user started and then reasonably stopped watching — so five minutes mostly denied the final, wanted action of a long run (the same observation that motivated notify mode).

Per-tool approval MODE (#1153)

The window above answers "how long do we wait". It does not answer "should we have asked at all", and one global critical_tools list forces one policy onto operations with very different undo stories. A card lands whenever the agent reaches it — often many minutes into a run the user started and then reasonably stopped watching — so the thing the 300s default most reliably blocked was the final, wanted action of a long analysis.

agent_policy.critical_tool_modes sets the mode per tool suffix:

Mode Behavior
approve (default) Stage a card and block until a human decides. What every critical tool did before modes existed.
notify Execute immediately and post a card recording what happened. No countdown, nothing to miss. 
agent_policy:
  critical_tools: [deploy_page, create_deal]
  critical_tool_modes:
    deploy_page: notify        # the store keeps immutable versions
    create_deal: approve       # audit-gated; unchanged
  critical_tool_undo_hints:
    deploy_page: "Undo with mcp_pages_rollback_page(slug, version_id)."

Three properties keep this narrow:

  • Opt-in per suffix. A tool the bundle does not name keeps blocking, so this changes nothing for a deployment that ignores it.
  • Outbound email can never be notify. The entire case for running without asking is "we can always roll it back", and a sent message has no undo — the card is the review step. A manifest that declares it is logged and pinned back to approve.
  • Recording is load-bearing. If the transport cannot post the record card, or fails to, the call falls back to a blocking approval. Executing unrecorded would remove the only thing that makes notify defensible.

critical_tool_undo_hints is bundle-authored on purpose: fleet does not know any client's reversal verb and must not invent one. "We can always roll back" is only true in practice if the card says how.

The record survives a page reload: resolved approvals (records included) are re-hydrated into the transcript by the conversation GET and anchored to the message holding their tool call, so the user who was away — notify mode's entire audience — still finds the "ran without asking" card and its undo hint when they come back, not just a raw tool chip. See APPROVAL-CARDS.md for the card UX as a whole (the generic critical-action card, honest per-tool copy, timed-out recovery).

FLEET_AUTO_APPROVE_IN_TEST (default false) is a CI/test escape hatch that auto-approves every staged critical tool instead of waiting for a human. It bypasses the human-in-the-loop gate and is intended only for pipelines with no human present and a mocked backend — never enable it in production. fleet logs a loud warning at startup when it is on.

Context-window pressure (proactive compaction)

Before each round's model call the run loop compares the prompt size against the active model's context window (resolved by contextWindowForModel — observed provider ground truth, the live OpenRouter cache, then a static table) and acts before the provider rejects an oversized request, rather than only recovering after a context_length_exceeded error:

Env var Default Behavior at/above the fraction
FLEET_CONTEXT_PRESSURE_WARN_THRESHOLD 0.75 Emit a fleet.context_pressure SSE event (the chat UI shows a non-blocking "conversation is N% full" banner).
FLEET_CONTEXT_COMPACTION_THRESHOLD 0.90 Proactively summarize the oldest half of the history (pinned head + recent half kept verbatim) and emit fleet.context_compacted.

Both honor the usual CHAT_/CUTLASS_ prefix aliases, and a value outside (0,1] falls back to its default. The size signal is the per-call input size (LastStepPromptTokens), never the cumulative token total, so a long run does not ratchet the trigger into a compaction spiral; on a turn's first round — before any per-call size is known — it falls back to a char-heuristic estimate of the carried-over history so a single-round turn that starts near the limit is still covered.

Scheduled safeguard. Unattended runs must not silently rewrite their own transcript: in ModeScheduled the warn event still fires (and a breadcrumb is written to the session log), but proactive compaction is off unless the operator sets FLEET_SCHEDULED_AUTO_COMPACT=1. The summary uses the driver's compactionSummarizer (an LLM summary) when wired, else a deterministic placeholder — the same hook the reactive context_length_exceeded recovery path already uses, so a proactive compaction does not count toward the consecutive- compaction cap that guards against compaction loops.

The summary call is governed (#1118). The summarizer fires exactly when a run is already large, so its own model call meters into the run's usage/cost accounting (the same counters the ceilings and the chat cost chip read), and it pre-checks the run's cost/token ceiling first: at or over budget it skips the model entirely and inserts the deterministic placeholder — truncation instead of an unmetered paid summary.

The summary is structured and iterative (#990, borrowed from Prime Agent's compaction). The interactive summarizer prompt demands a fixed section skeleton — Goal / Constraints & Preferences / Progress (Done · In Progress · Blocked) / Key Decisions / Next Steps / Critical Context — instead of freeform prose, and the Critical Context section explicitly records file paths and variable names because the sandbox workspace and any persistent Python session survive the summarization that hides the messages that created them. On a repeat compaction (the droppable middle contains a previous [context compaction summary) the prompt switches to an update variant: treat the previous summary as the baseline, preserve everything the newer messages do not supersede, move Progress items forward, keep exact paths and error messages — so early facts stop eroding a little further on every compaction round.

The task plan is re-announced after compaction (#990). The task_tracker plan is host-side state — it survives the compaction that just rewrote the history, and checkFinishEnforcement keeps enforcing it — but the summary may not have preserved it. Both compaction paths (proactive and the reactive context_length_exceeded recovery) therefore insert a bounded [plan state after compaction] message right after the summary whenever the tracker has open items, so the model continues from the plan it is being held to instead of having to rediscover it via task_tracker view. Repeated compactions keep at most one live copy (a stale copy in the kept slice is dropped when the fresh one is inserted); a completed plan is not re-announced.

The one server-name keying rule (Optional opt-in ↔ prompt roster, #1272)

Every server-keyed MCP gate is keyed by manifest (spec) server name, but the name a server actually registers under is <server>_<account> for a named-account seat (the _ variant convention), and the system prompt sees a flattened mcp_<server>_<tool> roster name. One rule bridges all three, and every layer resolves through the same helper (agentcore.longestServerKey, surfaced as OptionalServerFor / OptionalServerForToolName):

A key K governs a name N iff N == K, or N begins with K_. When several keys qualify, the longest one wins.

Consequences worth stating plainly:

  • A variant seat is opt-in gated by its own key when the bundle declares one, and by its base server's key when it does not. So with only jira marked optional: true, the jira_prod seat is withheld until jira is opted in — it is not a silently-always-on server.
  • The rule is applied identically at registration and in the prompt. Gate-1 (buildFantasyTools) and the system-prompt roster filter (Manager.activeMCPToolNames) call the same helper, so a tool can no longer be registered-and-callable while hidden from the model, or advertised while absent from the roster. Before #1272, Gate-1 did an exact map lookup on the registered name: a base-only Optional key missed the variant seat entirely, and the seat registered unconditionally while the prompt (which has always prefix-matched) hid it. Gate-1 now fails closed.
  • Longest-wins is what keeps it deterministic, which is why it is also a prompt-cache concern: the roster feeds the cacheable prefix, and a map-iteration-order winner would silently bust it (see PROMPT-CACHE-CONTRACT.md and #1125).
  • Gate-2's per-server tool allowlist (mcpAllowlist.toolsFor) resolves through the same helper, so a variant seat is filtered by its manifest server's allowlist exactly like the default seat.

The whole-name (N == K) branch applies only to registered server names — a registered name can legitimately be a declared server. It is deliberately off for roster names, whose trailing _<tool> segment means a whole-name hit would be a server+tool coincidence: mcp_jira_search is server jira's search tool, never a server named jira_search (that server's roster names all carry a further _<tool>).

Per-task credential allowlist (least-privilege MCP)

A scheduled task's MCP selection (mcp_selection) controls which servers it sees; the credential allowlist (#184) additionally scopes which (server, account) credential pairs it may call — Gate-3, after the server opt-in (Gate-1) and per-server tool allowlist (Gate-2):

  • credential_allowlist: null (the default) → inherit global: any server in mcp_selection is permitted (unchanged behaviour).
  • credential_allowlist: [] → deny all MCP calls.
  • credential_allowlist: [{"server":"github","account":"client-a"}, {"server":"sendgrid"}] → only those pairs. A {"server":"sendgrid"} entry (no account) matches only the default seat; a named account must be enumerated explicitly.

A call to a non-permitted pair is denied before it executes: the tool is advertised but every invocation returns a governance message to the model (a tool result, not a transport error) and records an audit entry (credential_allowlist_denied). The allowlist stores pair names only — credential values never enter the database (they live in the host env file; see internal/creds).

Set or clear it with the admin CLI (the task must be pending/scheduled):

fleet sched task set-credentials <task_id> --allow github:client-a --allow sendgrid
fleet sched task set-credentials <task_id> --clear   # revert to global inherit

Remote MCP transport hardening (TLS pinning + mTLS, #280)

A manifest HTTP MCP server may opt into per-server TLS hardening under a tls: block — pin the trusted CA, present a client certificate (mTLS), and/or pin the server's public key. It defends a sensitive connection (internal API, credential store) against CA-substitution MITM, beyond default system-root verification:

mcp_servers:
  - name: internal_api
    type: http
    url: https://mcp.internal.example
    tls:
      ca_cert: /etc/fleet/mcp/internal-ca.pem      # pin trust to this CA bundle
      client_cert: /etc/fleet/mcp/fleet-client.pem # mTLS client cert (+ key)
      client_key:  /etc/fleet/mcp/fleet-client.key
      pinned_sha256: 9b8e…                          # hex SHA-256 of the server SPKI
      server_name: mcp.internal                     # SNI / verified-name override

The public-key pin is checked in addition to normal chain verification, so a substituted certificate is rejected even if it chains to an otherwise-trusted CA. fleet never disables verification — a self-signed server is reached by supplying its certificate as ca_cert. Compute a pin with:

openssl x509 -in server.pem -pubkey -noout |
  openssl pkey -pubin -outform der | openssl dgst -sha256

Every field is optional; omitting the block keeps default system TLS. See ADR-0015. (Per-user OAuth remote servers below are not covered — they dial through the SSRF-safe client.)

Per-user remote MCP servers (OAuth login, #443)

The bundle's MCP servers are operator-provisioned. fleet also lets each user add a remote (hosted) MCP server from the GUI and log in to it via the MCP OAuth handshake (spec revision 2025-06-18: OAuth 2.1 + PKCE S256, RFC 9728/8414 discovery, RFC 7591 dynamic client registration, RFC 8707 resource indicators). The connected server's tools then participate in that user's chat turns and their scheduled tasks. In chat they appear in the Tools picker as toggleable, default-on entries (gated per conversation exactly like a bundle Optional server, so they count against the tool ceiling only when selected); scheduled runs use all of the owner's connected servers. Local stdio servers are unchanged. See ADR-0009 for the OAuth rationale and ADR-0040 for the production process boundary.

The feature is off until configured and fails closed:

  • FLEET_MCP_OAUTH_ENCRYPTION_KEY — base64 of 32 random bytes (openssl rand -base64 32). Encrypts the per-user tokens / client secret at rest (AES-256-GCM, AAD-bound to (email, canonical server URL)). Unset → the feature is disabled and the endpoints report it.
  • FLEET_PUBLIC_BASE_URL — the externally-reachable web origin (e.g. https://fleet.example.com). The OAuth redirect URI is derived from it (<base>/api/oauth/mcp/callback) and must be byte-stable; it is never reconstructed from request headers. Required. scripts/bootstrap.sh --enable-web --domain fleet.example.com writes this value and the encryption key into the backend environment automatically; fleet update reconciles existing installs from the web tier's persisted public-origin stamp before restarting the backend.
  • FLEET_REMOTE_MCP_ALLOW_INSECURE_HTTP — dev only; permits http:// servers. Default false (https required).

How the invariants hold:

  • Credentials stay host-side (ADR-0003). Tokens live only in the fleet process + chat Postgres (encrypted) and reach a server only as the Authorization header the host-side MCP client writes — never the sandbox, model context, or logs.
  • No forked governance (ADR-0001). A user's servers are wired as a per-run overlay mcp.Client (built with a freshly-refreshed bearer) composed with the shared/bundle client via a compositeBroker. The shared long-lived client is never mutated with per-user secrets, so concurrent users can't cross-pollute. Chat and scheduled use the same overlay + refresh path.
  • One identity, two URLs. The connection URL is the canonical form of what the user typed (scheme and host lowercased, default port and fragment dropped, the path kept exactly as typed — a percent-escaped segment such as /tenant%2Fone/mcp included, since decoding it would name a different route) — the MCP endpoint fleet dials, the DB key, the encryption AAD and the broker routing name. The RFC 8707 resource indicator the authorize, exchange and refresh requests carry is what the server's protected-resource metadata declared, when it shares that origin (remote_mcp_servers.resource, '' = same as the URL). They usually coincide. Slack declares its bare origin while serving MCP at /mcp, and its origin root answers with a redirect the SSRF client refuses — so adopting the declared value as the connection URL made every Slack mount fail (#1006).
  • SSRF guard. User-supplied URLs are dialed through a client that rejects private/loopback/link-local/metadata IPs at connect time (DNS-rebinding safe) and refuses redirects.
  • Authorization-server metadata is looked for where both specs put it. An issuer with a path component (https://access.stripe.com/mcp, https://mcp.datadoghq.com/v1/mcp, GitHub's /login/oauth) may publish its document at RFC 8414 §3.1's inserted location (/.well-known/oauth-authorization-server/mcp) or at OpenID Connect Discovery's appended one (/mcp/.well-known/openid-configuration). mcpoauth.authServerMetadataCandidates tries, in the MCP spec's order, RFC 8414 inserted, OIDC inserted, OIDC appended, then the appended RFC 8414 form fleet historically asked for; a failed Add names every location it tried. Before this, only the appended forms were tried and thirteen official catalog vendors could not be added at all (#1006 audit).
  • Protected-resource metadata is asked for the way a client would, and a server without any still connects. Discovery probes the MCP URL with a GET and then with an unauthenticated JSON-RPC initialize POST, taking the resource_metadata pointer from whichever 401 carries one (Uptime Robot answers the GET with 404 and points only from the POST). Without a pointer it tries RFC 9728 §3.1's path-inserted well-known form, then the path-appended form, then the origin root — all built from the URL's path alone, never its query. When no document exists at any of those (every location answers 404 or 410), or the document names no authorization server (authorization_servers is optional in RFC 9728; its other fields — scopes, resource — are kept), the MCP spec's backwards-compatibility rule applies: the server's own origin is the authorization server, its RFC 8414 / OIDC document is fetched there, and the typed URL is the resource (Discovered.LegacyOrigin). A location the server itself advertised on the 401 and then could not serve, a well-known location that answered 5xx, timed out or returned malformed JSON, a probe that got no answer at all, or a document that names no authorization server and lacks a resource fleet can canonicalize (absolute, http(s), a host, no userinfo — the same bar every stored resource passes, deliberately looser than RFC 9728's https-only, fragment-free rule so development servers work), is an error, never a fallback — that is a modern server failing or misbehaving, not a legacy one. (A document that does name an authorization server is handled as before, whatever its resource says.) The probe reads only status and headers, so an event stream a server holds open is closed, not drained. The probe's initialize announces the same protocol revision as fleet's real transport (pinned by a test), and a session it happens to open is terminated before discovery moves on. Intercom, Plaid, Cartesia, GoCardless and Square publish only that shape and could not be added before (#1006 catalog audit).
  • A document that names another issuer is accepted only when every endpoint in it is vouched for. Five official vendors (DocuSign, ZoomInfo, Sprout Social, OVHcloud, Chargebee) name the MCP host as the authorization server and serve, from that host, a document whose issuer is another URL — a copy of the real server's metadata, a proxy in front of Okta, or a hybrid — so RFC 8414's issuer check refused all five. mcpoauth.confirmProxiedIssuer runs only as a last resort, after every metadata location failed the strict check, and accepts the document when each endpoint it names (authorization, token, registration, revocation) is either confirmed equal by the claimed issuer's own metadata or, when the resource named a bare host, on that host itself. Endpoint URLs compare the way URLs do — scheme and host case-insensitively, path and query byte-for-byte with one documented exception: a single trailing slash is tolerated, so /token and /token/ are the same endpoint while /token// is not — so a "confirmed" endpoint cannot differ from the vouched-for one in casing alone. An endpoint must also be an absolute http(s) URL with a hostname: a relative one parses without error and normalizes to an empty origin, so two of them would confirm each other. An endpoint embedding userinfo is refused outright — net/http turns URL userinfo into a Basic Authorization header when the request sets none itself, so confirming one would dial it with authentication fleet never chose to send. An authorization server scoped to one tenant — by a path (https://as.example.com/tenantA), a query, a fragment or userinfo — gets neither leg on another tenant's terms: the only document that may vouch for it is its own ORIGIN-level one, the measured Chargebee shape — and only for a PATH-scoped one: the well-known lookup keeps just scheme, host and path, so a query-, fragment- or userinfo-scoped server would have its scoping dropped before any fetch, let an origin document self-confirm, and be silently replaced by the unscoped issuer; those get no fallback at all. A sibling tenant is self-consistent too, and accepting it would send the user through the wrong tenant's authorization endpoint. "Scoped" is read off the escaped path, so an issuer path of /%2F — which url.Parse decodes to // — stays a tenant rather than reading as a bare origin. Origins compare canonically throughout (lowercase scheme and host, the scheme's default port dropped, an IPv6 literal's brackets kept so the host/port boundary stays unambiguous), so a PRM or a copy spelling a host in mixed case or with an explicit :443 still matches the endpoints it vouches for; paths and queries compare byte-for-byte, but for that one tolerated trailing slash. When the token endpoint turns out to be the claimed issuer's own, that issuer's document supplies token_endpoint_auth_methods_supported — a copy saying none against an endpoint whose owner requires a secret would otherwise open a secretless client — and fills in scopes_supported and mfa_challenge_endpoint where a trimmed copy left them out, since losing either costs the connection the refresh token the offline_access rule below exists to secure. A populated list is never overwritten (a proxy may offer fewer scopes than the issuer behind it), and a proxy's token endpoint is its own, so there the document's own values stand. The validated document's endpoints are what fleet dials (a proxy's registered clients only work with the proxy's endpoints), and its Issuer is recorded as the identity the vendor asserts. An endpoint belonging to neither party — a copy that borrows a real issuer's name but points the token endpoint elsewhere — is still refused (#1006 audit).
  • Dynamic registration asks to be a public client, and retries once as a confidential one if refused. Of the official servers whose metadata lists no none method, the two met live registered fleet anyway — one returning a secret fleet stores and uses. A server that instead answers RFC 7591's invalid_client_metadata gets one retry with client_secret_basic (or client_secret_post) if it lists one — and with client_secret_basic when it advertises no list at all, which RFC 8414 §2 defines to mean exactly that, the same default the token endpoint already applies. The retry must come back with a client_secret: a confidential registration without one cannot authenticate at the token endpoint, so it is refused at add time rather than after a consent screen. RFC 7591 §3.2.1 also lets the server substitute the metadata it granted, so when the response names an effective token_endpoint_auth_method that is what fleet stores for the client — a server advertising both Basic and Post may register this client as post-only, and choosing Basic off the advertised list would 401 every exchange and revocation. A none echo does not narrow it, since fleet asks to be public first and a server may echo none while returning a secret anyway, and a granted method fleet cannot perform at all (private_key_jwt, an mTLS method) is refused at registration rather than falling back to the advertised list and failing the exchange after consent. The secret comes back encrypted at rest like any other.
  • Auth0 is asked for offline_access, like Entra. An Auth0 tenant (recognized by its proprietary mfa_challenge_endpoint, or an *.auth0.com issuer) issues a refresh token only for that scope, which the resource's own scope list omits (Checkly). Discovered.RequestedScopes appends it when the server advertises it. Not generalized to "advertises offline_access": GitHub advertises it and refreshes without it.
  • Rotation-safe refresh. Tokens are refreshed under a SELECT … FOR UPDATE row lock with a post-lock expiry re-check, persisting any rotated (single-use) refresh token in the same transaction. A dead refresh token marks the connection needs_reauth and the server is skipped — the run still completes. So does an access token that expires with no refresh token to renew it: the row used to stay connected while every turn skipped the server as "token unavailable" (#1006).
  • Google is asked for a refresh token; nobody else needs asking. Most authorization servers issue a refresh token by default. Google issues one only when the authorize request carries access_type=offline, and only on the first consent for a (user, client) pair unless prompt=consent forces the consent screen — without both, a Google Workspace connector lived one hour and had to be reconnected by hand. mcpoauth.FlowConfig.AuthCodeURL adds both when the discovered issuer is accounts.google.com, and to no one else: prompt is an OpenID Connect parameter whose values another server may validate. A vendor with its own refresh-token contract gets its own clause there, keyed on the issuer.
  • Microsoft Entra ID: a templated issuer, and offline_access. Entra's multi-tenant metadata (login.microsoftonline.com/{common,organizations, consumers}/v2.0) says issuer: https://login.microsoftonline.com/{tenantid}/v2.0 — the literal template — because the tenant is only known after sign-in. The mix-up check (mcpoauth.issuerMatches) accepts exactly that shape: same https Entra host, same path, the template standing in for one of the three multi-tenant aliases. A tenant GUID in the PRM's issuer, another host, or the template anywhere else stays a mismatch, and the stored issuer is the PRM's real URL, never the template. Entra also issues a refresh token only when offline_access is requested, so Discovered.RequestedScopes appends it for Entra issuers that advertise it (the Microsoft-hosted Azure DevOps MCP server declares only its .default scope). Measured for #1006: without the first fix Azure DevOps could not be added at all; without the second it would have lived one hour.
  • A refused mount is recorded, not just skipped. When the vendor's server answers the per-turn connect with HTTP 401 — a revoked grant, GitHub's "Revoke all user tokens", a key rotated on the vendor side — the connection is marked needs_reauth with the status in its detail, so Settings → Connections shows "Reconnect needed" immediately instead of after the token's natural expiry (eight hours, measured for #1006). Only a 401 marks: a 5xx, a timeout or a TLS failure says nothing about the credential. The transport reports such answers as the status plus the body's first line (mcp.HTTPStatusError) rather than a JSON decode error — unless the body is itself a JSON-RPC response, which wins at any size the 2xx path would accept (Google's Drive MCP server answers tools/list with HTTP 403 and a complete result when the project has not enabled drivemcp.googleapis.com; the mount succeeds, one log line notes the disagreement, and the tool calls that follow carry Google's real "the caller does not have permission").
  • Refresh failures are classified, not lumped together. The split decides whether the user is asked to do something, so it errs toward not bothering them (mcpoauth.IsTerminalRefreshError):
    • Terminal — invalid_grant (refresh token revoked/expired/already rotated), invalid_client (the AS no longer recognizes our client credentials), unauthorized_client (this client may not use the refresh grant) — plus GitHub's own spellings, bad_refresh_token and incorrect_client_credentials, which GitHub returns with HTTP 200; the token-endpoint parser reads an error member on a 2xx body too, so those classify instead of surfacing as an opaque "no access_token" that was retried every turn. Re-issuing the same request can never succeed, so the connection is marked needs_reauth with a reason naming the actual cause (mcpoauth.ReauthDetail, rendered in Settings → Connections) and the user reconnects — which re-runs DCR through the normal connect flow.
    • Recoverable in place — invalid_target retries without the RFC 8707 resource parameter; invalid_scope retries once without scope. fleet stores the scopes it requested at connect time, and an AS may grant a narrower set; RFC 6749 §6 forbids refreshing a wider scope than was granted and defines an omitted scope as "identical to the scope originally granted". Without that retry a narrowed grant wedges the connection forever.
    • Transient — network failures and 5xx. The transaction rolls back and the next call retries; the connection is left connected, so a blip never costs the user a manual reconnect.

Scheduled tasks resolve the task owner's email (the orchestrator username) to look up that user's connected servers, so a headless run reaches them with no user present — as long as a valid refresh token exists. A headless run can't re-prompt the user to log in, so a needs-reauth server is skipped AND surfaced to the owner: a notice naming the unavailable connectors is prepended to the run (visible in the task transcript) so the agent doesn't silently rely on missing tools and the owner knows to reconnect.

Tool-dispatch panic containment (#795)

fleet is a single-host process running interactive chat, scheduled tasks, sandboxes, and workers together, and fantasy executes streamed tool calls in unsupervised goroutines (a coordinator goroutine plus per-tool goroutines for parallel calls) with no recover. So a panic in ANY tool — native, loader, direct-MCP, or deferred-MCP — or in a policy gate, an output guardrail, or an Observer callback would escape and terminate the whole process; internal/safe's per-goroutine recover in the runner/httpapi callers is goroutine-local and cannot catch it.

Every tool fleet hands fantasy is wrapped in an outermost panic-containment wrapper (panic_containment.go). A panic becomes exactly one in-band tool error result — err == nil, so fantasy pairs one result to the call id and the round continues instead of aborting the stream. Logs, Sentry, and panic_events receive only an opaque incident id, a value-free panic class, and non-content tool/run attribution; the recovered value and stack are discarded before telemetry. The model sees only the incident id and a possibly executed marker, and ADR-0035 blocks in-round provider re-drive once a tool ran.

Invocation-local phase state attributes policy/output failures and marks a RecordToolResult attempt before calling it. Before-call, execution, and output panics therefore receive one failed logical-tool policy record (including deferred MCP), while a panic in the record hook itself is never retried. The run wraps its Observer once, disables it after its first panic, and returns an ordinary opaque run error only after Fantasy's tool goroutines settle. See ADR-0037 for the complete boundary.

Per-persona tool allowlist (least-privilege by role)

Different personas have different roles and risk surfaces. A code-reviewer persona that can send email, or an executive-assistant that can run arbitrary shell commands, violates least privilege. The per-persona tool allowlist (#294) lets the bundle manifest declare, per persona, which tools that persona may see — Gate-4, layered on top of the server opt-in (Gate-1), the per-server tool allowlist (Gate-2), and the per-task credential allowlist (Gate-3).

Declare it in the manifest's personas: block (the persona name matches the basename of its personas/<name>.yaml file):

personas:
  - name: code-reviewer
    tool_permissions:
      allow:
        - bash
        - run_python
        - mcp:filesystem/*
      deny:
        - mcp:email/*
        - send_email
  - name: executive-assistant
    tool_permissions:
      deny:
        - bash
        - run_python

Pattern syntax (matched against the fantasy tool name — a native name like bash or the mcp_<server>_<tool> form discovered MCP tools register under):

Pattern Matches
bash the native tool named bash, exactly
mcp:server/tool one MCP tool (→ mcp_<server>_<tool>)
mcp:server/* every tool from one MCP server
prefix/* any tool whose fantasy name has that prefix
* every tool

Resolution rules:

  • No tool_permissions block (or both lists empty) → no narrowing; the persona sees every tool the earlier gates already permit (backward compatible — the generic bundle ships no personas: block, so behaviour is unchanged).
  • allow non-empty → default-deny: only matching tools are offered.
  • only deny → default-allow: every tool except matching ones is offered.
  • Deny takes precedence when a tool matches both lists.

This gate can only NARROW, never widen. The filter runs over the tool list that already survived Gates 1–3, so a tool a persona's allow names but that the server or credential gates already dropped never reappears — the allowlist subtracts from, but can never add to, what the run was already permitted to offer. Enforcement is at tool registration, before the first LLM call: a suppressed tool never enters the model's tool list (a tool the model cannot see cannot be hallucinated into a call). Each suppressed tool emits a persona_tool_blocked{persona, tool, reason} observer event for the audit trail. Credentials are unaffected — they stay host-side, brokered out-of-process; this gate only decides which tool schemas are advertised.

Scheduled execution: sealed by default

A scheduled task's bash / run_python execution sandbox runs with no outbound network egress (--network=none) by default — the same seal the interactive lockdown path applies. Unattended runs have no human on the loop, so the safe posture is the default: a scheduled task cannot fetch arbitrary URLs, pip install, reach host-local services, or exfiltrate unless you opt it in.

To let a specific task's sandbox reach the network, set allow_network: true on the task (the Allow network egress toggle in the task-create form, or the allow_network field on POST /tasks). The default is false (sealed); the opt-in is per-task, so one task needing egress does not open up the rest. This governs only the execution sandbox's --network; it never affects credential brokering, which always stays host-side.

Captain's Log: persistent task memory (#198, #285)

By default every run of a scheduled task starts cold — it has no knowledge of what prior runs observed or decided. A task can opt into persistent, task-scoped memory by setting instruction_self_improve: true (the Captain's Log toggle in the task-create form). When set, that task's scheduled runs get two extra native tools:

  • remember(key, value) — upsert a fact for this task. Committed immediately; scheduled runs are unattended, so there is no human-approval step.
  • recall(key?) — read one fact, or all of them as a JSON object.

At the start of every run, all of the task's stored facts are injected into the system prompt under a Your Persistent Memory section, so the agent sees prior state without having to call recall. This lets a recurring task track state across time — "alert only if the price changed since last week", "skip anomalies already triaged", "accumulate a running digest".

The store is bounded so a long-lived task cannot grow unbounded: FLEET_TASK_MEMORY_MAX_KEYS (default 100, oldest key evicted LRU-style on overflow) and FLEET_TASK_MEMORY_MAX_VALUE_BYTES (default 4096, a hard reject). Memories live in the scheduler database (task_memories, keyed by (task_id, key)), not the client-config bundle — this is runtime state, so it never touches the operator-owned, git-versioned bundle, and the reproducibility guarantee ("the setup that worked is the setup that runs again") is preserved. Inspect or clear a task's memory with fleet task memories list|clear|delete.

Off by default (the column is BOOLEAN NOT NULL DEFAULT FALSE), so a task that does not opt in behaves exactly as before — no extra tools, no injection.

Prompt/knowledge self-improvement is a separate, already-shipped path: the agent proposes edits to the admin-curated knowledge base via propose_note, an admin publishes or rejects them, and published notes are injected into every run's prompt. Agent-authored client-bundle skills are intentionally not part of this — skills stay operator-authored so the bundle remains a reproducible artifact; nothing fleet does ever writes the bundle or commits to git.

Dead-letter queue (#253)

A scheduled task retries on transient failures up to its max_retries budget (the backoff curve + which failure classes retry come from the per-task retry_policy, #201). Once that contract reaches a terminal failure — either a transient failure with the retry budget exhausted, or a non-retryable (deterministic) failure — the runner routes the task to a distinct dead_lettered terminal status instead of bare error, so the exhausted task is reviewable and replayable rather than silently failing. The row records when it was quarantined (dead_lettered_at), the final attempt's failure message (dead_letter_reason), and the total attempts made (dead_letter_attempts). The runner is the only writer of this status; a self-reporting worker cannot set it.

error is preserved for the non-final failure cases it always covered — per-attempt failures that will retry, an interrupted run (shutdown grace expired), and a panic during execution.

Review and replay are operator actions on the box, via the admin CLI:

fleet sched dlq list [--tag <tag>] [--limit N] [--offset N] [--json]
fleet sched dlq replay <task_id>   # reset to pending; the scheduler re-runs it

replay resets the same task to a fresh pending slate (attempt_count = 0, the dead-letter columns cleared) and the normal claim path re-runs it. Entry into the DLQ also increments the fleet_dead_letter_queued_total{reason} counter (reason is the bounded class retry_exhausted or non_retryable — deliberately not a per-task label, to avoid unbounded metric cardinality). Dead-lettered tasks are not subject to the automatic retention sweep — quarantine is for review, so a DLQ task persists until it is replayed (or explicitly removed).

Published output artifacts (#204)

A scheduled run's agent can mark files it produced in the workspace as named output artifacts via the publish_artifact tool — a curated manifest of deliverables (the report, the processed dataset, the rendered document), distinct from the raw per-run workspace the file-browser endpoints already expose. The agent writes a file, then publishes its workspace-relative path with an optional description; the tool validates the path stays inside the workspace (no traversal / symlink escape) and names an existing regular file, then records {name, path, description, size}. It never reads, copies, or moves the bytes — the file stays in the workspace.

The tool is scheduled-only (assembled beside create_task / the metadata tools) and ungated: it can only record files in the run's own workspace, which the operator can already browse, so it grants no new access. A per-run cap bounds the manifest; re-publishing a path updates it in place.

The manifest is persisted on the run's success path, under the held lease, just before the terminal transition (riding a running-status update like the structured-output capture, #244) into a nullable artifacts JSONB column. The column is deliberately excluded from the task upsert, so a later status update cannot clobber it. GET /tasks/{id}/artifacts returns the manifest (404 when the run published none; 409 while non-terminal); each entry's path is downloadable via the existing workspace file endpoint. Because the manifest indexes the creator-private workspace (#287), the endpoint is gated to the task's creator or an admin — the same ownership check as the workspace file endpoints, not the looser task-visibility used by /output. A re-run (lease recovery) clears the prior attempt's manifest, so a task only ever serves the artifacts of its latest, successful attempt.

Conversation memory auto-indexing (#234)

fleet's long-term memory normally grows only when the agent calls propose_memory or a user POSTs a memory. With FLEET_MEMORY_AUTOINDEX_ENABLED=true (default false), each completed interactive turn is additionally mined for durable, reusable facts — stable preferences, environment/config facts, standing instructions — by a short-lived cheap-model call (FLEET_MEMORY_MODEL, defaulting to the metadata/title-model chain), the same host-side pattern as auto-titling.

It never writes memory live. Extracted facts are surfaced as memory proposals through the exact seam the propose_memory tool uses — a memory.proposed SSE card the user Saves or dismisses — so the human stays on the loop. Extraction runs on the already-detached post-turn goroutine (its own LLM-call budget, errors swallowed) and is deduped against the user's existing memories and the conversation's still-pending proposals, so a fact stated across several turns is proposed once, not on every turn. A bounded number of facts per turn caps the batch. Off by default, so the memory-write paths are byte-for-byte unchanged unless an operator opts in.

The scheduled end-of-run verifier

Scheduled runs layer an extra host-side LLM re-check on top of the shared audit/finish enforcement. When the scheduled policy clears a run, the runEndOfRunVerifier runs on fleet's fallback model (host-side creds — the verifier's model call is just another host LLM call) and returns any missing required actions, which the loop turns into a final enforcement round before it is allowed to finish. A verifier error fails open (allow finish). So core governance — per-tool policy, audit, finish enforcement, MCP credential brokering, note staging, usage/cost, and the end-of-run verifier — applies to every scheduled run. An explicit terminal audit abort skips the extra model reviewers and remains a failed result. Conditional task branches are verified using bounded structured result evidence, not tool names alone; see Conditional scheduled tasks.

The verifier's own spend does not debit the run's cost/token ceilings (it is a host-side extra around the loop), but it is recorded per call in the session log's labeled aux_usage ledger (#1118) — see docs/AUX-MODEL-CALL-METERING.md.

Round-cap exhaustion keeps its partial transcript (#1125, #1271)

The enforcement loop is bounded: if the finish gates (audit → verifier → phone-a-friend) never clear within 20 enforcement rounds, agentcore.Run gives up and returns a hard error wrapping ErrMaxEnforcementRounds (message unchanged: max enforcement rounds (20) exceeded without task completion). Scheduled is the only mode that can reach the cap — the interactive policy can finish at round 1.

Those rounds were paid for, so the failure carries real work back: Run returns the accumulated Result — transcript entries, rounds, FinalText and usage — alongside the error (#1125), and the scheduled driver now persists it instead of discarding it on the way out (#1271). The session log ends up with a [truncated] notice naming the rounds burned and the spend, then the partial assistant text, both stamped message_type: round_cap_truncated, then the usual [fatal] line. The tool calls, enforcement nudges and token/cost counters were already written live (by the scheduled Observer and the orchestration accounting) — the assistant text was the half an operator could not see.

To be plain about what did not change: the run still failed. It reports the same error, classifies as the same terminal failure class, and retries and notifies exactly as it always did. Only transcript visibility improved — a round-capped task is never recorded as success or partial success.

Iterative verification loops

A scheduled task with a loop_config (#179) runs as a bounded worker → verify → retry loop instead of a single pass: each iteration runs the worker agent to completion, evaluates an exit condition, and — if it fails and budget remains — re-runs the worker with the prior output fed forward, up to max_iterations (default 5). A task with no loop_config is an ordinary one-shot run (unchanged).

The exit condition (each iteration is judged by exactly one):

  • shell:<cmd> — run <cmd> in the worker's sandbox; exit 0 = pass.
  • regex:<pattern> — match <pattern> against the worker's last assistant message.
  • llm — ask verifier_model (defaults to the task's fallback model) the verifier_prompt; a reply beginning with YES = pass. Its spend is not counted toward the iteration cost / max_cost_usd (the worker session is the accounting unit), but each call is recorded in the worker session's aux_usage ledger (#1118). One caveat: for a multi-iteration loop only the surviving (last) worker session persists — pre-existing session handling — so earlier iterations' verifier records survive as host log lines only.

Two ceilings stop a runaway loop, checked before each iteration so already-accrued cost counts: max_cost_usd (accumulated across iterations) and time_budget_seconds (absolute wall-clock). Each iteration is the same governed worker pass an ordinary scheduled task uses (the loop adds only the verify/retry control around agentcore.Run), so the sandbox, policy, audit, and cost gates apply per cycle — "governance is one core" holds. Per-iteration telemetry (status, exit result, cost, tokens) is recorded to task_iterations and embedded in the GET /tasks/{id} response for a looped task.

"Phone a friend": one-time super-LLM review (#175)

An optional, off-by-default quality gate that layers onto the SAME finish seam as the verifier. When FLEET_PHONE_A_FRIEND_ENABLED is set, a scheduled run that has already cleared audit/finish enforcement and the end-of-run verifier is reviewed once more by a configurable — typically stronger — reviewer model (inspired by Brad's lifeline MCP: a one-time second opinion from a more capable model). runPhoneAFriendReview sends the original task, the agent's final answer/work, and the executed-tool summary to the reviewer and asks for a JSON verdict ({"needs_revision", "issues", "reasoning"}); when the reviewer flags material problems, the loop turns the issue list into one more enforcement round so the agent revises before finishing.

What it is and is not, stated plainly (honesty in docs):

  • It is a host-side LLM call, exactly like the verifier — the reviewer's credentials are just another host model handle and never enter the sandbox, the agent's model context, or logs (raw output is clamped to a short preview before any log line). It is not a built-in agent tool and not an MCP server, so the agent cannot invoke the reviewer at will, unbudgeted, or surface it in the sandboxed tool roster — keeping governance one core.
  • It runs at most once per run and fails open: a reviewer error, an empty reply, or an unparseable verdict logs a skip and allows the run to finish, so a flaky reviewer never blocks otherwise-complete work. Like the verifier, its spend stays off the run's ceilings but is recorded in the session log's aux_usage ledger (#1118).
  • It is scheduled-only and gated: with the flag off (the default), the review never runs and behaviour is identical to before. The reviewer model slug comes from FLEET_PHONE_A_FRIEND_MODEL and falls back to the run's fallback model when unset; an unresolvable slug also falls back rather than failing the run.
  • Sub-agents (the other half of #175) are a separate capability — see below.

Because the critique re-enters through the verifier's existing enforcement-round channel (scheduledPolicy.CanFinish), no second governance path is created: the review is bounded by the same per-run audit, finish enforcement, cost/token ceilings, and round cap as everything else.

Governed sub-agents / agent delegation (#175, #264, finished by #1043)

An on-by-default capability (#1043, amending ADR-0007): the spawn_subagent native tool lets a governed run delegate scoped subtasks to child runs — the agent delegation issue #264 asks for, realized as this one tool rather than a second delegate_task entrypoint (a second tool would be the forked, weaker path ADR-0001/ADR-0007 forbid). The child is not a new or weaker loop — it is another agentcore.Run, governed exactly like the parent. The tool body (internal/agent/subagent.go) only adapts I/O around a fresh agent.Agent.Execute.

Registering the tool is the feature; the parent agent decides whether to spawn 0, 1, or N children — a sequential run that never delegates is a successful use of it. The operator only ever opts out, via two independent kill switches:

  • Per task — allow_delegation: false (the column defaults true; existing rows were backfilled by migration 061). Tri-state on create: an omitted field means the default, an explicit false sticks.
  • Fleet-wide — FLEET_SUBAGENTS_ENABLED=false or Admin → Features subagents_enabled off.

They compose as AND (FLEET_SUBAGENTS_ENABLED && task.allow_delegation). When the composed gate is off the tool is not even registered — structural, not a soft check. Interactive chat registers the tool too (#1043) whenever the fleet flag is on (chat has no per-conversation column): same walls, budget sliced from and charged back to the live turn's policy, so the chat cost chip includes child spend. When the tool is registered, a short delegation-policy section is appended to the system prompt (scheduled and interactive) teaching spawn / don't-spawn / prefer-explore / budget rules.

Typed children (#1043). role=explore — the default, and the fallback for any invalid role — is a read-only research child: a single unit-tested denylist strips write-capable native tools (write_file, edit_file, xlsx_workbook, generate_image, create_task, publish_artifact, remember, propose_note, propose_skill) from its final composed roster, a best-effort name denylist narrows its MCP Gate-2 allowlist (mutation verbs like create/update/delete/send/upload as whole snake_case segments; every catalog server covered explicitly; the parent's own allowlist only ever narrowed), and its system prompt carries the read-only instruction for mutators neither list recognizes — that layering is the honest posture. role=worker keeps the full scheduled roster. Either role, the roster drops the interactive-only staging tools. Every child gets an isolated working directory <workspace>/subagents/<child-session-id>/ forced as its bash/file default cwd — still inside the parent's sandbox and bind-mounted workspace, so privilege is unchanged; parallel children just never share a default write path.

Each spawn obeys these non-negotiable properties:

  • Governance is one core. The child runs through (*Agent).Execute → agentcore.Run — the same governed entrypoint, pinned by TestEntrypointConformance.
  • Monotonic privilege. The child inherits the parent's sandbox (so the same network-seal posture — it has no namespace of its own to widen), the parent's brokered MCP client, and the parent's MCP/credential allowlists, and may only subtract (an allow_servers request is intersected with what the parent has loaded; the credential allowlist is the parent's, copied). A per-child model is resolved host-side like the phone-a-friend reviewer, so credentials never enter the sandbox or model context.
  • Hard budget split. The child's cost/token ceiling is capped at a fraction of the parent's remaining budget (FLEET_SUBAGENTS_BUDGET_FRACTION, default 0.10 = the #264 "≤10% of remaining per child") and sliced from what the parent has left, and the child's actual spend is charged back into the parent. A request for max_cost_usd/max_total_tokens above the per-child cap is refused (not silently clamped). The parent's configured MaxCostUSD/MaxTotalTokens is the hard wall the collective spend of all descendants can never breach.
  • One-level delegation. FLEET_SUBAGENTS_MAX_DEPTH (default 1) means parent → sub-agent only: a child does not get the spawn_subagent tool registered at all, so it cannot delegate further. An operator can raise the depth to allow deeper trees.
  • Fan-out cap. FLEET_SUBAGENTS_MAX_CHILDREN (default 5) bounds per-parent fan-out; the (N+1)-th spawn is refused with "max concurrent sub-agents reached" as an error result rather than blocking.

Parallel fan-out (#264). The tool is marked parallel, so when the model emits several spawn_subagent calls in one turn, fantasy dispatches them concurrently (bounded by its parallel-tool semaphore) and the parent collects all results before its next LLM call. The result is machine-parseable JSON {result, cost_usd, tokens, success, role, child_session_id, workdir} so the parent can branch deterministically even when several children return at once — and knows where a worker child's outputs live. The budget split combines an atomic up-front reservation of each child's granted ceiling (held against the parent's remaining budget under the parent mutex for as long as the child runs) with charge-back of the child's actual spend on return — so even N concurrent spawns can never collectively be granted more than the parent has left (a concurrency regression test pins this under -race; the wall-clock test pins that fan-out actually runs in parallel). An optional per-child timeout_minutes bounds a child's wall-clock (spend is still charged back on timeout, success=false), and max_iterations caps its agent steps (clamped at the parent's). A spawned child's run is linked back to its owning task via parent_task_id (on the child's session log and a subagent_spawned entry — child id, role, workdir, spend, success — in the parent's persisted log) for traceability; the task page and chat transcript render those as child cards (id, role, status, spend), never raw JSON, each with a Transcript disclosure that loads the child's own session log through GET /logs/{task_id}/subagents/{child_session_id} (orchestrator; task transcript gate + linkage check) or GET /conversations/{id}/subagents/{child_session_id} (chat; conversation ownership + history linkage). FLEET_SUBAGENTS_MODEL names a default child model slug; empty means the child inherits the parent's model. See docs/SUBAGENTS.md for the #1043 design note.

Git worktree isolation (scheduled tasks)

A scheduled task with a worktree_config (#180) runs each occurrence in its own git worktree + branch, so two tasks targeting the same repository can't corrupt each other's working tree (dirty files, colliding checkouts). A task with no worktree_config shares the workspace root, unchanged.

{
  "worktree_config": {
    "enabled": true,
    "base_branch": "main",          // empty = repo HEAD
    "branch_prefix": "fleet/task-", // empty = "fleet/task-"
    "auto_cleanup": true,           // remove worktree + branch after the run
    "cleanup_delay_seconds": 0      // delay before removal (0 = immediate)
  }
}

The task's workspace must be the root of a git repository; a non-repo (or a non-root subdirectory) is rejected at task creation. Each run gets a deterministic, unique branch "<branch_prefix><task_id>-<run_id>" and a worktree checkout, so concurrent runs never collide and no locking is needed. For a looped task (#179) the worktree is created once per task and reused across iterations, so filesystem state accumulates the way it does for a shared-workspace loop.

Where the worktree lives — and why it is NOT /tmp. The worktree is created as a subdirectory of the workspace root (<workspace>/.fleet-worktrees/<task>-<run>), not at a standalone /tmp path. A git worktree's .git is a file pointing back to "<mainrepo>/.git/worktrees/<name>"; git only resolves it when both the worktree and the main repo are reachable at their host absolute paths inside the sandbox. The sandbox bind-mounts the workspace root at the same absolute path, so a subdir of it satisfies that linkage — a lone /tmp worktree would break git inside the container because the main repo would be unmounted. The subdir is kept out of the main tree's git status via .git/info/exclude (a local, never-committed exclude). The run is scoped into the worktree by two complementary host-side seams: the per-run sandbox's default working directory and a per-run forced working directory threaded into the in-process tool layer. Together they scope bash, run_python, and the relative-path file tools into the worktree — git operations (driven through bash, the point of the feature) are isolated.

Cleanup. With auto_cleanup: true the worktree and its branch are removed after the run (optionally after cleanup_delay_seconds); with false the branch is left in place for inspection or a manual push. Orphans from a crashed run (the process died between worktree creation and cleanup) are reclaimed by an operator with fleet worktree prune --older-than <dur> (and fleet worktree list shows all registered worktrees).