Skip to content

Console API calls from the sandbox get their own passthrough limits instead of internet-fetch limits (OPT-1586) - #23

Merged
Optale-ops merged 3 commits into
mainfrom
opt-1586/console-api-egress-limits
Oct 13, 2026
Merged

Optale-ops merged 3 commits into
mainfrom
opt-1586/console-api-egress-limits

Conversation

@Optale-ops

Copy link
Copy Markdown
Owner

Engine half of OPT-1586. Console API traffic from the sandbox gets its own passthrough limits (host field httpsPassthroughLimits, granted by the deployment policy and requested by the Console-signed policy, smaller of each), metered in its own per-host budget; internet fetches keep the current limits.

Files.

  • service/src/external-fetch-policy.ts: new optional host field httpsPassthroughLimits {headersTimeoutMs, maxFetchesPerGrant, maxAggregateBytesPerGrant}, capped at 300 s / 1000 / 512 MiB, with the header wait never above the host's passthrough total. The field is parsed, serialized and covered by the digest. It applies only when the Console-signed policy asks for it and the deployment policy grants it, at the smaller value of each.
  • service/src/external-fetch.ts: the passthrough header wait uses the host's own limit when one is set
  • service/src/egress-ledger.ts: per-host passthrough budgets in the ledger record (passthrough_budgets). consume, reserve, commit and release take an optional passthroughHost. Without it they use the run-wide budget, so internet fetches keep 8 per run, independent of Console calls.
  • service/src/egress-gateway.ts: /https-passthrough meters a host that has its own limits in that host's budget
  • service/config/external-fetch-policy.json and helm/codeapi/values.yaml: the Console host gets 55000 / 500 / 268435456
  • tests: service/src/egress-ledger.test.ts (one test) and service/src/external-fetch-policy.test.ts (one describe block)

Why. OPT-1586 (Figent agent stalls F2, spec /home/thor/optale-artifacts/FIGENT-AGENT-STALLS-2026-10-13/diagnosis-spec.md RC2): optale-agent tool call from code leaves through the sandbox egress gateway, which held calls to our own Console API to internet-fetch limits (5 s to headers, 8 fetches per run). Figent on 2026-10-13: 27 FETCH_TIMEOUT and 32 FETCH_BUDGET_EXCEEDED.

Production engine today (read-only on AX41). Checked read-only on AX41 (65.109.115.37) over ssh with ~/.ssh/optale-ci-microvm-proof. api, worker, egress_gateway and file_server run code-interpreter e9ad47c (main before #20), created 2026-10-04; sandbox-runner is 0b2e6a5. The mounted policy /etc/optale/code-interpreter/external-fetch-policy.v2.json has global limits of 5 s to headers, 15 s total and 8 fetches per run. console.optale.com, figent.optale.com and console-staging.optale.com are HTTPS passthrough hosts with a 300 s total, 8 fetches and 2 MiB per response. A passthrough call therefore waited at most 5 s for headers and shared the 8-fetch budget with internet fetches; that is RC2 as the spec describes it. A separate limit also bites: api and worker run with EXECUTION_MANIFEST_MAX_REQUESTS=100, which caps all gateway requests in one code run.

Root cause. service/src/external-fetch-policy.ts:82-91 (origin/main) sets HARD_EXTERNAL_FETCH_LIMITS to 5 s headers, 15 s total and 8 fetches per run. Every host's limits were capped at the global limits (parseLimits, intersectExternalFetchPolicies), so a passthrough host could never raise them. service/src/external-fetch.ts:399 waited for passthrough headers with target.policy.limits.headersTimeoutMs, which is 5 s. service/src/egress-gateway.ts:1913-1916 and service/src/egress-ledger.ts:353-363 consumed the single run-wide fetch_count, shared with internet fetches. On the Console side, ops/optale-agent/transport.cjs:137-143 (main) printed the gateway's raw JSON {"error":"FETCH_TIMEOUT","message":"External fetch timed out"}.

Real-stack proof (KS-7). KS-7 through seat-heavy, run3, 2026-10-13 18:04-18:22Z. Evidence in /home/thor/optale-artifacts/foundation/OPT-1586/ (README.md, scripts/, evidence/run3/).

  • Conversation 42d73498-f594-4670-bc91-29690211ec35, one execute_code run, turn 294 s, no assistant error
  • CLI_CALLS 60 OK 60 SANDBOX_REFUSALS 0: 54 wiki_search calls plus 6 real gmail_fetch_emails calls at 2.6-3.6 s each
  • Calls slower than 5 s all succeeded: six held calls at 8.8-9.1 s, and the first call at 14.2 s (cold Console)
  • Gateway log: 61 passthrough requests to the Console host, all success 200 (60 calls plus the tool listing), against the old cap of 8; gateway durations up to 13,989 ms. No FETCH_TIMEOUT or FETCH_BUDGET_EXCEEDED on the Console host except the forced hit.
  • Internet host, same gateway and same run, still on the old limits: /delay/7 got 504 FETCH_TIMEOUT at 5,178 ms; the 9th fetch got 429 FETCH_BUDGET_EXCEEDED
  • Forced limit: a call held 70 s ended at 55.2 s with exit 8 and the output: OPTALE_CLI_SANDBOX_LIMIT: the code sandbox's network gateway stopped this call (the sandbox's time limit for one Console call ran out, gateway code FETCH_TIMEOUT); this is a limit of the code sandbox, not an error from the wiki_search tool. The tool may still have run, so check its effect before repeating a write. Make this call as a direct tool call instead of retrying it from code.

Not proven.

  • Monday was not exercised: Lab 1's seed has no Monday connection in use. Gmail was real; the slow calls came from the delay stub.
  • The stack was not run with hundreds of calls in one run: the 100-request cap above was kept for faithfulness, and the run made 60 calls plus internet fetches. Above about 95 calls a run hits that cap and gets a 403 'Egress grant request budget exceeded' (the CLI names it as a sandbox limit). Raising it is an AX41 config change.
  • The full UI drive ran only against the new engine; the before state is shown by production's numbers and by the internet-host leg of the same run, which is configured exactly like production's Console hosts today.
  • The model's first execute_code result was cut by the Console's 64 KiB tool-result cap and the model read the full file with read_file. That belongs to F1, not this change.

Risks and rollout order.

  • Rollout order is required: an engine without this change refuses a signed policy that carries httpsPassthroughLimits (malformed_claims), so every code run fails. Order: (1) engine release on AX41 (api, worker, egress); it changes nothing on its own. (2) AX41 config: add httpsPassthroughLimits {55000, 500, 268435456} to console.optale.com, figent.optale.com and console-staging.optale.com in external-fetch-policy.v2.json, and raise EXECUTION_MANIFEST_MAX_REQUESTS from 100 to at least 600. (3) Console 443dd3280 in a release after that. 5ed8ca662 is independent and can go in 0.16.2.
  • The sandbox runner's socket proxy (api/src/tool-call-socket-proxy.ts, activeRequestTimeoutMs) drops a connection idle for more than 60 s with a plain 'bad gateway'. That is why the header wait is 55 s, so a slow call ends in the named FETCH_TIMEOUT. A Console call that takes longer than 55 s still fails from code; the CLI then tells the model to make a direct call.
  • Console responses through passthrough are still capped at 2 MiB each by the deployed host limits; this branch does not change that.
  • The CLI recognises gateway refusals by their body ({error: } or 'Egress grant ...'). The Console's own MCP errors use object-shaped errors, so they do not collide, but this is a matching rule rather than a header contract.

…r-run budget (OPT-1586)

A passthrough host may carry httpsPassthroughLimits {headersTimeoutMs,
maxFetchesPerGrant, maxAggregateBytesPerGrant}. It applies when the signed
policy asks for it and the deployment grants it (smaller value of each);
passthrough requests to that host then wait for headers up to its own
timeout and are metered in a per-host ledger budget, apart from the
grant-wide external-fetch budget. Internet fetches keep the frozen limits.
Policies without the field verify and behave exactly as before.
… than the grant fetch budget (OPT-1586 review)
…led ledger, not trailers (Bun 1.3.14 client exposes none)
@Optale-ops
Optale-ops merged commit 538ea32 into main Oct 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant