Relaycast has no storage-stats endpoint, so capacity questions are unanswerable during an incident

[kjgbot]

RelayFile can report its own storage; Relaycast cannot. That gap cost most of a day.

What exists

relayfile-cloud exposes GET /v1/workspaces/{id}/admin/storage-stats (added in relayfile-cloud#195), reachable through the relayfile-storage-stats dispatch workflow in cloud. It answered a capacity question in one call:

{"code":"internal_error","message":"Exceeded the maximum database size."}

What does not exist

Relaycast has no equivalent. Verified by code search with a positive control: searching AgentWorkforce/relaycast for sweepStaleAgents returns four files, so the index covers the repo; searching for storage-stats, dbstat, page_count and PRAGMA returns nothing.

Why it matters, concretely

For most of 2026-09-07 Relaycast returned 503 database_overloaded — "The database is temporarily overloaded. Retry after the interval in the Retry-After header." — on writes: node enrolment, agent registration, agent release. Reads succeeded throughout.

Downstream, all of it blocked:

• the Software Garden's factory container cannot enrol and has been down for hours
• agent-relay's release failed repeatedly on Smoke jobs that mint an agent
• relay has ~11 open PRs failing CI for the same reason
• agent-relay agent remove returns status: dispatched and never applies
• the agent registry keeps growing (~4,614 rows, ~39 online)

Work was aimed at that error text for hours — bounded retries honouring Retry-After (factory-cloud#135), broker release resilience, a registry prune. All defensible, none confirmed, because nothing could measure whether Relaycast's database was actually near a limit.

The one measurement available pointed at a different service, and reading it as though it applied to Relaycast produced a wrong conclusion that had to be retracted.

Ask

An admin-only, read-only storage endpoint for Relaycast's database, matching RelayFile's shape: total size, per-table sizes, row counts, and whatever the platform exposes about the configured ceiling. Enough to answer "are we near the limit?" without a Cloudflare dashboard login.

Two properties matter more than completeness:

1. It must work when the database is under pressure, which is the only time anyone will reach for it. An endpoint that fails exactly when the system is unhealthy is not an instrument.
2. It should distinguish "at capacity" from "busy". The current 503 text conflates them, and that conflation is what sent several people down a retry path when the question was capacity.

Related

• AgentWorkforce/cloud#3389 (registry ghosts)
• AgentWorkforce/cloud#3401 (sandbox agents spawn but never register)
• GET /v1/agents performs a database write on every roster read, and lost the SQL status pushdown #315 (roster read was a write — fixed in 8.5.1, did not resolve the 503s)

[khaliqgant]

Fresh production evidence on 2026-09-08: Relay PR #1705 Package Validation run 34244078714 failed Standalone macOS Smoke at 15:29 UTC because agent registration returned 503 database_overloaded after all three attempts. A separate isolated Agent Relay Cloud recovery probe 9c99aad0-1ca0-4d8e-b208-849c711abff3 at 16:01 UTC successfully provisioned the outer workflow sandbox and a fresh per-step Daytona sandbox, then failed before agent execution because mcp-args --register exhausted Relaycast retries (Invalid response: Max retries exceeded). This currently blocks the guarded Agent37 AWS→Cloudflare production cutover: both the standalone package smoke and exact RelayFlow proof depend on a successful registration. No credentials or payloads are included.

[khaliqgant]

Intermittency update: an unrelated Package Validation run 34249509006 completed successfully at ~16:22 UTC, including Standalone macOS Smoke, but the exact Relay #1705 attempt-2 smoke 34244078714 then failed at 16:30 UTC with the same 503 database_overloaded after three registration attempts. Exact failure command used a unique run/attempt-scoped instance name; broker API bound locally, then connect_relay failed during workspace-key registration. This is not a stale single outage sample: registration writes are currently flapping within an ~8-minute window.

[khaliqgant]

Fresh independent recurrence at 18:01 UTC: Relay Package Validation run 34259289395, Standalone macOS Smoke job 102173861543, failed during connect_relay registration with HTTP 503 code database_overloaded after all 3 attempts. Broker startup/handshake had reached the registration write; this is a new unrelated head after the prior 16:30 exact-head failure, so registration remains unavailable rather than merely stale CI.

[khaliqgant]

Exact recurrence after the apparent recovery: Relay #1705 Package Validation run 34244078714, attempt 3, Standalone macOS Smoke job 102215949142 failed at 20:06:20Z on unchanged head 111ae74bdffa721f166a73f791238bc081c0fca1. Broker API bound locally, then connect_relay registration exhausted all 3 attempts with HTTP 503 database_overloaded. This occurred after the two unrelated successful registration smokes at 18:19Z and 18:30Z, so those passes showed a healthy interval, not durable recovery. The guarded release remains paused; no further exact registration retries will be launched until service recovery is independently demonstrated again.

[khaliqgant]

Post-deploy recurrence: production Deploy run 34272356093 for merged relaycast#391 completed successfully at 20:09:04Z. The unchanged Relay #1705 exact Standalone macOS Smoke then failed again at 20:22:14Z (Package Validation 34244078714, attempt 4, job 102220588057): broker API bound, first handshake timed out after 12s, and registration ultimately exhausted 3 attempts with HTTP 503 database_overloaded. Therefore the #391 deployment did not restore registration durability. An unrelated post-deploy Package Validation smoke 34273559379 is in progress and will provide an independent witness.

[khaliqgant]

Independent post-deploy confirmation: unrelated Relay Package Validation run 34273559379, Standalone macOS Smoke job 102222203324, failed at 20:30:06Z with the same registration path: broker API bound, connect_relay registration exhausted all 3 attempts, HTTP 503 database_overloaded. Together with exact #1705 attempt 4 at 20:22Z, this is two consecutive real registration-write failures after relaycast#391 deployed at 20:09Z. Relaycast remains unavailable for the guarded release gate.

[khaliqgant]

Fresh independent recurrence after 20:59Z: Relay Package Validation run 34277932803, Standalone macOS Smoke job 102236280553, failed at 21:11:48Z. The local broker API bound successfully; the first connect_relay handshake timed out after 12s, then agent registration exhausted all 3 attempts with HTTP 503 database_overloaded. This is a new unrelated head after the prior 20:22Z/20:30Z failures, so hosted /v1/agents remains unavailable. Current investigation is adding safe stage-labeled overload telemetry to distinguish workspace-key auth lookup from registration work before choosing a database change; no more exact cutover retries will be launched until a credible state change or independent success.