What happened
A customer's persona deploy went red with the cloud's generic catch-all:
cloud: deploying persona bundle to https://agentrelay.com/cloud
...98 seconds...
agentworkforce deploy failed: cloud deploy failed: 500
{"error":"Failed to deploy persona bundle","code":"deployment_failed"}
Re-running it succeeded (201, 11.2s) with a byte-identical bundle. The cause was on the cloud side — the Worker lost its Neon pool socket mid-request (AgentWorkforce/cloud#2300) — but the interesting part is that the CLI already has a retry for exactly this and it did not save us.
The two defects
packages/deploy/src/modes/cloud/ — requestJsonWithRetry wraps the bundle POST ({ action: 'cloud deploy' }) and does retry 5xx, MAX_ATTEMPTS = 3. But:
1. The backoff is far too tight.
function backoffMs(attempt) {
const override = numberFromEnv('WORKFORCE_DEPLOY_RETRY_BACKOFF_MS');
return override ?? attempt * 500;
}500ms, then 1000ms — about 1.5 seconds of total spread across all three attempts. A severed database connection pool does not recover in 1.5 seconds, so all three attempts land inside the same fault window and the retry buys nothing. This is tuned like a rate-limit retry, but the faults it actually meets are infrastructure-level.
2. It is completely silent.
Nothing is logged when an attempt fails and is retried; lastError is just overwritten each pass. So three failed attempts and one failed attempt produce byte-identical output.
That second one is the expensive defect. Debugging today's failure, I could not tell from the log whether that 98 seconds was one slow attempt or three — and the absence of any retry line strongly implied "no retry exists here," which sent me looking for a regression in the user's repo. It wasn't there. A single line per retry would have said "the cloud is flaking, this is not yours" immediately.
Proposed fix
- Backoff: something like
1s → 5s → 15s (or exponential with jitter), spreading attempts across a window where a reconnect is plausible. Keep the WORKFORCE_DEPLOY_RETRY_BACKOFF_MS override.
- Log every retry through the existing
emitLog / args.io path, naming the status and the wait — e.g. cloud deploy: 500 from the cloud (attempt 1/3), retrying in 1s.
- Report the attempt count in the final error, so a 3× failure says so instead of looking like a single shot.
- Consider surfacing the retry count on the result so callers can distinguish "worked first time" from "barely worked."
The existing 4xx exclusion in isRetryableError is correct and should stay — client errors are the caller's to fix and must keep failing fast.
Why this is the right layer
The CLI holds the actual HTTP response, so it can branch on res.status as a number. I've landed a stopgap at AgentWorkforce/agents#144 that retries at the script level, and to decide whether to retry it has to regex-parse the CLI's human-readable error prose to recover a status code that existed as an integer one layer down. That's the smell that says this belongs here. Once this issue ships, that stopgap should be deleted.
Related
What happened
A customer's persona deploy went red with the cloud's generic catch-all:
Re-running it succeeded (201, 11.2s) with a byte-identical bundle. The cause was on the cloud side — the Worker lost its Neon pool socket mid-request (AgentWorkforce/cloud#2300) — but the interesting part is that the CLI already has a retry for exactly this and it did not save us.
The two defects
packages/deploy/src/modes/cloud/—requestJsonWithRetrywraps the bundle POST ({ action: 'cloud deploy' }) and does retry 5xx,MAX_ATTEMPTS = 3. But:1. The backoff is far too tight.
500ms, then 1000ms — about 1.5 seconds of total spread across all three attempts. A severed database connection pool does not recover in 1.5 seconds, so all three attempts land inside the same fault window and the retry buys nothing. This is tuned like a rate-limit retry, but the faults it actually meets are infrastructure-level.
2. It is completely silent.
Nothing is logged when an attempt fails and is retried;
lastErroris just overwritten each pass. So three failed attempts and one failed attempt produce byte-identical output.That second one is the expensive defect. Debugging today's failure, I could not tell from the log whether that 98 seconds was one slow attempt or three — and the absence of any retry line strongly implied "no retry exists here," which sent me looking for a regression in the user's repo. It wasn't there. A single line per retry would have said "the cloud is flaking, this is not yours" immediately.
Proposed fix
1s → 5s → 15s(or exponential with jitter), spreading attempts across a window where a reconnect is plausible. Keep theWORKFORCE_DEPLOY_RETRY_BACKOFF_MSoverride.emitLog/args.iopath, naming the status and the wait — e.g.cloud deploy: 500 from the cloud (attempt 1/3), retrying in 1s.The existing 4xx exclusion in
isRetryableErroris correct and should stay — client errors are the caller's to fix and must keep failing fast.Why this is the right layer
The CLI holds the actual HTTP response, so it can branch on
res.statusas a number. I've landed a stopgap at AgentWorkforce/agents#144 that retries at the script level, and to decide whether to retry it has to regex-parse the CLI's human-readable error prose to recover a status code that existed as an integer one layer down. That's the smell that says this belongs here. Once this issue ships, that stopgap should be deleted.Related