feat(openai): reduce the risk of rate-limit errors when many agents run at once

[bman654]

When you run a lot of agents at once on a ChatGPT/Codex plan, OpenAI sometimes starts rejecting
clodex's attempts to open new connections, and those turns come back as rate-limit errors. clodex
already handled the rejection gracefully — it retries — but nothing limited how fast it asked for
new connections. This change makes clodex pace itself, which should make a burst of parallel work
less likely to trip the limit. (In the traffic sampled, rejections clustered in the busiest minutes;
that the rate is what triggers them is a reasonable reading of that, not something proven.) A
follow-up turn in a conversation that already has a connection is never slowed down; only work that
needs a brand new connection goes through the limiter.

There is a real cost, and it is stated up front in the README rather than buried: this is a
throughput ceiling, not a brief pause. One new connection per second means roughly 20 agents settle
at about 20 seconds per turn instead of a few seconds. You wait longer, in exchange for a lower
chance of losing turns to rate-limit errors — a reduction in risk, not a guarantee. Under a heavy
enough fan-out the limiter itself answers a turn with a rate-limit response, and the README says so.

---

The failure this addresses

OpenAI's edge returns HTTP 403 on a Responses WebSocket upgrade when an account opens new
connections too fast. responses-websocket.ts maps every upgrade 403 to a retryable Anthropic 429
with a backoff hint, which the AI SDK honours — so the user sees latency and retries rather than a
hard failure, but under a big fan-out they see a lot of both.

Evidence

Measured by re-reading one machine's own ws_head_decision diagnostics (103,698 records spanning
about a day and a half), bucketing records that carry a createdConnectionId by wall-clock minute:

statistic	value
minutes that opened at least one connection	1,158
median / p90 / p99 / max per minute	6 / 22 / 48 / 82
minutes above 60	4
upgrade 403s	40, all inside three minutes
where they fell	39 in the two minutes that each opened 82; 1 in a minute that opened 41

On a 200k-line slice of the same log, 11,417 of 26,430 head decisions opened a connection and 10,229
of those were the non-reusable parallel fan-out kind — new connections move with the number of
concurrent agents, not with conversation length.

This is a correlation in one account's traffic over one window, not a published limit and not a
demonstrated cause. The instantaneous rate was 3-5/second in the bad minutes and the clean ones
alike, which is why the limiter shapes a sustained rate rather than a burst. The single rejection in
a 41/minute minute does not fit the pattern and is unexplained; it is left visible rather than
dropped, because it is the main reason to treat the sustained-rate inference as weak.

What changed

A process-wide token bucket (src/oauth/ws-upgrade-pacer.ts) gating connection creation only,
called from one site in responses-websocket.ts. Reuse of an established or nursery head never
consults it. Defaults: 60/minute sustained, burst 10. CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN
overrides the rate (1-600; 0 disables; out-of-range clamps and malformed values are ignored, each
with a one-time notice).

Overflow is refused, not queued indefinitely, using the same retryable 429 frame shape the 403
path already produces. A refusal debits no token — that is the load-bearing detail. A refused
request opens no connection and will be retried, so charging it would let every retry deepen the
deficit that caused the refusal. Because only an admitted request debits, and only within the wait
bound, tokens cannot fall below -bound × refill: the queue is bounded by construction.

The wait bound is derived, not chosen. Every attempt of one request shares one no-data deadline —
the timer starts before the SDK call and only a stream part resets it — so the whole ladder must fit:
(maxRetries + 1) × bound + totalBackoff < idleTimeout. Both terms are read, through the same
upstreamRequestBudget() call every SDK generation entry point makes, so the pacer sizes its wait
against the deadline the paced request will actually spend. At the shipped defaults that is 4833ms.
Where a configuration leaves no room to queue at all, the bound floors to zero and pacing turns
itself off with a notice; it degrades to less pacing, never to a request pushed past its deadline.

The design that was rejected, and why it matters

The first implementation admitted anyway once the bound expired. A review measured it against the
real class at shipping defaults and found sustained output equal to sustained input:

offered	admitted (old design)
70/min	71/min
82/min	82/min
120/min	120/min
240/min	240/min

82/min is the rate that produced 39 of the 40 observed rejections. The debt floor and the wait bound
were the same number by construction, so the shaping reservoir was exhausted exactly where the bound
took over: every connection past the first ~24 waited the full 15s and was then opened anyway. It
would have shipped a limiter that did not limit, plus 15s of latency. There is now a test that
measures the final minute of a four-minute overload; against the old behaviour it reads 82.

The defect an adversarial review found after the rebase, and the fix

Worth reading even if you skip the rest, because it invalidated the bound derivation this feature is
built on and it survived every earlier review.

The bound budgets the SDK's 2s/4s/8s/16s/32s ladder as the delay between attempts. A refusal sends
its own Retry-After, and the SDK spends that hint INSTEAD of the rung — getRetryDelayInMs in
ai@7.0.22 (util/retry-with-exponential-backoff.ts:46-56) returns the supplied delay whenever it
is under 60s; it does not take the larger of the two. The hint was derived from the token deficit,
which is by definition larger than the bound (that is why the request was refused), so the pacer was
replacing the exact term its own inequality was derived against.

Executed repro: at 2 connections/minute inside a 10s deadline, the inequality passes
(3 × 666ms + 6000ms = 7998ms < 10000ms) and the refusal still asked for 30 seconds. The request
died at the deadline with attemptCount: 1 — both retries its budget had paid for never ran. This
is a default-on hard abort of the same shape as the 152s-vs-120s error, reached by a different
route.

Fix. The hint is capped at the derived bound, floored at one second, in a new
pacedRetryAfterSeconds. requiredWaitMs still reports the true deficit to diagnostics; only the
number the client is asked to honour is capped. The one-second floor exists because Retry-After is
whole seconds and zero would mean "retry immediately" — a hot loop that burns the retry budget in
milliseconds — and it is safe because 1s is strictly below the SDK's smallest rung, so it stays
inside the ladder term the bound already budgets. That floor matters wherever the bound is
sub-second, e.g. IDLE=10000 gives a 666ms bound.

The property now asserted across all 88 environment combinations is the one that accounts for the
substitution, not just the ladder:

(maxRetries + 1) × bound + Σᵢ max(hintCapMs, 2000 × 2^(i-1))  <  idleTimeout

Cost, stated rather than buried. A refused request now comes back sooner than its deficit needs
and may be refused again. That is the right trade — it spends its retries inside its deadline
instead of spending its whole deadline on one oversized sleep — but it does not make refusals
succeed, and at rates far below the deficit a fan-out can still end in a rate-limit error.

A second blocker, created by that fix, and a false oracle that hid it

The re-review of the fix above found both. Recording them because the pattern matters more than
either defect.

Capping the hint fixed one failure and created its mirror image. With the hint capped at the
bound, a refused request comes back every ~4s. At CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=1 the
first free slot is 60 seconds away, so all six attempts are spent inside 20 seconds and the request
fails as a rate-limit error — manufactured by the feature that exists to reduce them. Executed
against the real pacer and real SDK, 20 concurrent, default 120s/5 retries:

rate	admitted	terminal paced 429s	attempts each
1/min	10	10 of 10	6
2/min	10	10 of 10	6

Both rates are documented as supported. The fix had traded 7 deadline failures for 10 terminal ones.

The pacer now refuses only when its retry schedule can outlast the wait for a slot
(canRefuseAtRate); below that it shapes the opening burst and then admits the remaining overflow
instead of failing it, with a notice. That is exactly the rule the zero-bound case already applied,
and the review correctly pointed out I had been applying it to a zero bound but not to a positive
bound that equally cannot make progress.

I first justified that with a claim that was false, and a later round disproved it. I wrote that
no hint strategy could fix 1/minute. A separately budgeted 12-second hint does reach the refill —
attempts at 0, 12, 24, 36, 48, 60s — and still fits the conservative mixed-gap bound at the shipped
budget: 6 × 4833 + max(12,2) + max(12,4) + max(12,8) + max(12,16) + max(12,32) = 112,998ms < 120,000ms. The true statement is narrower: no strategy admits all the overflow inside the
deadline while preserving the configured ceiling — at 1/minute, ten simultaneous overflow requests
need ten minutes of capacity, so a longer hint rescues one and still fails nine. A separately
budgeted hint is a reasonable follow-up, and is named as one in the source and the deep doc. Both
now carry the correction rather than the original claim.

A later round caught a second overstatement in the very sentence that replaced the first, which
is worth recording because it is the identical failure mode one level down. I had written that
shaping "spends the shortfall on latency the user configured rather than on errors." It does not:
the user configures a connection rate, not a latency, and the fallback mostly adds no latency at
all — at 1/minute with 20 simultaneous requests, 19 are admitted immediately, one waits out the
bound, and none is refused. What actually gives way is the configured ceiling. The accurate
statement, now in both places: it avoids guaranteed local failures while retaining bounded
opening-burst shaping, at the cost of relaxing the configured ceiling.

Verified in a real run — the notice reaches the terminal:

clodex: pacing new OpenAI connections at 1/minute without refusing overflow: a 5-retry schedule
spans only 20000ms, which cannot outlast the 60000ms wait for a free connection slot, so excess
connections are admitted late rather than failed

The false oracle is the more instructive finding. The matrix that was supposed to prove the cap
computed its expected value by calling the function under test — with Number.POSITIVE_INFINITY,
which that function deliberately maps to one second. So hintCapMs read 1000ms for every bound,
including the shipped 4833ms and the 15000ms ceiling, and the whole 88-combination matrix was
asserting one degenerate value. The reviewer proved the consequence: this mutation survived all 131
pacer tests —

if (maxWaitMs > 2_000) return wantSeconds;   // cap only below 2s
return Math.min(wantSeconds, capSeconds);

— while keeping the two explicitly tested bounds (666ms, 2000ms) green, and under it the original
deadline-overrun blocker returns in full at the shipped bound. The expected cap is now derived
independently of the implementation, with direct cases at 4833ms and 15000ms; that mutation now
fails 6 tests.

The lesson is the one this PR keeps relearning: an oracle that calls the implementation cannot
falsify the implementation. Three of the four defects found after the first "green" were of that
shape — verifying a model of the behaviour rather than the behaviour.

A third round: two more mutations that survived a green suite

Neither is an implementation defect — both are tests that could not tell the shipped code apart
from a broken one.

The serviceability rule was only ever tested at the default budget. Every committed case used
the shipped 4833ms bound and five retries, so this passed all 142 pacer tests, all 121 WebSocket
tests and typecheck:

return maxRetries > 0 && ratePerMinute >= 3 && refillIntervalMs > 0 && maxWaitMs >= 0;

It recreates the terminal-failure defect under a fully supported configuration —
CLODEX_UPSTREAM_IDLE_TIMEOUT_MS=10000 with rate 3, where a 2s retry schedule faces a 20s wait.
There is now a budget matrix across both the shipped and a shortened deadline, including the
threshold either side of 30/minute, plus a behavioural pair that proves the constructor consults the
rule rather than a truth table proving only that the rule exists. That mutation now fails 4 tests.

The low-rate fallback's shaping was unasserted. This also passed everything:

this.refillPerMs = this.enabled ? (maxRetries > 0 && !this.canRefuse ? 0 : ratePerMinute / 60_000) : 0;

It makes low-rate pacing identical to pacing switched off. The tests asserted that nothing was
refused, never that anything waited — including the end-to-end test whose own comment said a request
"really does wait out the ~4.8s bound". Both now assert the retained delay. The mutation fails 3
tests.

Writing that assertion corrected my own model of the code: I predicted one delayed request and
measured two. Simultaneous arrivals hit the debt floor and pass straight through, but sequential
arrivals refill roughly what they consume, so each one past the burst keeps paying the bound. At
1/minute the effective ceiling is therefore about one per bound — weaker than configured, but not
absent, which is the distinction the mutation erased.

Four smaller corrections from the same review

• Notice dedupe keys are namespaced. Rate-validation notices keyed off an arbitrary environment
  string and shared one set with the pacing-disabled notice, so
  CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=disabled:3:14001 suppressed the one notice that must never
  go missing — that pacing turned itself off. Reproduced, fixed, and pinned with that exact string.
• The docs no longer promise that rate-limit errors stop. README, the deep doc and the module
  all said the trade buys "not tripping the throttle", while the module header correctly calls the
  causal link an assumption. They now say it lowers the chance and can itself refuse a turn.
• A test comment claiming "an ordinary user never sees a refusal" sat in a test that observes a
  refusal at 15 simultaneous connections — inside the many-agent workload this feature targets. It
  now says that, and notes the rebase moved the threshold down from 26 to 15.
• The "the throttle is the account's" claim is dropped. One account on one machine cannot tell
  an account-, IP-, model- or edge-level limit apart. One shared bucket is now presented as the
  conservative choice under that uncertainty rather than as a modelled fact.

Deliberately not in this PR

• Pool caps, the existing 403 handling, isolated-connection reuse, and the retry policy — all
  out of scope by design.
• The two replacement paths are not paced. A transport retry and a previous_response_not_found
  retry each build a connection through createReplacement. Both recover a request that was already
  admitted, both are capped at one per request, and both run inside socket callbacks where an await
  would restructure the retry path. Each now has a negative test. This means the bound is on
  admissions, not on sockets, which the docs say explicitly.
• Re-matching a head that freed up during the wait — filed as a follow-up (see below).

Verification

Re-run in full after the rebase onto #171, on the merged tree rather than on either branch
alone. pnpm typecheck && pnpm test && pnpm build green: 2260 tests, 106 files. Repeated under a
throwaway CLODEX_HOME. Node 24.14.1, macOS arm64, pnpm 10.34.5.

Smoke tested through real Claude Code on the default proxy mode, not just the changed path. The
pre-rebase branch ran Anthropic passthrough (HAIKU-OK), the translated OAuth WebSocket leg
(MODEL-OK), CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=0 (DISABLED-OK), a malformed pacing value
which printed its one-time notice and still answered (MALFORMED-OK), and
CLODEX_UPSTREAM_MAX_RETRIES=0 (NORETRY-OK). All were re-run against the rebased build, plus two
legs that only exist now that #171 made the deadline configurable:

leg	result
Anthropic passthrough	HAIKU-OK
translated OAuth WebSocket (the path this changes)	MODEL-OK
CLODEX_UPSTREAM_IDLE_TIMEOUT_MS=30000 — bound derives to 2000ms	SHORTIDLE-OK
CLODEX_UPSTREAM_IDLE_TIMEOUT_MS=14001 — bound floors to 0, pacing disables itself	DEGRADE-OK
CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=0	DISABLED-OK
CLODEX_UPSTREAM_MAX_RETRIES=0	NORETRY-OK

A CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=1 leg also answered (PACED-OK), but that leg proves
only that the tightest configuration still serves a request — a single-turn prompt needs one
connection and the burst allowance covers it, so no refusal was produced on the wire. The refusal
path's evidence is the unit tests and the reviewer's executed repro, not this leg.

The DEGRADE-OK leg matters because it is the safe-degradation path, and #171 is what made it
reachable at all — on the pre-rebase branch the deadline was a constant, so no user configuration
could floor the bound. The run also printed the notice to the real terminal after Claude Code
exited, so the "it never turns itself off silently" claim is now an observation rather than a
reading of the code:

clodex: not pacing new OpenAI connections: a 3-retry budget leaves no room to queue inside the
resolved 14001ms request deadline

Mutations run, and what caught them

Every behaviour below was mutated individually and the suite re-run in full.

mutation	caught by
delete the pacing call site	8 tests
make reuse consult the pacer	never asks the pacer for a request that reuses an existing connection
debit a token on refusal (the livelock)	3 tests, incl. the sustained-rate test
admit everything	9 tests
admit anyway past the bound (the rejected design)	sustained-rate test, reading 82 vs 61
remove the post-admission demotion	demotes a concurrent sibling… ([nursery,nursery] vs [isolated,nursery])
remove the post-wait clock refresh	ages the heads it reports from after the wait
tax every request past the floor with retries off	shapes the opening burst but stops taxing…
assume 5 retries instead of reading	derived-bound test
refuse even with retries off	no-refuse test
flat 15000 bound	8 tests
burst 10→11; clamp >→>=; break warn-once dedupe	2 / 1 / 7 tests
refusal frame code, retry_after_seconds, retry after Ns prose	refusal test
drop or corrupt the retry-after header	refusal test
no-op the ?? sharedWsUpgradePacer() fallback	paces through the real process-wide bucket…
no-op sharedWsUpgradePacer() itself	same test

Two holes were found this way and closed: the retry-after header shipped with no test, and the
frame's retry_after_seconds is invisible to the SDK (its chunk schema strips unknown fields), so
the prose is the only channel that reaches the client — now asserted directly on the frame.

The last two rows matter because every other integration test injects a fake pacer through a
test-only option. One test uses no injected pacer, drives 11 real connections through the shared
bucket with the env var set, and is the sole evidence for the production singleton and env wiring.

Mutations re-run after the rebase

The table above was measured on the pre-rebase branch. The guards that the rebase could plausibly
have broken were re-mutated against the merged tree, running
tests/ws-upgrade-pacer.test.ts + tests/responses-websocket.test.ts each time:

mutation	result
flat 15s bound — the 152s-vs-120s error	67 tests red
derive the bound from a hardcoded 120000ms deadline	3 tests red (all of them new; nothing pre-existing catches this)
hardcode maxRetries at 5 instead of reading it	1 test red
hardcode maxRetries at 2 — the pre-rebase ?? SDK_DEFAULT_MAX_RETRIES behaviour	3 tests red
debit a token on refusal — the livelock	3 tests red, incl. the sustained-rate test
tax every request past the debt floor instead of returning a zero wait	1 test red
pace anyway when the bound is zero (remove safe degradation)	2 tests red
uncapped retry-after hint — the blocker above	2 tests red
drop the hint's 1s floor (hint reaches 0s)	survived, so an explicit assertion was added; now 1 test red
un-namespace the notice dedupe keys	1 test red
cap the hint only below a 2s bound (reviewer's surviving mutation)	6 tests red
refuse at any rate (remove the low-rate degradation rule)	2 tests red
budget-blind serviceability rule (reviewer's second surviving mutation)	4 tests red
low-rate fallback made equivalent to disabled pacing (reviewer's third)	3 tests red

Two mutations survived at some point and each produced a test. Dropping the one-second hint floor
was invisible until an explicit lower-bound assertion pinned it. Capping the hint only below a 2s
bound was invisible because of the false oracle described above, and is now pinned by direct cases
at the two bounds production actually uses.

What review attacked and could not break

Stated because it is the stronger half of the evidence:

• An ordinary user never sees a refusal. On the pre-rebase branch a reviewer's closed-loop
  simulation put the threshold at 26 simultaneous new connections. feat(timeouts): let you raise the 10-minute limit so long agent runs can finish #171's five-retry default
  tightens the bound and moves it to 15 simultaneous new connections, now measured directly
  rather than derived (refuses only past a large simultaneous fan-out at the shipped defaults).
  Ten concurrent agents produced zero hard failures over a 300-second closed-loop run on the
  pre-rebase branch.
• The refusal is genuinely retried in-process, and the retry-after header is genuinely read.
• No unbounded wait, no token leak, no double refund, no lock across an await, tokens never
  exceeds capacity, and admission order is FIFO with deadlines provably 1/rate apart.
• The pacer is non-compressive: output burstiness never exceeds input burstiness, so it cannot
  turn a spread-out arrival pattern into a herd.
• createdConnectionId remains an exact prediction — there is no await between the diagnostic and
  the connection.

Boundaries I could not verify

• The measurement is one account, one machine, one window. The 60/minute default is a conservative
  reading of it, not a known threshold.
• idleTimeoutMs is assumed at 120s — closed by the rebase. Both terms are now read from
  the resolved budget. What is still assumed is that the pacer's snapshot matches the request's: the
  pacer resolves the budget once when the process-wide bucket is built, and no production caller
  overrides idleTimeoutMs on that call (verified by grep — the override seam exists for direct
  adapter callers and is unused), so the two agree for any environment a user can set. A future
  caller that starts passing an override would break that agreement silently.
• The derived-bound test re-encodes the SDK's 2000 × (2^n − 1) backoff ladder rather than
  integrating with the SDK, so a change to the SDK's backoff policy would not turn it red. This is
  the exact class of gap that produced the Retry-After blocker above, and it is not closed.
• The worst-case gap model is max(hintCap, rung), which does not model a provider-supplied
  retry-after on a non-pacing failure. If an upstream 429 unrelated to pacing carries its own
  hint of up to 60s, the SDK spends that instead, and no bound clodex derives can prevent it. That
  is pre-existing and outside this feature's control, but it means the inequality is a guarantee
  about clodex's own contribution to the delay, not about total elapsed time.
• A cancellation refunds its token without rescheduling reservations already queued behind it, so
  each one permits one extra admission at that instant. The bound is on reservations, not wall-clock
  departures: a stalled event loop releases overdue timers together and the pacer neither observes
  nor corrects for that.

One test-harness defect found and fixed along the way

A concurrency test showed two connections constructed but only one in the fake socket registry. That
looked like it could be a socket leak in a connection-management change, so it was chased rather than
waved off. Instrumenting the product side settled it: createConnection ran twice and both
constructions returned, but the fake constructor ran once — the second call got the real ws
module. Two concurrent dynamic imports of a mocked module race in vitest, and one resolved unmocked,
which also meant the suite was quietly constructing a real socket to wss://chatgpt.com. Resolving
the mock before the concurrent phase makes the counts agree exactly. Harness, not product — the
product created exactly the two connections it should. Any future test that fans out two wsFetch
calls needs the same warm-up.

Rebased onto #171 (done — this is no longer a merge-order hazard)

#171 has landed. This branch was cut before it and has been rebased onto current main, which
includes #171, #175, #167 and the 2.9.0 release. Neither PR's CI had ever run against the other, so the
combination was treated as untested and re-verified from scratch rather than trusted. (2.9.0 went
out without this change so the 2.1.260 patch fix could ship; this lands in the next release.)

What #171 changed underneath this branch. It deleted upstreamMaxRetries() (which returned
number | undefined) in favour of
upstreamRequestBudget(options?): { idleTimeoutMs, totalTimeoutMs, maxRetries }, which always
returns numbers — so the old upstreamMaxRetries() ?? SDK_DEFAULT_MAX_RETRIES fallback would not
merely have failed to compile, it had no meaning left. It also raised the default retry count to 5
and made the idle timeout, total timeout and retry count user-configurable. That changes both
inputs to this feature's bound inequality, so the bound could no longer be derived from constants
the way this branch derived it.

What the reconciliation does. src/oauth/ws-upgrade-pacer.ts now resolves the deadline and the
retry count together from a single upstreamRequestBudget() call — the same call every SDK
generation entry point makes — instead of assuming a fixed 120s deadline. The constructor reads the
environment only when a term is not injected, so a fully injected test pacer never depends on
ambient configuration. ASSUMED_UPSTREAM_IDLE_TIMEOUT_MS and SDK_DEFAULT_MAX_RETRIES are gone.

The bound tightens on its own, which is the point of reading rather than assuming. At the
shipped defaults it goes from 15s to 4833ms: five retries cost a 62s backoff ladder, leaving 58s for
six attempts, half of which is reserved for the provider's own first byte.

Why the inequality now holds for every configuration rather than by coincidence at one of them:
pacing takes at most half of what the ladder leaves, so
attempts × bound + backoff ≤ (idle + backoff) / 2 < idle whenever backoff < idle, and
upstreamRequestBudget guarantees that side condition by capping maxRetries at the largest ladder
fitting the resolved deadline. Worst cases, all executed:

configuration	resolved budget	bound	ladder vs deadline
defaults	idle 120000, retries 5	4833	6×4833 + 62000 = 90998 < 120000
IDLE=30000	idle 30000, retries 3 (capped by the deadline)	2000	4×2000 + 14000 = 22000 < 30000
IDLE=10000 (floor)	idle 10000, retries 2 (capped)	666	3×666 + 6000 = 7998 < 10000
IDLE=3600000 (ceiling)	idle 3600000, retries 5	15000 (at the ceiling)	6×15000 + 62000 = 152000 < 3600000
IDLE=600000 TOTAL=60000	idle 60000, retries 4 — the pair rule lowered the idle timeout, which lowered the retry cap with it	3000	5×3000 + 30000 = 45000 < 60000
IDLE=14001	idle 14001, retries 3	0	pacing disables itself
MAX_RETRIES=0	idle 120000, retries 0	15000	1×15000 + 0 = 15000 < 120000
MAX_RETRIES=99 IDLE=3600000	idle 3600000, retries 10 (clamped)	15000	11×15000 + 2046000 = 2211000 < 3600000

Every row above is machine-produced from the real upstreamRequestBudget and
wsNewConnectionMaxWaitMs, not computed by hand — one row was wrong when I did compute it by hand.

The IDLE=14001 row is the degradation case, and it is the one #171 made reachable: a deadline
barely wider than its own 14000ms backoff ladder leaves nothing to queue in, so the bound floors to
zero and the pacer turns itself off with a notice rather than refusing everything past the burst.
Confirmed in a real run, not just in a test — see the smoke matrix below.

Doc conflicts. README.md and .claude/docs/oauth-continuation.md conflicted. #171's rewritten
"Provider timeouts" and "Upstream retries" bullets are kept verbatim and this branch's now-stale
pre-#171 "Upstream retries" bullet was dropped; the pacing bullet and pacing section are kept and
their bound arithmetic updated.

One harness is partially repaired, and I am not claiming more than that.
.claude/harnesses/fix-parent-notice-tui-and-epipe-round2.harness.ts called upstreamMaxRetries,
the last tracked reference to the symbol #171 deleted; that call is updated. It still does not
execute, for an unrelated and older reason: it imports
tests/helpers/register-ts-resolve-hook.mjs, which was never committed — git log --all for
that path is empty and no equivalent hook exists in the tree, so both probes have exited 1 at
module resolution since #117. I declined to fabricate an untested resolve hook to revive a harness
neither PR touches; the reference site now records why it cannot run. It is outside tsconfig's
include, so nothing typechecks or runs it either way.

Review coverage, stated honestly

Five adversarial reviews ran against this branch: a deadlock/starvation lens, a
test-discrimination lens, a counterexample lens against the refusal semantics, and a cold review of
the final state. Their blocking findings are what produced the current design — the rejected
admit-anyway table above, the no-debit refusal, the derived bound, and the classification demotion
all came out of them.

The fifth, a claims-and-documentation audit, reported after the first push and returned
FIX-BEFORE-MERGE on prose alone — no code defects. Its findings have been applied: the commit
summary no longer promises that rate-limit errors stop; correlation is no longer stated as cause in
the module header, the deep doc, the README or the user-facing error string; "every new connection"
became "primary" to match the documented exemptions; the admission bound is now scoped to
retries-enabled; "will be retried" is qualified because a refusal on the final attempt is terminal;
the burst rationale no longer cites a statistic the header says distinguishes nothing; and four test
names that claimed more than they proved were renamed.

A sixth ran against the rebased tree, cross-family (OpenAI) against an Anthropic author,
pointed specifically at the derived-bound arithmetic under user-configured timeouts and at whether
any claim here outruns its evidence. It ran three rounds, each against the previous round's fix,
and each found something the previous round's green suite did not:

1. the Retry-After blocker — the SDK substitutes the hint for its own rung, invalidating the
   bound derivation. Five earlier reviews and a green suite had all missed it, because every
   one of them, mine included, checked the ladder the bound budgeted rather than the delay the SDK
   actually takes.
2. the low-rate blocker the fix for (1) introduced, plus the false oracle that would have let (1)
   return undetected, plus the safety proof that was still false as written.
3. two more mutations that survived a green suite, both test-discrimination gaps rather than
   implementation defects, plus the disproof of my "no hint strategy can fix this" claim.
4. final verification.

Independent work by that reviewer that this PR did not do and should be credited: a behavioural
matrix over 52,800 resolved budget/rate combinations confirming that whenever the pacer refuses an
overflow request, retrying on its emitted hint always eventually reaches an admission; and
real-pacer/real-SDK runs confirming both blockers closed with elapsed times matching the derived
queue bounds.

Rounds 2 to 4 are the argument for not pushing after round 1. The fix for a blocker is exactly
where the next blocker comes from, and the suite was green after every single round — 2080,
2224, 2236 and 2260 tests, all passing, with a live defect present at three of those four points.

The lesson generalises past this PR: the inequality was verified against a model of the retry
policy, not against the retry policy — and then the test for the fix was verified against a model
of itself. The fix is pinned by tests that encode the SDK's real substitution rule, but they still
re-encode the 2s/4s/8s ladder rather than integrating with the SDK, so a change to the SDK's
backoff policy would not turn them red. That limitation is listed below and is unchanged.

One finding is worth calling out as a process failure on my part. It found the sentence claiming
the SDK's backoff "waits OUTSIDE the request's no-data deadline" still present in
ws-upgrade-pacer.ts after I had reported verifying it was gone. It was: a reviewer's
mutate-and-restore reinstated it from a snapshot taken before the correction, and my verification
predated the restore. Every claim in this PR was therefore re-audited from scratch afterwards rather
than trusted from an earlier check.

Failure and rollback

CLODEX_WS_MAX_NEW_CONNECTIONS_PER_MIN=0 disables pacing completely and restores today's behaviour.
If the derived bound leaves no room to queue, the pacer disables itself and says so rather than
turning off silently. With CLODEX_UPSTREAM_MAX_RETRIES=0 it cannot refuse — nothing would retry a
refusal — so it shapes the opening burst and then stops limiting; that is documented as a limitation,
not sold as a guarantee.

[bman654]

Follow-ups filed from this review

Both were found while reviewing this change and are deliberately not fixed here.

• Pacing: a queued request does not re-match a head that freed up while it waited #173 — a queued request does not re-match a head that freed up while it waited, so it opens a
  duplicate persistent head and can evict another conversation's. It partially defeats this feature
  under its own target workload. Not fixed here because it changes head selection in a file under
  the prime directive and needs its own tests.
• Throttled upgrades retry in a synchronised herd: the 403 path sends no retry-after header #174 — pre-existing, and likely a bigger real-world win than this PR for the 403 case. The real
  upgrade-403 path sends no retry-after header, and the SDK's error schema strips the
  retry_after_seconds it does send, so throttled requests all retry on the same un-jittered ladder.
  This PR fixes that for its own synthetic refusals; the genuine throttle path still has it. The
  value is already parsed at that site and simply never reaches the response headers.