Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions devlog/_plan/260915_workflow_budget_window/000_unit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# 260915 — the root workflow budget outlives the task it was sized for

## What happened

A Codex session spent several hours dispatching subagents. Every dispatch failed,
across three unrelated providers, with a 429 that reads as a provider rate limit.
The obvious readings were all wrong: not the model, not the account, not the
upstream. The refusal came from this proxy.

The reproduction is one line. With the same request body, a probe carrying the
long-running session id in `x-codex-parent-thread-id` was refused, and a probe
carrying a freshly invented root id was served. Restarting the proxy served both.
That is the whole diagnosis: the ceiling is per root, held in process memory, and
the session had reached it.

## Why the ceiling fired

`DEFAULT_WORKFLOW_BUDGET_POLICY` (`src/lib/workflow-budget.ts`) caps a root at 256
physical sends and 64 distinct children. `workflowSendCeilingReached` compares
`state.sends >= policy.maxPhysicalSends`, and `state.sends` is **cumulative for the
life of the process**. The root id is `x-codex-parent-thread-id`, which for Codex is
the session. So the cap is not a fan-out guard on a long session; it is an expiry.

The comment that justifies it says a per-request cap "cannot bound a fan-out that
sends once per child seven hundred times". That is a **burst** concern, and a burst
is bounded by a rate. A lifetime total cannot tell seven hundred sends in a minute
from two hundred and fifty-six sends spread over four hours, and it refuses both.
The second one is ordinary work.

Two things made it expensive to diagnose rather than merely annoying. The refusal
is a 429 that an operator reads as an upstream rate limit, so the first hours went
to providers and accounts. And there is no way out except restarting the proxy:
`resetWorkflowBudgetsForTest` exists, the name says who it is for, and
`workflowBudgetSnapshot` is never exposed, so the state that decided the refusal is
invisible from outside the process.

## The rule

> A root budget bounds a **rate**, and says so. A ceiling that fires is a local
> decision an operator can see, name and clear without restarting the proxy.

## Roadmap

| Doc | Work phase | Outcome |
| --- | --- | --- |
| `010_windowed_ceilings.md` | wfb | Sends and distinct children are counted over a bounded window, so a long session is never refused for work it did hours ago while a burst inside one window still is |
| `020_legible_refusal.md` | wfc | The refusal names the ceiling that fired, is marked as a proxy decision rather than an upstream one, and the root budget can be read and cleared through the management API |

## Write scope

Permitted: `src/lib/workflow-budget.ts`, the workflow call sites in
`src/server/responses/core.ts` and `src/server/index.ts`, the management read and
mutation surface under `src/server/management/`, `src/server/request-log.ts` for the
refusal provenance, their tests, and this unit.

## Verification posture

Local suite, typecheck, install and GUI build are **not run** for this unit by
explicit instruction. Proof is hosted CI at the exact final head SHA and nothing
else. Pushes use `--no-verify`.

## What would make this fail

Raising the numbers instead of fixing the shape. A bigger lifetime total is the
same defect further away: it still refuses a session for work it finished hours
ago, and it still cannot be seen or cleared. The window is the change; the numbers
are a consequence of it.


## Reproducing it

Both probes carry the same body and differ only in the root id. Against a proxy
whose process has been up long enough for a session to reach the ceiling:

```bash
BODY='{"model":"gpt-5.6-terra","input":[{"role":"user","content":[{"type":"input_text","text":"ok"}]}],"max_output_tokens":16,"stream":true}'

# the long-running session's own root: refused
curl -s -o /dev/null -w '%{http_code}\n' -N -X POST http://127.0.0.1:10100/v1/responses \
-H 'Content-Type: application/json' -H 'Accept: text/event-stream' \
-H "x-codex-parent-thread-id: <that session id>" -d "$BODY"

# any root the process has not seen: served
curl -s -o /dev/null -w '%{http_code}\n' -N -X POST http://127.0.0.1:10100/v1/responses \
-H 'Content-Type: application/json' -H 'Accept: text/event-stream' \
-H "x-codex-parent-thread-id: probe-$(date +%s)" -d "$BODY"
```

Two answers from one proxy, one body and one upstream, separated only by which
root the request claims. That is what rules out the provider, the account and the
model in a single step, and it is the check to run first the next time a fan-out
starts failing for no visible reason.

After a restart both return 200, which is the other half of the diagnosis: the
ceiling is process-memory only, so the evidence disappears the moment anyone tries
the obvious remedy.

Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# 010 — wfb: count sends and children over a window, not over a lifetime

## Today

```ts
export function workflowSendCeilingReached(rootId, policy) {
const state = roots.get(rootId);
return state !== undefined && state.sends >= policy.maxPhysicalSends;
}
```

`state.sends` only ever grows. `state.children` is a Set that only ever gains
members. Neither has a clock. A root that made 256 sends in its first hour is
refused for the rest of the process even if it sends nothing for a day.

## The change

Keep the counters, add a window. A root records its sends as timestamped buckets
and the ceiling compares the count **inside the window** against
`maxPhysicalSends`. Distinct children get the same treatment: a child seen once,
hours ago, and never again should not hold a slot forever.

The default window has to be argued for rather than picked. 256 sends is the
number already in the tree and it was chosen against a fan-out, so the window is
the interval over which that fan-out would be abusive. A ten-minute window keeps
the original intent — seven hundred sends in a minute is still refused several
times over — while an ordinary session that averages well under a send every two
seconds never approaches it.

`maxConcurrentChildren` stays as it is. Concurrency is already instantaneous; it
has no lifetime problem to fix.

## What must not change

An unconfigured install must not see a refusal it would not have seen before.
Windowing only ever admits more, never less, for the same traffic — the count
inside a window is bounded by the lifetime count — so this direction is safe by
construction. Say so in a test rather than trusting the argument.

The eviction rules from #4546 stay: a root is evicted only when it is both
inactive and not exhausted, and a full table refuses rather than laundering a
fan-out into a fresh allowance. A windowed root that has aged out of its window is
no longer exhausted, which is exactly the state that makes it evictable again.

## Acceptance

1. A root at the ceiling is admitted once its window rolls, without a restart.
2. A burst inside one window is still refused at the same count as before.
3. Distinct children age out of the window the same way sends do.
4. Bucket storage per root is bounded; a root that sends forever does not grow
forever.

50 changes: 50 additions & 0 deletions devlog/_plan/260915_workflow_budget_window/020_legible_refusal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# 020 — wfc: a refusal an operator can read, name and clear

## Today

The refusal is a 429 with `workflow_budget_exhausted` and a sentence about the
task's send budget. Two problems, and the first one cost hours.

A 429 from a proxy that also forwards provider 429s is ambiguous. There is nothing
on the record that says which one this was, so the first move is always to go look
at the provider. This is the same defect #4639 fixed for the synthetic 503: a
locally generated refusal presented under a field an operator reads as upstream.
The fix there was provenance on the record, and it applies unchanged here.

The second is that there is no way out. `resetWorkflowBudgetsForTest` is named for
its audience and `workflowBudgetSnapshot` has no caller outside the module, so the
state that decided the refusal cannot be read and cannot be cleared except by
restarting the proxy — which drops every other root's accounting with it.

## The change

Name the ceiling. The denial type already distinguishes
`workflow-sends-exhausted` from `workflow-children-exhausted` and the rest; carry
that through to the error body and onto the request log instead of collapsing it
into one sentence.

Mark it local. The request log gains the same origin treatment #4639 introduced, so
a proxy refusal and an upstream 429 are distinguishable on the record and on the
management read surface.

Expose and allow clearing. `GET` the root's budget through the management API so an
operator can see a ceiling approaching rather than discovering it, and allow a
bounded, recorded clear of one root. Clearing one root is not the same as
restarting: it is scoped, it is logged, and it leaves every other root's accounting
intact.

## What must not change

The clear is an operator action on the operator's own proxy, not a path a request
can take. It goes through the management surface, which already requires a
dashboard session or the admin token, and it must not be reachable from the data
plane. A fan-out cannot be allowed to clear its own ceiling — that would make the
budget a suggestion, which is the failure #4546 spent a release removing.

## Acceptance

1. The refusal body and the request log name which ceiling fired.
2. The record marks the refusal as proxy-origin, distinguishable from an upstream 429.
3. An operator can read one root's budget and clear it through the management API.
4. The clear is scoped to one root, is recorded, and is not reachable from the data plane.

2 changes: 1 addition & 1 deletion docs-site/src/content/docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ does not change `main`/`preview` review rules or allow direct pushes, force-push

## Adding a provider to the catalog

All provider pickers and seeds derive from the canonical registry (`src/providers/registry.ts`):
All provider pickers and seeds derive from the canonical registry (`src/providers/registry/entries-extended.ts`):

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/fr/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ du dépôt et des chemins sensibles du point de vue de la sécurité est déclar

## Ajout d'un fournisseur au catalogue

Tous les sélecteurs de fournisseurs et les graines proviennent du registre canonique (`src/providers/registry.ts`) :
Tous les sélecteurs de fournisseurs et les graines proviennent du registre canonique (`src/providers/registry/entries-extended.ts`) :

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/ja/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ Go ネイティブポートを担っていた `dev2-go` は廃止し、2 本の

## カタログにプロバイダーを追加

すべてのプロバイダー選択肢と seed は canonical レジストリ(`src/providers/registry.ts`)から派生します。
すべてのプロバイダー選択肢と seed は canonical レジストリ(`src/providers/registry/entries-extended.ts`)から派生します。

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/ko/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ Go 네이티브 포트를 담당했던 `dev2-go`는 정리했고, 두 라인을

## 카탈로그에 프로바이더 추가하기

모든 프로바이더 선택기와 seed는 canonical registry(`src/providers/registry.ts`)에서 파생됩니다.
모든 프로바이더 선택기와 seed는 canonical registry(`src/providers/registry/entries-extended.ts`)에서 파생됩니다.

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/ru/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Pull request'ы с ребейзом приветствуются: ребейз
## Добавление провайдера в каталог

Все селекторы провайдеров и seed-данные выводятся из канонического реестра
(`src/providers/registry.ts`):
(`src/providers/registry/entries-extended.ts`):

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/tr/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ sahipliği `.github/CODEOWNERS` dosyasında bildirilmiştir.
## Kataloğa sağlayıcı ekleme

Tüm sağlayıcı seçicileri ve tohumları kurallı kayıt defterinden
(`src/providers/registry.ts`) türetilir:
(`src/providers/registry/entries-extended.ts`) türetilir:

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/zh-cn/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ bun run release:watch # 观察最新的 Release workflow run

## 向目录中添加 provider

所有 provider picker 与 seed 都来自 canonical registry(`src/providers/registry.ts`):
所有 provider picker 与 seed 都来自 canonical registry(`src/providers/registry/entries-extended.ts`):

```ts
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/zh-tw/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ bun run release:watch # 觀察最新的 Release workflow run

## 向目錄中新增 provider

所有 provider picker 與 seed 都來自 canonical registry(`src/providers/registry.ts`):
所有 provider picker 與 seed 都來自 canonical registry(`src/providers/registry/entries-extended.ts`):

```ts
{
Expand Down
Loading
Loading