diff --git a/devlog/_plan/260911_account_pool_unification/020_phase2_shared_kernel.md b/devlog/_plan/260911_account_pool_unification/020_phase2_shared_kernel.md index 3b0fc74ba3..fd0b21c651 100644 --- a/devlog/_plan/260911_account_pool_unification/020_phase2_shared_kernel.md +++ b/devlog/_plan/260911_account_pool_unification/020_phase2_shared_kernel.md @@ -217,3 +217,205 @@ starts from them rather than rediscovering them. - `tests/adapters/anthropic/anthropic-account-pool.test.ts` — parity - `tests/providers/kiro/kiro-pool-rank.test.ts` — the kiro exhaustion special case in `account-quota-rank.ts:84-108` survives + +## wp2b implementation plan (re-verified against `dev` 29d632ff2) + +Every anchor below was re-read on the post-merge tree, after #4275/#4277/#4279/#4284 landed. + +| Symbol | File | Line | +|---|---|---| +| `isProactivePreferenceEnabled` | `src/oauth/generic-account-failover.ts` | 150 | +| `rotateGenericOAuthAccountOn429` | `src/oauth/generic-account-failover.ts` | 178 | +| `preferredInitialAccount` | `src/oauth/generic-account-failover.ts` | 246 | +| `forgetGenericFailoverRoster` | `src/oauth/generic-account-failover.ts` | 308 | +| `GenericPoolSettingsDto` / `inert: true` | `src/oauth/pool-settings-capability.ts` | 40 / 54, 65 | +| `PUT /api/oauth/accounts/active` | `src/server/management/oauth-account-routes.ts` | 325 | +| generic GET / PUT DTO | `src/server/management/oauth-account-routes.ts` | 360 / 422 | +| `stickyLimit` 400 | `src/server/management/oauth-account-routes.ts` | 396 | +| `genericPoolKey` / `pickRoundRobinAccount` / `peekRoundRobinAccount` / `notePoolRotationSuccess` | `src/oauth/pool-kernel.ts` | 12 / 198 / 210 / 222 | +| `genericFailoverAccountId = resolved.accountId` | `src/server/responses/core.ts` | 4407 | +| per-provider `oauthAccountFailover` | `src/types/provider.ts` | 520 | + +### The question 020 left open: where does a round-robin proposal commit? + +`peekRoundRobinAccount` exists and does not advance the ring, which is correct for +`preferredInitialAccount` — that answer is discardable, and the resolver drops it when the +account turns out to be removed, reauth-flagged, or missing a Cloud Code Assist project. But +a peek that never commits is a ring that never turns: every request would propose the same +account forever, and "round-robin" would be a label on a constant. + +So a commit site is mandatory, and it has to be the admission point, not the proposal. That +point already exists and already has a generic-only branch: + +``` +src/server/responses/core.ts:4405-4408 + if (isGenericFailoverProvider(route.providerName, route.provider)) { + genericFailoverAccountId = resolved.accountId; + } +``` + +One line joins it: `noteGenericPoolSelection(config, route.providerName, resolved.accountId)`. +The function lives in `generic-account-failover.ts` and does the flag read, the strategy read +and the `notePoolRotationSuccess(genericPoolKey(name), id, stickyLimit)` call itself. No policy +moves into `core.ts`, the import comes from a module `core.ts` already imports from, and the +core-path Lab boundary is untouched — `pool-kernel.ts` pulls only two types. + +This is the one file in the unit that sits on every user's request path, so it takes exactly +one statement and no branching of its own. + +### Change surface + +**`src/types/config.ts`** — add `pool?: { kernel?: boolean }` beside the existing optional flag +objects (`resetCreditAutoRedeem` at :833 is the nearest shape). **`src/config.ts`** — add +`pool: z.object({ kernel: z.boolean().optional() }).optional().catch(undefined)` next to +`resetCreditAutoRedeem` at :1304. `.catch(undefined)` matches the house rule: a malformed hand +edit turns the feature off rather than costing the operator their providers. + +**`src/types/provider.ts`** — add `stickyLimit?: number` to the per-provider +`oauthAccountFailover` block at :520, with the same 1..100 range the Anthropic pool documents. + +**`src/oauth/generic-account-failover.ts`** — branch BOTH paths on strategy, because branching +one leaves the setting inert in practice: + +| Strategy | `preferredInitialAccount` | `rotateGenericOAuthAccountOn429` | +|---|---|---| +| flag off, or absent/`quota` | unchanged: healthy-active return :262, `hasHeadroomEvidence` :267, `rankAccountsByHeadroom` | unchanged: ring after the failed id, then `rankAccountsByHeadroom` | +| `round-robin` | skip BOTH guards, `peekRoundRobinAccount(genericPoolKey(name), eligible, stickyLimit)` | `pickRoundRobinAccount` over the eligible ring | +| `fill-first` | skip the healthy-active return; keep active while its usage is under `autoSwitchThreshold`, else advance to the next eligible account | must NOT keep the failed account: advance to the next eligible one | + +The two guards are skipped deliberately and for different reasons, both measured in 020's audit: +`hasHeadroomEvidence` returns false for any provider with no quota data, so leaving it in front +of round-robin makes round-robin unreachable exactly where it is most useful; and the +healthy-active early return fires before `autoSwitchThreshold` can ever be read, so fill-first +would never reach its own threshold test. Keep the presence quorum, the `EXCLUDED_PROVIDERS` +guard and the per-provider `health` cooldown on every branch. + +**`src/oauth/pool-settings-capability.ts`** — `inert` becomes `boolean` computed from the flag +instead of the literal `true`. `genericPoolSettingsDto` takes the flag as a third argument +rather than reading config itself, so the DTO stays a pure projection. + +**`src/server/management/oauth-account-routes.ts`** — three edits. The active PUT at :325 gains +`seedPoolRotationAccount(genericPoolKey(provider), accountId)` beside `forgetGenericFailoverRoster`, +or the operator's pick immediately loses to sticky rotation — the same defect wp1b just fixed on +the Codex side, and `forgetGenericFailoverRoster` only drops the presence count, never the +cursor. The 400 at :396 narrows to `quotaWindow` alone. The pool PUT accepts and persists +`stickyLimit` with the 1..100 validation. + +**`src/cli/account-extended.ts`** — the generic branch at :395 currently hardcodes +`const enabled = false`. With the kernel on it reports the real state. + +### Acceptance + +Criterion c-3: a test asserts a configured strategy actually changes the selected account, and +the DTO stops reporting `inert` once the flag is on. + +- `tests/oauth/generic-oauth-failover.test.ts` — round-robin rotates across dispatches for a + provider with NO quota data (the case the evidence guard blocks today); fill-first holds the + active account under threshold and advances over it; quota is byte-identical to today; every + one of them is a no-op with `pool.kernel` off. +- `tests/server/account-pool-management-api.test.ts` — `inert` follows the flag, `stickyLimit` + round-trips, `quotaWindow` still 400s. The existing marker test at :435 reads the source for + the literal `inert: true;` and moves with the type. +- `tests/cli/cli-account-pool-verbs.test.ts` — the CLI reports the live threshold when on. +- Red control for each new case, as in wp1b: the assertion must fail with its production branch + removed. A test that passes either way is not coverage. + +### Reversibility + +`pool.kernel` defaults off, and off means the pre-kernel code path byte for byte: the guards +stay, the DTO still says `inert: true`, and `noteGenericPoolSelection` returns before touching +the ring. No migration writes on upgrade; the kernel reads keys that are already persisted. + +### A-phase findings folded into this plan + +Verified while auditing the plan above, before any code was written. + +**The `inert` contract is published, in seven languages.** Turning `inert` into a computed +field makes live documentation false, and AGENTS.md requires docs-site to stay in sync and +translated locales not to contradict the English source. The statements that change: +`docs-site/src/content/docs/reference/configuration/providers.md` :568 ("the generic selector +does not act on it yet, so omitted and set behave the same today"), :569 ("inert until the +selector consumes it") and :590 ("`inert: true` for those two fields only"); and +`reference/cli/providers-accounts.md` :351 ("Generic pool thresholds are currently inert") and +:355, whose signature literally reads `inert: true | null`. The same page exists under +`ko`, `ja`, `fr`, `ru`, `tr`, `zh-cn` and `zh-tw`. All of it moves in this PR: a flag-gated +feature still has to describe both states, not the old one. + +**The DTO marker test fails OPEN, which is worse than failing.** +`tests/server/account-pool-management-api.test.ts:435` locates its slice with +`source.indexOf("inert: true;", start)`. Once the type reads `inert: boolean;` that returns +`-1`, and `source.slice(start, -1)` happily returns almost the whole file — which still +contains "strategy", "autoSwitchThreshold" and "enabled", so all three assertions pass while +the test has stopped checking anything. It must be rewritten against the new literal, not +merely allowed to keep passing. This is the same failure mode wp1b was built on, so it gets +named rather than discovered later. + +**`src/server/management/provider-routes.ts`:1023-1024 is a reader the plan did not name.** +It carries `oauthAccountFailover` forward when a provider is overwritten, to stop an edit +silently enabling rotation. It copies the whole object, so a new `stickyLimit` rides along +with no change — verified, listed here so the next reader does not have to re-derive it. + +**The core-path import edge is already there.** `src/server/responses/core.ts` imports from +`../../oauth/generic-account-failover` at :150, so adding `noteGenericPoolSelection` to that +existing import creates no new module edge at all, and `pool-kernel.ts` imports only two +types. `bun test tests/lab/core-lab-boundary.test.ts` is green at 17 pass / 0 fail on this +branch and is re-run at Check. + +**Fill-first's stable order is `eligibleFailoverAccounts`:164**, which preserves +`set.accounts` order from the store and filters out reauth-flagged and cooled accounts. That +is the order the 429 ring already walks, so fill-first advances through the same sequence +rather than inventing a second one. + +### Plan audit round 2 — FAIL, three blockers folded + +A dispatched reviewer returned FAIL on the plan above. All three blockers are real and two of +them contradict what this document said one revision earlier. Recorded rather than quietly +edited, because the corrections are the useful part. + +**Blocker 1 — fill-first must walk the SORTED FULL roster, not the eligible subset.** +The "A-phase findings" note above claimed `eligibleFailoverAccounts`:164 is the order +fill-first advances through. That is wrong, and it is the exact bug 020's own earlier audit +already rejected when it added the `stableAll` argument to `pickFillFirst`. Both shipped +copies walk a stable roster sorted with `localeCompare` — `src/codex/routing.ts`:1443 and +`src/oauth/anthropic-routing.ts`:427 — and dropping to the eligible subset changes the wrap +order whenever an ineligible id sits between two eligible ones. The generic roster is worse +than unsorted-by-accident: `getAccountSet().accounts` is in LOGIN order, so two operators who +added the same accounts in a different sequence would get different rotation. The generic +fill-first sorts the full roster the same way, then skips ineligible ids while walking it. +Supersedes the paragraph above. + +**Blocker 2 — the commit site fires on every generic dispatch, so it must gate on +round-robin specifically.** `core.ts`:4407 is reached on every generic first dispatch, +including the preferred-null quota path and the fallback after a preferred account is dropped +at :4368-4388. The plan said `noteGenericPoolSelection` "does the flag read and the strategy +read" without saying what it does with them, which is not precise enough to implement: an +ungated call would advance round-robin sticky state for quota and fill-first pools too. +It returns immediately unless `pool.kernel` is on AND the resolved strategy is +`round-robin`. Anthropic already draws exactly this line — `anthropic-routing.ts`:791 notes +rotation only on its round-robin branch — so this is matching an existing contract, not +inventing one. + +**Blocker 3 — the CLI has three states, not two.** `src/cli/account-extended.ts`:402-409 +prints "unavailable" and "threshold support is unknown" whenever `inert !== true`, so a +kernel-on `inert: false` would render the live feature as an unknown capability — the +opposite of the truth. `tests/cli/cli-account-pool-verbs.test.ts`:393-403 also feeds +`inert: false` through a malformed-capability loop that expects `enabled: false`. The CLI +needs `true` (stored, not applied), `false` (applied) and `null`/absent (unknown) as three +distinct renderings, and that test's fixture must stop conflating the middle one with +malformed input. + +**Major folded — an exact-equality DTO assertion.** +`tests/server/account-pool-management-api.test.ts`:477 asserts the generic GET body with +`toEqual`, so adding `stickyLimit` breaks it. :484 uses `toMatchObject` and is safe. The PUT +round-trip at :486 breaks only once PUT actually persists the field. All three move with the +change. + +**Major folded — `src/cli/capabilities.ts`:331** also publishes the inert contract, alongside +the docs-site pages already listed. + +**Correction — anchor.** This document cited `hasHeadroomEvidence` at :267; that is where its +comment begins. The call is at :272. The anchor table itself was verified correct. + +**Confirmed, no action —** the reviewer independently reached the same conclusion on the Lab +boundary: `core.ts` already imports `generic-account-failover`, and `pool-kernel.ts` is +`import type` only, which the boundary walker skips. No new edge. diff --git a/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md b/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md index c41c2ee7cd..184923941f 100644 --- a/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md @@ -208,11 +208,11 @@ renvoient 1 ; une sonde de quota en amont qui échoue ou expire produit plutôt ### `ocx account auto-switch > [--json]` -Contrôle le seuil du pool Codex `openai`, ou enregistre celui d’un pool OAuth générique. `on` enregistre 80 %, `off` 0 % et `threshold ` accepte 0–100. Les seuils génériques sont actuellement inactifs : leur sauvegarde ne change ni le basculement par seuil, ni l’activation du fournisseur, ni la rotation réactive après une erreur 429. Pour les pools génériques, les sorties utilisent la réponse confirmée du serveur. Pour un pool générique, `poolEnabled` est le réglage enregistré (`null` signifie non spécifié), pas l’état effectif hérité. `inert: true` indique que le seuil ne s’applique pas ; une capacité inconnue ne produit jamais `enabled: true`. Les fournisseurs à clé API, Anthropic et les valeurs invalides sont refusés. +Contrôle le seuil du pool Codex `openai`, ou enregistre celui d’un pool OAuth générique. `on` enregistre 80 %, `off` 0 % et `threshold ` accepte 0–100. Un seuil générique n’oriente la sélection que si `pool.kernel` est activé avec `strategy: "fill-first"` ; le drapeau désactivé, sa sauvegarde n’active pas le basculement par seuil. Dans les deux cas, elle ne change ni l’activation du fournisseur, ni la rotation réactive après une erreur 429. Pour les pools génériques, les sorties utilisent la réponse confirmée du serveur. Pour un pool générique, `poolEnabled` est le réglage enregistré (`null` signifie non spécifié), pas l’état effectif hérité. `inert: true` indique un seuil enregistré mais non appliqué, `inert: false` un seuil que le pool applique réellement. L’absence d’`inert` signale une capacité inconnue, qui ne produit jamais `enabled: true`. Les fournisseurs à clé API, Anthropic et les valeurs invalides sont refusés. ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md index 3557cd5ea7..dc5fd0dccf 100644 --- a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md @@ -152,11 +152,11 @@ OAuth プロバイダーと API キー プロバイダーの場合、これに ### `ocx account auto-switch > [--json]` -`openai` Codex プールのしきい値を制御するか、汎用 OAuth プールのしきい値を保存します。`on` は 80%、`off` は 0%、`threshold ` は 0–100 を保存します。汎用プールのしきい値は現在適用されません。保存しても、しきい値による切り替え、プロバイダーの有効化設定、429 エラー時のローテーションは変更されません。汎用プールの照会と変更の結果はサーバーの確認値を使用します。汎用プールの `poolEnabled` は保存された設定で、`null` は未指定です。継承後の実効状態ではありません。`inert: true` は未適用を示し、機能が不明な場合も `enabled: true` とは表示しません。API キープロバイダー、Anthropic、不正な値は拒否されます。 +`openai` Codex プールのしきい値を制御するか、汎用 OAuth プールのしきい値を保存します。`on` は 80%、`off` は 0%、`threshold ` は 0–100 を保存します。汎用プールのしきい値は `pool.kernel` が有効で `strategy: "fill-first"` の場合にのみ選択へ反映されます。フラグが無効なら、保存してもしきい値による切り替えは有効になりません。いずれの場合もプロバイダーの有効化設定と 429 エラー時のローテーションは変更されません。汎用プールの照会と変更の結果はサーバーの確認値を使用します。汎用プールの `poolEnabled` は保存された設定で、`null` は未指定です。継承後の実効状態ではありません。`inert: true` は保存済みで未適用、`inert: false` はプールが適用中であることを示します。`inert` が無い場合は機能が不明であり、その場合も `enabled: true` とは表示しません。API キープロバイダー、Anthropic、不正な値は拒否されます。 ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md index c6eaeec910..5f691ae3ee 100644 --- a/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md @@ -218,11 +218,11 @@ OAuth 및 API 키 제공자에는 제공자의 할당량 보고 엔드포인트 ### `ocx account auto-switch > [--json]` -`openai` Codex 풀의 임계값을 제어하거나 일반 OAuth 풀의 임계값을 저장합니다. `on`은 80%, `off`는 0%, `threshold `은 0–100을 저장합니다. 일반 풀의 임계값은 현재 동작에 적용되지 않습니다. 저장해도 임계값 기반 전환이나 제공자 활성화 설정이 바뀌지 않고, 429 오류에 따른 회전도 비활성화되지 않습니다. 일반 풀의 조회와 변경 결과는 서버가 확인한 값을 사용합니다. 일반 풀의 `poolEnabled`는 저장된 제공자별 설정이며 `null`은 미지정입니다. 전역 설정을 상속한 실제 상태를 뜻하지 않습니다. `inert: true`이면 임계값이 적용되지 않으며, 기능 지원을 알 수 없을 때도 `enabled: true`로 표시하지 않습니다. API 키 제공자, Anthropic 및 잘못된 값은 거부합니다. +`openai` Codex 풀의 임계값을 제어하거나 일반 OAuth 풀의 임계값을 저장합니다. `on`은 80%, `off`는 0%, `threshold `은 0–100을 저장합니다. 일반 풀의 임계값은 `pool.kernel`이 켜져 있고 `strategy: "fill-first"`일 때만 선택에 반영됩니다. 플래그가 꺼져 있으면 저장해도 임계값 기반 전환이 켜지지 않습니다. 어느 쪽이든 제공자 활성화 설정은 바뀌지 않고, 429 오류에 따른 회전도 비활성화되지 않습니다. 일반 풀의 조회와 변경 결과는 서버가 확인한 값을 사용합니다. 일반 풀의 `poolEnabled`는 저장된 제공자별 설정이며 `null`은 미지정입니다. 전역 설정을 상속한 실제 상태를 뜻하지 않습니다. `inert: true`는 임계값이 저장만 되고 적용되지 않는 상태, `inert: false`는 풀이 실제로 적용하고 있는 상태를 뜻합니다. `inert`가 아예 없으면 기능 지원을 알 수 없는 경우이며, 이때도 `enabled: true`로 표시하지 않습니다. API 키 제공자, Anthropic 및 잘못된 값은 거부합니다. ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/reference/cli/providers-accounts.md b/docs-site/src/content/docs/reference/cli/providers-accounts.md index 7ec246a96f..18eab1a47d 100644 --- a/docs-site/src/content/docs/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/reference/cli/providers-accounts.md @@ -348,11 +348,11 @@ instead (exit 0), matching the dashboard's quota bars. ### `ocx account auto-switch > [--json]` -Controls the `openai` Codex pool threshold, or stores a threshold for a generic OAuth pool. `on` stores 80%, `off` stores 0%, and `threshold ` accepts 0–100. Generic pool thresholds are currently inert: saving one does not enable threshold-based switching, change the provider enablement override, or disable reactive 429 rotation. `status` and mutation output for generic pools use the confirmed server response. For generic pools, `poolEnabled` is the stored provider override (`null` means unspecified), not inherited effective state; `inert: true` means the threshold is not applied, and unknown capability never reports `enabled: true`. API-key providers, Anthropic and invalid values are rejected. +Controls the `openai` Codex pool threshold, or stores a threshold for a generic OAuth pool. `on` stores 80%, `off` stores 0%, and `threshold ` accepts 0–100. A generic pool threshold steers selection only while `pool.kernel` is on with `strategy: "fill-first"`; with the flag off, saving one does not enable threshold-based switching. It never changes the provider enablement override or disables reactive 429 rotation. `status` and mutation output for generic pools use the confirmed server response. For generic pools, `poolEnabled` is the stored provider override (`null` means unspecified), not inherited effective state; `inert: true` means the threshold is stored but not applied, `inert: false` means the pool is applying it, and an absent `inert` is an unknown capability, which never reports `enabled: true`. API-key providers, Anthropic and invalid values are rejected. ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 65e57f19ab..39b5bc7dfa 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -565,8 +565,9 @@ second account. | --- | --- | --- | --- | | `oauthAccountFailover.enabled?` | `boolean` | presence-driven | Global override for the **pre-dispatch account preference** only. `false` stops a healthy request being steered toward the account with more known headroom. It does **not** disable 429 rotation. | | `providers..oauthAccountFailover.enabled?` | `boolean` | inherits | Per-provider override for the same preference; beats the global setting in either direction. `false` declines the preference for this provider even when the global setting is `true`, and `true` opts this provider in even when the global setting is `false`. Reactive 429 rotation is unaffected either way. | -| `providers..oauthAccountFailover.strategy?` | `"quota" \| "round-robin" \| "fill-first"` | — | Declared pool strategy for a generic OAuth provider (#695). Persisted through `ocx account strategy ` or `PUT /api/oauth/accounts/pool`; the generic selector does not act on it yet, so omitted and set behave the same today. | -| `providers..oauthAccountFailover.autoSwitchThreshold?` | `number` | — | Declared 0–100 usage percent for a proactive switch on a generic OAuth provider (#695). Set with `ocx account auto-switch threshold `; inert until the selector consumes it. | +| `providers..oauthAccountFailover.strategy?` | `"quota" \| "round-robin" \| "fill-first"` | — | Pool strategy for a generic OAuth provider (#695). Persisted through `ocx account strategy ` or `PUT /api/oauth/accounts/pool`. The selector acts on it only while `pool.kernel` is on; with the flag off, omitted and set behave the same. `quota` is the pre-kernel behaviour either way. | +| `providers..oauthAccountFailover.autoSwitchThreshold?` | `number` | `80` | 0–100 usage percent at which `fill-first` advances off the active account (#695). Set with `ocx account auto-switch threshold `. Read only under `pool.kernel` with `strategy: "fill-first"`; an account with no measured usage counts as under the threshold. | +| `providers..oauthAccountFailover.stickyLimit?` | `number` | `1` | Successful dispatches retained on one `round-robin` selection, 1–100 (#695). Read only under `pool.kernel` with `strategy: "round-robin"`. | To decline proactive account steering for one provider whose terms you would rather not test, while still recovering from a rate limit: @@ -586,10 +587,10 @@ That setting survives logging in, adding an account, and reauthenticating. Generic OAuth providers (Google Antigravity, xAI, Cursor, Kimi, GitHub Copilot, Nous, and any other OAuth provider outside the Codex and Anthropic pools) also accept `strategy` and `autoSwitchThreshold` on the same key, through `GET`/`PUT /api/oauth/accounts/pool?provider=` -and the `ocx account strategy` / `ocx account auto-switch` verbs. The response carries -`"inert": true` for those two fields only — `enabled` is live and governs the pre-dispatch -preference. `stickyLimit` and -`quotaWindow` are not part of the generic contract. Codex (`/api/codex-auth`) and Anthropic +and the `ocx account strategy` / `ocx account auto-switch` / `ocx account sticky` verbs. The response carries +`"inert"` for those three fields only — `true` while they are stored but not consumed, +`false` once `pool.kernel` is on and they actually select an account — `enabled` is live and governs the pre-dispatch +preference. `quotaWindow` is not part of the generic contract. Codex (`/api/codex-auth`) and Anthropic (`anthropicAccountPool`) keep their own contracts unchanged. Deliberately narrower than `anthropicAccountPool`: no session affinity, no quota-ranked diff --git a/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md index 9ee81cb626..e612ee0529 100644 --- a/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md @@ -192,11 +192,11 @@ quota-bar'ов дашборда. ### `ocx account auto-switch > [--json]` -Управляет порогом пула Codex `openai` или сохраняет порог общего пула OAuth. `on` сохраняет 80 %, `off` — 0 %, а `threshold ` принимает 0–100. Пороги общих пулов пока не применяются: сохранение не включает переключение по порогу, не меняет настройку включения провайдера и не отключает ротацию после ошибки 429. Для общего пула результат чтения и изменения берётся из подтверждённого ответа сервера. Для общего пула `poolEnabled` — сохранённая настройка провайдера (`null` означает отсутствие настройки), а не итоговое унаследованное состояние. `inert: true` означает, что порог не применяется; неизвестная возможность также не даёт `enabled: true`. Провайдеры с ключом API, Anthropic и неверные значения отклоняются. +Управляет порогом пула Codex `openai` или сохраняет порог общего пула OAuth. `on` сохраняет 80 %, `off` — 0 %, а `threshold ` принимает 0–100. Порог общего пула влияет на выбор только при включённом `pool.kernel` и `strategy: "fill-first"`; при выключенном флаге сохранение не включает переключение по порогу. В обоих случаях оно не меняет настройку включения провайдера и не отключает ротацию после ошибки 429. Для общего пула результат чтения и изменения берётся из подтверждённого ответа сервера. Для общего пула `poolEnabled` — сохранённая настройка провайдера (`null` означает отсутствие настройки), а не итоговое унаследованное состояние. `inert: true` означает, что порог сохранён, но не применяется, а `inert: false` — что пул его применяет. Отсутствие `inert` означает неизвестную возможность, которая также не даёт `enabled: true`. Провайдеры с ключом API, Anthropic и неверные значения отклоняются. ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md b/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md index 729596215e..8ab2ca1647 100644 --- a/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md @@ -234,11 +234,11 @@ eşleşen null veya eski bir rapora düşer (çıkış 0). ### `ocx account auto-switch > [--json]` -`openai` Codex havuzunun eşiğini yönetir veya genel OAuth havuzunun eşiğini kaydeder. `on` %80, `off` %0 kaydeder; `threshold ` 0–100 kabul eder. Genel havuz eşikleri şu anda uygulanmaz: kayıt işlemi eşik tabanlı geçişi, sağlayıcının etkinlik ayarını veya 429 hatasından sonraki otomatik hesap değişimini etkilemez. Genel havuz çıktısı sunucunun doğruladığı değerleri kullanır. Genel havuzlarda `poolEnabled`, kaydedilmiş sağlayıcı ayarıdır (`null` belirtilmemiş demektir); devralınmış etkin durumu göstermez. `inert: true`, eşiğin uygulanmadığını belirtir; yetenek bilinmiyorsa `enabled: true` bildirilmez. API anahtarlı sağlayıcılar, Anthropic ve geçersiz değerler reddedilir. +`openai` Codex havuzunun eşiğini yönetir veya genel OAuth havuzunun eşiğini kaydeder. `on` %80, `off` %0 kaydeder; `threshold ` 0–100 kabul eder. Genel havuz eşiği yalnızca `pool.kernel` açıkken ve `strategy: "fill-first"` seçiliyken seçimi yönlendirir; bayrak kapalıyken kayıt işlemi eşik tabanlı geçişi etkinleştirmez. Her iki durumda da sağlayıcının etkinlik ayarını veya 429 hatasından sonraki otomatik hesap değişimini etkilemez. Genel havuz çıktısı sunucunun doğruladığı değerleri kullanır. Genel havuzlarda `poolEnabled`, kaydedilmiş sağlayıcı ayarıdır (`null` belirtilmemiş demektir); devralınmış etkin durumu göstermez. `inert: true` eşiğin kaydedildiğini ama uygulanmadığını, `inert: false` ise havuzun onu uyguladığını belirtir. `inert` yoksa yetenek bilinmiyordur ve bu durumda da `enabled: true` bildirilmez. API anahtarlı sağlayıcılar, Anthropic ve geçersiz değerler reddedilir. ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md b/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md index 6418792429..623c84996b 100644 --- a/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md @@ -173,11 +173,11 @@ token,也不是简单重读账号列表。`--json` 返回 ### `ocx account auto-switch > [--json]` -控制 `openai` Codex 账户池阈值,或保存通用 OAuth 账户池阈值。`on` 保存 80%,`off` 保存 0%,`threshold ` 接受 0–100。通用池的阈值目前不参与运行;保存阈值不会启用阈值切换、改变提供方启用设置或禁用 429 错误后的轮换。通用池的查询和修改结果使用服务器确认值。通用池的 `poolEnabled` 是已保存的提供方设置,`null` 表示未指定,并不代表继承后的实际状态。`inert: true` 表示阈值未应用;能力未知时也不会报告 `enabled: true`。API 密钥提供方、Anthropic 和无效值会被拒绝。 +控制 `openai` Codex 账户池阈值,或保存通用 OAuth 账户池阈值。`on` 保存 80%,`off` 保存 0%,`threshold ` 接受 0–100。通用池的阈值只有在 `pool.kernel` 打开且 `strategy: "fill-first"` 时才参与选择;标志关闭时,保存阈值不会启用阈值切换。两种情况下都不会改变提供方启用设置或禁用 429 错误后的轮换。通用池的查询和修改结果使用服务器确认值。通用池的 `poolEnabled` 是已保存的提供方设置,`null` 表示未指定,并不代表继承后的实际状态。`inert: true` 表示阈值已保存但未应用,`inert: false` 表示账户池正在应用它。没有 `inert` 字段表示能力未知,此时同样不会报告 `enabled: true`。API 密钥提供方、Anthropic 和无效值会被拒绝。 ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account priority [<-100..100|first|earlier|normal|later|last|reset>] [--json]` diff --git a/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md b/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md index 5677e79852..ad824e24e7 100644 --- a/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md @@ -132,11 +132,11 @@ Codex 池選擇套用於清除既有親和性後的下一個請求;進行中 ### `ocx account auto-switch > [--json]` -控制 `openai` Codex 帳戶池閾值,或儲存通用 OAuth 帳戶池閾值。`on` 儲存 80%,`off` 儲存 0%,`threshold ` 接受 0–100。通用池的閾值目前不參與執行;儲存閾值不會啟用閾值切換、改變供應商啟用設定或停用 429 錯誤後的輪替。通用池的查詢與修改結果使用伺服器確認值。通用池的 `poolEnabled` 是已儲存的供應商設定,`null` 表示未指定,並不代表繼承後的實際狀態。`inert: true` 表示閾值未套用;能力未知時也不會回報 `enabled: true`。API 金鑰供應商、Anthropic 與無效值會被拒絕。 +控制 `openai` Codex 帳戶池閾值,或儲存通用 OAuth 帳戶池閾值。`on` 儲存 80%,`off` 儲存 0%,`threshold ` 接受 0–100。通用池的閾值只有在 `pool.kernel` 開啟且 `strategy: "fill-first"` 時才參與選擇;旗標關閉時,儲存閾值不會啟用閾值切換。兩種情況下都不會改變供應商啟用設定或停用 429 錯誤後的輪替。通用池的查詢與修改結果使用伺服器確認值。通用池的 `poolEnabled` 是已儲存的供應商設定,`null` 表示未指定,並不代表繼承後的實際狀態。`inert: true` 表示閾值已儲存但未套用,`inert: false` 表示帳戶池正在套用它。沒有 `inert` 欄位表示能力未知,此時同樣不會回報 `enabled: true`。API 金鑰供應商、Anthropic 與無效值會被拒絕。 ```text openai: { provider, autoSwitchThreshold: number, enabled: boolean } -generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: true | null } +generic OAuth: { provider, autoSwitchThreshold: number | null, enabled: boolean, poolEnabled: boolean | null, inert: boolean | null } ``` ### `ocx account login|reauth|code|cancel ...` diff --git a/skills/ocx/references/01_management_surface.md b/skills/ocx/references/01_management_surface.md index 7fcf629e50..c7ae57c14b 100644 --- a/skills/ocx/references/01_management_surface.md +++ b/skills/ocx/references/01_management_surface.md @@ -539,7 +539,7 @@ JSON mode: `envelope`. - A bare invocation reads and never writes. - The APPLIED value is echoed, not the requested one, so a server-side normalization stays visible. - Values are not re-validated in the CLI: the server owns the strategy names and the 1-100 sticky bound. -- `anthropic` owns the full pool contract. Other OAuth providers reach the same endpoint with a generic subset (enabled/strategy/autoSwitchThreshold) whose settings persist but do not yet steer selection; `sticky` and `quotaWindow` are refused for them. +- `anthropic` owns the full pool contract. Other OAuth providers reach the same endpoint with a generic subset (enabled/strategy/autoSwitchThreshold/sticky); those settings steer selection only while `pool.kernel` is on, which is what the `inert` field reports. `quotaWindow` is still refused for them. ### `ocx account sticky` diff --git a/src/cli/account-extended.ts b/src/cli/account-extended.ts index a6fa3b707c..a8484f8b73 100644 --- a/src/cli/account-extended.ts +++ b/src/cli/account-extended.ts @@ -399,14 +399,22 @@ export async function cmdAutoSwitch(args: string[], deps: AccountDeps): Promise< const storedThreshold = typeof stored === "number" && Number.isInteger(stored) && stored >= 0 && stored <= 100 ? stored : null; const poolEnabled = typeof settings.enabled === "boolean" ? settings.enabled : null; - const inert = settings.inert === true ? true : null; - // This CLI understands only the current inert generic threshold contract. - const enabled = false; + // Three states, not two. `true` is stored-but-not-applied, `false` is applied by the + // shared kernel, and absent is a server that does not speak this field at all. Collapsing + // false into absent would render the live feature as an unknown capability. + const inert = typeof settings.inert === "boolean" ? settings.inert : null; + // A stored threshold only steers selection once the pool consumes it, which is exactly + // what `inert: false` reports. + const enabled = inert === false && storedThreshold !== null; if (wantsJson) { console.log(JSON.stringify({ provider: name, autoSwitchThreshold: storedThreshold, enabled, poolEnabled, inert }, null, 2)); } else { const value = storedThreshold === null ? "unset" : `${storedThreshold}%`; - console.log(`auto-switch: ${inert === true ? "inactive" : "unavailable"} (stored threshold ${value}; ${inert === true ? "not applied by this pool" : "threshold support is unknown"})`); + const state = inert === false ? (enabled ? "on" : "off") : inert === true ? "inactive" : "unavailable"; + const why = inert === false + ? (enabled ? "applied by this pool" : "no threshold stored") + : inert === true ? "not applied by this pool" : "threshold support is unknown"; + console.log(`auto-switch: ${state} (stored threshold ${value}; ${why})`); } return 0; } diff --git a/src/cli/capabilities.ts b/src/cli/capabilities.ts index d9aa8d0402..60dfc67380 100644 --- a/src/cli/capabilities.ts +++ b/src/cli/capabilities.ts @@ -328,7 +328,7 @@ export const CAPABILITIES: readonly Capability[] = [ "A bare invocation reads and never writes.", "The APPLIED value is echoed, not the requested one, so a server-side normalization stays visible.", "Values are not re-validated in the CLI: the server owns the strategy names and the 1-100 sticky bound.", - "`anthropic` owns the full pool contract. Other OAuth providers reach the same endpoint with a generic subset (enabled/strategy/autoSwitchThreshold) whose settings persist but do not yet steer selection; `sticky` and `quotaWindow` are refused for them.", + "`anthropic` owns the full pool contract. Other OAuth providers reach the same endpoint with a generic subset (enabled/strategy/autoSwitchThreshold/sticky); those settings steer selection only while `pool.kernel` is on, which is what the `inert` field reports. `quotaWindow` is still refused for them.", ], }, { diff --git a/src/config.ts b/src/config.ts index 106dadd2b6..3cdd699549 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1305,6 +1305,9 @@ const configSchema = z.object({ enabled: z.boolean().optional(), leadTimeMinutes: z.number().int().min(1).max(60).optional(), }).optional().catch(undefined), + // Same degrade-to-off rule as the flags above: a hand-edited typo in an opt-in pool + // feature must never cost the operator their providers. + pool: z.object({ kernel: z.boolean().optional() }).optional().catch(undefined), // Model ids excluded from the Grok Build managed block (dashboard switches). grokExcludedModels: z.array(z.string()).optional(), // Invalid values degrade to undefined ("auto") instead of failing the whole diff --git a/src/oauth/account-quota-rank.ts b/src/oauth/account-quota-rank.ts index 978f73de9d..0cae484fa5 100644 --- a/src/oauth/account-quota-rank.ts +++ b/src/oauth/account-quota-rank.ts @@ -67,6 +67,17 @@ function headroomOf(provider: string, accountId: string): number | null { return 100 - Math.max(...percents); } +/** + * Remaining headroom percent for one account, or null when nothing has measured it. + * + * Exported for the generic fill-first threshold, which needs the measurement itself rather + * than an ordering. Null stays null all the way out: a caller must decide what "unmeasured" + * means for its own rule instead of being handed a fabricated 0 or 100. + */ +export function accountHeadroomPercent(provider: string, accountId: string): number | null { + return headroomOf(provider, accountId); +} + /** Unknown usage is not exhaustion; Kiro's explicit overage verdict is authoritative. */ export function isAccountQuotaExhausted(provider: string, accountId: string): boolean { const exhaustion = provider === "kiro" ? getKiroAccountExhaustion(`${provider}\u0000${accountId}`) : null; diff --git a/src/oauth/generic-account-failover.ts b/src/oauth/generic-account-failover.ts index 1ccfaf71df..6b444ea435 100644 --- a/src/oauth/generic-account-failover.ts +++ b/src/oauth/generic-account-failover.ts @@ -16,7 +16,21 @@ */ import { getAccountSet } from "./store"; import { getValidAccessSnapshotForAccount, type OAuthAccessSnapshot } from "./index"; -import { exhaustedCooldownMs, hasHeadroomEvidence, isAccountQuotaExhausted, rankAccountsByHeadroom } from "./account-quota-rank"; +import { + accountHeadroomPercent, + exhaustedCooldownMs, + hasHeadroomEvidence, + isAccountQuotaExhausted, + rankAccountsByHeadroom, +} from "./account-quota-rank"; +import { + genericPoolKey, + normalizeAccountPoolStickyLimit, + notePoolRotationSuccess, + peekRoundRobinAccount, + pickRoundRobinAccount, + seedPoolRotationAccount, +} from "./pool-kernel"; import { parseRetryAfterMs } from "../combos/failover"; import { sweepExpiredOnWrite } from "../lib/state-store-sweeper"; import type { OcxConfig, OcxProviderConfig } from "../types"; @@ -169,6 +183,113 @@ export function eligibleFailoverAccounts(providerName: string, now = Date.now()) .map(account => account.id); } +/** Generic pool strategies the kernel can actually run. `quota` IS the pre-kernel path. */ +type ActiveGenericStrategy = "round-robin" | "fill-first"; + +/** Matches the Codex and Anthropic pools; the DTO still reports `null` for "not stored". */ +const DEFAULT_GENERIC_AUTO_SWITCH_THRESHOLD = 80; + +/** + * The strategy this provider's pool actually runs, or null for today's behaviour. + * + * Three different inputs answer null and they all mean the same thing to a caller: the flag is + * off, no strategy is stored, or the stored strategy is `quota` — which is precisely what the + * unflagged code already does. Collapsing them here is what keeps every call site a two-way + * branch instead of a four-way one. + */ +function activeGenericStrategy(config: OcxConfig, providerName: string): ActiveGenericStrategy | null { + if (config.pool?.kernel !== true) return null; + const raw = config.providers?.[providerName]?.oauthAccountFailover?.strategy; + return raw === "round-robin" || raw === "fill-first" ? raw : null; +} + +function genericStickyLimit(config: OcxConfig, providerName: string): number { + return normalizeAccountPoolStickyLimit(config.providers?.[providerName]?.oauthAccountFailover?.stickyLimit); +} + +/** + * The FULL roster in a stable order, not the eligible subset. + * + * Two load-bearing reasons. The store holds accounts in LOGIN order, so two operators who added + * the same accounts in a different sequence would otherwise rotate differently; sorting makes + * the ring a property of the accounts rather than of the history. And walking the eligible + * subset instead of the full roster changes the wrap order whenever an ineligible id sits + * between two eligible ones — the bug the Codex and Anthropic copies carry a `stableAll` + * argument to avoid. + */ +function stableGenericRoster(providerName: string): string[] { + const set = getAccountSet(providerName); + if (!set) return []; + return set.accounts.map(account => account.id).sort((left, right) => left.localeCompare(right)); +} + +/** + * Has this account spent enough of its allowance for fill-first to move on? + * + * An unmeasured account reads as UNDER the threshold, matching the Codex pool: a threshold is a + * statement about observed usage, and treating "no observation" as "spent" would evacuate every + * quota-less provider off its active account on the very first request. + */ +function isOverAutoSwitchThreshold(providerName: string, accountId: string, threshold: number): boolean { + const headroom = accountHeadroomPercent(providerName, accountId); + if (headroom === null) return false; + return 100 - headroom >= threshold; +} + +/** + * Fill-first: stay on the active account until it crosses its threshold, then take the next + * eligible account in the stable ring. Null means "keep the active account". + */ +function pickFillFirstGenericAccount( + config: OcxConfig, + providerName: string, + activeId: string | undefined, + now: number, +): string | null { + const stableAll = stableGenericRoster(providerName); + if (stableAll.length < 2) return null; + const eligible = new Set(eligibleFailoverAccounts(providerName, now)); + const stored = config.providers?.[providerName]?.oauthAccountFailover?.autoSwitchThreshold; + const threshold = typeof stored === "number" && Number.isInteger(stored) && stored >= 0 && stored <= 100 + ? stored + : DEFAULT_GENERIC_AUTO_SWITCH_THRESHOLD; + if (activeId && eligible.has(activeId) && !isOverAutoSwitchThreshold(providerName, activeId, threshold)) { + return null; + } + const start = activeId ? stableAll.indexOf(activeId) : -1; + const ring = start >= 0 ? [...stableAll.slice(start + 1), ...stableAll.slice(0, start)] : stableAll; + for (const id of ring) { + if (id !== activeId && eligible.has(id)) return id; + } + return null; +} + +/** + * Advance the round-robin cursor once a dispatch has actually been admitted on this account. + * + * The early return is the whole safety story for the core path: this is reached on EVERY + * generic first dispatch, including quota pools and the fallback after a preferred account was + * dropped, so anything but round-robin must leave the cursor untouched. + * + * The live pick belongs here rather than in the proposal, and that is not stylistic. + * `peekRoundRobinAccount` never creates the pool state and `notePoolRotationSuccess` returns + * immediately when there is none, so a peek-only path would leave the ring with nothing to + * advance and round-robin would propose the same account forever. This is the same shape + * `commitAnthropicSelectionRouting` already commits with. + */ +export function noteGenericPoolSelection(config: OcxConfig, providerName: string, accountId: string): void { + if (activeGenericStrategy(config, providerName) !== "round-robin") return; + const poolKey = genericPoolKey(providerName); + const limit = genericStickyLimit(config, providerName); + const picked = pickRoundRobinAccount(poolKey, eligibleFailoverAccounts(providerName), limit); + // The resolver may have admitted a different account than the ring proposed: a removal, a + // reauth verdict or a manual selection can land during credential resolution. Realign the + // cursor onto what actually served rather than leaving it on a road not taken. + if (picked !== accountId) seedPoolRotationAccount(poolKey, accountId); + notePoolRotationSuccess(poolKey, accountId, limit); +} + + /** * Cool the account that actually 429'd and name the next eligible one, or null. * @@ -212,6 +333,30 @@ export function rotateGenericOAuthAccountOn429( const ring = start >= 0 ? [...order.slice(start + 1), ...order.slice(0, start)] : order; const candidates = ring.filter(id => id !== failedAccountId && eligible.includes(id)); if (candidates.length === 0) return null; + // The 429 path branches too. Leaving it on the quota ranking would make a configured + // strategy inert in practice the moment anything actually failed, which is the case the + // operator chose the strategy for. + const strategy = activeGenericStrategy(config, providerName); + if (strategy === "round-robin") { + // PICK here, not peek: the failure already happened and this answer is the one being used, + // so the ring genuinely advances. + return pickRoundRobinAccount( + genericPoolKey(providerName), + candidates, + genericStickyLimit(config, providerName), + ); + } + if (strategy === "fill-first") { + // Not "keep the active account": the one that just 429'd is cooled, so fill-first takes + // the next eligible account in the stable ring rather than its usual hold. + const stableAll = stableGenericRoster(providerName); + const from = stableAll.indexOf(failedAccountId); + const walk = from >= 0 ? [...stableAll.slice(from + 1), ...stableAll.slice(0, from)] : stableAll; + for (const id of walk) { + if (id !== failedAccountId && candidates.includes(id)) return id; + } + return null; + } // With no quota evidence this returns the ring untouched, so providers without // per-account quota keep exactly the traversal they have today. return rankAccountsByHeadroom(providerName, candidates)[0] ?? null; @@ -259,6 +404,29 @@ export function preferredInitialAccount( const order = selected.accounts.filter(account => account.needsReauth !== true).map(account => account.id); if (order.length < 2) return null; + // A configured strategy answers this question itself. Both guards below exist to protect the + // QUOTA answer, and both are fatal to the other two: hasHeadroomEvidence refuses every + // provider with no quota data, which is exactly where round-robin is the point, and the + // healthy-active return fires before autoSwitchThreshold can ever be read, so fill-first + // would never reach its own test. Cooldowns and reauth are still honoured inside each pick. + const strategy = activeGenericStrategy(config, providerName); + if (strategy === "round-robin") { + const eligibleNow = eligibleFailoverAccounts(providerName, now); + if (eligibleNow.length === 0) return null; + // PEEK, not pick: this proposal is discardable, and advancing the ring for an account the + // resolver then rejects would skip a turn for nothing. noteGenericPoolSelection commits. + const picked = peekRoundRobinAccount( + genericPoolKey(providerName), + eligibleNow, + genericStickyLimit(config, providerName), + ); + return picked && picked !== active ? picked : null; + } + if (strategy === "fill-first") { + const picked = pickFillFirstGenericAccount(config, providerName, active, now); + return picked && picked !== active ? picked : null; + } + const activeRow = selected.accounts.find(account => account.id === active); if (activeRow && activeRow.needsReauth !== true && !isCooled(providerName, activeRow.id, now) diff --git a/src/oauth/pool-settings-capability.ts b/src/oauth/pool-settings-capability.ts index 210870b718..eb98a0d4dc 100644 --- a/src/oauth/pool-settings-capability.ts +++ b/src/oauth/pool-settings-capability.ts @@ -37,24 +37,37 @@ export function parseGenericAutoSwitchThreshold(value: unknown): number | null { return typeof value === "number" && Number.isInteger(value) && value >= 0 && value <= 100 ? value : null; } +export function parseGenericStickyLimit(value: unknown): number | null { + return typeof value === "number" && Number.isInteger(value) && value >= 1 && value <= 100 ? value : null; +} + export interface GenericPoolSettingsDto { provider: string; kind: "generic"; enabled: boolean | null; strategy: GenericPoolStrategy | null; autoSwitchThreshold: number | null; + stickyLimit: number | null; /** - * Slice-1 marker for `strategy` and `autoSwitchThreshold` only: persisted, not yet consumed - * by the selector. + * Marker for `strategy`, `autoSwitchThreshold` and `stickyLimit` only: true while they are + * persisted but not consumed by the selector, false once `pool.kernel` is on and they + * actually choose an account. * * It deliberately does NOT describe `enabled`, which governs the pre-dispatch preference. * Widening it to the whole DTO would tell a dashboard that `enabled` changes nothing, which * has been false since reactive and proactive activation were split. + * + * Computed, never a literal: the flag is reversible, so a DTO that hard-codes either answer + * would be lying in one of the two states. */ - inert: true; + inert: boolean; } -export function genericPoolSettingsDto(name: string, provider: OcxProviderConfig): GenericPoolSettingsDto { +export function genericPoolSettingsDto( + name: string, + provider: OcxProviderConfig, + kernelEnabled = false, +): GenericPoolSettingsDto { const failover = provider.oauthAccountFailover ?? {}; return { provider: name, @@ -62,6 +75,7 @@ export function genericPoolSettingsDto(name: string, provider: OcxProviderConfig enabled: typeof failover.enabled === "boolean" ? failover.enabled : null, strategy: parseGenericPoolStrategy(failover.strategy), autoSwitchThreshold: parseGenericAutoSwitchThreshold(failover.autoSwitchThreshold), - inert: true, + stickyLimit: parseGenericStickyLimit(failover.stickyLimit), + inert: kernelEnabled !== true, }; } diff --git a/src/server/management/oauth-account-routes.ts b/src/server/management/oauth-account-routes.ts index e2ba5a2029..6a1a1ae35b 100644 --- a/src/server/management/oauth-account-routes.ts +++ b/src/server/management/oauth-account-routes.ts @@ -331,6 +331,12 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< if (!(await setActiveAccount(provider, body.accountId))) return jsonResponse({ error: "account not found" }, 404); const { forgetGenericFailoverRoster } = await import("../../oauth/generic-account-failover"); forgetGenericFailoverRoster(provider); + // Seed the rotation cursor on the operator's pick, or a sticky round-robin ring hands the + // very next dispatch back to whatever the pool had chosen. forgetGenericFailoverRoster + // only drops the presence count; it has never touched the cursor. Same defect the Codex + // side carries resetCodexRoutingForManualSelection for. + const { genericPoolKey, seedPoolRotationAccount } = await import("../../oauth/pool-kernel"); + seedPoolRotationAccount(genericPoolKey(provider), body.accountId); if (provider === "anthropic") { const { resetAnthropicRoutingForManualSelection } = await import("../../oauth/anthropic-routing"); resetAnthropicRoutingForManualSelection(body.accountId); @@ -357,7 +363,7 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< if (!provider || !prov || poolSettingsCapability(provider, prov) !== "generic") { return jsonResponse({ error: "pool config is only supported for anthropic and generic OAuth providers" }, 400); } - return jsonResponse(genericPoolSettingsDto(provider, prov)); + return jsonResponse(genericPoolSettingsDto(provider, prov, config.pool?.kernel === true)); } const pool = config.anthropicAccountPool ?? {}; return jsonResponse({ @@ -387,13 +393,14 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< if (provider !== "anthropic") { const { poolSettingsCapability, genericPoolSettingsDto, parseGenericPoolStrategy, parseGenericAutoSwitchThreshold, + parseGenericStickyLimit, } = await import("../../oauth/pool-settings-capability"); const prov = config.providers[provider]; if (!provider || !prov || poolSettingsCapability(provider, prov) !== "generic") { return jsonResponse({ error: "pool config is only supported for anthropic and generic OAuth providers" }, 400); } - if (body.stickyLimit !== undefined || body.quotaWindow !== undefined) { - return jsonResponse({ error: "stickyLimit and quotaWindow are not part of the generic pool contract yet" }, 400); + if (body.quotaWindow !== undefined) { + return jsonResponse({ error: "quotaWindow is not part of the generic pool contract yet" }, 400); } const next = { ...(prov.oauthAccountFailover ?? {}) }; if (body.enabled !== undefined) { @@ -416,10 +423,18 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< next.autoSwitchThreshold = parsed; } } + if (body.stickyLimit !== undefined) { + if (body.stickyLimit === null) delete next.stickyLimit; + else { + const parsed = parseGenericStickyLimit(body.stickyLimit); + if (parsed === null) return jsonResponse({ error: "stickyLimit must be an integer 1-100" }, 400); + next.stickyLimit = parsed; + } + } if (Object.keys(next).length > 0) prov.oauthAccountFailover = next; else delete prov.oauthAccountFailover; saveConfigPreservingClaudeCode(config); - return jsonResponse({ ok: true, ...genericPoolSettingsDto(provider, prov) }); + return jsonResponse({ ok: true, ...genericPoolSettingsDto(provider, prov, config.pool?.kernel === true) }); } let enabled = config.anthropicAccountPool?.enabled === true; if (body.enabled !== undefined) { diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index cccd942026..4ca744264f 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -145,6 +145,7 @@ import { GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST, isGenericFailoverProvider, isGenericOAuthFailoverEnabled, + noteGenericPoolSelection, preferredInitialAccount, rotateGenericOAuthAccountOn429, } from "../../oauth/generic-account-failover"; @@ -4405,6 +4406,10 @@ async function handleResponsesInner( // whichever account is active by the time the response comes back (#2568). if (isGenericFailoverProvider(route.providerName, route.provider)) { genericFailoverAccountId = resolved.accountId; + // Advance the pool cursor only now that this account is actually admitted. The + // helper returns immediately unless the kernel is on AND the strategy is + // round-robin, so quota and fill-first pools reach it without being touched. + noteGenericPoolSelection(config, route.providerName, resolved.accountId); } // Anthropic is excluded from isGenericFailoverProvider -- its own pool owns affinity and // a fail-closed local-cli credential rule -- so without this stamp its identity is diff --git a/src/types/config.ts b/src/types/config.ts index 499ca59eb1..979310f80c 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -831,6 +831,15 @@ export interface OcxConfig { * spends a second credit. A malformed value reads as off. */ resetCreditAutoRedeem?: { enabled?: boolean; leadTimeMinutes?: number }; + /** + * Shared account-pool kernel, opt-in and off by default. + * + * `kernel: true` is what makes a generic OAuth provider's stored `strategy` and + * `autoSwitchThreshold` actually select an account instead of merely being persisted. + * Off restores the pre-kernel path exactly, which is why the DTO keeps reporting + * `inert: true` until this is on. A malformed value reads as off. + */ + pool?: { kernel?: boolean }; /** Active pool account id for next session. undefined = main (passthrough as-is). */ activeCodexAccountId?: string; /** Auto-switch threshold (0-100). Default 80. 0 = disabled. */ diff --git a/src/types/provider.ts b/src/types/provider.ts index e65130a4fa..fc56c88e19 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -521,11 +521,21 @@ export interface OcxProviderConfig { enabled?: boolean; /** * Generic OAuth pool selection strategy (#695). Persisted through the pool-settings - * contract; the selector does not consume it yet, so omitted keeps today's behavior. + * contract. Consumed by the selector only while `pool.kernel` is on; with the flag off + * it is still merely persisted, so omitted and set behave the same. */ strategy?: "quota" | "round-robin" | "fill-first"; - /** 0-100 usage percent at which a proactive switch may be considered (#695); inert today. */ + /** + * 0-100 usage percent at which fill-first advances off the active account (#695). + * Read only under `pool.kernel` with `strategy: "fill-first"`; 80 when unset, matching + * the Codex and Anthropic pools. + */ autoSwitchThreshold?: number; + /** + * Successful dispatches retained on one round-robin selection. Default 1; range 1..100. + * Read only under `pool.kernel` with `strategy: "round-robin"`. + */ + stickyLimit?: number; }; /** Allow an explicitly key/oauth provider to run without a credential (for keyless local proxies). */ keyOptional?: boolean; diff --git a/tests/cli/cli-account-pool-verbs.test.ts b/tests/cli/cli-account-pool-verbs.test.ts index 04be2e4fa6..0eacd8e1bf 100644 --- a/tests/cli/cli-account-pool-verbs.test.ts +++ b/tests/cli/cli-account-pool-verbs.test.ts @@ -391,8 +391,7 @@ describe("generic OAuth pool-settings contract (#695)", () => { test("generic missing or malformed capability stays unknown rather than enabled", async () => { for (const json of [null, [], {}, { enabled: "true", autoSwitchThreshold: "90", inert: "false" }, - { enabled: true, autoSwitchThreshold: 90 }, { enabled: true, autoSwitchThreshold: 101, inert: false }, - { enabled: true, autoSwitchThreshold: 90, inert: false }]) { + { enabled: true, autoSwitchThreshold: 90 }, { enabled: true, autoSwitchThreshold: 101, inert: false }]) { const out = capture(); try { expect(await cmdAutoSwitch(["google-antigravity", "status", "--json"], genericDeps(() => ({ json }), []))).toBe(0); @@ -403,6 +402,39 @@ describe("generic OAuth pool-settings contract (#695)", () => { } }); + test("a generic pool that reports inert false is live, not unknown", async () => { + // `inert: false` used to be lumped in with the malformed bodies above, which made the CLI + // render a threshold the kernel is actually applying as "threshold support is unknown" -- + // the opposite of the truth. The three states are distinct: true is stored-but-not-applied, + // false is applied, absent is a server that does not speak the field at all. + const out = capture(); + try { + expect(await cmdAutoSwitch( + ["google-antigravity", "status", "--json"], + genericDeps(() => ({ json: { enabled: true, autoSwitchThreshold: 90, inert: false } }), []), + )).toBe(0); + } finally { out.restore(); } + expect(JSON.parse(out.lines.join("\n"))).toEqual({ + provider: "google-antigravity", autoSwitchThreshold: 90, enabled: true, poolEnabled: true, inert: false, + }); + }); + + test("a live generic pool with no stored threshold reports off, not on", async () => { + // `inert: false` alone is not enablement: the kernel is consuming settings, but there is + // no threshold to consume. Reporting "on" here would invent a value nobody set. + const out = capture(); + try { + expect(await cmdAutoSwitch( + ["google-antigravity", "status", "--json"], + genericDeps(() => ({ json: { enabled: true, inert: false } }), []), + )).toBe(0); + } finally { out.restore(); } + const result = JSON.parse(out.lines.join("\n")); + expect(result.enabled).toBe(false); + expect(result.inert).toBe(false); + expect(result.autoSwitchThreshold).toBeNull(); + }); + test("a successful generic write with a null body reports unknown settings", async () => { const calls: Captured[] = []; const out = capture(); diff --git a/tests/oauth/generic-oauth-failover.test.ts b/tests/oauth/generic-oauth-failover.test.ts index 48fb2782ec..fbf47c776a 100644 --- a/tests/oauth/generic-oauth-failover.test.ts +++ b/tests/oauth/generic-oauth-failover.test.ts @@ -9,6 +9,7 @@ import { hasFailoverAccountQuorum, isGenericFailoverProvider, isGenericOAuthFailoverEnabled, + noteGenericPoolSelection, preferredInitialAccount, rotateGenericOAuthAccountOn429, } from "../../src/oauth/generic-account-failover"; @@ -525,3 +526,131 @@ describe("#2807 a 429 rotation pairs the bearer with its OWN origin", () => { expect(rotated.baseUrl).toBe(CANONICAL); }); }); + +describe("#695 the generic pool consumes its persisted strategy behind pool.kernel", () => { + /** Proactive preference on, plus whichever strategy this case is about. */ + function kernelConfig(strategy?: "quota" | "round-robin" | "fill-first", extra: Record = {}): OcxConfig { + return { + pool: { kernel: true }, + providers: { + xai: { + ...OAUTH_PROVIDER, + oauthAccountFailover: { enabled: true, ...(strategy ? { strategy } : {}), ...extra }, + }, + }, + } as unknown as OcxConfig; + } + + test("round-robin rotates a provider with no quota data at all", async () => { + const ids = await seed(3); + await setActiveAccount("xai", ids[0]!); + // Deliberately NO quota is cached. This is the case the evidence guard refuses outright, + // and it is exactly where round-robin is the point: with nothing measured there is no + // ranking to make, only a turn to take. + const cfg = kernelConfig("round-robin"); + + const served: string[] = []; + for (let i = 0; i < 4; i += 1) { + const preferred = preferredInitialAccount(cfg, "xai"); + const account = preferred ?? getAccountSet("xai")!.activeAccountId!; + served.push(account); + // Admission is what advances the ring; the proposal above only peeks. + noteGenericPoolSelection(cfg, "xai", account); + } + expect(new Set(served).size).toBeGreaterThan(1); + }); + + test("round-robin is a no-op while pool.kernel is off", async () => { + const ids = await seed(3); + await setActiveAccount("xai", ids[0]!); + const off = { + providers: { xai: { ...OAUTH_PROVIDER, oauthAccountFailover: { enabled: true, strategy: "round-robin" } } }, + } as unknown as OcxConfig; + + for (let i = 0; i < 4; i += 1) { + // Same roster, same strategy, flag off: the pre-kernel answer is null every time, + // because no quota was ever measured. Reversibility is the whole point of the flag. + expect(preferredInitialAccount(off, "xai")).toBeNull(); + noteGenericPoolSelection(off, "xai", ids[0]!); + } + }); + + test("fill-first holds the active account under its threshold and advances over it", async () => { + const ids = await seed(3); + const sorted = [...ids].sort((left, right) => left.localeCompare(right)); + const active = sorted[0]!; + await setActiveAccount("xai", active); + const cfg = kernelConfig("fill-first", { autoSwitchThreshold: 80 }); + + setCachedProviderAccountQuotaForTests("xai", active, { weeklyPercent: 40, updatedAt: Date.now() }); + // Under threshold: fill-first is supposed to keep filling this one. + expect(preferredInitialAccount(cfg, "xai")).toBeNull(); + + setCachedProviderAccountQuotaForTests("xai", active, { weeklyPercent: 90, updatedAt: Date.now() }); + // Over threshold: it advances, and to the NEXT account in the sorted roster rather than + // to whichever id the login order happened to put first. + expect(preferredInitialAccount(cfg, "xai")).toBe(sorted[1]!); + }); + + test("fill-first advances through the sorted roster, not the eligible subset", async () => { + const ids = await seed(3); + const sorted = [...ids].sort((left, right) => left.localeCompare(right)); + const active = sorted[0]!; + await setActiveAccount("xai", active); + const cfg = kernelConfig("fill-first", { autoSwitchThreshold: 80 }); + setCachedProviderAccountQuotaForTests("xai", active, { weeklyPercent: 95, updatedAt: Date.now() }); + + // The successor is out of service, so the walk has to step OVER it and land on the third + // account. Walking the eligible subset instead would wrap from a shorter list and pick a + // different account -- the bug the shared kernel carries a stableAll argument to avoid. + await markAccountNeedsReauth("xai", sorted[1]!, true); + expect(preferredInitialAccount(cfg, "xai")).toBe(sorted[2]!); + }); + + test("quota keeps its pre-kernel answer with the flag on", async () => { + const ids = await seed(2); + await setActiveAccount("xai", ids[0]!); + // 100, not 99: the quota path only leaves an active account once it is exhausted or + // cooled. A merely busy account keeps serving, and that is the pre-kernel rule this case + // is here to pin. + setCachedProviderAccountQuotaForTests("xai", ids[0]!, { weeklyPercent: 100, updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("xai", ids[1]!, { weeklyPercent: 10, updatedAt: Date.now() }); + + // An explicit "quota" and no strategy at all must answer identically: quota IS the + // pre-kernel path, so the flag must not change it. + expect(preferredInitialAccount(kernelConfig("quota"), "xai")).toBe(ids[1]!); + expect(preferredInitialAccount(kernelConfig(), "xai")).toBe(ids[1]!); + }); + + test("a 429 under round-robin rotates instead of ranking", async () => { + const ids = await seed(3); + await setActiveAccount("xai", ids[0]!); + // Quota evidence pointing SOMEWHERE ELSE is what makes this case mean anything. With no + // evidence the pre-kernel path hands the ring back untouched and lands on the same + // account round-robin would, so the test would pass whether or not the branch exists. + setCachedProviderAccountQuotaForTests("xai", ids[1]!, { weeklyPercent: 80, updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("xai", ids[2]!, { weeklyPercent: 5, updatedAt: Date.now() }); + const next = rotateGenericOAuthAccountOn429(kernelConfig("round-robin"), "xai", ids[0]!, null); + expect(next).not.toBeNull(); + expect(next).not.toBe(ids[0]!); + // Round-robin takes its turn. Quota would have chased the roomier third account. + expect(next).toBe(ids[1]!); + }); + + test("a 429 under fill-first leaves the cooled account rather than holding it", async () => { + const ids = await seed(3); + const sorted = [...ids].sort((left, right) => left.localeCompare(right)); + await setActiveAccount("xai", sorted[0]!); + const cfg = kernelConfig("fill-first", { autoSwitchThreshold: 80 }); + // Well under threshold: the initial-preference rule would keep this account. The 429 path + // must not, because the account it would hold is the one that just failed. + setCachedProviderAccountQuotaForTests("xai", sorted[0]!, { weeklyPercent: 10, updatedAt: Date.now() }); + // The successor is the BUSIER of the two survivors, so quota ranking would skip past it. + // Fill-first still takes it: filling one account before opening the next is the point. + setCachedProviderAccountQuotaForTests("xai", sorted[1]!, { weeklyPercent: 70, updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("xai", sorted[2]!, { weeklyPercent: 5, updatedAt: Date.now() }); + + const next = rotateGenericOAuthAccountOn429(cfg, "xai", sorted[0]!, null); + expect(next).toBe(sorted[1]!); + }); +}); diff --git a/tests/server/account-pool-management-api.test.ts b/tests/server/account-pool-management-api.test.ts index e8eed3c05d..502062c1a5 100644 --- a/tests/server/account-pool-management-api.test.ts +++ b/tests/server/account-pool-management-api.test.ts @@ -439,7 +439,13 @@ describe("Anthropic account pool strategy management API", () => { // reading `inert` as covering `enabled` would render a live control as decorative. const source = await Bun.file("src/oauth/pool-settings-capability.ts").text(); const start = source.indexOf("autoSwitchThreshold: number | null;"); - const marker = source.slice(start, source.indexOf("inert: true;", start)); + // Anchored on the CURRENT literal. When this type read `inert: true;` and the field became + // `inert: boolean;`, indexOf returned -1 and slice(start, -1) handed back almost the whole + // file -- which still contains all three words, so every assertion below passed while the + // test had stopped checking anything. Fail closed on a missing anchor instead. + const end = source.indexOf("inert: boolean;", start); + expect(end).toBeGreaterThan(start); + const marker = source.slice(start, end); expect(marker).toContain("strategy"); expect(marker).toContain("autoSwitchThreshold"); expect(marker).toContain("enabled"); @@ -474,7 +480,10 @@ describe("generic OAuth pool-settings contract (#695)", () => { try { const absent = await fetch(new URL("/api/oauth/accounts/pool?provider=google-antigravity", server.url)); expect(absent.status).toBe(200); - expect(await absent.json()).toEqual({ provider: "google-antigravity", kind: "generic", enabled: null, strategy: null, autoSwitchThreshold: null, inert: true }); + expect(await absent.json()).toEqual({ + provider: "google-antigravity", kind: "generic", enabled: null, strategy: null, + autoSwitchThreshold: null, stickyLimit: null, inert: true, + }); const put = await fetch(new URL("/api/oauth/accounts/pool", server.url), { method: "PUT", headers: { "Content-Type": "application/json" }, @@ -488,7 +497,8 @@ describe("generic OAuth pool-settings contract (#695)", () => { for (const body of [ { provider: "google-antigravity", strategy: "weighted" }, { provider: "google-antigravity", autoSwitchThreshold: 101 }, - { provider: "google-antigravity", stickyLimit: 3 }, + { provider: "google-antigravity", stickyLimit: 0 }, + { provider: "google-antigravity", quotaWindow: "weekly" }, { provider: "deepseek", strategy: "quota" }, ]) { const bad = await fetch(new URL("/api/oauth/accounts/pool", server.url), {