fix(diagnostics): distinguish spill ACL timeout origins

docs-site/src/content/docs/reference/management-api.md

@@ -347,7 +347,7 @@ whether to star the repository.
 
 | Method and path | Purpose | Notable errors |
 | --- | --- | --- |
-| `GET /api/system/memory` | Return scalar process, heap, stream, response-state, watchdog, and active-turn metrics. Response-state diagnostics include spill-write status, consecutive failures, fixed privacy-safe failure class, and last failure/success timestamps; raw errors and paths are never returned. | — |
+| `GET /api/system/memory` | Return scalar process, heap, stream, response-state, watchdog, and active-turn metrics. Response-state diagnostics include spill-write status, consecutive failures, fixed privacy-safe failure class, and last failure/success timestamps. `spillLastWriteFailureOrigin` is `retry_returned_timeout`, `timeout_memo_refusal`, or null; cumulative `spillAclRetryReturnedTimeouts` and `spillAclTimeoutMemoRefusals` count terminal failed publications. See [Windows spill diagnostics](/troubleshooting/windows-memory/) for process-local semantics. Raw errors and paths are never returned. | — |
 | `POST /api/system/restart` | Begin a drain-aware process restart without removing client injection | Returns 202; repeated calls report the existing drain |
 | `POST /api/stop` | Stop the service, restore native Codex, remove managed Grok injection, and drain the proxy | 409 service ownership conflict; 409 `respawnable_service` when a Windows Task Scheduler wrapper could respawn the proxy and the caller is not `ocx stop` (nothing is changed); 409 when the installed manager refuses to stop; 409 `service_state_unknown` when the Task Scheduler state cannot be read (nothing is changed; repair the query and retry) |
 | `GET /api/system/codex-app-server` | Report whether running Codex app-servers predate the current model catalog | — |

docs-site/src/content/docs/troubleshooting/windows-memory.md

@@ -57,7 +57,22 @@ runtime the leak itself remains an upstream problem:
   time show whether failures are accumulating or recovering in the same process.
   The last failure is a fixed privacy-safe class such as `EACCES`, `ENOSPC`,
   `ETIMEDOUT`, or `EACLRETRYEXHAUSTED`; raw error messages and filesystem paths
-  are never returned. These diagnostics stay on the authenticated management
+  are never returned. `spillLastWriteFailureOrigin` adds a fixed origin or null:
+  `retry_returned_timeout` means the existing second spill attempt returned a
+  timeout; `timeout_memo_refusal` means the ACL helper refused through its
+  remembered timeout state. Other failures use null. The cumulative
+  `spillAclRetryReturnedTimeouts` and `spillAclTimeoutMemoRefusals` count terminal
+  failed publications, not individual ACL commands or transient first attempts.
+  Success clears the failure streak but retains the last failure fields and
+  cumulative counts; a later unrelated failure sets the last origin to null.
+  These values are process-local, so compare snapshots from the same process.
+  Neither origin identifies an OS command: the attempt budget can expire before
+  a command starts, and an optional compliance inspection can run before a memo
+  refusal. A separate process succeeding does not prove that the live process's
+  memo recovered. These observations do not add retries, clear memos, weaken
+  required ACLs, or automatically restart the service.
+
+  These diagnostics stay on the authenticated management
   endpoint and are intentionally absent from `/healthz`, which remains a liveness
   signal. The dashboard's **Memory observability** card renders the memory and
   continuation-size fields from this endpoint and offers a confirm-gated

skills/ocx/SKILL.md

@@ -1,12 +1,12 @@
 ---
 name: ocx
-description: Drive a running opencodex (`ocx`) proxy from the CLI — account pools, provider routing, model catalog, usage and cost attribution, request logs, access keys, storage cleanup, and the management API. Use when a task involves controlling or inspecting an opencodex proxy rather than editing the opencodex codebase. Triggers: ocx, opencodex, proxy control, account pool, pause account, pool strategy, provider routing, usage report, cost attribution, access key, request log, conversation trace, storage cleanup, management API.
+description: "Drive a running opencodex (`ocx`) proxy from the CLI — account pools, provider routing, model catalog, usage and cost attribution, request logs, access keys, storage cleanup, and the management API. Use when a task involves controlling or inspecting an opencodex proxy rather than editing the opencodex codebase. Triggers: ocx, opencodex, proxy control, account pool, pause account, pool strategy, provider routing, usage report, cost attribution, access key, request log, conversation trace, storage cleanup, management API."
 ---
 
 # Operating `ocx`
 
 `ocx` controls a locally running opencodex proxy. The CLI covers the dashboard's operational
-surface, with one consent exception (starring) recorded under Consent below. `ocx capabilities`
+surface, subject to Consent and Secret-bearing commands below. `ocx capabilities`
 lists the *declared* index, not every verb.
 
 Be precise about the gap, because guessing costs you more than reading: the capability index below
@@ -90,6 +90,27 @@ starring would be useful, say so and let the user decide.
 The same boundary covers the session-gated `/api/codex-prompt` writes: read them with
 `ocx inspect codex-prompt`, and leave the writes to the dashboard.
 
+## Secret-bearing commands
+
+**Do not create an access key or start an access-key rotation from an agent session.**
+This covers the create and rotation-start operations under `ocx access key`,
+`ocx access keys`, and `ocx api-key`, their `opencodex` equivalents and executable
+wrappers, and direct POST requests to `/api/keys` and `/api/keys/rotate`.
+Both text and JSON responses contain a one-time plaintext data-plane credential,
+which can enter the agent transcript. Ask the user to perform that step in a
+human-operated terminal outside the agent session, configure and verify the
+replacement, and report only confirmation plus non-secret key/rotation IDs.
+Never ask for the plaintext key in chat or offer a pipe, redirection, or API
+workaround to perform the secret-returning step inside the agent session.
+
+Configuration confirmation is not approval to revoke the existing credential.
+Identify the existing key ID and obtain separate explicit revocation approval
+before committing an in-place rotation or removing an old, separately replaced key.
+An existing explicit approval for that exact revocation remains valid; setup
+confirmation alone does not supply it. Commit and abort return no plaintext key,
+but still require authority for their state changes. Follow
+[recipe 5](references/03_recipes.md#5-prepare-an-access-key-rotation-without-exposing-the-new-key).
+
 ## Destructive verbs
 
 `storage trash restore` and `storage policy run` refuse without `--yes` (exit 2, nothing sent).

skills/ocx/references/03_recipes.md

@@ -94,20 +94,61 @@ Read `accounts[]`. Two things to respect:
 `providers[]` and `models[]` carry `estimatedCostUsd`. Costs are estimates; `estimateReasons` in the
 log rows tells you why (for example `usage_estimated`, `expected_price_overlay`).
 
-## 5. Rotate an access key and confirm it went quiet
+## 5. Prepare an access-key rotation without exposing the new key
 
 ```bash
 ocx access key list --json
-ocx access key create rotated --json          # the plaintext key is in THIS response only
+```
+
+Creating a key or starting a rotation returns a one-time plaintext credential in both text and
+JSON output. **Do not perform either operation in an agent session**, including through the
+aliases, executable wrappers, or management POST routes named in
+[Secret-bearing commands](../SKILL.md#secret-bearing-commands). Ask the user to perform that step
+in a terminal outside the agent session, configure and verify the replacement, and report only
+configuration confirmation and the non-secret key/rotation IDs. Never ask for the key itself.
+
+Configuration confirmation is not revocation approval. Identify the existing key ID and obtain
+separate explicit revocation approval before taking either path below. An existing explicit
+approval for that exact revocation remains valid; do not ask again for the same action and ID.
+
+For an in-place rotation, commit the pending replacement on the same ID:
+
+```bash
+ocx access key rotate commit <id> <rotation-id> --json
+```
+
+For a separately created replacement, remove only the old ID:
+
+```bash
 ocx access key remove <old-id> --yes --json
-ocx access key list --json                    # the old id is gone; check usage on the rest
 ```
 
-Note the argument style: `create <name>` and `remove <id>` are **positionals**, not `--label` and
-`--id`. `remove` also refuses without `--yes`.
+After the command succeeds, inspect the matching result:
+
+```bash
+ocx access key list --json
+```
+
+For an in-place rotation, the same ID remains and `pendingRotation` disappears. For a separately
+created replacement, the old ID disappears. The list alone does not prove the replacement accepts
+traffic; use the user's successful connection verification as that evidence. `remove <id>` is
+positional, not `--id`, and refuses without `--yes`.
+
+To cancel a pending rotation, with authority to discard the replacement:
+
+```bash
+ocx access key rotate abort <id> <rotation-id> --json
+```
+
+Abort retains the old credential and removes the pending replacement. Re-list to inspect pending
+state. On stale, mismatched, or expired rotation IDs, or an uncertain commit result, inspect
+non-secret state and report the refusal or uncertainty. Do not start another rotation, delete the
+entry, or retrieve a secret as automatic recovery. Missing pending state alone is not proof of a
+successful commit: expiry and abort also clear it.
 
-The list carries per-key usage, so a key whose count stops advancing is genuinely unused. The
-plaintext key appears once, in the `create` response, and is never retrievable again.
+The list carries per-key usage. A count that stops advancing shows no recorded new usage in that
+observation window; it does not prove no client still needs the key. Creation and rotation-start
+return the plaintext once; list does not return the full plaintext.
 
 An `ambiguous` footer on the list means two configured keys share an id, so per-key totals do not
 exist for them — do not attribute usage to either.

skills/ocx/references/05_remote_hub.md

@@ -114,6 +114,12 @@ The ordering is not ceremony. If the old key died at issuance, a client that had
 received the new key would be disconnected — and a disconnected client cannot be given a
 new key. So the contract is: apply the new key, verify the connection, then commit.
 
+Raw access-key creation and rotation-start return plaintext and belong outside the agent
+session; follow [recipe 5](03_recipes.md#5-prepare-an-access-key-rotation-without-exposing-the-new-key)
+for the human handoff and separate revocation approval. The managed `ocx connect rotate`
+flow returns non-secret status and is a distinct command, not permission to invoke the raw
+secret-returning endpoint from an agent tool.
+
 The token backup (`<tokenfile>.prev`) is not deleted while a rotation is in flight, and
 commits only once both sides are confirmed to have accepted.
 

src/lib/windows-secret-acl.ts

@@ -715,18 +715,22 @@ function sanitizedAclError(diagnostics: string, cause: unknown): NodeJS.ErrnoExc
   return error;
 }
 
-function previousTimeoutError(retryConsumed: boolean): NodeJS.ErrnoException {
+type TimeoutMemoRefusalError = NodeJS.ErrnoException & {
+  aclFailureOrigin: "timeout_memo_refusal";
+};
+
+function previousTimeoutError(retryConsumed: boolean): TimeoutMemoRefusalError {
   if (retryConsumed) {
     const error = new Error(
       "ACL hardening skipped — the previous timeout recovery was already consumed",
     ) as NodeJS.ErrnoException;
     error.code = "EACLRETRYEXHAUSTED";
-    return error;
+    return Object.assign(error, { aclFailureOrigin: "timeout_memo_refusal" as const });
   }
-  return sanitizedAclError(
+  return Object.assign(sanitizedAclError(
     "ACL hardening skipped — previous attempt timed out",
     Object.assign(new Error("timeout"), { code: "ETIMEDOUT" }),
-  );
+  ), { aclFailureOrigin: "timeout_memo_refusal" as const });
 }
 
 /** Consume, but never reset, the single explicit recovery attempt for this key. */

src/responses/state.ts

@@ -169,7 +169,10 @@ async function snapshotOnDiskMatches(path: string, payload: string, payloadBytes
     return false;
   }
 }
-const spillCounters = { writes: 0, writeFailures: 0, readFailures: 0 };
+const spillCounters = {
+  writes: 0, writeFailures: 0, readFailures: 0,
+  aclRetryReturnedTimeouts: 0, aclTimeoutMemoRefusals: 0,
+};
 
 export type ResponseSpillWriteFailureCode =
   | "EACLRETRYEXHAUSTED"
@@ -184,16 +187,22 @@ export type ResponseSpillWriteFailureCode =
 
 export type ResponseSpillWriteStatus = "initial" | "healthy" | "degraded";
 
+export type ResponseSpillWriteFailureOrigin =
+  | "retry_returned_timeout"
+  | "timeout_memo_refusal";
+
 interface ResponseSpillWriteHealth {
   consecutiveFailures: number;
   lastFailureCode: ResponseSpillWriteFailureCode | null;
+  lastFailureOrigin: ResponseSpillWriteFailureOrigin | null;
   lastFailureAt: number | null;
   lastSuccessAt: number | null;
 }
 
 const spillWriteHealth: ResponseSpillWriteHealth = {
   consecutiveFailures: 0,
   lastFailureCode: null,
+  lastFailureOrigin: null,
   lastFailureAt: null,
   lastSuccessAt: null,
 };
@@ -226,6 +235,20 @@ function classifySpillWriteFailure(error: unknown): ResponseSpillWriteFailureCod
   return "EUNKNOWN";
 }
 
+/** The spill writer preserves ACL errors in cause; only a fixed memo marker is diagnostic. */
+function spillAclMemoRefusalOrigin(error: unknown): "timeout_memo_refusal" | null {
+  let cursor = error;
+  for (let depth = 0; depth < 4 && cursor && typeof cursor === "object"; depth += 1) {
+    const record = cursor as { code?: unknown; aclFailureOrigin?: unknown; cause?: unknown };
+    if ((record.code === "ETIMEDOUT" || record.code === "EACLRETRYEXHAUSTED")
+      && record.aclFailureOrigin === "timeout_memo_refusal") {
+      return "timeout_memo_refusal";
+    }
+    cursor = record.cause;
+  }
+  return null;
+}
+
 function noteSpillWriteSuccess(): void {
   spillCounters.writes += 1;
   spillWriteHealth.consecutiveFailures = 0;
@@ -235,11 +258,20 @@ function noteSpillWriteSuccess(): void {
 function noteSpillWriteFailure(
   error: unknown,
   override?: ResponseSpillWriteFailureCode,
+  retryOrigin: ResponseSpillWriteFailureOrigin | null = null,
 ): void {
+  const code = override ?? classifySpillWriteFailure(error);
+  const origin = code === "ETIMEDOUT" || code === "EACLRETRYEXHAUSTED"
+    ? spillAclMemoRefusalOrigin(error) ?? retryOrigin
+    : null;
   spillCounters.writeFailures += 1;
   spillWriteHealth.consecutiveFailures += 1;
-  spillWriteHealth.lastFailureCode = override ?? classifySpillWriteFailure(error);
+  spillWriteHealth.lastFailureCode = code;
+  spillWriteHealth.lastFailureOrigin = origin;
   spillWriteHealth.lastFailureAt = now();
+  // Count terminal publications, not ACL calls or a transient first attempt.
+  if (origin === "retry_returned_timeout") spillCounters.aclRetryReturnedTimeouts += 1;
+  else if (origin === "timeout_memo_refusal") spillCounters.aclTimeoutMemoRefusals += 1;
 }
 /**
  * Admission-boundary observability (test-visible). directSpills: oversized
@@ -418,6 +450,7 @@ async function runPendingResponseSpill(job: PendingResponseSpill): Promise<void>
   const candidate = job.candidate;
   let ref: ResponseSpillRef | null = null;
   let exhaustedAclRetry = false;
+  let aclRetryFailureOrigin: ResponseSpillWriteFailureOrigin | null = null;
   try {
     const state = spillPayloadForResident(candidate);
     try {
@@ -437,6 +470,9 @@ async function runPendingResponseSpill(job: PendingResponseSpill): Promise<void>
         });
       } catch (retryError) {
         exhaustedAclRetry = isAclTimeout(retryError);
+        // A returned timeout can also mean an exhausted budget before the next OS command.
+        aclRetryFailureOrigin = spillAclMemoRefusalOrigin(retryError)
+          ?? (exhaustedAclRetry ? "retry_returned_timeout" : null);
         throw retryError;
       }
     }
@@ -460,7 +496,7 @@ async function runPendingResponseSpill(job: PendingResponseSpill): Promise<void>
   } catch (error) {
     if (ref) deleteResponseSpill(ref);
     if (states.get(job.id) === candidate && !job.cancelled) {
-      noteSpillWriteFailure(error, exhaustedAclRetry ? "EACLRETRYEXHAUSTED" : undefined);
+      noteSpillWriteFailure(error, exhaustedAclRetry ? "EACLRETRYEXHAUSTED" : undefined, aclRetryFailureOrigin);
       replaceWithSpillFailure(job.id, candidate);
       deferSupersededSpill(job.supersededSpill);
     }
@@ -2188,6 +2224,9 @@ export interface ResponseStateMetrics {
   spillWriteStatus: ResponseSpillWriteStatus;
   spillWriteConsecutiveFailures: number;
   spillLastWriteFailureCode: ResponseSpillWriteFailureCode | null;
+  spillLastWriteFailureOrigin: ResponseSpillWriteFailureOrigin | null;
+  spillAclRetryReturnedTimeouts: number;
+  spillAclTimeoutMemoRefusals: number;
   spillLastWriteFailureAt: number | null;
   spillLastWriteSuccessAt: number | null;
   spillReadFailures: number;
@@ -2240,6 +2279,9 @@ export function responseStateMetrics(): ResponseStateMetrics {
         : "initial",
     spillWriteConsecutiveFailures: spillWriteHealth.consecutiveFailures,
     spillLastWriteFailureCode: spillWriteHealth.lastFailureCode,
+    spillLastWriteFailureOrigin: spillWriteHealth.lastFailureOrigin,
+    spillAclRetryReturnedTimeouts: spillCounters.aclRetryReturnedTimeouts,
+    spillAclTimeoutMemoRefusals: spillCounters.aclTimeoutMemoRefusals,
     spillLastWriteFailureAt: spillWriteHealth.lastFailureAt,
     spillLastWriteSuccessAt: spillWriteHealth.lastSuccessAt,
     spillReadFailures: spillCounters.readFailures,
@@ -2360,8 +2402,11 @@ export function clearResponseStateMemoryForTests(): void {
   spillCounters.writes = 0;
   spillCounters.writeFailures = 0;
   spillCounters.readFailures = 0;
+  spillCounters.aclRetryReturnedTimeouts = 0;
+  spillCounters.aclTimeoutMemoRefusals = 0;
   spillWriteHealth.consecutiveFailures = 0;
   spillWriteHealth.lastFailureCode = null;
+  spillWriteHealth.lastFailureOrigin = null;
   spillWriteHealth.lastFailureAt = null;
   spillWriteHealth.lastSuccessAt = null;
   replayScopeMismatchDrops = 0;