Skip to content

fix(credential-proxy): give each sandbox its own share of in-progress operations - #3284

Merged
freeqaz merged 3 commits into
mainfrom
fix-2237
Oct 23, 2026
Merged

freeqaz merged 3 commits into
mainfrom
fix-2237

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

CredentialProxy.resolve() charges its in-progress operation cap (1024 per API process) before it authenticates the peer, and a ready handle keeps its slot until it closes or its deadline passes (at most 30 s). Every Sandbox shares that one cap. A single Sandbox that keeps 1024 handles or pending policy and Secret lookups open makes resolve refuse every other Agent in the installation with unavailable/resolver, for up to 30 s at a time.

#3281 fixed the same problem for the replay table only. The operation cap was left alone there because it is charged before authentication, when the only Sandbox identity is the unauthenticated sandboxId parameter. A share keyed on that value could be spent by a peer claiming another Sandbox's ID.

Change

packages/occ/src/credential-proxy.ts. I followed #3281's pattern.

  • Before verification: resolve still charges one slot before any await. That slot counts toward the 1024 cap and toward a new global authentication budget of 256 (OPERATION_AUTHENTICATION_BUDGET). This keeps unauthenticated floods bounded. The bundled OpenShell verification runs in memory, so in practice the budget only limits same-tick bursts.
  • After verification: once verifyPeer succeeds, peer.sandboxId matches, and the peer has not expired, the slot moves in the same synchronous turn out of the authentication budget and into a share keyed on the verified sender and Sandbox (OPERATION_SANDBOX_SHARE, 256: a quarter of the cap). The replay table already keys its share the same way.
  • Refusal: hitting the cap, the budget or the share returns the existing unavailable/resolver refusal.
    • The cap and budget checks still run before the request is copied.
    • A request refused at its share starts no installation, policy or Secret work and reserves no request ID.
  • Release: the existing OperationCharge release point returns the slot exactly once. That is the last hold on every path: refusal, throw, abort, handle close, and deadline discard. The release returns the slot to the budget if verification never settled, or to the share if it did.
  • Docs: the "Request-time capacity" table in docs/reference/drivers/credential-proxy.md now lists the three operation bounds. The request-time trace in docs/flows/credential-source-egress.md now describes the slot handoff. That doc stays under its word cap (2498 of 2500).

Sizing: a quarter of the cap, as in #3281. No existing test needed more once the tests that filled the cap from one Sandbox were rewritten to use four. Replay protection, the replay table and the OpenShell exchange caps are unchanged.

Provisional decision. The share identity (sender, Sandbox), the quarter share and the 256 authentication budget are provisional, so a maintainer can overrule them. One alternative is to authenticate at the transport, before resolve, and pass an authenticated identity in. That would remove the global pre-authentication budget, but it changes the driver/transport boundary, so it is out of scope here.

Verification

Tests in tests/integration/static-egress.test.mjs use the shared syntheticDispatchFixture and its sandboxes() helper:

  • New: "one Sandbox's open handles leave operation capacity to other Sandboxes". It fails on main ("another Sandbox's first request" is refused once one Sandbox holds 1024 open handles). It also covers release on handle close and on deadline discard.

  • New: "requests still authenticating share a small budget charged before verification". It covers the global budget, refusal before verification, and the move to the share after verification.

  • New: "every refusal returns its operation slot". It runs 257 refusals on each of ten paths:

    • before verification: aborted, unknown peer, verification throws, Sandbox ID mismatch;
    • after verification: deadline passed, installation missing, unsupported route, policy denies, policy throws, Secret read throws.

    After the loop it checks that the share and the authentication budget still hold exactly. A slot released twice would show up as extra capacity.

  • Updated: "in-flight Secret selection and outstanding handles consume the same bounded capacity" now fills the 1024 cap with four Sandboxes and checks that a fifth is refused before its request is copied.

  • Updated: "cancelled and denied exchanges retain capacity until receipt work settles" now holds the target Sandbox's share instead of the global cap.

  • 16 mutants, all killed. They cover each guard (the cap, the budget and the share, each dropped and off by one; the share refusal ignored; the share keyed on the sender only) and each release path (the budget, the share and the global count each left unreleased; the authentication slot not taken; the move keeping the authentication slot; a release on every hold; the budget or the share released twice).

  • The capacity and replay tests (12 cases) pass 20/20, each run pinned to 2 CPUs.

  • Internal security review: approved, with no must-fix. I applied its should-fixes: a vacuous "never verified" assertion, double-release coverage, and docs wording.

  • static-egress, credential-proxy-peer-pin and credential-proxy-withdrawal: 230 pass and 1 skipped (pre-existing) locally on Node 24.

  • tsc --build packages/occ, eslint and prettier are clean on the touched files.

Risks

  • Per-Sandbox limit: one Sandbox can now hold at most 256 operations at once (was 1024). At the 30 s maximum deadline that is still far above an Agent's normal concurrency.
  • Four busy Sandboxes can still fill the installation-wide cap together. As with the replay table, there is no per-Namespace share yet.
  • Same-tick bursts: a burst of more than 256 requests from all Sandboxes in one tick, before any verification settles, is refused with unavailable/resolver.

… operations

Requests still authenticating draw on a global budget of 256. Once the peer is
verified and its Sandbox matches, the slot moves to a share keyed on the
verified sender and Sandbox (256, a quarter of the 1024 cap), so one Sandbox's
open handles no longer refuse every other Agent.
@freeqaz
freeqaz requested a review from a team as a code owner October 23, 2026 08:16
@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

…ation

Review follow-ups: the refusal test now checks after its loop that the share and
the authentication budget still hold exactly, adds the missing-installation path,
and the budget test counts every verification call. The docs name the
installation lookup in the release rule and the cost of a Driver whose
verification does I/O.
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 23, 2026
@freeqaz

freeqaz commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

What this changes

This PR adds a shared authentication budget and verified per-Sandbox shares to Credential Proxy operation accounting.

Example: Sandbox A attempts to retain 1024 operation handles while Sandbox B starts a request.

  • Before: A can occupy all 1024 slots, causing B to receive unavailable/resolver.
  • After: A can retain 256 operations, leaving capacity for B.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The focused accounting repair has meaningful lifecycle coverage; confidence is limited by harness-only verification and the missing required flow update.
Proof confidence 🦐 gold shrimp (3/6) This collaborator-authored PR is outside the ordinary contributor proof gate; supplied verification exercises the production resolver with synthetic identity, policy, Secret, and receipt dependencies rather than the shipped transport.
Patch quality 🐚 platinum hermit (4/6) 1 actionable review finding remain.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: One Sandbox's retained handles or pending lookups can prevent every other Agent from resolving protected requests.
Reason: The collaborator-authored direction repairs a concrete isolation failure within the existing owner and records its narrower concurrency limits.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep open: this PR repairs operation exhaustion still present on main. No introduced runtime defect was found; one required flow-documentation update remains.

Priority: P2
Reviewed head: 34d47a0741bc1b430565ac2a216a2ed313ab2b85

Before merge

  • Document the operation-charge handoff in the existing request flow (P3) - This adds a new admission decision and accounting transition before installation lookup, but docs/flows/credential-source-egress.md still describes only generic slot reservation and release. AGENTS.md requires an owning flow update for non-trivial runtime changes. Extend its existing request-time trace to explain that verification transfers the charge from the global authentication budget to the verified Sandbox share, and that the final outstanding hold releases it; link the reference for numeric limits.

Findings

  • [P3] Document the operation-charge handoff in the existing request flow — packages/occ/src/credential-proxy.ts:353-357

Tests

  • Missing end-to-end proof: The supplied verification does not demonstrate cross-Sandbox operation saturation and recovery through the shipped authenticated transport; base/head regression results are author-reported.
Agent review details

How this fits together

OCC Credential Proxy receives transport-owned Sandbox requests, verifies identity, checks installation and policy, and selects credential material before separate send authorization.

flowchart TB
A[Sandbox request] --> B[Global authentication budget]
B --> C[Peer verification]
C --> D[Verified Sandbox operation share]
D --> E[Installation and replay checks]
E --> F[Policy and material selection]
F --> G[Handle and receipt lifecycle]
Loading

Technical review

Best possible solution:

Retain the existing registry and final-hold release mechanism, add verified Sandbox shares, and document the accounting handoff in the owning flow.

Do we have a high-confidence way to reproduce the issue?

Current-main source permits one verified Sandbox to retain all 1024 operation slots; the supplied regression targets that exact failure, but tests were not run during this read-only review.

Is this the best way to solve the issue?

The repair uses the existing operation owner, authenticates before assigning a share, preserves atomic admission, and keeps pending work charged until settlement.

Full review comments:

  • [P3] Document the operation-charge handoff in the existing request flow — packages/occ/src/credential-proxy.ts:353-357
    This adds a new admission decision and accounting transition before installation lookup, but docs/flows/credential-source-egress.md still describes only generic slot reservation and release. AGENTS.md requires an owning flow update for non-trivial runtime changes. Extend its existing request-time trace to explain that verification transfers the charge from the global authentication budget to the verified Sandbox share, and that the final outstanding hold releases it; link the reference for numeric limits.
    Confidence: 0.98

Overall correctness: patch is correct
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against f2a5b4b745ab.

Provenance checked

  • Operation admission and release accounting changes intended behavior with a stated reason (3115679: The registry shares one charge between a live handle and pending phases so cancellation cannot release capacity while work remains outstanding.)
  • Capacity integration tests keeps the original intent (3115679: Existing tests protect the shared capacity ceiling and retention through receipt settlement; the replacements preserve those invariants while adding Sandbox isolation.)
  • Request-time capacity reference changes intended behavior with a stated reason (6fa1aa0: The earlier replay repair documents process-wide bounds and explicitly defers operation-slot isolation.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Likely related people:

  • Free Wortley: Raw commit 6fa1aa0 adds packages/occ/src/credential-proxy.ts:95 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 6fa1aa00c9f9; files: packages/occ/src/credential-proxy.ts)

Review metrics

Metric Value Why it matters
Change distribution Production +51/-7; tests +239/-54; documentation +18/-10 Most growth protects distinct capacity and release failures using the existing shared fixture.

Labels

Label changes:

  • add P2: One Sandbox can exhaust shared request capacity and refuse other Agents' protected egress on the affected execution surface.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

  • P2: One Sandbox can exhaust shared request capacity and refuse other Agents' protected egress on the affected execution surface.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • For stronger confidence, demonstrate one Sandbox reaching its operation share while another completes a protected request through the authenticated transport.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 23, 2026, 4:35 AM ET / 08:35 UTC.

@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@freeqaz
freeqaz merged commit 9db4478 into main Oct 23, 2026
79 checks passed
@freeqaz
freeqaz deleted the fix-2237 branch October 23, 2026 08:37
@freeqaz

freeqaz commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

ClawSweeper could not start a re-review for this item.
Reason: re-review requires an open issue or PR.

freeqaz added a commit that referenced this pull request Oct 23, 2026
…st flow (#3287)

The request flow now says the Credential Proxy reserves an authentication slot first and moves it to the verified Sandbox's share before the installation lookup, matching #3284.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant