Repository navigation
Conversation
…x gateway outage #2957 treated every gRPC UNAVAILABLE as a gateway outage, but the pinned OpenShell gateway itself answers UNAVAILABLE for failures that do not clear by themselves (resource admission's Kubernetes 401/403/TLS errors, credential driver errors), and grpc-js returns it for an untrusted gateway certificate. Those waited the whole convergence deadline instead of failing within the attempt budget. Only grpc-js's own UNAVAILABLE details for a gateway it never reached or lost under a call (no connection: refused, reset, unreachable, timed out, no address; connection dropped; stream refused) now count as an outage. Every other UNAVAILABLE, including TLS failures, is an ordinary failed request again, and CreateSandbox keeps naming the gateway's reason (finding 2042).
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
CI's Node refuses direct use of an HTTP/2 session's socket; destroy the accepted connection instead.
…tener Review nits: on a host without IPv6 whose localhost still maps to ::1, the last address's unreachable error would hide the certificate error. Also name the excluded handshake-close and proxy HTTP status shapes in the allowlist comment.
clawsweeper
Bot
added
clawsweeper:intake:bug
Contributor
|
Codex review: needs maintainer review before merge. What this changesThis PR distinguishes unanswered OpenShell requests from server refusals and TLS failures when deciding whether to defer deployment retries. Example: CreateSandbox returns gRPC status 14 because Kubernetes denies the gateway access to Secret tenant/model.
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Regression provenancePossible regression — probable (reviewed change; known regression link). No predecessor PR is attributed. Merge readiness✅ Ready for maintainer review Keep this PR open: it repairs a source-confirmed classification regression on main, and no actionable introduced defect was found. Priority: P2 Before mergeNone. FindingsNone. Tests
Agent review detailsHow this fits togetherThe controller's OpenShell client converts gRPC failures into dependency errors that the revision worker uses to choose deadline-bound waiting or budgeted retries. flowchart TD
Revision[Agent revision work] --> Worker[Controller worker]
Worker --> Compute[Compute Driver]
Compute --> Client[OpenShell gateway client]
Client --> Gateway[OpenShell gateway]
Gateway --> Classification[Failure classification]
Classification --> Worker
Worker --> Status[Deployment status]
Technical reviewBest possible solution: Retain the focused transport classification in the existing gateway client and preserve the worker's established retry policies. Do we have a high-confidence way to reproduce the issue? Main's unconditional status-14 classification, the pinned gateway's refusal responses, and the worker's subtype-based retry branch establish the failure path; this review executed no target code. Is this the best way to solve the issue? The positive transport-detail allowlist is a narrow repair using existing owners, preserves sanitization, and has targeted real-wire coverage for the demonstrated failures. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 5f494a0fdf8f. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
Likely related people:
Review metrics
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 19, 2026, 4:31 PM ET / 20:31 UTC. |
freeqaz
added a commit
that referenced
this pull request
Oct 20, 2026
The branch's closed diagnostics keep reflected gateway text out of the CreateSandbox failure message, which HTTP dependency logging also emits; postgres-worker-agent-revision proves a reflected secret never reaches logs. Restoring main's sanitized detail (#2963) broke that, so the detail stays out and main's two wire expectations follow the closed message instead. The outage classification from #2963 is unchanged.
freeqaz-openai
added a commit
that referenced
this pull request
Oct 22, 2026
* Select the mandatory installed custody qualification and identify the actual authorized update before dropping its COMMIT reply
* Enroll the encrypted prepared credential family, proactive renewal and fenced recovery with explicit execution authority
* Join fresh-execution Conduit contracts, real signed prepared reads and qualified PAT-to-custody migration history
Keep credentialed qualification separate from installed enforcement. Use only installation-owned verified HTTPS trust for the synthetic issuer and reject fallback before enrolling. Preserve the human-only RFC merge boundary.
* Require the authentic execution scope, policy digest and reply custody in the prepared dispatch receipt
Proved by the real signed InstallationExchange from enrolled, worker-prepared PostgreSQL family through acknowledged permit, HTTPS metadata read and protected response. Keep legacy static receipts unchanged.
* Authorize explicitly credential-free protected repository reads through the same acknowledged dispatch
* Bind prepared GitHub token custody and retirement to the issuing HTTPS authority
* Reject an unbounded prepared issuer port and keep OpenShell client and authority Secrets distinct
* Exercise custody with authorized competing recipients and one actual final trailer
* Qualify static inference for the admitted execution and distinguish unmanaged bypass
* Bound prepared credential selection and decryption to the signed access deadline
* Check protected OpenShell registration and public CA in the rendered Gateway before creating k3d
* Resolve the installed issuer inspector client from its controller package
* Accept the OpenShell discovery capability set without reordering enforcement evidence
* Withdraw the admitted reader when Conduit refuses release
Keep an authorized active reconciliation pending until Compute closes the revision policy and observes termination of its exact workload. Preserve the release cutoff and the original preparation, stop and retire withdrawal paths.
Recognize the governed credential-free repository grant in protected Kubernetes runtime projection, with inference tools still disabled.
* Launch installed egress qualification clients through the OpenShell workload boundary
* Observe installed response custody in the supplier’s structured Sandbox log
* Report custody as unobservable when its in-flight log read reaches the deadline
* Inspect live API and worker issuer routing after an authority restart
* Specify warm-read custody, renewal retirement and released-catalog upgrade proof
* Wipe detached GitHub warm material when its read cannot complete
* Enforce durable PostgreSQL revision admission and preserve snapshot constraint errors
* test(kubernetes): declare and verify durable execution admission in k3d fixtures
* test(worker): declare execution admission for PostgreSQL revision fixtures
* test(compute): declare unprotected admission in legacy execution fixtures
* fix(controller): clear retirement bytes when claim handoff fails
* Fix the generated egress receipt dispatch constraint
* docs: preserve stopped 0055 Agent and resume development with a new Agent
* Remove the unused credential source adoption rehearsal
* Clarify the owning-release cutoff and independently authorized new-Agent recovery
* docs(credential-sources): drop the egress_proxy refusal sentence from the t06 merge
The sentence named the old egress_proxy Driver and said only openai sources are accepted. This branch selects the Conduit Driver and also admits prepared GitHub and credential-free metadata sources, so the sentence is no longer accurate. The file now matches the branch head before the merge.
* docs(repository-credentials): drop the expires_at sentence from the t10 merge to stay under the word cap
This flow doc is close to the 2,500-word docs limit on this branch (local estimate about 2,465 before the merge, 2,490 after). The conformance row added by the merge still pins the Z-form-only rule, so the sentence is not needed. The file now matches the branch head before the merge.
* test(egress): require committed refusal and selected chart validation
* Name the Driver Credential Proxy and select it with drivers.credential_proxy
* Name the protected provider capability Credential Proxy in RFC-0021
* Require Credential Proxy selection and OpenShell Backend membership to agree
* Stop credential access when the exchange closes or its access deadline passes
* Enforce the absolute credential deadline at Kubernetes dispatch, not just when its timer runs
* Start the protected development worker after tenant access and Gateway rollout
* Preserve ordered worker evidence when installed custody deployment fails
* Make exchange ownership, credential read lifetime, and request capture explicit
* Catch receipt digest regressions locally and name Credential Proxy refusals
* Do not open expired or superseded prepared GitHub material on a request
* Clear prepared token drafts when PostgreSQL publication rolls back
* Use the prepared GitHub credential envelope for metadata injection
* Keep ambiguous GitHub issuance out of use while preserving identifiable cleanup custody
* Reject conflicting GitHub token repository selection
* Bound installation openings and retain proxy capacity until work settles
Authenticate supervisors before allocating a challenge. Account for each credential operation until both its handle and any in-flight policy or receipt work have released it.
* Protect assembled Responses output text
* Protect incremental Responses text snapshots
* Protect assistant text visible from independent Responses items
* Check prepared send currentness in the active platform transaction
* Assert both OpenShell endpoint trust and the selected permission profile
* Protect completed and streamed Codex Plan text before response release
* Document the finite response custody boundary and keep integrated references concise
* Update installed custody refusal expectation
* Record whether a failed revision reached candidate preparation
* Check visible text in each completed Responses message
* Count the citation-projection refusal and its distinct recovery in the installed test
* Reconcile installed source cleanup only after confirmed prepared retirement
* Report an empty retirement lookup without claiming custody was observed
* Describe the paired protected-admission path that Kubernetes Compute implements
* Exercise stock OpenShell authorization against current OCE grants without releasing credentials
* Enforce a Git-only App grant at the repository broker
Bind the repository, allowed push refs, transport and contents permission in the grant. Keep ordinary GitHub App and development-token grants unchanged. This is the transport policy slice, not adoption of this broker issuance path by protected egress.
* Let a dedicated Agent use native Git with an already prepared repository credential
* Qualify private Git and live access withdrawal on an installed Skipper
* Let an admitted Agent open one draft pull request and continue work on it
* Share protected Responses exchange setup across scenarios
* Centralize send receipt source evidence projection
* Bind prepared Git ingress to the broker Service hostname
* Explain draft handoff and what the Agent receipt can establish
* Recognize the prepared draft migration on upgrade and on the next startup
* Parse the certificate bundle in one pass and take explicit pull request text
* Keep protected Harness credentials out of the legacy attachment path
* Distinguish token retirement from Git traffic in the installed withdrawal proof
* Keep the installed proof controller credential out of child processes
* Add the Git request and credential components for OpenShell qualification
Authorize against the live repository grant without returning credential material. Exercise Git operations, a bounded draft and withdrawal while the controlled provider still accepts the credential. Production continues to refuse this unqualified composition.
* Verify the operator-owned namespace used by an OpenShell installation
* Authorize ordinary Git and draft pull requests from the admitted OpenShell sandbox
* Specify installation-wide GitHub access through the same Credential Proxy
* fix(egress): allow gzip Git fetch negotiation through OpenShell
Git's remote-curl gzips every upload-pack request body over 1024 bytes,
and stock OpenShell shows content-encoding to the request middleware.
The OpenShell Git authority's visible-header allowlist omitted it, so the
fail-closed middleware denied clone/fetch of any repository with more than
about 20 refs, or any fetch with a longer have list.
Accept exactly one `content-encoding: gzip` (case-insensitive) and only on
the git-fetch (upload-pack) route, whose body this authority never parses:
it bounds the size and binds the encoded bytes and header in the request
digest. Discovery, receive-pack and draft create still refuse any
content-encoding, and other encodings (br, deflate, x-gzip, identity,
stacked or repeated gzip) are refused everywhere.
The new test fails on 924e0532a (gzip upload-pack denied) and passes here.
* fix(state): read pre-Credential-Proxy revisions after upgrade
Problem: 0058 adds agent_revisions_execution_admission NOT VALID and backfills
nothing, while revisionFromRow refuses any admitted_spec without
execution_admission. Upgrading a released catalog therefore left every
pre-existing Agent unreadable: createRevision decoded the earlier revisions and
failed, and the worker's reconcile, stop, retire and delete paths and the
active-revision lookup all failed the same way, so no new revision could replace
them (board finding 2120).
Change: migration 0066_legacy_revision_execution_admission records
{version: 2, mode: unprotected} on revisions that carry no decision, no egress
plan and no protected reader marker, which is the mode those revisions were
admitted under, restores the immutability trigger, then validates the constraint
so every stored row is proven consistent or the upgrade fails as a whole. The
read check is unchanged: a row that claims protection without a consistent
decision still fails closed, and a recorded decision cannot later pick up a
protected route. The history script, canonical manifest and compat suite learn
the prefix-65 catalog, the rollback case aborts 0066 at its VALIDATE, and the
upgrade expectation adds exactly that one field.
Test: every populated-history subtest now decodes the seeded revision through
findRevision and listRevisions before comparing stored rows. On the unfixed head
all 32 histories seeded before 0058 fail with the reader's refusal.
* fix(egress): accept documented Responses envelope fields
The streamed `response` envelope validator declared only the fields the
synthetic fixture ever carried. Live provider envelopes also carry
`top_logprobs`, `conversation: null`, `billing` and other documented
members, so every real `response.created`/`response.completed` refused
and the authority recorded `response_blocked/response_policy`: no model
output passed the Credential Proxy against the real provider.
Declare the `Response` model fields from openai-python
src/openai/types/responses/response.py (0f73f1d5, 2026-10-12), each with
a type check, restricted to what the admitted request can produce.
Provider-side state the request cannot ask for (conversation, prior
response, prompt template, moderation, background) must still echo its
empty value. The envelope stays closed: an undeclared field, a wrong
type or reflected material still refuses the whole response.
Test: a live-shaped created/in_progress/.../completed sequence passes
byte-for-byte; undeclared, mistyped and credential-bearing envelope
fields refuse. Fails on the previous head at the first live envelope.
* Renumber the Credential Proxy migrations to 0060-0068 after main's 0059
main landed 0058_sandbox_policies and 0059_administrator_sandbox_policy_grants
(#2599), which collided with this branch's 0058-0065 and the 0066 legacy
admission backfill (board finding 2051). The nine unpublished migrations keep
their bytes except for two literals: 0060 and 0065 re-create
agent_revisions_admitted_snapshot and now keep main's sandbox_policy key, so
the constraint still admits every OpenShell revision carrying a policy snapshot
and the upgrade does not fail at ADD CONSTRAINT.
Updated together: the journal and manifest entries (idx 59-67, when ...060-068,
sha256 recomputed), the history classifier (57 preSandboxPolicies,
58 preSandboxPolicyAdministrator, 59 preStaticEgress ... 67
preLegacyRevisionExecutionAdmission; the experimental-slot detector reads
static egress at entries[59]), the compat suite's three history loops, seed
thresholds and rollback tags (prefix 58 aborts in 0060 at GRANT), the
experimental 0057 slot case (shift by three), and the docs and schema comments
that cite the numbers. Catalog digests for the renumbered shapes follow in the
next commit.
* fix(state): read pre-Credential-Proxy revisions after upgrade
Problem: 0058 adds agent_revisions_execution_admission NOT VALID and backfills
nothing, while revisionFromRow refuses any admitted_spec without
execution_admission. Upgrading a released catalog therefore left every
pre-existing Agent unreadable: createRevision decoded the earlier revisions and
failed, and the worker's reconcile, stop, retire and delete paths and the
active-revision lookup all failed the same way, so no new revision could replace
them (board finding 2120).
Change: migration 0066_legacy_revision_execution_admission records
{version: 2, mode: unprotected} on revisions that carry no decision, no egress
plan and no protected reader marker, which is the mode those revisions were
admitted under, restores the immutability trigger, then validates the constraint
so every stored row is proven consistent or the upgrade fails as a whole. The
read check is unchanged: a row that claims protection without a consistent
decision still fails closed, and a recorded decision cannot later pick up a
protected route. The history script, canonical manifest and compat suite learn
the prefix-65 catalog, the rollback case aborts 0066 at its VALIDATE, and the
upgrade expectation adds exactly that one field.
Test: every populated-history subtest now decodes the seeded revision through
findRevision and listRevisions before comparing stored rows. On the unfixed head
all 32 histories seeded before 0058 fail with the reader's refusal.
* Record the catalog digests for the renumbered migration chain
Computed on a disposable PostgreSQL 18.6 server with stock Drizzle execution
as occ_migrator, one shape per journal prefix 57-68. Prefixes 57 and 58
reproduce main's preSandboxPolicies and preSandboxPolicyAdministrator digests,
which validates the method; 59 (preStaticEgress) equals main's completed
catalog because 0059 only rewrites role permissions. The Drizzle ledger digest
is unchanged for every prefix. 0068's VALIDATE leaves
agent_revisions_execution_admission validated at the completed shape.
* Refresh the manifest hash of 0064 after its comment edit
The cross-reference comment in 0064_protected_credential_free_metadata was
renumbered after the manifest entries were computed, so the history check
refused the chain with 'source bytes differ'. Every entry hash now matches its
file again; the catalog digests are unaffected by a comment.
* test: pin expected errors after the merge with main
Main's lint rule (#2859) refuses matcher-less assert.throws/rejects; the
branch's tests predate it. Each site now names the error it expects:
AssertionError codes for the connected-preflight input guards, AbortError
for the aborted material read, provider/ownership messages for the egress
and prepared-Git paths, and the Secret/ConfigMap namespace guards.
Also from the merge: configuration-startup lost its spawnSync import when
main moved the other child-process calls to captureChild; platform-state.ts
carried isDeepStrictEqual twice, which failed every Postgres test file at
module load; the draft-pull privilege history case now seeds prefix 67
(0067_prepared_repository_draft_pulls after the renumbering).
* fix(egress): close the merge findings against main (1599, 2054, 2055)
1599: main's transient startup-probe retry (#2639) sat beside the model
timeout retry that the protected profile already refuses, so a protected
execution (provider tools none) regained one retry through a 5xx, a dropped
stream or a SIGKILL. Both the event-loop candidate and the retry now carry
the same guard; two protected scenarios prove a single business request.
2054: the egress-enrollment startup step held its failure under a check
outside RUNTIME_FAILURE_CHECKS (#2763), so OCC recorded UNRECOGNIZED. The
vocabulary names it and the diagnostics flow explains it.
2055: protected removeRevision recorded a withdrawal on a bare GetSandbox
NOT_FOUND. It now confirms absence through GetWorkspace and a second read
like the Sandbox and credential-gateway drivers (#2757, #2762), and the
withdrawal chains the claim signal instead of a detached 30 s timeout. The
static-egress fake answers GetWorkspace and the test holds on a refusal.
* fix(sandbox): restore CreateSandbox's sanitized gateway detail; pin four tests
The merge with main dropped the CreateSandbox branch of
gatewayRequestFailureMessage, so a refused CreateSandbox lost the gateway's
reason (the wire test expects it, as before finding 2040). The sanitized
detail helpers return with it.
Tests: the v0.1.4 wire fixture on main already declares GetWorkspace, so
the prepared-Git provider test no longer injects it; the preprovisioned
Namespace guard, protected execution admission and the connected database
URL validator are pinned to the errors they actually raise.
* test: await the async Codex startup fixture; expect the CreateSandbox detail
This branch made checkCodexStartup async for the protected enrollment
exchange, while main's two output-class suites (#2962) call it as a sync
function and read its diagnostics; they now await it. The create-time
exposure case expects the sanitized CreateSandbox detail main restored.
* test: accept both admission refusals from validateExecutionAdmission
A legacy or unselected revision is refused as a missing durable admission
(assertExecutionAdmission), another Compute as an unavailable protected
execution; the pin names both.
* fix(sandbox): keep CreateSandbox diagnostics closed (finding 2040)
The branch's closed diagnostics keep reflected gateway text out of the
CreateSandbox failure message, which HTTP dependency logging also emits;
postgres-worker-agent-revision proves a reflected secret never reaches logs.
Restoring main's sanitized detail (#2963) broke that, so the detail stays
out and main's two wire expectations follow the closed message instead.
The outage classification from #2963 is unchanged.
* docs: bring three flow pages back under the 2500-word limit after the merge
The merge left agent-repository-credentials, openshell-sandbox-provisioning
and repository-credentials over main's hard limit. The branch's additions
are condensed; reference material moves where it belongs (CA trust and
closed worker diagnostics to the Sandbox driver reference, OAuth/Git
boundaries and the bounded metadata-read policy to the repository
credential references); the two longest changelogs keep their most recent
entries with a pointer to Git history. The scoped-token line main added to
the provisioning page, dropped in the merge, is restored.
* test(egress): protected admission refuses an Agent's network policy additions
The main sync made createRevision refuse Agent network policy additions on
a protected execution (finding 2053) instead of dropping them from the
protected profile, but nothing exercised it. static-egress now deploys
through the real route with additions saved (409 RESOURCE_CONFLICT, no
revision) and again without them (protected revision admitted with the
Driver-defaults snapshot); the suite entry pins the case.
* fix(occ): refuse Agent network policy additions and live application while the active revision is protected
A protected execution runs under its exact Credential Proxy profile, so the
sync made createRevision refuse Agent network policy additions and the worker
fail their application (SANDBOX_POLICY_UNSUPPORTED). Both are late: the policy
endpoint still accepted additions for an Agent whose running version is
protected, and the console only learned it after save or deploy.
updateAgentSandboxPolicy now refuses additions, and applyAgentSandboxPolicy
refuses application, with 409 RESOURCE_CONFLICT naming protected execution
and the Credential Proxy while the active revision is protected. Clearing
additions stays allowed. The policy view carries protectedExecution so the
console shows the refusal up front and disables the editor; the Driver-
defaults snapshot on protected revisions and the worker backstop are unchanged.
Provisional rule from the captain pending the owner (fail early at save,
not deploy). static-egress covers GET/PUT/apply through the real routes; the
case fails on f71f4a7f0 (PUT 409 names the ceiling or saves, apply enqueues).
* fix(repository-credentials): run the draft helper inside OpenShell sandboxes
`oce-github pr-create --draft`, the documented OpenShell draft step, could
not run in an OpenShell sandbox: Landlock never admitted
/opt/oce/repository-credentials and PATH lacked its bin, and the helper
refused to send without https_proxy, which OpenShell strips from every
sandbox process because it mediates connections transparently.
For a git-and-draft-pr grant, the OpenShell Git runtime now adds the
image's repository client to the sandbox's read-only paths and appends its
bin to PATH (after the image's own gh). In OpenShell mode the helper
connects to api.github.com like Git does there, and the sandbox routes it
through the middleware; it ignores proxy variables, which only the Agent
could have set. The broker path keeps its proxy. Draft only, explicit
--body, one request, attempt ID before the send and no retry are
unchanged.
The OpenShell Git integration test now checks the policy path and PATH
and runs the helper with the sandbox's environment against a mediator
standing in for OpenShell: one POST to api.github.com:443 with the
placeholder bearer and a confirmed draft receipt, the same route with an
Agent-set proxy variable, and a refusal before any send on a mismatched
push ref. It fails on the previous head.
Finding 2126.
* test(fixture): let the private Git origin inflate gzip upload-pack bodies
git remote-curl gzips every upload-pack request body over 1024 bytes,
which a clone or fetch of a repository with more than about 20 refs
reaches. The synthetic private origin piped that body to git
http-backend without HTTP_CONTENT_ENCODING, so http-backend read gzip
bytes as pkt-lines and the fixture answered 502. Any installed check
against the fixture therefore could not exercise the gzip path that
18a797f5e admits in the OpenShell Git authority; a dogfood walk on a
remote install saw clone, fetch and pull fail with HTTP 502 while the
supervisor logged the request as allowed.
Pass the request's Content-Encoding to http-backend and add a seedRefs
option that creates extra refs at the seed commit. The new test clones
a 41-ref repository through the fixture; it fails on 7c1b6b64d (git
exit 128, upload-pack 502) and passes here.
* fix(openshell-git): keep the Git route through a Gateway restart and surface route failures
Problem: OpenShellGitRuntime.reconcile() and lookup() treated every failure as an
unknown provider write. A read the worker could not complete, such as GetSandbox
while the OpenShell Gateway Pod is being replaced or a timed-out observation, set
the entry's tombstone, closed the running revision's Git route for good (403 on
clone, fetch and push) and reported a bare OPENSHELL_GIT_RECONCILIATION_REQUIRED,
until the Agent was redeployed. A deployment during in-flight pushes lost the new
revision's route the same way. An App key rejection, a blocked route and the
worker's own denials left only a bare code or nothing in the log, and a revision
with a closed route still deployed as succeeded.
Change:
- The runtime's own verifications (replaced execution or supervisor, Sandbox or
material mismatch, grant change, overlapping provider) and an unknown provider
write keep the tombstone, with a reason. A failed read (gateway unavailable,
another failed gateway request, timeout, a material store read) only closes the
route: the entry stays, requests are denied, the worker observes again with a
bounded backoff (1 s doubling to 10 s) and reopens the route once the same
Sandbox, supervisor instance and provider generation verify. Nothing is
rewritten or replayed. The provider error keeps the failed read's class (never
the gateway's text) so a write refused only because the gateway did not answer
is told from one the provider refused.
- route(revisionId) exposes the state. Kubernetes Compute readiness holds an
OpenShell Git revision as REPOSITORY_GIT_ROUTE_PENDING until the route is
installed and fails it with held evidence (check repository-git-route, code
REPOSITORY_ROUTE_BLOCKED, deployment error REPOSITORY_GIT_ROUTE_BLOCKED) while
the route is blocked.
- Structured worker events: worker.error OPENSHELL_GIT_RECONCILIATION_REQUIRED
and PREPARED_GITHUB_RECOVERY_REQUIRED carry reason, namespace, agent, revision
and repositoryRef; worker.openshell-git-route-warning / worker.openshell-git-route
report a route closed by a failed read and its restoration; one
worker.openshell-git-denied line per denial, with a reason class, from the
runtime (route state) and the authority (request shape, approval).
- Operator guide section on route state, restarts and recovery.
Verification: tests/conformance/openshell-git-runtime.test.mjs drives reconcile
and lookup with a fake gateway client, provider and owner (outage, backoff,
restoration without a second rotate, each tombstone reason, route states, denial
reasons); the authority test asserts the denial reasons. On the previous head the
same gateway outage sets blocked=true after one failed read and never reopens.
tsc --build, eslint and prettier on the touched files pass locally.
Risks: Compute readiness now waits for the route, so a deployment whose material
never commits (quarantined preparation) stays pending until the convergence
deadline instead of succeeding over a dead route. A worker restart still requires
a new revision (unchanged, now documented).
* fix(openshell-git): let a deploying revision activate once its Git route verifies
Problem: since 2e80a949a, Kubernetes Compute readiness held an OpenShell Git
revision as REPOSITORY_GIT_ROUTE_PENDING until its route was installed. Prepared
material is discovered, claimed and read only for the Agent's active revision
(the sessions snapshot in prepared-repository-git.ts joins
agents.active_revision_id), so a revision that is still deploying never gets
material, never installs, and never activates. Every deployment of an Agent
with an OpenShell Git binding stayed pending (a fresh install's first deploy
sat at REPOSITORY_GIT_ROUTE_PENDING for 15+ minutes with no prepared attempt).
Change: route() reports a verified route that has never had material as
pending/awaiting_material, and openShellGitRouteReadiness() maps route states
for readiness: installed and awaiting_material are ready (the material is
issued once the revision is active and installs within seconds); bootstrapping,
unverified, unobserved and material_unavailable (an active revision that lost
its material) wait; blocked fails the deployment. The operator guide says so,
and that a new version with a rejected App key deploys, then Git answers 403
and the worker logs PREPARED_GITHUB_RECOVERY_REQUIRED.
Verification: a conformance case drives the runtime through unverified,
awaiting_material, installed, unobserved, material_unavailable and blocked and
checks readiness; on 2e80a949a the file fails (no readiness mapping). Live on a
k3d install on a remote dogfood host: see the PR comment.
Risks: a new version whose App key the provider rejects no longer holds the
deployment pending; it deploys with Git denied (fail closed) and the worker
logs the recovery code. Letting material cover the deploying revision instead
would keep that visible but changes custody (tokens minted for a revision that
may never activate) and is left for a follow-up.
* fix(egress): report a second Sandbox for one revision as the same state conflict in PostgreSQL
recordEgressInstallation in the PostgreSQL store handled only the
(sender_id, sandbox_id) conflict. A second Sandbox for the same revision hit
UNIQUE (namespace_id, revision_id) and surfaced as the generic
ResourceConflictError, while the memory store raises
ResourceStateConflictError('Egress revision already has an installed
execution.'). Map that constraint to the shared message so both stores
report it alike; the write stays refused either way.
* fix(egress-proxy): accept documented Responses refusal parts and protect their text
A provider safety refusal arrives as a refusal content part (and
response.refusal.delta/.done events). The closed output validator admitted
only output_text, so a refusal blocked the whole response and the Agent saw a
failed turn instead of the refusal. Admit the refusal part and its stream
events, and treat refusal text as assistant-visible text: it goes through the
same per-part snapshot, joined-message and terminal-output scans as
output_text.
* fix(ci): resolve the image config digest on Docker's containerd image store
skipper-prepared-git-installed checkImage read the config digest from the
platform descriptor's config.digest annotation, else Id. Docker 29's default
containerd image store reports neither: Id and the descriptor name the
platform manifest, so the identity check failed on a stock Docker 29 host.
When the annotation is absent, read the platform manifest blob from the k3d
node's content store (the import copied it there), verify it against its
digest and take its config descriptor.
* fix(openshell-git): log why a restarted worker cannot serve a revision's Git route
After a worker restart the runtime refuses to adopt a Sandbox it did not
prepare, but its 'requires a new revision after worker restart' error never
reached the log: maintenance logged only a generic
KUBERNETES_PREPARATION_FAILED every pass, while the Agent stayed active and
Git answered 403 (no_binding). Emit worker.error
OPENSHELL_GIT_RECONCILIATION_REQUIRED with reason worker_restarted and the
Namespace, Agent, revision and repositoryRef, once per revision (the rejected
bootstrap is retained, so later passes rethrow without logging again), and
document what the operator sees until a new version deploys.
* fix(openshell-git): refuse a sandbox network policy that overlaps the Git route at startup
With drivers.repo.openShell enabled, a Sandbox Driver network policy that
also allows github.com or api.github.com (for example a stock source-control
policy) was refused only when the first Agent deployed. Check the configured
policies when the OpenShell Sandbox Driver is built with the Git route, so
the Installation fails at startup and the message names the policy. The
per-deploy check stays as a backstop. Document that the Git hosts come only
from the middleware route.
* docs(openshell-git): list the prerequisites a live trial depends on
The operator guide omitted environment prerequisites hit on a k3d install on
a remote dogfood host: Sandboxes need Landlock ABI v3 (Linux 6.2 or newer),
the OpenShell supervisor trusts only its image's CA bundle for GitHub and the
model endpoint, and a Codex approvalPolicy of on-request leaves Git commands
unanswered. Add a short prerequisites section, pointing to the existing PATH
note for draft creation.
* fix(repository-credentials): give oce-github pr-create a distinct message per failure class
Every failure printed 'Draft not confirmed. Check the pull requests at the
repository; do not submit another create.', including usage errors and
local refusals that never sent anything, so neither the Agent nor an
operator could tell what happened. Classify by what is known about GitHub's
state:
- usage errors (--help, unknown option, missing --draft) print the error and
usage and exit 2;
- local refusals (branch or base outside the binding, expired session,
oversized body, no connection) say 'Draft not sent';
- an OpenShell refusal made before forwarding (layer l7: request-phase
middleware denial or failure, L7 policy) and a GitHub 4xx (with GitHub's
message) say 'Draft not created';
- anything that may follow a create, including a response-phase middleware
block, keeps the 'not confirmed' line.
GitHub and OpenShell text is reduced to bounded printable text; the
credential placeholder is never printed.
* fix(openshell-git): refuse Agent network grants to the Git hosts for repository-bound Agents
With the OpenShell Git route, github.com and api.github.com are reachable
only through the oce-prepared-git middleware, which authorizes, audits and
denies each request. A Git-bound revision's own sandbox policy (approved
Agent network grants and its defaults snapshot) was merged into the
sandbox's network_policies unchecked, so an approved grant for github.com
(on any port, or a matching wildcard) added a direct route beside the
middleware (finding 2144).
SandboxDriver.admitNetworkPolicy takes an optional scope with
repositoryCredentials; the OpenShell Driver with its Git route refuses a
grant or default that reaches the Git hosts with a 409 naming it, using the
route's own overlap check (as at startup, 2131). OCC passes the scope when
the policy is saved for an Agent with repository bindings, when a binding
is saved beside already saved grants (the reverse order), at deploy, and at
live application to a Git-bound revision. Backstops: provisionHarness fails
such a revision permanently before any Sandbox exists, and
applyNetworkPolicy refuses before updating the policy.
The protected-execution 409 does not cover this: OpenShell Git requires an
Installation without the Credential Proxy, so its revisions are unprotected.
* fix(openshell-git): name the Git route policy refusal and fail its live application at once
Review follow-ups for finding 2144:
- A dedicated SandboxNetworkPolicyConflictError (a ResourceStateConflictError)
carries the Driver's named refusal; OCC passes only that class through, so
other Driver conflicts keep the generic message.
- The OpenShell Driver checks the Git route overlap before its other grant
checks, so the reverse-order check cannot be skipped by an unrelated
refusal it ignores.
- The policy worker fails an application the Driver refused this way as
SANDBOX_POLICY_UNSUPPORTED at once, instead of retrying until it reports
the outcome unconfirmed.
* fix(openshell-git): re-adopt the Git route when the same supervisor reconnects
Problem: a replaced OpenShell Gateway Pod gives every running Sandbox's
supervisor a new control session when it reconnects. The worker treated any
session change as a replaced supervisor and closed the Git route for good,
so one Gateway restart required redeploying every Git-bound Agent (2136).
Separately, a route that lost its material because the GitHub App key was
rejected at the provider was only visible as a bare 403 and a generic
route_pending denial (2135).
Change:
- observation(): a new session id is adopted when the rest of the verified
observation reads exactly as before (same Sandbox id, process and network
instance ids, provider generation, revisions, policy hash, credentials
installed and policy active). A changed process or network instance,
Sandbox id, or any other difference still tombstones as supervisor_replaced.
No material is issued or replayed. The restoration logs
worker.openshell-git-route {status: restored, reason: supervisor_reconnected,
durationMs}. The supervisor-asserted instance ids are a stated limit
(OpenShell 0.1.4 5a7728c3; an attested reconnect identity is a planned
OpenShell change), documented in the code and the operator guide.
- An open route that loses the material it had logs one
worker.openshell-git-route-warning {status: pending, reason:
material_unavailable} with the revision, and each refused request is denied
as material_unavailable instead of route_pending. A version that never had
material (deploying) is unchanged.
- Operator guide: the reconnect rule and its limit; a "GitHub App key rotated
or revoked" section (known gap: a new deploy still reports success) with the
log lines to alert on and the recovery steps.
Verification: tests/conformance/openshell-git-runtime.test.mjs gains a
reconnect test (fails on the previous head with blocked/supervisor_replaced)
and a lost-material test (fails on the previous head: no warning); the
tombstone test now covers a changed process id, network id, generation and
Sandbox id under a new session. tsc, eslint and prettier are clean.
* fix(openshell-git): adopt a reconnected session on identity, not a stale generation
Review of the reconnect change found two defects.
A credential rotation moves the provider generation and its verifying read
often lags the write by seconds, so comparing a reconnected session's full
observation against the previous read closed healthy routes for good on every
Gateway restart that overlapped a rotation. The session is now adopted when
the installation's identity reads as before (Sandbox id, name and workspace,
execution, provider id, profile version, attachment epoch, process and network
instance ids); its generation is what observe() already verifies against the
Gateway's receipt and the provider version this worker last wrote. Only the
worker's own reconcile pass adopts a session; a request's lookup verifies
against the reference and never moves it, so late answers from concurrent
reads cannot move it backwards. A reconnect before the route ever opened is
adopted without a "restored" line.
The material_unavailable warning fired at the moment of loss, which a version
superseded by a newer one also hits a moment before it is withdrawn. It now
rides on the first request refused for that loss (a Sandbox still asking),
once per loss, and so also covers material lost while the route was closed by
a failed read.
Tests: a rotation-window reconnect (fails with the previous comparison), a
silent pre-install reconnect, loss reported on the first denial and again
after a restoration, loss during an outage; durationMs asserted finite. Guide:
the adoption rule as implemented, the recovery-required line's real cadence
(about once a minute per binding), and the lag when the Secret still holds a
key GitHub no longer accepts.
* fix(openshell-git): report a reconnect as a restoration only for a route that had opened
A reconnect between the first credential write and the first verified install
(deploy-time lag) set the reconnected marker because the material was already
recorded, so the first install logged a restoration. Key it on the route having
been open at least once instead.
* fix(openshell-git): refuse IP-address hosts beside the Git route
An Agent network grant or sandbox network policy naming an IP address
(e.g. 140.82.112.3) passed the OpenShell Git overlap check, which only
matched github.com, api.github.com and their wildcards, so it could open a
route to GitHub beside the oce-prepared-git middleware (finding 2145).
With the Git route on, the overlap check now treats any IP-address host as
overlapping: IPv4 (any host whose last label is numeric, including hex and
single-number forms and wildcards over one), IPv6 in any form, and CIDR
ranges. It reports which kind it found, so the Installation startup check
(2131) and the repository-bound grant checks (2144: save, binding, deploy,
live apply, provisioning) refuse with a message naming the rule and host
and asking for DNS hosts. Non-Git installations and Agents without a
repository binding are unchanged. The repository-bound refusal now names
the host for default policies as well.
* fix(openshell-git): review follow-ups for the IP-address refusal
- Detect IPv6 with node:net isIP (bracketed or with a range) instead of
any colon, so a host:port such as github.com:443 keeps the generic
overlap message; an IPv4 address with a port still counts as an IP.
- Shorten the IP refusal so the rule and host fit the 256-character
HTTP message cap.
- Say why provider endpoints keep the name check only.
- Docs: private and loopback addresses are included, use DNS names, and
the check compares names only (a DNS name resolving to GitHub is not
detected); clarify the guide wording.
- Tests: an IP-address grant is also refused when a binding is added
beside it and at live application; host:port and loopback cases.
* fix(openshell-git): hold activation until the first token install and the node's reconnection
Problem: the worker issues a version's first prepared Git token only once the
version is active (finding 2139). OpenShell 0.1.4 (5a7728c3) folds every
attached provider's record version into the Sandbox's provider environment
revision (openshell-server grpc/policy.rs) and answers a change with a policy
runtime reload that advances the generation and closes every connection out of
the Sandbox (supervisor-network opa.rs, l7/relay.rs), the workspace node's
WebSocket to the Gateway included. The deployment had already reported success,
so chat answered 500 for 1-7 s right after (finding 2141). No OCE-side change
keeps that reset from happening: stable_placeholder keeps only the environment
placeholder stable, middleware cannot write credential headers, token grants
have a different lifecycle owner, and installation tokens expire hourly.
Change: the Git route records when it verified its current material installed
(materialInstalledAt). Dedicated activation waits as
REPOSITORY_GIT_MATERIAL_PENDING until that first install is verified, as
WORKSPACE_NODE_RECONNECT_PENDING for a 2 s settle margin after it (a node read
inside the margin can predate the Gateway seeing the drop), then reads the node
(the existing WORKSPACE_NODE_PENDING) and requires the same install to still be
verified once that read returns (a route that closed meanwhile waits as
REPOSITORY_GIT_MATERIAL_PENDING again; a newer install as the reconnect wait).
Both codes are activation waits (D330) on the readiness recheck cadence,
bounded by the deployment's convergence deadline, with fixed status copy that
points at the worker log. The operator guide states the limit: every later
renewal (about every 50 minutes) repeats the reset until OpenShell scopes it;
that is being taken upstream.
Behaviour change: a version whose material is never issued (the App key was
rejected, finding 2135) now waits as REPOSITORY_GIT_MATERIAL_PENDING until the
convergence deadline instead of succeeding with every Git request denied; the
guide's "GitHub App key rotated or revoked" section says so.
Verification: new conformance tests (route install time with the wall clock
moving between reads; activation sequence with a provisioning Sandbox driver and
a fake route, including a renewal landing and the route closing during the node
read; a version without repository credentials never consults the route) fail
on 56967fa20 and pass here; tsc, eslint and prettier clean on the touched files.
Risks: a maintenance pass for an active version that lands within 2 s of a
renewal ends pending (for a maintenance claim that is a failed work row and a
skipped predecessor retirement for that pass; the next maintenance pass, one
interval later, reads it again). The margin is a constant, not a measurement of
the Gateway's close detection. docs/reference/agents/deployment.md (2487 of
2500 words) does not list the two new codes.
* fix(openshell-git): keep a running route open across a sibling Agent's deploy
Finding 2146: deploying another Agent in the same namespace denied one Git
request of a running Agent with installation_changed. The running Sandbox's
supervisor log shows why: at that moment its settings poll reports
provider_env_changed (policy and config revision unchanged). The provider
environment revision is Sandbox-wide, so it moves when another provider
attached to the Sandbox changes. Until the supervisor's next poll installs
it, the Gateway reports this provider as pending (snapshot mismatch), and
once installed the observation no longer deep-equals the admitted one.
- observe() takes the installation it last verified. When only the provider
environment revision moved ahead of the supervisor (state pending, reason
snapshot mismatch, same attachment, config, policy, session, process and
network instance, this provider's id, resource version and profile version
pinned to the binding), it answers that installation unchanged.
- A request's lookup compares the installation without the provider
environment revision; the reconcile adopts the new one as before.
Anything else that moves still closes the route until the reconcile
verifies it again.
* fix(openshell-git): refuse every github.com name beside the Git route
Finding 2148 (part 2): the overlap check refused only github.com and
api.github.com, so a grant, default or Driver policy for another GitHub front
end (ssh.github.com:443, codeload.github.com, uploads.github.com) was admitted
beside the oce-prepared-git middleware and its requests skipped the
middleware's authorization, audit and denial.
The check now covers github.com and every name under it, plus a wildcard over
github.com's parent (*.com), at Installation startup, grant save, binding
save, deploy, provisioning and live application. githubusercontent.com hosts
stay allowed: the route sends nothing there. Messages say "github.com and its
subdomains".
Part 1 (a DNS name that resolves to a GitHub address) stays a documented
limit; the guide now says so explicitly.
* fix(openshell-git): say when a binding already opened its draft
Finding 2147: the one-draft limit is per grant (repository, review branch
and base), shared by every version and Agent bound to it and never reset.
Once spent, every later create on the OpenShell route got the generic
"the repository route refused it (online_authorization_denied)" and the
worker logged approval_denied, so neither the Agent nor the operator could
tell why.
- The store answers draft_already_opened when the grant's draft row exists
(after the live session and grant checks, nothing written).
- The online authority can deny with a stable reason code; the Git authority
denies with draft_already_opened and logs that reason.
- oce-github pr-create says the binding already opened its draft pull
request, that GitHub did not receive this one, and to ask for a new review
branch for another draft.
- The OpenShell Git guide states the per-grant scope, that closing the draft
does not reset it, and how to get a new one.
Not changed: on this route OCE never sees GitHub's reply (request-phase
middleware only), so the row cannot be settled or the PR number recorded
without a response-phase stage. Resetting the limit on PR close is an
owner decision.
* fix(openshell-git): bound the provider-environment lag a route tolerates
Review follow-ups for the sibling-deploy fix (finding 2146):
- The verified installation is taken as current while only the provider
environment waits for the supervisor for at most 60 s. A supervisor that
never installs it closes the route (pending) and logs
worker.openshell-git-route-warning reason=environment_not_installed once.
- The tolerance comment names what OpenShell 0.1.4 hashes into the provider
environment revision and why each part is pinned for this provider.
- Tests assert the runtime hands observe() the admitted installation on a
lookup and never on a credential rotation's verifying read, and cover the
lag limit.
* fix(openshell-git): carry the spent-draft refusal through the custody transaction
Follow-up for finding 2147: the custody transaction replaces every error
with a redacted one, so authorizeOpenShellRequest never saw the
spent-draft refusal and still answered false (caught by the PostgreSQL
journey and use tests). The refusal is now noted inside the transaction.
An aborted request whose draft was spent logs aborted, not approval_denied.
* docs(openshell-git): name the GitHub hosts the Git route now refuses
Follow-up for finding 2148: gist.github.com, docs.github.com and GitHub
Packages registries (*.pkg.github.com) are refused beside the Git route
too; ghcr.io and githubusercontent.com hosts are not.
* feat(openshell-git): let an operator opt a draft grant out of the one-draft limit
Owner decision 10-20 on finding 2147: offer a way to trust the Agent
instead of enforcing one draft per grant, documented as a known limit
with an opt-out.
- Registry namespace policy accepts draftPullLimit: "one-per-grant"
(the default, not stored, same grant) or "none", only beside
transport git-and-draft-pr. "none" is part of the grant fingerprint, so
turning it on is a new grant (redeploy).
- On the OpenShell route an opted-out grant admits every draft create its
live session, grant and material authorize; the draft-only, head and
base checks are unchanged. The grant's row keeps the latest create's
receipt (no migration: the table stays keyed by grant).
- The native broker route keeps its one replayable receipt per grant.
- The OpenShell Git guide documents the limit and the opt-out.
* docs(repository-credentials): state that the Agent may force-push or delete its review branch
Finding 2132, owner decision 10-20 ("allow both and document"): the push
authority admits a force update and a delete of the Agent's own review
ref. Deleting the branch closes its draft pull request. The OpenShell Git
and prepared native Git guides now say so and point to GitHub branch
protection or rulesets as the operator's way to stop it.
* fix(openshell-git): compare the lag tolerance against the installation the read began with
Re-review follow-ups for finding 2146:
- observation() snapshots the admitted installation before the read and
compares observe()'s answer with that snapshot. A concurrent close (a
lookup past the limit) no longer makes a reconcile's tolerated read look
fresh, which reset the 60 s window and reopened the route.
- Closing a route at the limit sets closedAt, so its reopening is reported
as a restoration with its duration.
- Tests: two concurrent lookups past the limit warn once; the race above
(fails with the unsnapshotted comparison); the restoration event.
* fix(openshell-git): keep the first draft receipt when a grant opts out of the limit
Re-review follow-up for the draftPullLimit opt-out (finding 2147): the
upsert that replaced the grant's receipt needs UPDATE on columns the
application role is not granted (migration 0067 grants UPDATE on outcome,
projection and settled_at only), so every opted-out create was refused
(PostgreSQL test failed on the remote runner). It could also have replaced
a broker receipt sharing the grant.
An opted-out create now inserts the receipt only if the grant has none and
is allowed either way. The grant keeps the first create's receipt;
OpenShell's log records each later request. No migration, no new grant.
The guide says so and that the grant is shared by every Agent in the
namespace bound to that repository.
* docs: say which log records opted-out draft creates (2147 re-review)
The opt-out paragraph now names the supervisor's HTTP log as the record of
later creates and scopes the shared limit to Agents bound to the grant. The
store comment says the primary key keeps one receipt rather than one create.
* docs(openshell-git): tighten the host, draft-limit and review-branch notes
After the activation-hold change the guide was 2565 words, over the 2500-word
hard limit. Shorten the paragraphs this lane added without dropping a fact.
* fix(credential-proxy): lease a permitted model stream past the operation deadline
The whole model operation had to finish within the 30-second operation
deadline: the permitted record was discarded at the deadline, so a reasoning
turn that streamed for minutes found no record when its complete response
reached the authority, the exchange failed and the client re-sent the turn.
The operation deadline now bounds access, send start and the first response
byte only. The supervisor reports progress on the installation exchange
(InstallationExchangeProgress, no bytes); each frame with new body bytes
renews the permitted operation for five minutes, never past fifteen minutes
from the request's UUIDv7 creation or the peer's expiry
(InstallationExchangeLease). Current policy is checked again at every
renewal, so a withdrawal cuts the stream there and records a blocked
response. No progress before the deadline, an idle stream or the ceiling
discard the record as before, and a supervisor that never sends progress
keeps today's behaviour.
* fix(openshell-git): let a restarted worker re-adopt running Git routes
Problem: the OpenShell Git route runtime was worker-process-local. After
any worker restart (crash, rollout, node drain) every running Agent's Git
route was gone: each request answered 403 `no_binding` until the Agent was
redeployed, and prepare() refused the existing Sandbox with "requires a new
revision after worker restart" (finding 2128).
Change: persist the route's non-secret identity and custody facts, and let
the next worker re-adopt the route only when the full observation verifies
the way the supervisor-reconnect adoption (finding 2136) does.
- What is persisted: migrations/0069_openshell_git_routes.sql creates
occ.openshell_git_routes keyed by revision id with workspace, sandbox
name, binding id, the provider generation this worker wrote (provider id,
profile version, resource version, credential state, expiry), the
material version reference (material_version + preparation_attempt_id),
the last verified observation (sandbox id, execution, attachment epoch,
process and network instance ids, revisions, policy hash: the 16
identity keys the CHECK pins), the fence, the writing worker instance and
a tombstone reason. No token material: the CHECK on `observed` rejects
any other key, occ_app cannot DELETE or rewrite revision_id/binding_id,
and tests/integration/postgres-openshell-git-routes.test.mjs asserts the
column list. packages/occ/src/state/postgres-schema.ts mirrors the table;
canonical-history, the journal, scripts/migration-history.mjs and the
migration compatibility test carry the new preOpenShellGitRoutes shape.
- Re-adoption: prepare() asks the store to adopt the row first
(openshell-git-runtime.ts:459); a row restores the provider generation
without IO (openshell-prepared-git-provider.ts:355 restore(); an expired
or bootstrap generation restores as bootstrap so the route reopens only
after the normal rotation) and the entry starts `unverified`
(openshell-git-runtime.ts:655 adopt()). The first observation compares
the full identity key set regardless of session id (:1114); a mismatch
tombstones `sandbox_mismatch` (:1052), a persisted tombstone, a
mid-write row (`write_unknown`, :696) or a conflicting provider refuses
with its reason, and a store that does not answer is retried, not
retained. When the persisted material reference equals the material the
owner reads back and the binding is active, no provider write happens;
a newer material rotates through the normal preparation path. Material
is only ever re-issued, never replayed. The restoration logs
`worker.openshell-git-route` with `status: adopted` then
`reason: worker_restarted` (:764).
- Fencing: adopt() bumps `fence` under CAS
(openshell-git-route-store.ts:192); every holder write carries it
(writing() :216 marks write_pending before a provider write, record()
:233 clears it, block() :284), and the runtime probes ownership at the
top of each reconcile pass and on every lookup (fenced(), :804; owns() is one primary-key read per request). A stale
fence makes the old worker drop its copy of the route (`worker_replaced`,
:1228: no tombstone, nothing written, the next prepare() by the claim
holder re-adopts). A store that does not answer a request's fence check
denies that request only (`store_unavailable`, :1501). An adoption the
gateway refuses (`gateway_unauthorized`) is never retried; a refused
bootstrap records `bootstrap_failed` (:514) instead of a pending write.
- Sandboxes prepared before this change have no row: prepare() keeps the
"requires a new revision after worker restart" refusal, logged
`worker_restarted` (:482). The worker wires the PostgreSQL store in
apps/controller/src/worker.mjs:182.
Verification: tests/conformance/openshell-git-runtime.test.mjs adds the
restart re-adoption, the changed process/network id tombstone and the
two-worker fence tests (all fail on the previous head);
tests/integration/postgres-openshell-git-routes.test.mjs (postgres-platform
suite) exercises the store against PostgreSQL and asserts no secret
column; sandbox-driver-startup asserts the bootstrap row and the recorded
material reference. Remote lanes checks-baseline-1 (static),
checks-baseline-2, postgres, postgres-platform and postgres-application
pass on this head; the conformance file pins 20/20 at 2 CPUs; tsc clean.
Risks: a denial window remains between the worker restart and the first
verified read (requests answer 403 until the maintenance pass adopts the
route); the operator guide's restart section documents it.
* Fix credential packaging and smoke checks, export Git events, and shorten the egress guide
* fix(openshell-git): name each refusal for the Agent and retire earlier-issuer tokens
Finding 2149: every refusal on the OpenShell Git route reached the Agent as the
same 403 online_authorization_denied, so a policy refusal, withdrawn access, a
rejected App key and a route still starting read alike.
- The authority now returns a short reason code in OpenShell's 403 body:
request_not_allowed, draft_already_opened, access_withdrawn,
credential_unavailable, route_blocked or route_not_ready. A request the worker
cannot attribute to one of its routes stays online_authorization_denied.
- The worker's lookup names the code for its own routes; the material store says
whether material is absent because the session ended or because none is usable,
and approval answers access_withdrawn when the session is no longer live.
- worker.openshell-git-denied (and the lost-material warning) log the same
reasonCode beside the detailed reason.
- oce-github pr-create prints advice per code. A draft body that is not UTF-8 or
has ambiguous JSON members is now refused and logged as draft_rejected instead
of failing the RPC silently.
- New reference page: what Git shows (403 only, "Everything up-to-date" on a
refused push), how to read the code with GIT_TRACE_CURL, and the code table.
Finding 2150: discovery retired only families of the current issuer authority,
so after the registry moved to another App origin or CA the previous authority's
family for a replaced version stayed enabled with a current token. Discovery now
retires every non-live family. A token from an earlier authority is never sent
to the new one for revocation; the worker logs PREPARED_GITHUB_RETIRED_UNREVOKED
once with its expiry instead, and a worker with the earlier setting still
revokes it.
* fix(openshell-git): review fixes for the refusal codes and issuer retirement
- Docs: Git's curl trace splits the body into 60-byte lines, so the documented
command joins them before looking for reason_code (checked against git 2.56).
Describe what Git shows by stage: a refused first request is a bare 403, only
a refused push upload shows RPC failed / Everything up-to-date.
access_withdrawn also covers an expired session; credential_unavailable also
covers a failed renewal.
- Helper: credential_unavailable no longer says retrying will not help (a
failed renewal is transient); online_authorization_denied gets advice too.
- Issuer change: say that revocation is left to the issuer that prepared the
token, which an overlapping earlier worker still does.
- Runtime: keep read()'s doc comment on read(); clarify which requests stay
generic.
- Test fixes: a second owner needs its own socket directory, and the provider
probe counts as a provider request, so it runs after the no-send checks.
* docs(openshell-git): move the restart and recovery detail to its own page
The operator guide passed the 2500-word cap after the worker-restart section
grew (docs:check red on the branch head). The guide keeps a short summary of
what a failed read, a Gateway restart and a worker restart do to a running
route; the exact adoption rules, the multi-worker fence, the stated OpenShell
0.1.4 limit and the worker log vocabulary move to
docs/reference/repository-credentials/openshell-git-restarts.md (registered in
docs/docs.json). The migration-history shape list in operations.md is shortened
to fit as well. Nothing changes in meaning.
Verification: pnpm docs:check passes (openshell-git.md 2278 words,
operations.md 2499).
* Check current policy after recording a leased model response
* Clear the active revision after repository cleanup stops its runtime
* fix(occ): make protected execution refuse Agent network policy additions by name
Finding 2053. A protected execution keeps only its Credential Proxy egress
policy, but the rule was implicit: revision admission still stored a Driver
defaults sandboxPolicy snapshot beside the egress plan, save accepted additions
until the first protected revision was active, the refusals were generic
RESOURCE_CONFLICT, and the OpenShell Driver dropped any additions a protected
revision carried.
The rule is now explicit, with one named error,
ProtectedExecutionNetworkPolicyError (409 SANDBOX_POLICY_PROTECTED_EXECUTION):
- save refuses additions while the active revision is protected or the next
deploy will be (the Installation selects the Credential Proxy); saving an
empty policy stays allowed. The view's protectedExecution covers both.
- live application to a protected revision is refused before the generic
"live application unavailable" check.
- deploy refuses saved additions (after SandboxPolicy read) before Driver
admission, and a protected revision admits no sandboxPolicy snapshot.
- the policy worker fails queued application with
SANDBOX_POLICY_PROTECTED_EXECUTION, and OpenShell provisionHarness refuses a
protected revision carrying additions instead of dropping them.
Unprotected Agents keep main's behaviour.
The console states the rule and offers "Remove saved network rules" when a
protected Agent still has saved rules, since those block every protected
deploy. The operator guide (protected inference cutoff), topic guide and flow
doc say so; the OpenAPI contract and API reference are regenerated.
Tests: static-egress covers deploy (named code, no snapshot), save before the
first protected deploy, save/apply on a protected active revision, the worker
backstop and the Driver backstop; http-errors maps the code; a browser test
covers the removal action. The five static-egress cases and http-errors fail
on 81d03d8d1.
* fix(occ): review fixes for the protected execution policy refusal
- One helper, protectedExecutionApplies, decides the save refusal and the
view's protectedExecution; save skips the revision read when the
Credential Proxy is selected.
- Docs, the error code description and the view field say exactly what is
refused: save and deploy while the running or next revision is protected,
live application only to a protected revision.
- Console: a hint when saved rules block protected deploys but editing is
disabled, and the Driver-defaults caption says protected versions do not
use them.
- Storybook: a protected-execution policy story with a saved rule, the
named 409 on save, and the removal action.
- Backstop test comments say the states are unreachable through admission.
* test(egress): brace the leased-stream test conditionals for the curly lint rule
* test(egress): restructure fixture checks flagged by code scanning
Four test-only code-scanning alerts on this branch, none exploitable,
each removed by making the code plainly safe rather than suppressing it:
- egress-custody-repository-origin: the token-exchange route test
dominated the JWT signature check (js/user-controlled-bypass). The
JWT and body checks are now one helper; the route answers 403 when it
returns false. Same checks, same order.
- skipper-private-git-origin: `!url.search` sat in the pull-create route
condition ahead of the token check (js/user-controlled-bypass). A
query string now gets the same 404 early, with the same `other:404`
count the old fall-through produced.
- ssh-compute: recreate the namespace marker with the test's own
write-then-rename helper after asserting it is absent, instead of a
second write to a path stat() checked (js/file-system-race).
- repository-credentials-projected-inputs: assert the symlink target
with readlink() instead of lstat()…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#2957 (finding 2040) made a revision pass wait out a Sandbox gateway outage instead of spending its attempt budget. It recognised an outage as any gRPC
UNAVAILABLE(openshell-gateway-client.ts,gatewayRequestFailure). ButUNAVAILABLEsays two different things:UNAVAILABLE, with a reason, for failures that do not clear by themselves. At the pinned OpenShell (5a7728c3): resource admission (on by default) maps a Kubernetes 403 ("check gateway RBAC"), 401, TLS and auth errors to it; the Kubernetes-secrets driver maps most Secret errors to it; Vault auth HTTP errors and default credential failures do too.grpc-js also returns
UNAVAILABLEfor a gateway certificate the client's CA does not trust. Since #2957 all of these wait the full convergence deadline (15 min by default) with "did not answer" status text, instead of failing within the attempt budget as before. #2957's wire test even pinned a server-sentUNAVAILABLEas an outage. Finding 2042.Change
gatewayDidNotAnswerclassifies anUNAVAILABLEas an outage only when its details are one of grpc-js's own texts for a gateway that did not answer. This is a positive allowlist:Connection dropped,read|write ECONNRESET|ETIMEDOUT: the connection closed or was reset under an in-flight call.Stream refused by server: the gateway refused the stream before handling it.Name resolution failed for target …,No connection established. Last error: No addresses resolved: there is no address for the gateway.No connection established. Last error: [Error: ]connect ECONNREFUSED|ECONNRESET|EHOSTUNREACH|ENETUNREACH|ETIMEDOUT …orread|write ECONNRESET|ETIMEDOUT|EPIPE …: nothing answers at its address.OpenShellGatewayRequestFailureagain, as before fix(controller): wait out a sandbox gateway outage without spending the attempt budget #2957: a server-sentUNAVAILABLE, and a TLS failure (an untrusted certificate, a wrong name or IP, a refused client certificate). It retries within the attempt budget asDEPENDENCY_UNAVAILABLE, andCreateSandboxstill names the gateway's (sanitised) reason in its message.ExecSandboxclassifies from the original grpc-js error. Its message still carries only the code.UNAVAILABLE". The flow doc gets shorter (it is near its word cap).The worker side is unchanged: it still waits only on
SandboxGatewayUnavailableError. Out of scope: the paths #2957 left on the budget (finding 2041), and runtime entrypoints.Verification
tests/conformance/openshell-gateway-wire.test.mjs, real local gRPC servers, no wall-clock windows:Health,CreateSandbox,ExecSandbox); a gateway that drops the connection under a call it received (held until the test sees the request, then its socket is destroyed); a gateway that refuses every stream.UNAVAILABLE: the pinned gateway's resource-admission 403 text onHealth,CreateSandboxandExecSandbox. It is not an outage and keeps code 14.CreateSandbox's message is exactlyOpenShell CreateSandbox failed with gRPC status 14: <reason>, and the other methods don't carry the text.untrusted-dnscase (thecreateSslpath, on a dual-stack listener so every addresslocalhostresolves to answers with the certificate; skipped without IPv6 loopback, like the other IPv6 cases), plus the existingwrong-ip,name-for-ipanduntrusted-beside-trustedcases, now also assert "not an outage".Connection dropped,ECONNREFUSEDorStream refusedentries fails the test, and so does classifying everyUNAVAILABLEas an outage. The defensiveread ECONNRESETentry has no deterministic local trigger.tsc --build, eslint and prettier are clean on the touched files.ClusterIP, reached by Service DNS; at zero replicas kube-proxy rejects the connection (connect ECONNREFUSED), and a headless name would fail resolution. Both are on the allowlist, so the proof should still seeSANDBOX_GATEWAY_UNAVAILABLEwithattempt_count0.untrusted-dns, a comment naming the excluded shapes, and this k3d note.Risks