From 9743f8b2b4ac322b12fe5043a36f55852786eef7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 12:39:09 +0000 Subject: [PATCH] guardrails: containment as a required companion; verify-the-agent rule; retire a removed subagent cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the five suggestions confirmed from PR #20's "Routine self-improvements" section (recommendations 1B, 2A, 3A, 4B, 5B). All are human-authored changes outside the routine's auto-apply envelope, which is why they could not ride the knowledge PR. Containment (S1, option B) — guardrails/README.md, guardrails/checklist.md The three hard stops bound what a loop spends and how long it runs. Neither bounds what an attacker-supplied input makes the harness do, and the known mechanisms fire before any stop condition or verification step runs: GitSpawn executes on the agent's startup git status, before the workspace-trust prompt; instruction privilege escalation lands before automatic permission review sees the content. Added as a required companion section rather than a fourth numbered hard stop, deliberately: the three are single checkable numbers, containment is a posture. Defaults cover scoped network egress, unreachable credentials, deny-by-default unattended runs, and real isolation. Deny-by-default for headless loops (S5, option B) — guardrails/README.md Documents --permission-prompts none (v2.1.259+) as the cheapest containment win available, including that it removes tools needing a human answer and surfaces denials in stream-json output. Also records that --restricted is a permission gate, not a sandbox: the primary docs say nothing about OS-level isolation and it is absent from Anthropic's own sandbox-environments comparison page. Routine threat model (S2, option A) — runbooks/staying-current.md Writes down what update-knowledge can reach and why it is still safe to run. The research it does now describes an attack on the shape it is: instruction privilege escalation reproduces through /goal- and /schedule-shaped features, and neither paper proposes a mitigation. The defenses are structural — never merges, blast radius limited to knowledge/, CI-enforced fence, claude/-only pushes — so they do not depend on the agent noticing it was manipulated. Also lists what would invalidate that reasoning. Verify the agent, don't relay it (S3, option A) — update-knowledge SKILL.md Step 4 now requires machine-verifying every arXiv citation against the arXiv API, and checking version attributions against the feature's own doc page rather than the nearest changelog heading. Step 4 is a protected region, so this is a normal human-authored commit, not a self-edit: one — the route the guard's own error message names. Consistency and drift fixes loop-guardrails gains containment as check 6, since a required section that nothing audits is not required. And guardrails/README.md, staying-current.md (three places) and artifact-audit SKILL.md all still recommended CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION, which was removed in v2.1.224 and now does nothing — replaced with the concurrency and depth caps that remain, and noting neither bounds a session's total lifetime spawns. S4 (recording the whats-new digest as unreliable in the skill) was declined: sources.md already carries it and is read every pass at step 1. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ca7z6e9N5SAhNjqM8gFM5R --- .claude/skills/artifact-audit/SKILL.md | 8 ++- .claude/skills/loop-guardrails/SKILL.md | 23 +++++-- .claude/skills/update-knowledge/SKILL.md | 19 ++++++ guardrails/README.md | 87 ++++++++++++++++++++++-- guardrails/checklist.md | 13 ++++ runbooks/staying-current.md | 63 +++++++++++++++-- 6 files changed, 196 insertions(+), 17 deletions(-) diff --git a/.claude/skills/artifact-audit/SKILL.md b/.claude/skills/artifact-audit/SKILL.md index 0ac781a..27b500e 100644 --- a/.claude/skills/artifact-audit/SKILL.md +++ b/.claude/skills/artifact-audit/SKILL.md @@ -89,8 +89,12 @@ drifted: make no commit and report "no artifact drift since ." A scheduled **one-shot** (one trigger → one audit pass → one PR → exit), not a `while`-loop. Same envelope as `update-knowledge`: -- **Iteration** — one trigger, one pass, no self-re-invocation; the native - `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION` cap bounds the ~4-agent fan-out. +- **Iteration** — one trigger, one pass, no self-re-invocation; bound the + ~4-agent fan-out with `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS` (default 20, + concurrency) and `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH` (default 3, depth). + The old per-session `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION` total cap was + **removed in v2.1.224 (Aug 7 2026)** and can no longer be relied on — there is + no native cap on a session's *total* lifetime spawns. - **Budget** — the subscription / daily-run cap is the ceiling, real only with metered overage off (see `runbooks/staying-current.md` → "Budget & caps"). - **Stall** — N/A for a one-shot; the no-op exit is the termination. diff --git a/.claude/skills/loop-guardrails/SKILL.md b/.claude/skills/loop-guardrails/SKILL.md index 8aef665..41e0098 100644 --- a/.claude/skills/loop-guardrails/SKILL.md +++ b/.claude/skills/loop-guardrails/SKILL.md @@ -42,11 +42,26 @@ For the given loop (file path, command, or config), verify all of: if the task needs human judgment / design decisions / one-shot ops / prod debugging — those are anti-patterns for autonomous loops. +6. **Containment.** The four above bound what the loop *spends*, *how long it + runs*, and whether it *checks its own work*. None of them bounds what an + attacker-supplied input can make the harness do — and the known mechanisms + fire *before* any of them run. Check the containment block in + `guardrails/checklist.md`: scoped network egress, no long-lived credentials + reachable, `--permission-prompts none` for unattended headless runs, and real + isolation (container/VM — `--restricted` is a permission gate, not a + sandbox). Weight this heavily if the loop **clones, reads, or reviews + untrusted repositories**, which is the premise both GitSpawn and instruction + privilege escalation need. Rationale and sources: `guardrails/README.md` → + "Containment", `knowledge/00-primer.md` §5A. + Unlike 1–3 this is a posture, not a single number, so report it as + ✅ / ⚠️ partial / ❌ rather than pass-fail, and say which specific control is + missing. + ## Output -Report a checklist: ✅ present / ❌ missing for each of the five, with the exact -line or flag that satisfies it. For each ❌, apply the fix (edit the file or -rewrite the command) and show the diff. End with the corrected, runnable loop -and a one-line cost expectation. +Report a checklist: ✅ present / ❌ missing for each of the six (containment may +also be ⚠️ partial), with the exact line or flag that satisfies it. For each ❌, +apply the fix (edit the file or rewrite the command) and show the diff. End with +the corrected, runnable loop and a one-line cost expectation. Pull default thresholds from `guardrails/budget.env`. diff --git a/.claude/skills/update-knowledge/SKILL.md b/.claude/skills/update-knowledge/SKILL.md index 89400c0..ee8c795 100644 --- a/.claude/skills/update-knowledge/SKILL.md +++ b/.claude/skills/update-knowledge/SKILL.md @@ -83,6 +83,25 @@ Treat search snippets skeptically. Only promote a claim to **High** if a primary source was read directly or it appears verbatim across independent sources. Anything you cannot verify against a primary source: mark **Low** and add it to the "things to re-verify" section of `sources.md` rather than the primer. + +**Verify a research agent's report; don't relay it.** A subagent's summary is not +a primary source — it is a claim *about* one. Before promoting anything it +returned, re-check it yourself. Two classes are mandatory: + +- **Every arXiv citation** must be machine-verified against the arXiv API + (`https://export.arxiv.org/api/query?id_list=`) for its **ID, exact title, + v1 `published` date, and primary category**. Report only what the API returns. + Past passes have carried a misdated ID and a hallucinated one; this check is + cheap and catches both. Attribute a paper's findings to its authors, never + assert them as fact. +- **Every version number and attribution** must be checked against the doc page + for the feature itself, not the nearest changelog heading. A version with no + changelog entry still exists (silent/staged releases), so absence of a heading + proves nothing. + +Beyond those, re-read the primary for any claim you are about to make load-bearing +— a correction to this knowledge base, a headline finding, or anything that +changes guidance — and quote it verbatim rather than paraphrasing a summary. ### 5. Diff and draft diff --git a/guardrails/README.md b/guardrails/README.md index c6e8891..5b069fe 100644 --- a/guardrails/README.md +++ b/guardrails/README.md @@ -20,6 +20,69 @@ the ceiling has to live in your loop harness. Plus, always: **verification inside the loop** — one deterministic check (test/lint/typecheck) the agent runs itself, declaring done only on pass. +## Containment — the required companion (not a fourth hard stop) + +The three hard stops bound what a loop **spends** and **how long it runs**. None +of them bounds **what an attacker-supplied input can make the harness do** — and +in 2026 that gap stopped being theoretical. Containment is therefore a +**required section of every loop's setup**, deliberately *not* numbered as a +fourth hard stop: the three are single, checkable numbers, while containment is +a posture made of several settings. Keeping the list at three preserves its +force; skipping this section is still incomplete. + +**Why it's required.** Two mechanisms, both documented in 2026, land *before* +any of the three hard stops or the verification step can run: + +- **GitSpawn** (Sep 2026) — a malicious repo's `.git/config` sets + `core.fsmonitor`, which git executes during the agent's startup `git status`: + before the workspace-trust prompt, outside the command sandbox, with no + approval prompt. Triggered by *cloning the target repo*. +- **Instruction privilege escalation** (arXiv:2608.27299, replicated + independently across 12 harnesses by arXiv:2609.01222) — a harness + reconstructing context drops provenance and re-labels attacker text that + arrived as a *file the agent read* into a genuine `user` message. It + reproduces through `/goal`- and `/schedule`-shaped features, and automatic + permission review does not stop it. + +This repo's model is "loops run *from here* against other repositories," i.e. +**we clone untrusted code by design.** That is exactly the premise both attacks +need. See `knowledge/00-primer.md` §5A. + +**The standard to hold** (arXiv:2609.00267): *"a correct system is one in which a +fully prompt-injected agent still cannot exceed the authority explicitly +delegated to it."* + +### Containment defaults + +- [ ] **Network egress is scoped.** Allowlist the domains the loop actually + needs rather than granting full access. In Claude Code, + `sandbox.network.strictAllowlist` denies non-allowlisted hosts without + prompting. Note that a proxy is not sufficient on its own — agents have + been observed editing `/etc/hosts` to masquerade blocked domains as + allowed ones. +- [ ] **No long-lived credentials in the environment.** Assume anything in env + vars is readable by whatever the loop runs. Isolate SSH keys, cloud + credentials and the home directory; prefer short-lived tokens. +- [ ] **Unattended runs deny by default.** Use `--permission-prompts none` + (Claude Code v2.1.259+) on headless hosts: *"anything that would prompt is + denied automatically while the active permission mode (including auto + mode) keeps deciding."* It also removes the tools that need a human answer + (e.g. `AskUserQuestion`), and under `--output-format stream-json` denials + surface as `permission_denied` messages with a `permission_denials` list + on the final result — so a harness can *see* what it blocked. This is the + single cheapest containment win available today. +- [ ] **The blast radius is a container/VM, not a flag.** `--restricted` + (v2.1.248+) removes command-running tools and confines file tools to the + working directory — but the primary docs describe it only as tool removal + and settings scoping, say **nothing** about OS-level isolation, and it is + **absent from Anthropic's own `sandbox-environments` comparison page**. + Treat it as a *permission gate, not a sandbox*, and don't rely on it to + protect secrets. For real isolation use a container, VM, or Claude Code on + the web's ephemeral VMs. +- [ ] **Review of untrusted code is a privileged operation.** "Have an agent + review it" is not a free safety check when the reviewer is the attack + surface (cf. the "Friendly Fire" disclosure, primer §5A). + ## Enforce them tool-agnostically (not just in Claude Code) The three hard stops are a *discipline*, not a Claude Code feature — the same @@ -37,13 +100,23 @@ Claude Code"). Two enforcement locations, both agent-independent: (`BudgetGuard`/`LoopGuard`/`TimeoutGuard`) and **LoopGain** (convergence early stop + rollback; adapters for LangGraph/CrewAI/AutoGen/Claude Agent SDK) ship the three stops as a kill-switch you drop into any loop. -- **Native in Claude Code** (v2.1.212, Jul 17 2026): a session-wide WebSearch - cap (default 200, `CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION`), a per-session - subagent-spawn cap (default 200, `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION`, - reset by `/clear`), and MCP calls running >2 min auto-moved to background. - Product-level backstops against runaway loops — they complement the three - hard stops, they don't replace them (none is a dollar ceiling, and the - defaults are far above a sane per-loop cap — set them lower explicitly). +- **Native in Claude Code**: a session-wide WebSearch cap (default 200, + `CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION`, v2.1.212), MCP calls running + >2 min auto-moved to background, a **concurrent**-subagent cap (default 20, + `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS`) and a subagent **depth** cap (default + 3, `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH`). Product-level backstops — they + complement the three hard stops, they don't replace them (none is a dollar + ceiling, and the defaults sit far above a sane per-loop cap — set them lower + explicitly). + **Corrected 2026-09-07 — do not rely on `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION`.** + This file previously recommended it as a per-session spawn cap with a default + of 200. That cap was **removed in v2.1.224 (Aug 7, 2026)** and the variable is + now gone from the docs entirely; the `sub-agents` page states *"There's no + limit on the total number of subagents Claude can spawn over a session."* The + concurrency and depth caps above are the only native subagent backstops left, + and **neither bounds a session's total lifetime spawns** — so that ceiling is + yours to enforce. A native backstop being deleted is exactly why the three + hard stops live in your harness rather than in a vendor default. Confidence on specific gateway/library flags is **Medium** — verify against live docs before relying on one (see `knowledge/sources.md`). The principle is firm; diff --git a/guardrails/checklist.md b/guardrails/checklist.md index 5abda2c..18d769a 100644 --- a/guardrails/checklist.md +++ b/guardrails/checklist.md @@ -9,6 +9,19 @@ it. Don't start until every box is checked. records blockers (default N=3). - [ ] **Budget ceiling** set as a *hard enforcement* stop (not just an alert). +## Containment (required companion to the stop conditions) +- [ ] **Network egress scoped** to the domains the loop needs, not full access + (`sandbox.network.strictAllowlist` denies the rest without prompting). +- [ ] **No long-lived credentials reachable** — SSH keys, cloud creds and the + home directory isolated from the loop. +- [ ] **Unattended runs deny by default** — `--permission-prompts none` on + headless hosts (v2.1.259+). +- [ ] **Isolation is a container/VM**, not a flag. `--restricted` is a + permission gate, not a sandbox. +- [ ] If the loop **clones or reviews untrusted repos**, you've read the + containment section of `guardrails/README.md` — attacks like GitSpawn fire + *before* any stop condition or verification step runs. + ## Correctness - [ ] **Deterministic success check** exists (test/lint/typecheck) returning clear pass/fail. diff --git a/runbooks/staying-current.md b/runbooks/staying-current.md index 7f22221..016d1c9 100644 --- a/runbooks/staying-current.md +++ b/runbooks/staying-current.md @@ -76,7 +76,7 @@ Routines can't be created from inside a Claude Code *web* session (the | **Trigger** | Schedule → Weekly (e.g. Monday 08:00 local) | | **Environment / network** | Needs web research. **Trusted** (default) lets `WebSearch` work but blocks `WebFetch` to arbitrary domains; set **Full** (or **Custom** allowing `code.claude.com`, `platform.claude.com`, `github.com`, `simonwillison.net`, `anthropic.com`) so it can read primary docs. | | **Branch pushes** | Leave default (`claude/`-prefixed only) — this forces a PR instead of touching `main`. The skill opens its PR from a `claude/knowledge-update-*` branch, so it stays inside this restriction; **do not** enable "Allow unrestricted branch pushes" (that would remove the floor). | -| **Env vars** | `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION=12` (hard-caps the ~5-agent fan-out against runaway recursion); optionally `CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION=100`. | +| **Env vars** | `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS=6` and `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH=1` (bound the ~5-agent fan-out and forbid recursion); optionally `CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION=100`. **Corrected 2026-09-07:** this row previously said `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION=12` — that variable was **removed in v2.1.224** and setting it now does nothing. There is no native per-session *total* spawn cap any more; concurrency + depth are what remain. | | **Connectors** | None required; remove extras. The PR is opened as your GitHub identity. | ### Budget & caps @@ -86,8 +86,10 @@ Routines can't be created from inside a Claude Code *web* session (the they only hold if the Routine is configured to let them: - **Iteration:** the weekly schedule bounds how often it fires; the skill never - re-invokes itself; `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION` (above) hard-caps - the fan-out. Keep the schedule **weekly**, not hourly. + re-invokes itself; the concurrency and depth caps (above) bound the fan-out. + Note they bound *width and recursion, not lifetime total* — no native + per-session spawn cap exists since v2.1.224. Keep the schedule **weekly**, + not hourly. - **Budget:** the account subscription usage limit + the per-account daily routine-run cap *reject* runs when exhausted — a real ceiling **only if metered overage / usage credits is OFF** for the account (otherwise spend @@ -140,7 +142,7 @@ weekly, with these differences: |---|---| | **Name** | Monthly loops artifact audit | | **Trigger** | Schedule → there is no Monthly preset (presets are hourly / daily / weekdays / weekly): pick the closest preset (Weekly), then run `/schedule update` from a local CLI session to set a monthly cron (e.g. 1st of the month, 08:00 local) | -| **Everything else** | Same as the weekly (repo `espi/loops`; env with `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION=12`; **Full/Custom** network for primary-doc checks; `claude/`-only branch pushes; overage off). See "Budget & caps" above. | +| **Everything else** | Same as the weekly (repo `espi/loops`; env with `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS=6` and `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH=1`; **Full/Custom** network for primary-doc checks; `claude/`-only branch pushes; overage off). See "Budget & caps" above. | Prompt (paste verbatim — points at the skill file, same anti-staleness reason as the weekly): @@ -180,6 +182,59 @@ run `/artifact-audit` on demand rather than waiting. Each applied edit is its own commit with a `git revert` written down — drop it wholesale if you don't want it, without unpicking the knowledge changes. +## Threat model: what these routines can reach, and why that's survivable + +Written down 2026-09-07 because the research these routines *do* now describes an +attack on the shape these routines *are*. Previously this reasoning existed only +in reviewers' heads. + +**The exposure.** `update-knowledge` is a scheduled, unattended job that reads +the open web, runs with no permission prompts, and can write to a repository. +Two 2026 results land on exactly that: + +- **Instruction privilege escalation** (arXiv:2608.27299, read in full; + independently replicated across 12 harnesses by arXiv:2609.01222). A harness + reconstructing context for a new invocation drops provenance and re-labels + content that arrived at *tool* level — a fetched web page, say — as a genuine + `user` message. It reproduces **through `/goal`- and `/schedule`-shaped + features**, automatic permission review does not stop it, and on Claude Code + specifically *"scheduled-task content is delivered to the working agent but + omitted from the context seen by Auto PR"* — the reviewer never sees the + injected instruction. **Neither paper proposes a mitigation.** +- **The web-research channel is the entry point.** Primer §5A already treats it + as a prompt-injection surface ("Friendly Fire"); the papers above make it a + *privilege* surface too. + +**Why the routine is still safe to run.** Not because the attack doesn't apply — +it does — but because the defenses are structural, sit outside the agent's own +judgement, and don't depend on the agent correctly noticing it has been +manipulated: + +1. **It never merges.** A human reviews and merges every PR. An escalated agent + can propose text; it cannot land it. +2. **The blast radius is `knowledge/`.** `guardrails/`, `budget.env`, + `CLAUDE.md`, `.claude/settings*` and the gate itself are outside the + auto-edit envelope — so the routine cannot widen its own permissions, + disable its own caps, or rewrite the rule that stops it. +3. **The fence is machine-checked, not self-graded.** `self-edit-guard` enforces + scope, protected regions and the per-pass cap in CI. A compromised agent + asserting "this edit is in scope" changes nothing. +4. **Branch pushes are `claude/`-prefixed only**, so it physically cannot touch + `main`. + +**What would invalidate this.** Treat any of these as a stop-and-rethink, not a +config tweak: + +- Enabling unrestricted branch pushes, or making the routine able to merge. +- Widening the auto-edit envelope beyond `update-knowledge/SKILL.md`, or + removing `self-edit-guard` as a check. +- Giving the routine connectors or credentials that reach beyond this repo. +- Granting it broader network access than the primary-doc allowlist needs. + +The load-bearing point: **the routine's autonomy is bounded by things it cannot +edit.** That is the property to preserve — every item above is a way of +accidentally trading it away. See `guardrails/README.md` → "Containment." + ## Self-guardrails `update-knowledge` and `artifact-audit` are both scheduled one-shots with a