Skip to content

knowledge: Aug 24–31 pass — gateway spend limits fail open by default, CC v2.1.243–251, two loop benchmarks converge at ~25% - #19

Merged
espi merged 1 commit into
mainfrom
claude/focused-clarke-y4kbqb
Sep 1, 2026
Merged

espi merged 1 commit into
mainfrom
claude/focused-clarke-y4kbqb

Conversation

@espi

@espi espi commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Routine self-improvements

Applied (auto-gated) — none this pass.

No auto-apply candidate arose. I re-checked this skill's own repo-internal state against the may-auto-apply categories and everything resolves correctly: the step cross-references in steps 7 and 8 point at the right steps, the relative link ../../../knowledge/archive/resolved-caveats.md resolves to a tracked file, and .github/workflows/self-edit-guard.yml (plus .github/self-edit-guard.py) exist as the gate text claims. No stale cross-reference, broken path, or typo to fix — so the default-deny outcome is "nothing to apply," not "something suppressed."

Worth noting the guardrail text was checked and held: nothing in-window changed CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS (20) or CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH (3), and the Routines docs still confirm the daily-run-cap-plus-metered-overage behaviour the "Budget" bullet depends on.

Suggested (needs your confirm) — three, all outside this SKILL.md and therefore always-suggest, never auto-apply:

  1. guardrails/: consider --restricted as a documented default for untrusted-input loops. v2.1.248 added --restricted / CLAUDE_CODE_RESTRICTED=1, which "removes the built-in tools that run commands or code and WebFetch… keeps file tools inside the working directory, refuses bypassPermissions, and ignores user, project and local settings files." That is a cheap blast-radius floor for the §5A "Friendly Fire" threat model. Routed to a human: edits to guardrails/** are on the always-suggest list, and the supporting claim that it is not an OS-level sandbox is secondary-sourced only (now a re-verify item).
  2. runbooks/staying-current.md: record that cloud routines cannot use locally configured MCP servers. v2.1.251's /schedule now says so explicitly — a connector must live on the claude.ai account or in a committed .mcp.json. This routine is unaffected (its GitHub access is an account connector), but the runbook documents the Routine setup path and someone following it with claude mcp add would hit this. Routed to a human: runbooks/** is always-suggest.
  3. A future §6 revision may want "non-decaying loop safety state." arXiv:2608.27141 argues that safety monitors which reset state between loop iterations are structurally defeatable. It is a good fit for the hard stops but it is a semantic change to a non-negotiable section, and it is web-sourced — both always-suggest triggers.

Baseline and scope

Last pass: 2026-08-24. Window: Aug 24 – Aug 31. Per step 1, I checked for an open update-knowledge PR before starting: none open (PR #18 merged; the branch sat at that HEAD), so this is a clean baseline with no expected merge conflict.

Five parallel research agents (tooling & versions, ecosystem & techniques, key voices, guardrails & cost, verification & skills); all five returned cleanly. A substantive window in every lane.

The headline is a correction to our own §6

Anthropic ships real spend enforcement, not only alerts — §6 previously read the 75%/90% figures as the whole story, but those are just the Claude Code warning layer. Underneath sit the Spend Limits API (Enterprise) and Claude apps gateway spend limits, which genuinely block: 429, error.type: billing_error, x-should-retry: false, daily/weekly/monthly each enforced independently, with client aborts billed on a floor estimate "so aborting requests early doesn't evade a cap."

But the default failure mode is the thing. The gateway's pre-check queries Postgres on a two-second timeout, and — verbatim — "if the store is unreachable or times out, enforcement fails open by default: the request proceeds, the gateway logs a warning." You need enforcement.fail_closed_on_error: true for "no unmetered spend."

So the ceiling silently degrades into an alert precisely when infrastructure is unhealthy — which is when a runaway loop is most likely to be running. It is the same shape as LiteLLM's fail_closed_budget_enforcement: on both gateways, the true ceiling is behind a non-default flag. I read and verbatim-verified this primary doc myself rather than relying on the research agent.

What's new

  • Claude Code v2.1.243–251 (7 releases, no new model, nothing touching --max-budget-usd/max_turns/subagent caps). Loop-relevant: /usage Loops breakdown (first per-loop cost attribution — visibility, not enforcement); a subagent stopping at maxTurns now returns output marked partial "instead of appearing finished", closing a real false-"done" hazard; /goal idle sessions capped at three check-ins per goal; and --restricted. Constraint we run into: locally configured MCP servers can't attach to cloud routines.
  • Sonnet 5's introductory $2/$10 pricing is now permanent — verified on 2026-08-31, the stated last day of the promotion: "The previously scheduled increase to $3/$15… on September 1, 2026 will not occur." The default model's cost basis is not rising 50%.
  • Anthropic weekly Claude Code limits move Sept 14: +25% vs the pre-promotion baseline, but ≈−17% vs today once the +50% boost expires Sept 13. Weekly limits are enforcement, and loops hit them first.
  • Two new key-voice essays pointing opposite directions. Osmani's "Audit your Agent files" (Aug 27) is the first in his arc to argue for subtracting scaffolding ("Your coding agent's configuration has a half-life"). Yegge's "Fences, not Sandboxes" (Aug 24) rejects containment for governance-by-law ending in "mechanical enforcement." §3 records the tension rather than smoothing it: Yegge's endpoint is our position, but he arrives by escalation from soft norms — and a rule that hardens only after re-violation is a bill already paid.
  • Two independent loop benchmarks agree on the ceiling. LoopsBench (the backlog read, 25.00%) and in-window LoopArena (24.69%), both finding verification and regression management, not coding ability, is the binding constraint — the strongest empirical backing our central claim has.
  • A verification arXiv cluster with v1 dates machine-verified via an explicit arXiv API date-range query (not inferred from the ID prefix). Best of it: EvoUndo — 197 of 600 self-evolution modifications fail recoverability verification and conventional repair recovers 0 of 197, the sharpest external argument yet for a CI-enforced rather than self-graded self-edit gate.
  • Skills-as-durable-asset gets numbers: inside skills/ directories, instruction files and their scripts co-evolve with 78% of co-changes functionally coupled; and agent-first repos without committed AI config grow cognitive complexity ~2× faster, with 73.8% of AI-config artifacts committed once and never modified. Flagged explicitly as observational, not causal.
  • roborev v0.67.0, recorded with an explicit null result: nothing on budget, gating, or skill-invocation policy.

Corrections, rejections and traps caught

  • A research agent flagged the v2.1.246 /goal item as possibly a restatement of v2.1.239's back-off. Spot-verification against the primary shows it is a separate, new cap — the opposite of the usual one-version-off error.
  • Why that error kept happening: v2.1.242, v2.1.244 and v2.1.249 do not exist in the changelog. The gaps are what prior passes were tripping over. (npm publish dates also run a day earlier for part of this batch; changelog dates used.)
  • The "Agent SDK billing split went live July 10" claim resurfaced a third time. Rejected again, and the paused status upgraded Medium → High against a directly-read primary.
  • A summariser attributed "27% reactive… n=573" to the VB Pulse survey; the article says n=107, 21%. Our existing figure stands.
  • RedHub AI's Aug 28 piece — the only in-window article squarely on our thesis — was read and rejected: no provider named, no primary source, no testing, mismatched dates, funnels to a $49 product.
  • arXiv:2608.27086 states it has no implementation, experiment or result. Logged as a position paper, not citable as evidence.

Quiet lanes, stated rather than padded

Gas Town/Gas City, Huntley/ralph/Loom, OpenHands, LangGraph/ADK/CrewAI/AG2, Devin, OpenRouter, Databricks, Cloudflare, AgentGuard, LoopGain, Helicone/Portkey, Agent Plugins, AAIF, Gartner/Forrester/IDC, and Steinberger/Cherny/Huntley all produced nothing in-window. No new cost-overrun incident. No Week 35 whats-new digest.

Archived

LoopsBench-not-yet-read → archive/resolved-caveats.md (read in full, promoted, Low → High). Per step 6 the superseded text was removed, not annotated: the expired "Sonnet 5 promo through Aug 31" wording and §5B's "alerting layer, not confirmed to hard-stop" characterization were both replaced in place.

Standing re-verify backlog — every currently-open item

Per step 2, the full live backlog, not only what is new this pass.

Carried forward unchanged (12):

  1. SKILL.md cross-tool execution is still not fully settled
  2. The "costliest thing is managing the agent loop" slogan is a community paraphrase, not a sourced Cherny quote
  3. The /goal "Codex invented it, Claude copied in 11 days" timeline rests on one secondary
  4. $47K / 11-day loop and overnight-billing figures are self-reported anecdotes
  5. $500M in one month — unnamed company, no primary disclosure
  6. June 15 2026 billing split — paused status upgraded to High this pass; still open pending a revised plan
  7. Microsoft dropping Claude Code — no primary Microsoft announcement read
  8. Huntley's Loom as an "orchestrator of ralph loops" — README more modest than the talk
  9. AI Engineer World's Fair 2026 sessions — aggregator-sourced, no verbatim quotes
  10. roborev.io/changelog 403s to automated fetch while GitHub releases read fine
  11. EvoAgentBench (arXiv:2607.05202) — too thin to promote
  12. SkillCheck (getskillcheck.com) — no in-window release found

New this pass (7):
13. modelPricing vs --max-budget-usd — the docs say contracted rates drive /cost, status line and telemetry, but not whether the harness ceiling meters at contract or list price. Changes what a given budget number buys.
14. The Sept 14 weekly-limit change — help-center page not updated as of Aug 31; re-verify after it lands
15. Claude Managed Agents' $0.08/session-hour runtime billing — a wall-clock cost dimension token-only budgeting misses; ship date not established, so not promoted
16. --restricted "is not an OS-level sandbox" — secondary-sourced only; verify before recommending it in guardrails/
17. arXiv:2608.27299 "When Context Gets Root" — reports escalation via ordinary /goal- and /schedule-shaped features; only the abstract was read, and it bears on the routine-autonomy model this repo runs on
18. LiteLLM v1.99.0 — does it GA or get skipped for v1.100.0?
19. Unverified aggregator claims — notably an alleged Hugging Face agent-escape incident. Would matter to §5A's threat model if true; must not be promoted on current evidence.


Files: knowledge/00-primer.md, knowledge/sources.md, knowledge/CHANGELOG.md, knowledge/archive/resolved-caveats.md. No changes to guardrails/, budget.env, CLAUDE.md, permissions, or any skill file.


Generated by Claude Code

…43–251, LoopsBench+LoopArena ~25% convergence

Five-agent fan-out over the Aug 24–31 window. Headline is a correction to this
repo's own primer §6 rather than an external finding.

§6 correction (High, primary read verbatim): Anthropic ships real spend
enforcement, not only alerts — the Spend Limits API (Enterprise) and Claude apps
gateway spend limits both hard-stop requests (429 / billing_error). But gateway
enforcement FAILS OPEN by default when its Postgres spend store is unreachable or
times out; enforcement.fail_closed_on_error: true is required for "no unmetered
spend". The ceiling degrades into an alert exactly when infrastructure is
unhealthy. Same shape as LiteLLM's fail_closed_budget_enforcement: on both
gateways the true ceiling sits behind a non-default flag. §5B's "alerting layer,
not confirmed to hard-stop" characterization corrected in place.

Claude Code v2.1.243–251 (7 releases, no new model): /usage Loops breakdown
(first per-loop cost attribution); subagent maxTurns results now marked partial
instead of appearing finished (closes a false-"done" hazard); /goal idle sessions
capped at three check-ins per goal (a new cap on top of the v2.1.239 back-off, not
a restatement — confirmed by spot-verification); new --restricted containment
flag. Constraint for this repo: locally configured MCP servers cannot attach to
cloud routines. All four headline attributions spot-verified verbatim; v2.1.242,
244 and 249 do not exist, which explains prior passes drifting one version off.

Sonnet 5's introductory $2/$10 pricing made permanent — verified on the stated
last day of the promotion; the Sept 1 increase to $3/$15 will not occur. Expiring
primer fact replaced rather than annotated.

Also: Anthropic weekly Claude Code limits move Sept 14 (+25% vs baseline, ~-17%
vs today); Osmani "Audit your Agent files" and Yegge "Fences, not Sandboxes" as
opposing in-window positions on scaffolding, with the tension recorded; LoopsBench
(backlog read, 25.00%) paired with independent in-window LoopArena (24.69%), both
naming verification/regression management as the binding constraint; a verification
arXiv cluster with machine-verified v1 dates (EvoUndo: 0/197 conventional repair on
failed recoverability); skills co-evolution data (78% of SKILL.md/script co-changes
functionally coupled); roborev v0.67.0 with an explicit null result.

Rejections logged: the "Agent SDK billing split went live July 10" claim for a
third time (paused status upgraded to High against a directly-read primary); a
snippet conflating the VB Pulse survey as n=573/27% (actual n=107/21%); RedHub AI's
unsourced spending-limits piece; arXiv:2608.27086 as a no-result position paper.

Backlog: LoopsBench resolved and archived; 12 items carried forward unchanged;
7 new, including whether --max-budget-usd meters at contracted or list price under
modelPricing, and unverified aggregator claims held back pending primary sourcing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BGke13bsngqBGuPLGX8ueT
@espi
espi merged commit 6de5d5a into main Sep 1, 2026
1 check passed
@espi
espi deleted the claude/focused-clarke-y4kbqb branch September 1, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants