Repository navigation
Conversation
…ts a name Weekly update-knowledge pass, window 2026-10-12 to 2026-10-19. Stacked on PR #28 (which is stacked on #27, on #26) per the skill's step 1, because main is 2,000+ lines stale. Headline findings, each verified by the lead against a primary: - v2.1.299: a mod kept a headless run going past --max-turns AND --max-budget-usd. First defeat of the dollar ceiling by anything other than its meter, and the vector is the extension surface -- which the mods doc documents as capability ("submit a prompt as if you had typed it"; "Mods aren't sandboxed"). - A whole-changelog grep found v2.1.217 (21 Jul 2026) fixing --max-budget-usd "not stopping background subagents" -- a hard stop #3 failure that sat available for three months, invisible to a weekly window-scoped sweep. Corrects the primer's claim that the budget ceiling would have caught hard stop #1's failures. - Codex 0.163.0 fixed a structurally identical interrupt leak of hard stop #2 one day after v2.1.299, so "interrupt/resume is where stops leak" is now a cross-vendor property rather than one vendor's bug. - arXiv:2609.27871 names vacuity and self-attestation; six independent instances identified, including Anthropic Code Review's cap and Qodo 3.1's automatic rule deactivation. - Google Cloud Spend Caps went GA 2026-10-13, and GA did not fix the documented leakage. - A guard gated on the truthiness of its own limit is not a guard: LiteLLM (still in v1.106.0), ralph-tui, and Gemini CLI (0 means both "stop immediately" and "unlimited" in one binary), against agentgateway's correct unconditional compare. Corrections: "no tool both caps spend and gates merges" is false (CodeRabbit gates; roborev does neither); the Ellipsis and Anthropic Code Review quotes are no longer on their pages; Gemini maxSessionTurns is a real iteration cap; Qodo is at 3.1. Also measured: this repo's own main has no branch protection, so the "physically cannot touch main" claims in runbooks/ and SKILL.md step 8 are false. Nine backlog items resolved and archived, six resolved in place, ~47 opened (95 -> 127). Primer held at its 750-line budget by moving older per-item prose verbatim to archive/primer-detail-2026-10.md. knowledge/ only -- no change to guardrails/, CLAUDE.md, runbooks/, .claude/ or .github/, and no self-edit: commit this pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDEU1huZuPhiQizA1oggoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Weekly
update-knowledgepass, window Oct 12 – Oct 19 2026. Five research agents; every load-bearing claim below was re-fetched and verified by the lead against a primary, not relayed from an agent's report.Routine self-improvements
Applied (auto-gated): none this pass.
I ran the step 7 scan and found no auto-apply candidate. Specifically, I verified on disk that all thirteen step-number cross-references in
SKILL.mdpoint at the right steps, that every path it references resolves (kb-size.yml,self-edit-guard.yml, all fourknowledge/files,guardrails/budget.env,runbooks/staying-current.md,CLAUDE.md), that the one relative link../../../knowledge/archive/resolved-caveats.mdresolves from the skill's own directory, and that no line has unbalanced backticks. Nothing stale, no broken link, no typo — so default-deny applies andSKILL.mdis untouched.git diff --name-onlyconfirmsknowledge/only, and there is noself-edit:commit forself-edit-guardto inspect.Suggested (needs your confirm):
mainhas no branch protection, and that makes two of this repo's own declared gates not gates. This is the one item I'd act on first, because it is now measured rather than inferred: the GitHub API reportsespi/loops'smainas"protected": false. Read against the currentroutinesdoc — "Claude pushes its work to a branch prefixed withclaude/unless your prompt directs it to push to another branch. To control which branches a run can push to, use branch protection rules or rulesets on GitHub" — the claims inrunbooks/staying-current.md("it physically cannot touchmain") andSKILL.mdstep 8 ("the routine physically can't touch main") are false. ✅ Andself-edit-guard.yml's own header already says the same thing about itself: "To make this a hard merge gate (not just a visible red X the reviewer sees), add it as a required status check in the repo's branch-protection rules formain. Without that, human merge remains the enforcement floor." ➜ One action fixes both: a ruleset onmainthat the connected identity cannot bypass, withself-edit-guardas a required check. Confidence upgraded Medium → High (the doc is now affirmative, not merely silent).runbooks/**and step 8's branch rule are both always-suggest.CLAUDE.mdmandates a deterministic pass/fail check inside the loop but says nothing about its behaviour on absence. The pass found a name and a taxonomy for exactly that gap (vacuity, "satisfied by the absence of the thing checked") plus six shipping instances. Proposed wording to add alongside last pass's provenance clause: exhaustion, timeout and error must not be expressible as pass. →CLAUDE.mdand the three hard stops are always-suggest.guardrails/may still rest on the unqualified version. I corrected the primer, but hard stop knowledge: 2026-06-22 update pass (7-day delta from Jun 15) #3's enforcement has now failed open twice (v2.1.217 and v2.1.299) andguardrails//templates/were not reviewed against that. →guardrails/is always-suggest.guardrails/checklist.md, and make the harness record which stop fired. The first generalises three independent instances found this pass; the second comes fromarXiv:2610.21098("most failures end at a harness limit", and the failure record doesn't say which). →guardrails/.CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MSinguardrails/budget.envand the templates, beside the existing watchdog settings. Both watchdog bounds are opt-in, so the default is still an unbounded wall-clock wait. →guardrails/.templates/ralph/run.shalready applies a per-calltimeout, so the practice is ahead of the doctrine; Gemini exposes a per-agentmaxTimeMinutes. → Changes the three hard stops, so always-suggest.artifact-audithas no trigger for "the model or harness changed, so re-measure the skills library" — carried forward from last pass and reinforced by2610.08875v2's 17 authoring practices, which are directly usable for this repo's own.claude/skills/. → Another skill, so always-suggest.--max-budget-usdfailure surfaced after sitting available for three months. The standing "backfill sweep" suggestion is now evidence-backed, and I'd extend it to "grep the full history per hard-stop token every pass", which is cheap.The headline: hard stop #3 is not the backstop the doctrine assumed
§6 has argued that hard stop #1's repeated failures were survivable because "what would have caught them is the budget ceiling or the stall detector." That argument no longer holds.
--max-turns,--max-budget-usd, an interrupt, or the end of the host's conversation." One bug, two hard stops. Every prior leak was first-party; this is third-party installed code overriding the harness's own budget ceiling.--max-budget-usdmentions ever, and v2.1.217 (21 July 2026) is one: "Fixed--max-budget-usdnot stopping background subagents." That sat available for three months, invisible to a weekly window-scoped sweep — the strongest possible vindication of the standing "the window sweep misses older work" item.And "interrupt/resume is where stops leak" graduated from a Claude Code bug to a cross-vendor property. v2.1.299 is the third
CLAUDE_CODE_RESUME_INTERRUPTED_TURNturn-accounting defeat in 15 days (eight fixes to that setting across its life). One day later Codex 0.163.0 (Oct 15) fixed a structurally identical leak of hard stop #2 — before it, every interrupt reset the consecutive-empty-turn counter, so the no-activity breaker (hardcoded>= 3) could be deferred indefinitely by interrupting. I verified it from source at the tag:let empty = automatic && !turn.has_activity && (turn.empty_final || interrupted);Two vendors, two languages, two different hard stops, same week, same seam.🔴 The failure this repo kept rediscovering has a name, a taxonomy and an instrument
The highest-value result came from the backfill read, not the window.⚠️ Cite its apparatus, not its numbers — the authors say so themselves: a fresh mutant family "recovers a 23.3% false-accept rate where the exhausted corpus reported none", and "the apparatus, not our number, is the contribution."
arXiv:2609.27871("Bounded Loops", v1 2026-08-20, cs.SE) defines a bounded loop as "a worker, an independent gate the worker cannot write to, and a declared budget" — this repo's doctrine in three terms — with the budget "enforced inside an attempt, not between attempts." And it names the two ways "a sound-looking check [can] pass anything": vacuity ("satisfied by the absence of the thing checked") and self-attestation ("where the subject supplies the value the check is applied to"), finding 47 vacuous gates in 69 shipped, reviewed loops.Six independent instances, which is why the name earns its keep: Anthropic Code Review's cap ("skips the review", check "doesn't conclude as a failure" — a correctly-configured hard ceiling can produce an unreviewed merge); roborev's
successmeaning "a comment was posted, including comments that contain findings"; CodeRabbit's "Incomplete analysis alone does not cause failure"; Qodo 3.1 (Oct 15) shipping "automatic rule deactivation for rules whose findings are rarely fixed" — a gate that erodes its own surface through use;2610.23362in deployed safety monitors ("the evidence is already recognized yet disconnected from enforcement"); and2610.19264, where 59% of declaring skill listings understate what their bundle can do.✅ The safe design is attested twice — Ellipsis ("never as an approval"; "when no reviewer finishes, nothing is posted and nothing is marked reviewed") and Greyling (a verifier "makes rejection the default, so the agent's output must 'earn' approval").
Corrections this pass owes the knowledge base
errormode + a fail-closed Security required check) and reportedly caps spend. And roborev does neither — "The cap never stops jobs" — where the KB credited it with the spend-cap half. Surviving, sharper claim: no reviewer documents what its spend cap does to its merge gate.maxSessionTurns= "history retention, docs ambiguous"/goalevaluator on every non-Anthropic provider; check next pass"gascitygastownhall/gascity—steveyegge/gascityis not publicly readable.✅ Google Cloud Spend Caps went GA (Oct 13) — and GA did not fix the leak
Verified from the release-notes page's own dated entry: "October 13, 2026 — Spend cap budgets are now generally available for a limited set of services." The first GA'd project-level spend ceiling from a hyperscaler (AWS's is still limited-availability).⚠️ Every caveat is live verbatim at GA: "spend cap enforcement isn't instantaneous", "Any in-flight requests … are processed to completion, accruing charges as applicable", "You remain responsible for any overage costs accrued due to reporting latency", plus up to an hour to resume. Still one project × one eligible service, monthly, enforced on estimated costs. ✅ Google states this repo's runway rule itself: "Consider setting your budget slightly below your absolute limit." The strongest first-party statement yet of invariant (a): a ceiling bounds new work, not the bill.
🔴 Most transferable finding: a guard gated on the truthiness of its own limit is not a guard
Three instances, each read from source. LiteLLM reads⚠️ and the sharper fact is that the surrounding budget-window code was refactored in-window while the zero guard was left alone, so this is neglect of the bug, not of the file (#44259 still open, no activity since Oct 2). ralph-tui gates its cap on
max_budget = 0as unlimited in all three router paths and v1.106.0 (Oct 18) still ships it —maxIterations > 0, so the documented-as-valid0silently removes hard stop #1. Gemini CLI means both things with one value in one binary:maxSessionTurns: 0stops on turn 1 in two code paths (>= 0) and is no cap at all in a third (> 0) — the cleanest instance the KB has, because there is no doc ambiguity to hide behind. ✅ The correct form is attested in code:agentgateway(AAIF-hosted) uses an unconditionalused >= budget.limit.amountwith anAudit/Blockenum, and Gas City'swithDefaults"fills every unset or non-positive duration and threshold… so no Policy can pace at zero."The meter is wrong four ways now, across two vendors — two of them stop a loop early
Claude Code: under-counting (v2.1.286), over-pricing 2× (v2.1.296), and⚠️ a cloud session's cost and usage "restarting from zero after the session resumed in a new process" (v2.1.302 — the spend analogue of
/goal's baseline reset, changelog-only, and cloud sessions are what Routines run on). Agent Router: a systematic +1 token per request against a token quota, "resulting in29tokens deducted" for a 28-token request — unreleased. Relatedly, Anthropic's Spend Limits API disowns itself as a control:period_to_date_spend"may read as"0"if the spend reading is temporarily unavailable; treat it as informational, not transactional" — a fifth meaning of zero, and here it means unknown, so a harness comparing it to a ceiling proceeds during an outage.This routine's own substrate
Completedfor runs that died — a fourth way a routine appears to have run while doing nothing.dist-tags.stableis2.1.292whilelatestis2.1.303— a harness pinned tostableis eleven versions behind and has none of the fixes above.Other findings worth your time
arXiv:2608.21884(Treude, Baltes et al.) mined 36,710 repositories, confirming loops in 217 of 256 matched, and its headline is a rebuke to prescriptive loop advice: "The repositories commit the configuration around these loops, but almost none commits the state files the discourse prescribes." It dates Osmani's coinage (25 July 2026, revising Yegge's January 2026 stages) and concludes verification cost is the bound on delegation.arXiv:2607.00038defines a "loop specification" — trigger, goal, verification step, stopping rule, memory. ⭐ Two independent groups, different methods, agree: verification is the most mature part of the practice and durable state the least.arXiv:2610.18004(SWEPatchTest, n = 619 tasks / 82,579 patches) — "the best-performing configuration achieves only 9.21%… despite reproducing the reference transition on 59.29% of tasks." Reproduction alone, the usual bar, is nearly uninformative.REVIEW.mdbesideCLAUDE.md.arXiv:2610.23254(n = 33,000 trajectories): "most failed attacks are ignored (76-80%) rather than refused", and "changing only the system prompt shifts InjecAgent ASR from 0.2% to 43%" on identical weights.GOOSE_MAX_TURNS(default 1000) still "stops and prompts" — in an unattended loop with nobody to answer, that is a hang, not a stop.ext/goal(on by default, stable) splits exactly as this doctrine predicts: token budget harness-enforced ("Budget limits take precedence over pausing"), no-progress and completion left to prompt instructions the model grades. Cursor's/goalis still "Rolling out" after ~9 weeks; Factory shipped 7 releases with zero hard-stop changes; Aider has had one release in fourteen months.aaif.io/projectsconfirms while naming Agent Router as hosted. Agent Skills still has no tag, release, version, charter or TSC and has been quiet since 2026-08-09.--restrictedis still absent fromsandbox-environments(5th pass) —Verification method
changelog.md(1.1 MB, with explicit<Update label="…" description="October 18, 2026">date attributes) rather than the 4.8 MB rendered page — cheaper and immune to the year trap. In-window range sliced and grepped locally; the whole file was also grepped per hard-stop token, which is how v2.1.217 surfaced. Version dates cross-checked against the npmtimemap.id_listcalls (both HTTP 200): 22/22 resolved, zero unresolved, zero hallucinated, 15 with v1 in-window, each checked for ID ↔ exact title ↔ v1published↔ primary category ↔ author count. Two agent-reported corrections were confirmed:2610.09159is cs.AI, and2610.13065is titled TestSuiteBench.main/v1.106.0/v1.105.1/v1.107.0-rc.1;agentgateway's comparison at HEAD; Codex'saccounting.rsat tagrust-v0.163.0; Gas City's config and dispatch at HEAD.--restrictedhad appeared insandbox-environments(resolving a five-pass item); checking the context showed it was the ordinary word "restricts". The item stays open.espi/loops, and no agent calledadd_repoon a third-party repo. ✅ New:git ls-remote --tagsplus a filtered single-tag clone works through the proxy, which dated six third-party projects and retires the standing request for read access toopenai/codex,block/gooseandsst/opencode.Pruning and the size budget
The primer went 748 → 750 lines, exactly its budget, so
kb-sizeshould pass at 750/750. Holding it there while absorbing this much required moving older per-item prose verbatim into a newarchive/primer-detail-2026-10.md(§5A items 3–6 and 9–11, item 8's carrier detail, item 12's full vacuity dossier, and §3's fuller per-voice prose) and condensing §3, §4's peer matrix, §5B and §6's per-vendor paragraphs. Every claim removed from the primer is still carried, with its source and confidence, insources.md. Per step 6, superseded text was replaced, not left beside the correction.Nine backlog items archived (Google Spend Caps' GA question + its companion quote item; Gemini's
maxSessionTurnsambiguity; opencode's tag dates; "three peers not date-trackable"; the three unread backfill papers; Osmani's two uncited statistics; the AAIF membership question;plugin eval's budget masquerade), six more resolved in place because each left a narrower successor question.Per step 2, the full standing backlog, not only what is new. It lives in
sources.mdunder "Known caveats / things to re-verify".*marks the 35 opened this pass.Security (19) —
*the harness's ceilings are defeatable from the extension surface;*a guard skippable by prototype-key stripping + plan mode writing through an allowed hook;*frontmatterallowed-toolsremoves prompts but does not restrict reach (59% under-declare);*a low ASR is not evidence of a defence;*adopt the vacuity/self-attestation vocabulary and its three follow-on guardrail calls; the in-loop check needs a provenance clause; a metered verifier's ceiling is a kill-switch; scanning source is not admission control / popularity is not provenance; discount ASRs that presuppose selection; adoptonFailure: "block"; per-edit gates don't compose; a command-prefix allowlist is not a sandbox; GitSpawnultrareview(7th pass, unconfirmed); does the PR-review gate bind what the reviewer sees to what merges?; is every write path to a protected region covered byself-edit-guard?; vendor artifact for 2609.01222; does X-CPE reach this repo'sknowledge/?; adopt PROCTOR canaries / hash-pinning / provisional-persistent split / annealed edit budget?;--restrictedis not an OS sandbox (5th pass).Guardrails (41) —
*🔴mainhas no branch protection (measured);*🔴 hard stop #3's enforcement has failed open twice;*interrupt/resume is a cross-vendor leak;*🔴 a truthiness-gated guard is not a guard (3 instances);*Gemini's one-value-two-meanings zero;*the meter is wrong four ways across two vendors;*the Spend Limits API reads0for unknown;*record which stop fired;*a stall detector's false-positive side;*wall-clock as an unnamed fourth stop;*three findings on this routine's own substrate (duplicate re-run,Completedfor dead runs,-ppermission-mode change);*setRETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS(undocumented);*thedist-tags.stablelag;*budget ceilings should scale with code health; theclaude/prefix is a default not enforcement; the Routine caps are hourly and wait; three ways a routine can appear to run and do nothing; hard stop #1's fourth leak +/goal's resume reset; the meter erred both ways; setRETRY_WATCHDOG_MAX_WAIT_MS; an aborted call is still billed;/loopships none of the three stops; Haiku 5.5's cliff on the/goalevaluator (wants a measurement); the AWS cell's opt-in correction; the exhausted-budget header vs status code; Managed Agents' three boundaries; task budgets unsupported on Claude Code + too-small-budget refusals; AgentGuard's Claude Code hook + the 90-day price-table gate; an available stall policy is not an engaged one; classify failures to short-circuit the cap; a deterministic ceiling on a replay substrate; a ceiling can leak through its meter; four status codes for an exhausted budget; a per-run budget is not a campaign ceiling (this routine's shape); AWS's limited availability + destructive tail; OpenRouter's zero-budget semantics; fixed caps vs progress-authorized continuation + context growth; verify an injected guardrail reached the model; zero-valued ceilings (LiteLLM still shipping it in v1.106.0); the LiteLLM fail-open family; Anthropic's undelivered spend controls (3rd pass); adopt OpenRouter's cost endpoint.Tooling (24) —
*``plugin eval's residual gap, narrowed and now documented as deliberate;*MCP allow rules narrowed silently;*Factory's 10-runs/hour cap unlocated;*Amp is a content block;*the Haiku tripwire must be re-pointed at the Bedrock/Vertex tables;*``experimental-ext-server-carddormant while SEP-2127 merged;*usegastownhall/gascity;*Gas City's unreleasedworkladder;*Agent Router'sserviceQuotastill unenforced;*AG2 v1.2.0 tagged-but-unpublished (8 days);*add tracking foragentgatewayand A2A;*Qodo is 3.1 + roborev v0.72.0 + Kodus not reached + CodeRabbit's cap needs first-party confirmation; the peer-harness matrix correction; the Codex row overstated; the MCP standards corrections + satellite repos; the Agent Skills governance asymmetry; the first-party plugin evaluator reads the maker's trace; the MCP metadata budget growth; URL and naming hygiene; two unreleased things to confirm; opencode's v2.0.x line (dates now resolved); the 1 GB tool-results cap (5th pass, changelog-only); the/loop//goalversion-minimum human call;SKILL.mdcross-tool execution.Research / infrastructure (43) —
*three fetch techniques that remove coverage holes (full-text feeds, the.mdchangelog, the npm tarball);*the byline trap moved into thedatetimeattribute;*grep the author's own archive before calling a statistic uncited;*two host migrations + two path corrections;*arXiv API behaviour + the raw-host directory-listing limit;*blacklist additions +dev.to/mech_app_aias a distortion source pattern;*dated tripwires (Gartner Oct 19–22 now live; Hype Cycle for Agentic AI Oct 20; Sonnet 4.5 retires Nov 30; GPT-5.6 promo to Nov 21; LiteLLM v1.107.0 / #44259);*the Ronacher wording rule; the YYMM trap in both directions; shallow-clone date fabrication + GitHub topic-page strings; the ghuntley byline skew; stop citing an aggregate skill count; skill utility is configuration-dependent (anartifact-audittrigger); the Mandiant distortion watch; annotate the two $6,000 figures; blacklist additions + the per-claim nuance; the unpromoted Cherny lead; two research-path techniques; Jev corroborated as a classifier; Osmani's.comblog lags Substack (now ~30 days, and it has no working feed); magnitudes don't cross papers; Steinberger's permanent coverage gap; the x.com note correction + snowflake decoding; Osmani's Sep 7 X post (still unread on his surface); Cherny's structural limit; Huntley's recap dating; the YC-hackathon field report; the "Jev may be two things" question; the do-not-cite SkillsBench figures; the marketing-dominated query space; the four recycled dollar anecdotes; the Jun 15 billing split; Microsoft dropping Claude Code; the/goalorigin timeline; the "costliest thing" slogan; roborev's 403s (raw-mirror workaround confirmed); the arXiv 406/301 modes;openai.comPDF primaries; the GitHub access shape (now improved bygit ls-remote).Worth a closer look
mainis measurably unprotected, and the same single action also turnsself-edit-guardfrom a visible red X into a real gate. This repo currently asserts two enforced gates it does not have, which is precisely the error it exists to warn about.loop-guardrails) are small and concrete.kb-sizeshould pass at 750/750;self-edit-guardhas noself-edit:commit to inspect.guardrails/,budget.env,CLAUDE.md,.claude/settings*,runbooks/,templates/or the gate was touched — this PR isknowledge/only.Generated by Claude Code