Skip to content

artifact-audit (2026-10-25): correct drifted tool claims; flag guardrail/hard-stop drift - #30

Open
espi wants to merge 1 commit into
mainfrom
claude/keen-archimedes-woryzc
Open

espi wants to merge 1 commit into
mainfrom
claude/keen-archimedes-woryzc

Conversation

@espi

@espi espi commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Monthly artifact audit — 2026-10-25

First audit since PR #24 (merged 2026-09-29). Four parallel slice audits (skills / templates / guardrails / runbooks+docs) against the merged KB (last pass 2026-09-28), the pending unmerged KB chain (#26→#29, through Oct 19), and primary docs fetched this pass: code.claude.com/docs/en/{goal,routines,scheduled-tasks,cli-reference,claude-code-on-the-web}.md. Every fix below was verified by the lead against the quoted primary, not relayed from an agent.

Yield rule applied: open PR #26 mid-edits most practice artifacts (guardrails/*, both other loop skills, update-knowledge/SKILL.md, three runbooks, templates/ralph/run.sh, CLAUDE.md) and #27–#29 stack on knowledge/. Nothing those PRs touch was edited here — findings on those files are in Flagged for you. Branch note: the skill names a claude/artifact-audit-YYYY-MM-DD branch, but this scheduled session is pinned to push only to claude/keen-archimedes-woryzc, so the PR ships from there.

Artifact drift

Fixed

File What was stale Evidence
templates/goal/recipes.md "an impossible condition loops until you stop it" — wrong: the evaluator clears a goal it judges Impossible (recorded as failed) and a no-progress run (no tool use for several turns) pauses with the goal still set goal.md (primary, this pass): "Claude Code clears the goal and records a failed entry… You don't need to clear it yourself"; "no tool use for several turns in a row… stops the loop… goal still set"
templates/goal/recipes.md No recipe carried any bound — an uncapped loop pattern in a repo whose one rule is "never run an uncapped loop" goal.md: "To bound how long a goal runs, include a turn or time clause in the condition, such as or stop after 20 turns". Each recipe now carries one, plus guidance to add external --max-turns / --max-budget-usd (print-mode flags, cli-reference.md) on headless runs, with the caveat that the in-condition clause is evaluator-judged from the agent's narration, not an enforced cap
templates/goal/recipes.md "~4,000 characters" (exact documented limit); "Interrupt a run: Ctrl+C" (only true for non-interactive -p runs) goal.md: "up to 4,000 characters"; "Interrupt the process with Ctrl+C to stop a non-interactive goal"
runbooks/staying-current.md "Branch pushes are claude/-prefixed only, so it physically cannot touch main" and the "default push restriction / do not enable unrestricted pushes" framing — no such restriction setting exists; the prefix is a prompt-steered default routines.md (primary, this pass): "Claude pushes its work to a branch prefixed with claude/ unless your prompt directs it to push to another branch. To control which branches a run can push to, use branch protection rules or rulesets on GitHub… a rule that access can bypass doesn't block a run's push." Settings row, threat-model point 4, invalidation list, and the "Branch prefix — reconciled" note all rewritten: the floor is a ruleset on main + the skill's prompt, not the prefix. ⚠️ See operational flag below
runbooks/staying-current.md "per-account daily routine-run cap rejects runs" — caps are hourly and over-limit scheduled runs wait, they don't fail; only the subscription limit (usage credits off) rejects routines.md limits table: "Scheduled runs… 100 per hour… The run waits until the limit resets"; "Without usage credits, additional runs are rejected"
runbooks/staying-current.md "Routines run autonomously with no approval prompts" — runs start in auto mode and a prompt (ask rule, repeated blocks) waits in the run's session routines.md: "A run still stops and waits for an answer"
runbooks/staying-current.md "no Monthly preset → pick Weekly, then /schedule update from CLI" — the web edit dialog now takes a Custom cron directly; also noted :07-style minutes (documented late starts on :00) and the New task button rename routines.md: "select Custom in its edit dialog. Then enter a cron expression"; "pick a few minutes past the hour, for example 9:07"
runbooks/babysit-prs.md "Stop early with Esc" (blanket) — Esc stops only a self-paced loop; a fixed-interval loop is cancelled by asking Claude ("Esc and Ctrl+C don't stop it") scheduled-tasks.md "Stop a loop"
runbooks/babysit-prs.md "Enable via the web CI bar…" — no CI bar; paths are the PR's ⋮ menu on the session's Changes card, /autofix-pr on the PR's branch, mobile, or pasting a PR URL. Added the doc's warnings: replies post as your GitHub account and can trigger comment-driven automation (Atlantis etc.); no webhook for merge conflicts claude-code-on-the-web.md "Auto-fix pull requests"
templates/loop/loop.md Session-scoping comment over-strong: backgrounding the session carries /loop over; a self-paced loop is not restored on --resume scheduled-tasks.md limitations: "Backgrounding the session carries /loop tasks over"; "A self-paced /loop isn't restored, so run /loop again"
.claude/skills/artifact-audit/SKILL.md Its own budget-backstop line said "subscription / daily-run cap is the ceiling" — same hourly/waits correction as above routines.md limits table
README.md / templates/ui-audit/FINDINGS.md Templates table omitted the ui-audit kit; FINDINGS.md told you to rename a file already so named Repo-internal

Rename note: the feature is now officially "cloud scheduled tasks, formerly routines" (routines.md title). Fixed files got a one-line note at first mention only; wholesale renaming should wait until the KB records it — even pending PR #29 (Oct 19) missed the rename, so it's a job for the next update-knowledge pass.

Flagged for you

1. Operational, highest priority: with the push "restriction" gone, the only enforcement keeping these routines off main is a GitHub branch-protection ruleset on main that the connected identity cannot bypass — the pending KB (#29) measured espi/loops main as "protected": false. Enabling one also lets you make self-edit-guard a required check (as staying-current.md already suggests). Until then, "never merge to main" is held by prompts and review habits only. The same false "physically can't touch main" claim sits in .claude/skills/update-knowledge/SKILL.md step 8 — not edited here (#26 mid-edits that file, and its branch rule is in that skill's protected always-suggest set); fold the correction into #26 or a follow-up.

2. loop-guardrails/SKILL.md:23-24 (yield to #26; NOT fixed by #26): "/goal → bounded by the validator" is wrong per primary — /goal's only documented bound is an in-condition clause the worker narrates and the tool-less evaluator grades, and a resume "resets the turn count, timer, and token-spend baseline" (goal.md). Proposed replacement: treat /goal as a stop condition, not hard stop #1; require an external cap (--max-turns per call on v2.1.281+, or a harness counter) alongside it.

3. new-loop/SKILL.md (yield to #26; not fixed by it): step 3.3's --max-budget-usd guidance omits that the flag is print mode only (cli-reference.md), spend can pass the cap, and resumed totals don't count. Also the bare "/goal v2.1.139+" and "/loop v2.1.72+" floors (here, in CLAUDE.md, and in templates/goal/recipes.md) are no longer verifiable against any primary — docs state per-feature minimums only; already on the KB re-verify backlog, left as-is.

4. guardrails/README.md + checklist.md (yield to #26 + hard-stop texts; none of these are in #26's diff):

  • Two do-not-cite receipts in the credibility section: the Uber $1,500/month figure is KB-downgraded to Low/do-not-cite (Willison confirms a cap, no figure), and the "$47K / 11 days" anecdote is the class the KB "refuses to cite" — replace with the two citable primaries it now holds: Mandiant Case study 6 ($50K in under an hour, 15,000+ calls, availability incident) and METR ($600K, "no way to put a spending limit on keys like this one").
  • Missing claude ultrareview containment rule — the KB backlog's standing ask of guardrails/: GitSpawn's fsmonitor path was patched in v2.1.196 but the ultrareview path remains unpatched (re-checked through Oct 19); the trigger needs only a working tree with .git/ intact, not a clone. Add the "never run claude ultrareview in a loop against an untrusted tree" checkbox + the v2.1.196/patch nuance.
  • Gateway/library drift: LiteLLM's real pre-flight primitive is budget reservation (on by default), fail_closed_budget_enforcement is a separate non-default backstop (and pending-KB: max_budget = 0 = unlimited; exhaustion surfaces as 422, not 429); OpenRouter workspace budgets return 403, key budgets 402; Portkey was acquired into Prisma AIRS; Helicone is in maintenance mode — stop recommending "confirm it blocks", the KB's answer is "don't build on it"; AgentGuard needs disambiguation (bmdhodl/agent47, pip agentguard47; v1.3.1 moved enforcement pre-dispatch); pending-KB says LoopGain covers stop knowledge: 2026-06-15 update pass (6-day delta) #2 only — stop crediting it with all three.
  • Emphasis: the 20-concurrency cap has documented holes (ultracode exempt, /subtask and resumed subagents bypass, workflows uncounted) — KB's rule: "say the spend cap, not 'a cap of 20'". WebSearch 200-cap now holds only for headless runs (interactive refills 100/hr since v2.1.290, pending-KB). Tombstone wording: the env-vars page carries CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION as an explicit "Removed in v2.1.224 and now a no-op", not "gone entirely". Egress bullet could name CLAUDE_GATEWAY_PROXY_IS_EGRESS_BOUNDARY=1 (v2.1.277) as the shipped fix for the /etc/hosts hole. Opening "billing… does not auto-disable" should be qualified per the KB's own 2026-08-31 correction (Enterprise Spend Limits API / gateway limits / Managed Agents budgets exist, each scoped, gateway fails open by default).
  • checklist.md:46-48: auto mode is the default starting mode on every plan and provider since v2.1.283 (pending-KB, permission-modes.md read raw), not just Bedrock/Vertex/Foundry since v2.1.207 — and docs say auto mode's containment is "None".

5. Already fixed by open PR #26 — endorse, don't duplicate: all four substantive templates/ralph/run.sh findings (unwired default-0 cost meter behind an "enforced" comment; no per-call --max-turns/timeout; HEAD-based stall detection fooled by empty commits; the awk budget check failing open on a malformed meter value), the self-edit-guard.yml base-branch trigger hole (guard never ran on stacked PRs — the mode the skill itself prescribes), budget.env's missing CLAUDE_CODE_MAX_TURNS/per-call caps, and the loop-guardrails/new-loop hard-stop rewrites. One reviewer note for #26: pending-KB records the CLI cost meter wrong in both directions in-window (v2.1.286 under-count, v2.1.296 2× over-price, v2.1.302 cloud reset-to-zero) — #26's belt-and-braces design (external meter + per-call --max-budget-usd) is right; a gateway-side ceiling remains the stronger form.

6. CLAUDE.md (protected — human-only): "defaults to Haiku" → docs now say "your configured small fast model"; the v2.1.72/v2.1.139 floors (see flag 3); "Routines" rename at first mention.

7. Minor flags: templates/loop/loop.md's /loop 15m /code-review 1234 example may arrive as plain text in a scheduled fire if /code-review is disable-model-invocation: true (docs name only /verify — verify empirically); pending-KB: a backgrounded/cloud /loop that loses a wakeup post-v2.1.293 "stays asleep" (silent stop); /goal evaluator cost caveat (haiku alias → Haiku 5.5 long-prompt pricing, pending-KB); runbooks/overnight-build.md "Gas Town productized" overstates (superseded by Gas City; yield to #26); runbooks/first-loop.md's blanket "Esc (/loop)" has the same self-paced-only scoping as babysit-prs (yield to #26, not in its diff).

Clean (verified, no change needed)

README's "one rule" paragraph; templates/ralph/PROMPT.md; the ui-audit template set (minus the FINDINGS nit); /goal clear aliases, trust-dialog/hooks interaction, headless example; /loop mechanics (scoping, 7-day expiry, .claude/loop.md precedence); claude --cloud + deprecated --remote; --permission-prompts none (v2.1.259) and --restricted-is-not-a-sandbox bullets; the Tesla receipt; arXiv 2608.27299 / 2609.01222 / 2609.00267 citations; all internal cross-references and step numbers in all four skills.



Generated by Claude Code

…books, docs

Monthly artifact-audit pass (first since PR #24, merged 2026-09-29). All fixes
verified this pass against primary docs (code.claude.com goal.md, routines.md,
scheduled-tasks.md, cli-reference.md, claude-code-on-the-web.md) and/or
High-confidence knowledge-base entries; files mid-edited by open PRs #26-#29
were left untouched and their findings flagged in the PR body instead.

- templates/goal/recipes.md: the evaluator clears impossible goals and pauses
  no-progress runs (not "loops until you stop it"); exact 4,000-char condition
  limit; every recipe now carries an in-condition turn bound plus guidance to
  add external --max-turns/--max-budget-usd caps on headless runs; Ctrl+C
  scoped to non-interactive goals.
- templates/loop/loop.md: backgrounding carries /loop over; self-paced loops
  are not restored on --resume; scheduled-task rename noted.
- runbooks/staying-current.md: the claude/ branch prefix is a prompt-steered
  default, NOT an enforced push restriction - the real floor is a GitHub
  branch-protection ruleset on main (threat-model point 4, settings table and
  the "reconciled" note corrected accordingly); platform scheduling caps are
  hourly and waiting, not a daily rejecting cap; Routine runs start in auto
  mode and can stall on a waiting permission prompt (not "no approval
  prompts"); monthly cron now settable via the Custom edit dialog; rename
  and New task button noted.
- runbooks/babysit-prs.md: Esc only stops a self-paced loop; auto-fix enable
  paths corrected (Changes-card menu / /autofix-pr on the PR branch / mobile /
  paste a PR URL); added the replies-as-your-account + comment-triggered-
  automation warning and the no-merge-conflict-webhook caveat.
- .claude/skills/artifact-audit/SKILL.md: budget backstop description
  corrected (hourly waiting caps + subscription limit, not a daily-run cap).
- README.md: templates table now lists the ui-audit kit.
- templates/ui-audit/FINDINGS.md: removed stale self-referential rename note.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016net4GTtLwuYBtz584SKby
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants