Skip to content

fix(factory): enforce the prompt refuse-lists with --disallowedTools - #619

Open
JakubAnderwald wants to merge 1 commit into
mainfrom
fix/factory-tool-denies
Open

JakubAnderwald wants to merge 1 commit into
mainfrom
fix/factory-tool-denies

Conversation

@JakubAnderwald

Copy link
Copy Markdown
Owner

Why

Every factory prompt carries a prose "Refuse:" list — no gh pr merge, no gh workflow run, no fastlane, no claude/node scripts/... subprocess. Nothing enforced any of it. All five Claude invocations run --dangerously-skip-permissions with no tool flags at all, so the "allow-list" was markdown honoured by model compliance alone. A drifting run could merge a PR, dispatch a release, or reset its own retry budget with nothing in the way.

The intuitive fix doesn't work

Measured against the installed CLI, not assumed:

Flags Result
--dangerously-skip-permissions --allowedTools "Read" Bash ran anyway — ignored under bypass
--allowedTools "Read" (no bypass flag) Bash still ran — the flag grants, it doesn't restrict
--dangerously-skip-permissions --disallowedTools "Bash" Bash removed entirely
--dangerously-skip-permissions --disallowedTools "Bash(ls:*)" ls denied, echo ran

So enforcement is a deny-list, and it composes with the bypass flag. Four more probes shaped the design:

  • Multi-word prefixes match — Bash(git log:*) denies git log, git status still runs.
  • Compound commands match per clause — cd /tmp && ls trips Bash(ls:*).
  • Task subagents inherit the deny set — not bypassable by delegation, which matters most on the two ultracode stages that fan out.
  • :* does not span a colon-suffixed token — Bash(pnpm release:*) is inert against pnpm release:beta. The chokepoint has to be fastlane/xcodebuild/gradlew, invoked name-first.
  • A malformed pattern is silently ignored — Bash(ls:* (unclosed) let ls run with no error.

What changed

Deny sets built as plain comma-separated bash strings next to the effort block (never arrays — an empty one under set -u throws on the mini's bash 3.2), layered:

  • core (all stages) — release verbs, host control, shell escapes (bash -c), WebFetch/WebSearch, and each dangerous scripts/lib/*.mjs CLI in both relative and $REPO_ROOT-absolute form.
  • coding (--implement/--watch) — plus history-rewriting git verbs. Plain git push and pnpm stay allowed; they're how work lands and gets verified.
  • read-only (--plan/replan, In Test) — plus git, pnpm, node wholesale, and Edit. In Test also denies Write ("you post one comment; that is your entire write surface"); plan keeps it for /tmp/factory-replan-body.md.

The two highest-value patterns stop an agent running state-cli to reset its own retry budget, and factory-project to write its own board Status — the loop guard the pipeline depends on.

Because malformed patterns fail silently, each stage logs the set it passed and the tests validate every pattern's syntax.

Guardrails against over-tightening

The realistic failure mode is denying something a stage needs and finding out at 03:00. factory-deny-grounding.test.mjs carries an explicit UNSAFE table per stage — Bash(git push:*), Bash(gh pr:*), Bash(pnpm:*), Bash(curl:*), Task, Write — and fails if any appears in that stage's set.

Since git checkout/git reset are now blocked, git restore / git restore --staged were added to the implement and watch allow-lists. gh pr edit was added to the implement tools list — the revision-run instructions at line 179 already required it but the list omitted it, a pre-existing inconsistency.

Not a sandbox

Stated in the ADR because the headline invites the wrong reading. gh api can mutate with no method flag (gh api repos/x/y/issues -f title=z, gh api graphql -f query='mutation{…}'); argument-value and cardinality constraints are inexpressible; env/xargs indirection isn't covered. The controls that hold are architectural and predate this: bash validates the summary line, owns every board write and the merge.

Testing

845 tests pass (from 828 — 17 new), lint 0 errors, typecheck 4/4, format clean, bash -n clean.

Verified end-to-end with the real FACTORY_DENY_PLAN string, not a synthetic one:

echo hello        => RAN
git status --short => DENIED
node --version    => DENIED

One existing test needed updating: factory-agent-effort.test.mjs pinned --dangerously-skip-permissions --effort as adjacent strings, and the deny flag now sits between them. Rewritten to match per invocation line rather than by adjacency.

See ADR-0034.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lw3K4txt5h2GQmWvAAw4Ju

Every factory prompt carries a prose "Refuse:" list, and nothing enforced any
of it. All five Claude invocations run --dangerously-skip-permissions with no
tool flags at all, so the "allow-list" was markdown honoured by the model's
compliance alone — a drifting run could merge a PR, dispatch a release, or
reset its own retry budget with nothing in the way.

An allow-list does not work here. Measured against the installed CLI:
--allowedTools is ignored entirely under --dangerously-skip-permissions, and
merely additive without it; Bash ran in both cases. --disallowedTools is the
only lever that restricts, and it composes with the bypass flag.

Four further probes shaped the design: multi-word prefixes match
(Bash(git log:*) denies git log, not git status); compound commands are
matched per clause (cd x && ls trips Bash(ls:*)); Task subagents INHERIT the
deny set, so it cannot be bypassed by delegation; but `:*` does not span a
colon-suffixed token, so Bash(pnpm release:*) is inert against
`pnpm release:beta` — the chokepoint has to be fastlane/xcodebuild/gradlew,
which are invoked name-first. And a malformed pattern is silently ignored,
which is why each stage logs its deny set and the tests validate syntax.

Deny sets are layered: a shared core (release verbs, host control, shell
escapes, WebFetch/WebSearch, and each dangerous scripts/lib CLI in both
relative and absolute form), plus history-rewriting git verbs on the coding
stages, plus wholesale git/pnpm/node on the read-only stages. The two core
patterns that matter most stop an agent resetting its own retry budget
(state-cli) or writing its own board Status (factory-project) — the loop
guard the pipeline depends on.

Plain git push, pnpm, gh pr create/view/edit and the screenshot curl carve-out
stay allowed on the coding stages; a test pins those as UNSAFE-to-deny so a
future tightening cannot brick the pipeline overnight. Since git checkout and
git reset are now blocked, git restore / git restore --staged were added to
the implement and watch allow-lists, and gh pr edit was added to the implement
tools list — the revision-run instructions already required it.

This is defence-in-depth against drift, NOT a sandbox: gh api can mutate with
no method flag, and argument-value and cardinality constraints are
inexpressible. ADR-0034 says so explicitly.

See ADR-0034.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lw3K4txt5h2GQmWvAAw4Ju
@vercel

vercel Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
drafto Ready Ready Preview Sep 8, 2026 9:41pm UTC

@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 6548928d-95bf-4347-88b4-404517a2df00


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch was successfully deployed

1 active deployment
Preview — 9fc213b0 Deployed Sep 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant