Skip to content

Guard: language/javascript blocks npm run dev, and the denial message sends the agent to fix the wrong thing #297

Description

@slowdini

Version: 0.9.1, dev @ 715b454
Severity: medium — ordinary verification is blocked, and the reported cause misleads

Both arms of a codebase-backed run against the shipped Weeknight fixture tried to
start the dev server to check their UI change. All four guard denials in the run
were this one intention.

1. The packaged profile has no way to start a dev server

language/javascript allows install, CI, test, build, lint, and typecheck. It
does not allow npm run dev. framework/nextjs does allow dev/build/start, but
it only activates when package.json declares a next dependency — so a plain
Vite + React project (which is what eval-magic init pins as the default
example) has no packaged path to running its own dev server.

Looking at a web feature in a browser is a normal part of verifying it, so an
eval measuring verification behavior against the default fixture blocks the most
natural verification step by default. The eval author's escape hatch — a
hand-written guard.allow_commands: ["npm run dev"] — is fine once you know, but
you only find out after a run has already been shaped by the denials.

Worth considering: a framework/vite profile detected from a vite dependency,
or moving the dev/start scripts into language/javascript, since npm run dev
is a lifecycle script name, not a Next.js-specific one.

2. The denial message diagnoses the wrong cause, and the agent follows it

Observed sequence in the without_skill arm, consecutive ordinals 36-38:

ordinal 36  npm run dev 2>&1 &
            → "command not allowed by eval guard policy"

ordinal 37  npm run dev > /tmp/dev-server.log 2>&1 &
            → "output redirection to a file — runs outside the eval sandbox.
               For temporary or scratch files, use <env>/tmp."

ordinal 38  npm run dev > <env>/tmp/dev-server.log 2>&1 &
            → "command not allowed by eval guard policy"

At ordinal 37 the containment check fires first and reports the redirect target,
which masks the command-policy denial that was already true at ordinal 36. The
message then offers a specific, actionable remedy — use the task-local tmp — and
the agent does exactly that at ordinal 38, only to be blocked again for the
reason that was there the whole time.

The guidance is individually correct and collectively misleading: it named a real
problem that was not the blocking one. Where both a containment check and the
command policy would deny a command, saying so in one verdict would save the
retry and stop the agent from burning turns on a fix that cannot work.

Note for the denial log

guard-denials.jsonl records reason and resolved_targets but not the command,
which is the right privacy default. ingest does surface the command text in its
console warnings, which is what made this diagnosable — that pairing works well
and is worth keeping.


Found during a pre-release shakedown of the #244 work: a four-skill treatment (hardening-plans, working-in-isolation, working-with-tdd, verifying-development-work) implementing a real feature in the Weeknight fixture, driven end to end through run → dispatch → ingest → compare → judges → finalize on claude-code with a haiku-4-5 agent and sonnet-5 judges.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions