Version: 0.9.1, dev @ 715b454
Severity: medium — ordinary verification is blocked, and the reported cause misleads
Both arms of a codebase-backed run against the shipped Weeknight fixture tried to
start the dev server to check their UI change. All four guard denials in the run
were this one intention.
1. The packaged profile has no way to start a dev server
language/javascript allows install, CI, test, build, lint, and typecheck. It
does not allow npm run dev. framework/nextjs does allow dev/build/start, but
it only activates when package.json declares a next dependency — so a plain
Vite + React project (which is what eval-magic init pins as the default
example) has no packaged path to running its own dev server.
Looking at a web feature in a browser is a normal part of verifying it, so an
eval measuring verification behavior against the default fixture blocks the most
natural verification step by default. The eval author's escape hatch — a
hand-written guard.allow_commands: ["npm run dev"] — is fine once you know, but
you only find out after a run has already been shaped by the denials.
Worth considering: a framework/vite profile detected from a vite dependency,
or moving the dev/start scripts into language/javascript, since npm run dev
is a lifecycle script name, not a Next.js-specific one.
2. The denial message diagnoses the wrong cause, and the agent follows it
Observed sequence in the without_skill arm, consecutive ordinals 36-38:
ordinal 36 npm run dev 2>&1 &
→ "command not allowed by eval guard policy"
ordinal 37 npm run dev > /tmp/dev-server.log 2>&1 &
→ "output redirection to a file — runs outside the eval sandbox.
For temporary or scratch files, use <env>/tmp."
ordinal 38 npm run dev > <env>/tmp/dev-server.log 2>&1 &
→ "command not allowed by eval guard policy"
At ordinal 37 the containment check fires first and reports the redirect target,
which masks the command-policy denial that was already true at ordinal 36. The
message then offers a specific, actionable remedy — use the task-local tmp — and
the agent does exactly that at ordinal 38, only to be blocked again for the
reason that was there the whole time.
The guidance is individually correct and collectively misleading: it named a real
problem that was not the blocking one. Where both a containment check and the
command policy would deny a command, saying so in one verdict would save the
retry and stop the agent from burning turns on a fix that cannot work.
Note for the denial log
guard-denials.jsonl records reason and resolved_targets but not the command,
which is the right privacy default. ingest does surface the command text in its
console warnings, which is what made this diagnosable — that pairing works well
and is worth keeping.
Found during a pre-release shakedown of the #244 work: a four-skill treatment (hardening-plans, working-in-isolation, working-with-tdd, verifying-development-work) implementing a real feature in the Weeknight fixture, driven end to end through run → dispatch → ingest → compare → judges → finalize on claude-code with a haiku-4-5 agent and sonnet-5 judges.
Version: 0.9.1,
dev@715b454Severity: medium — ordinary verification is blocked, and the reported cause misleads
Both arms of a codebase-backed run against the shipped Weeknight fixture tried to
start the dev server to check their UI change. All four guard denials in the run
were this one intention.
1. The packaged profile has no way to start a dev server
language/javascriptallows install, CI, test, build, lint, and typecheck. Itdoes not allow
npm run dev.framework/nextjsdoes allow dev/build/start, butit only activates when
package.jsondeclares anextdependency — so a plainVite + React project (which is what
eval-magic initpins as the defaultexample) has no packaged path to running its own dev server.
Looking at a web feature in a browser is a normal part of verifying it, so an
eval measuring verification behavior against the default fixture blocks the most
natural verification step by default. The eval author's escape hatch — a
hand-written
guard.allow_commands: ["npm run dev"]— is fine once you know, butyou only find out after a run has already been shaped by the denials.
Worth considering: a
framework/viteprofile detected from avitedependency,or moving the dev/start scripts into
language/javascript, sincenpm run devis a lifecycle script name, not a Next.js-specific one.
2. The denial message diagnoses the wrong cause, and the agent follows it
Observed sequence in the
without_skillarm, consecutive ordinals 36-38:At ordinal 37 the containment check fires first and reports the redirect target,
which masks the command-policy denial that was already true at ordinal 36. The
message then offers a specific, actionable remedy — use the task-local tmp — and
the agent does exactly that at ordinal 38, only to be blocked again for the
reason that was there the whole time.
The guidance is individually correct and collectively misleading: it named a real
problem that was not the blocking one. Where both a containment check and the
command policy would deny a command, saying so in one verdict would save the
retry and stop the agent from burning turns on a fix that cannot work.
Note for the denial log
guard-denials.jsonlrecordsreasonandresolved_targetsbut not the command,which is the right privacy default.
ingestdoes surface the command text in itsconsole warnings, which is what made this diagnosable — that pairing works well
and is worth keeping.
Found during a pre-release shakedown of the #244 work: a four-skill treatment (
hardening-plans,working-in-isolation,working-with-tdd,verifying-development-work) implementing a real feature in the Weeknight fixture, driven end to end through run → dispatch → ingest → compare → judges → finalize onclaude-codewith a haiku-4-5 agent and sonnet-5 judges.