Problem
The benchmark collector can accept a run whose agent reads coordinator files outside the prompt's allowed paths. Device isolation and Settings proof can pass despite that scope violation.
Evidence
During the Android campaign tracked by #450, manual review found Opus native control ran cat <campaign-root>/env.sh and cat <campaign-root>/pins.env. The prompt permits the fixture, run worktree, current run evidence directory, and explicit proof scratch paths, not the coordinator root. The automatic audit accepted the run, but it is being excluded manually before publication. No credential contents were read in the observed command.
Cause
The collector audits device sessions, setup commands, tool provenance, cache evidence, and proof. It does not enforce the prompt's coordinator-file read boundary.
Fix idea
Add auditable detection for actual shell/file-tool reads of coordinator config, other runs, and golden state. Preserve necessary SDK/skill reads, quoted searches, and current-run proof paths; do not use an indiscriminate substring scan. Consider an explicit persisted manual-audit override so a later re-collection cannot erase a reviewed exclusion. Add regressions from the observed command and legitimate neighboring paths.
Problem
The benchmark collector can accept a run whose agent reads coordinator files outside the prompt's allowed paths. Device isolation and Settings proof can pass despite that scope violation.
Evidence
During the Android campaign tracked by #450, manual review found Opus native control ran
cat <campaign-root>/env.shandcat <campaign-root>/pins.env. The prompt permits the fixture, run worktree, current run evidence directory, and explicit proof scratch paths, not the coordinator root. The automatic audit accepted the run, but it is being excluded manually before publication. No credential contents were read in the observed command.Cause
The collector audits device sessions, setup commands, tool provenance, cache evidence, and proof. It does not enforce the prompt's coordinator-file read boundary.
Fix idea
Add auditable detection for actual shell/file-tool reads of coordinator config, other runs, and golden state. Preserve necessary SDK/skill reads, quoted searches, and current-run proof paths; do not use an indiscriminate substring scan. Consider an explicit persisted manual-audit override so a later re-collection cannot erase a reviewed exclusion. Add regressions from the observed command and legitimate neighboring paths.