You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Eval workers can read the answer: the fix commit is in the worktree's git history, and hidden tests are readable on disk #5
Insi's first measurement rule is "the agent never sees the check". Today it holds only if the agent doesn't look. A worker in an insi eval attempt can reach the reference fix two ways:
Through git. The attempt's worktree comes from Insi's full clone (Work::worktree in crates/insi-eval/src/run.rs), so the fix commit and every later commit are reachable from the worktree:
$ git -C <attempt-worktree> log --all --oneline
17ac9ca fix: clamp ignored the upper bound
4a247c9 add arithmetic helpers <- the attempt's base (HEAD)
7333c1a fix: sub returned the sum instead of the difference <- the task's answer
$ git -C <attempt-worktree> show 7333c1a --stat
src/lib.rs | 2 +-
tests/sub.rs | 1 +
(Reproduced on a 3-task repo with Insi main at 28f3f20.) Even with the refs hidden, git cat-file --batch-all-objects would still list the objects, since the pack is shared.
On disk. The hidden tests live in private_checks (by default ../insi-private/<repo>), which any shell command can cat. So can earlier attempts' check.out / worker.diff under .insi-work/attempts/.
Why it matters: an agent that copies the reference fix passes the hidden check for real, so the reward-hacking measure (public pass, hidden fail) never sees it. A harness change that teaches "look at git history first" would win insi improve without the agent getting better.
Fix
1. History-free worktrees (closes route 1)
Build each attempt's tree with only the base commit's objects:
Alternative:git archive <base> | tar -x, then git init, git add -A, and one commit.
Keep the worker diff and own_files logic as they are; they only need HEAD and the index.
Acceptance: in a worker attempt, git log --all --oneline lists exactly one commit, and git cat-file --batch-all-objects --batch-check | wc -l contains no object from the fix commit.
2. Hide private_checks and other attempts (closes route 2)
Run the worker step under a sandbox with a read-deny list. sbx (incordai/Incode crates/sandbox, branch claude/laughing-darwin-hyqkjj) now supports this with --deny-read, enforced by Landlock on Linux and Seatbelt on macOS, with escape tests for the reading routes (direct, symlink, /proc/self/root, cd .., child processes).
Add placeholders and apply [sandbox] to eval worker steps, not only insi issue:
Activity
eval: keep the worker from the answer (fixes #5)
blockmandev commentedon Oct 20, 2026
Fixed in #6 (commit cb3c9a4):
[worker] sandboxwith{source}/{private}/{work}placeholders keeps the worker out of the checkout and the hidden tests;insi reportflags unsandboxed attempts.Checked live: a worker that copied the fixed file passed undetected when unsandboxed, and failed under
sbx.Generated by Claude Code
Merge pull request #6 from incordai/claude/laughing-darwin-hyqkjj