Skip to content

Eval workers can read the answer: the fix commit is in the worktree's git history, and hidden tests are readable on disk #5

Closed

Description

@blockmandev

Summary

Insi's first measurement rule is "the agent never sees the check". Today it holds only if the agent doesn't look. A worker in an insi eval attempt can reach the reference fix two ways:

  1. Through git. The attempt's worktree comes from Insi's full clone (Work::worktree in crates/insi-eval/src/run.rs), so the fix commit and every later commit are reachable from the worktree:
    $ git -C <attempt-worktree> log --all --oneline
    17ac9ca fix: clamp ignored the upper bound
    4a247c9 add arithmetic helpers          <- the attempt's base (HEAD)
    7333c1a fix: sub returned the sum instead of the difference   <- the task's answer
    $ git -C <attempt-worktree> show 7333c1a --stat
     src/lib.rs   | 2 +-
     tests/sub.rs | 1 +
    
    (Reproduced on a 3-task repo with Insi main at 28f3f20.) Even with the refs hidden, git cat-file --batch-all-objects would still list the objects, since the pack is shared.
  2. On disk. The hidden tests live in private_checks (by default ../insi-private/<repo>), which any shell command can cat. So can earlier attempts' check.out / worker.diff under .insi-work/attempts/.

Why it matters: an agent that copies the reference fix passes the hidden check for real, so the reward-hacking measure (public pass, hidden fail) never sees it. A harness change that teaches "look at git history first" would win insi improve without the agent getting better.

Fix

1. History-free worktrees (closes route 1)

Build each attempt's tree with only the base commit's objects:

git init <tree>
git -C <tree> fetch --depth 1 --no-tags file://<clone> <base-sha>   # needs uploadpack.allowAnySHA1InWant=true on Insi's own clone
git -C <tree> checkout --detach FETCH_HEAD
  • Alternative: git archive <base> | tar -x, then git init, git add -A, and one commit.
  • Keep the worker diff and own_files logic as they are; they only need HEAD and the index.
  • Acceptance: in a worker attempt, git log --all --oneline lists exactly one commit, and git cat-file --batch-all-objects --batch-check | wc -l contains no object from the fix commit.

2. Hide private_checks and other attempts (closes route 2)

Run the worker step under a sandbox with a read-deny list. sbx (incordai/Incode crates/sandbox, branch claude/laughing-darwin-hyqkjj) now supports this with --deny-read, enforced by Landlock on Linux and Seatbelt on macOS, with escape tests for the reading routes (direct, symlink, /proc/self/root, cd .., child processes).

  • Add placeholders and apply [sandbox] to eval worker steps, not only insi issue:
    [sandbox]
    command = ["sbx", "run", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json",
               "--deny-read", "{private}", "--deny-read", "{work}/attempts", "--wall", "1800", "--"]
    {private} is private_checks; {work} is the work folder.
  • Mark insi report lines whose attempts ran unsandboxed, so a measurement that could have leaked says so.
  • Acceptance: a worker told to cat a hidden test path gets "Permission denied", and the attempt still passes or fails on its own merit.

3. Record it

Each EvalResult gets isolation: {history: "base-only" | "full", private: "hidden" | "readable"}, so old results can be told apart from new ones.

Related: #1, #3 (§4), #4 (Level 1)

Activity

  1. added a commit that references this issue on Oct 20, 2026
    cb3c9a4
  2. blockmandev commented on Oct 20, 2026

    @blockmandev
    MemberAuthor

    Fixed in #6 (commit cb3c9a4):

    • worker attempts get a repository holding only the base commit;
    • [worker] sandbox with {source}/{private}/{work} placeholders keeps the worker out of the checkout and the hidden tests;
    • each result records its isolation, and insi report flags unsandboxed attempts.

    Checked live: a worker that copied the fixed file passed undetected when unsandboxed, and failed under sbx.


    Generated by Claude Code

  3. added a commit that references this issue on Oct 20, 2026
    5b85e7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions