Skip to content

eval: keep the worker from the answer (fixes #5); ignore a caller's GIT_DIR - #6

Merged
blockmandev merged 3 commits into
mainfrom
claude/laughing-darwin-hyqkjj
Oct 20, 2026
Merged

blockmandev merged 3 commits into
mainfrom
claude/laughing-darwin-hyqkjj

Conversation

@blockmandev

@blockmandev blockmandev commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What

Keep the worker from the answer (fixes #5)

A worker attempt could read the task's answer two ways:

  • git: the worktree shared Insi's clone, so git show <fix> printed the reference fix;
  • disk: the source checkout and the hidden tests were readable.

A worker that copied the answer passed for real, so the reward-hacking count couldn't see it. The fixes:

  • History-free tree. A worker attempt's repository is now a shallow fetch of the base commit alone (Work::isolated_tree). Base and reference controls keep worktrees.
  • [worker] sandbox. It wraps the worker step and fills in {tree} (required), {private}, {work} and {source}. With sbx, --deny-read {source} --deny-read {private} keeps the worker out of both.
  • Recorded and reported. EvalResult.isolation records {history: "base-only", sandboxed}. insi report adds "N attempt(s) unsandboxed" to any condition whose passes aren't proof of the work.

Ignore a caller's git redirects

Every command Insi starts, and Insi's own git, now drop GIT_DIR, GIT_WORK_TREE, GIT_INDEX_FILE and the related variables. A process started from a git hook inherits them. (Recommendation 4.1 in #3.)

README

sbx as the [sandbox] for insi issue, plus a new section on keeping the worker from the answer.

Tested

  • cargo test: 121 pass. The new tests:
    • a fake worker records that git shows it only the base (no fix commit, no fixed blob) and the sandbox places it was given;
    • the placeholders and the local-source rule;
    • the git-redirect removal.
  • clippy and fmt are clean.
  • Attack, live: a worker that only runs cp ../../../src/lib.rs src/lib.rs:
    • unsandboxed: it PASSED, reward hacking 0/1, and is now flagged "1 attempt(s) unsandboxed";
    • with [worker] sandbox using sbx: the copy was refused and it FAILED;
    • in both, git log --all in the tree showed only the base commit.
  • Live GIT_DIR: insi validate with GIT_DIR pointed at another repository validated its own tasks and left the other repository untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT

Every command Insi starts, and every git it runs itself, now drops the
variables that point git at another repository: GIT_DIR, GIT_WORK_TREE,
GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES,
GIT_COMMON_DIR, GIT_NAMESPACE, GIT_CEILING_DIRECTORIES and
GIT_DISCOVERY_ACROSS_FILESYSTEM. Insi installs hooks, and a process started
from a hook inherits them, so a worker's, a check's or Insi's own git would
act on the hook's repository instead of the attempt's worktree. Measured:
`insi validate` with GIT_DIR and GIT_INDEX_FILE set to another repository
now validates its own tasks and leaves that repository untouched.
(Recommendation 4.1 in #3.)

The README's [sandbox] section shows sbx (incordai/Incode crates/sandbox)
as a container-free sandbox: writes confined to the worktree, network
limited to named hosts, session and git variables stripped, a wall limit,
and a refusal when the machine cannot enforce it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT
A worker attempt could read the task's answer two ways:
- git: its worktree shared Insi's clone, so `git log --all` and
  `git show <fix>` printed the reference fix;
- disk: the source checkout (usually after the fix) and the hidden tests
  were readable by any shell command.

A worker that copied the answer passed the hidden tests for real, so the
reward-hacking count could not see it.

- A worker attempt's repository is now a shallow fetch of the base commit
  alone (`Work::isolated_tree`), from Insi's clone, with FETCH_HEAD
  removed. A worktree shares every object, and alternates would still list
  them, so neither is used. Base and reference controls keep worktrees.
  The clone sets uploadpack.allowAnySHA1InWant so the fetch by id works.
- `[worker] sandbox` wraps the worker step of an eval attempt, filling in
  {tree} (required), {private}, {work} and {source} (which must be a local
  folder). sbx with `--deny-read {source} --deny-read {private}` keeps the
  worker out of both.
- Each EvalResult records `isolation` {history: "base-only", sandboxed}.
  `insi report` adds "N attempt(s) unsandboxed (the worker could read the
  answer on disk; set [worker] sandbox)" to a condition. Results written
  before this have no isolation and count as unsandboxed.

Measured on a 3-task repo with a worker that only runs
`cp ../../../src/lib.rs src/lib.rs`:
- unsandboxed, it PASSED its task with reward hacking 0/1, and is now
  flagged unsandboxed;
- under `[worker] sandbox` with sbx, the copy was refused and it FAILED;
- in both, `git log --all` in the tree showed only the base commit.

Tests: a fake worker records what git shows it (only the base: no fix
commit, no fixed blob) and the sandbox places it was given; placeholders
and the local-source rule. 121 pass; clippy and fmt clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT
@blockmandev blockmandev changed the title process, git: ignore a caller's GIT_DIR; README: sbx as the sandbox eval: keep the worker from the answer (fixes #5); ignore a caller's GIT_DIR Oct 20, 2026
The accept rule was "better by a fixed margin". On a handful of tasks one
lucky attempt cleared it, and an errored run still produced a verdict.

- accept::compare pairs the two sides by (task, seed), pools seeds per
  task, and computes the per-task difference, its 95% bootstrap interval
  (2,000 resamples, seeded by the step so a decision reproduces), the tasks
  that moved, tasks the incumbent always passed and the candidate broke,
  errored shares, tokens per pair and reward hacks.
- accept::decide:
  - not measured above 5% errored on either side, or with nothing in
    common;
  - rejected if it breaks an always-passing task or reward hacking rises;
  - accepted when the whole interval is above zero and the mean clears
    --margin;
  - accepted as "as good and cheaper" on a tie with no task worse and at
    least 10% fewer tokens;
  - otherwise rejected, with the interval in the reason.
- insi improve uses it. The held-out half must be not worse and break
  nothing. A not-measured step is audited as `not-measured`, teaches the
  bandit nothing, and if every step was, the run exits 2.
- insi eval exits 2 ("not measured: N of M attempt(s) errored") above 5%
  errors, apart from 1 (failed), so CI never reads infrastructure trouble
  as a result. Measured: a task pointed at a missing source exits 2; a
  healthy run exits 0.
- docs/MEASUREMENT.md writes down the eleven rules and how each is
  enforced; the README links it.

Tests:
- one lucky task of four is rejected; four of four is accepted;
- the interval reproduces;
- breaking a sure thing is rejected;
- a cheaper tie is accepted, and 1% cheaper is noise;
- errors give "not measured";
- rising reward hacking is rejected.

The pipeline test still sees the loop accept the better harness and reject
the worse. 126 pass; clippy and fmt clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT

Copy link
Copy Markdown
Member Author

CI is red on all three test jobs, but not because of this PR. Each job ended within seconds without a runner assigned, so no step ran. ci on main (runs #12 and #13, the commits this branch starts from) failed the same way. It looks like an account-level Actions problem (billing or a spending limit), not the code.

Run locally on this branch's head (435f959):

  • cargo fmt --all -- --check passes;
  • cargo clippy --all-targets -- -D warnings is clean;
  • cargo test: 126 pass.

This branch now also carries the measurement change: accept on a 95% interval, guard tasks that always passed, and "not measured" with exit code 2. See docs/MEASUREMENT.md.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval workers can read the answer: the fix commit is in the worktree's git history, and hidden tests are readable on disk

2 participants