Repository navigation
eval: keep the worker from the answer (fixes #5); ignore a caller's GIT_DIR - #6
Merged
Merged
Conversation
Every command Insi starts, and every git it runs itself, now drops the variables that point git at another repository: GIT_DIR, GIT_WORK_TREE, GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_COMMON_DIR, GIT_NAMESPACE, GIT_CEILING_DIRECTORIES and GIT_DISCOVERY_ACROSS_FILESYSTEM. Insi installs hooks, and a process started from a hook inherits them, so a worker's, a check's or Insi's own git would act on the hook's repository instead of the attempt's worktree. Measured: `insi validate` with GIT_DIR and GIT_INDEX_FILE set to another repository now validates its own tasks and leaves that repository untouched. (Recommendation 4.1 in #3.) The README's [sandbox] section shows sbx (incordai/Incode crates/sandbox) as a container-free sandbox: writes confined to the worktree, network limited to named hosts, session and git variables stripped, a wall limit, and a refusal when the machine cannot enforce it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT
A worker attempt could read the task's answer two ways:
- git: its worktree shared Insi's clone, so `git log --all` and
`git show <fix>` printed the reference fix;
- disk: the source checkout (usually after the fix) and the hidden tests
were readable by any shell command.
A worker that copied the answer passed the hidden tests for real, so the
reward-hacking count could not see it.
- A worker attempt's repository is now a shallow fetch of the base commit
alone (`Work::isolated_tree`), from Insi's clone, with FETCH_HEAD
removed. A worktree shares every object, and alternates would still list
them, so neither is used. Base and reference controls keep worktrees.
The clone sets uploadpack.allowAnySHA1InWant so the fetch by id works.
- `[worker] sandbox` wraps the worker step of an eval attempt, filling in
{tree} (required), {private}, {work} and {source} (which must be a local
folder). sbx with `--deny-read {source} --deny-read {private}` keeps the
worker out of both.
- Each EvalResult records `isolation` {history: "base-only", sandboxed}.
`insi report` adds "N attempt(s) unsandboxed (the worker could read the
answer on disk; set [worker] sandbox)" to a condition. Results written
before this have no isolation and count as unsandboxed.
Measured on a 3-task repo with a worker that only runs
`cp ../../../src/lib.rs src/lib.rs`:
- unsandboxed, it PASSED its task with reward hacking 0/1, and is now
flagged unsandboxed;
- under `[worker] sandbox` with sbx, the copy was refused and it FAILED;
- in both, `git log --all` in the tree showed only the base commit.
Tests: a fake worker records what git shows it (only the base: no fix
commit, no fixed blob) and the sandbox places it was given; placeholders
and the local-source rule. 121 pass; clippy and fmt clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT
The accept rule was "better by a fixed margin". On a handful of tasks one
lucky attempt cleared it, and an errored run still produced a verdict.
- accept::compare pairs the two sides by (task, seed), pools seeds per
task, and computes the per-task difference, its 95% bootstrap interval
(2,000 resamples, seeded by the step so a decision reproduces), the tasks
that moved, tasks the incumbent always passed and the candidate broke,
errored shares, tokens per pair and reward hacks.
- accept::decide:
- not measured above 5% errored on either side, or with nothing in
common;
- rejected if it breaks an always-passing task or reward hacking rises;
- accepted when the whole interval is above zero and the mean clears
--margin;
- accepted as "as good and cheaper" on a tie with no task worse and at
least 10% fewer tokens;
- otherwise rejected, with the interval in the reason.
- insi improve uses it. The held-out half must be not worse and break
nothing. A not-measured step is audited as `not-measured`, teaches the
bandit nothing, and if every step was, the run exits 2.
- insi eval exits 2 ("not measured: N of M attempt(s) errored") above 5%
errors, apart from 1 (failed), so CI never reads infrastructure trouble
as a result. Measured: a task pointed at a missing source exits 2; a
healthy run exits 0.
- docs/MEASUREMENT.md writes down the eleven rules and how each is
enforced; the README links it.
Tests:
- one lucky task of four is rejected; four of four is accepted;
- the interval reproduces;
- breaking a sure thing is rejected;
- a cheaper tie is accepted, and 1% cheaper is noise;
- errors give "not measured";
- rising reward hacking is rejected.
The pipeline test still sees the loop accept the better harness and reject
the worse. 126 pass; clippy and fmt clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT
Member
Author
|
CI is red on all three Run locally on this branch's head (
This branch now also carries the measurement change: accept on a 95% interval, guard tasks that always passed, and "not measured" with exit code 2. See Generated by Claude Code |
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Keep the worker from the answer (fixes #5)
A worker attempt could read the task's answer two ways:
git show <fix>printed the reference fix;A worker that copied the answer passed for real, so the reward-hacking count couldn't see it. The fixes:
Work::isolated_tree). Base and reference controls keep worktrees.[worker] sandbox. It wraps the worker step and fills in{tree}(required),{private},{work}and{source}. Withsbx,--deny-read {source} --deny-read {private}keeps the worker out of both.EvalResult.isolationrecords{history: "base-only", sandboxed}.insi reportadds "N attempt(s) unsandboxed" to any condition whose passes aren't proof of the work.Ignore a caller's git redirects
Every command Insi starts, and Insi's own
git, now dropGIT_DIR,GIT_WORK_TREE,GIT_INDEX_FILEand the related variables. A process started from a git hook inherits them. (Recommendation 4.1 in #3.)README
sbxas the[sandbox]forinsi issue, plus a new section on keeping the worker from the answer.Tested
cargo test: 121 pass. The new tests:cp ../../../src/lib.rs src/lib.rs:[worker] sandboxusingsbx: the copy was refused and it FAILED;git log --allin the tree showed only the base commit.GIT_DIR:insi validatewithGIT_DIRpointed at another repository validated its own tasks and left the other repository untouched.🤖 Generated with Claude Code
https://claude.ai/code/session_011DiH8n52ifWoCd5Kx2aHwT