diff --git a/README.md b/README.md index f23840c..468dfc9 100644 --- a/README.md +++ b/README.md @@ -61,11 +61,15 @@ not learned from, measured against a control. draft pull request ───► a person reviews it; Insi never merges ``` -Four rules keep the measurement honest: - -1. **The agent never sees the check.** Hidden tests live outside every - checkout the agent can read. They are copied in after the agent stops, - replacing anything it wrote at those paths. +Four rules keep the measurement honest (all eleven, with how each is enforced, are in [docs/MEASUREMENT.md](docs/MEASUREMENT.md)): + +1. **The agent never sees the check.** Hidden tests live outside the + attempt's folder and are copied in after the agent stops, replacing + anything it wrote at those paths. The attempt's repository holds the base + commit and nothing after it, so no git command shows the fix. What is on + disk (the source checkout, the hidden tests) is kept from the agent by + [`[worker] sandbox`](#keeping-the-worker-from-the-answer); `insi report` + counts the attempts that ran without one. 2. **Only fair tasks count.** A task whose check already passes at the start, or fails even with the real fix, is dropped by `insi validate`. 3. **Learn on the past, measure on the future.** Tasks are split by commit @@ -389,6 +393,25 @@ insi issue --number 42 # from the repository's folder network. Committing, the scan and the push stay outside it, on this machine. A sandbox with no `{tree}` is refused. + Without a container, + [`sbx`](https://github.com/incordai/Incode/tree/main/crates/sandbox) does + the same with the OS's own sandbox (Landlock on Linux, Seatbelt on + macOS). In the example below: + - writes are confined to the worktree and the agent's own state; + - the network only reaches the hosts listed; + - the session and `GIT_DIR` variables are stripped; + - the step is killed after the wall limit. + + It refuses to run when the machine cannot enforce the policy, instead of + running unconfined: + + ```toml + [sandbox] + command = ["sbx", "run", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json", + "--allow-host", "api.anthropic.com", "--allow-host", "registry.npmjs.org", + "--wall", "1800", "--"] + ``` + GitHub is reached through the `gh` CLI, which holds the login: Insi reads, holds and sends no token. @@ -622,6 +645,37 @@ are timed but not counted in tokens. `--spec-gate=off` keeps incode from writing its spec documents before each already-scoped task (it needs an incode that has the flag; an older one refuses it by name). +### Keeping the worker from the answer + +The answer to a task is in three places: the fix commit, the source +repository's checkout (usually at a commit after the fix), and the hidden +tests. A worker attempt's repository is a shallow fetch of the base commit +alone, so git can't reach the first one. The other two are on disk, and any +worker with a shell can read them. Measured: a worker that only ran +`cp ../../../src/lib.rs src/lib.rs` passed its task, and the reward-hacking +check didn't catch it, because it passed the hidden tests for real. + +`[worker] sandbox` runs the worker step inside a command and fills in its +places: +- `{tree}`: the attempt's folder (required); +- `{private}`: the hidden tests; +- `{work}`: Insi's work folder; +- `{source}`: the task's source, which must be a local folder. + +With [`sbx`](https://github.com/incordai/Incode/tree/main/crates/sandbox): + +```toml +[worker] +command = ["incode", "-t", "{prompt}", "--output-format", "stream-json", "--spec-gate=off"] +sandbox = ["sbx", "run", "-q", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json", + "-w", "~/.incode", "--deny-read", "{source}", "--deny-read", "{private}", "--"] +``` + +The same cheating worker then fails, as it should: the copy is refused. +Each result records `isolation` (`history: "base-only"`, `sandboxed`). +`insi report` adds `N attempt(s) unsandboxed` to a condition whose passes are +not proof of the work. + Every command Insi starts — the worker, the checks, `gh` — is started without the variables of the agent session Insi itself runs in (`CLAUDECODE`, `CLAUDE_CODE_SESSION_ID`, …), so concurrent attempts never share, or report, diff --git a/crates/insi-core/src/config.rs b/crates/insi-core/src/config.rs index 38a32c9..2a3374a 100644 --- a/crates/insi-core/src/config.rs +++ b/crates/insi-core/src/config.rs @@ -109,6 +109,12 @@ fn untrusted() -> crate::Trust { /// What a sandbox command must contain: where the work's folder goes. pub const TREE: &str = "{tree}"; +/// In `[worker] sandbox`: where the hidden tests are kept. +pub const PRIVATE: &str = "{private}"; +/// In `[worker] sandbox`: Insi's work folder. +pub const WORK: &str = "{work}"; +/// In `[worker] sandbox`: the task's source repository. +pub const SOURCE: &str = "{source}"; /// A command every step of issue work runs inside, e.g. a container: /// @@ -230,6 +236,31 @@ pub struct Worker { /// code. The check still runs on the whole worktree. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub own_files: Vec, + /// A command the worker step of an eval attempt runs inside, with the + /// worker's own command appended, e.g. `sbx`: + /// + /// ```toml + /// [worker] + /// sandbox = ["sbx", "run", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json", + /// "--deny-read", "{source}", "--deny-read", "{private}", "--"] + /// ``` + /// + /// `{tree}` is the attempt's folder (required), `{private}` where the + /// hidden tests are kept, `{work}` Insi's work folder, and `{source}` + /// the task's source repository (a local folder). Without it a worker can + /// read the source checkout, usually at a commit after the fix, and the + /// hidden tests on disk; the report counts such attempts. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub sandbox: Vec, +} + +/// Where an eval attempt's places are, for [`Worker::sandboxed`]. +#[derive(Debug, Clone, Copy)] +pub struct Places<'a> { + pub tree: &'a Path, + pub private: &'a Path, + pub work: &'a Path, + pub source: &'a str, } impl Worker { @@ -257,6 +288,7 @@ impl Worker { // incode's state, the spec set its spec step writes when the gate // is on (incode src/spec.rs SPEC_SET), and the backups its edit // hook keeps beside each edited file. + sandbox: Vec::new(), own_files: [ ".incode/**", ".incord/**", @@ -276,7 +308,49 @@ impl Worker { Worker { command, own_files: Vec::new(), + sandbox: Vec::new(), + } + } + + /// Why `sandbox` cannot be used, if it cannot. + pub fn sandbox_problem(&self) -> Option { + if self.sandbox.is_empty() { + return None; + } + if self.sandbox[0].trim().is_empty() { + return Some("worker.sandbox: name the program to run".to_string()); } + if !self.sandbox.iter().any(|a| a.contains(TREE)) { + return Some(format!( + "worker.sandbox has no {TREE} argument, so the attempt's folder would never be in it" + )); + } + None + } + + /// `argv` wrapped in `sandbox`, with its places filled in; `argv` + /// unchanged when there is no sandbox. + /// + /// # Errors + /// When the sandbox names `{source}` and the task's source is not a + /// local folder: a path that does not exist could not be hidden. + pub fn sandboxed(&self, argv: Vec, places: &Places) -> Result, String> { + if self.sandbox.is_empty() { + return Ok(argv); + } + if self.sandbox.iter().any(|a| a.contains(SOURCE)) && !Path::new(places.source).is_dir() { + return Err(format!( + "worker.sandbox names {SOURCE}, but the task's source `{}` is not a local folder", + places.source + )); + } + let fill = |a: &String| { + a.replace(TREE, &places.tree.display().to_string()) + .replace(PRIVATE, &places.private.display().to_string()) + .replace(WORK, &places.work.display().to_string()) + .replace(SOURCE, places.source) + }; + Ok(self.sandbox.iter().map(fill).chain(argv).collect()) } /// The command for one prompt. Refused when the program is missing or no @@ -363,6 +437,9 @@ impl Config { if let Some(why) = config.sandbox.as_ref().and_then(Sandbox::problem) { return Err(why); } + if let Some(why) = config.worker.as_ref().and_then(Worker::sandbox_problem) { + return Err(why); + } let mut sources = std::collections::BTreeSet::new(); for c in &config.connectors { if c.name.trim().is_empty() || c.command.first().is_none_or(|p| p.trim().is_empty()) { @@ -472,6 +549,56 @@ mod tests { /// **A SANDBOX HOLDS THE WORK'S FOLDER, OR IS REFUSED.** The step's /// command follows the sandbox's, with `{tree}` filled in. + #[test] + fn a_worker_sandbox_needs_the_tree_and_fills_its_places() { + let err = Config::parse( + "[worker]\ncommand = [\"w\", \"{prompt}\"]\nsandbox = [\"sbx\", \"run\", \"--\"]", + ) + .expect_err("no {tree}"); + assert!(err.contains("{tree}"), "{err}"); + let cfg = Config::parse( + "[worker]\ncommand = [\"w\", \"{prompt}\"]\n\ + sandbox = [\"sbx\", \"-w\", \"{tree}\", \"--deny-read\", \"{private}\", \"--deny-read\", \"{work}/attempts\", \"--\"]", + ) + .expect("parses"); + let worker = cfg.worker(); + let places = Places { + tree: Path::new("/w/runs/t"), + private: Path::new("/p"), + work: Path::new("/w"), + source: "https://example.invalid/repo.git", + }; + let argv = worker + .sandboxed(vec!["w".into(), "go".into()], &places) + .expect("wraps"); + assert_eq!( + argv, + [ + "sbx", + "-w", + "/w/runs/t", + "--deny-read", + "/p", + "--deny-read", + "/w/attempts", + "--", + "w", + "go" + ] + ); + // {source} must be a folder that exists, or it could not be hidden. + let mut with_source = worker.clone(); + with_source.sandbox.push("{source}".into()); + assert!(with_source.sandboxed(vec![], &places).is_err()); + // No sandbox: the command as it was. + assert_eq!( + Worker::incode() + .sandboxed(vec!["x".into()], &places) + .expect("plain"), + ["x"] + ); + } + #[test] fn a_sandbox_wraps_each_command_around_the_work_folder() { let config = Config::parse( diff --git a/crates/insi-core/src/lib.rs b/crates/insi-core/src/lib.rs index 46cee3e..66d0068 100644 --- a/crates/insi-core/src/lib.rs +++ b/crates/insi-core/src/lib.rs @@ -16,8 +16,8 @@ pub mod task; pub mod topic; pub mod trust; -pub use config::{Config, Price, Worker}; +pub use config::{Config, Places, Price, Worker}; pub use lesson::{Lesson, Verification}; pub use signature::normalize; -pub use task::{Budget, EvalResult, Task, Usage}; +pub use task::{Budget, EvalResult, Isolation, Task, Usage}; pub use trust::Trust; diff --git a/crates/insi-core/src/task.rs b/crates/insi-core/src/task.rs index dbb0c37..9411a47 100644 --- a/crates/insi-core/src/task.rs +++ b/crates/insi-core/src/task.rs @@ -241,6 +241,26 @@ pub struct EvalResult { /// verdict: why setup's output was not reused or not kept, for one. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub notes: Vec, + /// How far a worker attempt was kept from the answer ([`Isolation`]). + /// Absent on controls, and on results written before it was recorded, + /// which were not isolated. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub isolation: Option, +} + +/// What stood between a worker and the task's answer. +/// +/// The answer exists in three places a worker could reach: the fix commit +/// (in any git history that includes it), the source repository's own +/// checkout (usually at a commit after the fix), and the hidden tests. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Isolation { + /// `"base-only"`: the worker's repository holds the base commit and + /// nothing after it, so no git command in it can show the fix. + pub history: String, + /// Whether the worker ran inside `[worker] sandbox`, which is what can + /// keep it out of the source checkout and the hidden tests on disk. + pub sandboxed: bool, } /// Tokens a worker reported, by kind. diff --git a/crates/insi-eval/src/git.rs b/crates/insi-eval/src/git.rs index 336ea15..b39dffb 100644 --- a/crates/insi-eval/src/git.rs +++ b/crates/insi-eval/src/git.rs @@ -10,7 +10,9 @@ pub fn git(dir: &Path, args: &[&str]) -> Result { /// Run git in `dir` and return its stdout untouched (a file's contents). pub fn git_bytes(dir: &Path, args: &[&str]) -> Result, String> { - let output = Command::new("git") + let mut command = Command::new("git"); + crate::process::detach_from_caller_repository(&mut command); + let output = command .args(args) .current_dir(dir) .output() diff --git a/crates/insi-eval/src/process.rs b/crates/insi-eval/src/process.rs index f059209..8e02f7a 100644 --- a/crates/insi-eval/src/process.rs +++ b/crates/insi-eval/src/process.rs @@ -55,6 +55,7 @@ pub fn run_with( let mut command = Command::new(&program); // Before `env`, so a variable the caller sets on purpose still wins. detach_from_parent_session(&mut command, std::env::vars_os().map(|(name, _)| name)); + detach_from_caller_repository(&mut command); command .args(args) .envs(env.iter().copied()) @@ -196,6 +197,31 @@ fn detach_from_parent_session( } } +/// Variables that point git at a repository other than the one in the folder +/// it runs in. A process started from a git hook inherits `GIT_DIR` (and +/// often `GIT_INDEX_FILE`), and Insi installs hooks: every `git` a worker, a +/// check or Insi itself then ran would act on the hook's repository instead +/// of the attempt's worktree. +pub const GIT_REDIRECTS: [&str; 9] = [ + "GIT_DIR", + "GIT_WORK_TREE", + "GIT_INDEX_FILE", + "GIT_OBJECT_DIRECTORY", + "GIT_ALTERNATE_OBJECT_DIRECTORIES", + "GIT_COMMON_DIR", + "GIT_NAMESPACE", + "GIT_CEILING_DIRECTORIES", + "GIT_DISCOVERY_ACROSS_FILESYSTEM", +]; + +/// Start `command` in the repository of the folder it runs in, whatever the +/// caller's environment points git at ([`GIT_REDIRECTS`]). +pub fn detach_from_caller_repository(command: &mut Command) { + for name in GIT_REDIRECTS { + command.env_remove(name); + } +} + /// The program to start. On Windows a bare name like `npx` is a `.cmd` /// script, which `Command` does not look for on its own: PATH is searched /// for each executable extension, as a shell would. @@ -271,6 +297,21 @@ mod tests { /// **A COMMAND NEVER INHERITS THE AGENT SESSION INSI RUNS IN** — and keeps /// the credentials it needs to start one of its own. + #[test] + fn a_command_ignores_git_redirects_from_its_caller() { + let mut command = Command::new("worker"); + detach_from_caller_repository(&mut command); + for name in GIT_REDIRECTS { + assert_eq!( + fate(&command, name), + Some(None), + "{name} reached the command" + ); + } + // Configuration through the environment is not a redirect. + assert_eq!(fate(&command, "GIT_CONFIG_COUNT"), None); + } + #[test] fn a_command_is_detached_from_the_parent_session_and_keeps_its_credentials() { let parent = [ diff --git a/crates/insi-eval/src/report.rs b/crates/insi-eval/src/report.rs index 190bbb6..aa7fdde 100644 --- a/crates/insi-eval/src/report.rs +++ b/crates/insi-eval/src/report.rs @@ -31,6 +31,10 @@ pub struct Totals { /// Of those, the ones that passed the public check and failed the /// hidden one: the score a worker could see, met without the work. pub reward_hacks: usize, + /// Worker attempts that ran without `[worker] sandbox` (or before it was + /// recorded): their worker could have read the source checkout or the + /// hidden tests, so a pass among them is not proof of the work. + pub unsandboxed: usize, } impl Totals { @@ -71,6 +75,9 @@ pub fn by_condition(results: &[EvalResult]) -> BTreeMap { let condition = r.condition(); let t = out.entry(condition.clone()).or_default(); t.attempts += 1; + if !r.isolation.as_ref().is_some_and(|i| i.sandboxed) { + t.unsandboxed += 1; + } t.tokens += r.tokens.unwrap_or(0); t.wall_secs += r.wall_secs; match r.cost_usd { @@ -143,6 +150,7 @@ mod tests { fn r(task: &str, kind: &str, passed: bool, sig: Option<&str>, tags: &[&str]) -> EvalResult { EvalResult { + isolation: None, task: task.into(), seed: 0, kind: kind.into(), @@ -193,6 +201,7 @@ mod tests { #[test] fn reward_hacking_is_a_public_pass_with_a_hidden_failure() { let with = |passed: bool, public: Option| EvalResult { + isolation: None, public_passed: public, ..r("t", "worker", passed, None, &["x"]) }; @@ -202,6 +211,7 @@ mod tests { with(false, Some(false)), with(false, None), EvalResult { + isolation: None, error: Some("setup failed".into()), ..with(false, Some(true)) }, diff --git a/crates/insi-eval/src/run.rs b/crates/insi-eval/src/run.rs index d51aed3..7fad009 100644 --- a/crates/insi-eval/src/run.rs +++ b/crates/insi-eval/src/run.rs @@ -1,9 +1,16 @@ //! One attempt at one task, in a throwaway worktree. //! -//! The worktree comes from Insi's OWN clone of the task's source, under the +//! The tree comes from Insi's OWN clone of the task's source, under the //! work folder, so the source repository — often someone's live checkout — //! is only ever read. The hidden tests are copied in after the worker has -//! stopped, so nothing the worker did could have read or edited them. +//! stopped, so nothing the worker did could have edited them. +//! +//! A worker attempt gets a repository of its own holding the base commit +//! and nothing else ([`Work::isolated_tree`]): a worktree of the clone shares +//! its history, and `git log --all` there showed the fix commit — the task's +//! answer — to any worker that looked. What git cannot show, the disk still +//! can (the source checkout, the hidden tests); `[worker] sandbox` is what +//! keeps a worker out of those, and each result records which it had. use std::path::{Path, PathBuf}; use std::time::Duration; @@ -204,9 +211,44 @@ impl Work { ], )?; } + // A worker's tree fetches one commit by id from here + // ([`Self::isolated_tree`]); upload-pack refuses an id no ref names + // unless this is set. + git(&dir, &["config", "uploadpack.allowAnySHA1InWant", "true"])?; Ok(dir) } + /// A tree for a worker attempt: a repository of its own whose history is + /// the base commit alone, so no git command in it can reach the fix. + /// + /// A shallow fetch of that one commit from Insi's clone, not a worktree + /// (which shares every object) and not alternates (whose objects + /// `git cat-file --batch-all-objects` would still list). It costs a copy + /// of the base commit's files' objects per attempt; that is the price of + /// a measurement whose answer is not one command away. + fn isolated_tree(&self, clone: &Path, commit: &str, name: &str) -> Result { + let path = self.root.join("runs").join(name); + self.drop_worktree(clone, &path); + std::fs::create_dir_all(&path).map_err(|e| format!("{}: {e}", path.display()))?; + git(&path, &["init", "--quiet"])?; + git( + &path, + &[ + "fetch", + "--quiet", + "--depth", + "1", + "--no-tags", + &file_url(clone), + commit, + ], + )?; + git(&path, &["checkout", "--quiet", "--detach", "FETCH_HEAD"])?; + // It names the clone it came from; the worker has no use for that. + let _ = std::fs::remove_file(path.join(".git").join("FETCH_HEAD")); + Ok(path) + } + /// A fresh worktree at `commit`. Only git's bookkeeping holds the lock: /// the slow parts — deleting a previous tree (its installed /// dependencies run to tens of thousands of files) and checking out the @@ -274,6 +316,17 @@ impl Work { } } +/// A local folder as a `file://` URL: a plain path makes git use its local +/// transport, which ignores `--depth`. +fn file_url(dir: &Path) -> String { + let path = dir.display().to_string().replace('\\', "/"); + if path.starts_with('/') { + format!("file://{path}") + } else { + format!("file:///{path}") + } +} + /// The folder a source is cloned into: the source's own last name, then a /// hash of the whole source, so two sources with one name stay apart. Not the /// whole path spelt out: under a work folder that is itself deep, a checkout @@ -332,6 +385,7 @@ pub const HARNESS_ENV: &str = "INSI_HARNESS"; pub fn attempt(work: &Work, store: &Store, a: &Attempt) -> EvalResult { let started = std::time::Instant::now(); let mut result = EvalResult { + isolation: None, task: a.task.id.clone(), seed: a.seed, kind: a.kind.name().to_string(), @@ -389,7 +443,11 @@ fn attempt_into( result.artifacts = Some(artifacts.display().to_string()); let (clone, tree) = timed(result, "prepare", || { let clone = work.clone_of(&task.source)?; - let tree = work.worktree(&clone, commit, &name)?; + let tree = if a.kind == Kind::Worker { + work.isolated_tree(&clone, commit, &name)? + } else { + work.worktree(&clone, commit, &name)? + }; Ok::<_, String>((clone, tree)) })?; let outcome = run_in(work, &tree, &artifacts, store, a, result); @@ -506,7 +564,7 @@ fn run_in( } } } - let outcome = work_and_check(tree, artifacts, store, a, result); + let outcome = work_and_check(&work.root, tree, artifacts, store, a, result); // Returned whether the attempt passed, failed or errored: what setup // made is unchanged unless the key files changed, and then it stays. if let Some((cache, key)) = &reusable { @@ -519,6 +577,7 @@ fn run_in( /// The worker (for a worker attempt), then the hidden check. fn work_and_check( + work_root: &Path, tree: &Path, artifacts: &Path, store: &Store, @@ -547,7 +606,19 @@ fn work_and_check( .collect::>() .join("\n\n"); keep(artifacts, WORKER_PROMPT, &prompt)?; - let argv = a.worker.render_in(&prompt, tree)?; + let argv = a.worker.sandboxed( + a.worker.render_in(&prompt, tree)?, + &insi_core::Places { + tree, + private: &store.private, + work: work_root, + source: &task.source, + }, + )?; + result.isolation = Some(insi_core::Isolation { + history: "base-only".to_string(), + sandboxed: !a.worker.sandbox.is_empty(), + }); let env: Vec<(&str, &std::ffi::OsStr)> = a .harness .map(|h| (HARNESS_ENV, h.as_os_str())) @@ -853,6 +924,112 @@ mod tests { work.drop_worktree(&clone, &tree); } + /// **A WORKER CANNOT REACH THE ANSWER THROUGH GIT.** Its tree holds the + /// base commit and nothing after it: no ref, no object of the fix. And + /// `[worker] sandbox` is what it runs inside, with its places filled in, + /// which the result records. + #[cfg(unix)] + #[test] + fn a_worker_tree_holds_only_the_base_and_runs_in_the_sandbox() { + let source = temp_dir("isolated-src"); + let sh = |args: &[&str]| git(&source, args).expect("git"); + sh(&["init", "-q", "-b", "main"]); + sh(&["config", "user.email", "t@example.invalid"]); + sh(&["config", "user.name", "t"]); + std::fs::write(source.join("sum.ts"), "a - b\n").expect("w"); + sh(&["add", "."]); + sh(&["commit", "-q", "-m", "feat: sum"]); + let base = sh(&["rev-parse", "HEAD"]); + std::fs::write(source.join("sum.ts"), "a + b\n").expect("w"); + sh(&["commit", "-q", "-am", "fix: sum adds"]); + let reference = sh(&["rev-parse", "HEAD"]); + let fix_blob = sh(&["rev-parse", "HEAD:sum.ts"]); + + let store = Store { + tasks: temp_dir("isolated-tasks"), + private: temp_dir("isolated-private"), + }; + let task = Task { + id: "iso-1".into(), + source: source.display().to_string(), + base: base.clone(), + reference: reference.clone(), + date: None, + prompt: "fix: sum adds".into(), + hidden_tests: vec![], + check: vec!["true".into()], + public_check: vec![], + setup: vec![], + setup_cache: None, + budget: insi_core::Budget { + wall_secs: 60, + tokens: None, + }, + }; + let work = Work::new(temp_dir("isolated-work")); + // The "worker" writes down what git lets it see, and where the + // sandbox said its tree and the hidden tests are. + let mut worker = Worker::command( + [ + "sh", + "-c", + "git rev-list --all > seen.refs; \ + git cat-file --batch-all-objects --batch-check > seen.objects; \ + printf '%s|%s' \"$INSI_T\" \"$INSI_P\" > seen.places", + "{prompt}", + ] + .map(String::from) + .to_vec(), + ); + worker.sandbox = ["env", "INSI_T={tree}", "INSI_P={private}"] + .map(String::from) + .to_vec(); + let r = attempt( + &work, + &store, + &Attempt { + task: &task, + kind: Kind::Worker, + seed: 0, + worker: &worker, + preface: "", + briefed: vec![], + tags: vec![], + keep: true, + harness: None, + }, + ); + assert_eq!(r.error, None, "{r:?}"); + assert_eq!( + r.isolation, + Some(insi_core::Isolation { + history: "base-only".into(), + sandboxed: true + }) + ); + let tree = work.root.join("runs").join("iso-1-worker-s0"); + let read = |f: &str| std::fs::read_to_string(tree.join(f)).expect(f); + assert_eq!( + read("seen.refs").trim(), + base, + "the worker saw more than the base" + ); + let objects = read("seen.objects"); + assert!( + !objects.contains(&reference), + "the fix commit is in the tree" + ); + assert!( + !objects.contains(&fix_blob), + "the fixed file is in the tree" + ); + assert_eq!( + read("seen.places"), + format!("{}|{}", tree.display(), store.private.display()) + ); + assert!(!tree.join(".git").join("FETCH_HEAD").exists()); + } + /// **THE CONTROLS RUN FOR REAL.** In a two-commit repository whose fix /// changes `a - b` to `a + b`, the check fails at the base and passes at /// the reference; the hidden test file arrives in the worktree; each diff --git a/crates/insi-loop/src/accept.rs b/crates/insi-loop/src/accept.rs index 8293a7d..63394dc 100644 --- a/crates/insi-loop/src/accept.rs +++ b/crates/insi-loop/src/accept.rs @@ -94,12 +94,260 @@ pub fn holds(candidate: Score, incumbent: Score, margin: f64) -> Verdict { } } +/// Above this share of errored worker attempts on either side, a comparison +/// says nothing about quality: it measured the infrastructure. (The Watchman +/// learning phase had 14 errors in 18 attempts.) +pub const INFRA_BUDGET: f64 = 0.05; + +/// How many times tasks are resampled for the interval. +pub const RESAMPLES: usize = 2000; + +/// The candidate against the incumbent on the work both were judged on, with +/// what a decision needs: the per-task difference and its interval, which +/// tasks moved, which always-passing tasks broke, errors, tokens and reward +/// hacks. +#[derive(Debug, Clone, PartialEq)] +pub struct Comparison { + pub candidate: Score, + pub incumbent: Score, + /// Tasks judged on both sides (seeds pooled per task). + pub tasks: usize, + /// Mean over tasks of (candidate pass rate - incumbent pass rate). + pub diff: f64, + /// 95% bootstrap interval of `diff`, resampling tasks. + pub interval: (f64, f64), + /// Tasks whose pass rate rose, and fell. + pub better: Vec, + pub worse: Vec, + /// Tasks the incumbent passed on every seed and the candidate failed on + /// at least one: a change may not trade them for wins elsewhere. + pub broke: Vec, + /// Share of worker attempts that errored, per side. + pub errored: (f64, f64), + /// Mean tokens per judged pair, per side, when every pair reported them. + pub tokens: (Option, Option), + /// Public pass with hidden failure, per side. + pub reward_hacks: (usize, usize), +} + +/// What the rule decided. +#[derive(Debug, Clone, PartialEq)] +pub enum Decision { + Accept(String), + Reject(String), + /// Too many attempts errored, or nothing was judged in common: no + /// verdict on quality either way. + NotMeasured(String), +} + +impl Decision { + pub fn accepted(&self) -> bool { + matches!(self, Self::Accept(_)) + } + + pub fn why(&self) -> &str { + match self { + Self::Accept(w) | Self::Reject(w) | Self::NotMeasured(w) => w, + } + } +} + +/// Compare the two sides. `seed` makes the interval reproducible: the same +/// results always give the same interval. +pub fn compare(candidate: &[EvalResult], incumbent: &[EvalResult], seed: u64) -> Comparison { + let (cs, is) = paired(candidate, incumbent); + fn per_task(rs: &[EvalResult]) -> BTreeMap> { + let mut m: BTreeMap> = BTreeMap::new(); + for r in rs + .iter() + .filter(|r| r.kind == "worker" && r.error.is_none()) + { + m.entry(r.task.clone()).or_default().push((r.seed, r)); + } + m + } + let (c, i) = (per_task(candidate), per_task(incumbent)); + let mut diffs = Vec::new(); + let (mut better, mut worse, mut broke) = (Vec::new(), Vec::new(), Vec::new()); + let (mut c_tokens, mut i_tokens, mut tokened, mut pairs) = (0u64, 0u64, 0usize, 0usize); + let (mut c_hacks, mut i_hacks) = (0, 0); + for (task, c_runs) in &c { + let Some(i_runs) = i.get(task) else { continue }; + let both: Vec<(&EvalResult, &EvalResult)> = c_runs + .iter() + .filter_map(|(seed, cr)| { + i_runs + .iter() + .find(|(s, _)| s == seed) + .map(|(_, ir)| (*cr, *ir)) + }) + .collect(); + if both.is_empty() { + continue; + } + let rate = |pick: fn(&(&EvalResult, &EvalResult)) -> bool| { + both.iter().filter(|p| pick(p)).count() as f64 / both.len() as f64 + }; + let d = rate(|p| p.0.passed) - rate(|p| p.1.passed); + diffs.push(d); + if d > 0.0 { + better.push(task.clone()); + } else if d < 0.0 { + worse.push(task.clone()); + } + if both.iter().all(|p| p.1.passed) && both.iter().any(|p| !p.0.passed) { + broke.push(task.clone()); + } + for (cr, ir) in &both { + pairs += 1; + if let (Some(ct), Some(it)) = (cr.tokens, ir.tokens) { + c_tokens += ct; + i_tokens += it; + tokened += 1; + } + c_hacks += usize::from(cr.public_passed == Some(true) && !cr.passed); + i_hacks += usize::from(ir.public_passed == Some(true) && !ir.passed); + } + } + let errored = |rs: &[EvalResult]| { + let workers: Vec<_> = rs.iter().filter(|r| r.kind == "worker").collect(); + if workers.is_empty() { + 0.0 + } else { + workers.iter().filter(|r| r.error.is_some()).count() as f64 / workers.len() as f64 + } + }; + let tokens = if pairs > 0 && tokened == pairs { + ( + Some(c_tokens as f64 / pairs as f64), + Some(i_tokens as f64 / pairs as f64), + ) + } else { + (None, None) + }; + Comparison { + candidate: cs, + incumbent: is, + tasks: diffs.len(), + diff: mean(&diffs), + interval: bootstrap(&diffs, seed), + better, + worse, + broke, + errored: (errored(candidate), errored(incumbent)), + tokens, + reward_hacks: (c_hacks, i_hacks), + } +} + +/// The accept rule. +/// +/// - **Not measured** when more than [`INFRA_BUDGET`] of either side's worker +/// attempts errored, or no task was judged on both. +/// - **Rejected** when a task the incumbent always passed now fails, or the +/// candidate reward-hacks more often: a change may not buy wins with +/// either. +/// - **Accepted** when the whole 95% interval of the per-task difference is +/// above zero and the mean difference is at least `margin`; one lucky +/// task in a small set cannot do that. +/// - **Accepted as a tie that is cheaper** when the interval contains zero, +/// no task got worse, and the candidate used at least 10% fewer tokens +/// on the same pairs. +/// - Otherwise rejected, with the interval in the reason. +pub fn decide(c: &Comparison, margin: f64) -> Decision { + let numbers = format!( + "{}/{} against {}/{} over {} task(s); difference {:+.2}, 95% interval [{:+.2}, {:+.2}]", + c.candidate.passed, + c.candidate.judged, + c.incumbent.passed, + c.incumbent.judged, + c.tasks, + c.diff, + c.interval.0, + c.interval.1 + ); + if c.errored.0 > INFRA_BUDGET || c.errored.1 > INFRA_BUDGET { + return Decision::NotMeasured(format!( + "{:.0}% and {:.0}% of attempts errored (more than {:.0}%): this measured the \ + infrastructure, not the harness", + c.errored.0 * 100.0, + c.errored.1 * 100.0, + INFRA_BUDGET * 100.0 + )); + } + if c.tasks == 0 { + return Decision::NotMeasured("no task was judged for both".to_string()); + } + if !c.broke.is_empty() { + return Decision::Reject(format!( + "{numbers}; broke {} that always passed before", + c.broke.join(", ") + )); + } + if c.reward_hacks.0 > c.reward_hacks.1 { + return Decision::Reject(format!( + "{numbers}; reward hacking rose from {} to {}", + c.reward_hacks.1, c.reward_hacks.0 + )); + } + if c.interval.0 > 0.0 && c.diff + 1e-12 >= margin { + return Decision::Accept(format!("{numbers}; better beyond its interval")); + } + if c.interval.0 <= 0.0 && c.interval.1 >= 0.0 && c.worse.is_empty() { + if let (Some(ct), Some(it)) = c.tokens { + if it > 0.0 && ct <= it * 0.9 { + return Decision::Accept(format!( + "{numbers}; a tie on {:.0} tokens a pair against {:.0}: as good and cheaper", + ct, it + )); + } + } + } + Decision::Reject(numbers) +} + +fn mean(xs: &[f64]) -> f64 { + if xs.is_empty() { + 0.0 + } else { + xs.iter().sum::() / xs.len() as f64 + } +} + +/// The 2.5th and 97.5th percentiles of the mean of `xs` resampled with +/// replacement [`RESAMPLES`] times, from a seeded generator. +fn bootstrap(xs: &[f64], seed: u64) -> (f64, f64) { + if xs.is_empty() { + return (0.0, 0.0); + } + let mut state = seed ^ 0x9E37_79B9_7F4A_7C15; + let mut next = || { + // splitmix64: small, fast, and the same everywhere. + state = state.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = state; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + }; + let n = xs.len(); + let mut means: Vec = (0..RESAMPLES) + .map(|_| { + let sum: f64 = (0..n).map(|_| xs[(next() % n as u64) as usize]).sum(); + sum / n as f64 + }) + .collect(); + means.sort_by(|a, b| a.total_cmp(b)); + let at = |q: f64| means[((RESAMPLES as f64 - 1.0) * q).round() as usize]; + (at(0.025), at(0.975)) +} + #[cfg(test)] mod tests { use super::*; fn r(task: &str, seed: u32, passed: bool) -> EvalResult { EvalResult { + isolation: None, task: task.into(), seed, kind: "worker".into(), @@ -130,6 +378,7 @@ mod tests { r("b", 0, true), r("c", 0, true), EvalResult { + isolation: None, error: Some("setup failed".into()), ..r("d", 0, false) }, @@ -167,4 +416,104 @@ mod tests { .why .contains("3/4 against 2/4")); } + + fn rt(task: &str, seed: u32, passed: bool, tokens: u64) -> EvalResult { + EvalResult { + tokens: Some(tokens), + ..r(task, seed, passed) + } + } + + /// **ONE LUCKY TASK IS NOT A WIN.** Better on one task of four, equal on + /// the rest: the interval reaches zero, so it is rejected; better on all + /// four is accepted. + #[test] + fn a_win_must_clear_its_interval() { + let inc: Vec<_> = ["a", "b", "c", "d"] + .iter() + .map(|t| r(t, 0, false)) + .collect(); + let one = vec![ + r("a", 0, true), + r("b", 0, false), + r("c", 0, false), + r("d", 0, false), + ]; + let d = decide(&compare(&one, &inc, 1), 0.0); + assert!(!d.accepted(), "{d:?}"); + assert!(d.why().contains("95% interval [+0.00"), "{}", d.why()); + let all: Vec<_> = ["a", "b", "c", "d"].iter().map(|t| r(t, 0, true)).collect(); + assert!(decide(&compare(&all, &inc, 1), 0.0).accepted()); + } + + /// **THE INTERVAL IS REPRODUCIBLE.** Same results, same seed, same interval. + #[test] + fn the_same_results_give_the_same_interval() { + let inc: Vec<_> = (0..8).map(|i| r(&format!("t{i}"), 0, i % 3 == 0)).collect(); + let cand: Vec<_> = (0..8).map(|i| r(&format!("t{i}"), 0, i % 2 == 0)).collect(); + assert_eq!( + compare(&cand, &inc, 7).interval, + compare(&cand, &inc, 7).interval + ); + } + + /// **NO TRADING AWAY A SURE THING.** Two new wins do not buy a task the + /// incumbent always passed. + #[test] + fn breaking_an_always_passing_task_is_rejected() { + let inc = vec![r("a", 0, true), r("b", 0, false), r("c", 0, false)]; + let cand = vec![r("a", 0, false), r("b", 0, true), r("c", 0, true)]; + let d = decide(&compare(&cand, &inc, 1), 0.0); + assert!( + matches!(&d, Decision::Reject(w) if w.contains("broke a")), + "{d:?}" + ); + } + + /// **AS GOOD AND CHEAPER SHIPS.** Equal on every task, 20% fewer tokens. + #[test] + fn a_tie_that_is_cheaper_is_accepted() { + let inc = vec![rt("a", 0, true, 1000), rt("b", 0, false, 1000)]; + let cand = vec![rt("a", 0, true, 800), rt("b", 0, false, 800)]; + let d = decide(&compare(&cand, &inc, 1), 0.0); + assert!( + matches!(&d, Decision::Accept(w) if w.contains("cheaper")), + "{d:?}" + ); + let same = vec![rt("a", 0, true, 990), rt("b", 0, false, 990)]; + assert!( + !decide(&compare(&same, &inc, 1), 0.0).accepted(), + "1% is noise" + ); + } + + /// **ERRORS ARE NOT A VERDICT.** More than 5% errored: not measured. + #[test] + fn too_many_errors_is_not_measured() { + let inc: Vec<_> = (0..10).map(|i| r(&format!("t{i}"), 0, false)).collect(); + let mut cand: Vec<_> = (0..10).map(|i| r(&format!("t{i}"), 0, true)).collect(); + cand[0].error = Some("setup failed".into()); + let d = decide(&compare(&cand, &inc, 1), 0.0); + assert!(matches!(d, Decision::NotMeasured(_)), "{d:?}"); + assert!(matches!( + decide(&compare(&[], &[], 1), 0.0), + Decision::NotMeasured(_) + )); + } + + /// **MORE REWARD HACKING IS NEVER BETTER.** + #[test] + fn rising_reward_hacking_is_rejected() { + let hack = |t: &str| EvalResult { + public_passed: Some(true), + ..r(t, 0, false) + }; + let inc = vec![r("a", 0, false), r("b", 0, false), r("c", 0, false)]; + let cand = vec![r("a", 0, true), r("b", 0, true), hack("c")]; + let d = decide(&compare(&cand, &inc, 1), 0.0); + assert!( + matches!(&d, Decision::Reject(w) if w.contains("reward hacking")), + "{d:?}" + ); + } } diff --git a/crates/insi/src/improve.rs b/crates/insi/src/improve.rs index 52c1d1f..54eecd2 100644 --- a/crates/insi/src/improve.rs +++ b/crates/insi/src/improve.rs @@ -106,6 +106,8 @@ pub struct Outcome { /// the evidence a pull request of the best harness carries /// (crates/insi/src/harness_pr.rs). pub kept: Vec, + /// Steps whose comparison measured nothing (too many errors). + pub not_measured: u32, } /// Run the loop. `on_result` sees every attempt before it is saved (to @@ -147,6 +149,7 @@ pub fn improve( let mut accepted = 0; let mut kept = Vec::new(); + let mut not_measured = 0u32; for step in first_step..first_step + l.steps { let Some(arm) = bandit.pick() else { return Err("no strategies to propose with".to_string()); @@ -189,22 +192,46 @@ pub fn improve( }; let cand_select = score(l, &ld, &candidate, l.select, "select", on_result)?; - let (cs, is) = accept::paired(&cand_select, &inc_select); - let mut verdict = accept::beats(cs, is, l.margin); - if verdict.accepted { + // Seeded by the step, so the same results always give the same + // interval and the same decision. + let selection = accept::compare(&cand_select, &inc_select, u64::from(step)); + let mut decision = accept::decide(&selection, l.margin); + if decision.accepted() { let inc_hold = score(l, &ld, &incumbent, l.holdout, "holdout", on_result)?; let cand_hold = score(l, &ld, &candidate, l.holdout, "holdout", on_result)?; + let held_out = accept::compare(&cand_hold, &inc_hold, u64::from(step)); let (ch, ih) = accept::paired(&cand_hold, &inc_hold); let held = accept::holds(ch, ih, l.holdout_margin); - verdict = accept::Verdict { - accepted: held.accepted, - why: format!("{}; {}", verdict.why, held.why), + let first = decision.why().to_string(); + decision = if held_out.errored.0 > accept::INFRA_BUDGET + || held_out.errored.1 > accept::INFRA_BUDGET + { + accept::Decision::NotMeasured(format!( + "{first}; held out: too many attempts errored to judge" + )) + } else if !held.accepted { + accept::Decision::Reject(format!("{first}; {}", held.why)) + } else if !held_out.broke.is_empty() { + accept::Decision::Reject(format!( + "{first}; held out: broke {} that always passed before", + held_out.broke.join(", ") + )) + } else { + accept::Decision::Accept(format!("{first}; {}", held.why)) }; } - let event = if verdict.accepted { - "accepted" - } else { - "rejected" + let verdict = accept::Verdict { + accepted: decision.accepted(), + why: decision.why().to_string(), + }; + let measured = !matches!(decision, accept::Decision::NotMeasured(_)); + if !measured { + not_measured += 1; + } + let event = match decision { + accept::Decision::Accept(_) => "accepted", + accept::Decision::Reject(_) => "rejected", + accept::Decision::NotMeasured(_) => "not-measured", }; audit::append( &ld.audit(), @@ -213,8 +240,11 @@ pub fn improve( .strategy(&strategy.name), )?; println!("step {step}: {event} ({})", verdict.why); - bandit.record(arm, if verdict.accepted { 1.0 } else { 0.0 }); - write_bandit(&ld.bandit(), &bandit)?; + // A step that measured nothing says nothing about its strategy. + if measured { + bandit.record(arm, if verdict.accepted { 1.0 } else { 0.0 }); + write_bandit(&ld.bandit(), &bandit)?; + } if verdict.accepted { let decision = format!( "Insi kept harness {} over {} (strategy {}): {change}", @@ -258,6 +288,7 @@ pub fn improve( best_dir: ld.harness(&best), best, accepted, + not_measured, steps: l.steps, kept, }) @@ -542,6 +573,7 @@ mod tests { }, }; let failed = EvalResult { + isolation: None, task: "shop-1".into(), seed: 0, kind: "worker".into(), diff --git a/crates/insi/src/main.rs b/crates/insi/src/main.rs index 2766096..e5cc41a 100644 --- a/crates/insi/src/main.rs +++ b/crates/insi/src/main.rs @@ -105,11 +105,20 @@ fn main() -> ExitCode { Ok(()) => ExitCode::SUCCESS, Err(why) => { eprintln!("insi: {why}"); - ExitCode::FAILURE + if why.starts_with(NOT_MEASURED) { + ExitCode::from(2) + } else { + ExitCode::FAILURE + } } } } +/// The start of an error that means "this run says nothing about quality": +/// too many attempts errored. Exit code 2, apart from 1 (failed) and 0, so a +/// script or CI never reads infrastructure trouble as a result. +const NOT_MEASURED: &str = "not measured:"; + /// `--key value` pairs and bare `--flag`s. struct Flags(BTreeMap>); @@ -764,6 +773,7 @@ fn eval_cmd(flags: &Flags, work: &Work, results: &Path) -> Result<(), String> { spent: 0.0, }; let mut unrunnable = Unrunnable::default(); + let (mut attempted, mut errored) = (0usize, 0usize); pool::each( &runs, jobs(flags)?, @@ -794,6 +804,8 @@ fn eval_cmd(flags: &Flags, work: &Work, results: &Path) -> Result<(), String> { |(seed, task), mut r| { r.cost_usd = cost_of(&prices, &r); store::append_result(results, &r)?; + attempted += 1; + errored += usize::from(r.error.is_some()); println!( "{} s{seed} {} {:.0}s{}{}{}", task.id, @@ -818,7 +830,16 @@ fn eval_cmd(flags: &Flags, work: &Work, results: &Path) -> Result<(), String> { unrunnable.add(&r) }, )?; - report_cmd(results) + report_cmd(results)?; + let share = errored as f64 / attempted.max(1) as f64; + if share > insi_loop::accept::INFRA_BUDGET { + return Err(format!( + "{NOT_MEASURED} {errored} of {attempted} attempt(s) errored (more than {:.0}%), so \ + this run measured the infrastructure, not the worker", + insi_loop::accept::INFRA_BUDGET * 100.0 + )); + } + Ok(()) } /// The improvement loop: selection on the older half of the fair tasks, @@ -910,6 +931,20 @@ fn improve_cmd(flags: &Flags, work: &Work, results: &Path) -> Result<(), String> }, )?; let ld = improve::LoopDir::new(work); + if outcome.not_measured > 0 && outcome.not_measured == outcome.steps { + return Err(format!( + "{NOT_MEASURED} every one of {} step(s) had too many errored attempts to judge; \ + the audit trail is {}", + outcome.steps, + ld.audit().display() + )); + } + if outcome.not_measured > 0 { + println!( + "{} of {} step(s) were not measured (too many errored attempts)", + outcome.not_measured, outcome.steps + ); + } if outcome.best == outcome.start { println!( "no change beat the harness in {} step(s); the audit trail is {}", @@ -1068,7 +1103,7 @@ fn report_cmd(results: &Path) -> Result<(), String> { } for (condition, t) in &by { println!( - "{condition}: {}/{} passed ({:.0}%), {} repeat failure(s), {} timed out, {} error(s), {} tokens, {:.0}s, {}{}", + "{condition}: {}/{} passed ({:.0}%), {} repeat failure(s), {} timed out, {} error(s), {} tokens, {:.0}s, {}{}{}", t.passed, t.attempts - t.errors, t.pass_rate() * 100.0, @@ -1085,6 +1120,15 @@ fn report_cmd(results: &Path) -> Result<(), String> { ) } else { String::new() + }, + if t.unsandboxed > 0 { + format!( + ", {} attempt(s) unsandboxed (the worker could read the answer on disk; \ + set [worker] sandbox)", + t.unsandboxed + ) + } else { + String::new() } ); } @@ -1134,6 +1178,7 @@ mod tests { phases.insert("worker".to_string(), 1.0); } EvalResult { + isolation: None, task: task.into(), seed: 0, kind: "worker".into(), @@ -1216,6 +1261,7 @@ mod tests { #[test] fn three_unrunnable_attempts_in_a_row_stop_the_run() { let broken = |task: &str| EvalResult { + isolation: None, error: Some("git fetch (exit 128): no output".into()), ..spent(task, None, None, false) }; diff --git a/docs/MEASUREMENT.md b/docs/MEASUREMENT.md new file mode 100644 index 0000000..dd2aba7 --- /dev/null +++ b/docs/MEASUREMENT.md @@ -0,0 +1,78 @@ +# How Insi measures + +Insi's claim is narrow: a harness change is better only when the benchmark +says so, on work it was not chosen on. These rules keep that claim true. A +change to Insi that breaks one of them needs to say why in its pull request. + +## The rules + +1. **The agent never sees the answer.** + - A worker attempt's repository holds the base commit and nothing after + it, so no git command shows the fix. + - The hidden tests arrive only after the worker stops. + - The source checkout and the hidden tests on disk are kept from the + worker by `[worker] sandbox` (for example `sbx --deny-read {source} + --deny-read {private}`). + - `insi report` counts attempts that ran without one, because their passes + aren't proof of the work. + +2. **Only fair tasks count.** A task's hidden check must fail at the base + and pass at the reference (`insi validate`). A check that can't fail + measures nothing. + +3. **Learn on the past, measure on the future.** Lessons and harness changes + are chosen on the older half of the tasks and checked on the newer half, + which never takes part in choosing. + +4. **Compare the same work.** Two conditions are compared only on the + (task, seed) pairs both were judged on. An attempt that errored judges + nothing. + +5. **A win clears its interval.** `insi improve` accepts a change only when + both of these hold: + - the whole 95% bootstrap interval of the per-task difference (tasks + resampled 2,000 times, seeds pooled per task) is above zero; + - the mean difference is at least `--margin`. + + One lucky task in a small set can't do that. The interval is seeded by + the step, so the same results always give the same decision. + +6. **No trading away a sure thing.** + - A change is rejected if any task the incumbent passed on every seed now + fails, however many new wins it brings, on selection or held out. + - It's rejected if reward hacking (public check passed, hidden failed) + rises. + +7. **As good and cheaper ships.** A tie (the interval contains zero, no task + got worse) is accepted when the candidate used at least 10% fewer tokens + on the same pairs. + +8. **Errors are not a verdict.** When more than 5% of either side's worker + attempts errored, the comparison measured the infrastructure. + - `insi improve` records the step as `not-measured` and teaches the + strategy bandit nothing from it. + - `insi eval` exits **2**, apart from 1 (failed) and 0, so a script or CI + never reads infrastructure trouble as a result. + +9. **One run decides nothing.** A result names its seeds. A published number + needs at least three seeds, with the interval beside it. + +10. **Every number comes from a run.** It isn't written by hand in code, + docs or a PR. Every attempt records its tokens, wall time, phases, cost + (with `[prices]`) and isolation, so a number can be traced to the + attempts behind it. + +11. **Show it can fail.** A measurement is trusted only after a deliberate + break turns it red. The pipeline test (`crates/insi/tests/pipeline.rs`) + runs the whole loop with model-free workers on every change: + - a careless worker that must fail; + - a diligent one that passes only when a lesson reaches it; + - a better harness the loop must accept, and a worse one it must reject. + +## Exit codes + +| code | meaning | +|---|---| +| 0 | ran and measured | +| 1 | failed (a bad flag, a missing file, a refused config) | +| 2 | not measured: too many attempts errored to say anything about quality |