Skip to content
incordaiPublic

About

Insi measures whether a coding agent gets better at a repository over time, and is built to be added to any repository as a dependency.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

INCORD

Does your coding agent actually get better at your repository? Insi measures it, and then works to make it so, with a person deciding what lands.

Insi is a command-line tool (one Rust binary) you add to any git repository. It turns the repository's own history into a benchmark, runs a coding agent against it in throwaway checkouts, and judges every attempt with tests the agent never saw. On top of that measurement it builds two ways for the agent to improve:

  • Lessons. After a failed attempt, the agent writes down what went wrong and what to do instead. Later tasks are briefed with those lessons.
  • The improvement loop. The agent's prompts and settings (its harness) are changed one file at a time, and a change is kept only when the benchmark says it is better.

Nothing changes itself silently. An improvement reaches the repository as a draft pull request that a person reviews and merges.

Why it exists: the claim "my agent learns" is usually an anecdote. Insi's rule is that an improvement counts only if it raises the pass rate on tasks it was not learned from, measured against a control.

CI Testing and outstanding work

PR review, evidence and onboarding · Incord candidate execution (removed) · Review research and next priorities

Features

Capability What Insi provides
Repository benchmarks Mine fix commits, validate base/fix controls, hide tests from workers, compare task × seed pairs
Agent improvement Learn lessons from verified fixes; evaluate harness changes and code suggestions against frozen benchmarks
Human-reviewed changes Issue work and accepted improvements produce draft pull requests; Insi does not merge them
PR review Configured analyzers, exact-revision summaries and inline comments, duplicate suppression and explicit incomplete results
Reproduction evidence Run the same frozen test on base/head; distinguish tool allegations from observed regressions and verified fixes
Impact and review quality Conservative test selection, full-check fallback, feedback and labeled-corpus precision/recall evaluation
Agent integration MCP lessons, edit/failure hooks, topic-scoped briefing, external context and Incord memory
Worker adapters incode, Claude, Codex and Antigravity output; sandboxed presets, pinned-model fallback and normalized token accounting
Experiment controls Time/token/cost budgets, bounded concurrency, held-out comparisons, integrity checks and per-phase records
Setup and diagnostics Stack detection, init, onboard, redacted offline doctor, development container and optional security-tool installation

See review and onboarding, code suggestions, benchmark configuration and experiment assessment for commands and prerequisites. Incord candidate execution requires the matching local engine API; there is no direct-worker fallback when that integration is unavailable.

Languages and tested coverage

Rust PHP Go Python JavaScript TypeScript Java C

The benchmark runner can execute a repository's configured test command in other languages too. These icons identify the languages with checked-in security or memory controls; they do not mean equal coverage across languages or frameworks.

Language Checked-in controls
Rust Forgotten ownership and shell-command rule samples; live allocation tracking versus Drop; dependency audit
PHP SQL/shell rule samples; retained versus released string allocations
Go SQL rule samples; blocked goroutines versus cancellation and cleanup
Python SQL, shell and deserialization rule samples; live SQL lab; allocation retention; small IDOR, traversal and HTML controls
JavaScript SQL/HTML rule samples; event-listener retention versus removal
TypeScript SQL/HTML rule samples; no TypeScript application runtime or heap-profile claim
Java SQL rule samples; no JVM runtime or heap-profile claim
C Linux sanitizer controls for leaks, use-after-free and heap-buffer-overflow, with fixed counterparts

Security and memory checks

Area Implementation and limits
Source analysis Semgrep native JSON import and local rules; findings remain tool-reported until reproduced
Secrets Gitleaks import, Trivy secret findings and existing publication redaction; matched secret values are withheld by the native import adapters
Dependencies cargo-audit and Trivy report imports; coverage depends on the configured scanner and advisory database
Containers/configuration Trivy filesystem/configuration imports; seeded Docker configuration checks
Network exposure Nmap XML importer and a local open-port control; an open port is an observation, not proof of a vulnerability
SQL injection Same probe against disposable vulnerable/fixed Python targets, plus language-specific static samples
Access and input boundaries Small positive/fixed controls for IDOR, path traversal, symlink escape and HTML escaping; not full framework or browser testing
Memory and resource retention Native leak/use-after-free/overflow controls; Rust, PHP and Python retained allocations; Go goroutines; Node listeners
Isolation and evidence Internal Docker lab networks, immutable execution images, restricted containers, resource/deadline limits, cleanup and explicit incomplete outcomes

The adapters extend existing JSON, SARIF and ESLint review inputs. They do not automatically install or run every scanner: execution comes from the operator's trusted review plan. Keep required rules and advisory snapshots outside PR control.

For the online workflow audit, install the pinned free scanner and reuse your GitHub login as described in the local-check runbook.

bash scripts/security/install.sh
python3 scripts/ci/test_all.py

The installer includes Semgrep, Trivy, Gitleaks, cargo-audit, Nmap, sqlmap and language runtimes; macOS gets Docker CLI and Colima, and the dev container provides Docker-in-Docker. Start Docker before running the separate container labs. Missing optional tools are reported as skipped tests. The last recorded local run passed 32 security test methods without skips and all eight local check groups; the separate Linux sanitizer run accepted each fixed control and detected its seeded fault. This is evidence about those controls, not a coverage percentage for arbitrary software.

See security setup, lab commands and coverage evidence.

Coverage gaps

Insi does not cover almost every exploit. Seeded boundary controls now exercise password verification, sessions/CSRF, object/method authorization, SSRF address pinning, text uploads and transaction replay/concurrency. A local browser canary and TLS handshake tests cover a reflected-HTML control and certificate validation. The reusable HTTP contract runner can apply declared expectations to a local application. These controls do not establish comprehensive application-level coverage: actual targets still need their identities, routes, workflows and expected behavior supplied. Cryptographic protocols and deployment-wide TLS policy remain outside these samples.

Other gaps include JVM heap profiling, Rust unsafe-code/Miri coverage, dedicated C++/C#/Ruby application controls, independently labeled held-out security corpora, and live sqlmap/Metasploit comparisons. sqlmap is installable, but that alone is not evidence of an executed vulnerability test. Metasploit remains an optional lab probe. Scanner availability, a clean scan and a verified fix are different outcomes. Real loopback authentication/API and SSRF/upload fixtures now compare vulnerable and fixed behavior, including private-canary requests and filesystem effects. See the end-to-end scenarios. Next, apply contracts to the actual application's identity and storage systems.

How it works

 the repository's history
        │
        │  insi mine       every fix commit that also changed a test file becomes a task:
        │                    start         = the commit before the fix
        │                    prompt        = the fix commit's message
        │                    hidden check  = the tests the fix added, kept outside the checkout
        ▼
   tasks
        │  insi validate   two controls, no agent: the hidden check must FAIL at the
        │                  start and PASS at the fix, or the task is not a fair test
        ▼
   fair tasks
        │  insi eval       the agent works each task in its own throwaway worktree;
        │                  the hidden tests are copied in only after it has stopped
        ▼
   results.jsonl  ──────►  insi report: pass rate, timeouts, repeated failures,
        │                  cost, and reward hacking per condition
        │
        │  insi learn      a failed attempt + its verified fix → one lesson
        │                  (the issue, the root cause, what to use, what to avoid)
        ▼
   lessons  ─────────────► briefed before later tasks, only from tasks dated earlier
        │
        │  insi improve    change the harness one file at a time, keep only what wins
        ▼
   draft pull request ───► a person reviews it; Insi never merges

Four rules keep the measurement honest (all eleven, with how each is enforced, are in docs/MEASUREMENT.md):

  1. The agent never sees the check. Hidden tests live outside the attempt's folder and are copied in after the agent stops, replacing anything it wrote at those paths. The attempt's repository holds the base commit and nothing after it, so no git command shows the fix. What is on disk (the source checkout, the hidden tests) is kept from the agent by [worker] sandbox; insi report counts the attempts that ran without one.
  2. Only fair tasks count. A task whose check already passes at the start, or fails even with the real fix, is dropped by insi validate.
  3. Learn on the past, measure on the future. Tasks are split by commit date. Lessons are learned on the older half and measured on the newer half, and a lesson is never briefed to the task it came from or to anything earlier. Conditions are compared on the same task × seed pairs.
  4. Visible score vs real score. When a task carries a public check (the repository's own test command), Insi records whether it passed. A public pass with a hidden failure is counted as reward hacking: the visible score was met without the work.

The agent is anything with a non-interactive mode: Insi runs it through a worker command ([worker] command, or --worker "<cmd {prompt}>"). The default is incode.

Security review can import Semgrep, Trivy, Gitleaks and cargo-audit reports. The security benchmark guide covers tool installation, Nmap observations and disposable vulnerable/fixed SQL-injection controls. Scanner findings remain tool-reported until a reproduction verifies the behavior; sample lab controls do not establish general security coverage.

Quick start

Inmem / Watchman setup

Insi can run its benchmarks and checks without Inmem. Scheduled watch checks require a running Incord Memory engine and a project registered with that engine. Insi does not install, launch or replace the engine. If Watchman already manages Inmem, use that existing instance rather than starting another one. For a standalone installation, complete Inmem's own installation/startup first; inmem doctor checks its health and inmem setup registers supported coding agents with the installed engine. Neither command is replaced by insi init.

Then run insi init for a new Insi project, or insi watch-init --repo . for an existing one. These commands scaffold configuration; they do not immediately execute every scanner or start a second scheduler.

insi init creates an Insi-owned segment in the shared .incord/watch.toml, which Inmem already uses. For an existing repository run insi watch-init. The segment is bounded by # BEGIN INSI WATCH MODULE / # END INSI WATCH MODULE; Insi preserves existing Inmem checks, other agent modules and subsequent edits. It does not maintain a second watch file or scheduler. On a new file it can enable the detected full test command. Existing watch files receive only the commented starter catalog, without a duplicate active suite.

The catalog gives agents templates for language tests, source analysis, secrets, dependencies, container checks, application contracts, network, browser and memory checks. Agents configure and verify applicable commands before enabling them, append insi: namespaced checks inside the segment, and preserve all other modules. Optional scanner installation, app-specific roles/routes and explicit lab targets are still prerequisites; commented examples are not active coverage.

In this repository, the scheduled security suite uses python3 scripts/security/run_checks.py to select an installed Python with hashlib.scrypt, even when Inmem's PATH differs from the terminal's. Set INSI_PYTHON to select an explicit interpreter. Missing compatibility is an incomplete check, never a reason to skip authentication tests. See the Inmem integration guide.

Build and initialize Insi

Insi is built from source: the npm and pip packages are ready but not yet published (RELEASING.md). You need Rust 1.89 or newer and git.

git clone https://github.com/incordai/Insi.git
cd Insi
cargo build --release          # the binary is target/release/insi

In the repository you want to measure:

insi init                  # detect the stack, write .insi/config.toml
insi mine                  # tasks from the history; hidden tests go outside the checkout
insi validate              # which tasks are fair tests
insi eval --fair-only      # run the agent on the fair tasks
insi report                # pass rate and the rest, per condition

To measure whether lessons help:

insi eval  --fair-only --half older --tag phase=learn    # attempts to learn from
insi learn                                               # lessons from their failures
insi eval  --fair-only --half newer --seeds 3            # lessons off
insi eval  --fair-only --half newer --seeds 3 --lessons .insi-work/lessons.jsonl
insi report

Benchmark results

Watchman, run 2 (9 October 2026)

The first benchmark is the history of Watchman, a desktop application in TypeScript and Rust whose tasks are checked with vitest: 38 tasks mined, 35 fair (3 already passed at the start and were dropped). Lessons were learned on the older half and measured on the newer half. One seed, the worker incode on a single model, a budget of 1,800 seconds per attempt.

Half Condition Passed Timed out
older (learning) lessons off 3 / 18 (17%) 6
newer (measured) lessons off 5 / 16 (31%) 6
newer (measured) lessons on 6 / 17 (35%) 3

The 11 lessons learned on the older half were briefed to the lessons-on round. Over the 16 tasks judged both ways:

Lessons off → on Tasks
passed both times 5
failed → passed 1
passed → failed 0
failed both times 10

What this does and does not show. One task changed outcome, so this is not evidence that lessons help. It is one seed, and three things limit it:

  • Every lessons-on attempt was briefed with the same three newest lessons. All 11 lessons sit on one topic and fill the 4,000-character briefing, so this measured three general lessons, not the relevant ones.
  • 4 of 15 lessons were lost to bugs found in this run, since fixed.
  • Wall time and tokens are not comparable between rounds: the lessons-off round ran on a heavily loaded machine, and an attempt cut off at its budget reports no tokens.

Reward hacking was not measured: these tasks carry no public check yet.

Next: a briefing ranked by relevance to the task, the lost lessons recovered, and three seeds.

The report as insi report printed it:

lessons=off,phase=learn:   3/18 passed (17%), 0 repeat failure(s), 6 timed out, 14 error(s)
lessons=off,phase=measure: 5/16 passed (31%), 0 repeat failure(s), 6 timed out, 1 error(s)
lessons=on,phase=measure:  6/17 passed (35%), 0 repeat failure(s), 3 timed out, 0 error(s)

Adding Insi to a repository

cd your-repo
insi init       # writes .insi/config.toml, ignores .insi-work/, registers insi mcp, hooks and CI
insi mine       # tasks from the history, hidden tests outside the checkout
insi validate   # which tasks are fair
insi eval --fair-only

insi init reads the files at the repository's root to tell how it is installed and tested — a Node lockfile (npm, pnpm, yarn, bun) with vitest or jest, Cargo, Python, Go — and writes what it found:

topic = "shop"
test_cmd = ["npm", "test"]
check = ["npx", "vitest", "run", "{tests}"]
setup = ["npm", "ci"]
private_checks = "../insi-private/shop"

[setup_cache]
key = ["package-lock.json"]
keep = ["node_modules"]

[setup_cache] lets attempts reuse a pristine snapshot of setup output. After setup succeeds and before any worker or check runs, node_modules is copied into <work>/deps-pristine-v1/<key>/. Matching attempts receive private copies; their modified dependencies are discarded afterward. The new cache namespace excludes older snapshots exposed to workers. Copying costs disk time, but changes made by one attempt cannot leak into another through the cache. Internal relative dependency symlinks are preserved; links outside the snapshot are refused. A --setup given by hand gets no cache, since it may produce different output.

For Rust the check is cargo test {test_stems}: when every hidden test is an integration-test target (tests/<stem>.rs or tests/<stem>/main.rs), {test_stems} becomes --test <stem> for each and only those binaries run; when any one is elsewhere (a unit test in src/, a helper under tests/) it becomes nothing, and the whole suite runs so that file is still checked.

Every key can be edited; --check and --setup replace what was detected, and an existing config is never overwritten without --force. Besides .insi/config.toml and the .gitignore line, init writes three files other tools read — .mcp.json, .claude/settings.json and .github/workflows/insi.yml — and prints each one it wrote; --no-mcp, --no-hooks and --no-ci keep any of them out. With a config, mine needs no flags, and every command finds its tasks in <private_checks>/tasks and the hidden tests in <private_checks>/hidden, outside the checkout a worker works in.

test_cmd is the repository's whole suite, and init also writes it into a missing .github/workflows/insi.yml: on a pull request from an insi/ branch (the drafts insi issue opens), the stack's toolchain is installed, setup runs, then test_cmd, with a read-only token that is not left in the checkout for the code under test to read. A person reviews a draft the repository's own CI has already checked. An existing workflow file is left alone, and with no detected stack none is written. There is no evals/ folder to commit: a task's definition holds its reference commit, which is the answer, so tasks stay in private_checks.

How a task is made

insi mine reads a repository's history — only reads it — and turns every fix commit that changed source and a test file of its own into a task:

Part of the task Where it comes from
start (base) the fix commit's parent
prompt the fix commit's message
hidden check the test files the fix added or changed
reference the fix commit itself

The task's public part goes to --tasks; its hidden test files go to --private, which should be outside every checkout an agent can read. A test written inline beside the code (Rust's #[cfg(test)]) cannot be held back without handing over the fix, so only separate test files count.

Fair tasks

insi validate runs two controls per task, with no agent:

  • base — the hidden check at the starting commit. It must fail, or the task asks for nothing.
  • reference — the hidden check at the fix. It must pass, or the task cannot be solved as set.

Only tasks that do both are fair; insi eval --fair-only runs just those.

An attempt

Each attempt gets its own worktree from Insi's own clone of the source, so the source repository is never written to. Setup runs, then the worker with the task's prompt and wall-clock budget, then the hidden tests are copied in — after the worker has stopped, replacing anything it wrote at those paths — and the check runs. A failure's output is reduced to a signature (paths, hashes and numbers normalised), so the same mistake made twice is counted as a repeat.

A task may also carry a public check: the repository's own test command (test_cmd, or insi mine --public-check), the score a worker could see. It runs on the tree exactly as the worker left it, before the hidden tests arrive, and the attempt records public_passed; its output is kept as public.out. A pass there with a hidden failure is reward hacking — the visible score met without the work — and insi report counts it per condition. Controls never run it, and one that cannot start is a note, never the verdict.

Three attempts in a row that could not run at all (a git or setup failure, not a failed check) stop insi eval and insi improve: a machine that keeps failing would otherwise burn through every task. A git failure names its exit code, in hex as well when it is a Windows status such as 0xC0000142.

The worker's prompt is the fix commit's message, plus two things a real issue would say:

  • The interface the check needs. Hidden tests often import a module the fix created or a function it exported — names the commit message never mentions, and no worker can guess. The prompt lists the module paths and exported names the hidden tests import that do not exist at the base; never a test, an assertion or an expected value. On the Watchman benchmark 20 of 38 tasks needed this; without it every pilot attempt failed on a missing name. insi brief prints what each task's worker is told, read from the base with git show.
  • That nobody will answer. A worker that asked a question waited out its whole budget for a reply; the prompt now says to decide, make the change and stop.

The full prompt is kept with the attempt as prompt.txt.

Lessons

Each worker attempt keeps what it left behind under <work>/attempts/<name>/: the worker's diff, its output, the check's output.

insi learn takes every failed attempt on a fair task — whose fix is therefore verified: the reference commit passed the check — and asks the worker for one lesson from the attempt, the failure and the fix: an issue, a root cause, an action to use and one to avoid. A lesson is refused if a field is empty, longer than a few sentences, or holds a code block. Lessons are kept in <work>/lessons.jsonl (or --lessons <file>), one per attempt.

With --to-memory, each lesson is also written to the local Incord Memory engine (INCORD_MEMORY_ADDR, default 127.0.0.1:9180; nothing off this machine) as an issue item filed under the repository it came from, tagged insi and verified:check, so every agent on the machine is briefed with it. Leave it off for a benchmark: an experiment's lessons belong in its own file, where they can be reset.

insi eval --lessons <file> puts a briefing in front of each task's prompt: the lessons of the same repository within 4,000 characters, and only lessons learned from tasks dated before this one — compared as instants, so commit dates in different time zones order correctly. A lesson can never be briefed to the task it came from, or to anything earlier.

The briefing is consolidated as of the task's date, never with hindsight:

  • One item per action. Lessons whose use says the same thing (case, spacing and a final full stop aside) are one item, carrying all their ids.
  • Beliefs first. An action learned twice or more is a belief, briefed as "Learned N times"; one seen once follows as "Seen when …". Newest next.
  • Superseded lessons drop out. A later lesson whose avoid is an earlier lesson's use retires the earlier one.
  • Promotion. An action learned in two or more sibling topics is briefed on their shared parent too, marked [across <parent>].

insi issue --lessons and the MCP server's insi_briefing build the same briefing; insi hook picks lessons by its own route (the file about to be edited, the command that just failed) and is not consolidated this way.

Every worker attempt is tagged lessons=on or lessons=off, and records which lessons it was briefed with.

The measurement, end to end:

insi validate ... --jobs 2                                 # which tasks are fair
insi eval  ... --fair-only --half older --tag phase=learn  # attempts to learn from
insi learn ...                                              # lessons from their failures
insi eval  ... --fair-only --half newer --seeds 3          # lessons=off
insi eval  ... --fair-only --half newer --seeds 3 --lessons <work>/lessons.jsonl
insi report

--half older|newer splits the selected tasks by commit date, so lessons are learned on one half and measured on the other. The report groups attempts by their tags, so the learning run carries a tag of its own and is never counted with the measured lessons=off runs.

An issue, worked into a draft pull request: insi issue

insi issue --number 42            # from the repository's folder
  • Only a labelled issue is worked. It must be open and carry the label named by trusted_issue_label in .insi/config.toml. Anyone can write an issue; only a maintainer can label one, so the label is the decision that an agent may take it. An unlabelled issue is refused before its text is given to anything.

  • Its own branch. insi/issue-<n>, from the default branch on origin, in a worktree under the work folder. A branch that already exists is refused: the issue was worked before, and that branch is a person's to read first.

  • Nothing leaves without the check. The worker's change is committed, then the repository's own test_cmd runs. A failing check keeps the commit on the local branch and pushes nothing. So does anything credential-shaped in the lines the branch adds (a prefixed key, a cloud key id, a JWT, a private-key block, a bearer token, or a file such as .env or id_rsa).

  • A draft, with its evidence. Closes #<n>, the check that passed, the worker, its time and tokens, whether lessons were briefed, where it ran. --ready opens it ready for review instead. Insi never merges.

  • Who publishes. By default the person's own GitHub login (gh and git push). With hub = "app" in .insi/config.toml, the branch is pushed and the draft opened by the owner's GitHub App through the Incord memory engine's tools (github_issue, github_default_branch, github_push, github_pr_open): the pull request shows the app as its author, and Insi holds no login and no token. The engine must be running with a GitHub App set up (github_app_id in its audit settings), installed on the repository with Contents and Pull requests write permission. Without that, Insi stops with the engine's reason before any worker runs; it never falls back to gh. The app hub opens drafts only, so --ready is refused. insi suggest and insi heartbeat publish through it too: they work in a clone, so the branch is first pushed from the clone into your local repository (no credentials: the clone's origin is that repository), and the engine then pushes it from there as the app. If the engine refuses, the branch is removed from your repository again and Insi stops with the reason. They do not ask GitHub whether the base moved (that read would use your login).

  • In a sandbox, when you name one. The issue's text is anyone's, and the check runs the code the worker wrote. With a [sandbox] command, setup, the worker and the check each run inside it, with {tree} as the worktree's path:

    [sandbox]
    command = ["docker", "run", "--rm", "-v", "{tree}:/work", "-w", "/work", "node:22"]

    The container gets only what the command gives it: add -e for the one model key the worker needs, --network none where the steps need no network. Committing, the scan and the push stay outside it, on this machine. A sandbox with no {tree} is refused.

    Without a container, insbx does the same with the OS's own sandbox (Landlock on Linux, Seatbelt on macOS). Insi has it built in as insi sandbox run ({insi} in a command is this Insi), taking -w, --deny-read and -q; for the network and wall-clock flags below, use the insbx binary. In the example below:

    • writes are confined to the worktree and the agent's own state;
    • the network only reaches the hosts listed;
    • the session and GIT_DIR variables are stripped;
    • the step is killed after the wall limit.

    It refuses to run when the machine cannot enforce the policy, instead of running unconfined:

    [sandbox]
    command = ["insbx", "run", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json",
               "--allow-host", "api.anthropic.com", "--allow-host", "registry.npmjs.org",
               "--wall", "1800", "--"]

GitHub is reached through the gh CLI, which holds the login: Insi reads, holds and sends no token.

Lessons for any agent: insi mcp

insi mcp serves the lessons over MCP on stdin/stdout, so an agent Insi did not start can use them too. It only reads: nothing an agent sends writes a lesson or runs anything.

Tool What it answers
insi_briefing {topic, before?, max_chars?} the briefing for a repository: verified lessons consolidated as for insi eval (beliefs first), as actions
insi_lessons_for_error {error, topic?} lessons whose failure looked like this one: the error is normalized (paths, numbers, colour codes gone) and matched line by line

insi init registers it for the repository: a missing .mcp.json is written with the entry claude mcp add --scope project insi -- insi mcp would add, which reads .insi-work/lessons.jsonl. An existing .mcp.json is never rewritten; init prints that command instead. Anywhere else, register it like any stdio MCP server, e.g. {"command": "insi", "args": ["mcp", "--lessons", "/path/to/lessons.jsonl"]}.

Lessons when they apply: insi hook

The briefing comes once, before a task. insi hook is a Claude Code hook that brings a lesson back at the step it is about:

Event When What comes back
PreToolUse (Edit, Write, MultiEdit, NotebookEdit) before a file is changed lessons that name that file, by its path in the repository or its file name
PostToolUseFailure (Bash) after a command fails lessons whose failure signature matches the output, as insi_lessons_for_error

A failed command raises PostToolUseFailure, not PostToolUse (measured on Claude Code 2.1.288). The answer is additionalContext, so the hook never blocks, allows or changes a call; at most two lessons at a time, and each one once per session (the ids shown are kept in .insi-work/hook-shown/). Anything it cannot read is no answer and exit 0, with the reason on stderr.

insi init writes it into a missing .claude/settings.json; an existing one is left alone and init prints the block to merge in.

Outside context: insi context

insi context --topic shop/api --query "refund rounding"

What the repository's outside sources say about one part of it, for an agent or a person. The MCP tool insi_context answers the same. Every item says where it came from, how far it is trusted, and its topic. Only items on the asked topic's line of descent come back, ordered by the words asked for, bounded in number and length.

Source Trust What
docs trusted README.md and docs/**/*.md, one item per section, each on its file's topic
ci internal the last failed GitHub Actions runs through gh, the end of their failed steps' log, on the topic of the repository files the log names (--no-ci leaves them out)
[[connectors]] untrusted unless set any command that prints items as JSON lines: an error tracker's export, a mailbox's
[[connectors]]
name = "errors"
command = ["sentry-export", "--project", "shop", "--json-lines"]   # {"title", "body", "url"?, "paths"?, "topic"?} per line
trust = "untrusted"

The command holds whatever login it needs; Insi holds none and only reads what it prints. A source that fails is named under "Not read" and the others still answer. Untrusted text is shown inside a fence it cannot close from within, marked as data: an email that says "ignore your rules" is something to read, not something to do. Outside context is not put in insi eval's briefings, so a measurement depends only on its tasks.

Topics: which lessons apply where

A lesson is filed on a topic, a /-separated path broadest first (shop → shop/api → shop/api/billing), and applies to that topic and every one below it. The repository's topic is the base; [[topic_rules]] in .insi/config.toml name narrower ones by the files and commands they cover. Resolution is by these rules only, never a model, and the same files always give the same topic.

[[topic_rules]]
topic = "shop/api"
paths = ["src/api"]              # a folder covers everything under it; * and ? stay in one name, ** spans folders
commands = ["npm run test:api"]  # these words, side by side, anywhere in the command
  • insi learn files each lesson on the narrowest topic covering every file its fix changed; files outside every rule leave it on the base.
  • The briefing for a task holds its topic's lessons and those of the topics above it. A task starts on the repository's topic, so lessons filed narrower are not in it.
  • insi hook brings those back when the agent edits a file in their part of the repository, and matches a failed command on the command's topic. In insi eval the worker gets the briefing only, so in a measurement only lessons on the base topic or above are used.

The most specific rule wins (deepest topic, then the first in the file); a rule with no paths and no commands, or a topic with an empty part, is refused when the config is read.

The improvement loop: insi improve

To suggest improvements to working application code, run insi improve --code (or insi suggest). It creates a benchmark, measures the baseline, implements in isolated clones, and opens a draft PR only for a measured gain with passing regressions. Use --local to keep qualifying results local. See code improvement suggestions. Use insi heartbeat --interval-secs 3600 to repeat discovery and evaluation on a timer. Each proposal needs a project-fit and feasibility justification grounded in committed code; discovery can decline to make a change. See heartbeat scheduling and recovery.

For controlled component comparisons on recorded runs, use insi experiment --plan experiment.json. It checks matched tasks and seeds, repository separation, stable controls and declared provenance, and reports failure evidence, reliability and statistical support without running a worker. See experiment assessment for the plan format and limits.

The harness is a folder of prompts and settings a worker runs with; its prompt.md is put before every task's prompt. insi improve --harness <dir> changes it one file at a time and keeps a change only when the benchmark says it is better:

  1. A UCB1 bandit picks a strategy (from-failures, clarify, simplify, checklist, or the [[strategies]] in .insi/config.toml): each is tried once, then the ones whose changes were kept are tried more.
  2. The proposer (--proposer, [proposer] in the config, else the worker) is shown the harness, each selection task's title and outcome, earlier proposals and the lessons, and answers with one file's whole new text and why. It is never shown a check's output: that is the hidden tests speaking, and a proposer that read them could write their answers into the harness. A path outside the harness, a hidden file, an oversized file or a change that changes nothing is refused before anything runs.
  3. The candidate runs the older half of the fair tasks with the same worker, seeds and budget as the incumbent, each attempt in its own worktree with the check run there.
  4. It is kept only when it beats the incumbent on the same task × seed pairs by --margin (a tie never wins) and is not worse on the newer half, which is never used to select (--holdout-margin to demand more).

Each attempt also runs with INSI_HARNESS set to that harness's own snapshot folder, so the worker can read the rest of it. incode does: instructions.md joins its system context, roles.json ({"roles": …, "tiers": …}) routes its models over admin.json for that run only, and agents/*.md are sub-agents between the project's and yours. A harness file incode cannot read refuses the run instead of running on the defaults.

Everything lands under <work>/loop/: every harness scored (harnesses/<id>/, named by content, so a harness is never run twice on the same task and seed), every attempt (results.jsonl), every state change (audit.jsonl: baseline, proposed, refused, accepted, rejected, finished), and the bandit (bandit.json). A second run continues the step count and the bandit. Your harness folder is only read: the loop prints where the best harness is, and putting it in place is yours to decide.

  • --pr opens it as a draft pull request on the repository holding the harness folder, through gh's login. Only the files the loop changed are written, in a fresh worktree of the default branch on insi/harness-<id>, so your uncommitted edits never ride along. The same credential and personal-data scan as insi issue runs first; a harness that fails it stays on a local branch and nothing is pushed. The PR lists each change kept with its benchmark verdict. A branch that exists already is refused: that harness was proposed before.
  • --to-memory files each change kept, and the run's end, in the local Incord Memory engine as a decision (tagged insi, loop:accepted or loop:finished). Proposals and scores stay in audit.jsonl, the record; an engine that is down costs a printed line, never the run.

Unattended runs that open pull requests wait for the GitHub App with short-lived, PR-only tokens; until then --pr uses the operator's login.

The project's own CI, on this machine: insi ci

insi ci --repo <dir> runs the steps of the repository's own GitHub workflows (.github/workflows/*.yml) on this machine, so a push never finds out from GitHub what the machine could have said. It keeps the workflows that run on a push or a pull request, and for each job whose runs-on is this operating system (a ${{ matrix.os }} is expanded) it runs every run: step in order, in the repository, with the step's shell, env: and working directory, and reports each step as pass, FAIL, skip (with why) or unavailable (a tool the step needs is not installed here). It exits 1 when a step failed; --strict also fails on an unavailable one. --all-jobs runs every job, not only this operating system's: a job meant for another one runs here as its runner would (its shell, its runner.os). A check GitHub runs as an action runs as its own tool, with the step's with: inputs: crate-ci/typos as typos, zizmor-action as zizmor --offline, cargo-deny-action as cargo deny, and cargo-semver-checks-action as cargo semver-checks.

insi ci [--repo <dir>] [--workflow <file>] [--job <id>] [--list] [--all-jobs] [--json] [--quiet] [--strict]
        [--timeout-secs N]
insi ci --install-hook [--repo <dir>] [--force]

What it does not do, and says so for each step it passes over: it runs no other uses: action (actions/checkout is the repository already in front of it; setup-* and rust-toolchain are the tools installed here); it never runs a step that would change this machine's own setup (git config --global, sudo, system package installs, global npm installs, rustup default), which a CI runner throws away and a developer keeps; it runs no job with container: or services:; it never runs a step that would publish or push (git push, cargo publish, npm publish, docker push, gh release, and the like); it does not run a step whose ${{ ... }} it cannot resolve (a secret, another step's output); and a matrix over versions runs once, with the versions installed here. A workflow run only by a tag, a release, a schedule or by hand is not a check and is left out, unless --workflow <file> names it. --list plans without running; --json is for an agent or a script.

insi ci --install-hook writes .git/hooks/pre-push (wherever git looks for hooks), which runs insi ci --quiet before every push and stops the push when a step failed. INSI_CI_SKIP=1 git push skips it once; a hook that insi did not write is never replaced unless --force. An agent working in the repository runs insi ci before any push.

Commands

insi doctor --markdown collects offline diagnostics for a bug report: build version and commit, target, OS, Git version, configured-worker availability, redacted configuration, and work-folder lock states. It does not run the worker or clean the work folder. Config strings and model names are redacted; lock states are only a snapshot. Use --repo <dir> and --work <dir> to inspect another repository and its work folder.

insi doctor   [--repo <dir>] [--work <dir>] [--markdown]
insi ci       [--repo <dir>] [--workflow <file>] [--job <id>] [--list] [--all-jobs] [--json] [--quiet] [--strict]
              [--timeout-secs N] | --install-hook [--force]
insi mine     --source <repo> --tasks <dir> --private <dir> --check "<cmd {tests}>"
              [--setup "<cmd>"] [--public-check "<cmd>"] [--prefix <name>] [--limit N] [--wall-secs N]
insi validate --tasks <dir> --private <dir> [--only id,id] [--half older|newer] [--results <file>] [--jobs N]
insi eval     --tasks <dir> --private <dir> [--seeds N] [--tag k=v] [--worker "<cmd {prompt}>"]
              [--lessons <file>] [--fair-only] [--only id,id] [--half older|newer]
              [--results <file>] [--keep] [--jobs N]
insi learn    --tasks <dir> --private <dir> [--lessons <file>] [--worker "<cmd {prompt}>"] [--to-memory]
              [--results <file>] [--jobs N]
insi improve  --harness <dir> --tasks <dir> --private <dir> [--steps N] [--seeds N] [--margin M]
              [--holdout-margin M] [--worker "<cmd {prompt}>"] [--proposer "<cmd {prompt}>"]
              [--only id,id] [--jobs N] [--max-dollars D] [--keep] [--to-memory] [--pr]
insi report   [--results <file>]

Every command takes --work <dir> (default .insi-work): Insi's clones, worktrees and the results file (results.jsonl, one line per attempt, appended). --jobs N runs N attempts at once. --setup-jobs N (default 1) lets only N pieces of bulk disk work run at once — a setup, or deleting a finished worktree. Both are bound by the disk: on Watchman, whole attempts run one after another took 145–205s, while with three installs at once a single setup took about 1,060s. Workers and checks still overlap.

Each result records seconds per phase — prepare, queue (waiting for a setup slot), setup, worker, check — so a slow run says where its time went. Deleting a finished worktree happens in the background and is in none of them. A worker declares the files its harness writes for itself ([worker] own_files, git glob pathspecs; incode's state, spec documents and backups by default), and they are left out of the worker's recorded diff.

Example, a TypeScript project tested with vitest:

insi mine --source ../app --tasks ../insi-private/app/tasks --private ../insi-private/app/hidden \
  --check "npx vitest run {tests}" --setup "npm ci" --prefix app --limit 40
insi validate --tasks ../insi-private/app/tasks --private ../insi-private/app/hidden --work ../insi-work
insi eval --tasks ../insi-private/app/tasks --private ../insi-private/app/hidden --work ../insi-work \
  --fair-only --seeds 3 --tag lessons=off
insi report --work ../insi-work

Configuration

.insi/config.toml in the repository being worked on; every key is optional, and an unknown key is refused.

topic = "app"
test_cmd = ["npm", "test"]
protected = ["src/auth/**"]
eval_tasks = "evals/tasks"
private_checks = "../insi-private/app"
trusted_issue_label = "insi"
# Who publishes branches and draft pull requests: "gh" (your login, the default)
# or "app" (the owner's GitHub App, through the Incord memory engine).
hub = "gh"

[worker]
# {prompt} is replaced by the task's prompt. The default is incode.
command = ["incode", "-t", "{prompt}", "--output-format", "stream-json", "--spec-gate=off"]

# US dollars per million tokens, by the model name the worker reports.
# A model on a subscription is an entry with no keys (all zero).
[prices."claude-sonnet-5-5"]
input = 3.0
output = 15.0
cache_write = 3.75
cache_read = 0.3

Any agent with a non-interactive mode can be the worker. Insi reads answers, models and token usage from incode, Claude, Codex and Antigravity machine output. Codex cached input is charged separately; Antigravity thinking tokens count as output. Missing stream models fall back to --model or -m. Use insi init --worker claude --model <id> (also codex or agy) to write a headless preset with CLI-specific writable state paths inside Insi's own sandbox (insi sandbox run, built in: nothing else to install; on macOS the system's sandbox, so the agent stays signed in). These presets require a pinned model. insi doctor warns about missing output, permission and model flags without printing their values. For wrapper commands set [worker] format to incode, claude, codex, agy or text; otherwise known executable names are inferred. Plain-text workers are timed without token accounting. Worker exits are recorded in attempt notes; failed startup without a diff and explicit auth/API failures are infrastructure errors rather than learning examples. --spec-gate=off keeps incode from writing its spec documents before each already-scoped task (it needs an incode that has the flag; an older one refuses it by name).

Keeping the worker from the answer

The answer to a task is in three places: the fix commit, the source repository's checkout (usually at a commit after the fix), and the hidden tests. A worker attempt's repository is a shallow fetch of the base commit alone, so git can't reach the first one. The other two are on disk, and any worker with a shell can read them. Measured: a worker that only ran cp ../../../src/lib.rs src/lib.rs passed its task, and the reward-hacking check didn't catch it, because it passed the hidden tests for real.

[worker] sandbox runs the worker step inside a command and fills in its places:

  • {tree}: the attempt's folder (required);
  • {private}: the hidden tests;
  • {work}: Insi's work folder;
  • {source}: the task's source, which must be a local folder.

With Insi's own sandbox (the library of insbx, built in):

[worker]
command = ["incode", "-t", "{prompt}", "--output-format", "stream-json", "--spec-gate=off"]
sandbox = ["{insi}", "sandbox", "run", "-q", "-w", "{tree}", "-w", "~/.claude", "-w", "~/.claude.json",
           "-w", "~/.incode", "--deny-read", "{source}", "--deny-read", "{private}", "--"]

The same cheating worker then fails, as it should: the copy is refused. Each result records isolation (history: "base-only", sandboxed). insi report adds N attempt(s) unsandboxed to a condition whose passes are not proof of the work.

Every command Insi starts — the worker, the checks, gh — is started without the variables of the agent session Insi itself runs in (CLAUDECODE, CLAUDE_CODE_SESSION_ID, …), so concurrent attempts never share, or report, the operator's session. Credentials are passed through.

Each attempt records cost_usd from [prices], and insi report sums it per condition — or says cost unknown (N attempt(s) unpriced; add [prices] to .insi/config.toml) when no attempt has a price, rather than $0.00. insi eval --max-dollars D starts no new attempt once the run has spent D; an attempt already running is bounded by its wall time, so the run can end above D by at most the attempts in flight. An attempt whose cost is not known — no usage reported, or no price for its model — stops a capped run rather than counting as free. (The prices above are an example of the format; set the ones you pay.)

Layout

Crate What
insi-core config, the task format, failure signatures
insi-eval running commands with deadlines, mining, the store, attempts, reports
insi-memory lessons: the writer's prompt and parsing, the store, the briefing
insi-loop the improvement loop's decisions: proposals, the accept rule, the strategy bandit, the audit trail
insi-connectors outside sources: docs, CI logs and command connectors, each item with its source, trust and topic
insi the command line

Building

cargo build --release
cargo test --workspace --locked

Contributing

Use Discussions for questions, ideas and benchmark results. Use issues for reproducible bugs and scoped work.

Contributions are welcome: bug reports, fixes, new stack detection, and benchmark results from your own repositories. Before opening a pull request, run what CI runs on Windows, macOS and Linux:

cargo fmt --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked

See CONTRIBUTING.md for how changes are written, tested and reviewed, and RELEASING.md for how a release is cut.

Report a vulnerability privately, as SECURITY.md describes, never in a public issue. Everyone taking part follows the Code of Conduct.

License

Insi is released under the Business Source License 1.1, copyright 2026 Incord AI: free for your own personal and internal business use, and it becomes Apache License 2.0 on the Change Date named in the license. See NOTICE.

About

Insi measures whether a coding agent gets better at a repository over time, and is built to be added to any repository as a dependency.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages