Repository navigation
Description
Status: recommendation, not a decision. Ideas taken from reading kodustech/kodus-ai, an open-source AI code-review product, for the maintainers to accept, change or reject.
Licence: Insi is Apache-2.0. Kodus is AGPLv3, and its
ee/folders and*.ee.*files are under a separate commercial licence. Borrow ideas only, never code or prompts, and don't open theee/code. Kodus paths are given below only as pointers to where each idea lives.
Out of scope: storing memory or knowledge. Incord Memory already holds facts, code, chat and edits pinned to topics in the code, so Kodus's rules/memories storage is not proposed. Section 2 only measures memory; it doesn't store any.
1. Measurement (largest gain)
1.1 Confidence interval instead of a fixed margin
- Idea: resample tasks (seeds pooled per task) to get a 95% interval for candidate minus incumbent. Accept only when the lower bound is above 0. Use a seeded RNG so the interval is reproducible. List the tasks that moved in the PR body.
- Where in Insi:
crates/insi-loop/src/accept.rs(beatsuses a fixedmargintoday). Keepmarginas an optional floor. - Kodus pointer:
evals/experiment.jscompare(). - Acceptance: same results in, same interval out; a candidate that's +1 task on 3 tasks is not accepted.
1.2 "Tie but cheaper" ships, with guards
- A change is accepted if the pass rate ties and tokens or cost drop.
- It is rejected if any task that passed on every seed for the incumbent now fails, or if the reward-hacking count (public pass, hidden fail) rises.
- With several worker models, accept only if none got worse and at least one got better.
- Kodus pointer:
experiment.jsdecide()/decideAcrossModels().
1.3 "Not measured" is its own outcome
- If more than ~5% of attempts on either side errored, report "not measured (infrastructure)", not a quality verdict.
- Exit code 2 for
insi eval/insi improve, distinct from 1 (below floor). - Today
paired()drops errored pairs silently. The Watchman learn phase had 14 errors out of 18. - Kodus pointer:
evals/README.mdrule 10,evals/shared/nightly-verdict.js.
1.4 Reuse the incumbent's results
- Cache the incumbent's
results.jsonl, keyed by (hash of the harness files, task-set hash, worker model, budget, seeds). Reuse it for ≤7 days, since providers drift. - Each
insi improvecandidate then only pays for its own side.
1.5 Scripted fake worker as Insi's own CI
- A worker command that applies the task's reference fix must pass. A no-op worker must fail.
- Running both through
insi evalchecks the whole pipeline (worktree, hidden-test copy, judging, token parsing) with no model spend. Add it to Insi's CI and as a preflight forinsi improve. - Kodus pointer:
evals/wiring-smoke.js.
1.6 insi watch: a nightly regression check
- Run a fixed task subset on the merged harness against
floors.toml. - A first miss triggers a confirming re-run, and only the pooled result alerts.
- Set floors at mean − 2·SEM from two runs of the same commit, and record the model and task set they were calibrated on.
- Add "collapse" floors (share of attempts with any diff, timeout share, mean turns) to catch a broken pipeline as opposed to noise.
1.7 Experiment log
insi improveappends one row per candidate to.insi/experiments.jsonl: diff hash, interval, moved tasks, decision reason, model, task-set version.- The draft PR body is generated from it.
- Refuse to run with uncommitted harness changes, so every result points at code that exists.
1.8 Re-score without re-running
insi report --rescorere-runs hidden checks on stored worker diffs, for example after fixing a hidden test, without paying for the agent again.
2. Measure memory, don't store it
2.1 Per-item effect of Incord Memory on results
- Insi has a stronger signal than thumbs up/down: the hidden check.
- For each memory item briefed to an attempt, record its id in the result. Then report each item's pass rate briefed vs control, and "tasks harmed by item X": passed with memory off, failed with it on.
- Feed this back to Incord Memory (for example a
helped/harmedcount on the item, or a flag for review). Insi stores nothing of its own.
3. Issues, CI and PR commands
3.1 Follow-ups that close themselves
- Kodus pointer: Kody Issues,
libs/issues/.../kodyIssuesManagement.service.ts. The idea only; its LLM part isee/. - After
insi issue, store the worker's out-of-scope notes and partial-pass failures per file. - When a later PR touching that file merges, re-check each one, preferring a deterministic test over a model verdict, and close the stale ones.
- Optionally file the survivors as unlabelled GitHub issues, so a maintainer still decides.
3.2 CI annotations before log tails
- For the CI fix loop (Recommendation: more ways in and out for
insi issue(CI fix loop, chat triage agent, email intake) #2 R1), read check-run annotations first (gh api repos/{o}/{r}/check-runs/{id}/annotations): exact file:line and rule id. - Fall back to the log tail (
LOG_TAIL = 4000incrates/insi-connectors/src/ci.rs). - Keep "not fetched" distinct from "fetched, none".
3.3 PR comment commands
/insi retryand/insi fix-cion aninsi/PR are accepted only from users whosegh api repos/{o}/{r}/collaborators/{u}/permissionis write or admin.- React with 👀 to acknowledge. Mark Insi's own comments with
<!-- insi -->so it never answers itself. - Make it idempotent by comment id, and on 403/429 re-queue until the rate-limit reset.
3.4 Commit trailers for provenance
Insi-Run: <attempt id>andInsi-Memory: <item ids>oninsi issuecommits tie every PR line back to the attempt record and the memory it was briefed with.
4. Sandbox safety (small, do early)
4.1 Remove git's environment variables from every child process
GIT_DIR,GIT_WORK_TREE,GIT_INDEX_FILE,GIT_OBJECT_DIRECTORY,GIT_ALTERNATE_OBJECT_DIRECTORIES,GIT_COMMON_DIR,GIT_NAMESPACE.insi initinstalls hooks, and a process started from a hook inherits these, so a worker's or check'sgitcan act on the caller's repository instead of the worktree.- Nothing in
crates/removes environment variables today. Add the removal incrates/insi-eval/src/process.rsandgit.rs, and also the Claude session variables from Findings from an end-to-end run with incode as the worker (session sharing, spec-doc tax, cost line) #1. - Test: run the worker command with
GIT_DIR=/elsewhere/.gitset.git rev-parse --git-dirinside it must point at the worktree.
4.2 Hardened [sandbox] example and a self-check
- Document a Docker command with
--cap-drop ALL --security-opt no-new-privileges --read-only --tmpfs /tmp --user <uid>. For network, suggest--network noneor a network whose only route is an allowlisting proxy. - Before trusting a sandbox, Insi runs a probe inside it:
sudo -n truemust fail and{tree}must be the only writable mount.
4.3 insi clean
- Label containers
--label insi.run=<id>. insi cleanremoves labelled containers and worktrees older than N hours from runs that ended.
Suggested order
4.1 → 1.3 → 1.1/1.2 → 1.5 → 3.2 (with #2 R1) → 2.1 → the rest.
Activity
blockmandev commentedon Oct 20, 2026
Closing at the owner's request.
Done in #6:
sbx(sandbox: sbx, a standalone sandbox any agent can run under Incode#12).Not started: §1.4–1.8 (baseline cache,
insi watch, experiment log, rescore), §2.1 (per-memory-item effect), §3 (follow-ups, CI annotations, PR comment commands, trailers) and §4.3 (insi clean). This issue stays as their reference.Generated by Claude Code