Skip to content

Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3

Closed

Description

@blockmandev

Status: recommendation, not a decision. Ideas taken from reading kodustech/kodus-ai, an open-source AI code-review product, for the maintainers to accept, change or reject.

Licence: Insi is Apache-2.0. Kodus is AGPLv3, and its ee/ folders and *.ee.* files are under a separate commercial licence. Borrow ideas only, never code or prompts, and don't open the ee/ code. Kodus paths are given below only as pointers to where each idea lives.

Out of scope: storing memory or knowledge. Incord Memory already holds facts, code, chat and edits pinned to topics in the code, so Kodus's rules/memories storage is not proposed. Section 2 only measures memory; it doesn't store any.


1. Measurement (largest gain)

1.1 Confidence interval instead of a fixed margin

  • Idea: resample tasks (seeds pooled per task) to get a 95% interval for candidate minus incumbent. Accept only when the lower bound is above 0. Use a seeded RNG so the interval is reproducible. List the tasks that moved in the PR body.
  • Where in Insi: crates/insi-loop/src/accept.rs (beats uses a fixed margin today). Keep margin as an optional floor.
  • Kodus pointer: evals/experiment.js compare().
  • Acceptance: same results in, same interval out; a candidate that's +1 task on 3 tasks is not accepted.

1.2 "Tie but cheaper" ships, with guards

  • A change is accepted if the pass rate ties and tokens or cost drop.
  • It is rejected if any task that passed on every seed for the incumbent now fails, or if the reward-hacking count (public pass, hidden fail) rises.
  • With several worker models, accept only if none got worse and at least one got better.
  • Kodus pointer: experiment.js decide() / decideAcrossModels().

1.3 "Not measured" is its own outcome

  • If more than ~5% of attempts on either side errored, report "not measured (infrastructure)", not a quality verdict.
  • Exit code 2 for insi eval / insi improve, distinct from 1 (below floor).
  • Today paired() drops errored pairs silently. The Watchman learn phase had 14 errors out of 18.
  • Kodus pointer: evals/README.md rule 10, evals/shared/nightly-verdict.js.

1.4 Reuse the incumbent's results

  • Cache the incumbent's results.jsonl, keyed by (hash of the harness files, task-set hash, worker model, budget, seeds). Reuse it for ≤7 days, since providers drift.
  • Each insi improve candidate then only pays for its own side.

1.5 Scripted fake worker as Insi's own CI

  • A worker command that applies the task's reference fix must pass. A no-op worker must fail.
  • Running both through insi eval checks the whole pipeline (worktree, hidden-test copy, judging, token parsing) with no model spend. Add it to Insi's CI and as a preflight for insi improve.
  • Kodus pointer: evals/wiring-smoke.js.

1.6 insi watch: a nightly regression check

  • Run a fixed task subset on the merged harness against floors.toml.
  • A first miss triggers a confirming re-run, and only the pooled result alerts.
  • Set floors at mean − 2·SEM from two runs of the same commit, and record the model and task set they were calibrated on.
  • Add "collapse" floors (share of attempts with any diff, timeout share, mean turns) to catch a broken pipeline as opposed to noise.

1.7 Experiment log

  • insi improve appends one row per candidate to .insi/experiments.jsonl: diff hash, interval, moved tasks, decision reason, model, task-set version.
  • The draft PR body is generated from it.
  • Refuse to run with uncommitted harness changes, so every result points at code that exists.

1.8 Re-score without re-running

  • insi report --rescore re-runs hidden checks on stored worker diffs, for example after fixing a hidden test, without paying for the agent again.

2. Measure memory, don't store it

2.1 Per-item effect of Incord Memory on results

  • Insi has a stronger signal than thumbs up/down: the hidden check.
  • For each memory item briefed to an attempt, record its id in the result. Then report each item's pass rate briefed vs control, and "tasks harmed by item X": passed with memory off, failed with it on.
  • Feed this back to Incord Memory (for example a helped/harmed count on the item, or a flag for review). Insi stores nothing of its own.

3. Issues, CI and PR commands

3.1 Follow-ups that close themselves

  • Kodus pointer: Kody Issues, libs/issues/.../kodyIssuesManagement.service.ts. The idea only; its LLM part is ee/.
  • After insi issue, store the worker's out-of-scope notes and partial-pass failures per file.
  • When a later PR touching that file merges, re-check each one, preferring a deterministic test over a model verdict, and close the stale ones.
  • Optionally file the survivors as unlabelled GitHub issues, so a maintainer still decides.

3.2 CI annotations before log tails

3.3 PR comment commands

  • /insi retry and /insi fix-ci on an insi/ PR are accepted only from users whose gh api repos/{o}/{r}/collaborators/{u}/permission is write or admin.
  • React with 👀 to acknowledge. Mark Insi's own comments with <!-- insi --> so it never answers itself.
  • Make it idempotent by comment id, and on 403/429 re-queue until the rate-limit reset.

3.4 Commit trailers for provenance

  • Insi-Run: <attempt id> and Insi-Memory: <item ids> on insi issue commits tie every PR line back to the attempt record and the memory it was briefed with.

4. Sandbox safety (small, do early)

4.1 Remove git's environment variables from every child process

  • GIT_DIR, GIT_WORK_TREE, GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_COMMON_DIR, GIT_NAMESPACE.
  • insi init installs hooks, and a process started from a hook inherits these, so a worker's or check's git can act on the caller's repository instead of the worktree.
  • Nothing in crates/ removes environment variables today. Add the removal in crates/insi-eval/src/process.rs and git.rs, and also the Claude session variables from Findings from an end-to-end run with incode as the worker (session sharing, spec-doc tax, cost line) #1.
  • Test: run the worker command with GIT_DIR=/elsewhere/.git set. git rev-parse --git-dir inside it must point at the worktree.

4.2 Hardened [sandbox] example and a self-check

  • Document a Docker command with --cap-drop ALL --security-opt no-new-privileges --read-only --tmpfs /tmp --user <uid>. For network, suggest --network none or a network whose only route is an allowlisting proxy.
  • Before trusting a sandbox, Insi runs a probe inside it: sudo -n true must fail and {tree} must be the only writable mount.

4.3 insi clean

  • Label containers --label insi.run=<id>.
  • insi clean removes labelled containers and worktrees older than N hours from runs that ended.

Suggested order

4.1 → 1.3 → 1.1/1.2 → 1.5 → 3.2 (with #2 R1) → 2.1 → the rest.

Related: #1, #2

Activity

  1. blockmandev commented on Oct 20, 2026

    @blockmandev
    MemberAuthor

    Closing at the owner's request.

    Done in #6:

    Not started: §1.4–1.8 (baseline cache, insi watch, experiment log, rescore), §2.1 (per-memory-item effect), §3 (follow-ups, CI annotations, PR comment commands, trailers) and §4.3 (insi clean). This issue stays as their reference.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions