Repository navigation
Description
Status: recommendation, not a decision. This is a gap analysis against kodustech/kodus-ai, written as a staged roadmap. #3 lists features to borrow; this issue is about the practices that put Kodus where it is, and what Insi would need to match them. It covers ideas only: Kodus is AGPLv3 (plus a commercial
ee/licence) and Insi is Apache-2.0, so no code or prompts are copied.
Not proposed: Kodus's SaaS parts (web dashboard, billing, feature flags in PostHog, multi-tenant cloud) and its memory/rules storage. Incord Memory already covers memory, pinned to topics.
Where the gap is
| Area | Kodus | Insi today |
|---|---|---|
| CI gates | ~40 workflows. They include: evals on every PR, nightly and weekly; a model benchmark; contract tests; a mutation-testing gate on changed files; e2e across self-hosted and cloud; env-drift and permissions-matrix checks; release-freeze; PR-title lint. | 2 workflows: fmt + clippy + test, and release. |
| Eval discipline | 13 written rules in evals/README.md (run production code, prove it can fail, floors come from runs, 0/1/2 exit codes, record tokens and time for every model call), plus pnpm eval:new to scaffold a new eval. |
Strong core measurement (hidden tests, validation, held-out halves), but no written rules for adding a measurement and no scaffolding. |
| Regression watch | Nightly and weekly runs with confirming re-runs and calibrated floors; alerts are rare and true. | None after merge. |
| Test layers | unit / integration / contract / e2e / mutation / fixtures. | Unit and one integration file (112 tests). |
| Forges | GitHub, GitLab, Bitbucket, Azure DevOps and Forgejo behind one interface with capability probes. | GitHub via gh only. |
| Release | Changelog generated and published from a feature catalog (release/features.yaml with stages draft → GA); self-hosted promote; preview deploys. |
Release workflow builds binaries; npm/pip packages ready but unpublished. |
| Docs | Docs site (how to use, how to deploy, cookbook, knowledge base) in 6 languages, a voice guide. | One long README plus RELEASING.md. |
| Distribution | Docker images, compose files for each deployment mode, a CLI with pre-push hook and --fail-on. |
Build from source. |
| Model choice | BYOK for any provider, token-usage panel. | Any headless worker; cost only if [prices] is set. |
| Privacy | Telemetry opt-out with a documented payload. | No telemetry (fine). |
Roadmap
Level 1: trustworthy numbers (do first; mostly #3 §1 and §4)
- Write Insi's measurement rules in
docs/MEASUREMENT.md, adapted from Kodus's 13 rules:- every number comes from runs, never from code or PR text;
- every check is shown it can fail;
- exit codes 0 / 1 / 2;
- "one run decides nothing";
- every attempt records its tokens, time and cost.
- Implement Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.1–1.3: a confidence interval, "tie but cheaper" with guards, and "not measured".
- Fake-worker smoke in CI (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.5): a fix-applier must pass and a no-op must fail, on a tiny fixture repo committed under
crates/insi/tests/fixtures/. This makes the whole pipeline a PR gate with no model spend. - Remove the inherited git and Claude-session environment variables from child processes (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §4.1, Findings from an end-to-end run with incode as the worker (session sharing, spec-doc tax, cost line) #1).
Done when: a PR that breaks judging, hidden-test copying or token parsing goes red in CI, and insi improve can say "not measured".
Level 2: a test pyramid and gates
- Contract tests for the worker protocol: incode's
stream-jsonresult line (the fields Insi reads) is pinned in a fixture, so an incode change that breaks Insi fails here, not in a benchmark. Mirror the test in incode. - Mutation testing on changed files (
cargo-mutants --in-diff) as a non-blocking PR job first, then blocking forinsi-loopandinsi-eval, where a silent wrong verdict costs most. - e2e matrix:
insi init → mine → validate → evalwith the fake worker on Linux, macOS and Windows, plus one fixture repo per detected stack (Rust, Node/vitest, Python, Go). - Environment-drift check: every config key in code is documented in the README, and vice versa.
- PR-title / conventional-commit lint, so the changelog can be generated (Level 4).
Done when: CI has unit, contract, e2e and mutation layers, and each stack insi init detects has an e2e fixture.
Level 3: a regression watch and real benchmarks
insi watchnightly on Insi's own benchmark repos (Watchman plus one public repo per stack), with confirming re-runs and floors at mean − 2·SEM from two runs (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.6).- A weekly model benchmark: the same fair tasks across the configured worker models, published as a table in the README with seeds ≥ 3 and intervals. This replaces the current single-seed Watchman table.
- Experiment log plus generated PR bodies (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.7). Every number in a PR links to its run.
scaffold:insi bench new <name>writes a benchmark folder with a README header (what it answers, how it runs, its cost), paired controls andfloors.toml.
Done when: a drop in the merged harness is reported within a day by a confirmed run, and the README's numbers come from the weekly job.
Level 4: reach and release
- More forges: split
HubintoIntake+Forge(Recommendation: more ways in and out forinsi issue(CI fix loop, chat triage agent, email intake) #2 R0), addglab(GitLab) and Forgejo (tea) next togh, each with capability probes (supports_draft_pr(), …). - Publish the npm/pip packages and a container image (
ghcr.io/incordai/insi) with a hardened sandbox profile (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §4.2). - A CI mode that runs as a GitHub Action / GitLab component with
--fail-on, and a sample workflow forinsi watch. - Changelog from a feature catalog: a
release/features.yamlwith a stage per feature (alpha → GA), and the changelog generated from it plus merged PR titles. - A docs site (mdBook is enough) split into how to use / how to deploy / cookbook / reference, generated from the README sections. English first.
Done when: Insi installs without Rust, works on GitHub and GitLab, and has a versioned changelog and docs site.
Level 5: product loops (after the above; see #2)
- A CI fix loop, the chat triage agent and email intake (Recommendation: more ways in and out for
insi issue(CI fix loop, chat triage agent, email intake) #2). - Follow-ups that close themselves, PR comment commands and commit trailers (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §3).
- Per-memory-item effect measurement fed back to Incord Memory (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §2.1).
How to use this issue
Each level is meant to become its own milestone, with one sub-issue per numbered item. Level 1 is a prerequisite for the rest: without trustworthy numbers, every later "improvement" claim is unproven, which is the problem Insi exists to fix.
Activity
blockmandev commentedon Oct 20, 2026
Closing at the owner's request.
Level 1 is done in #6: accept on a 95% interval, guard tasks that always passed, "not measured" with exit code 2, and
docs/MEASUREMENT.md. The existing pipeline test already covers the fake-worker smoke test.Levels 2–5 aren't started: contract and mutation tests, the e2e matrix,
insi watch, the weekly benchmark, more forges, packaging and the docs site. This issue stays as their reference.Generated by Claude Code