Skip to content

Recommendation: a roadmap to reach kodus-ai's level of engineering maturity #4

Closed

Description

@blockmandev

Status: recommendation, not a decision. This is a gap analysis against kodustech/kodus-ai, written as a staged roadmap. #3 lists features to borrow; this issue is about the practices that put Kodus where it is, and what Insi would need to match them. It covers ideas only: Kodus is AGPLv3 (plus a commercial ee/ licence) and Insi is Apache-2.0, so no code or prompts are copied.

Not proposed: Kodus's SaaS parts (web dashboard, billing, feature flags in PostHog, multi-tenant cloud) and its memory/rules storage. Incord Memory already covers memory, pinned to topics.

Where the gap is

Area Kodus Insi today
CI gates ~40 workflows. They include: evals on every PR, nightly and weekly; a model benchmark; contract tests; a mutation-testing gate on changed files; e2e across self-hosted and cloud; env-drift and permissions-matrix checks; release-freeze; PR-title lint. 2 workflows: fmt + clippy + test, and release.
Eval discipline 13 written rules in evals/README.md (run production code, prove it can fail, floors come from runs, 0/1/2 exit codes, record tokens and time for every model call), plus pnpm eval:new to scaffold a new eval. Strong core measurement (hidden tests, validation, held-out halves), but no written rules for adding a measurement and no scaffolding.
Regression watch Nightly and weekly runs with confirming re-runs and calibrated floors; alerts are rare and true. None after merge.
Test layers unit / integration / contract / e2e / mutation / fixtures. Unit and one integration file (112 tests).
Forges GitHub, GitLab, Bitbucket, Azure DevOps and Forgejo behind one interface with capability probes. GitHub via gh only.
Release Changelog generated and published from a feature catalog (release/features.yaml with stages draft → GA); self-hosted promote; preview deploys. Release workflow builds binaries; npm/pip packages ready but unpublished.
Docs Docs site (how to use, how to deploy, cookbook, knowledge base) in 6 languages, a voice guide. One long README plus RELEASING.md.
Distribution Docker images, compose files for each deployment mode, a CLI with pre-push hook and --fail-on. Build from source.
Model choice BYOK for any provider, token-usage panel. Any headless worker; cost only if [prices] is set.
Privacy Telemetry opt-out with a documented payload. No telemetry (fine).

Roadmap

Level 1: trustworthy numbers (do first; mostly #3 §1 and §4)

  1. Write Insi's measurement rules in docs/MEASUREMENT.md, adapted from Kodus's 13 rules:
    • every number comes from runs, never from code or PR text;
    • every check is shown it can fail;
    • exit codes 0 / 1 / 2;
    • "one run decides nothing";
    • every attempt records its tokens, time and cost.
  2. Implement Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.1–1.3: a confidence interval, "tie but cheaper" with guards, and "not measured".
  3. Fake-worker smoke in CI (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.5): a fix-applier must pass and a no-op must fail, on a tiny fixture repo committed under crates/insi/tests/fixtures/. This makes the whole pipeline a PR gate with no model spend.
  4. Remove the inherited git and Claude-session environment variables from child processes (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §4.1, Findings from an end-to-end run with incode as the worker (session sharing, spec-doc tax, cost line) #1).

Done when: a PR that breaks judging, hidden-test copying or token parsing goes red in CI, and insi improve can say "not measured".

Level 2: a test pyramid and gates

  1. Contract tests for the worker protocol: incode's stream-json result line (the fields Insi reads) is pinned in a fixture, so an incode change that breaks Insi fails here, not in a benchmark. Mirror the test in incode.
  2. Mutation testing on changed files (cargo-mutants --in-diff) as a non-blocking PR job first, then blocking for insi-loop and insi-eval, where a silent wrong verdict costs most.
  3. e2e matrix: insi init → mine → validate → eval with the fake worker on Linux, macOS and Windows, plus one fixture repo per detected stack (Rust, Node/vitest, Python, Go).
  4. Environment-drift check: every config key in code is documented in the README, and vice versa.
  5. PR-title / conventional-commit lint, so the changelog can be generated (Level 4).

Done when: CI has unit, contract, e2e and mutation layers, and each stack insi init detects has an e2e fixture.

Level 3: a regression watch and real benchmarks

  1. insi watch nightly on Insi's own benchmark repos (Watchman plus one public repo per stack), with confirming re-runs and floors at mean − 2·SEM from two runs (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.6).
  2. A weekly model benchmark: the same fair tasks across the configured worker models, published as a table in the README with seeds ≥ 3 and intervals. This replaces the current single-seed Watchman table.
  3. Experiment log plus generated PR bodies (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §1.7). Every number in a PR links to its run.
  4. scaffold: insi bench new <name> writes a benchmark folder with a README header (what it answers, how it runs, its cost), paired controls and floors.toml.

Done when: a drop in the merged harness is reported within a day by a confirmed run, and the README's numbers come from the weekly job.

Level 4: reach and release

  1. More forges: split Hub into Intake + Forge (Recommendation: more ways in and out for insi issue (CI fix loop, chat triage agent, email intake) #2 R0), add glab (GitLab) and Forgejo (tea) next to gh, each with capability probes (supports_draft_pr(), …).
  2. Publish the npm/pip packages and a container image (ghcr.io/incordai/insi) with a hardened sandbox profile (Recommendation: ideas from kodus-ai worth adopting (measurement, issues/CI, sandbox safety) #3 §4.2).
  3. A CI mode that runs as a GitHub Action / GitLab component with --fail-on, and a sample workflow for insi watch.
  4. Changelog from a feature catalog: a release/features.yaml with a stage per feature (alpha → GA), and the changelog generated from it plus merged PR titles.
  5. A docs site (mdBook is enough) split into how to use / how to deploy / cookbook / reference, generated from the README sections. English first.

Done when: Insi installs without Rust, works on GitHub and GitLab, and has a versioned changelog and docs site.

Level 5: product loops (after the above; see #2)

How to use this issue

Each level is meant to become its own milestone, with one sub-issue per numbered item. Level 1 is a prerequisite for the rest: without trustworthy numbers, every later "improvement" claim is unproven, which is the problem Insi exists to fix.

Related: #1, #2, #3

Activity

  1. blockmandev commented on Oct 20, 2026

    @blockmandev
    MemberAuthor

    Closing at the owner's request.

    Level 1 is done in #6: accept on a 95% interval, guard tasks that always passed, "not measured" with exit code 2, and docs/MEASUREMENT.md. The existing pipeline test already covers the fake-worker smoke test.

    Levels 2–5 aren't started: contract and mutation tests, the e2e matrix, insi watch, the weekly benchmark, more forges, packaging and the docs site. This issue stays as their reference.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions