feat: contract / definition-of-done + idempotent-write dedup + bench …

CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## [1.2.0] - 2026-05-23
+
+### feat: contract / definition-of-done + per-turn idempotent-write dedup + bench diff
+
+Three additions inspired by [jukefr/itsy](https://github.com/jukefr/itsy)
+(SmallCode's downstream Rust port):
+
+- **Contract / Definition-of-Done** — declarative per-project assertion list
+  the agent commits to up-front. The agent cannot deliver a final
+  "I'm done"-shaped response while any assertion is `pending` or `failed`.
+  Declared in MarrowScript at `marrow/contract.marrow` and implemented as a
+  hand-port at `src/session/contract.js` plus `contract_store.js` and
+  `contract_tools.js`. Five new tools — `contract_create`, `contract_status`,
+  `contract_assert_pass`, `contract_assert_fail`, `contract_assert_skip` —
+  plus a `/contract` slash command. State persists to
+  `.smallcode/contracts/<id>/state.json` with re-rendered `contract.md` and
+  `assertions.md` views and a `log.jsonl` audit trail. Disable with
+  `SMALLCODE_CONTRACT=false`. The done-guard heuristic lives in
+  `src/session/contract_guard.js`.
+- **Per-turn idempotent-write dedup** — closes the spam-loop gap where small
+  models could call `memory_remember` (or `memory_forget`) with identical
+  args 30+ times in a single turn. `PURE_TOOLS` dedup is sliding-window and
+  excludes writes; this set is per-turn, scoped to declared idempotent
+  writes, and resets between turns. Implemented in
+  `src/tools/dedup.js::IdempotentWriteSet` with hooks in
+  `bin/smallcode.js`'s `executeTool` wrapper and turn boundary. Disable with
+  `SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false`. Inspired by itsy commit 32653f3.
+- **Benchmark diff tool + BDD skill** — new `bench/diff.js` compares two
+  harness JSON outputs and exits `0` improved / `1` regressed / `2` noise
+  with optional `--json` for CI. Skill at
+  `skills/benchmark-driven-development.md` adapted from itsy's same-named
+  skill. Wired as `npm run bench:diff <baseline> <feature>`.
+
+### Verification
+
+- 46/46 new unit tests across `test/contract.test.js`,
+  `test/dedup_idempotent.test.js`, `test/bench_diff.test.js`
+- 14/14 existing SSRF guard tests still pass (60/60 total via `npm test`)
+- 11/11 E2E checks via `npm run test:e2e` against
+  `huihui-gemma-4-e4b-it-abliterated` on `http://10.0.0.20:1234/v1`. Live
+  trace: model created the contract, marked `a01` passed, tried to claim
+  done, done-guard fired (`⚠ contract guard: 1 unresolved assertion`),
+  model recovered by calling `contract_status` then `contract_assert_skip`
+  on `a02`, contract auto-completed.
+
+---
+
 ## [1.1.0] - 2026-05-23
 
 ### feat: tool-call recovery + /version command + SSRF guard hardening

README.md

@@ -185,12 +185,17 @@ Tracks which paths the model has read this session. First `write_file` to an exi
 ### Tool-Call Deduplication
 Identical pure-tool calls within a sliding window are short-circuited with a cached result instead of re-executing. Only applies to read-only tools (`read_file`, `search`, `graph_search`, etc.) — never to anything with side effects. Saves both context and latency on small models that loop. Disable with `SMALLCODE_DEDUP=false`.
 
+A second, stricter guard handles **idempotent-write tools** (`memory_remember`, `memory_forget`): identical calls in the same turn are short-circuited with `[already stored this turn]` instead of re-executing. Resets between turns rather than between sessions. Closes the spam-loop gap where small models could call `memory_remember` 30+ times with the same args. Disable with `SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false`.
+
 ### Evidence Store
 Automated capture of "what was tried, what worked, what failed" per task. Stored as searchable memory objects in the existing memory MCP module so they flow through FTS5 + staleness-decay loading on future tasks rather than always hogging context. The model learns from past sessions: it sees that `pip install` failed last time on this Python version, or that `npm test` hangs without `--run`. Disable with `SMALLCODE_EVIDENCE_DISABLE=true`.
 
 ### Plan-Then-Execute Mode
 For multi-step tasks (refactors, multi-file features, multi-imperative prompts), SmallCode asks the model to emit a numbered plan FIRST, then re-injects that plan as an anchor on subsequent turns. Reduces drift on long traces — the model can't "forget" step 3 by the time it finishes step 1. Heuristic-based — simple tasks like "create hello.py" don't trigger planning. Configure with `SMALLCODE_PLAN=true|false`.
 
+### Contract / Definition of Done
+For tasks where "done" should be hard-fail rather than self-reported, SmallCode supports per-project **contracts** — a declarative list of testable assertions the model commits to up-front. The agent cannot deliver a final "I'm done"-shaped response while any assertion remains `pending` or `failed`. The model uses `contract_create` to declare assertions, `contract_assert_pass` / `contract_assert_fail` / `contract_assert_skip` to record progress with command-line evidence, and `contract_status` to inspect remaining blockers. State persists to `.smallcode/contracts/<id>/` (state.json, contract.md, assertions.md, log.jsonl). Slash command `/contract` lists, activates, and aborts contracts. Inspired by [jukefr/itsy](https://github.com/jukefr/itsy)'s same-named feature. Disable the done-guard with `SMALLCODE_CONTRACT=false`.
+
 ### Snapshot & Auto-Rollback
 Before each agent turn, SmallCode opens a file snapshot checkpoint. Every `write_file` and `patch` records its pre-edit content. If validation hard-fails and all retries are exhausted, set `SMALLCODE_SNAPSHOT_AUTO_ROLLBACK=true` to automatically revert all edits in the turn back to the checkpoint state. All snapshots persisted to `.smallcode/snapshots/` for manual audit. Disable with `SMALLCODE_SNAPSHOT=false`.
 
@@ -239,6 +244,17 @@ npm run bench:polyglot
 npm run bench:tools
 ```
 
+### Benchmark Diff Tool
+Compare two harness runs (or a stored baseline against a fresh run) and get an exit-coded verdict you can use in CI: `0` improved, `1` regressed, `2` noise.
+
+```bash
+npm run bench:diff bench/baselines/main bench/baselines/feature
+# or with a custom threshold:
+node bench/diff.js bench/baselines/main bench/baselines/feature --threshold 0.05 --json
+```
+
+Reports mean reward delta, per-task pass-count moves (no task should regress), wall-clock delta, and tool-call delta. Pairs with the `benchmark-driven-development` skill at `skills/benchmark-driven-development.md` — the discipline of measure-first / change-second / measure-again before any agent-behaviour change ships. Adapted from [jukefr/itsy](https://github.com/jukefr/itsy).
+
 
 ## Commands
 
@@ -252,6 +268,7 @@ npm run bench:tools
 | `/trace` | List/show/export execution traces |
 | `/eval` | Run prompt evaluation suites |
 | `/memory` | Show working memory |
+| `/contract` | Definition-of-Done contract: list / activate / abort |
 | `/plan` | Show current task plan |
 | `/model` | Show/switch model |
 | `/profile` | Show detected model profile + routing mode |
@@ -324,6 +341,11 @@ Returns a structured `RunResult` with: response text, tool call records, files c
 | `memory_remember` | Save knowledge to memory |
 | `web_search` | Search the web via DuckDuckGo (requires `SMALLCODE_WEB_BROWSE=true`) |
 | `web_fetch` | Fetch and extract text from a URL (requires `SMALLCODE_WEB_BROWSE=true`) |
+| `contract_create` | Declare a Definition-of-Done with a list of testable assertions |
+| `contract_status` | Show the active contract: assertions, state, blockers |
+| `contract_assert_pass` | Mark an assertion passed (with command-line evidence) |
+| `contract_assert_fail` | Mark an assertion failed (with evidence) |
+| `contract_assert_skip` | Mark an assertion skipped (out of scope) |
 
 ### Web Browsing
 

bench/diff.js

@@ -0,0 +1,255 @@
+#!/usr/bin/env node
+// SmallCode — Benchmark Diff
+//
+// Compare two harness runs (or a stored baseline vs a fresh run) and decide
+// whether a change improved, regressed, or made no measurable difference.
+//
+// Usage:
+//   node bench/diff.js <baseline> <feature> [--threshold 0.02]
+//
+//   <baseline> / <feature> can be either:
+//     - a single .json file written by bench/harness.js
+//     - a directory containing one such .json (newest is used)
+//
+// Exit codes:
+//   0   improvement (mean reward delta >= +threshold AND no per-task hard regression)
+//   1   regression  (mean reward delta <= -threshold OR a task dropped to 0/N from >= 2/N)
+//   2   noise       (delta within ±threshold)
+//   3   usage / IO error
+//
+// Adapted from itsy's `.agents/skills/benchmark-driven-development/diff.py`,
+// rewritten in plain Node for SmallCode's bench harness format.
+
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+
+// ─── arg parsing ─────────────────────────────────────────────────────────────
+
+function parseArgs(argv) {
+  const args = { _: [], threshold: 0.02, json: false };
+  for (let i = 2; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === '--threshold' || a === '-t') {
+      args.threshold = parseFloat(argv[++i]);
+    } else if (a === '--json') {
+      args.json = true;
+    } else if (a === '--help' || a === '-h') {
+      args.help = true;
+    } else if (a.startsWith('--')) {
+      console.error(`unknown flag: ${a}`);
+      process.exit(3);
+    } else {
+      args._.push(a);
+    }
+  }
+  if (!Number.isFinite(args.threshold)) args.threshold = 0.02;
+  return args;
+}
+
+function usage() {
+  console.log('Usage: node bench/diff.js <baseline> <feature> [--threshold 0.02] [--json]');
+  console.log('');
+  console.log('  Each argument is a harness JSON file or a directory containing one.');
+  console.log('  Exit: 0 improved, 1 regressed, 2 noise, 3 usage/IO.');
+}
+
+// ─── result loading ──────────────────────────────────────────────────────────
+
+function loadRun(arg) {
+  if (!fs.existsSync(arg)) {
+    throw new Error(`not found: ${arg}`);
+  }
+  const stat = fs.statSync(arg);
+  let target = arg;
+  if (stat.isDirectory()) {
+    const candidates = fs.readdirSync(arg)
+      .filter((f) => f.endsWith('.json'))
+      .map((f) => path.join(arg, f))
+      .sort((a, b) => fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs);
+    if (candidates.length === 0) throw new Error(`no .json files in ${arg}`);
+    target = candidates[0];
+  }
+  const raw = fs.readFileSync(target, 'utf-8');
+  let data;
+  try {
+    data = JSON.parse(raw);
+  } catch (e) {
+    throw new Error(`malformed JSON in ${target}: ${e.message}`);
+  }
+  if (!Array.isArray(data.results)) {
+    throw new Error(`${target}: missing "results" array — not a harness output?`);
+  }
+  return { path: target, data };
+}
+
+// ─── metric extraction ───────────────────────────────────────────────────────
+
+function summarize(run) {
+  const results = run.data.results;
+  const total = results.length;
+  const passed = results.filter((r) => r.passed).length;
+  const reward = total > 0 ? passed / total : 0;
+  const totalMs = results.reduce((s, r) => s + (r.elapsedMs || 0), 0);
+  const totalToolCalls = results.reduce((s, r) => s + (r.toolCalls || 0), 0);
+  // Build a per-task pass map keyed by `id` for diffing.
+  const byId = {};
+  for (const r of results) {
+    byId[r.id] = {
+      passed: !!r.passed,
+      elapsedMs: r.elapsedMs || 0,
+      toolCalls: r.toolCalls || 0,
+      verifyError: r.verifyError || null,
+    };
+  }
+  return { passed, total, reward, totalMs, totalToolCalls, byId };
+}
+
+// ─── verdict ─────────────────────────────────────────────────────────────────
+
+const VERDICT = {
+  IMPROVED: 'IMPROVED',
+  REGRESSED: 'REGRESSED',
+  NOISE: 'NOISE',
+};
+
+function verdict(deltaReward, taskRegressions, threshold) {
+  if (taskRegressions.hard.length > 0) return VERDICT.REGRESSED;
+  if (deltaReward >= threshold) return VERDICT.IMPROVED;
+  if (deltaReward <= -threshold) return VERDICT.REGRESSED;
+  return VERDICT.NOISE;
+}
+
+function exitCodeFor(v) {
+  if (v === VERDICT.IMPROVED) return 0;
+  if (v === VERDICT.REGRESSED) return 1;
+  return 2;
+}
+
+// ─── per-task regression check ───────────────────────────────────────────────
+//
+// Mirrors itsy's "no task should drop from passing-majority to 0" rule.
+// `hard` regression: task passed in baseline, fails in feature.
+// `soft` regression: same task ID went from >=1 attempt-pass to 0 — we only
+//   have one attempt per task in the current harness, so soft == hard for now.
+
+function classifyTaskMoves(base, feat) {
+  const hard = []; // baseline pass → feature fail
+  const recovered = []; // baseline fail → feature pass
+  const allIds = new Set([...Object.keys(base.byId), ...Object.keys(feat.byId)]);
+  for (const id of allIds) {
+    const b = base.byId[id];
+    const f = feat.byId[id];
+    if (!b || !f) continue; // task added or removed — skip silently
+    if (b.passed && !f.passed) hard.push(id);
+    if (!b.passed && f.passed) recovered.push(id);
+  }
+  return { hard, recovered };
+}
+
+// ─── rendering ───────────────────────────────────────────────────────────────
+
+const C = {
+  reset: '\x1b[0m',
+  bold: '\x1b[1m',
+  dim: '\x1b[2m',
+  red: '\x1b[31m',
+  green: '\x1b[32m',
+  yellow: '\x1b[33m',
+  cyan: '\x1b[36m',
+};
+const colorize = (process.stdout.isTTY && !process.env.NO_COLOR);
+function paint(text, code) { return colorize ? `${code}${text}${C.reset}` : text; }
+
+function pct(n) { return `${(n * 100).toFixed(1)}%`; }
+function ms(n) { return `${(n / 1000).toFixed(1)}s`; }
+function signed(n, digits = 3) {
+  const s = n.toFixed(digits);
+  return n >= 0 ? `+${s}` : s;
+}
+
+function render(baseSum, featSum, moves, v, threshold, basePath, featPath) {
+  const deltaReward = featSum.reward - baseSum.reward;
+  const deltaMs = featSum.totalMs - baseSum.totalMs;
+  const deltaCalls = featSum.totalToolCalls - baseSum.totalToolCalls;
+
+  const verdictColor = v === VERDICT.IMPROVED ? C.green
+    : v === VERDICT.REGRESSED ? C.red
+    : C.yellow;
+
+  console.log('');
+  console.log(paint('  SmallCode bench diff', C.bold));
+  console.log(paint('  ──────────────────────────────────────────', C.dim));
+  console.log(`  baseline : ${paint(basePath, C.cyan)}`);
+  console.log(`  feature  : ${paint(featPath, C.cyan)}`);
+  console.log('');
+  console.log(`  reward   : ${pct(baseSum.reward)}  →  ${pct(featSum.reward)}   (Δ ${paint(signed(deltaReward), verdictColor)})`);
+  console.log(`  passed   : ${baseSum.passed}/${baseSum.total}  →  ${featSum.passed}/${featSum.total}`);
+  console.log(`  walltime : ${ms(baseSum.totalMs)}  →  ${ms(featSum.totalMs)}   (Δ ${signed(deltaMs / 1000, 1)}s)`);
+  console.log(`  toolcall : ${baseSum.totalToolCalls}  →  ${featSum.totalToolCalls}   (Δ ${signed(deltaCalls, 0)})`);
+  console.log('');
+
+  if (moves.hard.length > 0) {
+    console.log(paint(`  ✗ Regressed tasks (${moves.hard.length}):`, C.red));
+    for (const id of moves.hard) console.log(`      ${id}`);
+  }
+  if (moves.recovered.length > 0) {
+    console.log(paint(`  ✓ Recovered tasks (${moves.recovered.length}):`, C.green));
+    for (const id of moves.recovered) console.log(`      ${id}`);
+  }
+  if (moves.hard.length === 0 && moves.recovered.length === 0) {
+    console.log(paint('  no per-task moves', C.dim));
+  }
+
+  console.log('');
+  console.log(`  threshold: ±${threshold}`);
+  console.log(`  verdict  : ${paint(v, verdictColor + C.bold)}`);
+  console.log('');
+}
+
+// ─── main ────────────────────────────────────────────────────────────────────
+
+function main() {
+  const args = parseArgs(process.argv);
+  if (args.help || args._.length !== 2) {
+    usage();
+    process.exit(args.help ? 0 : 3);
+  }
+  const [baseArg, featArg] = args._;
+  let baseRun, featRun;
+  try {
+    baseRun = loadRun(baseArg);
+    featRun = loadRun(featArg);
+  } catch (e) {
+    console.error(`error: ${e.message}`);
+    process.exit(3);
+  }
+
+  const baseSum = summarize(baseRun);
+  const featSum = summarize(featRun);
+  const moves = classifyTaskMoves(baseSum, featSum);
+  const deltaReward = featSum.reward - baseSum.reward;
+  const v = verdict(deltaReward, moves, args.threshold);
+  const code = exitCodeFor(v);
+
+  if (args.json) {
+    process.stdout.write(JSON.stringify({
+      verdict: v,
+      exitCode: code,
+      threshold: args.threshold,
+      delta: { reward: deltaReward, totalMs: featSum.totalMs - baseSum.totalMs, totalToolCalls: featSum.totalToolCalls - baseSum.totalToolCalls },
+      baseline: { path: baseRun.path, passed: baseSum.passed, total: baseSum.total, reward: baseSum.reward, totalMs: baseSum.totalMs },
+      feature: { path: featRun.path, passed: featSum.passed, total: featSum.total, reward: featSum.reward, totalMs: featSum.totalMs },
+      regressed: moves.hard,
+      recovered: moves.recovered,
+    }, null, 2) + '\n');
+  } else {
+    render(baseSum, featSum, moves, v, args.threshold, baseRun.path, featRun.path);
+  }
+  process.exit(code);
+}
+
+if (require.main === module) main();
+
+module.exports = { loadRun, summarize, classifyTaskMoves, verdict, VERDICT };