diff --git a/CHANGELOG.md b/CHANGELOG.md index c660e511..a80a9a7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,52 @@ # Changelog +## [1.2.0] - 2026-05-23 + +### feat: contract / definition-of-done + per-turn idempotent-write dedup + bench diff + +Three additions inspired by [jukefr/itsy](https://github.com/jukefr/itsy) +(SmallCode's downstream Rust port): + +- **Contract / Definition-of-Done** — declarative per-project assertion list + the agent commits to up-front. The agent cannot deliver a final + "I'm done"-shaped response while any assertion is `pending` or `failed`. + Declared in MarrowScript at `marrow/contract.marrow` and implemented as a + hand-port at `src/session/contract.js` plus `contract_store.js` and + `contract_tools.js`. Five new tools — `contract_create`, `contract_status`, + `contract_assert_pass`, `contract_assert_fail`, `contract_assert_skip` — + plus a `/contract` slash command. State persists to + `.smallcode/contracts//state.json` with re-rendered `contract.md` and + `assertions.md` views and a `log.jsonl` audit trail. Disable with + `SMALLCODE_CONTRACT=false`. The done-guard heuristic lives in + `src/session/contract_guard.js`. +- **Per-turn idempotent-write dedup** — closes the spam-loop gap where small + models could call `memory_remember` (or `memory_forget`) with identical + args 30+ times in a single turn. `PURE_TOOLS` dedup is sliding-window and + excludes writes; this set is per-turn, scoped to declared idempotent + writes, and resets between turns. Implemented in + `src/tools/dedup.js::IdempotentWriteSet` with hooks in + `bin/smallcode.js`'s `executeTool` wrapper and turn boundary. Disable with + `SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false`. Inspired by itsy commit 32653f3. +- **Benchmark diff tool + BDD skill** — new `bench/diff.js` compares two + harness JSON outputs and exits `0` improved / `1` regressed / `2` noise + with optional `--json` for CI. Skill at + `skills/benchmark-driven-development.md` adapted from itsy's same-named + skill. Wired as `npm run bench:diff `. + +### Verification + +- 46/46 new unit tests across `test/contract.test.js`, + `test/dedup_idempotent.test.js`, `test/bench_diff.test.js` +- 14/14 existing SSRF guard tests still pass (60/60 total via `npm test`) +- 11/11 E2E checks via `npm run test:e2e` against + `huihui-gemma-4-e4b-it-abliterated` on `http://10.0.0.20:1234/v1`. Live + trace: model created the contract, marked `a01` passed, tried to claim + done, done-guard fired (`⚠ contract guard: 1 unresolved assertion`), + model recovered by calling `contract_status` then `contract_assert_skip` + on `a02`, contract auto-completed. + +--- + ## [1.1.0] - 2026-05-23 ### feat: tool-call recovery + /version command + SSRF guard hardening diff --git a/README.md b/README.md index db1bf3c4..e3eefd3a 100644 --- a/README.md +++ b/README.md @@ -185,12 +185,17 @@ Tracks which paths the model has read this session. First `write_file` to an exi ### Tool-Call Deduplication Identical pure-tool calls within a sliding window are short-circuited with a cached result instead of re-executing. Only applies to read-only tools (`read_file`, `search`, `graph_search`, etc.) — never to anything with side effects. Saves both context and latency on small models that loop. Disable with `SMALLCODE_DEDUP=false`. +A second, stricter guard handles **idempotent-write tools** (`memory_remember`, `memory_forget`): identical calls in the same turn are short-circuited with `[already stored this turn]` instead of re-executing. Resets between turns rather than between sessions. Closes the spam-loop gap where small models could call `memory_remember` 30+ times with the same args. Disable with `SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false`. + ### Evidence Store Automated capture of "what was tried, what worked, what failed" per task. Stored as searchable memory objects in the existing memory MCP module so they flow through FTS5 + staleness-decay loading on future tasks rather than always hogging context. The model learns from past sessions: it sees that `pip install` failed last time on this Python version, or that `npm test` hangs without `--run`. Disable with `SMALLCODE_EVIDENCE_DISABLE=true`. ### Plan-Then-Execute Mode For multi-step tasks (refactors, multi-file features, multi-imperative prompts), SmallCode asks the model to emit a numbered plan FIRST, then re-injects that plan as an anchor on subsequent turns. Reduces drift on long traces — the model can't "forget" step 3 by the time it finishes step 1. Heuristic-based — simple tasks like "create hello.py" don't trigger planning. Configure with `SMALLCODE_PLAN=true|false`. +### Contract / Definition of Done +For tasks where "done" should be hard-fail rather than self-reported, SmallCode supports per-project **contracts** — a declarative list of testable assertions the model commits to up-front. The agent cannot deliver a final "I'm done"-shaped response while any assertion remains `pending` or `failed`. The model uses `contract_create` to declare assertions, `contract_assert_pass` / `contract_assert_fail` / `contract_assert_skip` to record progress with command-line evidence, and `contract_status` to inspect remaining blockers. State persists to `.smallcode/contracts//` (state.json, contract.md, assertions.md, log.jsonl). Slash command `/contract` lists, activates, and aborts contracts. Inspired by [jukefr/itsy](https://github.com/jukefr/itsy)'s same-named feature. Disable the done-guard with `SMALLCODE_CONTRACT=false`. + ### Snapshot & Auto-Rollback Before each agent turn, SmallCode opens a file snapshot checkpoint. Every `write_file` and `patch` records its pre-edit content. If validation hard-fails and all retries are exhausted, set `SMALLCODE_SNAPSHOT_AUTO_ROLLBACK=true` to automatically revert all edits in the turn back to the checkpoint state. All snapshots persisted to `.smallcode/snapshots/` for manual audit. Disable with `SMALLCODE_SNAPSHOT=false`. @@ -239,6 +244,17 @@ npm run bench:polyglot npm run bench:tools ``` +### Benchmark Diff Tool +Compare two harness runs (or a stored baseline against a fresh run) and get an exit-coded verdict you can use in CI: `0` improved, `1` regressed, `2` noise. + +```bash +npm run bench:diff bench/baselines/main bench/baselines/feature +# or with a custom threshold: +node bench/diff.js bench/baselines/main bench/baselines/feature --threshold 0.05 --json +``` + +Reports mean reward delta, per-task pass-count moves (no task should regress), wall-clock delta, and tool-call delta. Pairs with the `benchmark-driven-development` skill at `skills/benchmark-driven-development.md` — the discipline of measure-first / change-second / measure-again before any agent-behaviour change ships. Adapted from [jukefr/itsy](https://github.com/jukefr/itsy). + ## Commands @@ -252,6 +268,7 @@ npm run bench:tools | `/trace` | List/show/export execution traces | | `/eval` | Run prompt evaluation suites | | `/memory` | Show working memory | +| `/contract` | Definition-of-Done contract: list / activate / abort | | `/plan` | Show current task plan | | `/model` | Show/switch model | | `/profile` | Show detected model profile + routing mode | @@ -324,6 +341,11 @@ Returns a structured `RunResult` with: response text, tool call records, files c | `memory_remember` | Save knowledge to memory | | `web_search` | Search the web via DuckDuckGo (requires `SMALLCODE_WEB_BROWSE=true`) | | `web_fetch` | Fetch and extract text from a URL (requires `SMALLCODE_WEB_BROWSE=true`) | +| `contract_create` | Declare a Definition-of-Done with a list of testable assertions | +| `contract_status` | Show the active contract: assertions, state, blockers | +| `contract_assert_pass` | Mark an assertion passed (with command-line evidence) | +| `contract_assert_fail` | Mark an assertion failed (with evidence) | +| `contract_assert_skip` | Mark an assertion skipped (out of scope) | ### Web Browsing diff --git a/bench/diff.js b/bench/diff.js new file mode 100644 index 00000000..f17182ef --- /dev/null +++ b/bench/diff.js @@ -0,0 +1,255 @@ +#!/usr/bin/env node +// SmallCode — Benchmark Diff +// +// Compare two harness runs (or a stored baseline vs a fresh run) and decide +// whether a change improved, regressed, or made no measurable difference. +// +// Usage: +// node bench/diff.js [--threshold 0.02] +// +// / can be either: +// - a single .json file written by bench/harness.js +// - a directory containing one such .json (newest is used) +// +// Exit codes: +// 0 improvement (mean reward delta >= +threshold AND no per-task hard regression) +// 1 regression (mean reward delta <= -threshold OR a task dropped to 0/N from >= 2/N) +// 2 noise (delta within ±threshold) +// 3 usage / IO error +// +// Adapted from itsy's `.agents/skills/benchmark-driven-development/diff.py`, +// rewritten in plain Node for SmallCode's bench harness format. + +'use strict'; + +const fs = require('fs'); +const path = require('path'); + +// ─── arg parsing ───────────────────────────────────────────────────────────── + +function parseArgs(argv) { + const args = { _: [], threshold: 0.02, json: false }; + for (let i = 2; i < argv.length; i++) { + const a = argv[i]; + if (a === '--threshold' || a === '-t') { + args.threshold = parseFloat(argv[++i]); + } else if (a === '--json') { + args.json = true; + } else if (a === '--help' || a === '-h') { + args.help = true; + } else if (a.startsWith('--')) { + console.error(`unknown flag: ${a}`); + process.exit(3); + } else { + args._.push(a); + } + } + if (!Number.isFinite(args.threshold)) args.threshold = 0.02; + return args; +} + +function usage() { + console.log('Usage: node bench/diff.js [--threshold 0.02] [--json]'); + console.log(''); + console.log(' Each argument is a harness JSON file or a directory containing one.'); + console.log(' Exit: 0 improved, 1 regressed, 2 noise, 3 usage/IO.'); +} + +// ─── result loading ────────────────────────────────────────────────────────── + +function loadRun(arg) { + if (!fs.existsSync(arg)) { + throw new Error(`not found: ${arg}`); + } + const stat = fs.statSync(arg); + let target = arg; + if (stat.isDirectory()) { + const candidates = fs.readdirSync(arg) + .filter((f) => f.endsWith('.json')) + .map((f) => path.join(arg, f)) + .sort((a, b) => fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs); + if (candidates.length === 0) throw new Error(`no .json files in ${arg}`); + target = candidates[0]; + } + const raw = fs.readFileSync(target, 'utf-8'); + let data; + try { + data = JSON.parse(raw); + } catch (e) { + throw new Error(`malformed JSON in ${target}: ${e.message}`); + } + if (!Array.isArray(data.results)) { + throw new Error(`${target}: missing "results" array — not a harness output?`); + } + return { path: target, data }; +} + +// ─── metric extraction ─────────────────────────────────────────────────────── + +function summarize(run) { + const results = run.data.results; + const total = results.length; + const passed = results.filter((r) => r.passed).length; + const reward = total > 0 ? passed / total : 0; + const totalMs = results.reduce((s, r) => s + (r.elapsedMs || 0), 0); + const totalToolCalls = results.reduce((s, r) => s + (r.toolCalls || 0), 0); + // Build a per-task pass map keyed by `id` for diffing. + const byId = {}; + for (const r of results) { + byId[r.id] = { + passed: !!r.passed, + elapsedMs: r.elapsedMs || 0, + toolCalls: r.toolCalls || 0, + verifyError: r.verifyError || null, + }; + } + return { passed, total, reward, totalMs, totalToolCalls, byId }; +} + +// ─── verdict ───────────────────────────────────────────────────────────────── + +const VERDICT = { + IMPROVED: 'IMPROVED', + REGRESSED: 'REGRESSED', + NOISE: 'NOISE', +}; + +function verdict(deltaReward, taskRegressions, threshold) { + if (taskRegressions.hard.length > 0) return VERDICT.REGRESSED; + if (deltaReward >= threshold) return VERDICT.IMPROVED; + if (deltaReward <= -threshold) return VERDICT.REGRESSED; + return VERDICT.NOISE; +} + +function exitCodeFor(v) { + if (v === VERDICT.IMPROVED) return 0; + if (v === VERDICT.REGRESSED) return 1; + return 2; +} + +// ─── per-task regression check ─────────────────────────────────────────────── +// +// Mirrors itsy's "no task should drop from passing-majority to 0" rule. +// `hard` regression: task passed in baseline, fails in feature. +// `soft` regression: same task ID went from >=1 attempt-pass to 0 — we only +// have one attempt per task in the current harness, so soft == hard for now. + +function classifyTaskMoves(base, feat) { + const hard = []; // baseline pass → feature fail + const recovered = []; // baseline fail → feature pass + const allIds = new Set([...Object.keys(base.byId), ...Object.keys(feat.byId)]); + for (const id of allIds) { + const b = base.byId[id]; + const f = feat.byId[id]; + if (!b || !f) continue; // task added or removed — skip silently + if (b.passed && !f.passed) hard.push(id); + if (!b.passed && f.passed) recovered.push(id); + } + return { hard, recovered }; +} + +// ─── rendering ─────────────────────────────────────────────────────────────── + +const C = { + reset: '\x1b[0m', + bold: '\x1b[1m', + dim: '\x1b[2m', + red: '\x1b[31m', + green: '\x1b[32m', + yellow: '\x1b[33m', + cyan: '\x1b[36m', +}; +const colorize = (process.stdout.isTTY && !process.env.NO_COLOR); +function paint(text, code) { return colorize ? `${code}${text}${C.reset}` : text; } + +function pct(n) { return `${(n * 100).toFixed(1)}%`; } +function ms(n) { return `${(n / 1000).toFixed(1)}s`; } +function signed(n, digits = 3) { + const s = n.toFixed(digits); + return n >= 0 ? `+${s}` : s; +} + +function render(baseSum, featSum, moves, v, threshold, basePath, featPath) { + const deltaReward = featSum.reward - baseSum.reward; + const deltaMs = featSum.totalMs - baseSum.totalMs; + const deltaCalls = featSum.totalToolCalls - baseSum.totalToolCalls; + + const verdictColor = v === VERDICT.IMPROVED ? C.green + : v === VERDICT.REGRESSED ? C.red + : C.yellow; + + console.log(''); + console.log(paint(' SmallCode bench diff', C.bold)); + console.log(paint(' ──────────────────────────────────────────', C.dim)); + console.log(` baseline : ${paint(basePath, C.cyan)}`); + console.log(` feature : ${paint(featPath, C.cyan)}`); + console.log(''); + console.log(` reward : ${pct(baseSum.reward)} → ${pct(featSum.reward)} (Δ ${paint(signed(deltaReward), verdictColor)})`); + console.log(` passed : ${baseSum.passed}/${baseSum.total} → ${featSum.passed}/${featSum.total}`); + console.log(` walltime : ${ms(baseSum.totalMs)} → ${ms(featSum.totalMs)} (Δ ${signed(deltaMs / 1000, 1)}s)`); + console.log(` toolcall : ${baseSum.totalToolCalls} → ${featSum.totalToolCalls} (Δ ${signed(deltaCalls, 0)})`); + console.log(''); + + if (moves.hard.length > 0) { + console.log(paint(` ✗ Regressed tasks (${moves.hard.length}):`, C.red)); + for (const id of moves.hard) console.log(` ${id}`); + } + if (moves.recovered.length > 0) { + console.log(paint(` ✓ Recovered tasks (${moves.recovered.length}):`, C.green)); + for (const id of moves.recovered) console.log(` ${id}`); + } + if (moves.hard.length === 0 && moves.recovered.length === 0) { + console.log(paint(' no per-task moves', C.dim)); + } + + console.log(''); + console.log(` threshold: ±${threshold}`); + console.log(` verdict : ${paint(v, verdictColor + C.bold)}`); + console.log(''); +} + +// ─── main ──────────────────────────────────────────────────────────────────── + +function main() { + const args = parseArgs(process.argv); + if (args.help || args._.length !== 2) { + usage(); + process.exit(args.help ? 0 : 3); + } + const [baseArg, featArg] = args._; + let baseRun, featRun; + try { + baseRun = loadRun(baseArg); + featRun = loadRun(featArg); + } catch (e) { + console.error(`error: ${e.message}`); + process.exit(3); + } + + const baseSum = summarize(baseRun); + const featSum = summarize(featRun); + const moves = classifyTaskMoves(baseSum, featSum); + const deltaReward = featSum.reward - baseSum.reward; + const v = verdict(deltaReward, moves, args.threshold); + const code = exitCodeFor(v); + + if (args.json) { + process.stdout.write(JSON.stringify({ + verdict: v, + exitCode: code, + threshold: args.threshold, + delta: { reward: deltaReward, totalMs: featSum.totalMs - baseSum.totalMs, totalToolCalls: featSum.totalToolCalls - baseSum.totalToolCalls }, + baseline: { path: baseRun.path, passed: baseSum.passed, total: baseSum.total, reward: baseSum.reward, totalMs: baseSum.totalMs }, + feature: { path: featRun.path, passed: featSum.passed, total: featSum.total, reward: featSum.reward, totalMs: featSum.totalMs }, + regressed: moves.hard, + recovered: moves.recovered, + }, null, 2) + '\n'); + } else { + render(baseSum, featSum, moves, v, args.threshold, baseRun.path, featRun.path); + } + process.exit(code); +} + +if (require.main === module) main(); + +module.exports = { loadRun, summarize, classifyTaskMoves, verdict, VERDICT }; diff --git a/bin/commands.js b/bin/commands.js index 2ee282a2..bd455770 100644 --- a/bin/commands.js +++ b/bin/commands.js @@ -746,6 +746,63 @@ module.exports = function createCommandHandler(config, conversationHistory, impr return; } + case '/contract': { + const { getStore } = require('../src/session/contract_store'); + const { formatStatus, statusPayload } = require('../src/session/contract_tools'); + const store = getStore(process.cwd()); + const sub = parts[1]; + + if (!sub || sub === 'status') { + const payload = statusPayload(store); + console.log(''); + console.log(formatStatus(payload)); + console.log(''); + } else if (sub === 'list') { + const all = store.list(); + if (all.length === 0) { + console.log(chalk.gray(' No contracts. Use /contract create ...')); + } else { + const activeId = store.activeId(); + for (const row of all) { + const marker = row.id === activeId ? chalk.green('●') : ' '; + const ds = row.doneStatus; + console.log(` ${marker} ${chalk.cyan(row.id)} [${row.status}] ${row.title} ${chalk.gray(`(${ds.passed}/${ds.total})`)}`); + } + } + console.log(''); + } else if (sub === 'activate') { + const id = parts[2]; + if (!id) { console.log(chalk.gray(' Usage: /contract activate <id>')); } + else { + try { store.activate(id); console.log(chalk.green(` ✓ Activated ${id}`)); } + catch (e) { console.log(chalk.red(` ${e.message}`)); } + } + console.log(''); + } else if (sub === 'deactivate') { + store.deactivate(); + console.log(chalk.gray(' Active contract cleared.')); + console.log(''); + } else if (sub === 'abort') { + const reason = parts.slice(2).join(' ') || ''; + try { const c = store.abort(reason); console.log(chalk.yellow(` ⊘ Aborted ${c.id}`)); } + catch (e) { console.log(chalk.red(` ${e.message}`)); } + console.log(''); + } else { + console.log(chalk.gray(' /contract Show active contract status')); + console.log(chalk.gray(' /contract list List all contracts')); + console.log(chalk.gray(' /contract activate <id> Switch active contract')); + console.log(chalk.gray(' /contract deactivate Clear active contract')); + console.log(chalk.gray(' /contract abort <reason> Abort the active contract')); + console.log(chalk.gray('')); + console.log(chalk.gray(' Note: contracts are normally created by the agent via the')); + console.log(chalk.gray(' contract_create tool. The model can\'t deliver "done"')); + console.log(chalk.gray(' while any assertion is pending or failed.')); + console.log(''); + } + rl.prompt(); + return; + } + case '/help': console.log(''); console.log(chalk.bold(' Commands')); @@ -758,6 +815,7 @@ module.exports = function createCommandHandler(config, conversationHistory, impr console.log(` ${chalk.cyan('/git')} <cmd> ${chalk.gray('Run any git command')}`); console.log(` ${chalk.cyan('/loop')} <file> ${chalk.gray('Validate + auto-fix')}`); console.log(` ${chalk.cyan('/memory')} ${chalk.gray('View/manage project memory')}`); + console.log(` ${chalk.cyan('/contract')} ${chalk.gray('Definition-of-Done contract')}`); console.log(` ${chalk.cyan('/undo')} ${chalk.gray('Revert uncommitted changes')}`); console.log(` ${chalk.cyan('/compact')} ${chalk.gray('Trim conversation history')}`); console.log(` ${chalk.cyan('/escalation')} ${chalk.gray('View model escalation status')}`); diff --git a/bin/executor.js b/bin/executor.js index 4a9c3c9d..b4260a95 100644 --- a/bin/executor.js +++ b/bin/executor.js @@ -773,6 +773,19 @@ async function executeTool(name, args, ctx) { return { result: `Category: ${category}. Proceed with your tool call.`, category }; } + case 'contract_status': + case 'contract_create': + case 'contract_assert_pass': + case 'contract_assert_fail': + case 'contract_assert_skip': { + try { + const { executeContractTool } = require('../src/session/contract_tools'); + return await executeContractTool(name, args, { cwd }); + } catch (e) { + return { error: `${name} failed: ${e.message}` }; + } + } + default: { if (mcpClient && mcpClient.isMCPTool(name)) { const mcpResult = await mcpClient.callTool(name, args); diff --git a/bin/smallcode.js b/bin/smallcode.js index f79f449c..d3db6f34 100755 --- a/bin/smallcode.js +++ b/bin/smallcode.js @@ -407,6 +407,20 @@ async function executeTool(name, args) { if (cached) return ToolDedup.markCached(cached); } catch {} + // Per-turn idempotent-write dedup: stop spirals where the model calls + // memory_remember (or memory_forget) with the same args repeatedly within + // a single turn. PURE_TOOLS dedup doesn't cover these — they DO mutate state + // and can't be cached across turns — but inside a turn they're idempotent + // and re-calling is always wasted work. + let writeSet = null; + try { + const { getIdempotentWriteSet } = require('../src/tools/dedup'); + writeSet = getIdempotentWriteSet(); + if (writeSet.has(name, args)) { + return writeSet.shortCircuitResult(name); + } + } catch {} + const result = await _executeToolModule(name, args, { _fullscreenRef, mcpCall, @@ -419,6 +433,7 @@ async function executeTool(name, args) { }); try { if (dedup) dedup.record(name, args, result); } catch {} + try { if (writeSet) writeSet.record(name, args, result); } catch {} return result; } @@ -483,6 +498,15 @@ async function runAgentLoop(userMessage, config) { // Reset early-stop state for new turn earlyStop.newTurn(); + // Reset per-turn idempotent-write dedup. PURE_TOOLS dedup uses a sliding + // window across turns; this set is *per-turn* and stops `memory_remember` + // (and friends) from being called with identical args repeatedly in a + // single turn. See src/tools/dedup.js for the full rationale. + try { + const { newTurnIdempotentWriteSet } = require('../src/tools/dedup'); + newTurnIdempotentWriteSet(); + } catch {} + // Start trace recording for this turn traceRecorder.start(userMessage, config.model.name); @@ -1560,6 +1584,22 @@ Read the FULL file above carefully. Fix ALL errors. Use the patch tool with the // Stream the final response for better UX if (message.content) { + // Contract done-guard: if a contract is active and the model claims + // completion while assertions are still pending/failed, inject a + // [CONTRACT-GUARD] system message and continue the loop instead of + // pushing the wrap-up text. See src/session/contract_guard.js. + try { + const { checkDoneGuard } = require('../src/session/contract_guard'); + const guard = checkDoneGuard(message.content, process.cwd()); + if (guard) { + conversationHistory.push({ role: 'assistant', content: message.content }); + conversationHistory.push({ role: 'user', content: guard.injection }); + if (_fullscreenRef) _fullscreenRef.addTool('contract', 'warn', `${guard.blockers.length} blockers`); + else console.log(` \x1b[33m⚠ contract guard: ${guard.blockers.length} unresolved assertion${guard.blockers.length === 1 ? '' : 's'}\x1b[0m`); + continue; + } + } catch {} + conversationHistory.push({ role: 'assistant', content: message.content }); // Reviewer agent (Feature #18): async critique of the response when files @@ -2495,8 +2535,9 @@ async function runNonInteractive(config, prompt) { resetFileStateTracker(); } catch {} try { - const { resetDedup } = require('../src/tools/dedup'); + const { resetDedup, resetIdempotentWriteSet } = require('../src/tools/dedup'); resetDedup(); + resetIdempotentWriteSet(); } catch {} try { const { resetSnapshotManager } = require('../src/session/snapshot'); diff --git a/bin/tools.js b/bin/tools.js index a1a691d1..0f275f03 100644 --- a/bin/tools.js +++ b/bin/tools.js @@ -25,6 +25,12 @@ const TOOLS = [ { type: 'function', function: { name: 'web_fetch', description: 'Fetch and extract readable text content from a URL. Requires SMALLCODE_WEB_BROWSE=true.', parameters: { type: 'object', properties: { url: { type: 'string', description: 'URL to fetch' } }, required: ['url'] } } }, { type: 'function', function: { name: 'memory_list', description: 'List all stored memory objects. Optionally filter by type.', parameters: { type: 'object', properties: { type: { type: 'string', description: 'Filter by type: decision, workflow, gotcha, convention, context (optional)' } }, required: [] } } }, { type: 'function', function: { name: 'memory_forget', description: 'Delete a memory object by ID.', parameters: { type: 'object', properties: { id: { type: 'string', description: 'Memory object ID to delete' } }, required: ['id'] } } }, + // ─── Contract tools (Definition of Done) ───────────────────────────────── + { type: 'function', function: { name: 'contract_status', description: 'Show the active contract: assertions, their state (pending/passed/failed/skipped), and remaining blockers. Use this BEFORE claiming a task is complete.', parameters: { type: 'object', properties: {}, required: [] } } }, + { type: 'function', function: { name: 'contract_create', description: 'Create a new Definition-of-Done contract for the current task. Pass a brief and a list of testable assertions; the agent cannot deliver "done" while any assertion remains pending. Activates the new contract automatically.', parameters: { type: 'object', properties: { title: { type: 'string', description: 'Short contract title' }, brief: { type: 'string', description: 'Free-form description of the task' }, assertions: { type: 'array', items: { type: 'string' }, description: 'List of testable assertions, one per item. Phrase each as a verifiable claim (e.g. "npm test exits 0", "POST /users returns 201 for valid input").' } }, required: ['title', 'assertions'] } } }, + { type: 'function', function: { name: 'contract_assert_pass', description: 'Mark a contract assertion as passed, with command-line evidence. Use the assertion id from contract_status (e.g. "a01"). evidence should be a short (<240 char) summary of what was run and what it returned.', parameters: { type: 'object', properties: { assertion_id: { type: 'string', description: 'Assertion id (e.g. a01)' }, evidence: { type: 'string', description: 'Short summary of command output proving the assertion holds' }, command: { type: 'string', description: 'The command run (optional)' }, exit_code: { type: 'integer', description: 'Exit code of the command (optional)' } }, required: ['assertion_id'] } } }, + { type: 'function', function: { name: 'contract_assert_fail', description: 'Mark a contract assertion as failed, with evidence. Used when a check ran and the result was wrong — not for skipping checks.', parameters: { type: 'object', properties: { assertion_id: { type: 'string', description: 'Assertion id (e.g. a01)' }, evidence: { type: 'string', description: 'Short summary of why the check failed' }, command: { type: 'string', description: 'The command run (optional)' }, exit_code: { type: 'integer', description: 'Exit code of the command (optional)' } }, required: ['assertion_id', 'evidence'] } } }, + { type: 'function', function: { name: 'contract_assert_skip', description: 'Mark an assertion as skipped (not applicable in current scope). Skipped assertions count as resolved for the done-guard.', parameters: { type: 'object', properties: { assertion_id: { type: 'string', description: 'Assertion id' }, reason: { type: 'string', description: 'Why this assertion is being skipped' } }, required: ['assertion_id', 'reason'] } } }, ]; // ─── Compound Tools ────────────────────────────────────────────────────────── diff --git a/marrow/contract.marrow b/marrow/contract.marrow new file mode 100644 index 00000000..8230f33d --- /dev/null +++ b/marrow/contract.marrow @@ -0,0 +1,132 @@ +// SmallCode — Contract (Definition of Done) MarrowScript Declaration +// +// Compiles to a deterministic per-project contract runtime. Inspired by itsy's +// `crates/itsy/src/session/contract.rs` — a list of testable assertions the +// model commits to up-front, scaled for a single-session agent on a small +// quantised model. The agent works through them, marking each `passed` or +// `failed` with command-line evidence. The model can't claim "done" while any +// assertion is still `pending` — there is a hard guard. +// +// Per AGENTS.md: declared in MarrowScript so the runtime (caching, validation, +// retry) is generated for free; the JS runtime under bin/contract.js is the +// hand-port until the compiler emits Node output. + +system SmallCodeContract { + domain: cognitive_scaffold + + // ═══════════════════════════════════════════════════════════════════════════ + // ENTITIES + // ═══════════════════════════════════════════════════════════════════════════ + + entity Assertion { + owns: [ + id: string, + text: string, + // pending | passed | failed | skipped + state: string, + evidence: string, + // last command-line evidence (stringified CommandEvidence) + last_check: string + ] + } + + entity CommandEvidence { + owns: [ + command: string, + exit_code: int, + observation: string, + timestamp: string + ] + } + + entity Contract { + owns: [ + id: string, + title: string, + created_at: string, + // draft | active | completed | aborted + status: string, + brief: string, + assertions: string, + features: string + ] + } + + // ═══════════════════════════════════════════════════════════════════════════ + // PROMPT — propose assertions from a brief + // Used by /contract create when the user supplies a brief but no + // pre-written assertion list. Constrained output: numbered list, one + // assertion per line, each phrased as a verifiable claim. + // ═══════════════════════════════════════════════════════════════════════════ + + prompt propose_assertions(brief: string) { + model: TinyClassifier + template: "extension_point:tmpl_propose_assertions" + returns: string + timeout: 10s + idempotent: true + validate: schema_only + on_invalid: retry + retry: { max_attempts: 2, backoff: fixed, interval: 200ms } + cache: { key: hash(brief), ttl: 30m } + } + + extension_point tmpl_propose_assertions(brief: string) { + returns: string + stable: true + } + + // ═══════════════════════════════════════════════════════════════════════════ + // CAPABILITIES — declarative state transitions + // ═══════════════════════════════════════════════════════════════════════════ + + capability create_contract(title: string, brief: string) { + cognition: semantic_slice using { + task: brief, + max_files: 4, + hop_depth: 1 + } + returns: Contract + sync: transactional + } + + capability mark_assertion(contract_id: string, assertion_id: string, state: string) { + cognition: semantic_slice using { + task: assertion_id, + max_files: 0, + hop_depth: 0 + } + returns: string + sync: transactional + } + + capability check_done(contract_id: string) { + cognition: semantic_slice using { + task: contract_id, + max_files: 0, + hop_depth: 0 + } + returns: string + sync: eventual + idempotent: true + } + + // ═══════════════════════════════════════════════════════════════════════════ + // POLICY — done-guard + // The agent cannot deliver a final assistant message claiming the work is + // complete while any assertion is still `pending` or `failed`. This is the + // hard-fail upgrade over the existing snapshot/auto-rollback feature. + // ═══════════════════════════════════════════════════════════════════════════ + + policy done_guard { + audit: true + cost_budgets: [ + { + scope: per_feature: "check_done" + window: 1m + cap_calls: 30 + on_exceeded: { action: error, code: "CONTRACT_CHECK_RATE_LIMIT" } + } + ] + } +} diff --git a/package.json b/package.json index c8ba4742..82762120 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "smallcode", - "version": "1.1.0", + "version": "1.2.0", "description": "AI coding agent optimized for small LLMs (8B-35B parameters)", "main": "src/api/index.js", "bin": { @@ -17,7 +17,10 @@ "bench:smoke": "node bench/harness.js --suite smoke", "bench:polyglot": "node bench/harness.js --suite polyglot-mini", "bench:tools": "node bench/harness.js --suite tool-use", - "bench:list": "node bench/harness.js --list" + "bench:list": "node bench/harness.js --list", + "bench:diff": "node bench/diff.js", + "test": "node --test test/*.test.js", + "test:e2e": "node test/e2e_smoke.js" }, "dependencies": { "bonescript-compiler": "0.14.0", diff --git a/skills/benchmark-driven-development.md b/skills/benchmark-driven-development.md new file mode 100644 index 00000000..aad4d948 --- /dev/null +++ b/skills/benchmark-driven-development.md @@ -0,0 +1,91 @@ +--- +name: benchmark-driven-development +trigger: manual +keywords: [bench, benchmark, regression, performance, scoreboard, prompt-tuning, knob] +--- + +# Benchmark-Driven Development + +**No agent-behaviour change ships without a measured before/after.** Vibes are +not evidence. "It should help" is not a result. A change is allowed to land +only if a side-by-side run shows it moved the metric we care about — or if it +didn't, the change is reverted or kept behind a flag that defaults off. + +Adapted from itsy's [`benchmark-driven-development`](https://github.com/jukefr/itsy/blob/main/.agents/skills/benchmark-driven-development/SKILL.md) +skill, scoped to SmallCode's bench harness. + +## Use when + +- About to commit any change that could shift the agent's decision making: + new tool, schema change, prompt edit, system-prompt section, plan-tracker + tweak, dedup rule, max-tokens heuristic, model-client retry logic, new + config flag. +- A change "feels obviously right" — that's exactly when measurement was + skipped. +- Re-tuning an existing knob (bumping `SMALLCODE_THINKING_BUDGET` from 2000 + to 4000, widening `SMALLCODE_DEDUP_WINDOW`, etc.). +- Comparing two implementations of the same thing. + +**Don't use for:** pure bug fixes that match an existing test, dependency +bumps, doc-only changes, internal refactors that don't change behaviour. + +## The flow + +1. **Pin a baseline commit.** `git rev-parse HEAD` before touching anything. +2. **Run a baseline benchmark.** Pick one of the suites: + - `npm run bench:smoke` — 5 trivial tasks, ~30 s + - `npm run bench:polyglot` — 19 tasks across Python / JS / TS / Bash / Markdown / JSON + - `npm run bench:tools` — 10 multi-step tool sequencing tasks +3. **Snapshot results.** Each suite writes to `.smallcode/benchmarks/`. + Copy that dir into `bench/baselines/<short-name>/` — that's your reference. +4. **Implement the change.** No rush — the baseline is now ground truth. +5. **Run the same suite on the feature branch.** Same model, same tasks. + Snapshot under `bench/baselines/<feature-name>/`. +6. **Compare.** `node bench/diff.js bench/baselines/<base> bench/baselines/<feature>`. + It prints mean reward delta, per-task pass-count diff, wall-clock delta, + and a verdict line. Exit code: `0` improvement, `1` regression, `2` noise. +7. **Decide.** Pass → commit. Fail → revert or gate behind a default-off flag. + Mixed → write the trade-off in the commit body so future-you can re-evaluate. + +## What counts as moving the right way + +| Metric | Good | Suspicious | Bad | +|---|---|---|---| +| Mean reward | +0.03 or more | ±0.01 (noise) | regression | +| Per-task pass count | no task regresses | one task swings −1 | any task drops to 0/N from ≥2/N | +| Wall clock | within ±15% | +30–50% | +2× | +| Cost ($) | within ±10% | +20% | +50% | + +A 0.01 delta on a 10-task suite is noise. Bump attempts (`--attempts 5`) if +you need a tighter estimate. + +## Common rationalizations + +| Excuse | Reality | +|---|---| +| "It's just a prompt tweak, no need to bench" | Every prompt tweak in this codebase has moved scores ±10% in *both* directions. You can't predict the sign. Measure. | +| "I'll bench later when there's a slow moment" | Later = never. A baseline run is ~30 s for smoke. Do it before you start coding. | +| "It only affects feature X, not other tasks" | Cross-cutting changes (tool list, system prompt, plan tracker) touch every task. | +| "I'll just run one task to spot-check" | n=1 is the failure mode of this whole genre of changes. Run the suite. | +| "The baseline from last week is fine" | Baselines drift with model swaps, llama.cpp upgrades, dependency bumps. Re-baseline from the same commit you're branching from. | +| "It's obviously an improvement — look at the agent log" | One log is a single sample. The model is high-variance at IQ2. Use the verifier-pass numbers. | +| "Measurement adds friction to iteration" | Iteration without measurement is sliding sideways. The friction is the point. | + +## Red flags — stop and run the baseline + +- About to type `git commit` and there's no `bench/baselines/*` dir paired with + this branch. +- Haven't run a bench in the last hour but the diff is non-trivial. +- The PR description says "should improve X" without a number. +- Tuning a knob with no baseline to compare against. +- The change is "to fix a single failure I saw" — without checking it doesn't + break the other N. + +## SmallCode tips + +- Set `SMALLCODE_MODEL` and `SMALLCODE_BASE_URL` consistently across baseline + and feature runs. +- Use `--run` or `--ci` flags on test runners when applicable so jobs don't + hang in watch mode. +- After a green diff, `memory_remember` type `workflow` with the bench command, + the model used, and the deltas — future runs benefit. diff --git a/src/session/contract.js b/src/session/contract.js new file mode 100644 index 00000000..4e384350 --- /dev/null +++ b/src/session/contract.js @@ -0,0 +1,266 @@ +// SmallCode — Contract (Definition of Done) +// +// Compiled hand-port of marrow/contract.marrow. +// +// A contract is a list of testable **assertions** the model commits to up-front. +// The agent works through them, marking each `passed` or `failed` with command- +// line evidence. The model cannot deliver a final "I'm done" message while any +// assertion is still `pending` — there's a guard in bin/smallcode.js that +// refuses such final responses (see done_guard below). +// +// Layout on disk (per project, per contract): +// +// .smallcode/contracts/ +// .active <contract-id> the agent is currently working on +// <contract-id>/ +// contract.md proposal / brief (human-readable) +// assertions.md rendered assertion list (human-readable) +// state.json canonical machine-readable state (source of truth) +// log.jsonl append-only event log +// +// `state.json` is authoritative; `.md` files are re-rendered on each write so +// human readers see current state without needing a separate viewer. +// +// Inspired by itsy's `crates/itsy/src/session/contract.rs` (Rust). This is the +// JS implementation tuned for SmallCode's model loop. +// +// Configuration: +// SMALLCODE_CONTRACT=false disable the guard entirely (still allows /contract commands) +// SMALLCODE_CONTRACT_DIR=<path> override .smallcode/contracts location + +'use strict'; + +const fs = require('fs'); +const path = require('path'); +const crypto = require('crypto'); + +// ─── Constants ─────────────────────────────────────────────────────────────── + +const STATES = Object.freeze({ + PENDING: 'pending', + PASSED: 'passed', + FAILED: 'failed', + SKIPPED: 'skipped', +}); + +const STATUSES = Object.freeze({ + DRAFT: 'draft', + ACTIVE: 'active', + COMPLETED: 'completed', + ABORTED: 'aborted', +}); + +const FILE_MODE = 0o600; + +// ─── Paths ─────────────────────────────────────────────────────────────────── + +function contractsDir(cwd) { + if (process.env.SMALLCODE_CONTRACT_DIR) return process.env.SMALLCODE_CONTRACT_DIR; + return path.join(cwd || process.cwd(), '.smallcode', 'contracts'); +} + +function activeFile(cwd) { return path.join(contractsDir(cwd), '.active'); } +function contractDir(id, cwd) { return path.join(contractsDir(cwd), id); } +function statePath(id, cwd) { return path.join(contractDir(id, cwd), 'state.json'); } +function logPath(id, cwd) { return path.join(contractDir(id, cwd), 'log.jsonl'); } + +// ─── ID generation ─────────────────────────────────────────────────────────── +// +// Time-descending IDs so most-recent sorts first lexicographically. Same scheme +// as session/persistence.js. + +function newContractId() { + const t = (Number.MAX_SAFE_INTEGER - Date.now()).toString(36); + const r = crypto.randomBytes(3).toString('hex'); + return `${t}-${r}`; +} + +function newAssertionId(index) { + return `a${String(index + 1).padStart(2, '0')}`; +} + +// ─── Atomic writes ─────────────────────────────────────────────────────────── + +function writeAtomic(filePath, content) { + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + const tmp = filePath + '.' + crypto.randomBytes(4).toString('hex') + '.tmp'; + fs.writeFileSync(tmp, content, { mode: FILE_MODE }); + fs.renameSync(tmp, filePath); +} + +function appendJsonl(filePath, obj) { + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.appendFileSync(filePath, JSON.stringify(obj) + '\n', { mode: FILE_MODE }); +} + +// ─── Rendering ─────────────────────────────────────────────────────────────── + +function renderContractMd(contract) { + const lines = []; + lines.push(`# ${contract.title}`); + lines.push(''); + lines.push(`- **id**: \`${contract.id}\``); + lines.push(`- **status**: ${contract.status}`); + lines.push(`- **created**: ${contract.created_at}`); + lines.push(''); + if (contract.brief && contract.brief.trim()) { + lines.push('## Brief'); + lines.push(''); + lines.push(contract.brief.trim()); + lines.push(''); + } + return lines.join('\n'); +} + +function renderAssertionsMd(contract) { + const lines = []; + lines.push(`# Assertions for ${contract.title}`); + lines.push(''); + if (contract.assertions.length === 0) { + lines.push('_No assertions yet._'); + lines.push(''); + return lines.join('\n'); + } + for (const a of contract.assertions) { + const mark = a.state === STATES.PASSED ? '✅' + : a.state === STATES.FAILED ? '❌' + : a.state === STATES.SKIPPED ? '⊘' + : '⏳'; + lines.push(`- ${mark} **${a.id}** — ${a.text} _(${a.state})_`); + if (a.evidence) { + const trimmed = a.evidence.length > 240 ? a.evidence.slice(0, 240) + '…' : a.evidence; + lines.push(` > ${trimmed.replace(/\n/g, '\n > ')}`); + } + if (a.last_check && a.last_check.command) { + lines.push(` > _last check: \`${a.last_check.command}\` (exit ${a.last_check.exit_code})_`); + } + } + lines.push(''); + return lines.join('\n'); +} + +// ─── Core model ────────────────────────────────────────────────────────────── + +class Contract { + constructor(data) { + this.id = data.id; + this.title = data.title || ''; + this.created_at = data.created_at || new Date().toISOString(); + this.status = data.status || STATUSES.DRAFT; + this.brief = data.brief || ''; + this.assertions = (data.assertions || []).map((a) => ({ + id: a.id, + text: String(a.text || ''), + state: a.state || STATES.PENDING, + evidence: a.evidence || null, + last_check: a.last_check || null, + })); + this.features = data.features || []; + } + + toJSON() { + return { + id: this.id, + title: this.title, + created_at: this.created_at, + status: this.status, + brief: this.brief, + assertions: this.assertions, + features: this.features, + }; + } + + pending() { return this.assertions.filter((a) => a.state === STATES.PENDING); } + failed() { return this.assertions.filter((a) => a.state === STATES.FAILED); } + passed() { return this.assertions.filter((a) => a.state === STATES.PASSED); } + skipped() { return this.assertions.filter((a) => a.state === STATES.SKIPPED); } + + // The done-guard lives here. A contract is "complete" only when every + // assertion is PASSED or SKIPPED — pending and failed both block. + isDone() { + if (this.assertions.length === 0) return false; + return this.assertions.every((a) => a.state === STATES.PASSED || a.state === STATES.SKIPPED); + } + + doneStatus() { + const total = this.assertions.length; + const passed = this.passed().length; + const failed = this.failed().length; + const pending = this.pending().length; + const skipped = this.skipped().length; + return { + done: this.isDone(), + total, + passed, + failed, + pending, + skipped, + blockers: [...this.pending(), ...this.failed()].map((a) => ({ id: a.id, text: a.text, state: a.state })), + }; + } + + setAssertionState(assertionId, state, opts = {}) { + if (!Object.values(STATES).includes(state)) { + throw new Error(`invalid assertion state: ${state}`); + } + const a = this.assertions.find((x) => x.id === assertionId); + if (!a) throw new Error(`assertion not found: ${assertionId}`); + a.state = state; + if (opts.evidence !== undefined) a.evidence = opts.evidence || null; + if (opts.lastCheck !== undefined) a.last_check = opts.lastCheck || null; + return a; + } +} + +// ─── Parsing ───────────────────────────────────────────────────────────────── +// +// Accepts either a markdown bullet/numbered list or a plain newline-separated +// list. Emits a normalised assertion array. + +function parseAssertions(input) { + if (!input) return []; + if (Array.isArray(input)) { + return input + .map((s, i) => String(s || '').trim()) + .filter(Boolean) + .map((text, i) => ({ id: newAssertionId(i), text, state: STATES.PENDING, evidence: null, last_check: null })); + } + const lines = String(input).split(/\r?\n/); + const out = []; + for (const raw of lines) { + const line = raw.trim(); + if (!line) continue; + // Strip leading bullet/number markers + const cleaned = line.replace(/^[-*]\s+/, '').replace(/^\d+[.)]\s+/, '').trim(); + if (!cleaned) continue; + out.push({ + id: newAssertionId(out.length), + text: cleaned, + state: STATES.PENDING, + evidence: null, + last_check: null, + }); + } + return out; +} + +module.exports = { + // constants + STATES, + STATUSES, + // class + Contract, + // helpers + contractsDir, + activeFile, + contractDir, + statePath, + logPath, + newContractId, + newAssertionId, + writeAtomic, + appendJsonl, + renderContractMd, + renderAssertionsMd, + parseAssertions, +}; diff --git a/src/session/contract_guard.js b/src/session/contract_guard.js new file mode 100644 index 00000000..1e312ecf --- /dev/null +++ b/src/session/contract_guard.js @@ -0,0 +1,98 @@ +// SmallCode — Contract Done-Guard +// +// The hard-fail half of the Definition-of-Done feature. When a contract is +// active and any assertion is still `pending` or `failed`, the agent is not +// allowed to deliver a final "I'm done"–shaped assistant message. We don't +// edit the model's text; we recognise the shape and inject a system message +// that nudges the model to either: +// - call contract_status to see what's blocking, or +// - run/observe the right command and call contract_assert_pass. +// +// Heuristic-only — we never rewrite or block the *first* informational reply +// even if it looks like a wrap-up, because false positives hurt a lot. The +// guard only fires when the model claims completion without resolving every +// assertion. +// +// Configuration: +// SMALLCODE_CONTRACT=false skip the guard entirely + +'use strict'; + +const { getStore } = require('./contract_store'); + +// Phrases that strongly suggest the model is wrapping up. Conservative — we +// only act when at least one of these matches AND a contract is open AND +// real blockers exist. +const DONE_PATTERNS = [ + /\b(all\s+)?done\b/i, + /\btask (is\s+)?(now\s+)?(complete|completed|finished|done)\b/i, + /\b(everything|all)\s+(is\s+)?(done|working|set|complete)\b/i, + /\b(finished|completed)\s+(the\s+)?task\b/i, + /\b(implementation|feature|fix)\s+(is\s+)?(complete|done|finished)\b/i, + /\bsuccessfully\s+(implemented|completed|finished)\b/i, + /\bready\s+to\s+(ship|merge|use)\b/i, +]; + +function looksLikeDoneClaim(text) { + if (!text || typeof text !== 'string') return false; + // Skip if the text reads like a question or asks for input. + if (/[?]\s*$/.test(text.trim())) return false; + for (const re of DONE_PATTERNS) if (re.test(text)) return true; + return false; +} + +/** + * Inspect a candidate final assistant message. If it claims completion while + * the active contract still has blockers, return an injection payload the + * caller can splice into history. Otherwise return null. + * + * Caller wires in the agent loop (bin/smallcode.js): + * const guard = checkDoneGuard(message.content, cwd); + * if (guard) { + * conversationHistory.push({ role: 'assistant', content: message.content }); + * conversationHistory.push({ role: 'user', content: guard.injection }); + * continue; // re-prompt the model; do NOT break the loop + * } + */ +function checkDoneGuard(content, cwd) { + if (process.env.SMALLCODE_CONTRACT === 'false') return null; + if (!looksLikeDoneClaim(content)) return null; + + let store, c; + try { + store = getStore(cwd || process.cwd()); + c = store.active(); + } catch { + return null; + } + if (!c) return null; // no active contract → guard inactive + + const ds = c.doneStatus(); + if (ds.done) return null; // contract is fully resolved — let the response through + + const blockerLines = ds.blockers + .map((b) => ` - ${b.id} (${b.state}) ${b.text}`) + .join('\n'); + + const injection = +`[CONTRACT-GUARD] You claimed the task is complete, but the active contract "${c.title}" (${c.id}) still has unresolved assertions: + +${blockerLines} + +You cannot deliver a final response while assertions are pending or failed. Do one of the following before claiming done: + + 1. Run the right command(s) and call contract_assert_pass <id> with evidence + 2. Call contract_assert_fail <id> with the actual failure (if it really is broken) + 3. Call contract_assert_skip <id> with a reason (if it is genuinely out of scope) + 4. Call contract_status to refresh the assertion list + +If you are blocked from completing an assertion, explain the specific blocker and ask the user — do not claim completion. Disable this guard with SMALLCODE_CONTRACT=false.`; + + return { + injection, + contractId: c.id, + blockers: ds.blockers, + }; +} + +module.exports = { checkDoneGuard, looksLikeDoneClaim, DONE_PATTERNS }; diff --git a/src/session/contract_store.js b/src/session/contract_store.js new file mode 100644 index 00000000..14bcbc5d --- /dev/null +++ b/src/session/contract_store.js @@ -0,0 +1,214 @@ +// SmallCode — Contract Store +// +// File-backed CRUD layer for src/session/contract.js. Singleton per process +// because the active-contract pointer (`.active`) needs to round-trip across +// tool calls within a session without rebuilding the whole state. +// +// Public surface: +// getStore(cwd) +// .create({ title, brief, assertions }) → Contract +// .get(id) → Contract | null +// .save(contract) → void +// .list() → [{ id, title, status, doneStatus }] +// .activate(id) → void +// .deactivate() → void +// .activeId() → id | null +// .active() → Contract | null +// .markAssertion(id, state, opts) → Contract +// .abort(id, reason) → Contract +// .complete(id) → Contract + +'use strict'; + +const fs = require('fs'); +const path = require('path'); + +const { + Contract, + STATES, + STATUSES, + contractsDir, + activeFile, + contractDir, + statePath, + logPath, + newContractId, + writeAtomic, + appendJsonl, + renderContractMd, + renderAssertionsMd, + parseAssertions, +} = require('./contract'); + +class ContractStore { + constructor(cwd) { + this.cwd = cwd || process.cwd(); + } + + // ─── pointers ────────────────────────────────────────────────────────────── + + activeId() { + const p = activeFile(this.cwd); + if (!fs.existsSync(p)) return null; + const id = fs.readFileSync(p, 'utf-8').trim(); + return id || null; + } + + active() { + const id = this.activeId(); + if (!id) return null; + return this.get(id); + } + + activate(id) { + const c = this.get(id); + if (!c) throw new Error(`contract not found: ${id}`); + if (c.status === STATUSES.DRAFT) { + c.status = STATUSES.ACTIVE; + this.save(c); + } + fs.mkdirSync(contractsDir(this.cwd), { recursive: true }); + writeAtomic(activeFile(this.cwd), id); + this._log(c.id, 'activate', {}); + } + + deactivate() { + const p = activeFile(this.cwd); + if (fs.existsSync(p)) fs.unlinkSync(p); + } + + // ─── persistence ─────────────────────────────────────────────────────────── + + get(id) { + if (!id) return null; + const p = statePath(id, this.cwd); + if (!fs.existsSync(p)) return null; + try { + const raw = fs.readFileSync(p, 'utf-8'); + const data = JSON.parse(raw); + return new Contract(data); + } catch (e) { + throw new Error(`failed to load contract ${id}: ${e.message}`); + } + } + + save(contract) { + if (!(contract instanceof Contract)) contract = new Contract(contract); + const dir = contractDir(contract.id, this.cwd); + fs.mkdirSync(dir, { recursive: true }); + writeAtomic(statePath(contract.id, this.cwd), JSON.stringify(contract.toJSON(), null, 2) + '\n'); + writeAtomic(path.join(dir, 'contract.md'), renderContractMd(contract)); + writeAtomic(path.join(dir, 'assertions.md'), renderAssertionsMd(contract)); + } + + list() { + const root = contractsDir(this.cwd); + if (!fs.existsSync(root)) return []; + const ids = fs.readdirSync(root) + .filter((name) => !name.startsWith('.')) + .filter((name) => fs.statSync(path.join(root, name)).isDirectory()); + const out = []; + for (const id of ids) { + const c = this.get(id); + if (!c) continue; + out.push({ + id: c.id, + title: c.title, + status: c.status, + created_at: c.created_at, + doneStatus: c.doneStatus(), + }); + } + out.sort((a, b) => a.id.localeCompare(b.id)); + return out; + } + + // ─── lifecycle ───────────────────────────────────────────────────────────── + + create({ title, brief, assertions }) { + const id = newContractId(); + const parsed = parseAssertions(assertions); + if (parsed.length === 0) { + throw new Error('cannot create contract with zero assertions — supply at least one'); + } + const c = new Contract({ + id, + title: title || 'Untitled contract', + brief: brief || '', + assertions: parsed, + status: STATUSES.DRAFT, + }); + this.save(c); + this._log(id, 'create', { title: c.title, assertion_count: c.assertions.length }); + return c; + } + + markAssertion(assertionId, state, opts = {}) { + const id = opts.contractId || this.activeId(); + if (!id) throw new Error('no active contract — run /contract activate <id> first'); + const c = this.get(id); + if (!c) throw new Error(`active contract missing on disk: ${id}`); + c.setAssertionState(assertionId, state, opts); + this.save(c); + this._log(c.id, 'mark', { assertion_id: assertionId, state, evidence: opts.evidence ? opts.evidence.slice(0, 200) : null }); + // Auto-complete when every assertion is done + if (c.isDone() && c.status !== STATUSES.COMPLETED) { + c.status = STATUSES.COMPLETED; + this.save(c); + this._log(c.id, 'auto_complete', {}); + } + return c; + } + + abort(reason, opts = {}) { + const id = opts.contractId || this.activeId(); + if (!id) throw new Error('no active contract'); + const c = this.get(id); + if (!c) throw new Error(`contract missing: ${id}`); + c.status = STATUSES.ABORTED; + this.save(c); + this._log(c.id, 'abort', { reason: reason || '' }); + return c; + } + + complete(opts = {}) { + const id = opts.contractId || this.activeId(); + if (!id) throw new Error('no active contract'); + const c = this.get(id); + if (!c) throw new Error(`contract missing: ${id}`); + if (!c.isDone()) { + const status = c.doneStatus(); + throw new Error( + `cannot complete: ${status.pending} pending, ${status.failed} failed. ` + + `blockers: ${status.blockers.map((b) => b.id).join(', ')}`, + ); + } + c.status = STATUSES.COMPLETED; + this.save(c); + this._log(c.id, 'complete', {}); + return c; + } + + _log(id, event, data) { + try { + appendJsonl(logPath(id, this.cwd), { + ts: new Date().toISOString(), + event, + ...data, + }); + } catch { + // Logging is best-effort. Don't fail state mutations on log write errors. + } + } +} + +let _instance = null; +function getStore(cwd) { + if (!_instance || (cwd && _instance.cwd !== cwd)) { + _instance = new ContractStore(cwd); + } + return _instance; +} +function resetStore() { _instance = null; } + +module.exports = { ContractStore, getStore, resetStore, STATES, STATUSES }; diff --git a/src/session/contract_tools.js b/src/session/contract_tools.js new file mode 100644 index 00000000..94a661d6 --- /dev/null +++ b/src/session/contract_tools.js @@ -0,0 +1,143 @@ +// SmallCode — Contract tool dispatch +// +// Splits the contract-related tool branches out of bin/executor.js so the +// executor stays under the 600-line guideline. One entry point: +// `executeContractTool(name, args, { cwd })`. + +'use strict'; + +const { getStore, STATES, STATUSES } = require('./contract_store'); + +// Build the standard "contract status" payload — used by contract_status +// directly and appended to every other contract-tool response so the model +// always sees current state without a follow-up call. + +function statusPayload(store) { + const c = store.active(); + if (!c) { + const all = store.list(); + if (all.length === 0) { + return { + active: null, + contracts: [], + message: 'No contracts on disk. Use contract_create to declare a Definition of Done.', + }; + } + return { + active: null, + contracts: all.map((row) => ({ id: row.id, title: row.title, status: row.status })), + message: 'No active contract. Use /contract activate <id> to set one.', + }; + } + const ds = c.doneStatus(); + return { + active: c.id, + title: c.title, + status: c.status, + summary: `${ds.passed}/${ds.total} passed (${ds.failed} failed, ${ds.pending} pending, ${ds.skipped} skipped)`, + done: ds.done, + assertions: c.assertions.map((a) => ({ id: a.id, text: a.text, state: a.state })), + blockers: ds.blockers, + }; +} + +function formatStatus(payload) { + if (!payload.active) { + if (payload.contracts && payload.contracts.length > 0) { + const list = payload.contracts.map((c) => ` - ${c.id} [${c.status}] ${c.title}`).join('\n'); + return `${payload.message}\n${list}`; + } + return payload.message; + } + const lines = []; + lines.push(`Contract: ${payload.title} (${payload.active})`); + lines.push(`Status: ${payload.status} — ${payload.summary}`); + lines.push(''); + for (const a of payload.assertions) { + const mark = a.state === STATES.PASSED ? '[PASS]' + : a.state === STATES.FAILED ? '[FAIL]' + : a.state === STATES.SKIPPED ? '[SKIP]' + : '[ ]'; + lines.push(` ${mark} ${a.id} ${a.text}`); + } + if (payload.blockers && payload.blockers.length > 0) { + lines.push(''); + lines.push(`Blockers: ${payload.blockers.map((b) => b.id).join(', ')}`); + } else if (payload.done) { + lines.push(''); + lines.push('All assertions resolved — done-guard will allow final response.'); + } + return lines.join('\n'); +} + +// ─── tool branches ─────────────────────────────────────────────────────────── + +async function executeContractTool(name, args, ctx) { + const store = getStore(ctx.cwd || process.cwd()); + + switch (name) { + case 'contract_status': { + const payload = statusPayload(store); + return { result: formatStatus(payload), payload }; + } + + case 'contract_create': { + const title = String(args.title || '').trim() || 'Untitled contract'; + const brief = String(args.brief || '').trim(); + const assertions = args.assertions; + if (!assertions || (Array.isArray(assertions) && assertions.length === 0)) { + return { error: 'contract_create requires at least one assertion (array of strings).' }; + } + const c = store.create({ title, brief, assertions }); + store.activate(c.id); + const payload = statusPayload(store); + return { + result: `Created and activated contract ${c.id} with ${c.assertions.length} assertions.\n\n${formatStatus(payload)}`, + contract_id: c.id, + payload, + }; + } + + case 'contract_assert_pass': + case 'contract_assert_fail': + case 'contract_assert_skip': { + const aid = String(args.assertion_id || '').trim(); + if (!aid) return { error: `${name}: assertion_id is required.` }; + + const state = name === 'contract_assert_pass' ? STATES.PASSED + : name === 'contract_assert_fail' ? STATES.FAILED + : STATES.SKIPPED; + + const evidence = args.evidence || args.reason || ''; + const lastCheck = (args.command || args.exit_code !== undefined) + ? { + command: String(args.command || ''), + exit_code: Number.isFinite(args.exit_code) ? args.exit_code : 0, + observation: String(args.evidence || args.reason || '').slice(0, 200), + timestamp: new Date().toISOString(), + } + : null; + + let updated; + try { + updated = store.markAssertion(aid, state, { evidence, lastCheck }); + } catch (e) { + return { error: `${name}: ${e.message}` }; + } + + const payload = statusPayload(store); + const verb = state === STATES.PASSED ? 'passed' + : state === STATES.FAILED ? 'failed' + : 'skipped'; + return { + result: `Marked ${aid} as ${verb}. ${payload.summary}${updated.status === STATUSES.COMPLETED ? ' — contract complete.' : ''}`, + payload, + }; + } + + default: + return { error: `unknown contract tool: ${name}` }; + } +} + +module.exports = { executeContractTool, statusPayload, formatStatus }; diff --git a/src/tools/dedup.js b/src/tools/dedup.js index 519ac1be..11b2e7ee 100644 --- a/src/tools/dedup.js +++ b/src/tools/dedup.js @@ -40,6 +40,31 @@ const PURE_TOOLS = new Set([ 'memory_list', ]); +// Tools whose effect is *idempotent within a single turn* — calling them +// twice in the same turn with the same args is a model spiral, not real +// progress. We can't put them in PURE_TOOLS (they DO mutate state, just +// idempotently), but we want to short-circuit identical repeats per turn. +// +// Observed in the wild: small models calling memory_remember(same key) 36 +// times in a single turn until the tool-call cap kills the run. +// +// Configuration: +// SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false disable entirely +const IDEMPOTENT_WRITE_TOOLS = new Set([ + 'memory_remember', + 'memory_forget', +]); + +/** + * Compute a stable hash for (toolName, args). Same scheme used by lookup() + * — sorted-keys JSON so callers can re-derive the key without instantiating + * a ToolDedup. + */ +function idempotentWriteKey(name, args) { + const norm = JSON.stringify(args || {}, Object.keys(args || {}).sort()); + return crypto.createHash('sha1').update(name + '|' + norm).digest('hex').slice(0, 16); +} + class ToolDedup { constructor(options = {}) { this.windowSize = options.windowSize || parseInt(process.env.SMALLCODE_DEDUP_WINDOW) || 5; @@ -125,4 +150,80 @@ function getDedup() { } function resetDedup() { if (_instance) _instance.reset(); } -module.exports = { ToolDedup, getDedup, resetDedup, PURE_TOOLS }; +// ─── Per-turn idempotent-write dedup ───────────────────────────────────────── +// +// PURE_TOOLS dedup is sliding-window across N recent calls. This is a separate, +// stricter guard: identical calls to an idempotent-write tool within the same +// turn always short-circuit. Reset between turns, not between sessions. + +class IdempotentWriteSet { + constructor(options = {}) { + this.disabled = options.disable || process.env.SMALLCODE_IDEMPOTENT_WRITE_DEDUP === 'false'; + this.seen = new Set(); // hash strings + this.hits = 0; + } + + isIdempotent(name) { + return IDEMPOTENT_WRITE_TOOLS.has(name); + } + + // Returns true if this call was already executed *this turn*. + has(name, args) { + if (this.disabled) return false; + if (!this.isIdempotent(name)) return false; + return this.seen.has(idempotentWriteKey(name, args)); + } + + // Record a successful idempotent-write call. + record(name, args, result) { + if (this.disabled) return; + if (!this.isIdempotent(name)) return; + if (result && result.error) return; // don't lock out retries on error + this.seen.add(idempotentWriteKey(name, args)); + } + + // Build the canonical "skipped" result that replaces the real tool call. + shortCircuitResult(name) { + this.hits += 1; + return { + result: `[${name}: already stored this turn — skipped duplicate call]`, + _idempotentWriteSkipped: true, + }; + } + + // Call once at the start of every turn (NOT every session). + newTurn() { + this.seen.clear(); + } + + reset() { + this.seen.clear(); + this.hits = 0; + } + + stats() { + return { hits: this.hits, size: this.seen.size, disabled: this.disabled }; + } +} + +let _writeSetInstance = null; +function getIdempotentWriteSet() { + if (!_writeSetInstance) _writeSetInstance = new IdempotentWriteSet(); + return _writeSetInstance; +} +function resetIdempotentWriteSet() { if (_writeSetInstance) _writeSetInstance.reset(); } +function newTurnIdempotentWriteSet() { if (_writeSetInstance) _writeSetInstance.newTurn(); } + +module.exports = { + ToolDedup, + getDedup, + resetDedup, + PURE_TOOLS, + // Idempotent-write set — separate per-turn dedup for memory_remember etc. + IdempotentWriteSet, + IDEMPOTENT_WRITE_TOOLS, + idempotentWriteKey, + getIdempotentWriteSet, + resetIdempotentWriteSet, + newTurnIdempotentWriteSet, +}; diff --git a/test/bench_diff.test.js b/test/bench_diff.test.js new file mode 100644 index 00000000..e214e272 --- /dev/null +++ b/test/bench_diff.test.js @@ -0,0 +1,148 @@ +// Tests for bench/diff.js — verdict and metric extraction. + +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); + +const { loadRun, summarize, classifyTaskMoves, verdict, VERDICT } = require('../bench/diff'); + +const DIFF = path.join(__dirname, '..', 'bench', 'diff.js'); + +function makeRun(results) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'bench-diff-')); + const file = path.join(dir, 'run.json'); + fs.writeFileSync(file, JSON.stringify({ + runId: 'fake', + suite: 'smoke', + model: 'mock', + summary: { passed: results.filter((r) => r.passed).length, total: results.length, totalMs: 0, meanMs: 0 }, + results, + }, null, 2)); + return file; +} + +// ─── unit ─────────────────────────────────────────────────────────────────── + +test('summarize() computes pass rate, walltime, tool calls', () => { + const file = makeRun([ + { id: 'a', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 3 }, + { id: 'b', lang: 'py', passed: false, elapsedMs: 2000, toolCalls: 5 }, + { id: 'c', lang: 'py', passed: true, elapsedMs: 1500, toolCalls: 2 }, + ]); + const run = loadRun(file); + const s = summarize(run); + assert.equal(s.passed, 2); + assert.equal(s.total, 3); + assert.ok(Math.abs(s.reward - 2 / 3) < 1e-9); + assert.equal(s.totalMs, 4500); + assert.equal(s.totalToolCalls, 10); +}); + +test('classifyTaskMoves identifies hard regressions and recoveries', () => { + const baseline = makeRun([ + { id: 'pass-pass', lang: 'py', passed: true, elapsedMs: 0, toolCalls: 0 }, + { id: 'pass-fail', lang: 'py', passed: true, elapsedMs: 0, toolCalls: 0 }, + { id: 'fail-pass', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }, + { id: 'fail-fail', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }, + ]); + const feature = makeRun([ + { id: 'pass-pass', lang: 'py', passed: true, elapsedMs: 0, toolCalls: 0 }, + { id: 'pass-fail', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }, + { id: 'fail-pass', lang: 'py', passed: true, elapsedMs: 0, toolCalls: 0 }, + { id: 'fail-fail', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }, + ]); + const moves = classifyTaskMoves(summarize(loadRun(baseline)), summarize(loadRun(feature))); + assert.deepEqual(moves.hard, ['pass-fail']); + assert.deepEqual(moves.recovered, ['fail-pass']); +}); + +test('verdict — improvement above threshold', () => { + const v = verdict(0.05, { hard: [] }, 0.02); + assert.equal(v, VERDICT.IMPROVED); +}); + +test('verdict — regression below negative threshold', () => { + const v = verdict(-0.05, { hard: [] }, 0.02); + assert.equal(v, VERDICT.REGRESSED); +}); + +test('verdict — within threshold = noise', () => { + const v = verdict(0.01, { hard: [] }, 0.02); + assert.equal(v, VERDICT.NOISE); +}); + +test('verdict — hard regression overrides everything', () => { + // Even with a positive delta, a task dropping from pass→fail is a regression. + const v = verdict(0.10, { hard: ['x'] }, 0.02); + assert.equal(v, VERDICT.REGRESSED); +}); + +// ─── CLI ───────────────────────────────────────────────────────────────────── + +test('CLI exits 0 on improvement', () => { + const baseline = makeRun([ + { id: 'a', lang: 'py', passed: false, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ]); + const feature = makeRun([ + { id: 'a', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ]); + const r = spawnSync(process.execPath, [DIFF, baseline, feature, '--json'], { encoding: 'utf-8' }); + assert.equal(r.status, 0, r.stderr || r.stdout); + const out = JSON.parse(r.stdout); + assert.equal(out.verdict, 'IMPROVED'); +}); + +test('CLI exits 1 on regression', () => { + const baseline = makeRun([ + { id: 'a', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ]); + const feature = makeRun([ + { id: 'a', lang: 'py', passed: false, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ]); + const r = spawnSync(process.execPath, [DIFF, baseline, feature, '--json'], { encoding: 'utf-8' }); + assert.equal(r.status, 1); +}); + +test('CLI exits 2 on noise', () => { + const baseline = makeRun([ + { id: 'a', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ]); + const feature = makeRun([ + { id: 'a', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + { id: 'b', lang: 'py', passed: true, elapsedMs: 1100, toolCalls: 1 }, + ]); + const r = spawnSync(process.execPath, [DIFF, baseline, feature, '--json'], { encoding: 'utf-8' }); + assert.equal(r.status, 2); +}); + +test('CLI accepts a directory and picks the newest .json', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'bench-diff-dir-')); + const oldFile = path.join(dir, 'old.json'); + const newFile = path.join(dir, 'new.json'); + fs.writeFileSync(oldFile, JSON.stringify({ summary: {}, results: [{ id: 'a', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }] })); + // Sleep-equivalent: bump mtime explicitly + const future = new Date(Date.now() + 5000); + fs.writeFileSync(newFile, JSON.stringify({ summary: {}, results: [{ id: 'a', lang: 'py', passed: true, elapsedMs: 0, toolCalls: 0 }] })); + fs.utimesSync(newFile, future, future); + + const baseline = makeRun([{ id: 'a', lang: 'py', passed: false, elapsedMs: 0, toolCalls: 0 }]); + const r = spawnSync(process.execPath, [DIFF, baseline, dir, '--json'], { encoding: 'utf-8' }); + assert.equal(r.status, 0); + const out = JSON.parse(r.stdout); + assert.equal(out.feature.path, newFile); +}); + +test('CLI usage error returns exit 3', () => { + const r = spawnSync(process.execPath, [DIFF, 'only-one-arg'], { encoding: 'utf-8' }); + assert.equal(r.status, 3); +}); diff --git a/test/contract.test.js b/test/contract.test.js new file mode 100644 index 00000000..a3109826 --- /dev/null +++ b/test/contract.test.js @@ -0,0 +1,319 @@ +// E2E tests for the Contract / Definition-of-Done feature. +// +// Covers: +// - state model (Contract class invariants) +// - file-backed store (create / save / load / activate / list) +// - tool dispatch (contract_create, contract_assert_pass/fail/skip, contract_status) +// - done-guard (looksLikeDoneClaim and checkDoneGuard) + +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +const { + Contract, + STATES, + STATUSES, + parseAssertions, + newAssertionId, +} = require('../src/session/contract'); + +const { ContractStore, getStore, resetStore } = require('../src/session/contract_store'); +const { executeContractTool, statusPayload, formatStatus } = require('../src/session/contract_tools'); +const { checkDoneGuard, looksLikeDoneClaim } = require('../src/session/contract_guard'); + +// ─── helpers ──────────────────────────────────────────────────────────────── + +function freshTmpRoot() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'smallcode-contract-')); + // The store reads .smallcode/contracts under cwd. Each test uses a fresh dir. + return dir; +} + +// ─── state model ──────────────────────────────────────────────────────────── + +test('parseAssertions handles markdown bullets and numbered lists', () => { + const a1 = parseAssertions('- foo\n- bar\n* baz\n \n'); + assert.equal(a1.length, 3); + assert.equal(a1[0].text, 'foo'); + assert.equal(a1[2].text, 'baz'); + + const a2 = parseAssertions('1. one\n2) two\n3. three\n'); + assert.equal(a2.length, 3); + assert.equal(a2[1].text, 'two'); + + const a3 = parseAssertions(['x', '', 'y']); + assert.equal(a3.length, 2); +}); + +test('parseAssertions returns empty array on empty input', () => { + assert.equal(parseAssertions('').length, 0); + assert.equal(parseAssertions(null).length, 0); + assert.equal(parseAssertions(' \n ').length, 0); +}); + +test('Contract.isDone() requires every assertion resolved', () => { + const c = new Contract({ + id: newAssertionId(0), + title: 't', + assertions: [ + { id: 'a01', text: 'x', state: STATES.PENDING }, + { id: 'a02', text: 'y', state: STATES.PASSED }, + ], + }); + assert.equal(c.isDone(), false); + c.setAssertionState('a01', STATES.PASSED); + assert.equal(c.isDone(), true); +}); + +test('Contract.isDone() — skipped counts as resolved, failed does not', () => { + const c = new Contract({ + id: 'x', + title: 't', + assertions: [ + { id: 'a01', text: 'x', state: STATES.SKIPPED }, + { id: 'a02', text: 'y', state: STATES.PASSED }, + ], + }); + assert.equal(c.isDone(), true); + c.setAssertionState('a02', STATES.FAILED); + assert.equal(c.isDone(), false); +}); + +test('Contract.setAssertionState rejects invalid state', () => { + const c = new Contract({ id: 'x', title: 't', assertions: [{ id: 'a01', text: 'x', state: STATES.PENDING }] }); + assert.throws(() => c.setAssertionState('a01', 'made-up'), /invalid assertion state/); + assert.throws(() => c.setAssertionState('missing', STATES.PASSED), /assertion not found/); +}); + +// ─── store ────────────────────────────────────────────────────────────────── + +test('ContractStore.create persists and activates a contract', () => { + const cwd = freshTmpRoot(); + resetStore(); + const store = new ContractStore(cwd); + const draft = store.create({ title: 'T', brief: 'B', assertions: ['one', 'two'] }); + store.activate(draft.id); + const c = store.get(draft.id); // re-read after activation flips status + + assert.equal(c.assertions.length, 2); + assert.equal(c.status, STATUSES.ACTIVE); // activate flips DRAFT → ACTIVE + assert.equal(store.activeId(), c.id); + // Files were written + assert.ok(fs.existsSync(path.join(cwd, '.smallcode', 'contracts', c.id, 'state.json'))); + assert.ok(fs.existsSync(path.join(cwd, '.smallcode', 'contracts', c.id, 'contract.md'))); + assert.ok(fs.existsSync(path.join(cwd, '.smallcode', 'contracts', c.id, 'assertions.md'))); +}); + +test('ContractStore.create rejects empty assertion list', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + assert.throws(() => store.create({ title: 'x', brief: '', assertions: [] }), /at least one/); +}); + +test('ContractStore round-trips state.json correctly', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + const c = store.create({ title: 'roundtrip', brief: '', assertions: ['a', 'b'] }); + store.activate(c.id); + store.markAssertion('a01', STATES.PASSED, { evidence: 'cargo test passed', lastCheck: { command: 'cargo test', exit_code: 0, observation: 'ok', timestamp: 'now' } }); + + // New store instance, same cwd: should see the persisted state + const fresh = new ContractStore(cwd); + const reloaded = fresh.get(c.id); + assert.equal(reloaded.assertions[0].state, STATES.PASSED); + assert.equal(reloaded.assertions[0].evidence, 'cargo test passed'); + assert.equal(reloaded.assertions[1].state, STATES.PENDING); +}); + +test('ContractStore.markAssertion auto-completes when all resolved', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + const c = store.create({ title: 'auto', brief: '', assertions: ['a', 'b'] }); + store.activate(c.id); + store.markAssertion('a01', STATES.PASSED); + let now = store.get(c.id); + assert.equal(now.status, STATUSES.ACTIVE); + store.markAssertion('a02', STATES.PASSED); + now = store.get(c.id); + assert.equal(now.status, STATUSES.COMPLETED); +}); + +test('ContractStore.complete refuses while blockers remain', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + const c = store.create({ title: 't', brief: '', assertions: ['x', 'y'] }); + store.activate(c.id); + store.markAssertion('a01', STATES.PASSED); + assert.throws(() => store.complete(), /pending|failed|cannot complete/i); +}); + +test('ContractStore.list sorts by ID and reports doneStatus', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + store.create({ title: 'A', brief: '', assertions: ['x'] }); + store.create({ title: 'B', brief: '', assertions: ['y', 'z'] }); + const list = store.list(); + assert.equal(list.length, 2); + for (const row of list) { + assert.ok(row.doneStatus); + assert.equal(typeof row.doneStatus.total, 'number'); + } +}); + +test('ContractStore log.jsonl appends events', () => { + const cwd = freshTmpRoot(); + const store = new ContractStore(cwd); + const c = store.create({ title: 'log', brief: '', assertions: ['x'] }); + store.activate(c.id); + store.markAssertion('a01', STATES.PASSED, { evidence: 'ok' }); + const log = fs.readFileSync(path.join(cwd, '.smallcode', 'contracts', c.id, 'log.jsonl'), 'utf-8'); + const entries = log.trim().split('\n').map(JSON.parse); + // create + activate + mark + auto_complete = 4 entries + assert.ok(entries.length >= 3); + assert.equal(entries[0].event, 'create'); +}); + +// ─── tool dispatch ────────────────────────────────────────────────────────── + +test('contract_create tool creates and activates a contract', async () => { + const cwd = freshTmpRoot(); + resetStore(); + const r = await executeContractTool('contract_create', { + title: 'feature X', + brief: 'add the new endpoint', + assertions: ['route is registered', 'handler returns 201', 'tests pass'], + }, { cwd }); + assert.ok(r.contract_id); + assert.match(r.result, /Created and activated/); + assert.equal(r.payload.assertions.length, 3); +}); + +test('contract_create rejects empty assertions array', async () => { + const cwd = freshTmpRoot(); + resetStore(); + const r = await executeContractTool('contract_create', { title: 't', assertions: [] }, { cwd }); + assert.ok(r.error); +}); + +test('contract_assert_pass marks an assertion and returns updated status', async () => { + const cwd = freshTmpRoot(); + resetStore(); + await executeContractTool('contract_create', { title: 'x', assertions: ['one', 'two'] }, { cwd }); + const r = await executeContractTool('contract_assert_pass', { + assertion_id: 'a01', + evidence: 'ran cargo test, exit 0', + command: 'cargo test', + exit_code: 0, + }, { cwd }); + assert.match(r.result, /Marked a01 as passed/); + assert.equal(r.payload.assertions[0].state, STATES.PASSED); +}); + +test('contract_assert_fail records failure with evidence', async () => { + const cwd = freshTmpRoot(); + resetStore(); + await executeContractTool('contract_create', { title: 'x', assertions: ['one'] }, { cwd }); + const r = await executeContractTool('contract_assert_fail', { + assertion_id: 'a01', + evidence: 'tests failed: 2 errors', + }, { cwd }); + assert.match(r.result, /failed/); + assert.equal(r.payload.assertions[0].state, STATES.FAILED); +}); + +test('contract_assert_skip skips an assertion', async () => { + const cwd = freshTmpRoot(); + resetStore(); + await executeContractTool('contract_create', { title: 'x', assertions: ['one', 'two'] }, { cwd }); + const r = await executeContractTool('contract_assert_skip', { + assertion_id: 'a02', + reason: 'out of scope this PR', + }, { cwd }); + assert.match(r.result, /skipped/); + assert.equal(r.payload.assertions[1].state, STATES.SKIPPED); +}); + +test('contract_status with no contracts returns explanatory message', async () => { + const cwd = freshTmpRoot(); + resetStore(); + const r = await executeContractTool('contract_status', {}, { cwd }); + assert.match(r.result, /No contracts/); +}); + +test('formatStatus renders a contract overview', () => { + const fakePayload = { + active: 'abc', + title: 'feature X', + status: 'active', + summary: '1/2 passed', + done: false, + assertions: [ + { id: 'a01', text: 'one', state: STATES.PASSED }, + { id: 'a02', text: 'two', state: STATES.PENDING }, + ], + blockers: [{ id: 'a02', text: 'two', state: STATES.PENDING }], + }; + const out = formatStatus(fakePayload); + assert.match(out, /Contract: feature X/); + assert.match(out, /\[PASS\] a01/); + assert.match(out, /Blockers: a02/); +}); + +// ─── done-guard ───────────────────────────────────────────────────────────── + +test('looksLikeDoneClaim recognises wrap-up phrasing', () => { + assert.equal(looksLikeDoneClaim('All done.'), true); + assert.equal(looksLikeDoneClaim('The task is now complete.'), true); + assert.equal(looksLikeDoneClaim('Successfully implemented the feature.'), true); + assert.equal(looksLikeDoneClaim('Ready to ship.'), true); + assert.equal(looksLikeDoneClaim('Should I proceed?'), false); + assert.equal(looksLikeDoneClaim('Reading file utils.py'), false); + assert.equal(looksLikeDoneClaim(''), false); + assert.equal(looksLikeDoneClaim(null), false); +}); + +test('checkDoneGuard fires only when active contract has blockers', async () => { + const cwd = freshTmpRoot(); + resetStore(); + // No contract yet — guard should be silent + assert.equal(checkDoneGuard('All done.', cwd), null); + + await executeContractTool('contract_create', { title: 't', assertions: ['x', 'y'] }, { cwd }); + // Now there's an active contract with 2 pending blockers + const guard = checkDoneGuard('All done — task is complete.', cwd); + assert.ok(guard, 'guard should fire on a done claim with pending blockers'); + assert.match(guard.injection, /CONTRACT-GUARD/); + assert.equal(guard.blockers.length, 2); + + // After resolving them, the guard should pass + await executeContractTool('contract_assert_pass', { assertion_id: 'a01' }, { cwd }); + await executeContractTool('contract_assert_pass', { assertion_id: 'a02' }, { cwd }); + assert.equal(checkDoneGuard('All done.', cwd), null); +}); + +test('checkDoneGuard ignores non-completion text', async () => { + const cwd = freshTmpRoot(); + resetStore(); + await executeContractTool('contract_create', { title: 't', assertions: ['x'] }, { cwd }); + assert.equal(checkDoneGuard('Reading the file now.', cwd), null); + assert.equal(checkDoneGuard('What should the API return on 404?', cwd), null); +}); + +test('checkDoneGuard respects SMALLCODE_CONTRACT=false', async () => { + const cwd = freshTmpRoot(); + resetStore(); + await executeContractTool('contract_create', { title: 't', assertions: ['x'] }, { cwd }); + const prev = process.env.SMALLCODE_CONTRACT; + process.env.SMALLCODE_CONTRACT = 'false'; + try { + assert.equal(checkDoneGuard('All done.', cwd), null); + } finally { + if (prev === undefined) delete process.env.SMALLCODE_CONTRACT; + else process.env.SMALLCODE_CONTRACT = prev; + } +}); diff --git a/test/dedup_idempotent.test.js b/test/dedup_idempotent.test.js new file mode 100644 index 00000000..d41566bd --- /dev/null +++ b/test/dedup_idempotent.test.js @@ -0,0 +1,122 @@ +// Regression tests for per-turn idempotent-write dedup. +// +// Closes the gap where small models could spam memory_remember with the same +// args dozens of times in one turn (observed: 36 calls before the tool-call +// cap killed the run). Inspired by itsy commit 32653f3. + +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const { + IdempotentWriteSet, + IDEMPOTENT_WRITE_TOOLS, + idempotentWriteKey, + getIdempotentWriteSet, + resetIdempotentWriteSet, + newTurnIdempotentWriteSet, +} = require('../src/tools/dedup'); + +test('memory_remember and memory_forget are flagged idempotent', () => { + const set = new IdempotentWriteSet(); + assert.equal(set.isIdempotent('memory_remember'), true); + assert.equal(set.isIdempotent('memory_forget'), true); + assert.equal(set.isIdempotent('write_file'), false); + assert.equal(set.isIdempotent('bash'), false); +}); + +test('IDEMPOTENT_WRITE_TOOLS export contains the right tools', () => { + assert.ok(IDEMPOTENT_WRITE_TOOLS.has('memory_remember')); + assert.ok(IDEMPOTENT_WRITE_TOOLS.has('memory_forget')); + assert.ok(!IDEMPOTENT_WRITE_TOOLS.has('write_file')); +}); + +test('first call passes; second identical call is short-circuited', () => { + const set = new IdempotentWriteSet(); + const args = { type: 'context', title: 'x', content: 'y' }; + assert.equal(set.has('memory_remember', args), false); + set.record('memory_remember', args, { result: 'ok' }); + assert.equal(set.has('memory_remember', args), true); +}); + +test('different args do NOT collide', () => { + const set = new IdempotentWriteSet(); + set.record('memory_remember', { type: 'a', title: '1' }, { result: 'ok' }); + assert.equal(set.has('memory_remember', { type: 'a', title: '1' }), true); + assert.equal(set.has('memory_remember', { type: 'a', title: '2' }), false); + assert.equal(set.has('memory_remember', { type: 'b', title: '1' }), false); +}); + +test('argument key order does NOT change the hash', () => { + const k1 = idempotentWriteKey('memory_remember', { type: 'context', title: 'x' }); + const k2 = idempotentWriteKey('memory_remember', { title: 'x', type: 'context' }); + assert.equal(k1, k2); +}); + +test('non-idempotent tools are NOT recorded', () => { + const set = new IdempotentWriteSet(); + set.record('write_file', { path: 'x.txt', content: 'y' }, { result: 'ok' }); + assert.equal(set.has('write_file', { path: 'x.txt', content: 'y' }), false); +}); + +test('errored calls are NOT recorded — model can retry', () => { + const set = new IdempotentWriteSet(); + set.record('memory_remember', { title: 'x' }, { error: 'boom' }); + assert.equal(set.has('memory_remember', { title: 'x' }), false); +}); + +test('newTurn() clears the set', () => { + const set = new IdempotentWriteSet(); + set.record('memory_remember', { title: 'x' }, { result: 'ok' }); + assert.equal(set.has('memory_remember', { title: 'x' }), true); + set.newTurn(); + assert.equal(set.has('memory_remember', { title: 'x' }), false); +}); + +test('shortCircuitResult returns the canonical skipped marker', () => { + const set = new IdempotentWriteSet(); + const r = set.shortCircuitResult('memory_remember'); + assert.match(r.result, /already stored this turn/); + assert.equal(r._idempotentWriteSkipped, true); + assert.equal(set.stats().hits, 1); +}); + +test('SMALLCODE_IDEMPOTENT_WRITE_DEDUP=false disables the set', () => { + const prev = process.env.SMALLCODE_IDEMPOTENT_WRITE_DEDUP; + process.env.SMALLCODE_IDEMPOTENT_WRITE_DEDUP = 'false'; + try { + const set = new IdempotentWriteSet(); + set.record('memory_remember', { title: 'x' }, { result: 'ok' }); + assert.equal(set.has('memory_remember', { title: 'x' }), false); + } finally { + if (prev === undefined) delete process.env.SMALLCODE_IDEMPOTENT_WRITE_DEDUP; + else process.env.SMALLCODE_IDEMPOTENT_WRITE_DEDUP = prev; + } +}); + +test('singleton helpers reset/newTurn work', () => { + const a = getIdempotentWriteSet(); + const b = getIdempotentWriteSet(); + assert.strictEqual(a, b); + a.record('memory_remember', { title: 'q' }, { result: 'ok' }); + assert.equal(a.has('memory_remember', { title: 'q' }), true); + newTurnIdempotentWriteSet(); + assert.equal(a.has('memory_remember', { title: 'q' }), false); + resetIdempotentWriteSet(); +}); + +// 36-call simulation — the actual observed pattern in itsy fix-git trial 1. +test('spam loop short-circuits after first call', () => { + resetIdempotentWriteSet(); + const set = getIdempotentWriteSet(); + const args = { type: 'context', title: 'wedge', content: 'foo' }; + let executed = 0; + for (let i = 0; i < 36; i++) { + if (set.has('memory_remember', args)) continue; + executed += 1; + set.record('memory_remember', args, { result: 'ok' }); + } + assert.equal(executed, 1, 'only the first call should run; the other 35 are short-circuited'); + resetIdempotentWriteSet(); +}); diff --git a/test/e2e_smoke.js b/test/e2e_smoke.js new file mode 100644 index 00000000..ef644603 --- /dev/null +++ b/test/e2e_smoke.js @@ -0,0 +1,256 @@ +#!/usr/bin/env node +// SmallCode — E2E smoke for the three new features +// +// Drives the real agent (huihui-gemma-4-e4b-it-abliterated on +// http://10.0.0.20:1234/v1) through scenarios that exercise: +// 1. Per-turn idempotent-write dedup — tries to make the model spam +// memory_remember; the runtime must short-circuit duplicates. +// 2. Contract create + done-guard — creates a contract, leaves an +// assertion pending, asks the model to claim done; the guard must +// catch the wrap-up and reject the final response. +// 3. bench/diff.js — already covered by unit tests +// via spawnSync; we re-run the smoke unit tests here for parity. +// +// Run with: node test/e2e_smoke.js +// +// Reads SMALLCODE_MODEL / SMALLCODE_BASE_URL from .env (loaded by the agent +// itself). No assertions are silently optional — the script exits 1 on +// failure. + +'use strict'; + +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { spawn, spawnSync } = require('node:child_process'); + +const ROOT = path.resolve(__dirname, '..'); +const SMALLCODE_BIN = path.join(ROOT, 'bin', 'smallcode.js'); +const MODEL = process.env.SMALLCODE_MODEL || 'huihui-gemma-4-e4b-it-abliterated'; +const BASE_URL = process.env.SMALLCODE_BASE_URL || 'http://10.0.0.20:1234/v1'; + +const TURN_TIMEOUT_S = parseInt(process.env.E2E_TURN_TIMEOUT || '180', 10); + +const C = { + reset: '\x1b[0m', bold: '\x1b[1m', dim: '\x1b[2m', + red: '\x1b[31m', green: '\x1b[32m', yellow: '\x1b[33m', cyan: '\x1b[36m', +}; +const paint = (t, code) => process.stdout.isTTY ? `${code}${t}${C.reset}` : t; + +// ─── helpers ──────────────────────────────────────────────────────────────── + +function freshTmpDir(prefix) { + return fs.mkdtempSync(path.join(os.tmpdir(), `${prefix}-`)); +} + +function runAgent(prompt, opts = {}) { + return new Promise((resolve) => { + const cwd = opts.cwd || freshTmpDir('e2e'); + const env = { + ...process.env, + SMALLCODE_MODEL: MODEL, + SMALLCODE_BASE_URL: BASE_URL, + SMALLCODE_PROVIDER: 'openai', + SMALLCODE_AUTO_APPROVE: 'true', + NO_COLOR: '1', + FORCE_COLOR: '0', + // Keep features deterministic for the smoke run + SMALLCODE_REVIEWER: 'false', + SMALLCODE_PLAN: 'false', + SMALLCODE_BOOTSTRAP: 'false', + ...(opts.env || {}), + }; + const child = spawn('node', [SMALLCODE_BIN, '--non-interactive', '-P', prompt], { + cwd, + env, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (d) => { stdout += d.toString(); }); + child.stderr.on('data', (d) => { stderr += d.toString(); }); + const t = setTimeout(() => { + try { child.kill('SIGKILL'); } catch {} + }, TURN_TIMEOUT_S * 1000); + child.on('exit', (code) => { + clearTimeout(t); + resolve({ code, stdout, stderr, cwd }); + }); + }); +} + +let failures = 0; +function check(name, ok, details) { + if (ok) { + console.log(` ${paint('✓', C.green)} ${name}`); + } else { + failures += 1; + console.log(` ${paint('✗', C.red)} ${name}`); + if (details) console.log(` ${paint(details, C.dim)}`); + } +} + +// ─── case 1: per-turn idempotent-write dedup ──────────────────────────────── + +async function caseIdempotentDedup() { + console.log(''); + console.log(paint('Case 1 — per-turn idempotent-write dedup', C.bold)); + + // Pre-seed memory so the dedup short-circuit message becomes visible to us. + // We ask the model to remember THE SAME thing several times in one message. + // Even if the model only emits two memory_remember calls, the second must + // hit the [already stored this turn] short-circuit. + const prompt = + 'Use the memory_remember tool to save a single context note titled ' + + '"smoke-key" with content "smoke-value", then do it again with the ' + + 'exact same arguments. Then reply with a one-line summary.'; + + const res = await runAgent(prompt); + // We expect the runtime to have logged either the canonical short-circuit + // marker OR the underlying tool response showing only one real call. + const merged = (res.stdout + res.stderr); + const sawShortCircuit = /already stored this turn/i.test(merged); + + // Fallback signal: count memory_remember tool indicators (`⚙ memory_remember` + // or · memory_remember in the TUI) — a model that emits 2 calls but only + // 1 is executed real would still be a correct outcome; we just want to make + // sure the short-circuit path is reachable end-to-end. + const remembersExecuted = (merged.match(/Remembered\s+\[/g) || []).length; + + check('agent ran without crashing', res.code === 0, + `exit=${res.code}; stderr tail: ${res.stderr.slice(-300)}`); + check('short-circuit message OR single execution', + sawShortCircuit || remembersExecuted <= 1, + `sawShortCircuit=${sawShortCircuit}, remembersExecuted=${remembersExecuted}`); +} + +// ─── case 2: contract create + done-guard ─────────────────────────────────── + +async function caseContractGuard() { + console.log(''); + console.log(paint('Case 2 — contract create + done-guard', C.bold)); + + // Single-turn, multi-step prompt: create a contract with two assertions, + // mark only the first one passed, then claim "all done". The guard must + // intercept the wrap-up. We expect to see a [CONTRACT-GUARD] injection + // OR — if the model recovers correctly after the guard fires — the on-disk + // state should show the assertions resolved through the contract tools. + const prompt = + 'Step 1: Use contract_create to declare a Definition of Done with title ' + + '"smoke" and these two assertions: "smoke step one passes", "smoke step ' + + 'two passes". ' + + 'Step 2: Use contract_assert_pass on a01 with evidence "verified by smoke ' + + 'test". ' + + 'Step 3: Reply with the single line "All done — task is complete." (do ' + + 'NOT mark a02 as passed; leave it pending intentionally).'; + + const res = await runAgent(prompt); + const merged = (res.stdout + res.stderr); + // The fullscreen TUI emits "⚙ <tool> ✓ <ms>" for tool calls. Use the tool + // names as the success signal — they're the closest thing to a structured + // event we can observe from outside the agent. + const toolFired = (name) => new RegExp(`⚙\\s*${name}`).test(merged); + const sawCreate = toolFired('contract_create'); + const sawPass = toolFired('contract_assert_pass'); + const sawGuard = /CONTRACT-GUARD/.test(merged) || /contract guard:/.test(merged); + + // Inspect the contract on disk for ground truth — the agent's tool calls + // should have left a state.json behind. Two valid outcomes: + // (a) a01 passed, a02 pending → guard fired, model didn't recover + // (b) every assertion resolved → guard fired, model used skip/pass to + // recover (this is the correct behaviour for the agent) + const contractsRoot = path.join(res.cwd, '.smallcode', 'contracts'); + let stateOk = false; + let stateDetail = '(no state.json)'; + try { + if (fs.existsSync(contractsRoot)) { + const ids = fs.readdirSync(contractsRoot).filter((f) => !f.startsWith('.')); + if (ids.length > 0) { + const state = JSON.parse(fs.readFileSync(path.join(contractsRoot, ids[0], 'state.json'), 'utf-8')); + if (state.assertions && state.assertions.length === 2) { + const a01 = state.assertions[0].state; + const a02 = state.assertions[1].state; + const a01Resolved = ['passed', 'skipped'].includes(a01); + const a02Resolved = ['passed', 'skipped', 'failed'].includes(a02); + // a01 must be marked, and a02 must either still be pending (guard + // fired and model honoured the failure) or resolved (guard fired + // and model recovered). + stateOk = a01Resolved && (a02 === 'pending' || a02Resolved); + stateDetail = `a01=${a01} a02=${a02}`; + } + } + } + } catch (e) { + stateDetail = `(read error: ${e.message})`; + } + + check('agent ran without crashing', res.code === 0, + `exit=${res.code}; stderr tail: ${res.stderr.slice(-300)}`); + check('contract_create tool fired', sawCreate, + `output tail:\n${merged.slice(-600)}`); + check('contract_assert_pass tool fired', sawPass); + check('done-guard intercepted wrap-up', sawGuard, + `expected CONTRACT-GUARD or "contract guard:" in output. Tail:\n${merged.slice(-600)}`); + check('contract state.json reflects tool calls', stateOk, stateDetail); +} + +// ─── case 3: bench/diff.js sanity ─────────────────────────────────────────── + +function caseBenchDiff() { + console.log(''); + console.log(paint('Case 3 — bench/diff.js exit codes', C.bold)); + + const baseDir = freshTmpDir('e2e-bench-base'); + const featDir = freshTmpDir('e2e-bench-feat'); + const baseFile = path.join(baseDir, 'run.json'); + const featFile = path.join(featDir, 'run.json'); + + fs.writeFileSync(baseFile, JSON.stringify({ + summary: {}, + results: [ + { id: 't1', lang: 'py', passed: false, elapsedMs: 1000, toolCalls: 1 }, + { id: 't2', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ], + })); + fs.writeFileSync(featFile, JSON.stringify({ + summary: {}, + results: [ + { id: 't1', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + { id: 't2', lang: 'py', passed: true, elapsedMs: 1000, toolCalls: 1 }, + ], + })); + + const r = spawnSync(process.execPath, [path.join(ROOT, 'bench', 'diff.js'), baseFile, featFile, '--json'], { encoding: 'utf-8' }); + check('exit 0 on improvement', r.status === 0, `status=${r.status}`); + let parsed = null; + try { parsed = JSON.parse(r.stdout); } catch {} + check('emitted JSON verdict IMPROVED', parsed && parsed.verdict === 'IMPROVED', + `stdout: ${r.stdout?.slice(0, 200)}`); +} + +// ─── main ──────────────────────────────────────────────────────────────────── + +(async () => { + console.log(paint('SmallCode E2E smoke', C.bold)); + console.log(` model: ${paint(MODEL, C.cyan)}`); + console.log(` base : ${paint(BASE_URL, C.cyan)}`); + + try { + caseBenchDiff(); + await caseIdempotentDedup(); + await caseContractGuard(); + } catch (e) { + console.log(''); + console.log(paint(`fatal: ${e.message}`, C.red)); + failures += 1; + } + + console.log(''); + if (failures === 0) { + console.log(paint('All E2E checks passed.', C.green + C.bold)); + process.exit(0); + } else { + console.log(paint(`${failures} E2E check(s) failed.`, C.red + C.bold)); + process.exit(1); + } +})();