The claim is a hypothesis until an instrumented comparison exists. A deterministic copy demonstration is evidence of correct copying, not evidence of an AI cost or speed advantage.
- Freeze a corpus of owned/permitted source repositories and source commits. Keep target requirements and hidden acceptance tests separate from source selection.
- Fix the coding agent, model/version, settings, tools, sandbox limits and task prompt. Record dependency caches and machine characteristics.
- Run A (agent without CodeRecall) and B (same agent plus CodeRecall) on isolated targets. Randomize order and repeat with multiple seeds/tasks. Include unrelated targets where generation should win.
- Report cold indexing cost and warm-memory performance separately. B includes indexing, retrieval, inspection, planning, token-consuming tool output, adaptation, verification, repair and human interventions. Do not silently move costs outside the measurement window.
- Run the same hidden functional and regression tests on both outputs. A solution that copied more but fails correctness is not a successful reuse.
- Report per-task results, medians, dispersion, failed runs and excluded runs with reasons. Savings comparisons must use equivalent successful outcomes, with unsuccessful outcomes separately visible.
| Metric | Measurement | Limitation |
|---|---|---|
| Files/bytes/lines copied | Accepted file inventory and exact transferred buffers | Counts accepted text, including docs/config; not just executable code |
| Files unchanged | Source transfer hash equals current target hash | Whole-file reuse only; cannot identify retained lines inside changed files |
| Files modified/added/removed | Accepted target inventory compared to copied baseline | Excluded/binary files are not covered; removal can also mean newly excluded |
modifiedFileLines |
Current lengths of modified files | Not lines edited or regenerated |
addedFileLines |
Current lengths of added eligible files | An addition is not necessarily LLM-generated |
| Verification | Chosen argv, snapshot, exit status, duration, timeout | No coverage guarantee or vulnerability audit |
| Source tokens generated by copy | Zero: ordinary file writes transfer existing buffers | Does not measure external agent output tokens |
| Agent usage | null until independently instrumented |
Never convert unknown usage to zero |
Future harness records must include input/output/generated tokens, total tool calls, elapsed time, generated/copied/modified files and lines, tests passed, regression count, human intervention, confidence, indexing costs and estimated API cost using an explicitly recorded price source/date. Hashes alone cannot reconstruct exact semantic or line ancestry. A proper agent trace and textual patch measurement are needed.
Cost comparisons must include model calls made by semantic providers if any are added later. Measure metadata/context tokens as well as patch output. CodeRecall fails its core hypothesis if selecting and adapting old code consumes more total work than generating a correct result, even when the copy counter is large.
pnpm demo creates a new synthetic corpus, verifies the Next.js fixture's domain tests, selects it, copies it, applies a fixed synthetic delta, runs the original plus new tests and emits .work/demo-*/report.json. The fixed patch is explicitly separate from the engine. No coding-agent A/B comparison or savings percentage is produced.