Skip to content

Latest commit

 

History

History
33 lines (24 loc) · 3.52 KB

benchmark.md

File metadata and controls

33 lines (24 loc) · 3.52 KB

Measuring whether reuse reduces AI work

The claim is a hypothesis until an instrumented comparison exists. A deterministic copy demonstration is evidence of correct copying, not evidence of an AI cost or speed advantage.

Paired protocol

  1. Freeze a corpus of owned/permitted source repositories and source commits. Keep target requirements and hidden acceptance tests separate from source selection.
  2. Fix the coding agent, model/version, settings, tools, sandbox limits and task prompt. Record dependency caches and machine characteristics.
  3. Run A (agent without CodeRecall) and B (same agent plus CodeRecall) on isolated targets. Randomize order and repeat with multiple seeds/tasks. Include unrelated targets where generation should win.
  4. Report cold indexing cost and warm-memory performance separately. B includes indexing, retrieval, inspection, planning, token-consuming tool output, adaptation, verification, repair and human interventions. Do not silently move costs outside the measurement window.
  5. Run the same hidden functional and regression tests on both outputs. A solution that copied more but fails correctness is not a successful reuse.
  6. Report per-task results, medians, dispersion, failed runs and excluded runs with reasons. Savings comparisons must use equivalent successful outcomes, with unsuccessful outcomes separately visible.

What the engine records today

Metric Measurement Limitation
Files/bytes/lines copied Accepted file inventory and exact transferred buffers Counts accepted text, including docs/config; not just executable code
Files unchanged Source transfer hash equals current target hash Whole-file reuse only; cannot identify retained lines inside changed files
Files modified/added/removed Accepted target inventory compared to copied baseline Excluded/binary files are not covered; removal can also mean newly excluded
modifiedFileLines Current lengths of modified files Not lines edited or regenerated
addedFileLines Current lengths of added eligible files An addition is not necessarily LLM-generated
Verification Chosen argv, snapshot, exit status, duration, timeout No coverage guarantee or vulnerability audit
Source tokens generated by copy Zero: ordinary file writes transfer existing buffers Does not measure external agent output tokens
Agent usage null until independently instrumented Never convert unknown usage to zero

Future harness records must include input/output/generated tokens, total tool calls, elapsed time, generated/copied/modified files and lines, tests passed, regression count, human intervention, confidence, indexing costs and estimated API cost using an explicitly recorded price source/date. Hashes alone cannot reconstruct exact semantic or line ancestry. A proper agent trace and textual patch measurement are needed.

Cost comparisons must include model calls made by semantic providers if any are added later. Measure metadata/context tokens as well as patch output. CodeRecall fails its core hypothesis if selecting and adapting old code consumes more total work than generating a correct result, even when the copy counter is large.

Current executable experiment

pnpm demo creates a new synthetic corpus, verifies the Next.js fixture's domain tests, selects it, copies it, applies a fixed synthetic delta, runs the original plus new tests and emits .work/demo-*/report.json. The fixed patch is explicitly separate from the engine. No coding-agent A/B comparison or savings percentage is produced.