Evaluation harness for measuring rule effectiveness

[thomas-bartlett]

An initial study was done at Cisco, but we have no consistent, repeatable workflow in this repo for quantitatively measuring the effectiveness of the security rules. This discussion is to agree on a standard approach before building the harness.

Questions for input

• Approach: with-rules vs without-rules comparison, rubric scoring, static analysis of generated code, or a combination?
• Inference: which models are in scope, which APIs or providers, where does it run (CI, hosted, local), and how is cost covered?
• Ground truth: what decides pass/fail for a generated sample?
• Tooling: PromptFoo, Inspect, Semgrep, something else?
• What does the smallest useful first result look like?

[thomas-bartlett]

Closing as superseded. This broad evaluation-harness thread has been replaced by the more specific literature-review discussion in #70 and the install-route evaluation issue in #61. New harness research should continue in #70.