Calibrate HARP with baselines, repeated trials, and confidence reporting #8

Problem

A 15-prompt, single-run composite score cannot justify readiness labels such as “daily driver.” There are no published reference runs, prompt variants, variance estimates, cost/latency data, or evidence that thresholds distinguish useful behavior from benchmark overfitting.

Goal

Turn HARP into a calibrated diagnostic that reports uncertainty and actionable capability-level results.

Scope

• Define versioned prompt variants and repeated-trial policy for each capability.
• Publish baseline results for a small, representative set of reference models.
• Report per-phase and per-capability outcomes alongside composite score, success rate, variance/confidence interval, latency, token usage, and cost when supplied by the provider.
• Establish threshold calibration criteria grounded in baseline data and failure traces.
• Introduce a held-out regression set or rotation policy that prevents tuning exclusively to public prompts.
• Document limitations, nondeterminism policy, and interpretation guidance.

Acceptance Criteria

• A benchmark run can execute a configured number of trials and report aggregate statistics.
• The report distinguishes reliable failures from statistically uncertain results.
• At least three documented reference-model baseline reports are committed or published with manifests.
• Readiness thresholds are accompanied by rationale and calibration evidence.
• The report surfaces the failed probe/assertion traces needed to recommend operating constraints.
• A held-out or rotating evaluation mechanism is documented and enforced for official baseline updates.

Non-Goals

• Building a general-purpose model leaderboard.
• Claiming cross-provider cost comparisons where providers do not return comparable usage data.

Implementation Notes

Do not collapse diagnostics into only one number. The principal artifact should make it clear whether a model fails at skill selection, instruction fidelity, safe tool use, or multi-step context preservation.

Dependencies

• Depends on Restore a self-contained, runnable benchmark pipeline #5, Make HARP scoring deterministic and behaviorally valid #6, Add reproducibility manifests and model configuration validation #7, and Calibrate HARP with baselines, repeated trials, and confidence reporting #8 so results are runnable, deterministic, isolated, and reproducible.