diff --git a/CHANGELOG.md b/CHANGELOG.md index 57deee3a..d6e93e95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,14 @@ All notable changes to SkillEvaluator are documented in this file. ### Fixed +- Tier 3 now uses one versioned, equal-weight five-dimension overall-score + policy across Harbor rewards, pass@k, Skill Lift, reports, comparisons, and + best-agent selection. Previously, some paths averaged all six evaluators + directly while reports averaged five dimensions, which gave Effectiveness + two votes and could reverse the reported lift direction. Current artifacts + persist `skill-evaluator-dimension-mean-v1`; legacy and partial artifacts + retain their historical semantics + (Relates to [#61](https://github.com/NVIDIA/SkillEvaluator/issues/61)). - `--llm-verify` now refuses to send file context from paths outside the skill root, including `..`, absolute paths, and outbound file symlinks. - Gitleaks path allowlist now skips test/example/fixture/mock directories diff --git a/docs/reports.mdx b/docs/reports.mdx index c1d3993a..7ece8ada 100644 --- a/docs/reports.mdx +++ b/docs/reports.mdx @@ -192,25 +192,49 @@ command. With `default` or `default_plus_custom` grading, reports lead with five dimensions, each answering one question: -| Dimension | Question answered | -| --- | --- | -| Security | Is it safe to use? | -| Correctness | Is the answer correct? | -| Discoverability | Was the right skill loaded when needed? | -| Effectiveness | Did the skill help complete the task? | -| Efficiency | Did it avoid wasted tool or skill usage? | - -Dimension scores are 0.0–1.0 rollups of the underlying evaluation signals. The -verdict bands are fixed: a dimension **passes** at 0.50 or above, is -**neutral** from 0.40 to below 0.50, and **fails** below 0.40. The judge runs on -the configured provider model; override it per run as described in -[Tier 3: Live Evaluation](tier3-live-evaluation.mdx). +| Dimension | Question answered | Signals and source | +| --- | --- | --- | +| Security | Is it safe to use? | `security`, from deterministic trace checks | +| Correctness | Is the answer correct? | `accuracy`, from the `judge_accuracy` LLM judge | +| Discoverability | Was the right skill loaded when needed? | `skill_execution`, from deterministic trajectory checks | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` and `behavior_check`, from LLM judges | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency`, from deterministic trajectory checks | + +Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated +standard grader rejects non-finite values and clamps numeric scores with +`max(0.0, min(1.0, score))`. A failed required judge produces no score instead +of a numeric zero. The fixed dimension bands are **pass** at 0.50 or above, +**neutral** from 0.40 to below 0.50, and **fail** below 0.40. + +`run_config.json` records the configured judge provider, model, source, and +catalog-verification status separately from the agent model. `result.json` +embeds that run configuration alongside the scores. This provenance makes the +LLM-judged values traceable, but it is not evidence that the judges are +validated. The repository does not publish inter-rater agreement, calibration +results, or a comparison against human-labeled ground truth. + +The **Quality Score** displayed out of 100 in the HTML report is the Tier 1 +static-analysis score. It is not an input to the Tier 3 dimensions, overall +score, or Skill Lift. + +The standard overall score gives each dimension one equal vote: + +```text +overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5 +Effectiveness = (goal_accuracy + behavior_check) / 2 +``` + +This formula is shared by the report, per-attempt pass@k thresholding, Skill +Lift, comparisons, and best-agent selection. Standard artifacts identify it as +`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persisted +attempt policy. Older metric sets and partial historical artifacts keep their +legacy policy rather than being rescored as current data. ### Skill Lift -Skill Lift is the with-skill score minus the without-skill baseline — the -direct measurement of what your skill contributes. Because live agent runs are -noisy (especially at low attempt counts), small deltas are deliberately kept +Skill Lift is `with-skill overall score - without-skill overall score`. It uses +the same score units as both arms and ranges from -1.0 to +1.0. Because live +agent runs are noisy, especially at low attempt counts, small deltas are kept neutral. The verdict bands: | Lift | Verdict | Read it as | @@ -236,7 +260,8 @@ see [Custom Graders & Tasks](custom-graders.mdx). pass@k is the reliability signal, reported separately from the dimension scores. With `--n-attempts k`, each eval case runs k times per arm, and a case counts as passed when at least one attempt clears the `--pass-threshold` -score. Comparing pass@k across arms (in `pass_at_k_lift.json`) tells you +score using the same overall formula above. Comparing pass@k across arms (in +`pass_at_k_lift.json`) tells you whether the skill makes success more *repeatable*, not just whether the average score moved. Each arm also records a case-level 95% Wilson score interval for its pass rate. When both arms contain the same identified cases, diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx index 924ab2d5..7759dff8 100644 --- a/docs/tier3-live-evaluation.mdx +++ b/docs/tier3-live-evaluation.mdx @@ -414,13 +414,48 @@ rollups, followed by a compact Artifacts panel pointing at the report and output directory. A failed run renders the same structured display instead of a bare exception. -| Dimension | Key question | Maps to | Weights | -| --- | --- | --- | --- | -| Security | Is it safe to use? | `security` | 1.0 | -| Correctness | Is the answer correct? | `accuracy` | 1.0 | -| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 | -| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 | -| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 | +Every evaluator score and dimension score is on a 0.0 to 1.0 scale. The +generated standard grader rejects non-finite values and clamps numeric scores +with `max(0.0, min(1.0, score))`. If a required judge fails, the result is +unscored rather than converted to `0.0`. + +| Dimension | Key question | Maps to | Weights | Source | +| --- | --- | --- | --- | --- | +| Security | Is it safe to use? | `security` | 1.0 | Deterministic checks of the trace and agent actions | +| Correctness | Is the answer correct? | `accuracy` | 1.0 | LLM judge (`judge_accuracy`) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 | Deterministic checks of the trajectory and tool calls | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 | LLM judges (`judge_goal_accuracy`, `judge_behavior_check`) | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 | Deterministic checks of the trajectory and tool calls | + +The run's configured judge provider and model are stored under `judge` in +`run_config.json`, separately from the agent's provider and model. The same run +configuration is embedded in `result.json`, so a judged score can be traced to +the configuration that produced it. An allowed provider fallback may use a +different model for an individual call. + +This provenance does not establish that the LLM judgments are correct. The +repository does not publish inter-rater agreement, calibration results, or a +comparison against human-labeled ground truth for these judges. + +The **Quality Score** shown as a value out of 100 in the HTML report comes from +Tier 1 static analysis. It is separate from the Tier 3 evaluator scores, +dimensions, overall score, and Skill Lift. + +For standard grading, the overall score is the equal-weight mean of these five +dimensions: + +```text +overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5 +Effectiveness = (goal_accuracy + behavior_check) / 2 +``` + +Effectiveness therefore enters the overall score once; its two source +evaluators do not receive an extra combined vote. The same aggregate drives +per-attempt pass@k, Skill Lift, comparison output, and best-agent selection. +Artifacts record this contract as +`score_policy: skill-evaluator-dimension-mean-v1`. Legacy metric sets and +historical partial reports retain their recorded scoring semantics instead of +being silently reinterpreted. `token_efficiency` is a standalone report signal, not a dimension source. It never changes a dimension score or the overall verdict. @@ -435,8 +470,9 @@ agent passes only when every dimension passes. The top-level Tier 3 verdict is: | Neutral | No agent passes, but at least one has no dimension below 0.40 | | Fail | Every successful agent has a dimension below 0.40 | -Skill Lift is the signed difference between the with-skill result and the -without-skill baseline. Its independent band is **PASS** at +0.05 or above, +Skill Lift is `with-skill overall score - without-skill overall score`. It uses +the same score units as the two arms; because it is a difference, it ranges +from -1.0 to +1.0. Its independent band is **PASS** at +0.05 or above, **NEUTRAL** between -0.10 and +0.05, and **FAIL** at -0.10 or below. Lift is diagnostic evidence and does not override the every-dimension gate. @@ -455,7 +491,7 @@ The canonical Tier 3 payload embedded by both standalone `evaluate` and `validate --agent-eval` reports uses schema version 2.0. A completed standard-grading payload includes the summary, five dimensions, per-agent results, trials, pass@k, attempt policy, run-owned dataset summary and digest, -evaluation timestamp, evaluator version, and verdict policy; advisory or +evaluation timestamp, evaluator version, score policy, and verdict policy; advisory or skipped payloads may leave live-evidence sections empty. Generated `BENCHMARK.md` cards put the verdict first, label each result column diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.py index 5926ba41..c5401fb4 100644 --- a/src/skillevaluator/constants.py +++ b/src/skillevaluator/constants.py @@ -481,6 +481,7 @@ "overall = mean(Security, Correctness, Discoverability, Effectiveness, Efficiency) " "dimensions; Security maps to security evaluator (fallback: behavior_check)" ) +DEFAULT_SCORE_POLICY = "skill-evaluator-dimension-mean-v1" AGENT_EVAL_VERDICT_PASS = "pass" AGENT_EVAL_VERDICT_NEUTRAL = "neutral" diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.py index 0dc50134..c0ed7497 100644 --- a/src/skillevaluator/evaluation/tier3_report.py +++ b/src/skillevaluator/evaluation/tier3_report.py @@ -30,6 +30,7 @@ from skillevaluator.constants import ( AGENT_EVAL_EVALUATORS, AGENT_EVAL_SCORE_DEFINITION, + DEFAULT_SCORE_POLICY, DIMENSION_HINTS, DIMENSION_MAPPING, DIMENSION_VERDICT_NEUTRAL_THRESHOLD, @@ -80,6 +81,18 @@ def _finite_float(value: object) -> float | None: return numeric if math.isfinite(numeric) else None +def _unit_interval_score(value: object) -> float | None: + """Return a finite score in the documented 0.0 to 1.0 range.""" + numeric = _finite_float(value) + return numeric if numeric is not None and 0.0 <= numeric <= 1.0 else None + + +def _skill_lift_value(value: object) -> float | None: + """Return a finite Skill Lift value in its documented -1.0 to 1.0 range.""" + numeric = _finite_float(value) + return numeric if numeric is not None and -1.0 <= numeric <= 1.0 else None + + def _sanitize_json_numbers(value: Any) -> Any: """Copy a canonical payload while replacing non-finite floats with JSON null.""" if isinstance(value, float): @@ -218,6 +231,7 @@ def _advisory_agent_eval_payload( "dataset_summary": dataset_summary, "dataset_digest": None, "dataset_digest_algorithm": None, + "score_policy": attempt_policy["score_policy"], "verdict_policy": verdict_policy, "execution_status": "skipped", "execution_errors": [message], @@ -257,6 +271,7 @@ def _advisory_agent_eval_payload( "dataset_summary": dataset_summary, "dataset_digest": None, "dataset_digest_algorithm": None, + "score_policy": attempt_policy["score_policy"], "verdict_policy": verdict_policy, "provenance": { "source": "advisory", @@ -441,11 +456,13 @@ def agent_eval_result_from_directory( run_truth = _run_truth_metadata(run_dir, engine_result, load_dataset_snapshot(run_dir)) dataset = run_truth.get("dataset") or load_staged_harbor_dataset(run_dir) + attempt_policy = _read_attempt_policy(run_dir) payload = build_agent_eval_payload( skill_path.name, agents, dataset=dataset, - attempt_policy=_read_attempt_policy(run_dir), + attempt_policy=attempt_policy, + historical_missing_score_policy="score_policy" not in attempt_policy, run_config=_read_run_config(run_dir), env_mode=env_mode, runtime_seconds=_runtime_seconds(engine_result), @@ -549,6 +566,7 @@ def build_agent_eval_payload( persisted_dataset_summary: dict[str, Any] | None = None, dataset_digest: str | None = None, dataset_digest_algorithm: str | None = None, + historical_missing_score_policy: bool = False, use_llm_judge: bool = True, ) -> dict[str, Any] | None: """Assemble the canonical Tier 3 ``agent_eval`` payload from loaded agent data. @@ -571,12 +589,40 @@ def build_agent_eval_payload( ) metrics = metrics_for_agents(agents) + policy_metrics = _policy_metrics_from_agents(agents, metrics) + + from skillevaluator.tier3.harbor.metrics import ( + DEFAULT_METRICS, + LEGACY_SCORE_POLICY, + score_policy_for_metrics, + ) + + policy = dict(attempt_policy) if attempt_policy else _default_attempt_policy() + if not attempt_policy: + policy.pop("score_definition", None) + policy.pop("score_policy", None) + stored_policy = policy.get("score_policy") + if isinstance(stored_policy, str) and stored_policy.strip(): + policy["score_policy"] = stored_policy.strip() + else: + recorded_policy = _recorded_score_policy(agents) + if recorded_policy is not None: + policy["score_policy"] = recorded_policy + elif historical_missing_score_policy and policy_metrics == DEFAULT_METRICS: + policy["score_policy"] = LEGACY_SCORE_POLICY + else: + policy["score_policy"] = score_policy_for_metrics(policy_metrics) + policy.setdefault( + "score_definition", + _score_definition_for_policy(policy_metrics, policy["score_policy"]), + ) + report_budget = _ReportBudget(artifact_loading=_artifact_loading_reasons(agents, dataset)) agent_payloads: dict[str, dict[str, Any]] = {} for name in sorted(agents): info = agents[name] model = _agent_model(name, info, run_config) - agent_payloads[name] = _build_agent(name, info, metrics, model) + agent_payloads[name] = _build_agent(name, info, metrics, model, policy["score_policy"]) if not agent_payloads: return None @@ -613,8 +659,6 @@ def build_agent_eval_payload( metric_ids = list(best.get("evaluators", {}).keys()) metric_labels = _metric_labels(metric_ids) - - policy = attempt_policy or _default_attempt_policy() canonical_trials = _flatten_trials(agent_payloads) public_dataset = deduplicate_dataset_entries([entry for entry in (dataset or []) if isinstance(entry, dict)]) computed_dataset_truth = ( @@ -654,6 +698,7 @@ def build_agent_eval_payload( "dataset_summary": dataset_summary, "dataset_digest": effective_dataset_digest, "dataset_digest_algorithm": effective_dataset_digest_algorithm, + "score_policy": policy["score_policy"], "verdict_policy": verdict_policy, "execution_status": execution_status, "execution_errors": execution_errors, @@ -713,6 +758,7 @@ def build_agent_eval_payload( "dataset_summary": dataset_summary, "dataset_digest": effective_dataset_digest, "dataset_digest_algorithm": effective_dataset_digest_algorithm, + "score_policy": policy["score_policy"], "verdict_policy": verdict_policy, "agents": agent_payloads, "dimensions": best_dimensions, @@ -1071,6 +1117,14 @@ def _serialized_payload_size(payload: dict[str, Any]) -> int: def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _ReportBudget) -> None: """Last-resort bounded shape for pathological single-field payloads.""" summary = payload.get("summary") if isinstance(payload.get("summary"), dict) else {} + attempt_policy = payload.get("attempt_policy") if isinstance(payload.get("attempt_policy"), dict) else {} + compact_attempt_policy = { + key: attempt_policy[key] for key in ("max_attempts", "pass_threshold", "stop_on_pass") if key in attempt_policy + } + for key, limit in (("score_definition", 1024), ("score_policy", 256)): + value = attempt_policy.get(key) + if isinstance(value, str): + compact_attempt_policy[key] = value[:limit] compact_summary = { key: value for key, value in summary.items() @@ -1082,6 +1136,7 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor "overall_lift", "environment", "runtime_seconds", + "score_policy", "execution_status", "expected_attempts", "scored_attempts", @@ -1089,6 +1144,8 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor } compact_summary["skill_name"] = str(summary.get("skill_name") or payload.get("skill_name") or "")[:256] compact_summary["best_agent"] = str(summary.get("best_agent") or payload.get("best_agent") or "")[:256] + raw_score_policy = payload.get("score_policy", summary.get("score_policy")) + compact_summary["score_policy"] = raw_score_policy[:256] if isinstance(raw_score_policy, str) else None compact_summary["agents_run"] = [str(name)[:256] for name in (summary.get("agents_run") or [])[:64]] compact_summary["execution_errors"] = [str(error)[:1024] for error in (summary.get("execution_errors") or [])[:16]] @@ -1109,6 +1166,8 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor "expected_attempts": payload.get("expected_attempts", 0), "scored_attempts": payload.get("scored_attempts", 0), "runtime_seconds": payload.get("runtime_seconds", 0.0), + "score_policy": compact_summary["score_policy"], + "attempt_policy": compact_attempt_policy, "agents": {}, "dimensions": [], "evaluators": {}, @@ -1141,6 +1200,58 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor # --------------------------------------------------------------------------- +def _policy_metrics_from_agents( + agents: dict[str, dict[str, Any]], + metrics: list[str], +) -> tuple[str, ...]: + available = tuple( + metric + for metric in metrics + if any( + _finite_float(scores.get(metric)) is not None + for info in agents.values() + for scores in (info.get("with_skill"), info.get("without_skill")) + if isinstance(scores, dict) + ) + ) + return available or tuple(metrics) + + +def _recorded_score_policy(agents: dict[str, dict[str, Any]]) -> str | None: + recorded: set[str] = set() + for info in agents.values(): + for condition, scores_key, policy_key in ( + ("with_skill", "with_skill", "score_policy_with_skill"), + ("without_skill", "without_skill", "score_policy_without_skill"), + ): + scores = info.get(scores_key) + policy = info.get(policy_key) + if ( + _condition_quality_available(info, condition) + and isinstance(scores, dict) + and scores + and isinstance(policy, str) + and policy.strip() + ): + recorded.add(policy.strip()) + return next(iter(recorded)) if len(recorded) == 1 else None + + +def _score_definition_for_policy(metrics: tuple[str, ...], score_policy: str) -> str: + from skillevaluator.tier3.harbor.metrics import LEGACY_SCORE_POLICY, score_definition + + if score_policy == LEGACY_SCORE_POLICY and metrics: + return f"overall = mean({', '.join(metrics)}) [{LEGACY_SCORE_POLICY}]" + return score_definition(metrics) + + +def _complete_metric_mean(scores: dict[str, Any], metrics: list[str]) -> float | None: + values = tuple(_finite_float(scores.get(metric)) for metric in metrics) + if not values or any(value is None for value in values): + return None + return round(sum(value for value in values if value is not None) / len(values), 4) + + def _condition_quality_available(info: dict[str, Any], condition: str) -> bool: """Return whether a condition may contribute score-bearing report fields.""" conditions = info.get("conditions") @@ -1156,7 +1267,15 @@ def _build_agent( info: dict[str, Any], metrics: list[str], model: str | None, + score_policy: str, ) -> dict[str, Any]: + from skillevaluator.tier3.harbor.metrics import ( + LEGACY_METRICS, + LEGACY_SCORE_POLICY, + canonical_dimension_mean, + overall_score_from_metrics, + ) + with_scores = info.get("with_skill") or {} without_scores = info.get("without_skill") or {} lift_data = info.get("lift") or {} @@ -1174,17 +1293,46 @@ def _build_agent( info.get("dimensions_with_skill") or {}, info.get("dimensions_without_skill") or {}, ) - overall_ws = _mean([d["with_skill"] for d in dimensions]) - overall_bl = _mean([d["baseline"] for d in dimensions]) + with_dimension_values = [d["with_skill"] for d in dimensions] + baseline_dimension_values = [d["baseline"] for d in dimensions] + overall_ws = None + overall_bl = None + if score_policy == LEGACY_SCORE_POLICY: + overall_ws = _unit_interval_score(info.get("overall_with_skill")) if with_quality_available else None + overall_bl = _unit_interval_score(info.get("overall_without_skill")) if baseline_quality_available else None + if overall_ws is None: + overall_ws = _complete_metric_mean(with_scores, metrics) + if overall_bl is None: + overall_bl = _complete_metric_mean(without_scores, metrics) + elif tuple(metrics) == LEGACY_METRICS: + if overall_ws is None: + overall_ws = overall_score_from_metrics(with_scores, LEGACY_METRICS) + if overall_bl is None: + overall_bl = overall_score_from_metrics(without_scores, LEGACY_METRICS) + else: + if overall_ws is None: + overall_ws = canonical_dimension_mean(with_dimension_values) + if overall_bl is None: + overall_bl = canonical_dimension_mean(baseline_dimension_values) + if overall_ws is None: + overall_ws = _mean(with_dimension_values) + if overall_bl is None: + overall_bl = _mean(baseline_dimension_values) if overall_ws is None and not metrics and with_quality_available: - overall_ws = _finite_float(info.get("overall_with_skill")) + overall_ws = _unit_interval_score(info.get("overall_with_skill")) if overall_ws is None and info.get("rewards_complete") is not False: overall_ws = _logical_reward_mean(info.get("rewards"), "overall") if overall_bl is None and not metrics and baseline_quality_available: - overall_bl = _finite_float(info.get("overall_without_skill")) + overall_bl = _unit_interval_score(info.get("overall_without_skill")) if overall_bl is None and info.get("rewards_baseline_complete") is not False: overall_bl = _logical_reward_mean(info.get("rewards_baseline"), "overall") overall_lift = round(overall_ws - overall_bl, 4) if overall_ws is not None and overall_bl is not None else None + if score_policy == LEGACY_SCORE_POLICY: + recorded_overall_lift = lift_data.get("overall") + if isinstance(recorded_overall_lift, dict): + recorded_delta = _skill_lift_value(recorded_overall_lift.get("delta")) + if recorded_delta is not None: + overall_lift = recorded_delta trials = _normalize_trials(info.get("rewards") or [], metrics) baseline_trials = _normalize_trials(info.get("rewards_baseline") or [], metrics) @@ -2544,6 +2692,11 @@ def _read_attempt_policy(run_dir: Path) -> dict[str, Any]: loaded = json.loads(policy_file.read_text(encoding="utf-8")) if isinstance(loaded, dict): policy.update(loaded) + if "score_policy" not in loaded: + policy.pop("score_policy", None) + return policy + policy.pop("score_definition", None) + policy.pop("score_policy", None) return policy @@ -2668,6 +2821,7 @@ def _default_attempt_policy() -> dict[str, Any]: "pass_threshold": 0.50, "stop_on_pass": False, "score_definition": AGENT_EVAL_SCORE_DEFINITION, + "score_policy": DEFAULT_SCORE_POLICY, } diff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.py index 6a7823a8..b5e52408 100644 --- a/src/skillevaluator/tier3/commands.py +++ b/src/skillevaluator/tier3/commands.py @@ -6,6 +6,7 @@ from __future__ import annotations import json +import math import os import shutil import subprocess @@ -36,7 +37,13 @@ HARBOR_AGENTS_SUPPORTED, canonical_agent_name, ) -from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, LEGACY_METRICS +from skillevaluator.tier3.harbor.metrics import ( + DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, + LEGACY_METRICS, + LEGACY_SCORE_POLICY, + overall_score_from_metrics, +) from skillevaluator.tier3.harbor.progress import ( NullProgressReporter, ProgressEvent, @@ -996,16 +1003,25 @@ def compare_results(skill_path: Path, *, results_dir: Path | None = None) -> int "timestamp": ts_dir.name, "path": str(agent_dir), "num_trials": data.get("num_trials", "?"), + "overall_with_skill": _summary_overall(data), + "score_policy_with_skill": _summary_score_policy(data), } wo_summary = agent_dir / "without-skill" / "summary.json" if wo_summary.exists(): try: + wo_data = json.loads(wo_summary.read_text(encoding="utf-8")) wo_scores = _summary_scores( - json.loads(wo_summary.read_text(encoding="utf-8")), + wo_data, allow_missing_status=allow_missing_status, ) if wo_scores: root_without[agent_name] = wo_scores + root_meta[agent_name].update( + { + "overall_without_skill": _summary_overall(wo_data), + "score_policy_without_skill": _summary_score_policy(wo_data), + } + ) except (ValueError, OSError): pass if root_with: @@ -1045,11 +1061,20 @@ def compare_results(skill_path: Path, *, results_dir: Path | None = None) -> int table.add_row(*[""] * (1 + sum(2 if agent in agent_without else 1 for agent in agents))) overall_row: list[str | Text] = [Text("Overall", style="bold")] for agent in agents: - with_avg = sum(_safe_score(agent_with[agent], metric) for metric in overall_metrics) / len(overall_metrics) + meta = agent_meta[agent] + with_avg = _overall_score_for_display( + agent_with[agent], + overall_metrics, + persisted_overall=meta.get("overall_with_skill"), + score_policy=meta.get("score_policy_with_skill"), + ) overall_row.append(Text(f"{with_avg:.2f}", style=f"bold {_score_style(with_avg)}")) if agent in agent_without: - without_avg = sum(_safe_score(agent_without[agent], metric) for metric in overall_metrics) / len( - overall_metrics + without_avg = _overall_score_for_display( + agent_without[agent], + overall_metrics, + persisted_overall=meta.get("overall_without_skill"), + score_policy=meta.get("score_policy_without_skill"), ) delta = with_avg - without_avg delta_text = f"+{delta:.2f}" if delta > 0 else f"{delta:.2f}" @@ -1089,6 +1114,15 @@ def _summary_scores(data: dict[str, Any], *, allow_missing_status: bool = False) return scores +def _summary_overall(data: dict[str, Any]) -> float | None: + return _unit_interval_score(data.get("overall_score")) + + +def _summary_score_policy(data: dict[str, Any]) -> str | None: + value = data.get("score_policy") + return value.strip() if isinstance(value, str) and value.strip() else None + + def _display_metrics(agent_with: dict[str, dict[str, float]]) -> tuple[tuple[str, ...], tuple[str, ...]]: has_default_scores = any(any(metric in scores for metric in DEFAULT_METRICS) for scores in agent_with.values()) if has_default_scores: @@ -1108,6 +1142,48 @@ def _safe_score(scores: dict[str, float], metric: str) -> float: return float(value) if isinstance(value, int | float) else 0.0 +def _unit_interval_score(value: object) -> float | None: + """Return a finite score in the documented 0.0 to 1.0 range.""" + if not isinstance(value, int | float) or isinstance(value, bool): + return None + try: + numeric = float(value) + except OverflowError: + return None + return numeric if math.isfinite(numeric) and 0.0 <= numeric <= 1.0 else None + + +def _overall_score_for_display( + scores: dict[str, float], + metrics: tuple[str, ...], + *, + persisted_overall: object = None, + score_policy: object = DEFAULT_SCORE_POLICY, +) -> float: + """Honor persisted score truth before inferring a policy from metric names.""" + recorded_overall = _unit_interval_score(persisted_overall) + + if score_policy == DEFAULT_SCORE_POLICY: + score = overall_score_from_metrics(scores, metrics) + if score is not None: + return score + + if recorded_overall is not None: + return recorded_overall + + if score_policy == LEGACY_SCORE_POLICY or (score_policy is None and metrics == DEFAULT_METRICS): + values = tuple(scores.get(metric) for metric in metrics) + if values and all( + isinstance(value, int | float) and not isinstance(value, bool) and math.isfinite(value) for value in values + ): + return round(sum(float(value) for value in values) / len(values), 4) + + score = overall_score_from_metrics(scores, metrics) + if score is not None: + return score + return sum(_safe_score(scores, metric) for metric in metrics) / len(metrics) + + def _score_style(score: float) -> str: if score >= 0.8: return "green" diff --git a/src/skillevaluator/tier3/harbor/collector.py b/src/skillevaluator/tier3/harbor/collector.py index a5a6985e..82fabafd 100644 --- a/src/skillevaluator/tier3/harbor/collector.py +++ b/src/skillevaluator/tier3/harbor/collector.py @@ -26,6 +26,7 @@ from skillevaluator.tier3.harbor.metrics import ( DEFAULT_METRIC_SET, DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, LEGACY_METRIC_SET, LEGACY_METRICS, average_custom_metrics, @@ -35,7 +36,9 @@ metric_set_for_reward, metric_value, overall_score, + overall_score_from_metrics, score_definition, + score_policy_for_metrics, ) from skillevaluator.tier3.output_provenance import write_output_file_atomically from skillevaluator.utils.redaction import is_sensitive_key, redact_sensitive_data, redact_sensitive_text @@ -2324,8 +2327,12 @@ def _compute_lift( "direction": "up" if delta > 0 else ("down" if delta < 0 else "flat"), } if metrics in {DISPLAY_METRICS, LEGACY_METRICS}: - overall_with = sum(with_scores[m] for m in metrics) / len(metrics) - overall_without = sum(without_scores[m] for m in metrics) / len(metrics) + overall_with = overall_score_from_metrics(with_scores, metrics) + overall_without = overall_score_from_metrics(without_scores, metrics) + else: + overall_with = None + overall_without = None + if overall_with is not None and overall_without is not None: lift["overall"] = { "with_skill": round(overall_with, 4), "without_skill": round(overall_without, 4), @@ -2342,6 +2349,18 @@ def _average_overall(rewards: list[dict[str, Any]]) -> float | None: return round(sum(value for value in values if value is not None) / len(values), 4) +def _aggregate_condition_overall( + scores: dict[str, float], + metrics: tuple[str, ...], + rewards: list[dict[str, Any]], +) -> float | None: + """Aggregate a condition consistently with lift and canonical reports.""" + if not metrics: + return _average_overall(rewards) + score = overall_score_from_metrics(scores, metrics) + return round(score, 4) if score is not None else None + + def _compute_custom_lift( with_custom_scores: dict[str, float], without_custom_scores: dict[str, float], @@ -3041,11 +3060,13 @@ def collect_harbor_results( "agents": {}, "metric_set": DEFAULT_METRIC_SET, "metrics": list(DISPLAY_METRICS), + "score_policy": DEFAULT_SCORE_POLICY, "attempt_policy": { "max_attempts": n_attempts, "pass_threshold": pass_threshold, "stop_on_pass": stop_on_pass, "score_definition": score_definition(DISPLAY_METRICS), + "score_policy": DEFAULT_SCORE_POLICY, }, } @@ -3087,6 +3108,8 @@ def collect_harbor_results( all_results["metric_set"] = with_metric_set all_results["metrics"] = list(with_metrics) all_results["attempt_policy"]["score_definition"] = score_definition(with_metrics) + all_results["score_policy"] = score_policy_for_metrics(with_metrics) + all_results["attempt_policy"]["score_policy"] = score_policy_for_metrics(with_metrics) with_custom_scores = average_custom_metrics(with_logical_rewards) with_pass = _pass_summary( with_logical_rewards, @@ -3112,7 +3135,9 @@ def collect_harbor_results( with_custom_scores = {} with_pass = {} with_overall_score = ( - _average_overall(with_logical_rewards) if with_execution["execution_status"] == "succeeded" else None + _aggregate_condition_overall(with_scores, with_metrics, with_logical_rewards) + if with_execution["execution_status"] == "succeeded" + else None ) _save_trials( with_collected_rewards, @@ -3135,6 +3160,7 @@ def collect_harbor_results( "overall_score": with_overall_score, "metric_set": with_metric_set, "metrics": list(with_metrics), + "score_policy": score_policy_for_metrics(with_metrics), "dimensions": dimension_scores(with_scores), "num_trials": len(with_rewards), "pass_at_k": with_pass, @@ -3270,7 +3296,7 @@ def collect_harbor_results( without_custom_scores = {} without_pass = {} without_overall_score = ( - _average_overall(without_logical_rewards) + _aggregate_condition_overall(without_scores, without_metrics, without_logical_rewards) if without_execution["execution_status"] == "succeeded" else None ) @@ -3295,6 +3321,7 @@ def collect_harbor_results( "overall_score": without_overall_score, "metric_set": without_metric_set, "metrics": list(without_metrics), + "score_policy": score_policy_for_metrics(without_metrics), "dimensions": dimension_scores(without_scores), "num_trials": len(without_rewards), "pass_at_k": without_pass, @@ -3333,6 +3360,7 @@ def collect_harbor_results( "overall_score": None, "metric_set": DEFAULT_METRIC_SET, "metrics": list(DISPLAY_METRICS), + "score_policy": DEFAULT_SCORE_POLICY, "dimensions": {}, "num_trials": 0, "pass_at_k": {}, @@ -3369,6 +3397,7 @@ def collect_harbor_results( "custom_scores": {}, "overall_score": None, "metrics": [], + "score_policy": DEFAULT_SCORE_POLICY, "dimensions": {}, "num_trials": 0, "pass_at_k": {}, diff --git a/src/skillevaluator/tier3/harbor/metrics.py b/src/skillevaluator/tier3/harbor/metrics.py index 98ed8658..f53d3d23 100644 --- a/src/skillevaluator/tier3/harbor/metrics.py +++ b/src/skillevaluator/tier3/harbor/metrics.py @@ -6,14 +6,19 @@ from __future__ import annotations import math +from collections.abc import Iterable, Mapping from typing import Any -from skillevaluator.constants import DIMENSION_MAPPING +from skillevaluator.constants import DEFAULT_SCORE_POLICY, DIMENSION_MAPPING DEFAULT_METRIC_SET = "skill-evaluator-default-v2" LEGACY_METRIC_SET = "skill-evaluator-default-v1" CUSTOM_ONLY_METRIC_SET = "custom-only" +LEGACY_SCORE_POLICY = "skill-evaluator-metric-mean-v1" +CUSTOM_SCORE_POLICY = "custom-overall-v1" +PARTIAL_SCORE_POLICY = "legacy-partial-dimension-mean-v1" + DEFAULT_METRICS = ( "security", "skill_execution", @@ -182,19 +187,59 @@ def average_metrics(rewards: list[dict[str, Any]]) -> tuple[dict[str, float], st return averages, metric_set, metrics +def score_policy_for_metrics(metrics: tuple[str, ...]) -> str: + """Return the versioned overall-score policy for one metric set.""" + if metrics == DEFAULT_METRICS: + return DEFAULT_SCORE_POLICY + if metrics == LEGACY_METRICS: + return LEGACY_SCORE_POLICY + if not metrics: + return CUSTOM_SCORE_POLICY + return PARTIAL_SCORE_POLICY + + +def canonical_dimension_mean(values: Iterable[object]) -> float | None: + """Average one complete canonical dimension set, failing closed on gaps.""" + raw_values = tuple(values) + if len(raw_values) != len(DIMENSION_DEFINITIONS): + return None + numeric = tuple(_finite_number(value) for value in raw_values) + if any(value is None for value in numeric): + return None + return sum(value for value in numeric if value is not None) / len(numeric) + + +def overall_score_from_metrics( + scores: Mapping[str, object], + metrics: tuple[str, ...] = DEFAULT_METRICS, +) -> float | None: + """Aggregate one metric set using its versioned score policy.""" + values = tuple(_finite_number(scores.get(metric)) for metric in metrics) + if not values or any(value is None for value in values): + return None + + if metrics != LEGACY_METRICS and all(metric in DEFAULT_METRICS for metric in metrics): + dimensions = dimension_scores(dict(zip(metrics, values, strict=True))) + dimension_values = tuple( + dimensions[dimension]["score"] for dimension in DIMENSION_DEFINITIONS if dimension in dimensions + ) + if metrics == DEFAULT_METRICS: + return canonical_dimension_mean(dimension_values) + return sum(dimension_values) / len(dimension_values) if dimension_values else None + + return sum(value for value in values if value is not None) / len(values) + + def overall_score(reward: dict[str, Any]) -> float | None: """Compute pass@k/lift overall score for a reward payload. - SkillEvaluator default rewards use the mean of their active SkillEvaluator metric set. Custom - rewards without SkillEvaluator metrics can still pass through by emitting numeric - ``overall``. + Default rewards use the equal-weight mean of the five canonical dimensions. + Legacy metric sets retain their historical metric mean. Custom rewards without + SkillEvaluator metrics can still pass through by emitting numeric ``overall``. """ _, metrics = metric_set_for_reward(reward) - values = [metric_value(reward, m) for m in metrics] if metrics: - if not values or any(value is None for value in values): - return None - return sum(value for value in values if value is not None) / len(values) + return overall_score_from_metrics({metric: metric_value(reward, metric) for metric in metrics}, metrics) return _finite_number(reward.get("overall")) @@ -202,8 +247,15 @@ def overall_score(reward: dict[str, Any]) -> float | None: def score_definition(metrics: tuple[str, ...] = DEFAULT_METRICS) -> str: """Human-readable definition for the SkillEvaluator overall score.""" if not metrics: - return "overall = user-provided reward overall" - return "overall = mean(" + ", ".join(metrics) + ")" + return f"overall = user-provided reward overall [{CUSTOM_SCORE_POLICY}]" + if metrics == DEFAULT_METRICS: + return ( + "overall = mean(Security, Correctness, Discoverability, Effectiveness, Efficiency) " + f"[{DEFAULT_SCORE_POLICY}]" + ) + if metrics == LEGACY_METRICS: + return "overall = mean(" + ", ".join(metrics) + f") [{LEGACY_SCORE_POLICY}]" + return f"overall = mean(available canonical dimensions) [{PARTIAL_SCORE_POLICY}]" def dimension_scores(scores: dict[str, float]) -> dict[str, dict[str, Any]]: diff --git a/src/skillevaluator/tier3/harbor/report.py b/src/skillevaluator/tier3/harbor/report.py index 91d557f1..594b06f0 100644 --- a/src/skillevaluator/tier3/harbor/report.py +++ b/src/skillevaluator/tier3/harbor/report.py @@ -24,6 +24,7 @@ METRIC_DISPLAY, METRIC_QUESTIONS, extract_custom_metrics, + overall_score_from_metrics, ) from skillevaluator.utils.redaction import redact_sensitive_data, redact_sensitive_text @@ -92,8 +93,10 @@ def _pick_best_agent( with_scores = data.get("with_skill", {}) if not with_scores: continue - metrics = [m for m in DISPLAY_METRICS if m in with_scores] or list(DISPLAY_METRICS) - overall = sum(with_scores.get(m, 0.0) for m in metrics) / len(metrics) + metrics = tuple(m for m in DISPLAY_METRICS if m in with_scores) or DISPLAY_METRICS + overall = overall_score_from_metrics(with_scores, metrics) + if overall is None: + continue if overall > best_score: best_score = overall best_agent = agent diff --git a/src/skillevaluator/tier3/harbor/report_data.py b/src/skillevaluator/tier3/harbor/report_data.py index 31316322..9b76c537 100644 --- a/src/skillevaluator/tier3/harbor/report_data.py +++ b/src/skillevaluator/tier3/harbor/report_data.py @@ -529,6 +529,11 @@ def load_agent_data( overall_key = "overall_with_skill" if variant == "with-skill" else "overall_without_skill" if "overall_score" in data: agent_info[overall_key] = data.get("overall_score") + score_policy_key = ( + "score_policy_with_skill" if variant == "with-skill" else "score_policy_without_skill" + ) + if isinstance(data.get("score_policy"), str) and data["score_policy"].strip(): + agent_info[score_policy_key] = data["score_policy"].strip() dimension_key = "dimensions_with_skill" if variant == "with-skill" else "dimensions_without_skill" if "dimensions" in data: agent_info[dimension_key] = data.get("dimensions", {}) diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.py index ff6d1b7d..bf077309 100644 --- a/src/skillevaluator/tier3/harbor/runner.py +++ b/src/skillevaluator/tier3/harbor/runner.py @@ -54,7 +54,7 @@ harbor_job_passed, validate_harbor_job_result, ) -from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, score_definition +from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, score_definition, score_policy_for_metrics from skillevaluator.tier3.harbor.progress import ( NullProgressReporter, ProgressEvent, @@ -2567,6 +2567,7 @@ def _emit_started_agents() -> None: "pass_threshold": float(pass_threshold), "stop_on_pass": bool(stop_on_pass), "score_definition": score_definition(tuple(results.get("metrics", DEFAULT_METRICS))), + "score_policy": score_policy_for_metrics(tuple(results.get("metrics", DEFAULT_METRICS))), }, } ) diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.py index 8627908d..15162f5a 100644 --- a/src/skillevaluator/tier3/harbor/templates/eval.py +++ b/src/skillevaluator/tier3/harbor/templates/eval.py @@ -3904,8 +3904,14 @@ def main(): logger.error("Required LLM judging failed for: %s", ", ".join(sorted(judge_errors))) raise SystemExit(1) - scores = [float(result[metric]) for metric in DISPLAY_METRICS] - overall = round(sum(scores) / len(scores), 4) + dimensions = ( + security_score, + acc_score, + se_score, + round((ga_score + bc_score) / 2, 4), + sef_score, + ) + overall = round(sum(dimensions) / len(dimensions), 4) write_reward_outputs(result, overall) diff --git a/src/skillevaluator/tier3/harbor/templates/metric.py b/src/skillevaluator/tier3/harbor/templates/metric.py index 5cb35c11..91cce43e 100644 --- a/src/skillevaluator/tier3/harbor/templates/metric.py +++ b/src/skillevaluator/tier3/harbor/templates/metric.py @@ -9,7 +9,7 @@ python metric.py -i rewards.jsonl -o metrics.json Input: JSONL where each line is one task's reward.json content. -Output: JSON with averaged 5-eval scores. +Output: JSON with six averaged evaluator scores and their five-dimension overall. """ import argparse @@ -72,7 +72,17 @@ def main(input_path: Path, output_path: Path) -> None: c = counts[metric] result[metric] = round(sums[metric] / c, 4) if c > 0 else 0.0 - result["overall"] = round(sum(result.values()) / len(result), 4) if result else 0.0 + if metrics == DEFAULT_METRICS: + dimensions = ( + result["security"], + result["accuracy"], + result["skill_execution"], + round((result["goal_accuracy"] + result["behavior_check"]) / 2, 4), + result["skill_efficiency"], + ) + result["overall"] = round(sum(dimensions) / len(dimensions), 4) + else: + result["overall"] = round(sum(result.values()) / len(result), 4) if result else 0.0 result["metric_set"] = "skill-evaluator-default-v2" if "security" in metrics else "skill-evaluator-default-v1" output_path.write_text(json.dumps(result, indent=2)) diff --git a/src/skillevaluator/tier3/result_display.py b/src/skillevaluator/tier3/result_display.py index 1c2a1920..f7b7adc2 100644 --- a/src/skillevaluator/tier3/result_display.py +++ b/src/skillevaluator/tier3/result_display.py @@ -25,6 +25,7 @@ DEFAULT_METRICS, DIMENSION_DISPLAY, METRIC_DISPLAY, + overall_score_from_metrics, ) from skillevaluator.tier3.harbor.progress import redact_progress_detail, secret_values_from_environment from skillevaluator.tier3.harbor.runner import format_harbor_view_command @@ -97,13 +98,8 @@ def _default_with_skill_overall(data: Mapping[str, Any]) -> float | None: scores = data.get("with_skill") if not isinstance(scores, Mapping): return None - values: list[float] = [] - for metric in DEFAULT_METRICS: - value = _finite_number(scores.get(metric)) - if value is None: - return None - values.append(value) - return round(sum(values) / len(values), 4) + score = overall_score_from_metrics(scores, DEFAULT_METRICS) + return round(score, 4) if score is not None else None def _custom_only_with_skill_overall(data: Mapping[str, Any]) -> float | None: diff --git a/src/skillevaluator/tier3/results_location.py b/src/skillevaluator/tier3/results_location.py index 09c30fb5..c793b81d 100644 --- a/src/skillevaluator/tier3/results_location.py +++ b/src/skillevaluator/tier3/results_location.py @@ -379,6 +379,18 @@ def _current_result_identity_is_valid(candidate: Path, run_config: dict[object, or not isinstance(attempt_policy.get("stop_on_pass"), bool) or not isinstance(attempt_policy.get("score_definition"), str) or not attempt_policy["score_definition"] + or ( + "score_policy" in attempt_policy + and (not isinstance(attempt_policy["score_policy"], str) or not attempt_policy["score_policy"]) + ) + ): + return False + result_score_policy = result.get("score_policy") + attempt_score_policy = attempt_policy.get("score_policy") + if "score_policy" in result and ( + not isinstance(result_score_policy, str) + or not result_score_policy + or (isinstance(attempt_score_policy, str) and result_score_policy != attempt_score_policy) ): return False return _recorded_path_matches(result.get("run_dir"), candidate) and _recorded_path_matches( diff --git a/tests/evaluation/test_tier3_benchmark_contract.py b/tests/evaluation/test_tier3_benchmark_contract.py index dcba337f..64d4c582 100644 --- a/tests/evaluation/test_tier3_benchmark_contract.py +++ b/tests/evaluation/test_tier3_benchmark_contract.py @@ -10,6 +10,7 @@ from skillevaluator import __version__ from skillevaluator.constants import ( + DEFAULT_SCORE_POLICY, DIMENSION_VERDICT_NEUTRAL_THRESHOLD, DIMENSION_VERDICT_PASS_THRESHOLD, TIER3_LIFT_FAIL_THRESHOLD, @@ -24,6 +25,7 @@ agent_eval_result_from_run, build_agent_eval_payload, ) +from skillevaluator.tier3.harbor.metrics import LEGACY_METRICS, LEGACY_SCORE_POLICY from skillevaluator.tier3.harbor.report_data import build_dataset_snapshot, load_dataset_snapshot @@ -81,6 +83,8 @@ def test_payload_exposes_report_truth_metadata() -> None: assert payload["evaluator_version"] == __version__ assert payload["dataset_digest"].startswith("sha256:") assert payload["dataset_digest_algorithm"] == "skill-evaluator-dataset-snapshot/1" + assert payload["score_policy"] == DEFAULT_SCORE_POLICY + assert payload["attempt_policy"]["score_policy"] == DEFAULT_SCORE_POLICY assert payload["dataset_summary"] == { "total_tasks": 3, "positive_tasks": 1, @@ -97,9 +101,36 @@ def test_payload_exposes_report_truth_metadata() -> None: "overall_pass_rule": "one_supported_agent_all_dimensions_pass", } assert payload["summary"]["dataset_summary"] == payload["dataset_summary"] + assert payload["summary"]["score_policy"] == payload["score_policy"] assert payload["summary"]["verdict_policy"] == payload["verdict_policy"] +def test_legacy_payload_preserves_the_versioned_metric_mean() -> None: + scores = { + "skill_execution": 0.2, + "skill_efficiency": 0.4, + "accuracy": 0.6, + "goal_accuracy": 0.8, + "behavior_check": 1.0, + } + + payload = build_agent_eval_payload( + "legacy-demo", + { + "opencode": { + "execution_status": "succeeded", + "metrics_with_skill": list(LEGACY_METRICS), + "with_skill": scores, + } + }, + use_llm_judge=False, + ) + + assert payload is not None + assert payload["overall_score"] == 0.6 + assert payload["score_policy"] == LEGACY_SCORE_POLICY + + def test_advisory_payload_exposes_same_report_truth_metadata() -> None: payload = _advisory_agent_eval_payload( "Tier 3 was skipped", @@ -110,6 +141,7 @@ def test_advisory_payload_exposes_same_report_truth_metadata() -> None: assert payload["evaluated_at"] is None assert payload["evaluator_version"] == __version__ + assert payload["score_policy"] == DEFAULT_SCORE_POLICY assert payload["dataset_summary"] == { "total_tasks": 0, "positive_tasks": 0, diff --git a/tests/golden/test_tier3_schema.py b/tests/golden/test_tier3_schema.py index daa58853..9e214a16 100644 --- a/tests/golden/test_tier3_schema.py +++ b/tests/golden/test_tier3_schema.py @@ -15,8 +15,10 @@ from skillevaluator.tier3.harbor.metrics import ( DEFAULT_METRIC_SET, DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, metric_set_for_reward, score_definition, + score_policy_for_metrics, ) @@ -47,7 +49,9 @@ def test_metric_set_for_reward_defaults_to_skill_evaluator() -> None: assert metrics == DEFAULT_METRICS -def test_score_definition_mentions_metrics() -> None: +def test_score_definition_identifies_the_versioned_dimension_policy() -> None: definition = score_definition(DEFAULT_METRICS) assert isinstance(definition, str) - assert "security" in definition + assert "Security" in definition + assert DEFAULT_SCORE_POLICY in definition + assert score_policy_for_metrics(DEFAULT_METRICS) == DEFAULT_SCORE_POLICY diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.py index 832d0f84..828134ec 100644 --- a/tests/reporting/test_unified_tier3_report.py +++ b/tests/reporting/test_unified_tier3_report.py @@ -19,6 +19,7 @@ _metric_evidence, _normalize_trials, _raw_trial_rewards, + _replace_with_minimal_payload, _ReportBudget, agent_eval_result_from_directory, build_agent_eval_payload, @@ -29,7 +30,12 @@ from skillevaluator.reporting import html as html_module from skillevaluator.reporting.html import PackageLoader, _compact_json from skillevaluator.tier3.harbor.collector import _paired_pass_comparison, _wilson_score_interval -from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS +from skillevaluator.tier3.harbor.metrics import ( + DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, + LEGACY_SCORE_POLICY, + PARTIAL_SCORE_POLICY, +) from skillevaluator.tier3.harbor.report_data import build_dataset_snapshot @@ -294,9 +300,179 @@ def test_authenticated_pre_status_rerender_preserves_historical_scores(tmp_path: payload = tier3.metadata["agent_eval"] assert payload["execution_status"] == "succeeded" assert payload["overall_score"] is not None + assert payload["score_policy"] == PARTIAL_SCORE_POLICY assert payload["agents"]["opencode"]["evaluators"]["security"]["with_skill"] == 0.8 +@pytest.mark.parametrize( + ("stored_overall", "recorded_policy", "expected_policy", "expected_score"), + [ + pytest.param(0.5583, None, LEGACY_SCORE_POLICY, 0.5583, id="stored-historical-overall"), + pytest.param(None, None, LEGACY_SCORE_POLICY, 0.5583, id="recomputed-historical-overall"), + pytest.param(2.0, None, LEGACY_SCORE_POLICY, 0.5583, id="out-of-range-overall"), + pytest.param(10**4000, None, LEGACY_SCORE_POLICY, 0.5583, id="overflowing-overall"), + pytest.param(None, DEFAULT_SCORE_POLICY, DEFAULT_SCORE_POLICY, 0.49, id="current-policy"), + ], +) +def test_default_v2_rerender_respects_recorded_or_historical_policy( + tmp_path: Path, + stored_overall: object, + recorded_policy: str | None, + expected_policy: str, + expected_score: float, +) -> None: + skill = tmp_path / "demo" + skill.mkdir() + run_dir = tmp_path / "results" / "20260709_120009" + summary = run_dir / "opencode" / "with-skill" / "summary.json" + summary.parent.mkdir(parents=True) + scores = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + summary_payload = { + "execution_status": "succeeded", + "execution_errors": [], + "expected_attempts": 1, + "scored_attempts": 1, + "scores": scores, + "metric_set": "skill-evaluator-default-v2", + "metrics": list(DEFAULT_METRICS), + "num_trials": 0, + } + if stored_overall is not None: + summary_payload["overall_score"] = stored_overall + if recorded_policy is not None: + summary_payload["score_policy"] = recorded_policy + summary.write_text(json.dumps(summary_payload), encoding="utf-8") + if recorded_policy is None: + comparison_summary = run_dir / "codex" / "with-skill" / "summary.json" + comparison_summary.parent.mkdir(parents=True) + comparison_summary.write_text( + json.dumps( + { + **summary_payload, + "scores": dict.fromkeys(DEFAULT_METRICS, 0.55), + "overall_score": 0.55, + } + ), + encoding="utf-8", + ) + (run_dir / "attempt_policy.json").write_text( + json.dumps( + { + "max_attempts": 1, + "pass_threshold": 0.5, + "stop_on_pass": False, + "score_definition": "overall = mean(security, skill_execution, skill_efficiency, accuracy, goal_accuracy, behavior_check)", + } + ), + encoding="utf-8", + ) + + tier3 = agent_eval_result_from_directory(skill, run_dir, use_llm_judge=False) + + assert tier3 is not None + payload = tier3.metadata["agent_eval"] + assert payload["overall_score"] == pytest.approx(expected_score) + assert payload["agents"]["opencode"]["with_skill"] == pytest.approx(expected_score) + assert payload["score_policy"] == expected_policy + assert payload["attempt_policy"]["score_policy"] == expected_policy + if expected_policy == LEGACY_SCORE_POLICY: + assert payload["best_agent"] == "opencode" + definition = payload["attempt_policy"]["score_definition"] + if expected_policy == LEGACY_SCORE_POLICY: + assert definition.startswith("overall = mean(security,") + else: + assert "mean(Security, Correctness, Discoverability, Effectiveness, Efficiency)" in definition + + +def test_default_v2_rerender_preserves_historical_baseline_lift_and_pass_at_k(tmp_path: Path) -> None: + skill = tmp_path / "demo" + skill.mkdir() + run_dir = tmp_path / "results" / "20260709_120010" + agent_dir = run_dir / "opencode" + with_scores = { + "security": 0.5, + "skill_execution": 0.5, + "skill_efficiency": 0.5, + "accuracy": 0.5, + "goal_accuracy": 1.0, + "behavior_check": 1.0, + } + without_scores = { + "security": 0.75, + "skill_execution": 0.75, + "skill_efficiency": 0.75, + "accuracy": 0.75, + "goal_accuracy": 0.25, + "behavior_check": 0.25, + } + with_pass = {"rate": 0.75, "passed_cases": 3, "total_cases": 4} + without_pass = {"rate": 0.25, "passed_cases": 1, "total_cases": 4} + + for condition, scores, overall, pass_at_k in ( + ("with-skill", with_scores, 0.6667, with_pass), + ("without-skill", without_scores, 0.5833, without_pass), + ): + summary = agent_dir / condition / "summary.json" + summary.parent.mkdir(parents=True) + summary.write_text( + json.dumps( + { + "execution_status": "succeeded", + "execution_errors": [], + "expected_attempts": 1, + "scored_attempts": 1, + "scores": scores, + "overall_score": overall, + "metric_set": "skill-evaluator-default-v2", + "metrics": list(DEFAULT_METRICS), + "num_trials": 0, + "pass_at_k": pass_at_k, + } + ), + encoding="utf-8", + ) + + (agent_dir / "lift.json").write_text( + json.dumps({"overall": {"with_skill": 0.6667, "without_skill": 0.5833, "delta": 0.0833}}), + encoding="utf-8", + ) + pass_lift = {"with_skill": 0.75, "without_skill": 0.25, "delta": 0.5} + (agent_dir / "pass_at_k_lift.json").write_text(json.dumps(pass_lift), encoding="utf-8") + (run_dir / "attempt_policy.json").write_text( + json.dumps( + { + "max_attempts": 1, + "pass_threshold": 0.5, + "stop_on_pass": False, + "score_definition": "overall = mean(security, skill_execution, skill_efficiency, accuracy, goal_accuracy, behavior_check)", + } + ), + encoding="utf-8", + ) + + tier3 = agent_eval_result_from_directory(skill, run_dir, use_llm_judge=False) + + assert tier3 is not None + payload = tier3.metadata["agent_eval"] + agent = payload["agents"]["opencode"] + assert payload["score_policy"] == LEGACY_SCORE_POLICY + assert agent["with_skill"] == pytest.approx(0.6667) + assert agent["baseline"] == pytest.approx(0.5833) + assert agent["lift"] == pytest.approx(0.0833) + assert agent["pass_at_k"] == { + "with_skill": with_pass, + "without_skill": without_pass, + "lift": pass_lift, + } + + def test_canonical_report_prefers_agentskills_dataset_fields() -> None: payload = build_agent_eval_payload( "hld-documents", @@ -986,6 +1162,24 @@ def test_canonical_payload_enforces_total_serialized_budget() -> None: assert len(encoded) <= truncation["payload_budget_bytes"] +def test_minimal_payload_preserves_score_policy() -> None: + payload = { + "summary": {"score_policy": DEFAULT_SCORE_POLICY}, + "score_policy": DEFAULT_SCORE_POLICY, + "attempt_policy": { + "score_definition": "x" * (3 * 1024 * 1024), + "score_policy": DEFAULT_SCORE_POLICY, + }, + } + + _replace_with_minimal_payload(payload, _ReportBudget()) + + assert payload["summary"]["score_policy"] == DEFAULT_SCORE_POLICY + assert payload["score_policy"] == DEFAULT_SCORE_POLICY + assert payload["attempt_policy"]["score_policy"] == DEFAULT_SCORE_POLICY + assert len(payload["attempt_policy"]["score_definition"]) == 1024 + + def test_unpaired_id_artifact_survives_loader_report_budget_and_html(tmp_path: Path) -> None: skill = tmp_path / "demo" skill.mkdir() diff --git a/tests/test_harbor_metrics_truth.py b/tests/test_harbor_metrics_truth.py index eaf33479..1ebdfaac 100644 --- a/tests/test_harbor_metrics_truth.py +++ b/tests/test_harbor_metrics_truth.py @@ -3,12 +3,15 @@ from __future__ import annotations +import json import math import os import subprocess import sys import time from fractions import Fraction +from itertools import product +from pathlib import Path import pytest @@ -30,11 +33,19 @@ CUSTOM_ONLY_METRIC_SET, DEFAULT_METRIC_SET, DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, + LEGACY_METRIC_SET, + LEGACY_METRICS, average_metrics, + canonical_dimension_mean, extract_custom_metrics, metric_value, overall_score, + overall_score_from_metrics, + score_definition, + score_policy_for_metrics, ) +from skillevaluator.tier3.harbor.report import _pick_best_agent @pytest.mark.parametrize("invalid", [float("nan"), float("inf"), float("-inf")]) @@ -73,6 +84,173 @@ def test_overall_score_requires_a_complete_finite_metric_set() -> None: assert overall_score({"metric_set": CUSTOM_ONLY_METRIC_SET, "overall": float("nan")}) is None +def test_default_overall_score_uses_the_canonical_dimension_mean() -> None: + reward = { + "metric_set": DEFAULT_METRIC_SET, + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + + # Effectiveness owns goal_accuracy + behavior_check as one of five + # dimensions. Averaging all six evaluators would incorrectly return 0.5583. + assert overall_score(reward) == pytest.approx(0.49) + assert score_policy_for_metrics(DEFAULT_METRICS) == DEFAULT_SCORE_POLICY + assert "mean(Security, Correctness, Discoverability, Effectiveness, Efficiency)" in score_definition( + DEFAULT_METRICS + ) + + +def test_best_agent_selection_uses_the_canonical_dimension_mean() -> None: + direction_reversing = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + agents = { + "metric-mean-winner": {"execution_status": "succeeded", "with_skill": direction_reversing}, + "dimension-mean-winner": { + "execution_status": "succeeded", + "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.5), + }, + } + + assert sum(direction_reversing.values()) / len(direction_reversing) > 0.5 + assert _pick_best_agent(agents) == "dimension-mean-winner" + + +def test_standalone_harbor_metric_aggregation_uses_the_canonical_dimension_mean(tmp_path: Path) -> None: + reward = { + "metric_set": DEFAULT_METRIC_SET, + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + input_path = tmp_path / "rewards.jsonl" + output_path = tmp_path / "metrics.json" + input_path.write_text(json.dumps(reward) + "\n", encoding="utf-8") + metric_script = Path(__file__).parents[1] / "src/skillevaluator/tier3/harbor/templates/metric.py" + + completed = subprocess.run( + [sys.executable, str(metric_script), "-i", str(input_path), "-o", str(output_path)], + check=False, + capture_output=True, + text=True, + ) + + assert completed.returncode == 0, completed.stderr + assert json.loads(output_path.read_text(encoding="utf-8"))["overall"] == 0.49 + + precision_reward = { + "metric_set": DEFAULT_METRIC_SET, + "security": 0.0001, + "skill_execution": 0.0001, + "skill_efficiency": 0.0001, + "accuracy": 0.0001, + "goal_accuracy": 0.0, + "behavior_check": 0.0007, + } + input_path.write_text(json.dumps(precision_reward) + "\n", encoding="utf-8") + completed = subprocess.run( + [sys.executable, str(metric_script), "-i", str(input_path), "-o", str(output_path)], + check=False, + capture_output=True, + text=True, + ) + + assert completed.returncode == 0, completed.stderr + assert json.loads(output_path.read_text(encoding="utf-8"))["overall"] == 0.0001 + + +def test_legacy_overall_score_preserves_the_historical_metric_mean() -> None: + reward = { + "metric_set": LEGACY_METRIC_SET, + "skill_execution": 0.2, + "skill_efficiency": 0.4, + "accuracy": 0.6, + "goal_accuracy": 0.8, + "behavior_check": 1.0, + } + + assert overall_score(reward) == pytest.approx(0.6) + assert "mean(skill_execution, skill_efficiency, accuracy, goal_accuracy, behavior_check)" in score_definition( + LEGACY_METRICS + ) + + +def test_partial_score_averages_available_dimensions_instead_of_evaluators() -> None: + scores = {"accuracy": 0.0, "goal_accuracy": 1.0, "behavior_check": 1.0} + metrics = ("accuracy", "goal_accuracy", "behavior_check") + + assert overall_score_from_metrics(scores, metrics) == pytest.approx(0.5) + + +def test_lift_and_pass_at_k_share_the_canonical_default_score() -> None: + baseline = dict.fromkeys(DEFAULT_METRICS, 0.5) + with_skill = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + + lift = _compute_lift(with_skill, baseline) + summary = _pass_summary( + [{"entry_id": "case-001", "_trial_name": "case-001__attempt1", **with_skill}], + n_attempts=1, + pass_threshold=0.5, + expected_cases=1, + expected_case_ids=["case-001"], + ) + + assert lift["overall"] == { + "with_skill": 0.49, + "without_skill": 0.5, + "delta": -0.01, + } + assert summary["cases"]["case-001"]["attempts"] == [ + { + "attempt": 1, + "trial": "case-001__attempt1", + "score": 0.49, + "passed": False, + } + ] + + +def test_default_score_policy_matches_the_five_dimension_contract_exhaustively() -> None: + for values in product((0.0, 0.5, 1.0), repeat=len(DEFAULT_METRICS)): + scores = dict(zip(DEFAULT_METRICS, values, strict=True)) + expected = ( + scores["security"] + + scores["accuracy"] + + scores["skill_execution"] + + (scores["goal_accuracy"] + scores["behavior_check"]) / 2 + + scores["skill_efficiency"] + ) / 5 + + assert overall_score_from_metrics(scores) == pytest.approx(expected) + + +@pytest.mark.parametrize( + "values", + [[], [0.5] * 4, [0.5] * 6, [0.5, 0.5, 0.5, 0.5, float("nan")]], +) +def test_canonical_dimension_mean_rejects_partial_or_invalid_dimension_sets(values: list[float]) -> None: + assert canonical_dimension_mean(values) is None + + def test_lift_omits_unpaired_metrics_and_incomplete_overall() -> None: lift = _compute_lift( {"security": 1.0, "accuracy": 0.8}, diff --git a/tests/test_issue55_collector_truth.py b/tests/test_issue55_collector_truth.py index 82feaae7..86b1d83d 100644 --- a/tests/test_issue55_collector_truth.py +++ b/tests/test_issue55_collector_truth.py @@ -14,7 +14,7 @@ import pytest -from skillevaluator.evaluation.tier3_report import render_agent_eval_html_report +from skillevaluator.evaluation.tier3_report import build_agent_eval_payload, render_agent_eval_html_report from skillevaluator.tier3.harbor import collector as collector_module from skillevaluator.tier3.harbor import report, report_data from skillevaluator.tier3.harbor.collector import ( @@ -27,6 +27,7 @@ CUSTOM_ONLY_METRIC_SET, DEFAULT_METRIC_SET, DEFAULT_METRICS, + DEFAULT_SCORE_POLICY, LEGACY_METRIC_SET, LEGACY_METRICS, overall_score, @@ -163,6 +164,59 @@ def _collect( ) +def test_collection_and_canonical_report_share_one_versioned_overall_score_policy(tmp_path: Path) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + baseline = {"entry_id": "case-001", "metric_set": DEFAULT_METRIC_SET, **dict.fromkeys(DEFAULT_METRICS, 0.5)} + with_skill = { + "entry_id": "case-001", + "metric_set": DEFAULT_METRIC_SET, + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + for variant, reward in (("with", with_skill), ("without", baseline)): + job_dir = tmp_path / "jobs" / f"demo-opencode-{variant}" + _write_reward(job_dir, "case-001__attempt", reward) + _write_complete_job_result(job_dir, ["case-001__attempt"]) + + result = _collect(tmp_path, skip_baseline=False, case_ids=["case-001"]) + agent = result["agents"]["opencode"] + payload = build_agent_eval_payload( + "demo", + result["agents"], + attempt_policy=result["attempt_policy"], + use_llm_judge=False, + ) + + assert result["score_policy"] == DEFAULT_SCORE_POLICY + assert result["attempt_policy"]["score_policy"] == DEFAULT_SCORE_POLICY + assert agent["lift"]["overall"] == {"with_skill": 0.49, "without_skill": 0.5, "delta": -0.01} + assert agent["pass_at_k"]["with_skill"]["cases"]["case-001"]["best_score"] == 0.49 + with_summary = json.loads( + (tmp_path / "results" / "opencode" / "with-skill" / "summary.json").read_text(encoding="utf-8") + ) + assert with_summary["overall_score"] == 0.49 + assert with_summary["score_policy"] == DEFAULT_SCORE_POLICY + assert payload is not None + assert payload["score_policy"] == DEFAULT_SCORE_POLICY + assert payload["summary"]["score_policy"] == DEFAULT_SCORE_POLICY + assert payload["overall_score"] == 0.49 + assert payload["overall_lift"] == -0.01 + + report_path = render_agent_eval_html_report(skill_path, tmp_path / "results", use_llm_judge=False) + report_html = report_path.read_text(encoding="utf-8") + assert DEFAULT_SCORE_POLICY in report_html + assert "-0.01" in report_html + + persisted_policy = json.loads((tmp_path / "results" / "attempt_policy.json").read_text(encoding="utf-8")) + assert persisted_policy["score_policy"] == DEFAULT_SCORE_POLICY + assert DEFAULT_SCORE_POLICY in persisted_policy["score_definition"] + + def test_failed_judge_sidecar_is_merged_but_never_scored_and_reason_is_safe(tmp_path: Path) -> None: job_dir = tmp_path / "jobs" / "demo-opencode-with" trial_name = "case-001__attempt" diff --git a/tests/test_results_location.py b/tests/test_results_location.py index 758b6287..26acbebe 100644 --- a/tests/test_results_location.py +++ b/tests/test_results_location.py @@ -13,6 +13,7 @@ import pytest +from skillevaluator.constants import DEFAULT_SCORE_POLICY from skillevaluator.tier3 import results_location from skillevaluator.tier3.output_provenance import mark_generated_output_root from skillevaluator.tier3.results_location import external_results_root, resolve_latest_results @@ -99,11 +100,13 @@ def _write_authenticated_current_run( "result_path": recorded_result_path or str((run_dir / "result.json").resolve()), "run_config": run_config, "agents": {"opencode": agent_result}, + "score_policy": DEFAULT_SCORE_POLICY, "attempt_policy": { "max_attempts": 1, "pass_threshold": 0.5, "stop_on_pass": False, "score_definition": "test", + "score_policy": DEFAULT_SCORE_POLICY, }, "execution_status": "succeeded", "execution_errors": [], @@ -267,6 +270,68 @@ def test_latest_results_rejects_authenticated_current_run_with_malformed_result_ assert resolve_latest_results(skill_path, cli_results_dir, environ={}) == historical +@pytest.mark.parametrize("invalid_score_policy", ["", 1, None]) +def test_latest_results_rejects_invalid_recorded_score_policy(tmp_path: Path, invalid_score_policy: object) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + cli_results_dir = tmp_path / "results" + root = external_results_root(cli_results_dir, skill_path) + historical = _write_completed_run(root, "20260705_120000") + current = _write_authenticated_current_run(root, "20260705_130000_222_bbbbbbbbbbbb") + result_path = current / "result.json" + result = json.loads(result_path.read_text(encoding="utf-8")) + result["attempt_policy"]["score_policy"] = invalid_score_policy + result_path.write_text(json.dumps(result), encoding="utf-8") + + assert resolve_latest_results(skill_path, cli_results_dir, environ={}) == historical + + +@pytest.mark.parametrize("invalid_score_policy", ["", 1, None]) +def test_latest_results_rejects_invalid_top_level_score_policy(tmp_path: Path, invalid_score_policy: object) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + cli_results_dir = tmp_path / "results" + root = external_results_root(cli_results_dir, skill_path) + historical = _write_completed_run(root, "20260705_120000") + current = _write_authenticated_current_run(root, "20260705_130000_222_bbbbbbbbbbbb") + result_path = current / "result.json" + result = json.loads(result_path.read_text(encoding="utf-8")) + result["score_policy"] = invalid_score_policy + result_path.write_text(json.dumps(result), encoding="utf-8") + + assert resolve_latest_results(skill_path, cli_results_dir, environ={}) == historical + + +def test_latest_results_rejects_mismatched_score_policies(tmp_path: Path) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + cli_results_dir = tmp_path / "results" + root = external_results_root(cli_results_dir, skill_path) + historical = _write_completed_run(root, "20260705_120000") + current = _write_authenticated_current_run(root, "20260705_130000_222_bbbbbbbbbbbb") + result_path = current / "result.json" + result = json.loads(result_path.read_text(encoding="utf-8")) + result["score_policy"] = "different-policy-v1" + result_path.write_text(json.dumps(result), encoding="utf-8") + + assert resolve_latest_results(skill_path, cli_results_dir, environ={}) == historical + + +def test_latest_results_accepts_authenticated_current_run_without_historical_score_policy(tmp_path: Path) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + cli_results_dir = tmp_path / "results" + root = external_results_root(cli_results_dir, skill_path) + current = _write_authenticated_current_run(root, "20260705_130000_222_bbbbbbbbbbbb") + result_path = current / "result.json" + result = json.loads(result_path.read_text(encoding="utf-8")) + result["attempt_policy"].pop("score_policy") + result.pop("score_policy") + result_path.write_text(json.dumps(result), encoding="utf-8") + + assert resolve_latest_results(skill_path, cli_results_dir, environ={}) == current + + def test_latest_results_accepts_authenticated_current_run_with_relative_recorded_path_after_cwd_change( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_tier3_compare.py b/tests/test_tier3_compare.py index 3298fc39..67845232 100644 --- a/tests/test_tier3_compare.py +++ b/tests/test_tier3_compare.py @@ -11,7 +11,8 @@ import pytest -from skillevaluator.tier3.commands import compare_results +from skillevaluator.tier3.commands import _overall_score_for_display, _summary_overall, compare_results +from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, DEFAULT_SCORE_POLICY from skillevaluator.tier3.output_provenance import mark_generated_output_root @@ -23,6 +24,117 @@ def _complete_current_run(run_dir: Path) -> None: ) +def test_compare_overall_uses_the_canonical_dimension_mean() -> None: + scores = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + + assert _overall_score_for_display(scores, DEFAULT_METRICS) == pytest.approx(0.49) + + +def test_overall_display_preserves_pre_policy_default_v2_semantics() -> None: + scores = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + + assert _overall_score_for_display(scores, DEFAULT_METRICS, score_policy=None) == pytest.approx(0.5583) + assert _overall_score_for_display( + scores, + DEFAULT_METRICS, + persisted_overall=0.5583, + score_policy=None, + ) == pytest.approx(0.5583) + assert _overall_score_for_display( + scores, + DEFAULT_METRICS, + persisted_overall=0.5583, + score_policy=DEFAULT_SCORE_POLICY, + ) == pytest.approx(0.49) + + +@pytest.mark.parametrize("invalid_overall", [2.0, -0.1, float("inf"), 10**4000, True]) +def test_historical_overall_rejects_invalid_persisted_scores(invalid_overall: object) -> None: + scores = { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + + assert _summary_overall({"overall_score": invalid_overall}) is None + assert _overall_score_for_display( + scores, + DEFAULT_METRICS, + persisted_overall=invalid_overall, + score_policy=None, + ) == pytest.approx(0.5583) + + +def test_compare_carries_persisted_historical_overall( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + skill_path = tmp_path / "demo" + skill_path.mkdir() + run_dir = tmp_path / "results" / "demo" / "20260709_010000" + summary = run_dir / "opencode" / "with-skill" / "summary.json" + summary.parent.mkdir(parents=True) + summary.write_text( + json.dumps( + { + "execution_status": "succeeded", + "scores": { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + }, + "overall_score": 0.5583, + "metric_set": "skill-evaluator-default-v2", + "metrics": list(DEFAULT_METRICS), + "num_trials": 1, + } + ), + encoding="utf-8", + ) + baseline_summary = run_dir / "opencode" / "without-skill" / "summary.json" + baseline_summary.parent.mkdir(parents=True) + baseline_summary.write_text( + json.dumps( + { + "execution_status": "succeeded", + "scores": dict.fromkeys(DEFAULT_METRICS, 0.5), + "overall_score": 0.5, + "metric_set": "skill-evaluator-default-v2", + "metrics": list(DEFAULT_METRICS), + "num_trials": 1, + } + ), + encoding="utf-8", + ) + _complete_current_run(run_dir) + + assert compare_results(skill_path, results_dir=tmp_path / "results") == 0 + output = capsys.readouterr().out + assert "0.56" in output + assert "0.49" not in output + assert "+0.06" in output + + def _write_authentic_pre_status_run(root: Path, run_id: str, *, score: float = 0.8) -> Path: run_dir = root / run_id summary_dir = run_dir / "opencode" / "with-skill" diff --git a/tests/test_tier3_result_display.py b/tests/test_tier3_result_display.py index 5b246b7d..77e88bd2 100644 --- a/tests/test_tier3_result_display.py +++ b/tests/test_tier3_result_display.py @@ -91,6 +91,21 @@ def test_skip_baseline_success_uses_canonical_default_metric_aggregate() -> None assert "skipped" in output +def test_skip_baseline_default_overall_uses_the_dimension_score_policy() -> None: + agent = _skip_baseline_agent( + { + "security": 0.3875, + "skill_execution": 0.3875, + "skill_efficiency": 0.3875, + "accuracy": 0.3875, + "goal_accuracy": 0.9, + "behavior_check": 0.9, + } + ) + + assert _with_skill_overall(agent, DEFAULT_METRIC_SET) == pytest.approx(0.49) + + def test_skip_baseline_custom_only_uses_persisted_attempt_scores() -> None: agent = _skip_baseline_agent({}, pass_rate=0.5) agent["pass_at_k"] = { diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.py index fbe8225f..715764b7 100644 --- a/tests/tier3/test_judge_failure_artifacts.py +++ b/tests/tier3/test_judge_failure_artifacts.py @@ -190,8 +190,8 @@ def judge(*_args, **_kwargs): "goal_accuracy": 0.0, "behavior_check": 0.0, } - assert numeric["overall"] == 0.5 - assert overall_score(numeric) == 0.5 + assert numeric["overall"] == 0.6 + assert overall_score(numeric) == 0.6 def test_verifier_main_recovers_malformed_accuracy_and_goal_judges( diff --git a/tests/tier3/test_negative_control_invocation_evidence.py b/tests/tier3/test_negative_control_invocation_evidence.py index a6536483..4ffbb68f 100644 --- a/tests/tier3/test_negative_control_invocation_evidence.py +++ b/tests/tier3/test_negative_control_invocation_evidence.py @@ -13,6 +13,7 @@ import pytest from skillevaluator.tier3.eval_core import checks as shared_checks +from skillevaluator.tier3.harbor.metrics import overall_score def _load_template(): @@ -498,11 +499,7 @@ def passing_judge(*_args, **_kwargs): result = json.loads(rich_reward_json.read_text(encoding="utf-8")) harbor_reward = json.loads(reward_json.read_text(encoding="utf-8")) expected_reward_keys = {*TEMPLATE.DISPLAY_METRICS, "overall"} - expected_overall = round( - sum(float(result.get(metric, 0.0) or 0.0) for metric in TEMPLATE.DISPLAY_METRICS) - / len(TEMPLATE.DISPLAY_METRICS), - 4, - ) + expected_overall = overall_score(result) assert result["skill_execution"] == 0.0 if include_trusted_identity: