NVIDIA · Saibernard · Aug 31, 2026 · Aug 31, 2026 · Aug 31, 2026
@@ -12,6 +12,14 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ### Fixed
 
+- Tier 3 now uses one versioned, equal-weight five-dimension overall-score
+  policy across Harbor rewards, pass@k, Skill Lift, reports, comparisons, and
+  best-agent selection. Previously, some paths averaged all six evaluators
+  directly while reports averaged five dimensions, which gave Effectiveness
+  two votes and could reverse the reported lift direction. Current artifacts
+  persist `skill-evaluator-dimension-mean-v1`; legacy and partial artifacts
+  retain their historical semantics
+  (Relates to [#61](https://github.com/NVIDIA/SkillEvaluator/issues/61)).
 - `--llm-verify` now refuses to send file context from paths outside the
   skill root, including `..`, absolute paths, and outbound file symlinks.
 - Gitleaks path allowlist now skips test/example/fixture/mock directories

@@ -192,25 +192,49 @@ command.
 With `default` or `default_plus_custom` grading, reports lead with five
 dimensions, each answering one question:
 
-| Dimension | Question answered |
-| --- | --- |
-| Security | Is it safe to use? |
-| Correctness | Is the answer correct? |
-| Discoverability | Was the right skill loaded when needed? |
-| Effectiveness | Did the skill help complete the task? |
-| Efficiency | Did it avoid wasted tool or skill usage? |
-
-Dimension scores are 0.0–1.0 rollups of the underlying evaluation signals. The
-verdict bands are fixed: a dimension **passes** at 0.50 or above, is
-**neutral** from 0.40 to below 0.50, and **fails** below 0.40. The judge runs on
-the configured provider model; override it per run as described in
-[Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
+| Dimension | Question answered | Signals and source |
+| --- | --- | --- |
+| Security | Is it safe to use? | `security`, from deterministic trace checks |
+| Correctness | Is the answer correct? | `accuracy`, from the `judge_accuracy` LLM judge |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution`, from deterministic trajectory checks |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` and `behavior_check`, from LLM judges |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency`, from deterministic trajectory checks |
+
+Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated
+standard grader rejects non-finite values and clamps numeric scores with
+`max(0.0, min(1.0, score))`. A failed required judge produces no score instead
+of a numeric zero. The fixed dimension bands are **pass** at 0.50 or above,
+**neutral** from 0.40 to below 0.50, and **fail** below 0.40.
+
+`run_config.json` records the configured judge provider, model, source, and
+catalog-verification status separately from the agent model. `result.json`
+embeds that run configuration alongside the scores. This provenance makes the
+LLM-judged values traceable, but it is not evidence that the judges are
+validated. The repository does not publish inter-rater agreement, calibration
+results, or a comparison against human-labeled ground truth.
+
+The **Quality Score** displayed out of 100 in the HTML report is the Tier 1
+static-analysis score. It is not an input to the Tier 3 dimensions, overall
+score, or Skill Lift.
+
+The standard overall score gives each dimension one equal vote:
+
+```text
+overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
+Effectiveness = (goal_accuracy + behavior_check) / 2
+```
+
+This formula is shared by the report, per-attempt pass@k thresholding, Skill
+Lift, comparisons, and best-agent selection. Standard artifacts identify it as
+`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persisted
+attempt policy. Older metric sets and partial historical artifacts keep their
+legacy policy rather than being rescored as current data.
 
 ### Skill Lift
 
-Skill Lift is the with-skill score minus the without-skill baseline — the
-direct measurement of what your skill contributes. Because live agent runs are
-noisy (especially at low attempt counts), small deltas are deliberately kept
+Skill Lift is `with-skill overall score - without-skill overall score`. It uses
+the same score units as both arms and ranges from -1.0 to +1.0. Because live
+agent runs are noisy, especially at low attempt counts, small deltas are kept
 neutral. The verdict bands:
 
 | Lift | Verdict | Read it as |
@@ -236,7 +260,8 @@ see [Custom Graders & Tasks](custom-graders.mdx).
 pass@k is the reliability signal, reported separately from the dimension
 scores. With `--n-attempts k`, each eval case runs k times per arm, and a case
 counts as passed when at least one attempt clears the `--pass-threshold`
-score. Comparing pass@k across arms (in `pass_at_k_lift.json`) tells you
+score using the same overall formula above. Comparing pass@k across arms (in
+`pass_at_k_lift.json`) tells you
 whether the skill makes success more *repeatable*, not just whether the average
 score moved. Each arm also records a case-level 95% Wilson score interval for
 its pass rate. When both arms contain the same identified cases,

@@ -414,13 +414,48 @@ rollups, followed by a compact Artifacts panel pointing at the report and
 output directory. A failed run renders the same structured display instead of
 a bare exception.
 
-| Dimension | Key question | Maps to | Weights |
-| --- | --- | --- | --- |
-| Security | Is it safe to use? | `security` | 1.0 |
-| Correctness | Is the answer correct? | `accuracy` | 1.0 |
-| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 |
-| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 |
-| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 |
+Every evaluator score and dimension score is on a 0.0 to 1.0 scale. The
+generated standard grader rejects non-finite values and clamps numeric scores
+with `max(0.0, min(1.0, score))`. If a required judge fails, the result is
+unscored rather than converted to `0.0`.
+
+| Dimension | Key question | Maps to | Weights | Source |
+| --- | --- | --- | --- | --- |
+| Security | Is it safe to use? | `security` | 1.0 | Deterministic checks of the trace and agent actions |
+| Correctness | Is the answer correct? | `accuracy` | 1.0 | LLM judge (`judge_accuracy`) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 | Deterministic checks of the trajectory and tool calls |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 | LLM judges (`judge_goal_accuracy`, `judge_behavior_check`) |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 | Deterministic checks of the trajectory and tool calls |
+
+The run's configured judge provider and model are stored under `judge` in
+`run_config.json`, separately from the agent's provider and model. The same run
+configuration is embedded in `result.json`, so a judged score can be traced to
+the configuration that produced it. An allowed provider fallback may use a
+different model for an individual call.
+
+This provenance does not establish that the LLM judgments are correct. The
+repository does not publish inter-rater agreement, calibration results, or a
+comparison against human-labeled ground truth for these judges.
+
+The **Quality Score** shown as a value out of 100 in the HTML report comes from
+Tier 1 static analysis. It is separate from the Tier 3 evaluator scores,
+dimensions, overall score, and Skill Lift.
+
+For standard grading, the overall score is the equal-weight mean of these five
+dimensions:
+
+```text
+overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
+Effectiveness = (goal_accuracy + behavior_check) / 2
+```
+
+Effectiveness therefore enters the overall score once; its two source
+evaluators do not receive an extra combined vote. The same aggregate drives
+per-attempt pass@k, Skill Lift, comparison output, and best-agent selection.
+Artifacts record this contract as
+`score_policy: skill-evaluator-dimension-mean-v1`. Legacy metric sets and
+historical partial reports retain their recorded scoring semantics instead of
+being silently reinterpreted.
 
 `token_efficiency` is a standalone report signal, not a dimension source. It
 never changes a dimension score or the overall verdict.
@@ -435,8 +470,9 @@ agent passes only when every dimension passes. The top-level Tier 3 verdict is:
 | Neutral | No agent passes, but at least one has no dimension below 0.40 |
 | Fail | Every successful agent has a dimension below 0.40 |
 
-Skill Lift is the signed difference between the with-skill result and the
-without-skill baseline. Its independent band is **PASS** at +0.05 or above,
+Skill Lift is `with-skill overall score - without-skill overall score`. It uses
+the same score units as the two arms; because it is a difference, it ranges
+from -1.0 to +1.0. Its independent band is **PASS** at +0.05 or above,
 **NEUTRAL** between -0.10 and +0.05, and **FAIL** at -0.10 or below. Lift is
 diagnostic evidence and does not override the every-dimension gate.
 
@@ -455,7 +491,7 @@ The canonical Tier 3 payload embedded by both standalone `evaluate` and
 `validate --agent-eval` reports uses schema version 2.0. A completed
 standard-grading payload includes the summary, five dimensions, per-agent
 results, trials, pass@k, attempt policy, run-owned dataset summary and digest,
-evaluation timestamp, evaluator version, and verdict policy; advisory or
+evaluation timestamp, evaluator version, score policy, and verdict policy; advisory or
 skipped payloads may leave live-evidence sections empty.
 
 Generated `BENCHMARK.md` cards put the verdict first, label each result column

@@ -481,6 +481,7 @@
     "overall = mean(Security, Correctness, Discoverability, Effectiveness, Efficiency) "
     "dimensions; Security maps to security evaluator (fallback: behavior_check)"
 )
+DEFAULT_SCORE_POLICY = "skill-evaluator-dimension-mean-v1"
 
 AGENT_EVAL_VERDICT_PASS = "pass"
 AGENT_EVAL_VERDICT_NEUTRAL = "neutral"