Alexeybalexeyb/spa 89 restore loop axis badge

docs/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 Формат: `YYYY-MM-DD — что изменилось — ссылка на блок плана / PR`.
 
+## 2026-08-21 (4)
+
+- **SPA-89 — The one reliability badge that costs no human annotation is restored, and stops being computed on the judge's silence.** The deterministic loop counter (SPA-75) scores repetition on every trajectory-scored trace, so judge↔counter Cohen's κ is available for free on the `loop_detection` axis where the other five stay `not_calibrated` until somebody rates runs by hand. That mechanism was written and **never ran**: schema v11 removed the loop axis from the badged set, and the structural fallback lives *inside the same loop* the axis was skipped by — `struct_ok = key == "loop_detection" and …` could not be true. Everything else said otherwise: the function's own docstring, `api.md` («6 E-07 trajectory axes»), the report UI heading («6-axis process judge per config») and its footnote («the loop axis instead anchors to the deterministic counter»). Five texts described a mechanism; one — the metrics glossary — described the code.
+  - **The gate decides which axes are weighed, not a constant.** v11 hardcoded `_AGG_EXCLUDED_AXES = {'loop_detection'}` because SPA-76 promised a quarantined axis is not weighed and the stored `overall_score` averaged it in anyway — a stand-in for a gate that did not exist yet. Since SPA-88 the gate acts, so the exception is deleted: `_traj_score` averages every axis the judge scored and the trusted view's `allowed` set is the only thing that removes one. The loop axis is dropped from a conclusion exactly when its own anchor says the judge cannot be trusted on it — which on every corpus measured so far it does, so the *outcome* is v11's; what changed is that it is now earned per experiment instead of asserted once. Keeping the hardcode while restoring the badge would have been worse than either: SPA-88 builds its numeric set from the badges, so the axis would have been listed as trusted and then silently ignored.
+  - **A judge that never answered is not a judge that said «no loop».** `loop_detected` is derived from the loop axis and is `False` both when the judge scored it above the threshold and when it never scored it at all — axis absent, or returned not-applicable. The report read that flag straight into the judge↔counter 2×2, so every silent run joined the both-clean cell whenever the counter also found nothing, and lifted the κ. Priced on the ~715-run corpus the anchor was validated against: **κ ≈ 0.33 under that convention, ≈ 0.28 over the runs both raters actually rated** — and the error runs toward *more* agreement than was earned, which since SPA-88 is the direction that grants trust. A run now enters the 2×2 only when both raters answered; the rest are reported as `n_judge_unscored` rather than dropped quietly, `loop_rate` is over `n_judge_scored`, and `structural_loop_rate` keeps its own wider denominator because the counter ran on them regardless. Same principle as SPA-114's token split: absence is not zero.
+  - **A structural row can never be rank-rescued.** The anchor is binary, so it carries no Spearman ρ — SPA-79's rescue of a scale-shifted judge has nothing to work with, and a low structural κ lands on `unreliable` with no second chance. Stated in code and docs so it is not mistaken for an omission. A human who rated the axis still outranks the counter: the anchor is a fallback, not a competing gold standard.
+  - **Post-review: a boolean reference cannot certify a 0–10 score.** Restoring the axis to the aggregate opened a hazard the v11 hardcode had been accidentally covering: the anchor compares two *booleans*, but a high κ used to climb to `reliable_absolute`, which admits the axis to the numeric set — and then the judge's unvalidated 0–10 loop score lands in the trusted trajectory mean. A perfect binary κ is compatible with the judge scoring 0 where a human would score 4 and 10 where a human would score 5: the threshold-crossing agrees, the magnitude is untouched. Reproduced before fixing (κ = 0.85 → `reliable_absolute` → trusted trajectory mean 0.0, driven entirely by the uncertified axis). A structurally-sourced axis now caps at a seventh trust zone, **`binary_only`**, which drives the binary loop rate and neither a mean nor a rank test on the score — the `rank_only` rule one step further down, since an axis trusted for one dichotomy cannot claim an ordering either. Below the bar it is still plainly `unreliable`: the cap cuts one way. Each axis row carries `evidence: graded | binary`, so the ceiling follows what was measured rather than which axis it is — a human rating the loop axis climbs normally.
+  - **Post-review: a source that produced nothing is not the absence of a source.** With the κ now admitting only runs both raters answered, an experiment whose judge never scored the loop axis yields zero pairs — and the axis was reporting `not_calibrated`, i.e. «nobody has ever checked this», while the counter was in fact running on every trace. It reports `source: structural`, `n: 0`, `insufficient` instead. The two states call for opposite actions: fix the judge, or go and annotate.
+  - **`SCHEMA_VERSION` 20→21** — the raw trajectory aggregate goes from a five-axis to a six-axis mean, so every cached report recomputes. The trajectory heatmap has six columns again, and «6-axis» in `api.md` and in the UI heading is now true as written rather than aspirational.
+  - **Coverage.** 8 tests: the restored branch is reached (a fallback nothing enters proves nothing — the lesson from SPA-111's vacuity guard), the anchor moves the badge in both directions and reports `insufficient` on too few pairs, both shapes of judge silence stay out of the κ and out of `loop_rate`, and an unreliable anchor quarantines the axis end-to-end from the trusted aggregate.
+
 ## 2026-08-21 (3)
 
 - **SPA-114 — The platform stops discarding most of what a reasoning model produces, and then grading it on the remainder.** A reasoning model returns its deliberation in a field separate from the answer, and `grep -rn "reasoning_content" agent-image/ backend/app/` returned zero. Measured live on MiniMax-M3 while building this: on a one-word reply, **44 of 48 output tokens were reasoning** — thrown away, and billed. The process judge was then asked how the agent worked across `efficiency`, `tool_selection`, `error_recovery` and `goal_alignment`, every one of which, on such a model, is answered largely in the stream nobody kept. Same shape as SPA-86 (arguments were never recorded, so `parameter_quality` had no subject) and SPA-113 (the trace was not in the order things happened): the recording layer was at fault and the model was charged for it.

docs/architecture.md

@@ -474,12 +474,17 @@ summary bit. `fit_trace_to_budget` is shared with the hallucination (E-16), fail
 - On-demand `POST /api/quality/records/{task_id}/evaluate-trajectory` + read
   `GET …/trajectory`; optional batch job `trajectory_judge_evaluate` (off by default,
   gated by `trajectory_eval_enabled`).
-- **Report aggregation caveat (v11).** The per-task profile still scores all six axes,
-  but the experiment report's **trajectory aggregate now excludes `loop_detection`**
-  (`_AGG_EXCLUDED_AXES = {'loop_detection'}` in `experiment_report.py`; `_traj_score`
-  is the mean over the five surviving axes). The deterministic SPA-75 loop counter
-  (below) is now the loop anchor, so loop signal is read from it rather than from the
-  judge's `loop_detection` axis.
+- **Which axes reach a conclusion (v11 → v21).** v11 removed `loop_detection` from the
+  report's trajectory aggregate by name, because SPA-76 promised a quarantined axis is
+  not weighed and the stored `overall_score` averaged it in regardless. That was a
+  stand-in for a gate that did not yet exist. Since SPA-88 the gate acts, so **v21
+  (SPA-89) deletes the exception**: `_traj_score` is the mean over every axis the judge
+  scored, and the only thing that removes one is the trusted view's `allowed` set —
+  i.e. the axis's own badge, recomputed per experiment. The loop axis is quarantined
+  exactly when its anchor says the judge cannot be trusted on it, which on the corpus
+  measured so far it does. The hardcoded set also had a side effect worth naming: it
+  skipped the loop axis before `_axis_reliability` could reach the structural fallback,
+  so the axis's calibration source was unreachable code for the whole of v11–v20.
 
 ### The model's own reasoning (SPA-114)
 
@@ -568,8 +573,34 @@ prose transforms, and `README.md` / `readme.md`, two commands diverging at chara
 Thresholds: `_MIN_REPEAT_RUN = 3` / `_MIN_CYCLE_REPEATS = 3`; `loop_detected` fires when
 either crosses its threshold. `LOOP_SCHEMA_VERSION = 1`. It feeds the report's
 `structural_loop_rate` and serves as the **E-07 loop anchor**: the report computes
-judge↔counter agreement (Cohen's κ ≈ 0.33 over ~715 runs) to badge how far the judge's
-`loop_detection` axis can be trusted.
+judge↔counter agreement (Cohen's κ) to badge how far the judge's `loop_detection` axis
+can be trusted. This is the only reliability badge in the platform that needs no human
+annotation — the counter runs on every trajectory-scored trace — which is why the axis
+is badged from it rather than left `not_calibrated` (SPA-89).
+
+**What enters that κ (SPA-89).** A run counts only when **both** raters answered. The
+judge's silence — the loop axis missing from the profile, or returned as not applicable
+— is not a "no loop" vote, and the derived `loop_detected` flag cannot tell the two
+apart: it is `False` for both. Folding silence into the both-clean cell moves κ toward
+agreement nobody earned; on the ~715-run corpus the anchor was validated against, the
+old convention read κ ≈ 0.33 where the runs both raters actually rated read ≈ 0.28. The
+excluded runs are reported as `n_judge_unscored` rather than dropped quietly, and
+`structural_loop_rate` keeps its own wider denominator — the counter ran on them anyway.
+A counter that ran and produced no pairs at all still counts as a *source*: the axis
+reports `source: structural` with `n: 0` and status `insufficient`, not
+`not_calibrated`. «The anchor found nothing to compare» and «nobody has ever checked
+this axis» call for opposite actions — fix the judge, or go and annotate.
+
+**What that κ may certify (SPA-89).** It compares two booleans, so however high it
+goes it certifies one dichotomy. Where the judge places a run *inside* either side of
+the threshold was never examined — a perfect binary κ is compatible with the judge
+scoring 0 where a human would score 4 — so the axis lands in its own trust zone,
+`binary_only`, which drives the binary loop rate and neither a numeric mean nor a rank
+test on the 0–10 score. This is the same rule that already governs `rank_only`
+(«trusted for order alone cannot make a claim about size»), one step further down: an
+axis trusted for one dichotomy cannot make a claim about order either. A human rating
+that axis is graded evidence and climbs the zones normally — the ceiling follows what
+was measured, not which axis it is.
 
 ### Evidence bank judge (E-08)
 
@@ -1323,13 +1354,17 @@ nothing on one side makes no claim at all.
   calibration via `_classify_reliability`. v13 extended the badge from the six E-07
   trajectory axes (`_axis_reliability`) to the **outcome** rubric axes
   (`_outcome_axis_reliability`); **v17 (SPA-88) makes it act**, and splits the four
-  zones into six because they license different claims:
+  zones into six because they license different claims; **v21 (SPA-89) restores the
+  loop axis to the badged set**, sourced from the deterministic counter when no human
+  rated it, and adds a seventh zone for what that source can support. A
+  structurally-sourced axis carries no ρ, so it can never be rank-rescued either:
 
   | status | rule | may drive |
   | -- | -- | -- |
   | `reliable_absolute` | κ ≥ `RELIABILITY_RELIABLE_KAPPA` = 0.6 | numeric aggregates, absolute claims |
   | `moderate_agreement` | 0.4 ≤ κ < 0.6 | numeric aggregates, flagged |
   | `rank_only` | κ < 0.4 **and** Spearman ρ ≥ `RELIABILITY_RANK_RHO` = 0.5 | ranks / paired comparisons **only** |
+  | `binary_only` | κ ≥ 0.4 against a **boolean** reference (the loop counter) | the binary loop rate **only** — never a mean, never a rank test on the 0–10 score |
   | `insufficient` | n < 3, or κ undefined | nothing |
   | `unreliable` | κ < 0.4 **and** ρ < 0.5 | nothing |
   | `not_calibrated` | no source | nothing |
@@ -1419,7 +1454,7 @@ nothing on one side makes no claim at all.
 | `app/quality/reproducibility.py` | E-20 Reproducibility Snapshot: `assemble_snapshot` + `snapshot_fingerprint` + `diff_snapshots` + `capture_snapshot`/`replay_from_snapshot` — per-record `experiment_snapshot` (captured/missing manifest), SHA-256 determinism fingerprint, snapshot diff + re-run via `clone_task_for_rerun` |
 | `app/quality/comparison.py` | E-21 Pairwise Comparison: `create_comparison`/`advance_comparison` + `judge_pair_llm` (both-orders, position-bias mitigation) + `comparisons_to_matches`/`run_pairwise_leaderboard` — real A/B verdicts on `pairwise_comparisons`, handed to E-19 as explicit ELO matches |
 | `app/quality/experiments.py` | SPA-40 Experiment Runner: `create_experiment` + `expand_matrix`/`normalize_dataset` + `advance_experiment` — frozen dataset × config matrix × n_runs, poll-driven `experiment_runs` cells (E-02 + optional E-07/E-14/E-20), cost-capped. Driven by the `experiment_run_tick` job |
-| `app/quality/experiment_report.py` | SPA-40 Experiment report: `build_report`/`compute_report` — per-config summary, heatmap, Pareto frontier, pairwise leaderboard, `significance_matrix`, failure-mode/orchestrator breakdowns, plus the per-axis reliability traffic light and the gate it drives (`_classify_reliability` six-way / `_axis_reliability` / `_outcome_axis_reliability` / `_trust_split` / `_trusted_weighted`; raw vs `trusted` views), and the paired,
+| `app/quality/experiment_report.py` | SPA-40 Experiment report: `build_report`/`compute_report` — per-config summary, heatmap, Pareto frontier, pairwise leaderboard, `significance_matrix`, failure-mode/orchestrator breakdowns, plus the per-axis reliability traffic light and the gate it drives (`_classify_reliability` seven-way / `_axis_reliability` / `_outcome_axis_reliability` / `_trust_split` / `_trusted_weighted`; raw vs `trusted` views), and the paired,
 multiplicity-corrected statistics on top of it (`_significance_cells` /
 `_compare_cells`; `SCHEMA_VERSION=18`) |
 | `app/api/data_lake.py` | `/api/data-lake` — records (filter), full blob, group-by query, export (json/parquet) |

frontend/src/components/quality/SummaryRadarPanel.tsx

@@ -1,4 +1,5 @@
 import { useState } from 'react'
+import type { AxisTrustStatus } from '@/types'
 import {
   Radar,
   RadarChart,
@@ -16,15 +17,11 @@ interface Row {
   cells: Record<string, { mean?: number | null } | undefined>
 }
 
-// SPA-88: the six-way trust taxonomy (see AxisTrustStatus in types). Only the
-// grey/non-grey distinction matters here — a radar is a shape, not a claim.
-type AxisStatus =
-  | 'reliable_absolute'
-  | 'moderate_agreement'
-  | 'rank_only'
-  | 'insufficient'
-  | 'unreliable'
-  | 'not_calibrated'
+// The trust taxonomy, ALIASED rather than restated: this file used to keep its own
+// copy of the union, so adding a zone in types/ left the radar silently behind
+// (caught by tsc when SPA-89 added 'binary_only'). Only the grey/non-grey
+// distinction matters here — a radar is a shape, not a claim.
+type AxisStatus = AxisTrustStatus
 
 /** Overlayable per-config radar over a set of axes (quality dimensions E-02, or
  *  trajectory axes E-07). Each config is a semi-transparent layer; checkboxes
@@ -84,8 +81,12 @@ export default function SummaryRadarPanel({
     const value = String(props.payload?.value ?? '')
     const st = statusByLabel[value]
     const struck = st === 'unreliable'
+    // Greyed whenever the shape at this axis rests on a number nothing certified —
+    // 'binary_only' included: its κ is about a yes/no, and a radar plots magnitudes.
     const fill =
-      st === 'unreliable' || st === 'not_calibrated' || st === 'insufficient' ? '#9ca3af' : '#6b7280'
+      st === 'unreliable' || st === 'not_calibrated' || st === 'insufficient' || st === 'binary_only'
+        ? '#9ca3af'
+        : '#6b7280'
     return (
       <text
         x={props.x}

frontend/src/pages/CheatSheet.tsx

@@ -121,7 +121,7 @@ const GROUPS: Group[] = [
       { name: 'The trajectory judge', desc: 'An LLM judge that scores HOW the agent worked across six axes: efficiency, tool selection, parameter quality, error recovery, goal alignment, loop detection.' },
       { name: 'The evidence-bank trace judge', desc: 'A trajectory-judge variant that walks the cleaned trace step by step against an accumulating bank of established facts, adding a groundedness signal.' },
       { name: 'The trajectory match', desc: "A deterministic, LLM-free comparison of the agent's tool-call sequence against a canonical/gold trajectory (exact, edit-distance, and graph metrics)." },
-      { name: 'The deterministic loop counter', desc: 'An LLM-free detector that counts repeated tool-calls over the full untrimmed trace — a structural lower bound on looping that replaces the unreliable judge loop signal.' },
+      { name: 'The deterministic loop counter', desc: "An LLM-free detector that counts repeated tool-calls over the full untrimmed trace — a structural lower bound on looping. Because it runs on every trace, it also calibrates the judge: agreement between the two is the loop axis's reliability badge, the only one that needs no human annotation." },
     ],
   },
   {