feat(cli): results matrix — behavior × arm comparison table

[Chang Liu (changliu2)]

Adds assert-ai results matrix, a 2D comparison view for multi-run eval results. Companion to the bank-manager one-behavior-per-yaml configs in #262 — it renders the full N-failure-modes × M-arms grid that the viewer's single-suite compare view can't.

What changed

• New assert-ai results matrix SUITE/RUN ... under the results group.
• Repeatable --suite <SUITE> auto-expands all scored runs in a suite; pass several to span multiple behavior-suites in one table.
• --metric (default policy_violation), --json, --no-color, --results-dir.
• Rows = behaviors, read from each run's config.yaml behavior.name (falls back to manifest, then suite id).
• Columns = arm labels derived from run ids, ordered baseline → prompted → acs then any others alphabetically (deterministic).
• Cells = the metric rate; missing behavior/arm cells render - in tables and null in JSON.

Example

assert-ai results matrix `
  --suite bank-1b-distortion --suite bank-1b-data-leak `
  --suite bank-1b-policy-fab --suite bank-1b-unauth-txn `
  --metric policy_violation

Demo output (rendered from real runs)

       Behavior × arm matrix (Policy violation)
Behavior                      baseline  prompted  acs
no_financial_distortion       96.0%     88.0%     89.8%
no_sensitive_data_leak        53.3%     57.8%     45.5%
no_fabricated_policy          92.0%     84.0%     87.8%
no_unauthorized_transactions  60.0%     50.0%     26.0%

      Behavior × arm matrix (Strict overrefusal)
Behavior                      baseline  prompted  acs
no_financial_distortion       2.0%      0.0%      8.2%
no_sensitive_data_leak        2.2%      6.7%      2.3%
no_fabricated_policy          0.0%      0.0%      0.0%
no_unauthorized_transactions  2.0%      0.0%      2.0%

Validation

8 passed, 4 skipped, 1163 deselected

[Chang Liu (changliu2)]

The split-vs-union default and the denominator note are the right calls, and the permissibility-split lookup fix is well justified by its tests. Two things block merge for me, both about the arm labeling being lossy on this repo's actual run-id conventions:

1. _run_arm_label falls back to the last hyphen-delimited segment of the run id, which mislabels most existing runs in artifacts/results (variant-a-unguarded → unguarded, variant-c-baseline-prompt → prompt, baseline-weak-prompt → prompt, langgraph-multi → multi).
2. Because cells are written with a plain dict assignment, two runs that collapse to the same (behavior, arm) key silently overwrite each other — one run's numbers just vanish from the table with no warning and exit code 0. The two examples above (variant-c-baseline-prompt and baseline-weak-prompt, in a --suite a --suite b invocation over behaviors that share a name) collide today.

Everything else is comment-level: an unvalidated --metric renders a full grid of - (indistinguishable from "no violations") on a typo, and the prompt-then-scenario fallback silently picks one half of a run that has both.

Inline notes (line numbers on the PR head b02773e, assert_ai/cli.py)

cli.py:1558 — must fix.

cells.setdefault(behavior, {})[arm] = _run_dimension_rate(run_summary, metric)

Two runs that resolve to the same (behavior, arm) key silently overwrite each other; the surviving cell looks authoritative and the dropped run leaves no trace. This is reachable today: with --suite bank-manager-agent-shield --suite incident-triage-agent-v1, variant-c-baseline-prompt and baseline-weak-prompt both label as prompt. It is also what you get from any repeated-seed setup (run-1, run-2 of the same arm) — exactly the "multi-run" case the command is for.
Suggested fix: detect an existing key and either error naming both run ids, or aggregate deliberately (mean over runs) and say so in the output. Silent last-write-wins is the one behavior that shouldn't survive.

cli.py:693 — must fix.

if "-" in run_id:
    return run_id.rsplit("-", 1)[-1] or run_id

The rsplit fallback only produces a meaningful label when the run id happens to end in the arm name. Against the run ids actually in artifacts/results it produces: variant-a-unguarded→unguarded, variant-b-guarded→guarded, variant-c-baseline-prompt→prompt, guarded-with-shield→shield, langgraph-multi→multi (next to langgraph→langgraph). Column headers become wrong-but-plausible, which is worse than verbose.
Suggested fix: when the run id does not carry the {suite_id}- prefix, return the full run_id rather than guessing a suffix. Optionally add an explicit --arm SUITE/RUN=LABEL override for the cases where users want short headers.

cli.py:1560 — should fix. _reject_ordinal_compare is the only validation applied to metric, and it only rejects ordinal dimensions. A typo (--metric policy_violaton) resolves to nothing, every cell renders -, and the command exits 0 under a correct-looking title. Suggested fix: after loading runs, if metric is neither a permissibility key nor present in any run's dimensions, error out with the available dimension names.

cli.py:478 — comment.

prompt_rate = _dimension_rate(prompt_metrics, metric)
if prompt_rate is not None:
    return prompt_rate
return _dimension_rate(scenario_metrics, metric)

For a run with both prompt and scenario rows, the cell is the prompt rate only and the scenario half is dropped without any indication. Worth either a --rows prompt|scenario selector or a footnote when the loaded runs are not homogeneous.

cli.py:670 — comment.

config = yaml.safe_load(config_path.read_text(encoding="utf-8")) if config_path.exists() else None

Unguarded, unlike _load_dimensions (line ~174) which wraps the same call in try/except. A malformed or non-UTF-8 config.yaml in any one run aborts the whole matrix with a raw traceback instead of a CLI error. Wrap and fall through to the manifest/suite-id fallbacks already implemented below.

cli.py:1466 — nice to have. --metric here lacks shell_complete=_complete_metric, which results compare has.

cli.py:1506 — nice to have. --suite expansion appends without deduping against the positional args, so matrix suite-a/run-1 --suite suite-a loads and re-parses run-1 twice. Harmless output-wise but doubles the file I/O.

Performance: no super-linear patterns — loading is O(runs), rendering O(behaviors × arms). The one scale concern is memory, not time: _load_run_summary retains prompt_rows/scenario_rows for every run, and these stay alive for the whole command even though the matrix only reads the computed metrics. Dropping the row lists after the metrics are computed would fix it.

Tests: the split coverage is genuinely good. Gaps worth closing alongside the fixes: arm-label collision, non-conforming run ids like variant-a-unguarded, unknown-metric typo, and --suite pointing at a missing suite. Also, test_results_matrix_missing_cell_renders_null_and_dash asserts "-" in text_result.output, which is satisfied by any hyphen in the table (including behavior names) and would not fail if missing cells rendered wrongly.

Verdict: Request Changes — the arm-label collision silently drops an entire run's numbers from a table whose only job is cross-arm comparison, and the suffix-based labeling mislabels the majority of run ids already in artifacts/results.

Must fix before merge

1. Silent cell overwrite on duplicate (behavior, arm) (cli.py:1558).
2. _run_arm_label suffix heuristic mislabels real run ids (cli.py:693).

Should fix
3. Unvalidated --metric renders an all-dash grid at exit 0 (cli.py:1560).

Nice to have
4. Prompt-over-scenario fallback silently picks one half (cli.py:478).
5. Unguarded yaml.safe_load on run config.yaml (cli.py:670).
6. Missing shell_complete on --metric; no dedupe between positional args and --suite; retained score rows inflate memory at scale; the four test gaps above.

[Chang Liu (changliu2)]

Synced current main into this branch in 79c3525, preserving the results-matrix fixes and picking up the Phoenix <19.18.0 pin. The prior correctness blockers are closed: duplicate cells no longer overwrite silently, descriptive arm IDs are preserved, malformed configs surface clean errors, and unknown metrics are rejected. Focused matrix/results tests pass (47 passed, 18 subtests). The full local suite reached 1178 passed before one viewer test hit missing local npm dependencies; after npm ci, the viewer citation suite passes (16 passed). Ready for CI/re-review.