[Eval > Run Summary / Compare] Count Test Cases Passed Threshold by score pass/fail, not execution status

[NastassiaKryshtal]

Description

The Run Summary Test Cases Passed Threshold KPI currently buckets eval-summary rows by execution_status: SUCCESS → pass, ERROR → error, FAILED/TIMEOUT → fail. That treats an execution failure as a threshold miss, and treats a successful run that scored below the suite threshold as a pass.

The same card is shown on Run Comparison → Summary, with a side-by-side fraction/legend per run and a delta on the passed count. It uses the same query/parser, so it has the same bug — including in Only matching test cases mode.

The evaluation backend now exposes per-row score and passed on eval_summaries (passed = score >= snapshotted overallScoreThreshold). Both surfaces must use those fields so pass/fail reflect the score threshold, and error is reserved for cases that did not complete successfully. No comparison-API change is required.

Actual result

• A test case with execution_status = FAILED (or TIMEOUT) is counted as fail.
• A test case that executed (SUCCESS) but did not meet overallScoreThreshold is counted as pass.

Expected result

• For each eval-summary row in the run (latest computation, same population as today — including Compare’s matched-only exclusion):
• pass — passed === true (score ≥ snapshotted threshold)
• fail — the case ran and was scored, but passed === false (score < threshold)
• error — execution/evaluation did not succeed (execution_status is ERROR, FAILED, or TIMEOUT), regardless of score
• Rows with execution_status = SUCCESS and passed == null (no threshold, no score, or pre-backfill history) count toward total only and do not increment pass/fail/error
• The card still shows passed / total plus the pass / fail / error legend (zeros included), matching the existing PassFailFraction / PassFailStatusBreakdown UI.
• Hide the card when the run has no snapshotted overallScoreThreshold (including null / omitted), consistent with Trends Runs Passed Threshold. Show it when the threshold is set, including 0.

Run Comparison
Apply the same pass / fail / error rules on Run Comparison Summary’s Test Cases Passed Threshold card (side-by-side primary vs compared, including the passed-count delta).

Backend is sufficient: each run already has passed / execution_status on eval_summaries. Compare already loads counts via buildTestCasesStatusQuery (full population, or matched-only by excluding unmatchedEvalSummaryIds). Do not use matchedSuccessRowCount from the comparison API — that is execution SUCCESS, not threshold pass. Do not extend the comparison endpoint.

All test cases: classify each run independently against that run’s snapshotted overallScoreThreshold (passed is already frozen per run at start).
Only matching test cases: same classification on the matched-only subset (existing NOT id IN unmatchedEvalSummaryIds filter). Matching does not change pass/fail/error meaning.
Card visibility: omit the card when both runs have no snapshotted threshold. If only one run has a threshold, still show the card; the side without a threshold shows — / zeros as appropriate (do not invent a shared live-suite threshold).
Keep the existing compare layout (PassFailFraction + compact PassFailStatusBreakdown per run, delta on passed count).

Acceptance criteria

• Run Summary Test Cases Passed Threshold counts pass/fail from Query DSL field passed, not from SUCCESS vs FAILED.
• Execution ERROR / FAILED / TIMEOUT increment error, not fail.
• passed === true increments pass; passed === false increments fail; exact threshold equality is pass (BE already encodes this).
• Successful but unscored rows (passed == null) only increase total.
• Run Comparison Summary uses the same rules (shared query/parser) for all test cases and only matching test cases.
• Compare does not derive the KPI from matchedSuccessRowCount or any new comparison-API field.
• Each compared run is classified against its own snapshotted threshold, not the live suite value.
• Run Summary: card is omitted when that run’s snapshot has no overallScoreThreshold; shown when the threshold is set (including 0).
• Run Comparison: card is omitted only when both runs have no snapshotted threshold; if exactly one has a threshold, the other side shows empty/—.
• Unit tests cover the new grouping/parser (pass/fail/error mix, exact threshold, failed-with-score → error, null passed on SUCCESS, Compare unmatched-id exclusion).