Ticket: P1.5-S5-07
Type: Execution | Est: 1d
Goal: AC-01 through AC-06 scored per incident; pass/fail report written with documented root cause for every failure.
Scope (depends on P1.5-S5-06 results):
- Score each AC by comparing
results_round2.json against ground_truth.json
- AC-03 (log recall) scored as N/A for UC3 incidents
- Write
tests/acceptance/report_round2.md:
- Per-incident per-criterion pass/fail table (30 rows × 6 ACs)
- Summary accuracy/latency stats per UC
- Miss analysis: categorise failures by root cause (wrong platform tag, no logs returned, LLM classification error, latency breach)
- UC3 graceful degradation quality section (partial notification rating)
Acceptance criteria:
Ticket:
P1.5-S5-07Type: Execution | Est: 1d
Goal: AC-01 through AC-06 scored per incident; pass/fail report written with documented root cause for every failure.
Scope (depends on P1.5-S5-06 results):
results_round2.jsonagainstground_truth.jsontests/acceptance/report_round2.md:Acceptance criteria:
mainbranch