The effectiveness dashboard defines "succeeded" as the complement of failed/blocked, which sweeps every other status into the success bucket.
agents/effectiveness.py:154
failed = [x for x in results if x.get("status") in {"failed", "blocked"}]
succeeded = [x for x in results if x.get("status") not in {"failed", "blocked"}]This flows into the headline KPI:
agents/effectiveness.py:259
item["change_success_rate"] = round(item["successful_changes"] / item["changes_total"], 4) if item["changes_total"] else 0
Failure scenario
A remediation runs in dry-run mode (or is queued/pending, or skipped, or returns an empty/unknown status). None of those are in {"failed", "blocked"}, so every one is counted as a success. A dry-run that mutated nothing reports change_success_rate = 1.0. Over a fleet of mostly-dry-run activity, the "effectiveness" number the dashboard advertises is inflated toward 100% and stops meaning "changes that actually worked."
There's a second distortion: changes_total is derived from planned changes, while succeeded is derived from results. If execution aborts early, results has fewer entries than changes, but the denominator still counts every planned change — quietly skewing the ratio.
Suggested fix
Count success by an explicit allow-list of terminal-success statuses, and decide the denominator deliberately:
succeeded = [x for x in results if x.get("status") in {"succeeded", "completed", "applied"}]If dry-runs should be excluded from the metric entirely, filter them out before computing the rate rather than scoring them as wins.
The effectiveness dashboard defines "succeeded" as the complement of failed/blocked, which sweeps every other status into the success bucket.
agents/effectiveness.py:154This flows into the headline KPI:
agents/effectiveness.py:259Failure scenario
A remediation runs in dry-run mode (or is queued/
pending, orskipped, or returns an empty/unknownstatus). None of those are in{"failed", "blocked"}, so every one is counted as a success. A dry-run that mutated nothing reportschange_success_rate = 1.0. Over a fleet of mostly-dry-run activity, the "effectiveness" number the dashboard advertises is inflated toward 100% and stops meaning "changes that actually worked."There's a second distortion:
changes_totalis derived from plannedchanges, whilesucceededis derived fromresults. If execution aborts early,resultshas fewer entries thanchanges, but the denominator still counts every planned change — quietly skewing the ratio.Suggested fix
Count success by an explicit allow-list of terminal-success statuses, and decide the denominator deliberately:
If dry-runs should be excluded from the metric entirely, filter them out before computing the rate rather than scoring them as wins.