Release 1.3.0: how the triage agent is graded

CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+Releases before 1.3.0 are recorded in the git history and their merge
+commits.
+
+## 1.3.0 - 2026-10-20
+
+- **Triage agent 0.3.0.** `backend/requirements.txt` pins the agent to
+  the commit its 0.3.0 release was merged as (`cc5f707`). 0.3.0 adds an
+  eval suite that grades what the agent's writers put on each card, and ships the
+  latest pass rates in the package.
+- **How this agent is graded.** A small panel at the foot of the Triage
+  tab: the template writer's pass rate on each of the agent's six graders
+  (grounding, root cause, escalation, uncertainty, next step, format),
+  how many of the agent's synthetic cases passed every grader, and a link
+  to the Evals section of the agent's showcase page. Collapsed until
+  opened.
+- **`GET /api/triage/grading`.** Returns those numbers, read from the
+  installed agent with `bms_triage_service.eval_scorecard()`. Nothing is
+  hard-coded in the dashboard; a 503 if the installed agent has no
+  scorecard.
+- **The template's wording is tighter in the agent.** Unexplained
+  electrical alarms now say "no confirmed" cause, as the evidence does,
+  and times past the limit stay in seconds. The live outage card is
+  unchanged: the utility failure first, the other 5 alarms collapsed
+  under it, and the frequency dip measured from the load step (58.88 Hz,
+  1.12 Hz below 60, back within 0.5 Hz in 2 s).
+- **Fixtures.** The Triage tab's fixtures were rebuilt from the simulator
+  with the new agent by `backend/scripts/make_triage_fixtures.py`, which
+  is now in the repository. Apart from the agent version and run timings
+  they are identical to 1.2.2's.
+
+Verification: backend 85 tests, front end 44 tests, production build.

README.md

@@ -58,12 +58,13 @@ What else is implemented:
 
 ISA 18.2 alarm handling keeps the list honest, but a utility outage still annunciates six alarms in twelve seconds: utility failure, both UPS systems on battery, the generator running, the transfer switch on the emergency source, and the generator frequency dip when the load lands. Only one of those is the thing that happened.
 
-The **Triage** tab hands the alarm journal and trend history for the window around the latest event to the [BMS Alarm Triage Agent](https://github.com/AddisonTech/bms-alarm-triage-agent), version 0.2.2, pinned in `backend/requirements.txt` to the commit that release was merged as, and shows what comes back:
+The **Triage** tab hands the alarm journal and trend history for the window around the latest event to the [BMS Alarm Triage Agent](https://github.com/AddisonTech/bms-alarm-triage-agent), version 0.3.0, pinned in `backend/requirements.txt` to the commit that release was merged as, and shows what comes back:
 
 - **Root cause first.** The agent's electrical rule set ranks the cause and folds its consequences under it. A utility failure explains UPS on battery, the generator start, the ATS transfer, and the frequency dip; a voltage sag explains the UPS ride through; a PDU overload explains its own lower stage and the load alarm on the UPS that feeds it; low power factor stands on its own.
 - **Trend evidence** for every item, the stretch it spent past its limit, and the rule that ranked it. The generator frequency dip is measured from the load step at the transfer, once the generator is at rated frequency, never across its run up from standstill: worst frequency, how far below 60 Hz, and how long until it was back within 0.5 Hz.
 - **A next step**, always a diagnostic one: a verb from a closed list, the item's own equipment as the target, never a control action.
 - **Who wrote the text.** A fixed template writes it from each item's own evidence. Claude Haiku writes it only when the server has `TRIAGE_ANTHROPIC_KEY` set, and the same checks apply to every reply either way. The hosted demo has no key, so the demo shows the template.
+- **How the agent is graded.** *How this agent is graded*, at the foot of the Triage tab, opens the template writer's pass rate on each of the agent's six write-up graders (grounding, root cause, escalation, uncertainty, next step, format), with a link to the [Evals section](https://addisontech.github.io/bms-alarm-triage-agent/#evals) of the agent's showcase page that explains them. The numbers come from the pinned agent package (`bms_triage_service.eval_scorecard()`), never from the dashboard, so they change only when the pin does. The agent's eval cases are synthetic.
 
 After a demo event finishes, a *Triage this event* button appears under the top bar. On the simulated utility outage the journal holds 24 entries (6 alarms, 6 returns, and 12 acknowledgements and suppressions) and the agent returns 1 ranked item, the utility failure, with the other 5 alarms under it. A run takes about 65 ms once the agent is loaded; the first one loads it on demand so startup stays fast on the free tier.
 
@@ -83,6 +84,7 @@ It is read only. The agent gets a copy of the journal and history for the window
 | `GET /api/export/journal?minutes=60` | Journal and trend history for the last N minutes (or `start` and `end`, ISO 8601), up to 24 hours |
 | `GET /api/triage/event` | The latest alarm event: first and last alarm and its window |
 | `POST /api/triage` | Run the agent on the latest event (or a `start` and `end` window) and return the ranked list |
+| `GET /api/triage/grading` | The pinned agent's eval pass rates for the template writer, per grader |
 
 ---
 
@@ -150,7 +152,7 @@ Demo scenarios:
 | API and WebSocket | Python 3.13, FastAPI 0.111, Uvicorn |
 | Storage | SQLite (WAL mode) |
 | Field protocol | Modbus TCP via pymodbus (optional) |
-| Alarm triage | BMS Alarm Triage Agent 0.2.2, pinned to its release commit |
+| Alarm triage | BMS Alarm Triage Agent 0.3.0, pinned to its release commit |
 | Front end | React 18, Vite 5, React Router 6 |
 | Charts and icons | Recharts (trend pages), inline SVG, Lucide |
 | Tests | pytest, FastAPI TestClient, Vitest |
@@ -191,7 +193,13 @@ cd backend && pip install -r requirements-dev.txt && python -m pytest -q
 cd frontend && npm test
 ```
 
-The backend suite covers the simulator (physical consistency, the outage sequence, every scenario), the alarm state machine (every ISA 18.2 transition, deadband, delays, shelving, suppression, KPIs), the WebSocket feed (snapshot, streaming, ping, live alarm updates after operator actions), the REST API, Modbus decoding, and the triage tie-in (the journal export checked against the agent's own parser, every demo scenario triaged from the real simulator, the event window, scan rate history during events, and the export and triage endpoints). The front end suite covers the feed reducer, URL handling, and the Triage tab (root cause first, consequences collapsed, trend evidence, the writer label).
+The backend suite covers the simulator (physical consistency, the outage sequence, every scenario), the alarm state machine (every ISA 18.2 transition, deadband, delays, shelving, suppression, KPIs), the WebSocket feed (snapshot, streaming, ping, live alarm updates after operator actions), the REST API, Modbus decoding, and the triage tie-in (the journal export checked against the agent's own parser, every demo scenario triaged from the real simulator, the event window, scan rate history during events, the export and triage endpoints, and the grading read from the agent package). The front end suite covers the feed reducer, URL handling, and the Triage tab (root cause first, consequences collapsed, trend evidence, the writer label, the grading panel).
+
+The Triage tab's test fixtures are a simulated outage triaged by the pinned agent. After changing the pin, rebuild them:
+
+```bash
+cd backend && python scripts/make_triage_fixtures.py
+```
 
 ---
 
@@ -251,8 +259,9 @@ backend/
   config.py             Settings, topology, master alarm database
   database.py           SQLite history and alarm journal
   triage_export.py      Alarm journal export in the triage agent's format, event window
-  triage.py             Runs the triage agent on the latest event
+  triage.py             Runs the triage agent on the latest event, and reads its grading
   modbus_source.py      Modbus TCP polling (optional)
+  scripts/              Rebuilds the Triage tab's test fixtures from the pinned agent
   tests/
 frontend/
   src/services/         WebSocket feed, REST client

backend/config.py

@@ -6,7 +6,7 @@
 
 import os
 
-VERSION = "1.2.2"
+VERSION = "1.3.0"
 
 
 def _env_bool(name: str, default: bool) -> bool:

backend/main.py

@@ -282,6 +282,15 @@ def triage_event(request: Request, time_zone: Optional[str] = None):
     }}
 
 
+@app.get("/api/triage/grading")
+def triage_grading():
+    """How the triage agent's writer is graded: its eval pass rates, from the pinned agent."""
+    try:
+        return triage.grading()
+    except triage.AgentUnavailable as exc:
+        raise HTTPException(status_code=503, detail=str(exc))
+
+
 class TriageRequest(BaseModel):
     start: Optional[str] = None
     end: Optional[str] = None

backend/requirements.txt

@@ -1,4 +1,4 @@
 fastapi==0.111.0
 uvicorn[standard]==0.29.0
 pydantic==2.14.1
-bms-alarm-triage-agent @ git+https://github.com/AddisonTech/bms-alarm-triage-agent@1c32b20747dd204f96ae25e3006151f8b178bb80
+bms-alarm-triage-agent @ git+https://github.com/AddisonTech/bms-alarm-triage-agent@cc5f707f329a20862dbb19318e6d428c8162bcfd

backend/scripts/make_triage_fixtures.py

@@ -0,0 +1,53 @@
+"""Rebuild the Triage tab's test fixtures from the real simulator and the pinned agent.
+
+    cd backend && python scripts/make_triage_fixtures.py
+
+Writes frontend/src/test/triage-outage.json (America/New_York) and
+triage-outage-utc.json: a simulated utility outage on the same fixed clock
+the backend tests use, triaged by the installed agent with the template
+writer. Also writes triage-grading.json, the agent's grading as
+GET /api/triage/grading returns it. Run it after the agent pin changes.
+"""
+import json
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BACKEND = HERE.parent
+OUT = BACKEND.parent / "frontend" / "src" / "test"
+
+# The same settings the backend tests run with, set before config.py is imported
+os.environ["DB_PATH"] = os.path.join(tempfile.mkdtemp(prefix="triage-fixtures-"), "fixtures.db")
+os.environ.setdefault("BACKFILL_HOURS", "1")
+os.environ.setdefault("SIM_SEED", "7")
+os.environ.setdefault("SIM_FIRST_EVENT", "100000")
+os.environ["DATA_SOURCE"] = "simulator"
+os.environ.pop("TRIAGE_ANTHROPIC_KEY", None)
+sys.path.insert(0, str(BACKEND))
+sys.path.insert(0, str(BACKEND / "tests"))
+
+import triage  # noqa: E402
+from test_triage_export import drive  # noqa: E402
+
+
+def write(name: str, data) -> None:
+    path = OUT / name
+    with path.open("w", encoding="utf-8", newline="\n") as handle:
+        json.dump(data, handle, indent=1, ensure_ascii=True)
+        handle.write("\n")
+    print("wrote %s" % path.relative_to(BACKEND.parent))
+
+
+def main() -> int:
+    _collector, now = drive("utility_outage", duration=110)
+    for zone, name in (("America/New_York", "triage-outage.json"), ("UTC", "triage-outage-utc.json")):
+        triage.clear_cache()
+        write(name, triage.run(now, time_zone=zone))
+    write("triage-grading.json", triage.grading())
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backend/tests/test_api_websocket.py

@@ -26,7 +26,7 @@ def test_health(client):
     body = client.get("/api/health").json()
     assert body["status"] == "ok"
     assert body["data_source"] == "simulator"
-    assert body["version"] == "1.2.2"
+    assert body["version"] == "1.3.0"
 
 
 def test_websocket_sends_snapshot_on_connect(client):

backend/tests/test_triage_api.py

@@ -21,7 +21,7 @@ def test_export_returns_the_journal_format(client):
     assert res.headers["content-disposition"].startswith('attachment; filename="alarm-journal-')
     doc = res.json()
     assert doc["format"] == "bms-triage-journal" and doc["version"] == 1
-    assert doc["source"] == "power-dashboard 1.2.2"
+    assert doc["source"] == "power-dashboard 1.3.0"
     parsed = agent_journal.parse(doc)
     assert parsed.points and parsed.series
 
@@ -126,10 +126,10 @@ def test_the_agent_is_pinned_to_one_release():
     assert re.fullmatch(r"v\d+\.\d+\.\d+|[0-9a-f]{40}", ref)
 
 
-def test_the_installed_agent_is_0_2():
+def test_the_installed_agent_is_0_3():
     import bms_alarm_triage
 
-    assert bms_alarm_triage.__version__ == "0.2.2"
+    assert bms_alarm_triage.__version__ == "0.3.0"
 
 
 @pytest.mark.parametrize("zone_name", ["America/New_York", "UTC"])
@@ -158,3 +158,27 @@ def test_the_event_and_the_text_use_the_viewers_zone(client, zone_name):
 def test_a_bad_zone_is_a_400(client):
     assert client.post("/api/triage", json={"time_zone": "Mars/Olympus"}).status_code == 400
     assert client.get("/api/triage/event", params={"time_zone": "Mars/Olympus"}).status_code == 400
+
+
+def test_the_grading_comes_from_the_pinned_agent(client):
+    import bms_triage_service
+
+    res = client.get("/api/triage/grading")
+    assert res.status_code == 200
+    body = res.json()
+    entry = bms_triage_service.eval_scorecard()["writers"]["template"]
+    assert body["writer"] == "template"
+    assert body["agent_version"] == bms_triage_service.__version__
+    assert body["date"] == entry["date"]
+    assert [g["name"] for g in body["graders"]] == list(entry["graders"])
+    for grader in body["graders"]:
+        assert grader["rate"] == entry["graders"][grader["name"]]["rate"]
+        assert grader["passed"] == entry["graders"][grader["name"]]["passed"]
+        assert grader["what"]
+    assert body["showcase"].endswith("#evals")
+    assert body["synthetic"] is True
+
+
+def test_grading_without_a_scorecard_is_a_503(client, monkeypatch):
+    monkeypatch.setattr(triage, "_service", lambda: object())
+    assert client.get("/api/triage/grading").status_code == 503

backend/triage.py

@@ -4,7 +4,7 @@
 list. This module finds the window around the most recent alarm event,
 exports the alarm journal and trend history for it (triage_export.py), and
 hands that document to the agent, which is installed as a package pinned to
-its 0.2.2 release commit.
+its 0.3.0 release commit.
 
 Light enough for a free hosting tier: the agent is imported on first use
 rather than at startup, one run happens at a time, a window that has not
@@ -143,6 +143,39 @@ def run(now: float, start: Optional[float] = None, end: Optional[float] = None,
     return value
 
 
+GRADED_WRITER = "template"
+
+
+def grading() -> Dict[str, Any]:
+    """How the agent's writer is graded, read from the installed agent.
+
+    The agent ships the latest pass rates of its write-up eval suite in its
+    package. This passes on the template writer's, which is the writer the
+    dashboard runs, with what each grader checks and a link to the agent's
+    showcase page that explains them. Nothing here is computed or stored by
+    the dashboard, so the numbers change only when the pinned agent does.
+    """
+    service = _service()
+    if not hasattr(service, "eval_scorecard"):
+        raise AgentUnavailable("The installed triage agent has no eval scorecard")
+    card = service.eval_scorecard()
+    entry = card["writers"][GRADED_WRITER]
+    return {
+        "agent_version": service.__version__,
+        "writer": GRADED_WRITER,
+        "writer_name": entry["writer_name"],
+        "date": entry["date"],
+        "cases_run": entry["cases_run"],
+        "cases_passed": entry["cases_passed"],
+        "graders": [
+            {"name": name, "what": card["graders"].get(name, ""), **{k: g[k] for k in ("passed", "applicable", "rate")}}
+            for name, g in entry["graders"].items()
+        ],
+        "showcase": card["showcase"],
+        "synthetic": True,
+    }
+
+
 def clear_cache():
     with _lock:
         _cache["key"] = _cache["value"] = None

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "power-dashboard",
   "private": true,
-  "version": "1.2.2",
+  "version": "1.3.0",
   "type": "module",
   "scripts": {
     "dev": "vite",

frontend/src/App.css

@@ -636,6 +636,17 @@ select:focus, input[type="text"]:focus { border-color: var(--accent); }
 .tri-intro { display: flex; flex-direction: column; gap: 10px; max-width: 80ch; color: var(--text-secondary); }
 
 .tri-list { display: flex; flex-direction: column; gap: 14px; }
+
+.tri-grading { padding-block: 12px; }
+.tri-grading-body { display: flex; flex-direction: column; gap: 10px; margin-top: 10px; max-width: 90ch; }
+.tri-grades { list-style: none; display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 8px; }
+.tri-grades li {
+  display: flex; flex-direction: column; gap: 1px; min-width: 0;
+  padding: 8px 10px; border: 1px solid var(--border); border-radius: var(--r-md); background: var(--bg-elevated);
+}
+.tri-grade-name { font-size: 12px; color: var(--text-secondary); font-weight: 600; }
+.tri-grade-rate { font-size: 18px; font-weight: 650; color: var(--text-primary); }
+.tri-grading-link { display: inline-flex; align-items: center; gap: 4px; color: var(--accent); align-self: flex-start; }
 .tri-item {
   background: var(--bg-card);
   border: 1px solid var(--border);
@@ -750,6 +761,7 @@ select:focus, input[type="text"]:focus { border-color: var(--accent); }
 }
 
 @media (max-width: 720px) {
+  .tri-grades { grid-template-columns: repeat(2, minmax(0, 1fr)); }
   .sidebar { display: none; }
   .mobile-only { display: inline-block; }
   .topbar { padding: 0 12px; height: 52px; }

frontend/src/lib/triage.js

@@ -136,3 +136,23 @@ export function chartGeometry(evidence, width, height, pad = { left: 6, right: 6
 export function eventFinished(previous, current) {
   return !!(previous && !current)
 }
+
+// How the agent's writer is graded, from its eval suite (GET /api/triage/grading)
+export const GRADERS = {
+  grounding: 'Grounding',
+  root_cause: 'Root cause',
+  escalation: 'Escalation',
+  uncertainty: 'Uncertainty',
+  next_step: 'Next step',
+  format: 'Format',
+}
+
+export function graderTitle(name) {
+  return GRADERS[name] || capitalize((name || '').replace(/_/g, ' '))
+}
+
+export function passRate(rate) {
+  if (rate == null) return '--'
+  const pct = rate * 100
+  return `${Number.isInteger(pct) ? pct : pct.toFixed(1)}%`
+}

frontend/src/lib/triage.test.js

@@ -1,6 +1,7 @@
 import { describe, expect, it } from 'vitest'
 import {
-  chartGeometry, dipText, eventFinished, evidenceText, floodParts, itemWriter, reduction, ruleLabel, seconds, valueText, writerLabel,
+  chartGeometry, dipText, eventFinished, evidenceText, floodParts, graderTitle, itemWriter, passRate, reduction, ruleLabel,
+  seconds, valueText, writerLabel,
 } from './triage'
 import outage from '../test/triage-outage.json'
 import outageUtc from '../test/triage-outage-utc.json'
@@ -95,3 +96,18 @@ describe('frequency dip', () => {
     expect(evidenceText(c.evidence, outage.zone)).toBe(dipText(c.evidence.dip, outage.zone))
   })
 })
+
+describe('grading', () => {
+  it('names each grader in plain words', () => {
+    expect(graderTitle('root_cause')).toBe('Root cause')
+    expect(graderTitle('next_step')).toBe('Next step')
+    expect(graderTitle('something_new')).toBe('Something new')
+  })
+
+  it('writes a pass rate as a percentage', () => {
+    expect(passRate(1)).toBe('100%')
+    expect(passRate(0.9483)).toBe('94.8%')
+    expect(passRate(null)).toBe('--')
+  })
+})
+