Repository navigation
fix(ticketing): treat prolonged comms loss as catastrophic when backup power is known-compromised - #202
Merged
Conversation
…p power is known-compromised The correlation judge already suppresses send_telegram when telemetry is merely "stale" -- correct in general, since a separate once-a-day comms-down alert already covers routine gateway flakiness. But that assumption breaks when a site's comms/DCU power path (normally Phoenix/Orion-inverter-backed, independent of grid downtime) is itself known to be degraded: on such a site, no report for 30+ minutes may mean the site lost power, not just visibility into it, and the daily comms-down alert alone doesn't say so. Adds an explicit exception: check the O&M message history already in the judge's context for a recent report that this grid's comms power path is degraded (dead Phoenix/Orion inverter, comms confirmed running off grid power). If found, stale telemetry stops being safely covered elsewhere -- send_telegram should be true, and the reason should say plainly that comms loss here can't be distinguished from a power outage. Prompt-only change; no new output field or code path needed since the existing send_telegram/notification.reason/grid_impact.summary fields already cover it -- om_messages_provider already surfaces the relevant O&M chat history to the judge today, so this only needed the missing reasoning rule, not a new data source. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
6 of 7 tasks
Merged
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of the same incident investigation as #201, but a separate, independent fix.
The correlation judge (
ticketing.correlation.prompt) already suppressessend_telegramwhen a grid's telemetry is merelystale— correct in general, since a separate once-a-day comms-down alert already covers routine gateway flakiness ("stalemeans the plant's gateway has stopped reporting, which a separate once-a-day comms-down alert already covers").That assumption breaks on a site whose comms/DCU power path is itself known to be degraded. Normally comms are powered separately from the battery via a Phoenix/Orion inverter, independent of grid downtime (already documented in
staff.system.prompt) — but when that backup path is dead and comms end up running off grid power instead, a stale telemetry reading stops meaning "routine monitoring gap" and starts meaning "the site may have lost power, and we can't tell the difference."Change
One prompt-only addition to
ticketing.correlation.prompt: check the O&M message history already in the judge's context for a recent report that this grid's comms power path is degraded. If found, treatstale+ known-degraded backup power as a real gap —send_telegram: true, and say plainly that comms loss here can't be distinguished from a power outage.No code change, no new output field, no new data source —
om_messages_provideralready surfaces the relevant O&M chat history to the judge today; this only needed the missing reasoning rule.Test plan
chat_orchestrator/tests/prompt_checksums.json— diff is exactly the oneticketing.correlationhash.chat_orchestratorsuite (uv run pytest tests/ -q, matching CI): 2524 passed, 1 pre-existing unrelated failure (confirmed identical onmainbefore this change — a local network/DNS issue intest_verification_criteria_library.py, nothing this PR touches).uv run pytest ../shared -q -n auto: pre-existing failures only, both confirmed unrelated (one scans.worktrees/for other in-progress sessions' unrelated code, one is-n autoparallel test-isolation flakiness — passes in isolation).pre-commit run --all-filesclean.🤖 Generated with Claude Code