feat: say when the AI budget runs out, on email or any other channel - #289
Merged
Conversation
The budget stops root-cause analysis and recorded that in a metric counter. On a self-hosted instance nobody is scraping it, so explanations simply stopped appearing at some point in the afternoon and nothing anywhere said why. It now goes through the notification path that already exists, so email, Slack, Discord and signed webhooks all carry it and it appears as a checkbox beside every other event. Deduplicated per project per day. The budget is re-checked on every run, so a busy afternoon would otherwise send one of these per failing suite. The message says analysis is paused until tomorrow rather than only reporting numbers — someone reading it in a mail client has no dashboard open, and "budget exceeded" alone does not tell them anything stopped. My first test for this proved nothing: it fired the event on the bus and watched a channel receive it, so deleting the emit inside RCA left it passing. Found by deleting the emit. The real test drives processFailures until the budget is exhausted, and fails when the emit is removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The alerting half of #73, on email as asked — and on every other channel, because it goes through the notification path that already exists rather than a new one.
What it fixes
The budget stops root-cause analysis and recorded that in a metric counter. On a self-hosted instance nobody is scraping it, so explanations simply stopped appearing partway through the afternoon and nothing anywhere said why. #288 made the state visible on a page you have to go and look at; this one comes to you.
The message says analysis is paused rather than only reporting numbers. Someone reading this in a mail client has no dashboard open, and "budget exceeded" alone does not tell them anything stopped.
Deduplicated per project per day
The budget is re-checked on every run, so without a key that collapses them a busy afternoon sends one of these per failing suite — which is how a useful alert becomes a filter rule.
My first test proved nothing
It emitted the event on the bus and watched a channel receive it. That tests the routing — bus name, notification type, subscription list — and all of it passed with the emit inside RCA deleted:
Found by deleting the emit, which is the only way that gap shows up. The real test drives
processFailuresuntil the budget is exhausted and fails when the emit is gone:Both tests are kept: one covers the wiring between three pieces that must agree, the other covers the thing that triggers it.
Getting the second one to fire took two attempts — the first used a second failure whose message normalised close enough to the first that it was not a new signature, so the loop skipped it before reaching the budget check. Worth knowing that the budget only applies to signatures that would actually call the model.
62/62 turbo tasks.