Skip to content

feat: say when the AI budget runs out, on email or any other channel - #289

Merged
AKogut merged 2 commits into
mainfrom
feat/budget-alerts
Aug 18, 2026
Merged

feat: say when the AI budget runs out, on email or any other channel#289
AKogut merged 2 commits into
mainfrom
feat/budget-alerts

Conversation

@AKogut

@AKogut AKogut commented Aug 18, 2026

Copy link
Copy Markdown
Owner

The alerting half of #73, on email as asked — and on every other channel, because it goes through the notification path that already exists rather than a new one.

What it fixes

The budget stops root-cause analysis and recorded that in a metric counter. On a self-hosted instance nobody is scraping it, so explanations simply stopped appearing partway through the afternoon and nothing anywhere said why. #288 made the state visible on a page you have to go and look at; this one comes to you.

Subject: [Flakemetry] AI budget spent for today

Root-cause analysis is paused until tomorrow — 200,000 of 200,000 tokens used today.

Spent: 200,000
Budget: 200,000

The message says analysis is paused rather than only reporting numbers. Someone reading this in a mail client has no dashboard open, and "budget exceeded" alone does not tell them anything stopped.

Deduplicated per project per day

The budget is re-checked on every run, so without a key that collapses them a busy afternoon sends one of these per failing suite — which is how a useful alert becomes a filter rule.

My first test proved nothing

It emitted the event on the bus and watched a channel receive it. That tests the routing — bus name, notification type, subscription list — and all of it passed with the emit inside RCA deleted:

$ # emit removed from rca.ts
Tests  1 passed        ← the routing test, still green

Found by deleting the emit, which is the only way that gap shows up. The real test drives processFailures until the budget is exhausted and fails when the emit is gone:

× announces that the budget is spent rather than only counting it
AssertionError: expected [] to have a length of 1

Both tests are kept: one covers the wiring between three pieces that must agree, the other covers the thing that triggers it.

Getting the second one to fire took two attempts — the first used a second failure whose message normalised close enough to the first that it was not a new signature, so the loop skipped it before reaching the budget check. Worth knowing that the budget only applies to signatures that would actually call the model.

62/62 turbo tasks.

AKogut added 2 commits August 18, 2026 15:25
The budget stops root-cause analysis and recorded that in a metric counter. On a
self-hosted instance nobody is scraping it, so explanations simply stopped
appearing at some point in the afternoon and nothing anywhere said why.

It now goes through the notification path that already exists, so email, Slack,
Discord and signed webhooks all carry it and it appears as a checkbox beside
every other event.

Deduplicated per project per day. The budget is re-checked on every run, so a
busy afternoon would otherwise send one of these per failing suite.

The message says analysis is paused until tomorrow rather than only reporting
numbers — someone reading it in a mail client has no dashboard open, and
"budget exceeded" alone does not tell them anything stopped.

My first test for this proved nothing: it fired the event on the bus and
watched a channel receive it, so deleting the emit inside RCA left it passing.
Found by deleting the emit. The real test drives processFailures until the
budget is exhausted, and fails when the emit is removed.
@AKogut
AKogut merged commit 6f71e52 into main Aug 18, 2026
5 checks passed
@AKogut
AKogut deleted the feat/budget-alerts branch August 18, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant