feat: show what a project is spending, and cap it per project - #288
Merged
Conversation
#73 asks that an operator can see and cap AI and storage spend per tenant. Capping half-existed and seeing did not exist at all. The daily token budget was enforced — the worker sums the day's tokenCost and stops calling the model — but read only from the environment, so `ai.dailyTokenBudget` in a project's flakemetry.yml was documented, validated and ignored. One instance could have exactly one budget, which is not what "per tenant" means. It is now a policy field with the same default/UI/env tiers as everything else. Nothing showed the spend. From the dashboard, a project with root-cause analysis switched on and a project that quietly stopped analysing at noon looked identical; the only trace was a metric counter. The Data page now shows tokens against budget, reports today, rows stored, artifact objects and bytes, and the oldest execution — with the paused state stated outright. Bucket listing is the slow part and the only way to know what storage holds, so an unreachable store degrades that one row rather than the page. Three guards written earlier fired on this change and all three were right: a new policy field with no environment tier, a new project-scoped query missing from the tenant isolation suite, and a new column matching the credential pattern needing classification. That is the third time the isolation guard has caught something added in the same session it was written for.
AKogut
added a commit
that referenced
this pull request
Aug 18, 2026
…289) The alerting half of #73, on email as asked — and on every other channel, because it goes through the notification path that already exists rather than a new one. ## What it fixes The budget stops root-cause analysis and recorded that in a **metric counter**. On a self-hosted instance nobody is scraping it, so explanations simply stopped appearing partway through the afternoon and nothing anywhere said why. #288 made the state visible on a page you have to go and look at; this one comes to you. ``` Subject: [Flakemetry] AI budget spent for today Root-cause analysis is paused until tomorrow — 200,000 of 200,000 tokens used today. Spent: 200,000 Budget: 200,000 ``` The message says *analysis is paused* rather than only reporting numbers. Someone reading this in a mail client has no dashboard open, and "budget exceeded" alone does not tell them anything stopped. ## Deduplicated per project per day The budget is re-checked on **every run**, so without a key that collapses them a busy afternoon sends one of these per failing suite — which is how a useful alert becomes a filter rule. ## My first test proved nothing It emitted the event on the bus and watched a channel receive it. That tests the routing — bus name, notification type, subscription list — and all of it passed with the emit inside RCA **deleted**: ``` $ # emit removed from rca.ts Tests 1 passed ← the routing test, still green ``` Found by deleting the emit, which is the only way that gap shows up. The real test drives `processFailures` until the budget is exhausted and fails when the emit is gone: ``` × announces that the budget is spent rather than only counting it AssertionError: expected [] to have a length of 1 ``` Both tests are kept: one covers the wiring between three pieces that must agree, the other covers the thing that triggers it. Getting the second one to fire took two attempts — the first used a second failure whose message normalised close enough to the first that it was not a new signature, so the loop skipped it before reaching the budget check. Worth knowing that the budget only applies to signatures that would actually call the model. 62/62 turbo tasks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Toward #73 — "an operator can see and cap AI + storage spend per tenant". Capping half-existed; seeing did not exist at all.
The cap was not per tenant
The budget is enforced — the worker sums the day's
tokenCostand stops calling the model, with a metric when it trips. But it was read only fromFLAKEMETRY_AI_DAILY_TOKEN_BUDGET, soai.dailyTokenBudgetin a project'sflakemetry.ymlwas documented in the config reference, validated by the schema, and ignored. One instance could have exactly one budget.It is now a policy field with the same default → UI → env tiers as everything else.
Nothing showed the spend
From the dashboard, a project with root-cause analysis switched on and a project that quietly stopped analysing at noon looked identical. The only trace was a metric counter nobody self-hosting is scraping.
The Data page now shows it, verified against a real database:
The paused state is stated outright rather than left to be inferred from a number. Listing a bucket is both the only way to know what storage holds and the slowest thing on the page, so an unreachable store degrades that one row rather than breaking the page.
Three guards fired, all correctly
Every one of them on code written in this same change:
policy-env.test.ts— a new policy field with no environment tiergetProjectUsagenot exercisedai_daily_token_budgetmatchestokenand needed classifying as not a credentialThird time the isolation guard has caught something added after it was written, which is the whole argument for coverage guards over a fixed list of cases.
Still open on #73
Alerts when a budget is approached, and storage tiering. Both need somewhere to send an alert and a decision about what tiering means here — not things to guess at.
62/62 turbo tasks.