tools(sentry): grade the noise gate against install-base uptake, not raw volume

[dzianisv]

Stacked on #172 (needs its --json/--by-reason volume report). Refs AGE-105.

The bug this fixes is in the measurement, not the code

The noise gate ships inside the app binary. It only runs on devices that installed v0.4.14 (Play versionCode 150/151, production 2026-08-14 14:22 UTC). So the scheduled re-reads on 08-17 / 08-21 / 09-05 are about to compare a raw event count against a target that assumes 100% install share — which Play reaches slowly and never fully.

Both readings of that mistake are wrong in a way that costs a heartbeat:

• still ~2,200/mo at 33% uptake reads as "the gate failed" — it is in fact exactly what the model predicts, i.e. the gate working;
• a dip because it was a quiet weekend reads as efficacy.

What it does

scripts/noise-gate-report.mjs grades against the uptake-adjusted expectation:

expected_post = baseline x (1 - gated_share x 0.969)

• 0.969 is measured, not guessed — the 90d replay of this project's real events in src/lib/sentry-noise.test.ts. It counts only the transport drop-list, ignoring dedup/rate-cap, so it is deliberately conservative.
• gated_share = share of active users on versionCode >= 150, from Play vitals (play-version-share.mjs). Sentry cannot answer this: while the org is over quota nothing is stored, so release tags stop.
• The 100%-uptake projection is reported separately, so "on track today" and "clears the 3,500/mo org gate" stop being the same question.
• It inverts the model and prints the implied on-device efficacy, so 96.9% is checked against reality instead of trusted.
• It refuses to grade windows that look like results but aren't: client_discard/before_send == 0 (nothing ran the gate) or 0% Play share → UNGRADED. Absence of evidence is neither pass nor fail.
• Org outlook takes the background rate from the longer window — the other projects emit ~0.7 events/h, so a short window contains zero of them and would flatter the org total.

Why a workflow

Neither credential exists outside GitHub Secrets. An agent picking up the 08-21 read on a laptop is stuck (I was). .github/workflows/sentry-noise-gate-report.yml runs it on dispatch and weekly, tests the grading model in the same job that publishes the number, and writes the table to the run summary.

gh workflow run "Sentry noise-gate report" -f post=2026-08-21T00:00:00Z..now

Verification

• 10 unit tests on the grading model, incl. the two symmetrical mistakes above, a real regression at low uptake (still OFF_TRACK), and a gate that does nothing at 90% uptake (implied efficacy 0).
• Run against the live org 1.2h after rollout: UNGRADED, before_send=0, 4.95/h vs 4.71/h baseline — correct, no device has the build yet.
• npm test 291 pass.

[dzianisv]

CI note: this PR gets no checks, and that is a repo config fact rather than a failure — every workflow here is pull_request: branches: [main], and this one is based on age-105-sentry-volume-report (#172). When #172 merges, GitHub retargets this PR to main and CI runs for real; do not merge it before that.

Ran the equivalent locally in the meantime:

• npm test — 291 pass, 0 fail (includes the 10 new grading-model tests)
• npm run typecheck — clean
• python3 -c 'import yaml; yaml.safe_load(...)' on the new workflow — parses
• the report itself against the live Sentry org (--no-play, 1.2h post-rollout): UNGRADED, before_send=0, 4.95/h vs a 4.71/h baseline, org outlook 628/mo. Correct answer — no device has v0.4.14 yet.