tools(sentry): anchor the measurement windows on the gate's rollout instant

.github/workflows/publish-play-store.yml

@@ -126,13 +126,16 @@ jobs:
         run: ./gradlew bundleRelease
 
       - name: Verify the Sentry noise gate is in the artifact
-        # AGE-105: the org error quota is defended ONLY by the client-side gate
-        # (every server-side lever on this Sentry plan was checked and is dead:
-        # per-key rate limit silently no-ops with a 200, custom inbound filters
-        # absent, spike protection 403). If a build ships without the gate — or
-        # without a DSN, which makes Sentry a silent no-op — the org goes back
-        # over quota, and while it is over quota Sentry stores nothing, so the
-        # regression is invisible in Sentry itself until the monthly reset.
+        # AGE-105: the org error quota is defended ONLY by the client-side gate.
+        # Every server-side lever on this Sentry plan was probed and is dead
+        # (docs/analytics.md): per-key rate limit answers 200 and silently drops
+        # the field, custom error-message filters answer 400 "You do not have
+        # that feature enabled", and spike protection is already on everywhere
+        # but only catches spikes, not this steady baseline. If a build ships
+        # without the gate — or without a DSN, which makes Sentry a silent
+        # no-op — the org goes back over quota, and while it is over quota
+        # Sentry stores nothing, so the regression is invisible in Sentry itself
+        # until the monthly reset.
         # Grep the shipped Hermes bundle instead. Verified to discriminate:
         # v0.4.14 passes, pre-gate v0.4.13 fails.
         run: node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab

.github/workflows/sentry-noise-gate-report.yml

@@ -83,6 +83,6 @@ jobs:
             echo ''
             echo '### Raw org volume'
             echo '```'
-            node scripts/sentry-volume-report.mjs --by-reason || true
+            node scripts/sentry-volume-report.mjs --by-reason --since-rollout || true
             echo '```'
           } >> "$GITHUB_STEP_SUMMARY"

docs/analytics.md

@@ -165,6 +165,23 @@ production rollout at 14:22 UTC), two windows agreeing to within 0.2%:
 Mobile was 87% of the org's post-box-bot demand. Target: under ~1,500/month, which puts the
 org under the 3,500/month gate.
 
+**Never measure across the rollout instant.** Use `--since-rollout`, which reads the v0.4.14
+production instant (2026-08-14 14:22Z) out of the release-history table in
+[`playstore.md`](./playstore.md) and splits the windows exactly there:
+
+```sh
+SENTRY_AUTH_TOKEN=... node scripts/sentry-volume-report.mjs --by-reason --since-rollout
+```
+
+A hand-rolled window that spans the instant mixes two populations — devices that have the
+gate and devices that do not — so its rate is neither a baseline nor a result, while looking
+exactly like both. This is not hypothetical: a `post=08-14T07:00Z..now` window (84% of it
+pre-rollout) was run against this very script and reported mobile *rising* to 4.44/h. Such
+windows now print `[mixed]` with the pre-rollout percentage, a clean but young post window
+prints how many hours of uptake it has, and an unparseable release table reports `unknown`
+rather than silently grading everything as post-gate
+(`scripts/sentry-volume-report.test.mjs`).
+
 ### Uptake is part of the measurement, not an excuse afterwards
 
 The gate ships **inside the app binary**, so it only runs on devices that installed v0.4.14.
@@ -235,18 +252,29 @@ box-bot at 0, mobile is now the dominant remaining demand.
 
 ### Server-side levers do not exist on this plan
 
-Checked directly against the API on 2026-08-14, so nobody re-litigates it:
+Probed directly against the API (re-verified 2026-08-14 15:45Z with a **write**-scoped
+token, so none of these is a permissions artifact), so nobody re-litigates it:
 
-| Lever | Result |
-|---|---|
-| Per-key rate limit (`PUT /projects/{org}/{proj}/keys/{id}/`) | **Silent no-op.** Returns HTTP 200 and drops the field; a follow-up GET always reads `rateLimit: null`. Reproduced with `window` = 60, 3600 and 86400. |
-| Custom inbound filters (error message / release) | Not present. Only the five generic browser filters exist. |
-| Spike protection (`/organizations/{org}/spike-protections/`) | HTTP 403. |
+| Lever | Call | Result |
+|---|---|---|
+| Per-key rate limit | `PUT /projects/{org}/{proj}/keys/{id}/` `{"rateLimit":{"window":86400,"count":50}}` | **HTTP 200 that lies.** The field is dropped; the follow-up GET reads `rateLimit: null`. Reproduced with `window` = 60, 3600, 86400. The success code is the trap — this is the one lever that looks like it worked. |
+| Custom inbound filter on error message | `PUT /projects/{org}/{proj}/` `{"options":{"filters:error_messages":"…"}}` | **HTTP 400 `{"detail":"You do not have that feature enabled"}`.** The option key exists and reads `''`; writing it is plan-gated (Business). This is the lever that *would* fix the residual risk, because it drops at ingest for **every** app version, including installs that never update. |
+| Generic inbound filters | `PUT /projects/{org}/{proj}/filters/{id}/` `{"active":true}` | **HTTP 204 — works.** Useless here: the only ids are `browser-extensions`, `legacy-browsers`, `localhost`, `web-crawlers`, `filtered-transaction`. None can match a React Native app error. |
+| Spike protection | `POST /organizations/{org}/spike-protections/` | **HTTP 201 — and it was already on.** Every project reads `quotas:spike-protection-disabled = false`. It did not prevent this overage because this is *sustained baseline* volume, not a spike. (The org-level GET is 403, which earlier read as "unavailable"; it is not unavailable, it is ineffective — do not spend a plan upgrade on it.) |
 
 Org `features` is `[]`. **The client-side gate is the only control that exists**, so its
 coverage is the entire safety margin — which is why `sentry-noise-production.test.ts` pins
 that coverage against real production data.
 
+Two consequences worth stating outright:
+
+1. Because the only control ships **inside the app binary**, Play install uptake is on the
+   critical path of the fix. That is not a reporting detail — it is why the verdict must be
+   normalised by uptake (`scripts/noise-gate-report.mjs`) instead of read off raw volume.
+2. Devices that never update are permanently ungated. Nothing on this plan can reach them.
+   If the org ever moves to Business, `filters:error_messages` is the row to revisit first;
+   re-probe it rather than assuming, since these answers are plan state, not physics.
+
 ### Gate coverage against 90d of real events
 
 Every issue in the project over the 90d to 2026-08-14 (648 events), replayed through the

scripts/noise-gate-report.mjs

@@ -51,9 +51,24 @@
  */
 
 import { execFileSync } from "node:child_process"
+import { readFile } from "node:fs/promises"
 import { fileURLToPath } from "node:url"
 import { dirname, join } from "node:path"
 
+import { gateRollout, parseRolloutHistory } from "./sentry-volume-report.mjs"
+
+/** Rollout instant of the first gated production build, or null if unrecorded.
+ *  Never guessed: an unrecorded rollout must surface as UNGRADED, not as a
+ *  window silently assumed to be post-gate. */
+async function gateRolloutAt() {
+  try {
+    const md = await readFile(new URL("../docs/playstore.md", import.meta.url), "utf8")
+    return gateRollout(parseRolloutHistory(md))?.at ?? null
+  } catch {
+    return null
+  }
+}
+
 const HERE = dirname(fileURLToPath(import.meta.url))
 const MONTH_HOURS = 730
 
@@ -96,7 +111,14 @@ export const TOLERANCE = Number(process.env.NOISE_GATE_TOLERANCE || 0.25)
  * @param {number} [p.efficacy]        share of events the gate drops on a gated device
  * @param {boolean} [p.gateLive]       did client_discard/before_send appear at all
  */
-export function grade({ baselinePerHour, actualPerHour, gatedShare, efficacy = GATE_EFFICACY, gateLive = true }) {
+export function grade({
+  baselinePerHour,
+  actualPerHour,
+  gatedShare,
+  efficacy = GATE_EFFICACY,
+  gateLive = true,
+  postPhase = "post",
+}) {
   const share = Math.min(Math.max(gatedShare ?? 0, 0), 1)
   const expectedPerHour = baselinePerHour * (1 - share * efficacy)
   const ceilingPerHour = expectedPerHour * (1 + TOLERANCE)
@@ -108,7 +130,15 @@ export function grade({ baselinePerHour, actualPerHour, gatedShare, efficacy = G
 
   let verdict
   let because
-  if (!gateLive) {
+  if (postPhase && postPhase !== "post") {
+    // A window that starts before the gate reached production contains devices
+    // that could not possibly have run it, so it dilutes the rate toward
+    // baseline and grades the gate as worse than it is.
+    verdict = "UNGRADED"
+    because =
+      `the post window is [${postPhase}] — it does not lie entirely after the gate's production rollout, ` +
+      "so it mixes gated and ungated devices. Re-run with a post window starting at the rollout instant."
+  } else if (!gateLive) {
     verdict = "UNGRADED"
     because =
       "client_discard/before_send is 0 in this window — no device ran the gate, so nothing here measures it. " +
@@ -132,6 +162,7 @@ export function grade({ baselinePerHour, actualPerHour, gatedShare, efficacy = G
   return {
     verdict,
     because,
+    postPhase,
     gatedShare: share,
     efficacy,
     baselinePerMonth: baselinePerHour * MONTH_HOURS,
@@ -172,7 +203,12 @@ export function gatedShareFromPlay(playJson, firstGatedVersionCode = GATE_FIRST_
   const gated = versions
     .filter((v) => Number(v.versionCode) >= firstGatedVersionCode)
     .reduce((s, v) => s + (v.users || 0), 0)
-  return { gated, total, share: total > 0 ? gated / total : 0, window: playJson?.window ?? null }
+  return {
+    gated,
+    total,
+    share: total > 0 ? gated / total : 0,
+    window: playJson?.window ?? null,
+  }
 }
 
 function runJson(script, args) {
@@ -194,8 +230,34 @@ function parseArgs(argv) {
     else if (a === "--json") out.json = true
     else if (a === "--no-play") out.play = false
   }
-  if (!out.pre) throw new Error("--pre START..END is required (the rate before the rollout reached devices)")
-  if (!out.post) out.post = `${new Date(Date.now() - 7 * 86_400_000).toISOString()}..now`
+  return out
+}
+
+/** Documented pre-rollout baseline window (docs/analytics.md): starts after the
+ *  AGE-55 box-bot fix landed (2026-08-14 06:19Z) and ends before the gate
+ *  rollout (14:22Z). It is deliberately NOT "7 days before the rollout" — that
+ *  spans the box-bot regime and would import ~22k/mo of dead volume into the
+ *  org background estimate, making the outlook miss the gate by a mile for a
+ *  reason that was already fixed. */
+export const BASELINE_WINDOW = "2026-08-14T07:00:00Z..2026-08-14T14:00:00Z"
+
+/** Fill in windows the caller did not pin, anchored on the rollout instant.
+ *
+ *  The old default (`post = now-7d..now`) straddles the 2026-08-14 14:22Z
+ *  rollout on every run before 08-21, so the grader's own default window mixed
+ *  gated and ungated devices and diluted the measured rate toward baseline —
+ *  i.e. it was biased toward reporting the gate as ineffective. The window may
+ *  never start before the gate reached production.
+ */
+export function resolveWindows(args, rolloutAt, now = new Date()) {
+  const out = { ...args }
+  if (!out.pre) out.pre = BASELINE_WINDOW
+  if (!out.post) {
+    if (!rolloutAt) throw new Error("--post START..END is required (no rollout instant in docs/playstore.md)")
+    const weekAgo = new Date(now.getTime() - 7 * 86_400_000)
+    const start = weekAgo > rolloutAt ? weekAgo : rolloutAt
+    out.post = `${start.toISOString()}..now`
+  }
   return out
 }
 
@@ -205,6 +267,7 @@ function pick(report, windowName, project) {
   const p = w.projects.find((x) => x.project === project)
   return {
     hours: w.hours,
+    phase: w.phase ?? null,
     orgPerHour: w.orgSubmitted / w.hours,
     perHour: p ? p.submittedPerHour : 0,
     submitted: p ? p.submitted : 0,
@@ -216,7 +279,7 @@ function pick(report, windowName, project) {
 const money = (n) => Math.round(n).toLocaleString("en-US")
 
 async function main() {
-  const args = parseArgs(process.argv.slice(2))
+  const args = resolveWindows(parseArgs(process.argv.slice(2)), await gateRolloutAt())
 
   const volume = runJson("sentry-volume-report.mjs", [
     "--json",
@@ -240,6 +303,7 @@ async function main() {
     actualPerHour: post.perHour,
     gatedShare: shareInfo.share,
     gateLive: post.gateDropped > 0,
+    postPhase: post.phase,
   })
   // Other projects are a background rate, not something this gate changes, so
   // estimate them from whichever window is long enough to contain any of them.
@@ -255,7 +319,10 @@ async function main() {
   const result = {
     project: args.project,
     generatedAt: new Date().toISOString(),
-    windows: { pre: { ...pre, spec: args.pre }, post: { ...post, spec: args.post } },
+    windows: {
+      pre: { ...pre, spec: args.pre },
+      post: { ...post, spec: args.post },
+    },
     uptake: shareInfo,
     grade: g,
     org,

scripts/noise-gate-report.test.mjs

@@ -5,6 +5,8 @@ import {
   GATE_EFFICACY,
   ORG_MONTHLY_GATE,
   gatedShareFromPlay,
+  BASELINE_WINDOW,
+  resolveWindows,
   grade,
   orgOutlook,
 } from "./noise-gate-report.mjs"
@@ -26,7 +28,11 @@ test("partial uptake is not gate failure — the mistake this script exists to p
 })
 
 test("full uptake at the replayed efficacy lands under the project target", () => {
-  const g = grade({ baselinePerHour: BASELINE, actualPerHour: BASELINE * (1 - GATE_EFFICACY), gatedShare: 1 })
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE * (1 - GATE_EFFICACY),
+    gatedShare: 1,
+  })
   assert.equal(g.verdict, "ON_TRACK")
   assert.ok(g.meetsProjectTargetAtFullUptake)
   assert.ok(g.fullUptakeProjection < 150, `expected ~107/mo, got ${g.fullUptakeProjection}`)
@@ -35,27 +41,48 @@ test("full uptake at the replayed efficacy lands under the project target", () =
 test("a real regression still fails even when uptake is low enough to excuse a lot", () => {
   // 10% uptake excuses almost nothing; volume that did not move at all is fine,
   // but volume that GREW is a finding.
-  const flat = grade({ baselinePerHour: BASELINE, actualPerHour: BASELINE, gatedShare: 0.1 })
+  const flat = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE,
+    gatedShare: 0.1,
+  })
   assert.equal(flat.verdict, "ON_TRACK", "flat volume at 10% uptake is within tolerance")
-  const worse = grade({ baselinePerHour: BASELINE, actualPerHour: BASELINE * 1.6, gatedShare: 0.1 })
+  const worse = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE * 1.6,
+    gatedShare: 0.1,
+  })
   assert.equal(worse.verdict, "OFF_TRACK")
   assert.match(worse.because, /new\s+noise class|not dropping/)
 })
 
 test("a gate that does nothing on device is caught once uptake is high", () => {
-  const g = grade({ baselinePerHour: BASELINE, actualPerHour: BASELINE, gatedShare: 0.9 })
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE,
+    gatedShare: 0.9,
+  })
   assert.equal(g.verdict, "OFF_TRACK")
   assert.equal(g.impliedEfficacy, 0, "0% of the drop happened, so implied efficacy is 0")
 })
 
 test("no before_send discards means the window measures nothing — refuse to grade it", () => {
-  const g = grade({ baselinePerHour: BASELINE, actualPerHour: 0.01, gatedShare: 0.9, gateLive: false })
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: 0.01,
+    gatedShare: 0.9,
+    gateLive: false,
+  })
   assert.equal(g.verdict, "UNGRADED")
   assert.match(g.because, /before_send/)
 })
 
 test("zero uptake is ungraded, not a pass — a quiet weekend is not efficacy", () => {
-  const g = grade({ baselinePerHour: BASELINE, actualPerHour: 0.2, gatedShare: 0 })
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: 0.2,
+    gatedShare: 0,
+  })
   assert.equal(g.verdict, "UNGRADED")
   assert.equal(g.impliedEfficacy, null)
 })
@@ -84,7 +111,11 @@ test("no Play rows at all reads as 0% uptake, never as a divide-by-zero pass", (
 
 test("org outlook subtracts this project before projecting it forward", () => {
   // org 5.43/h of which mobile is 4.71/h -> other projects 0.72/h ~= 526/mo
-  const g = grade({ baselinePerHour: BASELINE, actualPerHour: BASELINE, gatedShare: 0.5 })
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE,
+    gatedShare: 0.5,
+  })
   const o = orgOutlook({
     orgPerHour: 5.43,
     projectPerHour: BASELINE,
@@ -96,7 +127,63 @@ test("org outlook subtracts this project before projecting it forward", () => {
 })
 
 test("org gate can still be missed by other projects even with a perfect mobile gate", () => {
-  const o = orgOutlook({ orgPerHour: 6, projectPerHour: 0.5, projectFullUptakePerMonth: 100 })
+  const o = orgOutlook({
+    orgPerHour: 6,
+    projectPerHour: 0.5,
+    projectFullUptakePerMonth: 100,
+  })
   assert.equal(o.clearsOrgGate, false)
   assert.ok(o.headroom < 0)
 })
+
+test("a post window that straddles the rollout is UNGRADED, not a failing grade", () => {
+  // The old default post window (now-7d..now) started before the 08-14 14:22Z
+  // rollout on every run until 08-21, mixing devices that could not have run
+  // the gate into the "after" rate — which biases the verdict toward failure.
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: BASELINE * 0.9,
+    gatedShare: 0.4,
+    postPhase: "mixed",
+  })
+  assert.equal(g.verdict, "UNGRADED")
+  assert.match(g.because, /mixes gated and ungated/)
+})
+
+test("an unrecorded rollout is UNGRADED — never assumed to be post-gate", () => {
+  const g = grade({
+    baselinePerHour: BASELINE,
+    actualPerHour: 0.1,
+    gatedShare: 0.9,
+    postPhase: "unknown",
+  })
+  assert.equal(g.verdict, "UNGRADED", "a great-looking number from an unattributable window is not a pass")
+})
+
+test("resolveWindows never starts the post window before the gate reached production", () => {
+  const rollout = new Date("2026-08-14T14:22:00Z")
+  // Run three days in: a naive 7d lookback would reach back to 08-10, four days
+  // before any device could have had the gate.
+  const w = resolveWindows({}, rollout, new Date("2026-08-17T00:00:00Z"))
+  assert.equal(w.post, `${rollout.toISOString()}..now`)
+
+  // Well after rollout, a 7d lookback is entirely post-gate and is preferred:
+  // a longer window is what makes the other projects' background rate visible.
+  const later = resolveWindows({}, rollout, new Date("2026-09-01T00:00:00Z"))
+  assert.equal(later.post, "2026-08-25T00:00:00.000Z..now")
+})
+
+test("no rollout on record and no explicit post window is an error, not a default", () => {
+  assert.throws(() => resolveWindows({}, null), /--post/)
+})
+
+test("the default baseline excludes the box-bot era, which would swamp the org outlook", () => {
+  // A "7 days before the rollout" baseline spans the AGE-55 box-bot fix
+  // (2026-08-14 06:19Z) and drags ~22k/mo of already-fixed volume into the
+  // background estimate, so the org outlook MISSES for a dead reason.
+  const w = resolveWindows({}, new Date("2026-08-14T14:22:00Z"), new Date("2026-08-17T00:00:00Z"))
+  assert.equal(w.pre, BASELINE_WINDOW)
+  const [start, end] = BASELINE_WINDOW.split("..").map((s) => new Date(s))
+  assert.ok(start >= new Date("2026-08-14T06:19:00Z"), "baseline must start after the box-bot fix")
+  assert.ok(end <= new Date("2026-08-14T14:22:00Z"), "baseline must end before the gate rollout")
+})