tools(sentry): org-wide volume report + measure on submitted, not accepted

.github/workflows/publish-play-store.yml

@@ -125,6 +125,18 @@ jobs:
           RELEASE_KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
         run: ./gradlew bundleRelease
 
+      - name: Verify the Sentry noise gate is in the artifact
+        # AGE-105: the org error quota is defended ONLY by the client-side gate
+        # (every server-side lever on this Sentry plan was checked and is dead:
+        # per-key rate limit silently no-ops with a 200, custom inbound filters
+        # absent, spike protection 403). If a build ships without the gate — or
+        # without a DSN, which makes Sentry a silent no-op — the org goes back
+        # over quota, and while it is over quota Sentry stores nothing, so the
+        # regression is invisible in Sentry itself until the monthly reset.
+        # Grep the shipped Hermes bundle instead. Verified to discriminate:
+        # v0.4.14 passes, pre-gate v0.4.13 fails.
+        run: node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab
+
       - name: Upload AAB artifact
         uses: actions/upload-artifact@v7
         with:

docs/analytics.md

@@ -115,6 +115,158 @@ Rules are pure and unit-tested in `src/lib/sentry-noise.test.ts` (18 tests, incl
 the observed 1,126-event hour → 5 delivered events). Widening the drop-list is a deliberate
 act: add a test asserting the new pattern, and never add anything that could mask a crash.
 
+### Measuring whether it worked
+
+```sh
+SENTRY_AUTH_TOKEN=… node scripts/sentry-volume-report.mjs --by-reason \
+  --org vibetechnologies \
+  --window "before=2026-08-14T07:00:00Z..2026-08-14T14:00:00Z" \
+  --window "after=2026-08-17T00:00:00Z..now"
+```
+
+Read **`submitted` = `accepted` + `rate_limited`**, never `accepted` alone. The org is
+currently over its error quota, so Sentry rejects essentially everything and `accepted`
+reads ~0 for *every* project — a blown org and a fixed one look identical on that column.
+`submitted` is the demand the clients actually put on the wire, which is what the
+3,500/month gate is really about.
+
+**And do not read raw `client_discard` as "the gate is working" either — that is the same
+mistake one column over.** Split it by reason (`--by-reason`, and the split line prints
+unconditionally):
+
+| `client_discard` reason | what it means |
+|---|---|
+| `before_send` | **our noise gate dropped the event.** Recorded by `@sentry/core` `baseclient.js` whenever `beforeSend` returns `null`. The only proof the gate is live on real devices. |
+| `ratelimit_backoff` | the SDK is in 429 backoff because the **org** is over quota. A symptom of the overage; it goes UP when things get worse. |
+| `event_processor`, `network_error` | neither of the above. |
+
+On 2026-08-14, 100% of `opencode-mobile`'s `client_discard` was `ratelimit_backoff` and
+`before_send` was 0 — i.e. the pre-rollout `client_discard` number was entirely quota
+damage, not filtering. So the healthy shape is precisely: `submitted` falls **and**
+`client_discard/before_send` rises from zero.
+
+`before_send > 0` is also the **earliest** available evidence, because it does not depend on
+what share of the install base has updated: one device on v0.4.14 hitting one filtered error
+produces it. Check it before waiting days for the monthly rate to bend.
+
+**Do not try to segment the after-number by app release.** While the org is over quota,
+rate-limited events are never stored, so the project's `release`/`dist` tag values and issue
+list stop dead (last value: `opencode-mobile@0.4.12`, 2026-08-08) even though clients keep
+submitting. Version share comes from Play (`scripts/play-version-share.mjs`), not Sentry.
+
+Pre-rollout baseline for the v0.4.14 comparison (measured 2026-08-14 14:00 UTC, before
+production rollout at 14:22 UTC), two windows agreeing to within 0.2%:
+
+| Window | `opencode-mobile` submitted | → /month | Org submitted → /month |
+|---|---|---|---|
+| 7h, post-box-bot-fix (08-14 07:00–14:00Z) | 33 (4.71/h) | 3,441 | 3,963 |
+| 7d trailing (08-07–08-14) | 793 (4.72/h) | 3,446 | 25,450 (box-bot pre-fix) |
+
+Mobile was 87% of the org's post-box-bot demand. Target: under ~1,500/month, which puts the
+org under the 3,500/month gate.
+
+### The quota resets on the 4th — that is the real deadline
+
+Org-wide daily `accepted` shows a hard billing boundary:
+
+| Date | org `accepted` | org `rate_limited` | cumulative `accepted` |
+|---|---|---|---|
+| 08-03 | 2 | 427 | 24 |
+| **08-04** | **837** | 10 | 861 |
+| 08-07 | 1,820 | 0 | 3,812 |
+| **08-08** | 1,574 | 816 | **5,386** |
+| 08-09 → 08-14 | 0 | 155–672/day | 5,387 |
+
+The period reset on **2026-08-04**, the 5,000-error month was spent in **4.5 days**, and the
+org has been receiving *zero* error data since **2026-08-08**. Next reset: **2026-09-04**.
+Two consequences: (1) no `accepted`-based measurement is possible before then, which is why
+`submitted` is the metric; (2) 09-04 is the date the gate actually has to hold by.
+
+Who spent it, over the 30d to 08-14:
+
+| project | accepted | rate_limited | submitted |
+|---|---|---|---|
+| `openclaw-box-bot` | 4,401 (82%) | 11,741 | 16,142 |
+| `vibe-api-gateway` | 254 | 3,321 | 3,575 |
+| `opencode-mobile` | 664 (12%) | 2,148 | 2,812 |
+| `openclaw-ci` | 68 | 333 | 401 |
+
+`opencode-mobile` did not blow the quota — `openclaw-box-bot` did (fixed by AGE-55). But with
+box-bot at 0, mobile is now the dominant remaining demand.
+
+### Server-side levers do not exist on this plan
+
+Checked directly against the API on 2026-08-14, so nobody re-litigates it:
+
+| Lever | Result |
+|---|---|
+| Per-key rate limit (`PUT /projects/{org}/{proj}/keys/{id}/`) | **Silent no-op.** Returns HTTP 200 and drops the field; a follow-up GET always reads `rateLimit: null`. Reproduced with `window` = 60, 3600 and 86400. |
+| Custom inbound filters (error message / release) | Not present. Only the five generic browser filters exist. |
+| Spike protection (`/organizations/{org}/spike-protections/`) | HTTP 403. |
+
+Org `features` is `[]`. **The client-side gate is the only control that exists**, so its
+coverage is the entire safety margin — which is why `sentry-noise-production.test.ts` pins
+that coverage against real production data.
+
+### Gate coverage against 90d of real events
+
+Every issue in the project over the 90d to 2026-08-14 (648 events), replayed through the
+gate's own precedence (allowlist first, then drop-list) by
+`src/lib/sentry-noise-production.test.ts`:
+
+| Outcome | events | share |
+|---|---|---|
+| hard-dropped as transport noise | 628 | 96.9% |
+| always-send crash classes (OOM, ANR, `IllegalStateException`) | 13 | 2.0% |
+| deduped / rate-capped (the 401 storm) | 7 | 1.1% |
+
+Upper bound of surviving volume: 2,750/month × 3.1% ≈ **87/month**, ~17× under the
+1,500/month target, and that ignores dedup and the hourly cap, which only push it lower.
+Crash classes still come through — the test fails if any of them stops being allowlisted,
+because hitting the number by silencing real crashes is a failure, not a win.
+
+### The one piece of evidence available on release day: check the artifact
+
+Everything above is worthless if a build ships without the gate. That failure is
+*invisible in Sentry*: while the org is over quota nothing is stored, so release tags stop
+updating (opencode-mobile's stop at `0.4.12` / 2026-08-08 while clients keep submitting
+~4.7/h) and a `release:0.4.14` query returns an empty result that reads exactly like "no
+errors from the new build".
+
+The binary is checkable the same day. Hermes bytecode keeps string literals, so
+`scripts/verify-release-bundle.mjs` greps `base/assets/index.android.bundle` inside the AAB
+for the gate's own reason codes, the transport drop-list regex, and the
+`noise.dropped_since_last` tag that only `applyNoiseGate()` writes — plus the baked-in DSN,
+because a release built without `EXPO_PUBLIC_SENTRY_DSN` makes `Sentry.init()` a silent
+no-op. It runs in `publish-play-store.yml` **before** the Play upload step, so a gateless
+build cannot reach users.
+
+It discriminates — this is not a self-confirming assertion:
+
+| Artifact | Result |
+|---|---|
+| v0.4.14 AAB, versionCode 151 (the build now on Play production, run `31807432647`) | **passes**, all six markers, DSN = project `4511436292292608` |
+| v0.4.13 AAB (run `31786473735`, pre-gate) | **fails**, all six markers absent |
+
+### Rejected: persisting gate state across launches
+
+The rate caps (`maxPerHour`, `maxNewPerHour`) and the 6h per-fingerprint cooldown live in
+process memory, so every cold start resets them. That looks like a hole worth plugging
+with `AsyncStorage`; the session data says it is not.
+
+Sentry session envelopes for `opencode-mobile`, 7d to 2026-08-14: **2,633** (267–541/day),
+which is the install base's app-start rate. Persisting only pays off if a device launches
+the app *more often than the cooldown expires* — i.e. if `376 launches/day ÷ devices > 4`,
+so only below **~94 active devices**. One issue alone (`connect timeout`) has 104 distinct
+users over 90d, so the install base is above that line and the two rates are within
+rounding of each other. Persisting would add native storage I/O on the crash path to buy
+nothing measurable. Revisit only if the app-start rate rises well above ~4/device/day.
+
+(Those session envelopes are 100% `client_discard`, 96% `network_error` — sent at cold
+start and at process teardown, when the transport often can't complete. Sessions are not
+billed, so this costs no quota, but it does mean release health is not a usable signal for
+this app either.)
+
 ## Disclosure surfaces (must stay in sync)
 
 | Surface | File |

package.json

@@ -10,7 +10,7 @@
     "android": "expo run:android",
     "typecheck": "tsc --noEmit",
     "check:versions": "node scripts/check-version-parity.mjs",
-    "test": "node --test 'src/**/*.test.ts'"
+    "test": "node --test 'src/**/*.test.ts' 'scripts/*.test.mjs'"
   },
   "dependencies": {
     "@expo/vector-icons": "^15.0.3",