diff --git a/.github/workflows/publish-play-store.yml b/.github/workflows/publish-play-store.yml index 41361fa4..d9ca9eb5 100644 --- a/.github/workflows/publish-play-store.yml +++ b/.github/workflows/publish-play-store.yml @@ -125,6 +125,18 @@ jobs: RELEASE_KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }} run: ./gradlew bundleRelease + - name: Verify the Sentry noise gate is in the artifact + # AGE-105: the org error quota is defended ONLY by the client-side gate + # (every server-side lever on this Sentry plan was checked and is dead: + # per-key rate limit silently no-ops with a 200, custom inbound filters + # absent, spike protection 403). If a build ships without the gate — or + # without a DSN, which makes Sentry a silent no-op — the org goes back + # over quota, and while it is over quota Sentry stores nothing, so the + # regression is invisible in Sentry itself until the monthly reset. + # Grep the shipped Hermes bundle instead. Verified to discriminate: + # v0.4.14 passes, pre-gate v0.4.13 fails. + run: node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab + - name: Upload AAB artifact uses: actions/upload-artifact@v7 with: diff --git a/docs/analytics.md b/docs/analytics.md index d07ff6e4..dbf47532 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -115,6 +115,158 @@ Rules are pure and unit-tested in `src/lib/sentry-noise.test.ts` (18 tests, incl the observed 1,126-event hour → 5 delivered events). Widening the drop-list is a deliberate act: add a test asserting the new pattern, and never add anything that could mask a crash. +### Measuring whether it worked + +```sh +SENTRY_AUTH_TOKEN=… node scripts/sentry-volume-report.mjs --by-reason \ + --org vibetechnologies \ + --window "before=2026-08-14T07:00:00Z..2026-08-14T14:00:00Z" \ + --window "after=2026-08-17T00:00:00Z..now" +``` + +Read **`submitted` = `accepted` + `rate_limited`**, never `accepted` alone. The org is +currently over its error quota, so Sentry rejects essentially everything and `accepted` +reads ~0 for *every* project — a blown org and a fixed one look identical on that column. +`submitted` is the demand the clients actually put on the wire, which is what the +3,500/month gate is really about. + +**And do not read raw `client_discard` as "the gate is working" either — that is the same +mistake one column over.** Split it by reason (`--by-reason`, and the split line prints +unconditionally): + +| `client_discard` reason | what it means | +|---|---| +| `before_send` | **our noise gate dropped the event.** Recorded by `@sentry/core` `baseclient.js` whenever `beforeSend` returns `null`. The only proof the gate is live on real devices. | +| `ratelimit_backoff` | the SDK is in 429 backoff because the **org** is over quota. A symptom of the overage; it goes UP when things get worse. | +| `event_processor`, `network_error` | neither of the above. | + +On 2026-08-14, 100% of `opencode-mobile`'s `client_discard` was `ratelimit_backoff` and +`before_send` was 0 — i.e. the pre-rollout `client_discard` number was entirely quota +damage, not filtering. So the healthy shape is precisely: `submitted` falls **and** +`client_discard/before_send` rises from zero. + +`before_send > 0` is also the **earliest** available evidence, because it does not depend on +what share of the install base has updated: one device on v0.4.14 hitting one filtered error +produces it. Check it before waiting days for the monthly rate to bend. + +**Do not try to segment the after-number by app release.** While the org is over quota, +rate-limited events are never stored, so the project's `release`/`dist` tag values and issue +list stop dead (last value: `opencode-mobile@0.4.12`, 2026-08-08) even though clients keep +submitting. Version share comes from Play (`scripts/play-version-share.mjs`), not Sentry. + +Pre-rollout baseline for the v0.4.14 comparison (measured 2026-08-14 14:00 UTC, before +production rollout at 14:22 UTC), two windows agreeing to within 0.2%: + +| Window | `opencode-mobile` submitted | → /month | Org submitted → /month | +|---|---|---|---| +| 7h, post-box-bot-fix (08-14 07:00–14:00Z) | 33 (4.71/h) | 3,441 | 3,963 | +| 7d trailing (08-07–08-14) | 793 (4.72/h) | 3,446 | 25,450 (box-bot pre-fix) | + +Mobile was 87% of the org's post-box-bot demand. Target: under ~1,500/month, which puts the +org under the 3,500/month gate. + +### The quota resets on the 4th — that is the real deadline + +Org-wide daily `accepted` shows a hard billing boundary: + +| Date | org `accepted` | org `rate_limited` | cumulative `accepted` | +|---|---|---|---| +| 08-03 | 2 | 427 | 24 | +| **08-04** | **837** | 10 | 861 | +| 08-07 | 1,820 | 0 | 3,812 | +| **08-08** | 1,574 | 816 | **5,386** | +| 08-09 → 08-14 | 0 | 155–672/day | 5,387 | + +The period reset on **2026-08-04**, the 5,000-error month was spent in **4.5 days**, and the +org has been receiving *zero* error data since **2026-08-08**. Next reset: **2026-09-04**. +Two consequences: (1) no `accepted`-based measurement is possible before then, which is why +`submitted` is the metric; (2) 09-04 is the date the gate actually has to hold by. + +Who spent it, over the 30d to 08-14: + +| project | accepted | rate_limited | submitted | +|---|---|---|---| +| `openclaw-box-bot` | 4,401 (82%) | 11,741 | 16,142 | +| `vibe-api-gateway` | 254 | 3,321 | 3,575 | +| `opencode-mobile` | 664 (12%) | 2,148 | 2,812 | +| `openclaw-ci` | 68 | 333 | 401 | + +`opencode-mobile` did not blow the quota — `openclaw-box-bot` did (fixed by AGE-55). But with +box-bot at 0, mobile is now the dominant remaining demand. + +### Server-side levers do not exist on this plan + +Checked directly against the API on 2026-08-14, so nobody re-litigates it: + +| Lever | Result | +|---|---| +| Per-key rate limit (`PUT /projects/{org}/{proj}/keys/{id}/`) | **Silent no-op.** Returns HTTP 200 and drops the field; a follow-up GET always reads `rateLimit: null`. Reproduced with `window` = 60, 3600 and 86400. | +| Custom inbound filters (error message / release) | Not present. Only the five generic browser filters exist. | +| Spike protection (`/organizations/{org}/spike-protections/`) | HTTP 403. | + +Org `features` is `[]`. **The client-side gate is the only control that exists**, so its +coverage is the entire safety margin — which is why `sentry-noise-production.test.ts` pins +that coverage against real production data. + +### Gate coverage against 90d of real events + +Every issue in the project over the 90d to 2026-08-14 (648 events), replayed through the +gate's own precedence (allowlist first, then drop-list) by +`src/lib/sentry-noise-production.test.ts`: + +| Outcome | events | share | +|---|---|---| +| hard-dropped as transport noise | 628 | 96.9% | +| always-send crash classes (OOM, ANR, `IllegalStateException`) | 13 | 2.0% | +| deduped / rate-capped (the 401 storm) | 7 | 1.1% | + +Upper bound of surviving volume: 2,750/month × 3.1% ≈ **87/month**, ~17× under the +1,500/month target, and that ignores dedup and the hourly cap, which only push it lower. +Crash classes still come through — the test fails if any of them stops being allowlisted, +because hitting the number by silencing real crashes is a failure, not a win. + +### The one piece of evidence available on release day: check the artifact + +Everything above is worthless if a build ships without the gate. That failure is +*invisible in Sentry*: while the org is over quota nothing is stored, so release tags stop +updating (opencode-mobile's stop at `0.4.12` / 2026-08-08 while clients keep submitting +~4.7/h) and a `release:0.4.14` query returns an empty result that reads exactly like "no +errors from the new build". + +The binary is checkable the same day. Hermes bytecode keeps string literals, so +`scripts/verify-release-bundle.mjs` greps `base/assets/index.android.bundle` inside the AAB +for the gate's own reason codes, the transport drop-list regex, and the +`noise.dropped_since_last` tag that only `applyNoiseGate()` writes — plus the baked-in DSN, +because a release built without `EXPO_PUBLIC_SENTRY_DSN` makes `Sentry.init()` a silent +no-op. It runs in `publish-play-store.yml` **before** the Play upload step, so a gateless +build cannot reach users. + +It discriminates — this is not a self-confirming assertion: + +| Artifact | Result | +|---|---| +| v0.4.14 AAB, versionCode 151 (the build now on Play production, run `31807432647`) | **passes**, all six markers, DSN = project `4511436292292608` | +| v0.4.13 AAB (run `31786473735`, pre-gate) | **fails**, all six markers absent | + +### Rejected: persisting gate state across launches + +The rate caps (`maxPerHour`, `maxNewPerHour`) and the 6h per-fingerprint cooldown live in +process memory, so every cold start resets them. That looks like a hole worth plugging +with `AsyncStorage`; the session data says it is not. + +Sentry session envelopes for `opencode-mobile`, 7d to 2026-08-14: **2,633** (267–541/day), +which is the install base's app-start rate. Persisting only pays off if a device launches +the app *more often than the cooldown expires* — i.e. if `376 launches/day ÷ devices > 4`, +so only below **~94 active devices**. One issue alone (`connect timeout`) has 104 distinct +users over 90d, so the install base is above that line and the two rates are within +rounding of each other. Persisting would add native storage I/O on the crash path to buy +nothing measurable. Revisit only if the app-start rate rises well above ~4/device/day. + +(Those session envelopes are 100% `client_discard`, 96% `network_error` — sent at cold +start and at process teardown, when the transport often can't complete. Sessions are not +billed, so this costs no quota, but it does mean release health is not a usable signal for +this app either.) + ## Disclosure surfaces (must stay in sync) | Surface | File | diff --git a/package.json b/package.json index b3c661a7..1792df9b 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "android": "expo run:android", "typecheck": "tsc --noEmit", "check:versions": "node scripts/check-version-parity.mjs", - "test": "node --test 'src/**/*.test.ts'" + "test": "node --test 'src/**/*.test.ts' 'scripts/*.test.mjs'" }, "dependencies": { "@expo/vector-icons": "^15.0.3", diff --git a/scripts/sentry-volume-report.mjs b/scripts/sentry-volume-report.mjs new file mode 100644 index 00000000..512e0b4e --- /dev/null +++ b/scripts/sentry-volume-report.mjs @@ -0,0 +1,228 @@ +#!/usr/bin/env node +// Org-wide Sentry error-volume report. +// +// Answers one question: how many error events per month is this org actually +// consuming, per project, per outcome — and how did that change across a +// deploy boundary? +// +// Why it exists: the "Sentry event budget" work (see docs/analytics.md) is +// gated on a *measured* rate, not on "the filter is merged". Every heartbeat +// that re-checks the number should run the same query, or the before/after +// comparison is not a comparison. +// +// Usage: +// SENTRY_AUTH_TOKEN=... node scripts/sentry-volume-report.mjs \ +// --by-reason --org vibetechnologies \ +// --window "pre=2026-08-13T14:00:00Z..2026-08-14T06:00:00Z" \ +// --window "post=2026-08-14T14:22:00Z..now" +// +// Notes on reading the output: +// * The headline metric is `submitted` = accepted + rate_limited: every event +// the client actually put on the wire, i.e. real demand against quota. +// Do NOT headline `accepted`. Once the org is over quota, Sentry rejects +// everything and `accepted` collapses to ~0 for every project — which +// makes a broken org look identical to a fixed one. +// * `rate_limited` is demand that arrived after the quota was gone. It is +// evidence of a problem, not of a fix. +// * `client_discard` is NOT a gate metric on its own. Split it by reason +// (`--by-reason`, on by default in the summary line): +// - `before_send` -> our noise gate dropped the event. THIS is the +// only column that proves the gate is running on +// real devices, and it is independent of install +// -base share, so it shows up long before the +// monthly rate bends. +// - `ratelimit_backoff` -> the SDK is in 429 backoff because the ORG is +// over quota. Pure symptom of the overage. It +// rises when things get WORSE. Reading raw +// `client_discard` as "the gate is working" is +// the same class of error as reading `accepted`. +// - `event_processor` / `network_error` -> neither of the above. +// * Per-release attribution is impossible while the org is over quota: +// rate_limited events are never stored, so `release`/`dist` tag values (and +// issues) simply stop. Do not try to segment the after-number by app +// version from Sentry; use outcome+reason, and Play for version share. +// * A window shorter than ~24h cannot certify a monthly rate; it can only +// rank sources. Diurnal load is real. + +const API = "https://sentry.io/api/0" +const MONTH_HOURS = 730 + +function parseArgs(argv) { + const out = { org: process.env.SENTRY_ORG ?? "vibetechnologies", windows: [] } + for (let i = 0; i < argv.length; i++) { + const a = argv[i] + if (a === "--org") out.org = argv[++i] + else if (a === "--window") out.windows.push(argv[++i]) + else if (a === "--json") out.json = true + else if (a === "--by-reason") out.byReason = true + } + return out +} + +function parseWindow(spec) { + const eq = spec.indexOf("=") + if (eq < 0) throw new Error(`bad --window ${spec} (want name=START..END)`) + const name = spec.slice(0, eq) + const [rawStart, rawEnd] = spec.slice(eq + 1).split("..") + if (!rawStart || !rawEnd) throw new Error(`bad --window ${spec} (want name=START..END)`) + const at = (v) => (v === "now" ? new Date() : new Date(v)) + const start = at(rawStart) + const end = at(rawEnd) + if (Number.isNaN(start.getTime()) || Number.isNaN(end.getTime())) { + throw new Error(`bad --window ${spec} (unparseable date)`) + } + if (end <= start) throw new Error(`bad --window ${spec} (end <= start)`) + return { name, start, end, hours: (end - start) / 3_600_000 } +} + +async function sentry(path, token) { + const res = await fetch(`${API}${path}`, { headers: { Authorization: `Bearer ${token}` } }) + const body = await res.json() + if (!res.ok) throw new Error(`${path} -> ${res.status} ${JSON.stringify(body)}`) + return body +} + +async function projectSlugs(org, token) { + const projects = await sentry(`/organizations/${org}/projects/`, token) + return new Map(projects.map((p) => [String(p.id), p.slug])) +} + +async function windowStats(org, token, win) { + const qs = new URLSearchParams({ + field: "sum(quantity)", + category: "error", + start: win.start.toISOString().replace(/\.\d+Z$/, "Z"), + end: win.end.toISOString().replace(/\.\d+Z$/, "Z"), + project: "-1", + }) + qs.append("groupBy", "project") + qs.append("groupBy", "outcome") + qs.append("groupBy", "reason") + const data = await sentry(`/organizations/${org}/stats_v2/?${qs}`, token) + const rows = new Map() + for (const g of data.groups ?? []) { + const key = String(g.by.project) + if (!rows.has(key)) rows.set(key, { outcomes: {}, reasons: {} }) + const row = rows.get(key) + const qty = g.totals["sum(quantity)"] ?? 0 + if (!qty) continue + row.outcomes[g.by.outcome] = (row.outcomes[g.by.outcome] ?? 0) + qty + const reason = g.by.reason ?? "none" + row.reasons[`${g.by.outcome}/${reason}`] = (row.reasons[`${g.by.outcome}/${reason}`] ?? 0) + qty + } + return rows +} + +function fmt(n) { + return n.toLocaleString("en-US", { maximumFractionDigits: 0 }) +} + +async function main() { + const args = parseArgs(process.argv.slice(2)) + const token = process.env.SENTRY_AUTH_TOKEN + if (!token) { + console.error("SENTRY_AUTH_TOKEN is not set. This script is read-only; a scoped read token is enough.") + process.exit(2) + } + if (args.windows.length === 0) { + // Default: last 7 days, one window. Enough to certify a monthly rate. + const end = new Date() + const start = new Date(end.getTime() - 7 * 86_400_000) + args.windows.push(`7d=${start.toISOString()}..${end.toISOString()}`) + } + + const windows = args.windows.map(parseWindow) + const slugs = await projectSlugs(args.org, token) + const results = [] + for (const win of windows) { + results.push({ win, rows: await windowStats(args.org, token, win) }) + } + + const report = { org: args.org, generatedAt: new Date().toISOString(), windows: [] } + for (const { win, rows } of results) { + const projects = [] + let orgSubmitted = 0 + for (const [id, { outcomes, reasons }] of rows) { + const accepted = outcomes.accepted ?? 0 + const rateLimited = outcomes.rate_limited ?? 0 + const submitted = accepted + rateLimited + orgSubmitted += submitted + projects.push({ + project: slugs.get(id) ?? id, + accepted, + rateLimited, + submitted, + clientDiscard: outcomes.client_discard ?? 0, + // The gate. Everything else in client_discard is not us. + gateDropped: reasons["client_discard/before_send"] ?? 0, + backoffDropped: reasons["client_discard/ratelimit_backoff"] ?? 0, + reasons, + filtered: outcomes.filtered ?? 0, + submittedPerHour: submitted / win.hours, + submittedPerMonth: (submitted / win.hours) * MONTH_HOURS, + }) + } + projects.sort((a, b) => b.submitted - a.submitted) + report.windows.push({ + name: win.name, + start: win.start.toISOString(), + end: win.end.toISOString(), + hours: Number(win.hours.toFixed(2)), + projects, + orgSubmitted, + orgSubmittedPerMonth: (orgSubmitted / win.hours) * MONTH_HOURS, + }) + } + + if (args.json) { + console.log(JSON.stringify(report, null, 2)) + return + } + + for (const w of report.windows) { + console.log(`\n== ${w.name} ${w.start} -> ${w.end} (${w.hours}h)`) + console.log( + `${"project".padEnd(24)}${"submitted".padStart(11)}${"accepted".padStart(10)}${"rate_lim".padStart(10)}${"cli_disc".padStart(10)}${"sub/h".padStart(9)}${"sub/mo".padStart(10)}`, + ) + for (const p of w.projects) { + console.log( + `${p.project.padEnd(24)}${fmt(p.submitted).padStart(11)}${fmt(p.accepted).padStart(10)}${fmt(p.rateLimited).padStart(10)}${fmt(p.clientDiscard).padStart(10)}${p.submittedPerHour.toFixed(2).padStart(9)}${fmt(p.submittedPerMonth).padStart(10)}`, + ) + } + console.log( + `${"ORG TOTAL (submitted)".padEnd(24)}${fmt(w.orgSubmitted).padStart(11)}${"".padStart(30)}${(w.orgSubmitted / w.hours).toFixed(2).padStart(9)}${fmt(w.orgSubmittedPerMonth).padStart(10)}`, + ) + + // Always print the gate-liveness split, because raw `client_discard` is + // ambiguous: `ratelimit_backoff` (org over quota, a symptom) looks exactly + // like `before_send` (our gate, the fix) unless you split them. + const gate = w.projects.reduce((n, p) => n + p.gateDropped, 0) + const backoff = w.projects.reduce((n, p) => n + p.backoffDropped, 0) + console.log( + `client_discard split: before_send (our gate) ${fmt(gate)} | ratelimit_backoff (org over quota) ${fmt(backoff)}`, + ) + if (gate === 0) { + console.log(" before_send == 0 -> no device is running the noise gate yet in this window.") + } + + if (args.byReason) { + for (const p of w.projects) { + const entries = Object.entries(p.reasons).sort((a, b) => b[1] - a[1]) + if (entries.length === 0) continue + console.log(` ${p.project}`) + for (const [k, v] of entries) console.log(` ${fmt(v).padStart(8)} ${k}`) + } + } + } + console.log( + "\nGate: org submitted (accepted + rate_limited) < 3,500/month." + + "\nWindows under ~24h rank sources but do not certify a rate." + + "\nGate liveness: client_discard/before_send > 0 for opencode-mobile." + + "\nDo NOT segment by release: over-quota events are never stored, so release tags stop.", + ) +} + +main().catch((err) => { + console.error(err.message) + process.exit(1) +}) diff --git a/scripts/verify-release-bundle.mjs b/scripts/verify-release-bundle.mjs new file mode 100644 index 00000000..c05544f3 --- /dev/null +++ b/scripts/verify-release-bundle.mjs @@ -0,0 +1,137 @@ +#!/usr/bin/env node +// Verify that a release AAB actually ships the Sentry noise gate (AGE-105). +// +// Why this exists: the entire quota fix is client-side. Sentry's server-side +// levers were checked and are all dead on this plan (per-key rate limit returns +// HTTP 200 and silently drops the field; custom inbound filters absent; spike +// protection 403). So if a build ever ships without the gate — a Metro entry +// change, a refactor that stops importing sentry-noise, a release built without +// EXPO_PUBLIC_SENTRY_DSN — the org quietly goes back over quota and nobody +// finds out until the monthly reset. +// +// It also can't be caught by watching Sentry: while the org is over quota, +// events are rejected at ingest and never stored, so release tags stop +// updating (opencode-mobile's stop at 0.4.12 / 2026-08-08 while clients keep +// submitting ~4.7/h). A `release:0.4.14` query returns EMPTY, which reads like +// "no errors from the new build" but actually means "no data at all". +// +// The binary itself is the only same-day evidence. Hermes bytecode keeps string +// literals, so the gate's own reason strings and the drop-list regex are +// greppable in base/assets/index.android.bundle. +// +// Usage: +// node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab +// +// Optional env: +// EXPO_PUBLIC_SENTRY_DSN when set, the DSN baked into the bundle must point +// at the same project id (never printed). +// +// Read-only, dependency-free (uses the `unzip` CLI, present on the runner). + +import { execFileSync } from "node:child_process" +import { existsSync } from "node:fs" + +/** Literals that only exist because the noise gate is in the bundle graph. + * Each is unique to our code, not the Sentry SDK. */ +export const GATE_MARKERS = [ + // NoiseGate reason codes (src/lib/sentry-noise.ts) + "transport-noise", + "new-fingerprint-cap", + "hourly-cap", + "always-send", + // the transport drop-list regex source — the single biggest volume cut + "connect (?:timeout", + // applyNoiseGate() in src/lib/sentry.ts, i.e. the gate is wired to beforeSend + "noise.dropped_since_last", +] + +const DSN_RE = /https:\/\/[0-9a-f]{16,64}@[a-z0-9.-]*ingest[a-z0-9.-]*\/(\d+)/g + +/** Pure check over the raw JS/Hermes bundle bytes. Returns a list of problems; + * empty means the artifact is good. Never returns the DSN itself. */ +export function checkBundle(bundle, opts = {}) { + const text = Buffer.isBuffer(bundle) ? bundle.toString("latin1") : String(bundle) + const problems = [] + + const missing = GATE_MARKERS.filter((marker) => !text.includes(marker)) + if (missing.length) { + problems.push(`noise gate missing from bundle — absent markers: ${missing.join(", ")}`) + } + + const projectIds = [...text.matchAll(DSN_RE)].map((m) => m[1]) + if (projectIds.length === 0) { + // No DSN => Sentry.init() is a no-op => the gate never runs and no telemetry + // arrives at all. A release build must never be in this state. + problems.push("no Sentry DSN baked into the bundle — telemetry would be a silent no-op") + } else if (opts.expectedProjectId && !projectIds.includes(String(opts.expectedProjectId))) { + problems.push( + `bundled DSN points at project ${projectIds.join("/")}, expected ${opts.expectedProjectId}`, + ) + } + + return problems +} + +/** Extract the project id from a DSN without leaking the key. */ +export function dsnProjectId(dsn) { + if (!dsn) return null + const m = /\/(\d+)\/?$/.exec(String(dsn).trim()) + return m ? m[1] : null +} + +export function checkAppConfig(configJson, expectedVersion) { + if (!expectedVersion) return [] + let version + try { + version = JSON.parse(configJson)?.version + } catch { + return ["base/assets/app.config is not valid JSON"] + } + return version === expectedVersion ? [] : [`bundle app.config version ${version}, expected ${expectedVersion}`] +} + +function unzipEntry(archive, entry) { + return execFileSync("unzip", ["-p", archive, entry], { maxBuffer: 256 * 1024 * 1024 }) +} + +function main() { + const aab = process.argv[2] + if (!aab || !existsSync(aab)) { + console.error("usage: node scripts/verify-release-bundle.mjs ") + process.exit(2) + } + + let bundle + try { + bundle = unzipEntry(aab, "base/assets/index.android.bundle") + } catch { + console.error(`FAIL ${aab}: no base/assets/index.android.bundle inside the archive`) + process.exit(1) + } + + const problems = checkBundle(bundle, { expectedProjectId: dsnProjectId(process.env.EXPO_PUBLIC_SENTRY_DSN) }) + + let version = null + try { + const config = unzipEntry(aab, "base/assets/app.config").toString("utf8") + version = JSON.parse(config)?.version ?? null + } catch { + problems.push("could not read base/assets/app.config") + } + + console.log(`artifact: ${aab}`) + console.log(`bundle: ${(bundle.length / 1024 / 1024).toFixed(2)} MiB, app version ${version ?? "unknown"}`) + for (const marker of GATE_MARKERS) { + const ok = bundle.toString("latin1").includes(marker) + console.log(` ${ok ? "ok " : "MISS"} ${marker}`) + } + + if (problems.length) { + console.error("\nFAIL — this build must not go to Play:") + for (const p of problems) console.error(` - ${p}`) + process.exit(1) + } + console.log("\nOK — Sentry noise gate is present and pointed at the expected project.") +} + +if (import.meta.url === `file://${process.argv[1]}`) main() diff --git a/scripts/verify-release-bundle.test.mjs b/scripts/verify-release-bundle.test.mjs new file mode 100644 index 00000000..399db191 --- /dev/null +++ b/scripts/verify-release-bundle.test.mjs @@ -0,0 +1,64 @@ +import assert from "node:assert/strict" +import test from "node:test" + +import { GATE_MARKERS, checkAppConfig, checkBundle, dsnProjectId } from "./verify-release-bundle.mjs" + +const DSN = "https://4f21a8b3c4d5e6f708192a3b4c5d6e7f@o4510132673511424.ingest.us.sentry.io/4511436292292608" +const PROJECT = "4511436292292608" + +function goodBundle(extra = "") { + return Buffer.from(`\u0000\u0001hermes${GATE_MARKERS.join("\u0000")}\u0000${DSN}${extra}`, "latin1") +} + +test("a bundle with every gate marker and the right DSN passes", () => { + assert.deepEqual(checkBundle(goodBundle(), { expectedProjectId: PROJECT }), []) +}) + +test("a bundle built without the gate fails and names the missing markers", () => { + const stripped = Buffer.from(`hermes${DSN}`, "latin1") + const problems = checkBundle(stripped, { expectedProjectId: PROJECT }) + assert.equal(problems.length, 1) + for (const marker of GATE_MARKERS) assert.match(problems[0], new RegExp(marker.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"))) +}) + +test("dropping a single marker is enough to fail — a partial gate is not a gate", () => { + const partial = Buffer.from(`hermes${GATE_MARKERS.slice(1).join("|")}${DSN}`, "latin1") + const problems = checkBundle(partial, { expectedProjectId: PROJECT }) + assert.equal(problems.length, 1) + assert.match(problems[0], /transport-noise/) +}) + +test("a release built without EXPO_PUBLIC_SENTRY_DSN fails — Sentry would be a silent no-op", () => { + const noDsn = Buffer.from(`hermes${GATE_MARKERS.join("|")}`, "latin1") + const problems = checkBundle(noDsn, { expectedProjectId: PROJECT }) + assert.deepEqual(problems, ["no Sentry DSN baked into the bundle — telemetry would be a silent no-op"]) +}) + +test("a DSN for the wrong project fails, and the failure never prints the key", () => { + const wrong = goodBundle().toString("latin1").replace(PROJECT, "9999999999") + const problems = checkBundle(Buffer.from(wrong, "latin1"), { expectedProjectId: PROJECT }) + assert.equal(problems.length, 1) + assert.match(problems[0], /expected 4511436292292608/) + assert.ok(!problems[0].includes("4f21a8b3c4d5e6f708192a3b4c5d6e7f"), "problem text must not leak the DSN key") +}) + +test("without an expected project id the DSN only has to exist", () => { + assert.deepEqual(checkBundle(goodBundle(), {}), []) +}) + +test("dsnProjectId extracts the trailing id and tolerates junk", () => { + assert.equal(dsnProjectId(DSN), PROJECT) + assert.equal(dsnProjectId(`${DSN}/`), PROJECT) + assert.equal(dsnProjectId(""), null) + assert.equal(dsnProjectId(undefined), null) + assert.equal(dsnProjectId("not-a-dsn"), null) +}) + +test("app.config version mismatch is reported, match is silent", () => { + assert.deepEqual(checkAppConfig(JSON.stringify({ version: "0.4.14" }), "0.4.14"), []) + assert.deepEqual(checkAppConfig(JSON.stringify({ version: "0.4.13" }), "0.4.14"), [ + "bundle app.config version 0.4.13, expected 0.4.14", + ]) + assert.deepEqual(checkAppConfig("{", "0.4.14"), ["base/assets/app.config is not valid JSON"]) + assert.deepEqual(checkAppConfig("{", null), []) +}) diff --git a/src/lib/sentry-noise-production.test.ts b/src/lib/sentry-noise-production.test.ts new file mode 100644 index 00000000..b80bdb5f --- /dev/null +++ b/src/lib/sentry-noise-production.test.ts @@ -0,0 +1,168 @@ +// Replay of REAL production Sentry data through the AGE-105 noise gate. +// +// `sentry-noise.test.ts` proves the gate behaves as specified. This file proves +// the specification was aimed at the right targets — that the patterns match the +// event signatures `opencode-mobile` actually emits, at the volumes it actually +// emits them, and therefore that the gate lands under the org quota gate. +// +// Why freeze a census instead of querying Sentry in a test: the numbers below +// are the *justification* for the pattern lists. If someone later edits +// TRANSPORT_NOISE_PATTERNS or ALWAYS_SEND_PATTERNS and the projected volume +// leaves budget — or a genuine crash class stops being allowlisted — that is a +// regression, and it should fail here rather than on next month's Sentry bill. +// A live query would move under our feet and could never fail. +// +// Census: every issue in the `opencode-mobile` Sentry project over the 90 days +// ending 2026-08-14, sorted by event count, taken from +// GET /organizations/vibetechnologies/issues/?statsPeriod=90d&sort=freq. +// Reproduce with `node scripts/sentry-volume-report.mjs` + the issues endpoint. +// +// Volume context measured the same day (see docs/analytics.md): +// * `opencode-mobile` submitted = 2,812 events / 30d (~2,750/month) +// * org error quota 5,000/month, self-imposed gate 3,500/month (AGE-71) +// * AGE-105 target for this project: under 1,500/month +// +// NOTE: server-side levers were tested and are NOT available on this Sentry +// plan — per-key rate limits accept a PUT with HTTP 200 and silently discard +// the value, custom inbound filters are absent, spike protection returns 403. +// This client-side gate is the only control that exists, so its coverage is the +// whole safety margin. + +import { test } from "node:test" +import assert from "node:assert/strict" +import { eventText, isAlwaysSend, isTransportNoise } from "./sentry-noise.ts" + +type Observed = { + /** Sentry short id, so a failure is traceable to the real issue. */ + id: string + /** exception type as Sentry grouped it */ + type: string + /** exception value, verbatim (truncated exactly as Sentry stores it) */ + value: string + /** events in the 90d window */ + count: number + /** what the gate must decide, and why it is the correct decision */ + expect: "drop" | "always-send" | "capped" +} + +/** Frozen 90d production census (648 events across 11 issues). */ +const PRODUCTION_90D: Observed[] = [ + // --- client-side network conditions: already shown to the user as connection + // UI and already trended in PostHog as connection_failed{error_class}. --- + { id: "OPENCODE-MOBILE-3", type: "Error", value: "connect timeout", count: 462, expect: "drop" }, + { id: "OPENCODE-MOBILE-4", type: "Error", value: "connect server-unreachable", count: 157, expect: "drop" }, + { id: "OPENCODE-MOBILE-B", type: "Error", value: "Network request failed", count: 6, expect: "drop" }, + { id: "OPENCODE-MOBILE-7", type: "Error", value: "Request timed out after 30000ms", count: 3, expect: "drop" }, + + // --- genuine crash classes: must survive whatever the quota says. --- + { + id: "OPENCODE-MOBILE-A", + type: "Error", + value: + "Call to function 'ExponentImagePicker.launchImageLibraryAsync' has been rejected.\n→ Caused by: java.lang.IllegalStateException: Attempting to launch an unregistered ActivityResultLauncher with contract expo.modules.imagepicker.contracts.ImageLibraryC", + count: 5, + expect: "always-send", + }, + { + id: "OPENCODE-MOBILE-5", + type: "OutOfMemoryError", + value: "Failed to allocate a 126790640 byte allocation with 12451840 free bytes", + count: 4, + expect: "always-send", + }, + { + id: "OPENCODE-MOBILE-6", + type: "OutOfMemoryError", + value: "Failed to allocate a 8208 byte allocation with 1261584 free bytes", + count: 1, + expect: "always-send", + }, + { + id: "OPENCODE-MOBILE-8", + type: "ApplicationNotResponding", + value: "ANR", + count: 1, + expect: "always-send", + }, + { + id: "OPENCODE-MOBILE-9", + type: "IllegalStateException", + value: "The specified child already has a parent. You must call removeView() on the child's parent first.", + count: 1, + expect: "always-send", + }, + { + id: "OPENCODE-MOBILE-2", + type: "IllegalStateException", + value: "The specified child already has a parent. You must call removeView() on the child's parent first.", + count: 1, + expect: "always-send", + }, + + // --- reportable, but a storm of it must collapse: AGE-107 traced 498 of + // these to one user re-tapping Connect with a wrong password. Dedup + rate + // cap handle it; a hard drop would hide real server-side 401 regressions. --- + { id: "OPENCODE-MOBILE-1", type: "Error", value: "API Error: 401 - ", count: 7, expect: "capped" }, +] + +const asEvent = (o: Observed) => ({ + exception: { values: [{ type: o.type, value: o.value, mechanism: { handled: true } }] }, +}) + +function classify(o: Observed): Observed["expect"] { + const text = eventText(asEvent(o)) + // Mirrors NoiseGate.admit() precedence: allowlist wins over the drop-list. + if (isAlwaysSend(text)) return "always-send" + if (isTransportNoise(text)) return "drop" + return "capped" +} + +test("production replay: every observed issue is classified as intended", () => { + for (const o of PRODUCTION_90D) { + assert.equal( + classify(o), + o.expect, + `${o.id} (${o.count} events, ${o.type}: ${o.value.slice(0, 60)}) should be ${o.expect}`, + ) + } +}) + +test("production replay: no genuine crash class is ever filtered", () => { + // The failure mode this guards: hitting the volume target by silencing real + // crashes. A zero here would make the quota win meaningless. + const crashes = PRODUCTION_90D.filter((o) => o.expect === "always-send") + assert.ok(crashes.length >= 6, "census should still contain the observed crash classes") + for (const o of crashes) { + assert.equal(isAlwaysSend(eventText(asEvent(o))), true, `${o.id} must be allowlisted`) + assert.equal(classify(o), "always-send", `${o.id} must not be reachable by the drop-list`) + } +}) + +test("production replay: gate drops >=95% of observed volume", () => { + let dropped = 0 + let total = 0 + for (const o of PRODUCTION_90D) { + total += o.count + if (classify(o) === "drop") dropped += o.count + } + assert.equal(total, 648, "census total changed — re-derive the projections below") + const share = dropped / total + assert.ok(share >= 0.95, `expected >=95% of volume dropped, got ${(share * 100).toFixed(1)}%`) +}) + +test("production replay: projected monthly volume clears the AGE-105 target", () => { + const SUBMITTED_PER_MONTH = 2750 // measured 2026-08-14, pre-v0.4.14-uptake + const TARGET_PER_MONTH = 1500 // AGE-105 "done means" + let survivingShare = 0 + let total = 0 + for (const o of PRODUCTION_90D) { + total += o.count + if (classify(o) !== "drop") survivingShare += o.count + } + // Upper bound: assumes dedup + the hourly cap never fire, which they will. + const projected = SUBMITTED_PER_MONTH * (survivingShare / total) + assert.ok( + projected < TARGET_PER_MONTH, + `projected ${Math.round(projected)}/month must stay under ${TARGET_PER_MONTH}/month`, + ) +})