Skip to content

tools(sentry): anchor the measurement windows on the gate's rollout instant - #174

Closed
dzianisv wants to merge 1 commit into
age-105-uptake-normalized-verdictfrom
age-105-rollout-boundary-guard
Closed

tools(sentry): anchor the measurement windows on the gate's rollout instant#174
dzianisv wants to merge 1 commit into
age-105-uptake-normalized-verdictfrom
age-105-rollout-boundary-guard

Conversation

@dzianisv

Copy link
Copy Markdown
Owner

Stacked on #173 (which is stacked on #172). Review/merge in order.

The windows were free-form, so the reads scheduled for 08-17 and 08-21 could not be trusted

The gate reached Play production at 2026-08-14 14:22Z. A measurement window that spans that instant contains devices that could not possibly have run the gate. Its rate is neither a baseline nor a result — and it prints identically to both.

This is not hypothetical. I ran --window "post=2026-08-14T07:00:00Z..now" against this very script earlier today. 84% of that window predated the rollout, and it reported opencode-mobile rising to 4.44/h. Same tool, same data, wrong conclusion, no warning.

Worse: noise-gate-report.mjs — the script that produces the ticket's verdict — shipped with the identical defect as its default. post = now-7d..now straddles the rollout on every run before 08-21, which dilutes the after-rate toward baseline. The bias runs one way: it grades the gate as less effective than it is, on exactly the dates the ticket schedules its reads.

What changed

  • --since-rollout reads the rollout instant out of the release-history table in docs/playstore.md — the same table the publish workflow already makes humans update — and splits the windows exactly there. It takes the earliest production release with versionCode >= 150, so a later v0.4.15 does not silently restart the measurement window.
  • Every window prints its phase: [pre], [post], or [mixed] with the percentage that predates the gate. A young post window prints how many hours of uptake it actually has.
  • An unparseable or missing release table reports unknown, never a silent post. Absence of the record must not read as a clean measurement.
  • noise-gate-report.mjs returns UNGRADED for a mixed or unknown post window, consistent with how it already treats before_send == 0 and zero Play share.
  • Its baseline default is pinned to the documented post-box-bot-fix window. A 7d lookback spans the AGE-55 fix and dragged ~22k/mo of already-dead openclaw-box-bot volume into the org estimate: the report said "MISSES the org gate by 18,612/mo". With the correct baseline it reads 628/mo, clears. Both from live data, minutes apart.
  • before_send == 0 is now reported as expected in a pre/mixed/young window, and as a failure only after 24h+ of gated production. Previously it always printed as "no device is running the gate", which at 1.4h post-rollout is alarm noise.

Server-side levers re-probed, and the record corrected

Re-ran every lever with a write-scoped token, so none of these answers is a permissions artifact. Two of the three previously recorded conclusions were wrong in ways that would misdirect the next person:

Lever Result
Per-key rate limit HTTP 200 that lies — field silently dropped, GET reads rateLimit: null. The success code is the trap.
filters:error_messages HTTP 400 "You do not have that feature enabled" — plan-gated, not absent. This is the one lever that would reach installs that never update. Revisit first if the org ever moves to Business.
Generic inbound filters HTTP 204, works — but the only ids are browser/crawler/localhost. None can match a React Native error.
Spike protection Already enabled on every project (quotas:spike-protection-disabled = false), and it did not fire because this is sustained baseline volume, not a spike. Previously recorded as "403 / unavailable", which invites spending a plan upgrade on a feature that is already on and already ineffective.

Verification

  • scripts/sentry-volume-report.test.mjs — 8 new tests, including one that parses the real docs/playstore.md (so the table drifting breaks CI, not the 08-17 read) and one that pins the exact bad window from today.
  • 4 new tests in scripts/noise-gate-report.test.mjs for the UNGRADED paths and the baseline choice.
  • Full suite: 304 pass, 0 fail.
  • Both scripts run live against Sentry; output in the commit message and above.

@dzianisv
dzianisv force-pushed the age-105-uptake-normalized-verdict branch from c7542dd to ff5c784 Compare August 14, 2026 15:53
…nstant

A window that spans the 2026-08-14 14:22Z production rollout contains devices
that could not possibly have run the gate. Its rate is neither a baseline nor a
result, and it prints identically to both. This was not hypothetical: a
`post=08-14T07:00Z..now` window (84% of it pre-rollout) was run against this
script and reported opencode-mobile *rising* to 4.44/h.

`noise-gate-report.mjs` shipped with the same defect built into its default:
`post = now-7d..now` straddles the rollout on every run before 08-21, diluting
the after-rate toward baseline - biased toward grading the gate as ineffective
on exactly the dates the ticket schedules its reads (08-17, 08-21).

- sentry-volume-report: `--since-rollout` reads the instant from the release
  history table in docs/playstore.md (production versionCode >= 150, earliest
  such release, so a later v0.4.15 does not restart the window) and splits
  there. Every window is labelled [pre]/[post]/[mixed]; mixed prints how much
  of it predates the gate, a young post window prints its uptake age, and an
  unparseable table reports "unknown" rather than assuming post.
- noise-gate-report: defaults post to the rollout instant, returns UNGRADED for
  a mixed/unknown post window, and pins the baseline to the documented
  post-box-bot-fix window instead of a 7d lookback that dragged ~22k/mo of
  already-fixed box-bot volume into the org outlook (it read "MISSES by 18,612"
  for a dead reason; now 628/mo, clears).
- before_send == 0 is now reported as expected in a pre/mixed/young window and
  as a failure only after 24h+ of gated production.

Re-probed every server-side lever with a WRITE-scoped token so none of the
answers is a permissions artifact, and corrected the record in docs/analytics.md:
per-key rate limit returns 200 and silently drops the field; error-message
filters return 400 "You do not have that feature enabled" (a plan gate, not
absence - it is the one lever that would reach never-updating installs); spike
protection is not 403-unavailable, it is already enabled everywhere and simply
does not fire on sustained baseline volume.
@dzianisv

Copy link
Copy Markdown
Owner Author

Auto-closed when age-105-uptake-normalized-verdict was deleted on the #175 squash-merge. Branch rebased onto main and reopened as #178 — same commit content.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant