tools(sentry): anchor the measurement windows on the gate's rollout instant - #174
Closed
dzianisv wants to merge 1 commit into
Closed
tools(sentry): anchor the measurement windows on the gate's rollout instant#174dzianisv wants to merge 1 commit into
dzianisv wants to merge 1 commit into
Conversation
dzianisv
force-pushed
the
age-105-uptake-normalized-verdict
branch
from
August 14, 2026 15:53
c7542dd to
ff5c784
Compare
…nstant A window that spans the 2026-08-14 14:22Z production rollout contains devices that could not possibly have run the gate. Its rate is neither a baseline nor a result, and it prints identically to both. This was not hypothetical: a `post=08-14T07:00Z..now` window (84% of it pre-rollout) was run against this script and reported opencode-mobile *rising* to 4.44/h. `noise-gate-report.mjs` shipped with the same defect built into its default: `post = now-7d..now` straddles the rollout on every run before 08-21, diluting the after-rate toward baseline - biased toward grading the gate as ineffective on exactly the dates the ticket schedules its reads (08-17, 08-21). - sentry-volume-report: `--since-rollout` reads the instant from the release history table in docs/playstore.md (production versionCode >= 150, earliest such release, so a later v0.4.15 does not restart the window) and splits there. Every window is labelled [pre]/[post]/[mixed]; mixed prints how much of it predates the gate, a young post window prints its uptake age, and an unparseable table reports "unknown" rather than assuming post. - noise-gate-report: defaults post to the rollout instant, returns UNGRADED for a mixed/unknown post window, and pins the baseline to the documented post-box-bot-fix window instead of a 7d lookback that dragged ~22k/mo of already-fixed box-bot volume into the org outlook (it read "MISSES by 18,612" for a dead reason; now 628/mo, clears). - before_send == 0 is now reported as expected in a pre/mixed/young window and as a failure only after 24h+ of gated production. Re-probed every server-side lever with a WRITE-scoped token so none of the answers is a permissions artifact, and corrected the record in docs/analytics.md: per-key rate limit returns 200 and silently drops the field; error-message filters return 400 "You do not have that feature enabled" (a plan gate, not absence - it is the one lever that would reach never-updating installs); spike protection is not 403-unavailable, it is already enabled everywhere and simply does not fire on sustained baseline volume.
dzianisv
force-pushed
the
age-105-rollout-boundary-guard
branch
from
August 14, 2026 15:54
7ba2a7b to
210451e
Compare
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #173 (which is stacked on #172). Review/merge in order.
The windows were free-form, so the reads scheduled for 08-17 and 08-21 could not be trusted
The gate reached Play production at 2026-08-14 14:22Z. A measurement window that spans that instant contains devices that could not possibly have run the gate. Its rate is neither a baseline nor a result — and it prints identically to both.
This is not hypothetical. I ran
--window "post=2026-08-14T07:00:00Z..now"against this very script earlier today. 84% of that window predated the rollout, and it reportedopencode-mobilerising to 4.44/h. Same tool, same data, wrong conclusion, no warning.Worse:
noise-gate-report.mjs— the script that produces the ticket's verdict — shipped with the identical defect as its default.post = now-7d..nowstraddles the rollout on every run before 08-21, which dilutes the after-rate toward baseline. The bias runs one way: it grades the gate as less effective than it is, on exactly the dates the ticket schedules its reads.What changed
--since-rolloutreads the rollout instant out of the release-history table indocs/playstore.md— the same table the publish workflow already makes humans update — and splits the windows exactly there. It takes the earliest production release with versionCode >= 150, so a later v0.4.15 does not silently restart the measurement window.[pre],[post], or[mixed]with the percentage that predates the gate. A young post window prints how many hours of uptake it actually has.unknown, never a silentpost. Absence of the record must not read as a clean measurement.noise-gate-report.mjsreturnsUNGRADEDfor a mixed or unknown post window, consistent with how it already treatsbefore_send == 0and zero Play share.openclaw-box-botvolume into the org estimate: the report said "MISSES the org gate by 18,612/mo". With the correct baseline it reads 628/mo, clears. Both from live data, minutes apart.before_send == 0is now reported as expected in a pre/mixed/young window, and as a failure only after 24h+ of gated production. Previously it always printed as "no device is running the gate", which at 1.4h post-rollout is alarm noise.Server-side levers re-probed, and the record corrected
Re-ran every lever with a write-scoped token, so none of these answers is a permissions artifact. Two of the three previously recorded conclusions were wrong in ways that would misdirect the next person:
rateLimit: null. The success code is the trap.filters:error_messages"You do not have that feature enabled"— plan-gated, not absent. This is the one lever that would reach installs that never update. Revisit first if the org ever moves to Business.quotas:spike-protection-disabled = false), and it did not fire because this is sustained baseline volume, not a spike. Previously recorded as "403 / unavailable", which invites spending a plan upgrade on a feature that is already on and already ineffective.Verification
scripts/sentry-volume-report.test.mjs— 8 new tests, including one that parses the realdocs/playstore.md(so the table drifting breaks CI, not the 08-17 read) and one that pins the exact bad window from today.scripts/noise-gate-report.test.mjsfor the UNGRADED paths and the baseline choice.