Skip to content

tools(sentry): measure gate uptake from the population that emits the errors - #176

Merged
dzianisv merged 1 commit into
mainfrom
age-105-install-base-uptake
Aug 14, 2026
Merged

tools(sentry): measure gate uptake from the population that emits the errors#176
dzianisv merged 1 commit into
mainfrom
age-105-install-base-uptake

Conversation

@dzianisv

Copy link
Copy Markdown
Owner

The number that decides AGE-105 was coming from a population that cannot see most of our users

noise-gate-report.mjs grades the noise gate with

expected_post = baseline x (1 - gated_share x efficacy)

gated_share came from Play's version share. This app's marketed install paths are the self-hosted F-Droid repo and the direct APK (docs/marketing/*), and Play production had been serving versionCode 136 (2026-06-22) for eight weeks when the gate shipped. Play's denominator cannot contain those users; Sentry's can, because it is built from the devices that actually send events.

scripts/release-uptake.mjs reads gated share from Sentry release health. Sessions are billed against a different quota than errors, so they keep arriving while the org is over its error quota and every error is rate-limited away — precisely when this measurement is needed. It also needs no Play service account, so it runs on a laptop instead of only in CI.

First live reading — 7d to 2026-08-14

version       users   share    sessions   share   gated
0.4.10         801   78.5%      8175   75.3%   no
0.4.12         161   15.8%      2369   21.8%   no
0.4.13          30    2.9%       139    1.3%   no
0.4.14           7    0.7%        17    0.2%   yes

gated share (users):    0.7%  (7/1021)
gated share (sessions): 0.2%  (17/10855)
largest ungated cohort: v0.4.10 at 78.5% of active users

v0.4.10 has held ~78% for four weeks; v0.4.12 sat flat at ~16% for three. This install base does not auto-converge on the newest build, which is the assumption written into the Play-based path ("Play auto-updates, so its active base converges on the newest build fast"). Until that cohort moves, the ceiling on any client-side reduction is ~21.5%, whatever the gate does on-device.

What the tests pin

Every one of these fails silently in production if it regresses:

  • 0.4.9 vs 0.4.10 ordering — lexical compare puts them on the wrong side of the gate.
  • Unparseable release names stay in the denominator; dropping them inflates the share.
  • An empty window returns share: null, never 0, so "no data" cannot grade as "nobody upgraded".
  • User share and session share are both reported when they diverge (quota is consumed per event, not per user).
  • The projection refuses undefendable horizons and flat series instead of extrapolating uptake that is not happening.

node --test scripts/release-uptake.test.mjs — 10 pass. Already covered by npm test (scripts/*.test.mjs).

Deliberately not wired into noise-gate-report.mjs here

That file is modified by both #175 and #174. This PR is new files only, so it cannot conflict with or re-orphan that stack. Wiring the Sentry source in as the preferred gated_share (Play as cross-check) follows once they land.

Refs AGE-105.

… errors

The AGE-105 grader multiplies the baseline by `1 - gated_share * efficacy`, so
`gated_share` decides the verdict. Its only source was Play's version share,
which cannot see this app's primary install channels: the self-hosted F-Droid
repo and the direct-APK release, both of which are what the marketing pages
actually point at.

Sentry release health can. Sessions are stored under a separate quota from
errors, so they keep arriving while the org is over its error quota and every
error is rate-limited away — which is exactly when this number is needed. It is
also the right population by construction: a device that sends a session is a
device that can send an error.

First live reading (7d to 2026-08-14, org vibetechnologies):

    0.4.10   801 users  78.5%   ungated
    0.4.12   161 users  15.8%   ungated
    0.4.14     7 users   0.7%   gated
    gated share 0.7% (users) / 0.2% (sessions)

v0.4.10 has held ~78% for four weeks and v0.4.12 sat flat at ~16% for three,
so this install base does not auto-converge on the newest build. Until that
cohort moves, the ceiling on any client-side volume reduction is ~21.5%.

Tests pin the parts that fail silently: 0.4.9 vs 0.4.10 ordering, unparseable
releases staying in the denominator (dropping them inflates the share), an
empty window returning null rather than 0, user-vs-session divergence, and a
projection that refuses horizons and flat series instead of inventing uptake.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@dzianisv
dzianisv merged commit 1852eb8 into main Aug 14, 2026
7 checks passed
@dzianisv
dzianisv deleted the age-105-install-base-uptake branch August 14, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant