Skip to content

feat(observability): rate-limit rejections reach Sentry and PostHog, not just the container log - #246

Merged
BSalaeddin merged 1 commit into
mainfrom
puw/rate-limit-abuse-visibility
Sep 17, 2026
Merged

BSalaeddin merged 1 commit into
mainfrom
puw/rate-limit-abuse-visibility

Conversation

@BSalaeddin

Copy link
Copy Markdown
Collaborator

Why

Prod, last seven days, grep '\[ratelimit\]' inside compose-input-bluetooth-application-qvdfbq-web-1:

total: 3377      kind: burst 1750 / read 1627      path: /api/coupons 3377 (100%)
2690  ip=2a02:4780:2b:2103:0:1a12:44c:1   ua="DealsbuckCouponBot/1.0"

A competitor's coupon bot spent a week walking our 182,598-row catalog and getting throttled — and
every single one of those 3377 rejections was a console.warn inside the container and nothing
else.
No Sentry issue, no analytics event, nothing anyone could alert on or query. The guard in
rateLimit.ts did its job perfectly; the fact that it had to do its job never left the box.

That is the failure mode this repo's own rules call out: Sentry silence meant "no reporter", not
"no abuse". Found during the /paid-user-watch fleet run (2026-09-16).

What changes

New src/lib/abuseSignal.ts, called from rateLimit.ts's existing logAbuse (the log line is
unchanged):

  • Every rejection adds a Sentry breadcrumb, so any later error from that request carries the
    abuse context. Cheap, no network.
  • Sustained abuse from one client — 50 rejections inside one hour — escalates once per client
    per window
    to a Sentry.captureMessage (surface: rate-limit-abuse, carrying the IP, the
    rejection count, the paths and the UA) plus a server-side api_abuse_detected PostHog event.
  • The throttle is the point: 2690 Sentry events would be noise; one event carrying
    rejections_in_window: 2690 is a decision. The window resets, so a bot that comes back tomorrow
    reports again.
  • Analytics gets ip:<truncated sha256> as the distinct id, never the raw address. Sentry — our own
    ops surface, where acting on a scraper means writing an edge rule for that exact IP — does get it.
  • Fire-and-forget by design (void at the call site, documented): the 429 must not wait on
    telemetry, and the reporter reports its own failures rather than throwing into the request path.
  • The tracked-client map is capped at 5000 entries so a rotating-IP flood cannot grow it.

Tests

tests/unit/abuse-signal.test.ts — 5 cases: breadcrumbs below the threshold without escalating,
escalates exactly once at the threshold and stays quiet for the next 25 rejections, sends the hashed
id and never the raw IP, counts two clients separately, and never throws when the analytics capture
fails.

Local gates: tsc --noEmit clean · oxlint clean · prettier --check clean · knip clean ·
343 tests green across the 26 unit files that touch rateLimit.

Follow-up (not in this PR)

The app-level throttle holds but does not stop a determined catalog walk (120 req/min × 50 rows).
A Cloudflare rule for DealsbuckCouponBot/1.0 and that IP range is an owner item — this PR is what
makes the case for it visible without an SSH session.

🤖 Generated with Claude Code

…not just the container log

Prod ran 3377 rate-limit rejections in seven days (2690 from one IP running
DealsbuckCouponBot/1.0 against /api/coupons) and produced zero Sentry issues
and zero analytics events: every rejection was a console.warn inside the web
container. The guard worked; nobody outside an SSH session could know it had to.

Every rejection now leaves a Sentry breadcrumb, and sustained abuse from one
client escalates ONCE per client per hour to a Sentry event plus a server-side
`api_abuse_detected` PostHog event. Throttled on purpose — 2690 events are
noise, one event carrying rejections: 2690 is a decision. The analytics distinct
id is a truncated SHA-256 of the IP, never the raw address.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@BSalaeddin
BSalaeddin merged commit 44c7a51 into main Sep 17, 2026
16 checks passed
@BSalaeddin
BSalaeddin deleted the puw/rate-limit-abuse-visibility branch September 17, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant