Skip to content

fix(verify): treat a rate-limit answer as transient, not as a dead link - #52

Merged
Seungpyo1007 merged 1 commit into
mainfrom
fix/transient-429-not-cached-dead
Aug 3, 2026
Merged

fix(verify): treat a rate-limit answer as transient, not as a dead link#52
Seungpyo1007 merged 1 commit into
mainfrom
fix/transient-429-not-cached-dead

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

Problem

check-urls treats HTTP 429 as a liveness verdict and caches it for the full 30-day TTL.

Current cache contents by host:

host result
gsmarena.com 3,998 http-429, 44 http-200
en.wikipedia.org 68 alive / 6 404
techpowerup.com 61 http-403, 25 alive

Probed directly, every sampled 429'd GSMArena URL answers 200, even with no pacing at all:

--- pacing 0.0s, GET ---
  200 1.5s  blackview_bv5200_pro-12056.php
  200 1.1s  blackview_bv5300_pro-12410.php
  200 1.0s  blackview_bv8900-12372.php

So ~4,000 records are unpromotable not because their sources are bad, but because one run asked too fast and the refusal was written down as a fact. There is already a 1 req/s per-host limiter — but once the host starts refusing, every remaining URL on it cascades into 429 at that same pace.

Change

  • Retry 429/503 up to 3 times, honouring Retry-After when present (clamped to 15s).
  • Back off per host: a host that pushes back gets its interval multiplied for the rest of the run, so the cascade stops.
  • Never cache a rate limit: a still-throttled URL isn't written, and 429/503 entries already in the cache are ignored on load — which heals the 3,998 poisoned entries without a manual purge.

Liveness classification is otherwise unchanged: 404, 403, soft-404 redirects and connection errors are still dead.

4 regression tests added (retry-then-succeed, persistent-throttle-is-transient, host slowdown, poisoned-cache-entry-is-not-a-hit); tests/verify is 50 passed.

Refs #1

check-urls recorded HTTP 429 as a liveness verdict and cached it for the
full 30-day TTL. 3,998 of the 4,042 cited GSMArena URLs are currently
parked as dead that way — yet every one of them answers 200 when asked
at a civil pace. Those records can never be promoted, not because their
sources are bad but because we asked too fast once.

Three changes:

* 429/503 is retried up to 3 times, honouring Retry-After when the host
  sends one.
* A host that pushes back has its per-host interval multiplied for the
  rest of the run, so one rate-limited host stops cascading into 429s
  for every remaining URL on it.
* A still-rate-limited URL is not written to the cache, and existing
  cached 429s are ignored on load. A 429 is not an answer, so it must
  not occupy an answer's slot.

Refs #1
@Seungpyo1007 Seungpyo1007 added the bug Something isn't working label Aug 3, 2026
@Seungpyo1007 Seungpyo1007 self-assigned this Aug 3, 2026
@Seungpyo1007
Seungpyo1007 merged commit 4da91fc into main Aug 3, 2026
1 check passed
@Seungpyo1007
Seungpyo1007 deleted the fix/transient-429-not-cached-dead branch August 3, 2026 07:40
Seungpyo1007 added a commit that referenced this pull request Aug 3, 2026
The rate-limit penalty added in #52 is multiplicative with no ceiling,
so a host that refuses repeatedly walks its interval up without bound
(1s -> 4 -> 16 -> 64 -> ...). Over the ~4,000 cited GSMArena URLs that
turns a run into hours of sleeping and risks the job timeout.

Cap the per-host interval at 30s.

Refs #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant