fix(verify): treat a rate-limit answer as transient, not as a dead link - #52
Merged
Merged
Conversation
check-urls recorded HTTP 429 as a liveness verdict and cached it for the full 30-day TTL. 3,998 of the 4,042 cited GSMArena URLs are currently parked as dead that way — yet every one of them answers 200 when asked at a civil pace. Those records can never be promoted, not because their sources are bad but because we asked too fast once. Three changes: * 429/503 is retried up to 3 times, honouring Retry-After when the host sends one. * A host that pushes back has its per-host interval multiplied for the rest of the run, so one rate-limited host stops cascading into 429s for every remaining URL on it. * A still-rate-limited URL is not written to the cache, and existing cached 429s are ignored on load. A 429 is not an answer, so it must not occupy an answer's slot. Refs #1
Seungpyo1007
added a commit
that referenced
this pull request
Aug 3, 2026
The rate-limit penalty added in #52 is multiplicative with no ceiling, so a host that refuses repeatedly walks its interval up without bound (1s -> 4 -> 16 -> 64 -> ...). Over the ~4,000 cited GSMArena URLs that turns a run into hours of sleeping and risks the job timeout. Cap the per-host interval at 30s. Refs #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
check-urlstreats HTTP 429 as a liveness verdict and caches it for the full 30-day TTL.Current cache contents by host:
http-429, 44http-200http-403, 25 aliveProbed directly, every sampled 429'd GSMArena URL answers 200, even with no pacing at all:
So ~4,000 records are unpromotable not because their sources are bad, but because one run asked too fast and the refusal was written down as a fact. There is already a 1 req/s per-host limiter — but once the host starts refusing, every remaining URL on it cascades into 429 at that same pace.
Change
Retry-Afterwhen present (clamped to 15s).Liveness classification is otherwise unchanged: 404, 403, soft-404 redirects and connection errors are still dead.
4 regression tests added (retry-then-succeed, persistent-throttle-is-transient, host slowdown, poisoned-cache-entry-is-not-a-hit);
tests/verifyis 50 passed.Refs #1