Skip to content

fix(hardcover): defensive rate limiting and 429 backoff - #1439

Open
splitsec2 wants to merge 2 commits into
calibrain:mainfrom
splitsec2:fix/hardcover-429-upstream
Open

splitsec2 wants to merge 2 commits into
calibrain:mainfrom
splitsec2:fix/hardcover-429-upstream

Conversation

@splitsec2

Copy link
Copy Markdown
Contributor

Kept as small as I could so it's easy to review. It is still about 140 lines of code plus tests, in two commits so the second can be dropped on its own. Same kind of fix as #1429, so see the question there about separate PRs versus combined.

This is a defensive change. I found the problem on my own deployment, which has a more automated ingestion on top of this provider: Hardcover want-to-read items turn into requests, and each pending request gets one get_book lookup. One pass made around 38 lookups back to back, which is more than Hardcover's burst allows, and a few came back as 429. _execute_query swallowed the 429 and returned None, so get_book returned None, and the caller read that as "no such book". Nothing in the logs said a lookup had been throttled.

What it does

  • Every POST goes through one shared token bucket (burst 5, 0.8 requests per second), set under the documented Free limits. The limit is per user across all of the account's apps, so I kept well below it.
  • A 429 is retried up to 3 attempts in total. It waits for Retry-After when Hardcover sends one, otherwise 2^attempt seconds plus up to a second of jitter, and the wait holds back other threads too. A Retry-After over 30 seconds (a spent daily quota) is not waited out.
  • When the retries run out it raises HardcoverRateLimitError, a RuntimeError, instead of returning None.
  • The second commit makes get_book ask for that error, so a throttled or failed lookup is no longer taken for a missing book. GraphQL rejections still return None.

Behaviour change in the second commit. /api/metadata/book already turns a RuntimeError into a 503, so a timeout or HTTP error there now returns 503 instead of 404. In /api/releases the same error goes through the operational errors handler and returns 500 with the message, where it used to return 404 "Book not found". If you'd rather keep returning None and only have the pacing and backoff, I can drop the second commit.

Limits. I read them from the hardcover-docs repo (api/Getting-Started.mdx, Rate Limits), because the docs site blocks my headless browser: Free is 60 per minute, a burst of 10 and 5,000 a day, with Retry-After on a 429. Personal Access Tokens get double the burst. I didn't parse the RateLimit-Policy headers, so the bucket constants are fixed.

Testing. The tests use a fake clock and never sleep. I broke each rule in turn (pacing, Retry-After, the 30 second cap, the shared hold-back, and get_book raising) and each break failed a test. Ruff, basedpyright, tests/metadata and tests/core pass. I haven't seen a real 429 from Hardcover with this change running, so the Retry-After handling comes from the docs.

The one red check is the existing clock-dependent test test_a_queued_torrent_asks_for_a_grace_once, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change.

Every Hardcover call now takes a token from one shared bucket (burst 5,
0.8/s, under the documented 10 burst / 60 per minute) before it is sent.
A 429 is retried up to twice, waiting for Retry-After when the API sends
it and otherwise backing off exponentially with jitter. The wait also
empties the shared bucket so other threads hold back too. A Retry-After
longer than 30s (a spent daily quota) is not waited out. When retries run
out, _execute_query raises HardcoverRateLimitError with raise_on_error and
returns None without it, as before.
…ng None

A throttled or failed lookup came back as None, which callers read as
'no such book'. get_book now asks _execute_query to raise, so a 429 that
survives the retries reaches the caller as HardcoverRateLimitError.
GraphQL rejections still return None.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant