Repository navigation
Conversation
Every Hardcover call now takes a token from one shared bucket (burst 5, 0.8/s, under the documented 10 burst / 60 per minute) before it is sent. A 429 is retried up to twice, waiting for Retry-After when the API sends it and otherwise backing off exponentially with jitter. The wait also empties the shared bucket so other threads hold back too. A Retry-After longer than 30s (a spent daily quota) is not waited out. When retries run out, _execute_query raises HardcoverRateLimitError with raise_on_error and returns None without it, as before.
…ng None A throttled or failed lookup came back as None, which callers read as 'no such book'. get_book now asks _execute_query to raise, so a 429 that survives the retries reaches the caller as HardcoverRateLimitError. GraphQL rejections still return None.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kept as small as I could so it's easy to review. It is still about 140 lines of code plus tests, in two commits so the second can be dropped on its own. Same kind of fix as #1429, so see the question there about separate PRs versus combined.
This is a defensive change. I found the problem on my own deployment, which has a more automated ingestion on top of this provider: Hardcover want-to-read items turn into requests, and each pending request gets one
get_booklookup. One pass made around 38 lookups back to back, which is more than Hardcover's burst allows, and a few came back as 429._execute_queryswallowed the 429 and returnedNone, soget_bookreturnedNone, and the caller read that as "no such book". Nothing in the logs said a lookup had been throttled.What it does
Retry-Afterwhen Hardcover sends one, otherwise 2^attempt seconds plus up to a second of jitter, and the wait holds back other threads too. ARetry-Afterover 30 seconds (a spent daily quota) is not waited out.HardcoverRateLimitError, aRuntimeError, instead of returningNone.get_bookask for that error, so a throttled or failed lookup is no longer taken for a missing book. GraphQL rejections still returnNone.Behaviour change in the second commit.
/api/metadata/bookalready turns aRuntimeErrorinto a 503, so a timeout or HTTP error there now returns 503 instead of 404. In/api/releasesthe same error goes through the operational errors handler and returns 500 with the message, where it used to return 404 "Book not found". If you'd rather keep returningNoneand only have the pacing and backoff, I can drop the second commit.Limits. I read them from the
hardcover-docsrepo (api/Getting-Started.mdx, Rate Limits), because the docs site blocks my headless browser: Free is 60 per minute, a burst of 10 and 5,000 a day, withRetry-Afteron a 429. Personal Access Tokens get double the burst. I didn't parse theRateLimit-Policyheaders, so the bucket constants are fixed.Testing. The tests use a fake clock and never sleep. I broke each rule in turn (pacing,
Retry-After, the 30 second cap, the shared hold-back, andget_bookraising) and each break failed a test. Ruff, basedpyright,tests/metadataandtests/corepass. I haven't seen a real 429 from Hardcover with this change running, so theRetry-Afterhandling comes from the docs.The one red check is the existing clock-dependent test
test_a_queued_torrent_asks_for_a_grace_once, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change.