fix(cache): run sync handlers in the threadpool - #134
Merged
Merged
Conversation
The @cache wrapper is async, so FastAPI no longer offloads a def handler and get_response() called it on the event loop. Run sync handlers with run_in_threadpool, detect an async __call__ on the handler's type, and await any awaitable a sync callable returns. Closes #100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100.
Problem
@cachewraps every handler in anasyncwrapper, so FastAPI no longer sends adefhandler to its threadpool.get_response()then called the sync handler inline, on the event loop, and blocking I/O in it stalled every other request on the worker.The coroutine check also missed an object with an
async def __call__, and a sync callable that returns a coroutine; both produced an un-awaited coroutine that then failed to encode.Change
get_response()awaits coroutine handlers and runs everything else withstarlette.concurrency.run_in_threadpool._is_coroutine_callable():inspect.iscoroutinefunction(func)(which already unwrapsfunctools.partial), or the same check ontype(func).__call__. A class counts as sync, because its type's__call__istype.__call__.A note on the issue text: a
functools.partialof an async function was already handled, sinceinspect.iscoroutinefunctionunwraps partials. It stays in the test as a regression case.Tests
test_sync_handler_runs_off_the_event_loop_threadrecords the thread ident of a sync handler and an async handler, and asserts they differ.test_handler_returning_a_coroutine_is_awaitedis parametrized over a partial, an async__call__object, and a sync function returning a coroutine.Against the old
get_response(), 3 of these 4 cases fail; the partial case passes.Checks
fastapi_cachex,testsandscriptstestsandscriptsCACHEX_REQUIRE_LIVE_SERVERS=1): 720 passed, 100% coveragezensical build --strictThe CHANGELOG entry sits at the top of
### Fixed, next to #133's, so whichever merges second needs a trivial rebase.