Repository navigation
perf(redis): delete each SCAN page as it arrives in clear operations - #211
Merged
Merged
Conversation
clear(), clear_pattern() and clear_path() collected every matching key in memory before deleting any of them. Delete each SCAN page as it arrives and sum the DEL replies instead. A key SCAN repeats on a later page is already gone by then, and DEL counts only keys that existed, so it is not counted twice. clear_path() also ran EXISTS on the direct key before adding it to the delete. DEL already returns 0 for a missing key, so delete it directly. Closes #172
18 of 20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #172
Problem
clear(),clear_pattern()andclear_path()scanned the whole matching keyspace into memory before deleting anything.clear_path()also ran an extraEXISTSon the direct key. Forclear_path("/p", include_params=True)over 250 keys, the recorded client calls werescan scan scan exists delete delete delete.Change
_delete_matching(pattern)deletes each SCAN page as it arrives and sums theDELreplies. The same call order is nowscan delete scan delete scan delete delete.clear(),clear_path()and both passes ofclear_pattern()(including the deprecated prefix-stripped retry from 0.4.0: remove the Redisclear_patternprefix fallback #125) use it.clear_path()deletes the direct key with a plainDEL, sinceDELalready returns 0 for a missing key.DELdoes not count it twice._scan_keys(used byget_all_keys) and_delete_keys(used bydelete_many) are unchanged.Tests
test_redis_clear_deletes_each_scan_page_as_it_arrives[clear|clear_pattern|clear_path]: over three pages, the firstDELcomes before the lastSCAN.test_redis_clear_path_does_not_check_exists.test_redis_clear_counts_a_key_scan_repeats_once: a repeat is injected on every later page, andclear_patternstill reports the exact total.redis.pyreverted to master, the first four tests fail. The repeat test still passes, because master deduplicated keys beforeDEL.deleted += len(page)in place of theDELreply, only the repeat test fails.mypy --strict, full suite against live Redis and Memcached (CACHEX_REQUIRE_LIVE_SERVERS=1): 866 passed.backends/redis.pycoverage is 100%.CHANGELOG entry
Section: Changed (tracked in #206)