Skip to content

perf(redis): delete each SCAN page as it arrives in clear operations - #211

Merged
allen0099 merged 1 commit into
masterfrom
perf/redis-clear-by-page
Sep 26, 2026
Merged

allen0099 merged 1 commit into
masterfrom
perf/redis-clear-by-page

Conversation

@allen0099

Copy link
Copy Markdown
Owner

Closes #172

Problem

clear(), clear_pattern() and clear_path() scanned the whole matching keyspace into memory before deleting anything. clear_path() also ran an extra EXISTS on the direct key. For clear_path("/p", include_params=True) over 250 keys, the recorded client calls were scan scan scan exists delete delete delete.

Change

  • New _delete_matching(pattern) deletes each SCAN page as it arrives and sums the DEL replies. The same call order is now scan delete scan delete scan delete delete. clear(), clear_path() and both passes of clear_pattern() (including the deprecated prefix-stripped retry from 0.4.0: remove the Redis clear_pattern prefix fallback #125) use it.
  • clear_path() deletes the direct key with a plain DEL, since DEL already returns 0 for a missing key.
  • Deleting keys during a SCAN is safe, because SCAN still returns every key present for the whole iteration. If SCAN repeats a key on a later page (Redis get_all_keys: deduplicate keys returned twice by SCAN #173), that key is already gone, so DEL does not count it twice.
  • _scan_keys (used by get_all_keys) and _delete_keys (used by delete_many) are unchanged.

Tests

  • test_redis_clear_deletes_each_scan_page_as_it_arrives[clear|clear_pattern|clear_path]: over three pages, the first DEL comes before the last SCAN.
  • test_redis_clear_path_does_not_check_exists.
  • test_redis_clear_counts_a_key_scan_repeats_once: a repeat is injected on every later page, and clear_pattern still reports the exact total.
  • Mutation checks:
    • With redis.py reverted to master, the first four tests fail. The repeat test still passes, because master deduplicated keys before DEL.
    • With deleted += len(page) in place of the DEL reply, only the repeat test fails.
  • ruff, mypy --strict, full suite against live Redis and Memcached (CACHEX_REQUIRE_LIVE_SERVERS=1): 866 passed. backends/redis.py coverage is 100%.

CHANGELOG entry

Section: Changed (tracked in #206)

- **Redis clear operations delete page by page.** `clear()`, `clear_pattern()`
  and `clear_path()` delete each SCAN page as it arrives instead of first
  collecting every matching key in memory, and `clear_path()` no longer runs
  an extra `EXISTS` on the direct key.
  ([#172](https://github.com/allen0099/FastAPI-CacheX/issues/172))

clear(), clear_pattern() and clear_path() collected every matching key
in memory before deleting any of them. Delete each SCAN page as it
arrives and sum the DEL replies instead. A key SCAN repeats on a later
page is already gone by then, and DEL counts only keys that existed, so
it is not counted twice.

clear_path() also ran EXISTS on the direct key before adding it to the
delete. DEL already returns 0 for a missing key, so delete it directly.

Closes #172
@allen0099 allen0099 added this to the 0.3.8 milestone Sep 26, 2026
@allen0099 allen0099 added enhancement New feature or request backends Cache backends and their atomic primitives labels Sep 26, 2026
@allen0099
allen0099 merged commit 0a6df96 into master Sep 26, 2026
11 checks passed
@allen0099
allen0099 deleted the perf/redis-clear-by-page branch September 26, 2026 14:35
allen0099 added a commit that referenced this pull request Sep 27, 2026
Copies the CHANGELOG sections of #207, #208, #209, #211, #212, #272,
#273, #274, #275, #276, #279 and #284 into Unreleased. The #273 entry
drops expire_if_equals from its list of methods that changed, since that
primitive is new in 0.3.8.
allen0099 added a commit that referenced this pull request Sep 27, 2026
Copies the CHANGELOG sections of #207, #208, #209, #211, #212, #272,
#273, #274, #275, #276, #279 and #284 into Unreleased. The #273 entry
drops expire_if_equals from its list of methods that changed, since that
primitive is new in 0.3.8.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backends Cache backends and their atomic primitives enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redis clear operations: delete page by page and drop the extra EXISTS

1 participant