fix: reclaim expired item leases#5
Merged
Merged
Conversation
Allow expired item leases to become visible and be claimed by another worker. Also align retry backoff so the first retry uses the configured base delay instead of doubling it.
Extract the lease write path so expired lease reclamation keeps the same behavior while satisfying the strict Credo checks used by CI.
jallum
approved these changes
May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Leasing uses the item
vesting_timeto hide active work from other consumers. Once the lease expiry has passed, that same item should become visible again so another worker can reclaim stale work.Current behavior keeps expired items hidden because
Item.visible?/2rejects any item with alease_id, even whenlease_expires_athas passed.Store.obtain_lease/6also rejects those items for the same reason, sodequeue/5cannot recover a stale lease.This also fixes first-retry timing.
Store.requeue/4incrementserror_countbefore calculating delay, so usingbase_delay * 2^error_countmakes the first retry wait2x base_delay. The first retry should usebase_delay; later retries can grow from there.Steps To Reproduce
Expired lease recovery:
vesting_time = now.lease_duration = 1_000.lease_id,lease_expires_at = now + 1_000, andvesting_time = now + 1_000.now + 1_000, callStore.peek/4orStore.dequeue/5.lease_idis set. After this change, the expired lease is visible and can be claimed by a new worker.Retry timing:
error_count = 0.Store.requeue/4withbase_delay: 1_000.now + 1_000; it waits untilnow + 2_000.now + 1_000.What Changed
lease_expires_atas visible once theirvesting_timehas passed.obtain_lease/6to replace an expired item lease.2^(error_count - 1)so the first retry uses the configured base delay.