fix: enforce concurrency_limit across concurrent workers - #762
Closed
janbjorge wants to merge 2 commits into
Closed
Conversation
Two workers on separate connections both observe zero picked jobs and claim past the entrypoint limit: the capacity count reads the statement snapshot while SKIP LOCKED slides past rows locked by the concurrent claim. The interleaving is forced deterministically by holding worker A's claim transaction open and polling pg_stat_activity, so the test needs no timing sleeps. Fails until the dequeue capacity check is fixed. Refs #761
Two mechanisms let concurrent dequeues exceed a global entrypoint limit: the capacity count reads its own statement snapshot, so an uncommitted claim by another worker is invisible, and LIMIT with FOR UPDATE SKIP LOCKED applies the limit after skipping, sliding past locked rows onto the remaining queued ones. Two layered fixes: - Limited entrypoints fix a candidate window (top rows up to the remaining capacity) before any locking, then lock each window row by primary key in its own LATERAL. SKIP LOCKED and EvalPlanQual can only shrink the pick, never reach outside the window. - Every fresh claim on a limited entrypoint takes a capacity slot in 0..limit-1, guarded by a partial unique index on (entrypoint, slot) WHERE status = 'picked'. The btree uniqueness check sees uncommitted tuples, so the limit holds across snapshots even when a higher priority enqueue reshapes a racing worker's window. Losing the slot race reports an empty batch; the winner's commit fires the table-changed notification that re-wakes the loser. Unlimited entrypoints keep the previous direct locked scan. Rows picked before the migration hold no slot and are covered by the count gate only until they drain. Refs #761
janbjorge
force-pushed
the
fix/global-concurrency-limit-race
branch
from
August 29, 2026 19:48
7751654 to
c763071
Compare
janbjorge
marked this pull request as ready for review
August 30, 2026 11:44
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Global
concurrency_limitcan be exceeded when workers dequeue concurrently: the capacity count in the dequeue CTE reads its own statement snapshot (a concurrent uncommitted claim is invisible), andLIMIT ... FOR UPDATE SKIP LOCKEDslides past rows locked by another worker onto the remaining queued rows.test_concurrency_limit_holds_across_concurrent_dequeues)Closes #761