fix(qb): claim jobs through capacity slots - #777
Merged
Conversation
A picked job of a limited entrypoint will hold one slot in 0..limit-1, and a partial unique index on (entrypoint, slot) rejects a second claim of the same slot even when the first is invisible to the claimer's snapshot. Nothing writes the column yet; the dequeue statement starts using it separately. Fresh installs get the column inline. Existing installs get it through the upgrade path, where rows picked before the migration keep slot NULL, stay covered by the count gate alone, and drain out as they complete. Startup verification learns both objects, so a worker running the slot-aware dequeue against an un-upgraded database fails loudly rather than silently overshooting the limit.
The persistence layer had to ask whether a driver exception was a unique violation, and answered it by importing asyncpg and psycopg and testing isinstance against their error classes. SqlStateError declares the shape both drivers share: an exception carrying a PostgreSQL SQLSTATE. A new sqlstate module reads the code through that protocol and names the two codes the dequeue cares about. is_unique_violation keeps its name, since it is re-exported from the public shim, but delegates now instead of importing drivers. Dequeue uses it to report an empty batch when it loses a capacity slot race. Losing that race means the capacity went to another worker, which is the same outcome as finding nothing claimable; the winner's commit fires the notification that wakes the poll loop.
Windowing candidates before the lock (#774) closes the reported race, because both workers derive the same window and collide on the same row locks. It stops holding as soon as the ordering changes under them: a higher-priority job arriving inside another worker's in-flight claim gives the second worker a window the first never locked, and the limit is exceeded again. Every fresh claim on a limited entrypoint now takes a slot in 0..limit-1. Two workers racing for the last slot both pass the count gate, but the loser's index insertion waits on the winner and fails with 23505, which dequeue reports as an empty batch for the poll loop to retry. A stale re-pick keeps the slot it already holds, so recovery is not gated. The free-slot search generates held+batch seats rather than one row per configured seat, so a poll costs the same at any limit. Unbounded, concurrency_limit=5000 produced 286k plan rows and ran 67x slower than an ungated dequeue; bounded, it matches concurrency_limit=10. The ungated shapes render byte-identical SQL, which is what proves this touches only entrypoints that carry a limit. Closes #761
3 tasks
janbjorge
changed the title
janbjorge
marked this pull request as ready for review
August 30, 2026 12:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Concurrent workers could exceed an entrypoint's
concurrency_limit. #774fixed the case where both workers see the same queue ordering. This covers
the rest: a higher-priority job arriving inside another worker's in-flight
claim changes that ordering, so the second worker windows a row the first
never locked.
Every fresh claim on a limited entrypoint now takes a numbered slot in
0..limit-1, guarded by a partial unique index on(entrypoint, slot) WHERE status = 'picked'. Counting picked rows reads asnapshot and cannot see an uncommitted claim. A btree uniqueness check can,
so the loser of a slot race gets
23505and reports an empty batch, whichthe poll loop retries. Unlimited entrypoints carry no slot and stay outside
the index.
Rows picked before the migration hold no slot and are covered by the count
gate until they drain. Mixed limits across workers stay unsupported (#776).
Depends on #751: the
slotcolumn and its index need registering in theschema manifest, which replaces the probes this touches in
qm.py.Closes #761