Repository navigation
Claim a deferred pull by when it is due again; cap a refused host's gap at a day - #1475
Merged
Merged
Conversation
…ap at a day From 2026-11-16 Workday's slot on every address went to the same refused tenant (gdit) about every 15 minutes. A refused pull is deferred 5 s, and the claim order counted it from its creation, so it was the oldest again at every opening while 727 other Workday pulls waited, some of them answering 200. CLAIM_ORDER now counts from GREATEST(created_at, not_before). The refusal cap moves from the 900 s literal to host_pace_cap_seconds in app_config, default one day.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workday ingest has been near zero since 2026-11-16: 58 new Workday jobs on Nov 16, 361 on Nov 17, 0 so far on Nov 18. Non-Workday ingest is normal.
Cause in the queue. A 429 defers the pull by 5 s, but
CLAIM_ORDERsorts bycreated_at. So the refused pull was the oldest again at every slot the host opened. Production showedhost_budget.okflat at 36,899 for more than 19 hours whilerefusedkept climbing, and the worker logs show task 44813480 (gdit) being retried. Meanwhile 727 other Workday pulls waited, and a live request to kohls from the same address returned 200.Changes
CLAIM_ORDERcounts fromGREATEST(created_at, not_before), so a deferred task goes behind the pulls made after it.CAP_SECONDS = 900literal to thehost_pace_cap_secondsapp_config key, default 86400. A host that keeps refusing gets probed about once a day instead of four times an hour.Tests
test_a_deferred_pull_waits_behind_pulls_made_after_it.