Skip to content

Claim a deferred pull by when it is due again; cap a refused host's gap at a day - #1475

Merged
kensac merged 1 commit into
mainfrom
workday-unblock
Nov 18, 2026
Merged

kensac merged 1 commit into
mainfrom
workday-unblock

Conversation

@kensac

@kensac kensac commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Workday ingest has been near zero since 2026-11-16: 58 new Workday jobs on Nov 16, 361 on Nov 17, 0 so far on Nov 18. Non-Workday ingest is normal.

Cause in the queue. A 429 defers the pull by 5 s, but CLAIM_ORDER sorts by created_at. So the refused pull was the oldest again at every slot the host opened. Production showed host_budget.ok flat at 36,899 for more than 19 hours while refused kept climbing, and the worker logs show task 44813480 (gdit) being retried. Meanwhile 727 other Workday pulls waited, and a live request to kohls from the same address returned 200.

Changes

  • CLAIM_ORDER counts from GREATEST(created_at, not_before), so a deferred task goes behind the pulls made after it.
  • The refusal cap moves from the CAP_SECONDS = 900 literal to the host_pace_cap_seconds app_config key, default 86400. A host that keeps refusing gets probed about once a day instead of four times an hour.
  • Docs: sources-and-boards.md (pace section) and observability.md (claim order).

Tests

  • New: test_a_deferred_pull_waits_behind_pulls_made_after_it.
  • The cap test now reads the config key.
  • Full suite: 2861 passed.

…ap at a day

From 2026-11-16 Workday's slot on every address went to the same refused
tenant (gdit) about every 15 minutes. A refused pull is deferred 5 s, and the
claim order counted it from its creation, so it was the oldest again at every
opening while 727 other Workday pulls waited, some of them answering 200.

CLAIM_ORDER now counts from GREATEST(created_at, not_before). The refusal cap
moves from the 900 s literal to host_pace_cap_seconds in app_config, default
one day.
@kensac
kensac merged commit 6fb7e44 into main Nov 18, 2026
13 checks passed
@kensac
kensac deleted the workday-unblock branch November 18, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant