Skip to content

Recheck runs still recorded as running before the completed backlog - #22

Merged
zhaozhaozz merged 1 commit into
mainfrom
fix/pending-in-progress-first
Oct 26, 2026
Merged

zhaozhaozz merged 1 commit into
mainfrom
fix/pending-in-progress-first

Conversation

@zhaozhaozz

Copy link
Copy Markdown
Contributor

Last night's Nightly finished at 21:45 UTC, but collections at 00:47 UTC and later still showed it as in progress.

Why

A run's status changes only through the pending queue: run lists are read by creation time, so a run that was listed while running is updated only when its queued detail job is processed. The queue currently holds about 600 items, mostly completed runs whose reports are still to be read.

Each collection takes up to 30 items: runs the bot notified as reruns, then half "newest run ID" and half "oldest due". In practice a collection processes only 6–17 before the download budget runs out on report downloads.

Runs still recorded as running got no priority (the docs say they should). The Nightly was queued at 19:40, was neither among the newest IDs nor among the oldest due, and had not been processed once (tries: 0).

Change

Sync.pending now puts queued runs whose stored status is not completed (queued or in progress) right after the bot-notified reruns and ahead of the backlog, oldest due first. Re-reading them is what flips them to completed. Only the bounded rerun pass (only_priority) is unchanged. On the current queue this is three runs: last night's Nightly, a Real E2E run recorded as running since 2026-10-24, and a CI run.

docs/incremental-history.md states the order.

Validation

  • New tests/test_run_changes.py test: 39 completed runs wait, and run 20 was saved while running and has since finished. It fails on main (run 40 is inspected first and run 20 is never reached with count=4). With this change run 20 is inspected first, updated to completed/success, and leaves the queue.
  • python3 -m unittest discover -s tests: 167 passed.
  • test/ score tests: OK.
  • python3 test/journey-rerun-collection.py: OK.
  • Checked against the repository's stored queue: the three runs above are the ones now taken first.

Release

Release category: release:fix
User impact: a workflow that finishes is shown as finished at the next collection, even with a large detail backlog.
Compatibility and migration: none.

@zhaozhaozz
zhaozhaozz merged commit cf7f491 into main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant