ci: name the shard that stopped, and say what cancelled means - #878
Open
fujibee wants to merge 3 commits into
Open
ci: name the shard that stopped, and say what cancelled means#878fujibee wants to merge 3 commits into
fujibee wants to merge 3 commits into
Conversation
This was referenced Aug 18, 2026
fujibee
force-pushed
the
ci/name-the-shard-that-stopped
branch
from
August 18, 2026 23:28
add4b87 to
9b86d5e
Compare
The matrix roll-up is one word for twelve jobs, and the word it uses for a shard that ran out of timeout-minutes is 'cancelled'. Two pull requests from an outside contributor sat red for a day on exactly that. Every ubuntu shard, every Windows leg and three of four macOS shards were green; the fourth had been stopped at exactly 25 minutes. The only thing this job said was 'the bats shards did not all pass (result: cancelled)', which names no shard and does not connect 'cancelled' to a runner limit. From the outside it reads as the repository rejecting the change. The roll-up still decides the exit code. What is added is the listing that makes the exit code actionable, and a sentence for the cancelled case saying it is a limit in this repository's CI rather than a defect in the change. The listing is best-effort by construction: if the API call fails, the roll-up's own message stands. It needs actions:read beyond the workflow's contents:read, granted at the job, or that call is a 403 the step swallows and the listing becomes a feature that reads as present and never runs.
Two corrections from review. The emitted text asserted three things this code does not establish: that the cap had been reached, that the log ended mid-suite, and that the change under test was innocent. The query observes a conclusion and a job name, and nothing else. The last claim is worse than vague — a change that hangs drives the shard into the same 25-minute cancel, so the message would have told an outside contributor the opposite of what is known, in exactly the case where it matters. It now names the job and hands over the discriminator: open it, and if it ran about 25 minutes and ends mid-suite naming no failing case, the cap was reached — while saying plainly that the cap being reached does not by itself say whose fault it is. Second, the listing was said to be untestable without a failing shard. It is not. It now runs on every outcome and prints how many matrix jobs it saw, so a permissions or API problem surfaces on a green run as a warning rather than waiting for a red to discover the diagnosis was never able to speak. Also moved the scratch files to $RUNNER_TEMP.
Review found the query selecting a different set from the one it reports on.
`needs.bats-shard.result` rolls up 8 jobs: ubuntu and macos, four shards each.
The predicate was `startswith("bats (")`, which also matches three separately
gated Windows legs -- 11 jobs against the roll-up's 8. On a run where both the
matrix and a Windows leg were red, the diagnosis could have named the Windows
failure as though it explained the roll-up.
Match the matrix shape instead, and compare the count against SHARD_TOTAL times
the matrix's OS list. A mismatch suppresses the listing with a warning rather
than describing some other set of jobs: the control is the comparison, not the
fact that rows came back.
The control added in the previous commit is what exposed this. It printed 11 on
a green run, where the roll-up covers 8.
fujibee
force-pushed
the
ci/name-the-shard-that-stopped
branch
from
August 19, 2026 16:23
90b2690 to
4871ec1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI only. One file,
.github/workflows/tests.yml. Nothing a user installs or runs changes.What happened
Two pull requests from an outside contributor sat red for a day. Their results:
The cancelled shard had run for exactly 25 minutes on both — this workflow's
timeout-minutes. GitHub records a job that exceeds it ascancelled, not astimed out, and the matrix roll-up collapses twelve jobs into that one word.What the required check said, in full:
It names no shard. It does not say that
cancelledhere means a runner limit.From outside the project, that reads as the repository rejecting the change —
and there is nothing on the page to suggest otherwise, or to suggest a re-run
would help.
The change
The roll-up still decides the exit code. What is added is the job name, and a
sentence that hands the reader the discriminator rather than a conclusion:
An earlier revision of this PR said more than that: it asserted the cap had
been hit, that the log ended mid-suite, and that the change under test was
innocent. The query observes a conclusion and a job name. It establishes none of
those three, and the last is false precisely when a change hangs and drives its
own shard into the cap — the case where a wrong answer costs the most. Caught in
review.
Best-effort by construction. The listing sits inside an
if, so a failed APIcall leaves the roll-up's message standing. A diagnosis must never be able to
turn a red into a green, or into a different red.
It needs
actions: read, granted at the job. The workflow'scontents: readdoes not cover the jobs endpoint; without the grant the call is a 403 the step
swallows.
The listing proves it can see, on every run
It runs whatever the outcome and prints how many matrix jobs it found. On a green
run that is the positive control: a permissions or API problem shows up
immediately as
could not list the shard jobs, instead of lying dormant until ared arrives and the diagnosis turns out to have never been able to speak.
An earlier revision of this body claimed the path could not be exercised without
a failing shard. That was wrong, and the control costs one line.
What this does not do
It does not stop the timeout being hit. That is the eight slowest tests, seven of
them
sync start, at 30% of the suite's runtime — #876. And it does not reducethe queueing that makes a slow shard slower — #877.
This is the third of the three, and the only one about what a person sees when it
goes wrong.