Skip to content

ci: name the shard that stopped, and say what cancelled means - #878

Open
fujibee wants to merge 3 commits into
mainfrom
ci/name-the-shard-that-stopped
Open

ci: name the shard that stopped, and say what cancelled means#878
fujibee wants to merge 3 commits into
mainfrom
ci/name-the-shard-that-stopped

Conversation

@fujibee

@fujibee fujibee commented Aug 18, 2026

Copy link
Copy Markdown
Owner

CI only. One file, .github/workflows/tests.yml. Nothing a user installs or runs changes.

What happened

Two pull requests from an outside contributor sat red for a day. Their results:

ubuntu, all four shards      success
Windows, all three legs      success
macOS 2/4, 3/4, 4/4          success
macOS 1/4                    cancelled
bats  (the required check)   failure

The cancelled shard had run for exactly 25 minutes on both — this workflow's
timeout-minutes. GitHub records a job that exceeds it as cancelled, not as
timed out, and the matrix roll-up collapses twelve jobs into that one word.

What the required check said, in full:

::error::the bats shards did not all pass (result: cancelled)

It names no shard. It does not say that cancelled here means a runner limit.
From outside the project, that reads as the repository rejecting the change —
and there is nothing on the page to suggest otherwise, or to suggest a re-run
would help.

The change

The roll-up still decides the exit code. What is added is the job name, and a
sentence that hands the reader the discriminator rather than a conclusion:

Shards that did not pass:
  cancelled	bats (macos-latest 1/4)

A shard whose conclusion is 'cancelled' was stopped rather than failing a
case, and this job cannot tell you why: reaching the 25-minute job cap and
being cancelled from outside the run look identical from here. Open the job
named above. If it ran for about 25 minutes and its log ends mid-suite naming
no failing case, it reached the cap -- which a change that hangs can also
cause, so the cap being reached does not by itself say whose fault it is.

An earlier revision of this PR said more than that: it asserted the cap had
been hit, that the log ended mid-suite, and that the change under test was
innocent. The query observes a conclusion and a job name. It establishes none of
those three, and the last is false precisely when a change hangs and drives its
own shard into the cap — the case where a wrong answer costs the most. Caught in
review.

Best-effort by construction. The listing sits inside an if, so a failed API
call leaves the roll-up's message standing. A diagnosis must never be able to
turn a red into a green, or into a different red.

It needs actions: read, granted at the job. The workflow's contents: read
does not cover the jobs endpoint; without the grant the call is a 403 the step
swallows.

The listing proves it can see, on every run

It runs whatever the outcome and prints how many matrix jobs it found. On a green
run that is the positive control: a permissions or API problem shows up
immediately as could not list the shard jobs, instead of lying dormant until a
red arrives and the diagnosis turns out to have never been able to speak.

An earlier revision of this body claimed the path could not be exercised without
a failing shard. That was wrong, and the control costs one line.

What this does not do

It does not stop the timeout being hit. That is the eight slowest tests, seven of
them sync start, at 30% of the suite's runtime — #876. And it does not reduce
the queueing that makes a slow shard slower — #877.

This is the third of the three, and the only one about what a person sees when it
goes wrong.

The matrix roll-up is one word for twelve jobs, and the word it uses for a
shard that ran out of timeout-minutes is 'cancelled'.

Two pull requests from an outside contributor sat red for a day on exactly
that. Every ubuntu shard, every Windows leg and three of four macOS shards
were green; the fourth had been stopped at exactly 25 minutes. The only thing
this job said was 'the bats shards did not all pass (result: cancelled)',
which names no shard and does not connect 'cancelled' to a runner limit. From
the outside it reads as the repository rejecting the change.

The roll-up still decides the exit code. What is added is the listing that
makes the exit code actionable, and a sentence for the cancelled case saying
it is a limit in this repository's CI rather than a defect in the change.

The listing is best-effort by construction: if the API call fails, the
roll-up's own message stands. It needs actions:read beyond the workflow's
contents:read, granted at the job, or that call is a 403 the step swallows and
the listing becomes a feature that reads as present and never runs.
Two corrections from review.

The emitted text asserted three things this code does not establish: that the
cap had been reached, that the log ended mid-suite, and that the change under
test was innocent. The query observes a conclusion and a job name, and nothing
else. The last claim is worse than vague — a change that hangs drives the shard
into the same 25-minute cancel, so the message would have told an outside
contributor the opposite of what is known, in exactly the case where it matters.

It now names the job and hands over the discriminator: open it, and if it ran
about 25 minutes and ends mid-suite naming no failing case, the cap was
reached — while saying plainly that the cap being reached does not by itself
say whose fault it is.

Second, the listing was said to be untestable without a failing shard. It is
not. It now runs on every outcome and prints how many matrix jobs it saw, so a
permissions or API problem surfaces on a green run as a warning rather than
waiting for a red to discover the diagnosis was never able to speak.

Also moved the scratch files to $RUNNER_TEMP.
Review found the query selecting a different set from the one it reports on.

`needs.bats-shard.result` rolls up 8 jobs: ubuntu and macos, four shards each.
The predicate was `startswith("bats (")`, which also matches three separately
gated Windows legs -- 11 jobs against the roll-up's 8. On a run where both the
matrix and a Windows leg were red, the diagnosis could have named the Windows
failure as though it explained the roll-up.

Match the matrix shape instead, and compare the count against SHARD_TOTAL times
the matrix's OS list. A mismatch suppresses the listing with a warning rather
than describing some other set of jobs: the control is the comparison, not the
fact that rows came back.

The control added in the previous commit is what exposed this. It printed 11 on
a green run, where the roll-up covers 8.
@fujibee
fujibee force-pushed the ci/name-the-shard-that-stopped branch from 90b2690 to 4871ec1 Compare August 19, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant