Skip to content

Tight per-test timeouts flake on the Windows leg under thread contention #360

Description

@vyncint

Todaystress.yml turns up a recurring failure on the Windows leg
whose shape is always the same: a test with a sub-second builder timeout
times out with an empty screen, because the child had not written
anything yet. It is not the same test twice, which is what makes it a class
rather than a bug in any one of them.

Two measurements, on different branches and different thread counts:

date run shard tests iteration
2026-09-10 34527017048 windows-latest, 8 threads wait_idle_for_overrides_the_builder_default
2026-09-16 35101695488 windows-latest, 16 threads unanswerable_queries_turn_timeouts_into_diagnoses, the_responder_can_be_disabled_and_says_what_went_unanswered, all_unanswered_queries_are_named, wait_frame_timeouts_carry_the_query_note 21/25

The 2026-09-16 diagnostic, which is the whole story in four lines:

query not named in: timed out after 500ms while waiting for the screen predicate to hold
--- screen at timeout ---
size: 80x24  cursor: 0,0
                          ← nothing on the grid at all

The four queries.rs tests assert that a timeout names the query that
went unanswered
. They cannot: the child never emitted the --csi 14t or
--csi 6n within the 500ms the builder was given, so there was no query to
name. The assertion is about the diagnosis; what failed is process startup.

Why it is worth fixing — this is the failure mode that teaches people to
ignore a red stress run, and stress.yml is the instrument this repository
relies on for exactly the bugs it is best at finding. A shard that goes red
for contention rather than for a defect costs a re-run every time and
eventually costs the run its credibility.

It is also the only remaining stress failure not attributable to a test
added on 2026-09-16 (those two are fixed in #359, one of them proven
deterministically).

Fix — the honest options, roughly in order of preference:

  1. Give the Windows leg a spawn-latency floor: the 500ms in these tests
    is chosen to keep the suite fast, and ConPTY's process startup under
    16-way contention plainly does not fit in it. A platform-aware minimum
    (cfg!(windows) → a larger floor) keeps the test's meaning — the
    timeout still has to name the query — while not asserting a schedule.
  2. Wait for the child to be alive and drawing before starting the clock,
    so the deadline covers the behaviour under test and not the spawn.
  3. Failing both: #[cfg_attr(windows, ignore = …)] with the measurement
    written into the reason, which is what this repository does elsewhere for
    ConPTY divergences — but that is the weakest option, since the behaviour
    itself is not Windows-specific and would stop being tested there.

Option 1 or 2 keeps the assertion. Option 3 gives it up; prefer it only if
the first two prove not to hold.

Done when

  • stress.yml is green on windows-latest at 16 threads across a full
    25-iteration run, more than once.
  • Whatever floor or wait is introduced is justified by a measured
    number in a comment, not a guessed one.
  • No test stops asserting that a timeout names the query it was blocked
    on.

Found while clearing the 2026-09-16 issue backlog; filed separately rather
than folded into #359, which fixes only the two flakes that PR's own author
introduced.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions