Skip to content

a second wall-clock flaky gate: the coverage-barrier tests miss their deadline under full-suite load #707

Description

@0xLeif

Found incidentally while running change check for an unrelated documentation-only change — nothing in that change can reach this code.

commands::gradle_post_discovery_symlink_swap_is_inconclusive_for_every_coverage_gate
  tests/integration/commands.rs:1662
  "check did not reach the root-retained coverage barrier"

Re-run alone: passes in 1.72s. Failed only inside the full 405-test suite on a loaded machine.

Mechanism

The test spawns ten subprocesses and waits a bounded number of wall-clock seconds each for a barrier file to appear. Under the full suite — which runs tests in parallel, and today also alongside cargo builds and several agent worktrees — a spawn can miss its deadline. Nothing about the assertion is wrong; the deadline assumes the machine is otherwise idle.

Same class as #702, and that matters

#702 fixed three staleness_unmeasurable tests that lost a race on .git removal. This is the same failure family: a gate whose outcome depends on host load rather than on the behaviour it asserts.

The cost is not the red run, it is what a red run then means. A suite that fails for reasons unrelated to the change teaches everyone to re-run without reading — and this repository has spent the release chasing signals that stopped carrying information while still looking like signals.

What it needs

Not a longer timeout — that moves the threshold without removing the dependence on load. Options worth weighing:

  1. Wait on the event, not the clock. If the barrier is a file, watch for it (the repo already depends on notify) rather than polling to a deadline.
  2. Serialize this test group so ten concurrent spawns do not compete with the rest of the suite.
  3. If the deadline is genuinely part of what is being asserted, say so in the test and make the failure message distinguish "the barrier was never reached" from "the barrier was not reached in time" — today one message covers both, which is why this reads as a real failure rather than a missed deadline.

Option 3 is worth doing regardless of which of the others lands: a message that cannot distinguish a violated invariant from a slow machine is the same defect shape in the diagnostic itself.

Related: #702.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciRemote or local CI gate/workflow mismatcharea:test-coverageTests, verification, coverage, or lint coverage gapimpact:developer-experienceAffects authors, maintainers, or local workflowsimpact:maintainerPrimarily affects CI, release, or repository operatorspriority:p2Important but not immediately release-blockingscope:specsync-6Applies to current SpecSync 6 behavior or release contract

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions