Skip to content

Add native --rerun-failed[=N] to rerun only failed tests (closes #2) - #3

Open
mxschmitt wants to merge 1 commit into
mainfrom
feat/rerun-failed
Open

mxschmitt wants to merge 1 commit into
mainfrom
feat/rerun-failed

Conversation

@mxschmitt

Copy link
Copy Markdown
Owner

What

Implements Option B from #2: a native --rerun-failed[=N] flag that brings upstream-Playwright retry semantics to go test — stay green on a transient flake, but still surface it as flaky on Flakiness.io.

After the first go test -json pass, flakiness-go re-invokes go test on only the still-failing tests (scoped via an anchored -run regex + -count=1), up to N more times. Every round feeds the same converter, so each rerun becomes an extra RunAttempt — a fail-then-pass is recorded as mixed attempts (flaky) while the job stays green.

flakiness-go --rerun-failed -timeout 15m --race ./...   # bare flag → 2 reruns
flakiness-go --rerun-failed=3 ./...                     # up to 3 reruns
Outcome Exit code
A failed test passes on some attempt 0 (green-on-transient)
A test fails every attempt non-zero (real regression)
Build/setup failure, panic/timeout, benchmark, usage error non-zero (never masked)

Design

  • Per-test gating. The verdict is keyed by full test name: the job goes green only once every test that failed in round 0 has passed on some attempt. Rerunning a parent func re-executes already-passing sibling subtests as collateral — those outcomes are explicitly not allowed to gate the job.
  • Safeguards (borrowed from gotestsum): --rerun-failed-max-failures (default 10 — skip reruns on broad breakage) and --rerun-failed-abort-on-data-race (default true).
  • Wrapper-mode only. Warns (no-op) under --stdin. Flags also resolve from FLAKINESS_RERUN_FAILED[_MAX_FAILURES|_ABORT_ON_DATA_RACE].
  • Rerun flags are inserted before any -args/-- separator so they reach go test, not the test binary.

Tests

  • Scripted-runner unit tests (synthetic event streams) cover: fail-then-pass green, fails-every-attempt red, clean-pass no-rerun, early-stop on recovery, max-failures skip + =0 unlimited, data-race abort (and =false no-abort), build-fail not retried, collateral passing sibling doesn't gate, failing subtest not masked by passing sibling, unexplained rerun failure not masked, -args separator handling, space-separated-value rejection, stdin warning.
  • Real end-to-end test (testdata/flaky) drives actual go test -json through the rerun loop: fail-then-pass → exit 0 + [failed, passed]; stable test never rerun.
  • go test -race -count=1 ./..., go vet ./..., and gofmt all clean.

🤖 Generated with Claude Code

Brings upstream-Playwright retry semantics to `go test`: stay green on a
transient flake, but still surface it as flaky on Flakiness.io. After the
first pass, flakiness-go re-invokes `go test` on only the still-failing
tests (scoped via an anchored `-run` regex + `-count=1`), up to N more
times, feeding every round into the same converter so each rerun is an
extra RunAttempt.

The job's verdict is per-test: it goes green only once every test that
failed in round 0 has passed on some attempt; a test that fails every
attempt, a build/setup failure, a benchmark, or an unexplained non-zero
rerun exit still fails the job. Borrows gotestsum's safeguards:
--rerun-failed-max-failures (default 10) and
--rerun-failed-abort-on-data-race (default true).

Wrapper-mode only; warns under --stdin. Flags also resolve from
FLAKINESS_RERUN_FAILED[_MAX_FAILURES|_ABORT_ON_DATA_RACE].
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant