Skip to content

Group tests into per-file suites, not one import-path bucket - #1

Merged
mxschmitt merged 1 commit into
mainfrom
file-suites-by-source-file
Jun 25, 2026
Merged

mxschmitt merged 1 commit into
mainfrom
file-suites-by-source-file

Conversation

@mxschmitt

@mxschmitt mxschmitt commented Jun 25, 2026 •

Copy link
Copy Markdown
Owner

Problem

The flakiness.io maintainer reported:

your tests belong to the top-level "github.com/playwright-community/playwright-go/tests" suite

We emitted one type:"file" suite per Go package, titled by the import path and with no location. But the spec defines type:"file" as "Suite representing a test file" (with a file location; file suites use (0,0)). So every test in a package collapsed under one long import-path bucket, instead of being grouped by their actual source file the way the Playwright / JUnit reporters group.

Fix

Top-level suites are now per source file:

  • Each test's top-level function (TestXxx/Example/Benchmark/Fuzz) is located via go/ast.
  • Tests sharing a file are grouped under a file suite titled by the git-relative path, with location:{file, 0, 0}.
  • Tests whose file can't be resolved fall back to a package-titled file suite, so nothing is lost when source resolution is unavailable.

Subtest nesting (TestX/a/b → nested suite nodes) and parent-test-own-failure preservation are unchanged.

Before / After

Same fixture (testdata/example/sample_test.go):

Before — one import-path bucket, no location:

[file] example.test/sample (no loc)
  - TestPasses
  - TestFails
  - TestSkips
  [suite] TestWithSubtests @ sample_test.go
    - alpha
    - beta
    - TestWithSubtests

After — a file suite titled by the source file, with a location:

[file] sample_test.go @ sample_test.go
  - TestPasses
  - TestFails
  - TestSkips
  [suite] TestWithSubtests @ sample_test.go
    - alpha
    - beta
    - TestWithSubtests

And tests split across files now produce separate file suites (verified with an added extra_test.go fixture):

[file] extra_test.go
[file] sample_test.go

Tests

  • TestConvert_FileSuites — two top-level tests in different files → two file suites, each titled by + located at its path, location (0,0).
  • TestConvert_FileSuiteFallbackToPackage — no locator → single package-titled file suite (nothing lost).
  • e2e test updated to walk all file suites and assert the sample_test.go file suite carries a location.
  • features.md #5 updated.

Full suite green (go test -race ./...), go vet clean.

Flakiness.io viewer — before / after

The same fixture (testdata/example), served to the real Flakiness.io report viewer via flakiness show. Only the test grouping differs.

Before (main) — every test sits under one import-path suite, example.test/sample › …:

Before: tests grouped under the example.test/sample import-path suite

After (this PR) — tests are grouped by their source file, sample_test.go › … (and a separate extra_test.go suite), each with a file location:

After: tests grouped under per-file suites sample_test.go and extra_test.go

The flakiness.io maintainer reported that "your tests belong to the top-level
'github.com/.../tests' suite" — because we emitted one type:"file" suite per Go
PACKAGE, titled by the import path and with no location. The spec defines
type:"file" as "Suite representing a test file" (with a file location, (0,0)),
so every test in a package collapsed under one import-path bucket instead of
being grouped by their actual source file the way Playwright/JUnit reports are.

Now top-level suites are per source FILE: each test's top-level function is
located via go/ast, tests sharing a file are grouped under a file suite titled
by the git-relative path with location {file,0,0}. Tests whose file can't be
resolved fall back to a package-titled file suite so nothing is lost.

Added fixture file (testdata/example/extra_test.go) + unit tests for multi-file
grouping and the package fallback; updated the e2e test to walk all file
suites. features.md #5 updated.
@mxschmitt
mxschmitt merged commit 2632918 into main Jun 25, 2026
10 checks passed
@mxschmitt
mxschmitt deleted the file-suites-by-source-file branch June 25, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant