From 2dc2b0ba0fe49421774f0645b544e037af5eb329 Mon Sep 17 00:00:00 2001 From: Peter D Bethke Date: Wed, 18 Nov 2026 17:30:05 -0500 Subject: [PATCH 1/3] List PHPUnit's test files with --list-tests and have PHP name each test class's file, because --list-test-files exists only from PHPUnit 11 and a project on 9 or 10 (ramsey/uuid among them) got no unit evidence at all; and print a failed listing's output once, not three times. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 12 ++ cmd/corral/certify_repo.go | 13 +- cmd/corral/certify_repo_unit_evidence_test.go | 9 +- cmd/corral/certify_repo_units.go | 4 +- docs/corral/language-support.md | 2 +- internal/lang/runner.go | 15 ++- internal/lang/runner_phpunit.go | 125 +++++++++++++++--- internal/lang/runner_real_test.go | 16 ++- internal/lang/runner_units.go | 19 ++- internal/lang/runner_units_test.go | 49 ++++++- .../testdata/runners/phpunit-units-group.txt | 7 +- .../lang/testdata/runners/phpunit-units.txt | 9 +- .../lang/testdata/runners/phpunit10-units.txt | 9 ++ .../lang/testdata/runners/phpunit12-units.txt | 9 ++ .../testdata/runners/phpunit9-units-group.txt | 13 ++ .../lang/testdata/runners/phpunit9-units.txt | 9 ++ internal/reposcan/generation.go | 14 +- scripts/capture-runner-fixtures.sh | 29 +++- scripts/listunitscmd/main.go | 37 ++++++ .../src/content/docs/docs/language-support.md | 2 +- 20 files changed, 357 insertions(+), 45 deletions(-) create mode 100644 internal/lang/testdata/runners/phpunit10-units.txt create mode 100644 internal/lang/testdata/runners/phpunit12-units.txt create mode 100644 internal/lang/testdata/runners/phpunit9-units-group.txt create mode 100644 internal/lang/testdata/runners/phpunit9-units.txt create mode 100644 scripts/listunitscmd/main.go diff --git a/CHANGELOG.md b/CHANGELOG.md index dbafde0f..19775f59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,18 @@ still move between minor versions. Entries describe what changed for someone *using* the tool. For the full commit history of any release, `git log v0.3.4..v0.3.5`. +## [Unreleased] + +### Fixed + +- **PHP: a project on PHPUnit 9 or 10 gets test-file evidence.** To learn which test file executes which source file, corral asks the runner which test files your command collects and runs each one alone under coverage. For PHPUnit it asked with `--list-test-files`, an option that exists only from PHPUnit 11. On 9.6 and 10.5 the listing stopped at `Unknown option "--list-test-files"`, so no test file was run alone: a source file was paired with a test by file name only (one with no test of a matching name was not audited at all), and every file was graded by the whole suite. corral now asks with `--list-tests`, which every PHPUnit from 9.6 has, and has PHP itself name the file each listed test class was loaded from, so a test class needs no autoload entry to be found. In a PHPUnit 9.6 project whose only source file has no test of its name, the file went from not audited to paired by evidence with the one test file that runs it. + - *What the listing run sees that yours does not.* For the listing only, `PHP_INI_SCAN_DIR` gains a temporary directory after any value you set, holding one `auto_prepend_file` setting; the file it names prints each test class's file when PHP exits and does nothing in a process that never loaded PHPUnit. The directory is removed when the listing ends. No graded run is changed. + - *Group options on PHPUnit 9 and 10.* Those versions ignore `--group` and `--exclude-group` when listing and say so (`The --group and --list-tests options cannot be combined, --group is ignored`), so the list is of every test and not of the tests your command runs. corral reads that sentence as no listing: such a command gets no test-file evidence on 9 or 10, is graded by the whole suite as before, and the scan says why, with PHPUnit's own output. PHPUnit 11 and 12 apply the group options when listing, as `--list-test-files` did. + - *Where it was run.* PHPUnit 9.6.39, 10.5.67, 11.5.57 and 12.5.44 on PHP 8.5, listing and one-file runs on each; on 9.6.39 also whole scans in the bubblewrap sandbox (`certify --repo --dry-run --evidence`): the project above, and ramsey/uuid at `7a16331`, where the listing named 67 test files, 59 of them reported which source files they execute, and 10 source files became candidates that file names alone had not paired (the other 8 test files failed under coverage for a reason of corral's own, not yet fixed: it loads the pcov extension a second time, PHP warns, and a test PHPUnit runs in a separate process fails on the warning; those 8 are added to every file's selection, so no file is graded by fewer tests than execute it). No older PHPUnit and no older PHP was run, and no audit with models was run on a PHPUnit 9 or 10 project. + - Cached verdicts are measured again once (`VerdictGeneration` 47). + +- **A unit listing that fails prints the runner's output once.** When the listing of test units failed, the scan printed what the runner said on the `units:` line, again on the `selection:` line, and a third time inside the summary line, each copy as many lines long as the runner's output. The `units:` line still prints it in full; the two lines after it now give the reason in one line, the same sentence the record keeps. + ## [v1.0.0-rc.34] — 2026-11-18 ### Fixed diff --git a/cmd/corral/certify_repo.go b/cmd/corral/certify_repo.go index fc8e75f9..dca4e7c7 100644 --- a/cmd/corral/certify_repo.go +++ b/cmd/corral/certify_repo.go @@ -1509,13 +1509,22 @@ func runCertifyRepo(args []string, stdout, stderr io.Writer) int { } selectionNote := "" if ex != nil { - selectionNote = ex.selection.Said() + // The pass's Note, not Said(): the "selection:" or "units:" line + // above already printed what the terminal has to add (a failed + // listing's output, an error), and Said() here printed a runner's + // whole output a second and a third time (PHPUnit 9: its banner + // and "Unknown option" on the units line, the grading line and + // this one). The summary repeats the reason, once, in one line. + selectionNote = ex.selection.Note + if selectionNote == "" { + selectionNote = ex.selection.Said() + } // For a language the unit pass serves, ITS reason is why candidacy // is pairing-only; the selection note ("no selector for …") is // about grading. switch { case ex.units.Note != "": - selectionNote = ex.units.Said() + selectionNote = ex.units.Note case ex.unitLandingNote != "": selectionNote = ex.unitLandingNote case ex.units.Ran && !evidenceCandidacyOK: diff --git a/cmd/corral/certify_repo_unit_evidence_test.go b/cmd/corral/certify_repo_unit_evidence_test.go index 342860a1..6cd48621 100644 --- a/cmd/corral/certify_repo_unit_evidence_test.go +++ b/cmd/corral/certify_repo_unit_evidence_test.go @@ -226,11 +226,14 @@ func TestUnitPassDisclosesACommandWithNoUnitForm(t *testing.T) { root := unitEvidenceTSFixture(t) s := runUnitEvidenceScan(t, root, "--", "npm", "test") - const note = "typescript: `npm test` has no one-unit form corral can list and run, so no unit evidence" - if !strings.Contains(s, "units: "+note) { + const said = "typescript: `npm test` has no one-unit form corral can list and run, so no unit evidence" + const note = "typescript: the unit pass could not list units: its command has no one-unit form corral can list and run" + if !strings.Contains(s, "units: "+said) { t.Errorf("missing the units line with the pass's own note:\n%s", s) } - if !strings.Contains(s, "pairing-only candidacy (no selection evidence: "+note+")") { + // The summary gives the reason in the one line the record keeps + // (UnitEvidence.Note); what the terminal adds is on the units line, once. + if !strings.Contains(s, "pairing-only candidacy (no selection evidence: "+note+")") || strings.Count(s, said) != 1 { t.Errorf("the summary must say candidacy is pairing-only, and why:\n%s", s) } if !strings.Contains(s, "evidence-paired 0") || strings.Contains(s, "paired by evidence") { diff --git a/cmd/corral/certify_repo_units.go b/cmd/corral/certify_repo_units.go index 96fc84db..abba2bc7 100644 --- a/cmd/corral/certify_repo_units.go +++ b/cmd/corral/certify_repo_units.go @@ -438,7 +438,9 @@ func printUnitGrading(w io.Writer, ex *localExecutor, langName string) { } fmt.Fprintln(w, line+")") case !ev.Ran: - fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", ex.unitsNotRunSaid(langName)) + // The reason alone: the units line above printed what the pass + // said in full (printUnitPass), a failed listing's output included. + fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", ex.unitsNotRunReason(langName)) case reposcan.NoRunnerRow(ex.unitMap) != "": fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", reposcan.NoRunnerRow(ex.unitMap)) default: diff --git a/docs/corral/language-support.md b/docs/corral/language-support.md index aaa27fe0..4f71f136 100644 --- a/docs/corral/language-support.md +++ b/docs/corral/language-support.md @@ -60,6 +60,6 @@ A language has a runner capability when any of its runners does; this is which. | `node --test` (default) | `node-test` | javascript, typescript | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | node has no collect-only mode, so listing RUNS the test file once on the unmutated code; verified on node 22.23 and 24.22: listing, one-test, one-file and several-file runs, and salvage, on both (24.22.2 measured 2026-11-10). Through a package script, listing, one-test runs and salvage need `--test` to be the script's last word, since node reads options only before the first file | | `rspec` (default) | `rspec` | ruby | `--fail-fast` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on rspec-core 3.13 (one-test runs need 3.12 or later) | | `minitest` | `minitest` | ruby | | | | | | | ✓ | ✓ | | minitest has no list mode and no full-name selector; rspec is the Ruby runner that lists and runs one test; no unit evidence: no listing command; Ruby falls back to name pairing under minitest | -| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs (`--list-test-files`) verified on PHPUnit 11.5; other versions were not run | +| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on PHPUnit 11.5; the test-file listing (`--list-tests`, with PHP naming each test class's file) also on 9.6, 10.5 and 12.5, where one-file runs were run as well; PHPUnit 9 and 10 ignore `--group` and `--exclude-group` when listing, so a command carrying one gets no test-file listing there | | `cargo nextest` | `cargo-nextest` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | one-binary and several-binary runs (one `-E 'binary_id(=...)'` each) and their `list` listing verified on cargo-nextest 0.9.148; a command with its own filterset or a partition has no unit form | | `cargo test` (default) | `cargo-test` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | libtest has no retries; flaky tests are named when the output is nextest's (a wrapper that runs nextest under another command); one-target runs (`-p ` with `--lib`, `--test` or `--bin`, in place of the command's own package and target selection) verified on cargo 1.90; a run of several targets runs each named target in every named package, since cargo selects targets across the packages it is given (measured on cargo 1.90); doc tests, examples and benches are no unit | diff --git a/internal/lang/runner.go b/internal/lang/runner.go index 4c50940b..4cb0a0c5 100644 --- a/internal/lang/runner.go +++ b/internal/lang/runner.go @@ -125,8 +125,15 @@ type Runner struct { // second listed unit can match); nil means a unit run is exact. ListUnits func(argv []string) ([]string, bool) ParseUnits func(stdout string) []string - UnitCmd func(argv []string, unit string) ([]string, bool) - UnitCoruns func(unit, other string) bool + // ListUnitsSetup is shell the listing runs before its command, for a + // runner whose listing needs something beside it (PHPUnit: a file PHP + // loads into the listing). It may write only under "$d", the listing's + // own temporary directory, removed when the listing ends, and export + // what the command reads. "" on every row whose listing is the command + // alone. + ListUnitsSetup string + UnitCmd func(argv []string, unit string) ([]string, bool) + UnitCoruns func(unit, other string) bool // CollectionBase is argv with the operator's collection targets removed, // so appending a chosen set NARROWS the run instead of joining a // directory argv already names (pytest `tests/` plus a node id still @@ -391,9 +398,9 @@ var runners = []Runner{ Name: "phpunit", ID: "phpunit", Languages: []string{"php"}, Default: true, Words: []string{"phpunit"}, FailFast: []string{"--stop-on-failure"}, FirstFailure: phpFirstFailure, FailedTests: phpFailures, DeselectCmd: phpunitDeselectCmd, ListCmd: phpunitListCmd, ParseList: phpunitParseList, SingleCmd: phpunitSingleCmd, ResolveByList: true, - ListUnits: phpunitListUnits, ParseUnits: phpunitParseUnits, UnitCmd: oneUnit(phpunitUnitsCmd), + ListUnits: phpunitListUnits, ParseUnits: phpunitParseUnits, ListUnitsSetup: phpunitUnitsSetup, UnitCmd: oneUnit(phpunitUnitsCmd), UnitsCmd: phpunitUnitsCmd, UnitForFile: fileUnitForFile, UnitNoun: "test files", - Note: "listing, one-test, one-file and several-file runs (`--list-test-files`) verified on PHPUnit 11.5; other versions were not run", + Note: "listing, one-test, one-file and several-file runs verified on PHPUnit 11.5; the test-file listing (`--list-tests`, with PHP naming each test class's file) also on 9.6, 10.5 and 12.5, where one-file runs were run as well; PHPUnit 9 and 10 ignore `--group` and `--exclude-group` when listing, so a command carrying one gets no test-file listing there", }, { // FirstFailure and Flaky read the grammar off the output, not the diff --git a/internal/lang/runner_phpunit.go b/internal/lang/runner_phpunit.go index 278d133a..a96cc566 100644 --- a/internal/lang/runner_phpunit.go +++ b/internal/lang/runner_phpunit.go @@ -3,6 +3,7 @@ package lang import ( + "regexp" "slices" "strings" ) @@ -58,35 +59,127 @@ func phpunitSingleCmd(argv []string, testPath, id string) ([]string, bool) { return append(slices.Clone(argv), "--filter", filter, testPath), true } -// phpunitListUnits is PHPUnit's own `--list-test-files` over argv: the test -// files the command collects, after its configuration's suites, the -// operator's arguments and every group or filter option (measured on -// PHPUnit 11.5.57: under --group slow only the file with a slow test was -// listed, and under --exclude-group slow a file whose only test is slow was -// not). +// phpunitListUnits is PHPUnit's own `--list-tests` over argv, run under +// phpunitUnitsSetup: the tests the command collects, after its +// configuration's suites and the operator's arguments, and the file PHP +// loaded each test class from. The same listing serves every PHPUnit from +// 9.6 to 12.5 (measured on 9.6.39, 10.5.67, 11.5.57 and 12.5.44). +// `--list-test-files`, which this row used before, exists only from PHPUnit +// 11 (9.6 and 10.5: `Unknown option "--list-test-files"`), so a project on +// an older PHPUnit had no unit listing at all. func phpunitListUnits(argv []string) ([]string, bool) { if _, _, ok := phpunitUnitArgs(argv); !ok { return nil, false } - return append(slices.Clone(argv), "--list-test-files"), true + return append(slices.Clone(argv), "--list-tests"), true } -// phpunitParseUnits reads the ` - ` lines under `Available test -// files:`; with no such heading, PHPUnit listed nothing (an error, another -// mode), which is no listing. +// phpunitUnitClassMarker leads each line phpunitUnitsPrepend prints: a test +// class and the file it was loaded from, tab-separated. +const phpunitUnitClassMarker = "corral-unit-class " + +// phpunitUnitsPrepend is loaded by auto_prepend_file into the listing run. +// PHPUnit's `--list-tests` names classes and not files, and only PHP knows +// which file a class came from: a test class need not be autoloadable, since +// PHPUnit finds test files by walking the configured directories, so no +// reading of composer.json's PSR-4 map could name the file of every class. +// At shutdown, after PHPUnit has printed its list, this prints the file of +// every concrete TestCase subclass the process loaded. phpunitParseUnits +// keeps only the classes PHPUnit listed, so a test class that was loaded +// and not collected adds no file. +// +// GUARD FIRST, as phpCoveragePrepend does: the file is loaded into every PHP +// process the listing starts (`composer test` is one before phpunit is), and +// a process that never loaded PHPUnit prints nothing. +const phpunitUnitsPrepend = `isAbstract() || $r->getFileName() === false) { + continue; + } + fwrite(STDOUT, "` + phpunitUnitClassMarker + `" . $class . "\t" . $r->getFileName() . "\n"); + } +}); +` + +// phpunitUnitsSetup is the listing's Runner.ListUnitsSetup: it writes +// phpunitUnitsPrepend into the listing's temporary directory and adds that +// directory to PHP's ini scan path. PHP_INI_SCAN_DIR and not `php -d`, for +// CoverageCmd's reason: `vendor/bin/phpunit` and `composer test` take no +// `-d`. The operator's own PHP_INI_SCAN_DIR is kept in front, and with none +// set the value opens with a colon, which PHP reads as its default scan +// directory. +const phpunitUnitsSetup = `cat > "$d/corral_units.php" <<'CORRAL_PHP_EOF'` + "\n" + phpunitUnitsPrepend + "CORRAL_PHP_EOF\n" + + `printf 'auto_prepend_file=%s\n' "$d/corral_units.php" > "$d/zz-corral-units.ini" || exit 1` + "\n" + + `export PHP_INI_SCAN_DIR="${PHP_INI_SCAN_DIR-}:$d"` + +// phpunitListHeading is the line above `--list-tests`' list: "Available +// test(s):" on PHPUnit 9 and 10, "Available tests:" or "Available test:" on +// 11 and 12. An empty list still prints it (measured on 9.6 and 11.5). +var phpunitListHeading = regexp.MustCompile(`^Available tests?(\(s\))?:$`) + +// phpunitListIgnoredOption is what PHPUnit 9 and 10 print when the command +// carries a group option beside `--list-tests`: "The --group and +// --list-tests options cannot be combined, --group is ignored". The list +// under it is of every test, not of the tests the command runs. +var phpunitListIgnoredOption = regexp.MustCompile(`^The --\S+ and --list-tests options cannot be combined`) + +// phpunitParseUnits reads phpunitListUnits' output into the files of the +// classes PHPUnit listed, in list order. It is no listing (nil) with no +// heading (an error, another mode), when PHPUnit says it ignored an option +// of the command (the list is then not the command's own: PHPUnit 9 and 10 +// under --group or --exclude-group, where 11 and 12 apply them), and when a +// listed class has no file line, since a listing that cannot name every +// unit is not trusted to name any. func phpunitParseUnits(stdout string) []string { - var files []string + fileOf := map[string]string{} + var classes []string + seen := map[string]bool{} + listed := false for _, line := range strings.Split(stdout, "\n") { line = strings.TrimRight(line, "\r") - if files == nil { - if line == "Available test files:" { - files = []string{} + if phpunitListIgnoredOption.MatchString(line) { + return nil + } + if rest, ok := strings.CutPrefix(line, phpunitUnitClassMarker); ok { + if class, file, ok := strings.Cut(rest, "\t"); ok && class != "" && file != "" { + fileOf[class] = file } continue } - if f, ok := strings.CutPrefix(line, " - "); ok && f != "" { - files = append(files, f) + if phpunitListHeading.MatchString(line) { + listed = true + continue + } + if !listed { + continue + } + fq, ok := strings.CutPrefix(line, " - ") + if !ok { + continue + } + if class, _, ok := strings.Cut(fq, "::"); ok && class != "" && !seen[class] { + seen[class] = true + classes = append(classes, class) + } + } + if !listed { + return nil + } + files := []string{} + for _, class := range classes { + file, ok := fileOf[class] + if !ok { + return nil } + files = append(files, file) } return files } diff --git a/internal/lang/runner_real_test.go b/internal/lang/runner_real_test.go index 8b1ceeae..704924d0 100644 --- a/internal/lang/runner_real_test.go +++ b/internal/lang/runner_real_test.go @@ -423,6 +423,20 @@ func TestRealRunnersListAndRunOneUnitBeyondJS(t *testing.T) { []string{"vendor/bin/phpunit", "--version"}, []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, map[string]int{"units/phpunit/tests/ATest.php": 2, "units/phpunit/tests/BTest.php": 1}, ran, true}, + // The PHPUnit lines the capture script installs beside the fixtures + // (pv/): the same listing and one-file runs on each. + {"phpunit", "php", "", []string{"pv/9/vendor/bin/phpunit", "-c", "units/phpunit/phpunit.xml", "--testsuite", "unit"}, + []string{"pv/9/vendor/bin/phpunit", "--version"}, + []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + map[string]int{"units/phpunit/tests/ATest.php": 2, "units/phpunit/tests/BTest.php": 1}, ran, true}, + {"phpunit", "php", "", []string{"pv/10/vendor/bin/phpunit", "-c", "units/phpunit/phpunit.xml", "--testsuite", "unit"}, + []string{"pv/10/vendor/bin/phpunit", "--version"}, + []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + map[string]int{"units/phpunit/tests/ATest.php": 2, "units/phpunit/tests/BTest.php": 1}, ran, true}, + {"phpunit", "php", "", []string{"pv/12/vendor/bin/phpunit", "-c", "units/phpunit/phpunit.xml", "--testsuite", "unit"}, + []string{"pv/12/vendor/bin/phpunit", "--version"}, + []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + map[string]int{"units/phpunit/tests/ATest.php": 2, "units/phpunit/tests/BTest.php": 1}, ran, true}, // Two packages each with a lib and a tests/it.rs: every unit, the // shared names included, runs its own package's target alone. {"cargo test", "rust", "units/cargo", []string{"cargo", "test", "--workspace"}, []string{"cargo", "--version"}, @@ -432,7 +446,7 @@ func TestRealRunnersListAndRunOneUnitBeyondJS(t *testing.T) { []string{"units-a", "units-a::bin/units-a", "units-a::it", "units-b", "units-b::it"}, map[string]int{"units-a": 2, "units-b::it": 1}, nextestCount, true}, } { - t.Run(c.runner, func(t *testing.T) { + t.Run(c.runner+"/"+c.argv[0], func(t *testing.T) { dir := root if !c.needsRoot { dir = t.TempDir() diff --git a/internal/lang/runner_units.go b/internal/lang/runner_units.go index 1b2df48c..4f0d4436 100644 --- a/internal/lang/runner_units.go +++ b/internal/lang/runner_units.go @@ -31,6 +31,19 @@ const unitsCwdMarker = "corral-units-cwd " // re-read by the shell. const unitsPrelude = `printf 'corral-units-cwd %s\n' "$(pwd -P)" && exec "$@"` +// unitsSetupPrelude is unitsPrelude for a row with a ListUnitsSetup: the +// marker line, a temporary directory "$d" removed when the shell exits, the +// row's setup, then the listing command, whose exit status is the shell's. +// The command is run and not exec'd, so the shell outlives it to remove the +// directory. +func unitsSetupPrelude(setup string) string { + return `printf 'corral-units-cwd %s\n' "$(pwd -P)" || exit 1` + "\n" + + `d=$(mktemp -d) || exit 1` + "\n" + + `trap 'rm -rf "$d"' EXIT` + "\n" + + setup + "\n" + + `"$@"` +} + // ListUnitsCmd is the command that lists the units argv collects, from its // runner, under the working-directory prelude ParseUnitList reads. func ListUnitsCmd(p Plugin, inv Invocation) ([]string, bool) { @@ -42,7 +55,11 @@ func ListUnitsCmd(p Plugin, inv Invocation) ([]string, bool) { if !ok { return nil, false } - return append([]string{"sh", "-c", unitsPrelude, "corral-units"}, cmd...), true + prelude := unitsPrelude + if r.ListUnitsSetup != "" { + prelude = unitsSetupPrelude(r.ListUnitsSetup) + } + return append([]string{"sh", "-c", prelude, "corral-units"}, cmd...), true } // ParseUnitList reads ListUnitsCmd's stdout into repository-relative unit diff --git a/internal/lang/runner_units_test.go b/internal/lang/runner_units_test.go index e9ee6048..00f17d13 100644 --- a/internal/lang/runner_units_test.go +++ b/internal/lang/runner_units_test.go @@ -520,10 +520,13 @@ func TestMinitestHasNoUnitForm(t *testing.T) { } } -// PHPUnit's units are its own --list-test-files, which applies the -// configuration, --testsuite and the group options (captured: under --group -// slow only ATest is listed). One unit replaces the operator's file -// arguments, since PHPUnit runs them all (capture phpunit-args-union). +// PHPUnit's units are the files of the classes its own --list-tests names, +// each file told by PHP itself in the listing run (phpunitUnitsSetup). The +// listing applies the configuration and --testsuite on every PHPUnit from +// 9.6 to 12.5 (captured on each), and the group options on 11 and 12 +// (captured: under --group slow only ATest is listed, though PHP also +// loaded BTest and CTest). One unit replaces the operator's file arguments, +// since PHPUnit runs them all (capture phpunit-args-union). func TestPHPUnitUnitsFromListing(t *testing.T) { conf := []string{"vendor/bin/phpunit", "-c", "units/phpunit/phpunit.xml"} for _, c := range []unitCase{ @@ -533,10 +536,48 @@ func TestPHPUnitUnitsFromListing(t *testing.T) { {row: "phpunit", lang: "php", fixture: "phpunit-units-group.txt", argv: append(slices.Clone(conf), "--group", "slow", "units/phpunit"), want: []string{"units/phpunit/tests/ATest.php"}, wantRun: append(slices.Clone(conf), "--group", "slow", "units/phpunit/tests/ATest.php")}, + {row: "phpunit", lang: "php", fixture: "phpunit9-units.txt", argv: append(slices.Clone(conf), "--testsuite", "unit"), + want: []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + wantRun: append(slices.Clone(conf), "--testsuite", "unit", "units/phpunit/tests/ATest.php")}, + {row: "phpunit", lang: "php", fixture: "phpunit10-units.txt", argv: append(slices.Clone(conf), "--testsuite", "unit"), + want: []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + wantRun: append(slices.Clone(conf), "--testsuite", "unit", "units/phpunit/tests/ATest.php")}, + {row: "phpunit", lang: "php", fixture: "phpunit12-units.txt", argv: append(slices.Clone(conf), "--testsuite", "unit"), + want: []string{"units/phpunit/tests/ATest.php", "units/phpunit/tests/BTest.php"}, + wantRun: append(slices.Clone(conf), "--testsuite", "unit", "units/phpunit/tests/ATest.php")}, } { c.check(t) } php, _ := ByName("php") + // The listing is PHPUnit's --list-tests under the row's setup, which + // every version from 9.6 takes; --list-test-files is PHPUnit 11's. + if got, ok := listCmd(t, "php", conf); !ok || !slices.Equal(got, append(slices.Clone(conf), "--list-tests")) { + t.Errorf("listing command = %q, %v", got, ok) + } + if cmd, _ := ListUnitsCmd(php, Cmd(conf)); len(cmd) < 3 || !strings.Contains(cmd[2], phpunitUnitsPrepend) || !strings.Contains(cmd[2], `trap 'rm -rf "$d"' EXIT`) { + t.Errorf("the PHPUnit listing does not run under its setup: %q", cmd) + } + // PHPUnit 9 and 10 ignore a group option beside --list-tests and say + // so; the list is then every test, so it is no listing of the command + // (captured on 9.6: all four tests under --group slow). + groupInv := Cmd(append(slices.Clone(conf), "--group", "slow")) + if got := ParseUnitList(php, groupInv, unitFixture(t, "phpunit9-units-group.txt")); got != nil { + t.Errorf("PHPUnit 9 under --group = %q, want no listing: it ignored the option", got) + } + // A listed class PHP named no file for is no listing: every unit or none. + whole := unitFixture(t, "phpunit9-units.txt") + cut := strings.Replace(whole, phpunitUnitClassMarker+"BTest\t/fixture/units/phpunit/tests/BTest.php\n", "", 1) + if cut == whole { + t.Fatal("the fixture has no file line for BTest to remove") + } + if got := ParseUnitList(php, Cmd(conf), cut); got != nil { + t.Errorf("a listed class with no file line = %q, want no listing", got) + } + // An empty list under its heading is a listing of no units (captured: + // PHPUnit prints the heading alone). + if got := ParseUnitList(php, Cmd(conf), unitsCwdMarker+"/r\nPHPUnit 9.6.39 by Sebastian Bergmann and contributors.\n\nAvailable test(s):\n"); got == nil || len(got) != 0 { + t.Errorf("an empty list = %#v, want an empty listing", got) + } if got, ok := UnitRunCmd(php, Cmd([]string{"php", "-d", "memory_limit=1G", "vendor/bin/phpunit", "tests", "-dxdebug.mode=off", "--colors=never"}), "tests/ATest.php"); !ok || !slices.Equal(got, []string{"php", "-d", "memory_limit=1G", "vendor/bin/phpunit", "-dxdebug.mode=off", "--colors=never", "tests/ATest.php"}) { t.Errorf("UnitRunCmd = %q, %v", got, ok) diff --git a/internal/lang/testdata/runners/phpunit-units-group.txt b/internal/lang/testdata/runners/phpunit-units-group.txt index 237ad2dc..8ba30b3e 100644 --- a/internal/lang/testdata/runners/phpunit-units-group.txt +++ b/internal/lang/testdata/runners/phpunit-units-group.txt @@ -1,5 +1,8 @@ corral-units-cwd /fixture PHPUnit 11.5.57 by Sebastian Bergmann and contributors. -Available test files: - - /fixture/units/phpunit/tests/ATest.php +Available test: + - ATest::testTwo +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php +corral-unit-class CTest /fixture/units/phpunit/other/CTest.php diff --git a/internal/lang/testdata/runners/phpunit-units.txt b/internal/lang/testdata/runners/phpunit-units.txt index 5d3ad7d9..0d44aac8 100644 --- a/internal/lang/testdata/runners/phpunit-units.txt +++ b/internal/lang/testdata/runners/phpunit-units.txt @@ -1,6 +1,9 @@ corral-units-cwd /fixture PHPUnit 11.5.57 by Sebastian Bergmann and contributors. -Available test files: - - /fixture/units/phpunit/tests/ATest.php - - /fixture/units/phpunit/tests/BTest.php +Available tests: + - ATest::testOne + - ATest::testTwo + - BTest::testOne +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php diff --git a/internal/lang/testdata/runners/phpunit10-units.txt b/internal/lang/testdata/runners/phpunit10-units.txt new file mode 100644 index 00000000..5e6fc8c2 --- /dev/null +++ b/internal/lang/testdata/runners/phpunit10-units.txt @@ -0,0 +1,9 @@ +corral-units-cwd /fixture +PHPUnit 10.5.67 by Sebastian Bergmann and contributors. + +Available test(s): + - ATest::testOne + - ATest::testTwo + - BTest::testOne +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php diff --git a/internal/lang/testdata/runners/phpunit12-units.txt b/internal/lang/testdata/runners/phpunit12-units.txt new file mode 100644 index 00000000..75df9c0a --- /dev/null +++ b/internal/lang/testdata/runners/phpunit12-units.txt @@ -0,0 +1,9 @@ +corral-units-cwd /fixture +PHPUnit 12.5.44 by Sebastian Bergmann and contributors. + +Available tests: + - ATest::testOne + - ATest::testTwo + - BTest::testOne +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php diff --git a/internal/lang/testdata/runners/phpunit9-units-group.txt b/internal/lang/testdata/runners/phpunit9-units-group.txt new file mode 100644 index 00000000..46af47e9 --- /dev/null +++ b/internal/lang/testdata/runners/phpunit9-units-group.txt @@ -0,0 +1,13 @@ +corral-units-cwd /fixture +PHPUnit 9.6.39 by Sebastian Bergmann and contributors. + +The --group and --list-tests options cannot be combined, --group is ignored + +Available test(s): + - ATest::testOne + - ATest::testTwo + - BTest::testOne + - CTest::testOne +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php +corral-unit-class CTest /fixture/units/phpunit/other/CTest.php diff --git a/internal/lang/testdata/runners/phpunit9-units.txt b/internal/lang/testdata/runners/phpunit9-units.txt new file mode 100644 index 00000000..de737e47 --- /dev/null +++ b/internal/lang/testdata/runners/phpunit9-units.txt @@ -0,0 +1,9 @@ +corral-units-cwd /fixture +PHPUnit 9.6.39 by Sebastian Bergmann and contributors. + +Available test(s): + - ATest::testOne + - ATest::testTwo + - BTest::testOne +corral-unit-class ATest /fixture/units/phpunit/tests/ATest.php +corral-unit-class BTest /fixture/units/phpunit/tests/BTest.php diff --git a/internal/reposcan/generation.go b/internal/reposcan/generation.go index 8520b7e5..dd4ccad9 100644 --- a/internal/reposcan/generation.go +++ b/internal/reposcan/generation.go @@ -462,7 +462,19 @@ package reposcan // authored_together_output is the run's true last 2000 bytes rather than the // end of its first 16 KiB. It rides along: "46" has not left this branch. // Behaviour bump; the shape is unchanged. -const VerdictGeneration = "46" +// +// "47" (2026-11-18): PHPUnit's test-file listing is `--list-tests` with PHP +// naming each test class's file (lang.phpunitListUnits), where it was +// `--list-test-files`, an option PHPUnit 9 and 10 do not have. A "46" scan +// of a project on PHPUnit 9 or 10 had no unit listing: no unit evidence, so +// its files were candidates by name pairing only and every one was graded by +// the whole suite. Now those projects list, so a file can be paired by +// evidence and graded by the test files that execute it, which can change +// which files are audited and, through per-mutant time caps and fail-fast +// order, what a file's verdict holds. PHPUnit 11 and 12 list the same files +// as before (captured on both). "46" was released in rc.34, so this gets its +// own generation. +const VerdictGeneration = "47" // VerdictShapeSHA256 fingerprints advpool.Verdict's serialized shape: every // exported field's name, type and json tag, sorted by name and hashed. diff --git a/scripts/capture-runner-fixtures.sh b/scripts/capture-runner-fixtures.sh index c00cf689..a80fef06 100755 --- a/scripts/capture-runner-fixtures.sh +++ b/scripts/capture-runner-fixtures.sh @@ -30,7 +30,7 @@ DIR="${1:-$(mktemp -d)}" [ "$KEEP" = 1 ] || trap 'rm -rf "$DIR"' EXIT RSPEC="${RSPEC:-rspec}" -JEST=30.5.2 VITEST=5.1.0 MOCHA=12.0.3 PHPUNIT=11.5.57 +JEST=30.5.2 VITEST=5.1.0 MOCHA=12.0.3 PHPUNIT=11.5.57 PHPUNIT9=9.6.39 PHPUNIT10=10.5.67 PHPUNIT12=12.5.44 mkdir -p "$DIR" cp -r "$OUT/src/." "$DIR/" @@ -178,15 +178,34 @@ want rspec-no-profile-reads-no-word 2 "$(cd units/rspec && $RSPEC --no-profile s help rspec-help.txt $RSPEC --help # vendor/bin/phpunit -c units/phpunit/phpunit.xml --testsuite unit / ... --group slow -PHPC="vendor/bin/phpunit -c units/phpunit/phpunit.xml" -units phpunit-units.txt $PHPC --testsuite unit --list-test-files -units phpunit-units-group.txt $PHPC --group slow --list-test-files phpran() { sed -n 's/^OK (\([0-9]*\) tests\{0,1\}.*/\1/p'; } +PHPC="vendor/bin/phpunit -c units/phpunit/phpunit.xml" +# PHPUnit's listing runs under the row's own setup (phpunitUnitsSetup, which +# has PHP name each test class's file), so its script comes from the row and +# is not copied here. +PHPPRELUDE="$(cd "$REPO" && go run ./scripts/listunitscmd php vendor/bin/phpunit)" +phpunits() { # phpunits : units, under the PHPUnit row's listing script + local f="$1"; shift + sh -c "$PHPPRELUDE" corral-units "$@" 2>/dev/null | sed -e "s#$PDIR#/fixture#g" -e "s#$HOME#~#g" > "$OUT/$f" || true +} +phpunits phpunit-units.txt $PHPC --testsuite unit --list-tests +phpunits phpunit-units-group.txt $PHPC --group slow --list-tests +# The same listing on the other PHPUnit lines, each installed beside the +# fixtures. 9 and 10 have no --list-test-files, which is why the row lists +# tests and not files; under a group option both say they ignore it, and +# the row reads that as no listing (phpunit9-units-group.txt). +for line in "9 $PHPUNIT9" "10 $PHPUNIT10" "12 $PHPUNIT12"; do + set -- $line + [ -f "pv/$1/vendor/bin/phpunit" ] || { mkdir -p "pv/$1" && composer require -q --working-dir "pv/$1" --dev "phpunit/phpunit:$2"; } + phpunits "phpunit$1-units.txt" "pv/$1/vendor/bin/phpunit" -c units/phpunit/phpunit.xml --testsuite unit --list-tests + want "phpunit$1-unit" 2 "$("pv/$1/vendor/bin/phpunit" -c units/phpunit/phpunit.xml --testsuite unit units/phpunit/tests/ATest.php 2>&1 | phpran)" +done +phpunits phpunit9-units-group.txt pv/9/vendor/bin/phpunit -c units/phpunit/phpunit.xml --group slow --list-tests want phpunit-unit 2 "$($PHPC --testsuite unit units/phpunit/tests/ATest.php 2>&1 | phpran)" want phpunit-args-union 5 "$($PHPC units/phpunit/tests units/phpunit/tests/ATest.php 2>&1 | phpran)" want phpunit-file-replaces-testsuite 2 "$($PHPC --testsuite other units/phpunit/tests/ATest.php 2>&1 | phpran)" want phpunit-option-after-file 1 "$($PHPC units/phpunit/tests/ATest.php --group slow 2>&1 | phpran)" -want phpunit-exclude-group-lists-no-empty-file 0 "$($PHPC --exclude-group slow --list-test-files units/phpunit/excl units/phpunit/tests 2>&1 | grep -c DTest || true)" +want phpunit-exclude-group-lists-no-empty-file 0 "$($PHPC --exclude-group slow --list-tests units/phpunit/excl units/phpunit/tests 2>&1 | grep -c ' - DTest' || true)" want phpunit-colors-takes-no-next-word 'Test file "never" not found' "$($PHPC --colors never units/phpunit/tests/ATest.php 2>&1 | grep -o 'Test file "never" not found')" help phpunit-help.txt vendor/bin/phpunit --help diff --git a/scripts/listunitscmd/main.go b/scripts/listunitscmd/main.go new file mode 100644 index 00000000..f77a80ac --- /dev/null +++ b/scripts/listunitscmd/main.go @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: Elastic-2.0 + +// Command listunitscmd prints the shell script a runner row's unit listing +// runs under (lang.ListUnitsCmd's `sh -c` script) for a language and a test +// command: `go run ./scripts/listunitscmd php vendor/bin/phpunit`. +// +// scripts/capture-runner-fixtures.sh uses it for PHPUnit, whose listing runs +// under a setup the row owns (lang.Runner.ListUnitsSetup). The script used +// to keep its own copy of the listing prelude; a copy of a setup that writes +// a PHP file would be a second place to change it, and a fixture captured +// under the copy would prove nothing about the row. +package main + +import ( + "fmt" + "os" + + "github.com/pdbethke/corralai/internal/lang" +) + +func main() { + if len(os.Args) < 3 { + fmt.Fprintln(os.Stderr, "usage: listunitscmd ") + os.Exit(2) + } + p, ok := lang.ByName(os.Args[1]) + if !ok { + fmt.Fprintf(os.Stderr, "listunitscmd: no language %q\n", os.Args[1]) + os.Exit(2) + } + cmd, ok := lang.ListUnitsCmd(p, lang.Cmd(os.Args[2:])) + if !ok || len(cmd) < 4 || cmd[0] != "sh" || cmd[1] != "-c" { + fmt.Fprintf(os.Stderr, "listunitscmd: %q has no unit listing\n", os.Args[2:]) + os.Exit(1) + } + fmt.Print(cmd[2]) +} diff --git a/site/src/content/docs/docs/language-support.md b/site/src/content/docs/docs/language-support.md index 495cb91e..5f52402d 100644 --- a/site/src/content/docs/docs/language-support.md +++ b/site/src/content/docs/docs/language-support.md @@ -63,6 +63,6 @@ A language has a runner capability when any of its runners does; this is which. | `node --test` (default) | `node-test` | javascript, typescript | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | node has no collect-only mode, so listing RUNS the test file once on the unmutated code; verified on node 22.23 and 24.22: listing, one-test, one-file and several-file runs, and salvage, on both (24.22.2 measured 2026-11-10). Through a package script, listing, one-test runs and salvage need `--test` to be the script's last word, since node reads options only before the first file | | `rspec` (default) | `rspec` | ruby | `--fail-fast` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on rspec-core 3.13 (one-test runs need 3.12 or later) | | `minitest` | `minitest` | ruby | | | | | | | ✓ | ✓ | | minitest has no list mode and no full-name selector; rspec is the Ruby runner that lists and runs one test; no unit evidence: no listing command; Ruby falls back to name pairing under minitest | -| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs (`--list-test-files`) verified on PHPUnit 11.5; other versions were not run | +| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on PHPUnit 11.5; the test-file listing (`--list-tests`, with PHP naming each test class's file) also on 9.6, 10.5 and 12.5, where one-file runs were run as well; PHPUnit 9 and 10 ignore `--group` and `--exclude-group` when listing, so a command carrying one gets no test-file listing there | | `cargo nextest` | `cargo-nextest` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | one-binary and several-binary runs (one `-E 'binary_id(=...)'` each) and their `list` listing verified on cargo-nextest 0.9.148; a command with its own filterset or a partition has no unit form | | `cargo test` (default) | `cargo-test` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | libtest has no retries; flaky tests are named when the output is nextest's (a wrapper that runs nextest under another command); one-target runs (`-p ` with `--lib`, `--test` or `--bin`, in place of the command's own package and target selection) verified on cargo 1.90; a run of several targets runs each named target in every named package, since cargo selects targets across the packages it is given (measured on cargo 1.90); doc tests, examples and benches are no unit | From b01906363817f76027667d6fdcce38f66228934b Mon Sep 17 00:00:00 2001 From: Peter D Bethke Date: Wed, 18 Nov 2026 17:28:43 -0500 Subject: [PATCH 2/3] Load pcov in the PHP coverage run only when PHP does not already have it under the injected ini scan path, because the second load's start-up warning on stderr failed every test PHPUnit runs in a separate process (eight ramsey/uuid test files gave no unit evidence). Co-Authored-By: Claude Fable 5.2 --- internal/lang/php.go | 45 +++++++++++------- internal/lang/php_test.go | 97 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+), 17 deletions(-) diff --git a/internal/lang/php.go b/internal/lang/php.go index 20191ad0..62095d9e 100644 --- a/internal/lang/php.go +++ b/internal/lang/php.go @@ -656,25 +656,36 @@ func (phpPlugin) CoverageCmd(testCmd []string) (cmd []string, ok bool) { if len(testCmd) == 0 || !phpIsRunner(testCmd) { return nil, false } - // `extension=pcov` is written even though the machine may already load it, - // and that redundancy is the fix for a real failure. Overriding - // PHP_INI_SCAN_DIR replaces the scan path, and although a LEADING COLON is - // documented to mean "the default directory, then this one" — and does - // exactly that locally — on a GitHub runner the default was not preserved: - // auto_prepend_file loaded from this file while pcov did not load at all, - // so \pcov\start was undefined inside the instrumented run even though a - // plain `php -r` reported the extension present. Naming the extension here - // makes the injected ini self-sufficient instead of dependent on which - // directories survived the override. + // `extension=pcov` is written only when, under this same scan path, PHP + // does not already have pcov; both halves of that are fixes for real + // failures. // - // Loading it twice is safe: PHP warns "Module already loaded" and carries - // on, and an unresolvable name warns "Unable to load dynamic library" and - // carries on. Both land on stderr, where the suite's own output already - // goes, and both leave the prepend's function_exists guard to make the - // actual decision. Neither is fatal, which is the property that matters — - // see phpCoveragePrepend for why nothing here may kill the audited suite. + // Written when missing: overriding PHP_INI_SCAN_DIR changes the scan + // path, and although a LEADING COLON is documented to mean "the default + // directory, then this one" — and does exactly that locally — on a + // GitHub runner the default was not preserved: auto_prepend_file loaded + // from this file while pcov did not load at all, so \pcov\start was + // undefined inside the instrumented run even though a plain `php -r` + // reported the extension present. So the probe runs under the override, + // not plainly, and naming the extension when it reports none makes the + // injected ini self-sufficient. + // + // Not written when present: this used to name the extension always, on + // the reasoning that a second load only warns. The warning (`PHP + // Warning: Module "pcov" is already loaded`) goes to stderr at every + // PHP start-up, and PHPUnit fails a test it runs in a separate process + // when that process writes to stderr. On ramsey/uuid (PHPUnit 9.6.39, + // PHP 8.5.4, 2026-11-18) eight test files failed under coverage and + // passed without it, so they gave no unit evidence. + // + // The probe runs before auto_prepend_file is written, so it never loads + // the prepend. A probe that cannot run at all (no `php` word on PATH: a + // suite started through an absolute path or a wrapper) names the + // extension, which is what every run did before. setup := `mkdir -p "$d/cov" && cat > "$d/corral_prepend.php" <<'CORRAL_PHP_EOF'` + "\n" + phpCoveragePrepend + "CORRAL_PHP_EOF\n" + - `printf 'extension=pcov\npcov.enabled=1\nauto_prepend_file=%s\n' "$d/corral_prepend.php" > "$d/zz-corral.ini"` + "\n" + `printf 'pcov.enabled=1\n' > "$d/zz-corral.ini"` + "\n" + + `PHP_INI_SCAN_DIR=":$d" php -r 'exit(extension_loaded("pcov") ? 0 : 1);' >/dev/null 2>&1 || printf 'extension=pcov\n' >> "$d/zz-corral.ini"` + "\n" + + `printf 'auto_prepend_file=%s\n' "$d/corral_prepend.php" >> "$d/zz-corral.ini"` + "\n" // The LEADING COLON keeps the default scan directory — without it pcov // itself, and every other extension the suite relies on, stops loading. env := `PHP_INI_SCAN_DIR=":$d" CORRAL_COV_DIR="$d/cov"` diff --git a/internal/lang/php_test.go b/internal/lang/php_test.go index 31a21f1f..bfde4cb0 100644 --- a/internal/lang/php_test.go +++ b/internal/lang/php_test.go @@ -3,6 +3,7 @@ package lang import ( + "bytes" "os" "os/exec" "path/filepath" @@ -216,3 +217,99 @@ spl_autoload_register(function ($c) { func TestPHPCoverageCountsFromRealRun(t *testing.T) { assertFixtureCounts(t, capturedCounts(t, "php"), "lib/Calc.php", "lib/Dead.php", "lib/Unloaded.php", "lib/Shape.php") } + +// The coverage run loads pcov ONCE. The injected ini used to name +// `extension=pcov` whether or not the machine already loaded it, and a +// second load makes PHP print `PHP Warning: Module "pcov" is already loaded` +// on stderr at every start-up. PHPUnit fails a test it runs in a separate +// process when that process writes to stderr, so on ramsey/uuid (2026-11-18, +// PHPUnit 9.6.39, PHP 8.5.4) eight test files failed under coverage and +// passed without it, and gave no unit evidence. The run still measures: the +// extension is named when, under the same ini scan path, PHP does not have +// it. +func TestPHPCoverageLoadsPcovOnce(t *testing.T) { + if _, err := phpInterpreter(nil); err != nil { + t.Skipf("SKIP php coverage: no php on PATH: %v", err) + } + if out, err := exec.Command("php", "-r", `exit(extension_loaded("pcov") ? 0 : 1);`).CombinedOutput(); err != nil { + t.Skipf("SKIP php coverage: this php does not load pcov: %v %s", err, out) + } + dir := t.TempDir() + for rel, body := range map[string]string{ + "lib/Calc.php": "= c.Executable { + t.Errorf("lib/Calc.php: one function ran and one did not, got %+v (all: %v)", c, counts) + } +} + +// The other half of TestPHPCoverageLoadsPcovOnce: when the probe says PHP +// has no pcov under the injected scan path, the extension IS named. The +// probe is answered "no" here by a `php` shim that passes every other call +// to the real php; this machine does load pcov, so the line the wrapper +// then writes shows as PHP's "already loaded" warning, which is the proof +// it was written. +func TestPHPCoverageNamesPcovWhenTheProbeFindsNone(t *testing.T) { + real, err := exec.LookPath("php") + if err != nil { + t.Skipf("SKIP php coverage: no php on PATH: %v", err) + } + if out, err := exec.Command(real, "-r", `exit(extension_loaded("pcov") ? 0 : 1);`).CombinedOutput(); err != nil { + t.Skipf("SKIP php coverage: this php does not load pcov: %v %s", err, out) + } + dir := t.TempDir() + shim := filepath.Join(dir, "bin") + if err := os.MkdirAll(shim, 0o750); err != nil { + t.Fatal(err) + } + script := "#!/bin/sh\ncase \"$*\" in *extension_loaded*) exit 1;; esac\nexec " + real + " \"$@\"\n" + if err := os.WriteFile(filepath.Join(shim, "php"), []byte(script), 0o700); err != nil { // #nosec G306 -- an executable shim in a temp dir + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "main.php"), []byte(" Date: Wed, 18 Nov 2026 17:31:37 -0500 Subject: [PATCH 3/3] Say in the changelog and the generation note what the two PHP fixes measured on ramsey/uuid: 59 of 67 test files gave evidence with the listing fix alone and all 67 with pcov loaded once. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 5 ++++- internal/reposcan/generation.go | 7 ++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 19775f59..821d5b5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,9 +16,12 @@ history of any release, `git log v0.3.4..v0.3.5`. - **PHP: a project on PHPUnit 9 or 10 gets test-file evidence.** To learn which test file executes which source file, corral asks the runner which test files your command collects and runs each one alone under coverage. For PHPUnit it asked with `--list-test-files`, an option that exists only from PHPUnit 11. On 9.6 and 10.5 the listing stopped at `Unknown option "--list-test-files"`, so no test file was run alone: a source file was paired with a test by file name only (one with no test of a matching name was not audited at all), and every file was graded by the whole suite. corral now asks with `--list-tests`, which every PHPUnit from 9.6 has, and has PHP itself name the file each listed test class was loaded from, so a test class needs no autoload entry to be found. In a PHPUnit 9.6 project whose only source file has no test of its name, the file went from not audited to paired by evidence with the one test file that runs it. - *What the listing run sees that yours does not.* For the listing only, `PHP_INI_SCAN_DIR` gains a temporary directory after any value you set, holding one `auto_prepend_file` setting; the file it names prints each test class's file when PHP exits and does nothing in a process that never loaded PHPUnit. The directory is removed when the listing ends. No graded run is changed. - *Group options on PHPUnit 9 and 10.* Those versions ignore `--group` and `--exclude-group` when listing and say so (`The --group and --list-tests options cannot be combined, --group is ignored`), so the list is of every test and not of the tests your command runs. corral reads that sentence as no listing: such a command gets no test-file evidence on 9 or 10, is graded by the whole suite as before, and the scan says why, with PHPUnit's own output. PHPUnit 11 and 12 apply the group options when listing, as `--list-test-files` did. - - *Where it was run.* PHPUnit 9.6.39, 10.5.67, 11.5.57 and 12.5.44 on PHP 8.5, listing and one-file runs on each; on 9.6.39 also whole scans in the bubblewrap sandbox (`certify --repo --dry-run --evidence`): the project above, and ramsey/uuid at `7a16331`, where the listing named 67 test files, 59 of them reported which source files they execute, and 10 source files became candidates that file names alone had not paired (the other 8 test files failed under coverage for a reason of corral's own, not yet fixed: it loads the pcov extension a second time, PHP warns, and a test PHPUnit runs in a separate process fails on the warning; those 8 are added to every file's selection, so no file is graded by fewer tests than execute it). No older PHPUnit and no older PHP was run, and no audit with models was run on a PHPUnit 9 or 10 project. + - *Where it was run.* PHPUnit 9.6.39, 10.5.67, 11.5.57 and 12.5.44 on PHP 8.5, listing and one-file runs on each; on 9.6.39 also whole scans in the bubblewrap sandbox (`certify --repo --dry-run --evidence`): the project above, and ramsey/uuid at `7a16331`, where the listing named 67 test files and 10 source files became candidates that file names alone had not paired (measured before the pcov fix in the next entry, when 59 of the 67 reported which source files they execute). No older PHPUnit and no older PHP was run, and no audit with models was run on a PHPUnit 9 or 10 project. - Cached verdicts are measured again once (`VerdictGeneration` 47). +- **PHP: the coverage run no longer fails tests that PHPUnit runs in a separate process.** To measure which source files a test file executes, corral adds an ini file to the run that turns pcov on. That file named `extension=pcov` whether or not your PHP already loads it, and a second load makes PHP print `PHP Warning: Module "pcov" is already loaded` on its error stream at every start. PHPUnit fails a test it runs in a separate process (`@runInSeparateProcess`, process isolation) when that process writes to its error stream, so such tests failed under corral's coverage run and passed in yours, and their test files gave no evidence. corral now asks PHP, under the same ini scan path the run will use, whether pcov is loaded, and names the extension only when it is not. Only the coverage runs are changed; a graded run never carried the ini file. + - *Where it was run.* PHP 8.5.4 with pcov from the distribution package, PHPUnit 9.6.39, in the bubblewrap sandbox on ramsey/uuid at `7a16331`: of the 67 test files listed, 59 gave evidence before this change (the other 8 hold separate-process tests and failed on the warning) and all 67 after, and the source files paired by evidence went from 10 to 11. The branch that names the extension was run with a stand-in `php` that answers the probe "not loaded". The GitHub runner on which the extension was first found missing under the scan path was not re-run. + - **A unit listing that fails prints the runner's output once.** When the listing of test units failed, the scan printed what the runner said on the `units:` line, again on the `selection:` line, and a third time inside the summary line, each copy as many lines long as the runner's output. The `units:` line still prints it in full; the two lines after it now give the reason in one line, the same sentence the record keeps. ## [v1.0.0-rc.34] — 2026-11-18 diff --git a/internal/reposcan/generation.go b/internal/reposcan/generation.go index dd4ccad9..6138f7aa 100644 --- a/internal/reposcan/generation.go +++ b/internal/reposcan/generation.go @@ -472,7 +472,12 @@ package reposcan // evidence and graded by the test files that execute it, which can change // which files are audited and, through per-mutant time caps and fail-fast // order, what a file's verdict holds. PHPUnit 11 and 12 list the same files -// as before (captured on both). "46" was released in rc.34, so this gets its +// as before (captured on both). In the same generation, the PHP coverage +// run names `extension=pcov` only when PHP lacks it under the injected scan +// path: the second load's stderr warning failed every test PHPUnit runs in +// a separate process, so such a test file gave no unit evidence and was +// added to every file's selection instead (ramsey/uuid: 59 of 67 test files +// reported before, 67 after). "46" was released in rc.34, so this gets its // own generation. const VerdictGeneration = "47"