Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,21 @@ still move between minor versions.
Entries describe what changed for someone *using* the tool. For the full commit
history of any release, `git log v0.3.4..v0.3.5`.

## [Unreleased]

### Fixed

- **PHP: a project on PHPUnit 9 or 10 gets test-file evidence.** To learn which test file executes which source file, corral asks the runner which test files your command collects and runs each one alone under coverage. For PHPUnit it asked with `--list-test-files`, an option that exists only from PHPUnit 11. On 9.6 and 10.5 the listing stopped at `Unknown option "--list-test-files"`, so no test file was run alone: a source file was paired with a test by file name only (one with no test of a matching name was not audited at all), and every file was graded by the whole suite. corral now asks with `--list-tests`, which every PHPUnit from 9.6 has, and has PHP itself name the file each listed test class was loaded from, so a test class needs no autoload entry to be found. In a PHPUnit 9.6 project whose only source file has no test of its name, the file went from not audited to paired by evidence with the one test file that runs it.
- *What the listing run sees that yours does not.* For the listing only, `PHP_INI_SCAN_DIR` gains a temporary directory after any value you set, holding one `auto_prepend_file` setting; the file it names prints each test class's file when PHP exits and does nothing in a process that never loaded PHPUnit. The directory is removed when the listing ends. No graded run is changed.
- *Group options on PHPUnit 9 and 10.* Those versions ignore `--group` and `--exclude-group` when listing and say so (`The --group and --list-tests options cannot be combined, --group is ignored`), so the list is of every test and not of the tests your command runs. corral reads that sentence as no listing: such a command gets no test-file evidence on 9 or 10, is graded by the whole suite as before, and the scan says why, with PHPUnit's own output. PHPUnit 11 and 12 apply the group options when listing, as `--list-test-files` did.
- *Where it was run.* PHPUnit 9.6.39, 10.5.67, 11.5.57 and 12.5.44 on PHP 8.5, listing and one-file runs on each; on 9.6.39 also whole scans in the bubblewrap sandbox (`certify --repo --dry-run --evidence`): the project above, and ramsey/uuid at `7a16331`, where the listing named 67 test files and 10 source files became candidates that file names alone had not paired (measured before the pcov fix in the next entry, when 59 of the 67 reported which source files they execute). No older PHPUnit and no older PHP was run, and no audit with models was run on a PHPUnit 9 or 10 project.
- Cached verdicts are measured again once (`VerdictGeneration` 47).

- **PHP: the coverage run no longer fails tests that PHPUnit runs in a separate process.** To measure which source files a test file executes, corral adds an ini file to the run that turns pcov on. That file named `extension=pcov` whether or not your PHP already loads it, and a second load makes PHP print `PHP Warning: Module "pcov" is already loaded` on its error stream at every start. PHPUnit fails a test it runs in a separate process (`@runInSeparateProcess`, process isolation) when that process writes to its error stream, so such tests failed under corral's coverage run and passed in yours, and their test files gave no evidence. corral now asks PHP, under the same ini scan path the run will use, whether pcov is loaded, and names the extension only when it is not. Only the coverage runs are changed; a graded run never carried the ini file.
- *Where it was run.* PHP 8.5.4 with pcov from the distribution package, PHPUnit 9.6.39, in the bubblewrap sandbox on ramsey/uuid at `7a16331`: of the 67 test files listed, 59 gave evidence before this change (the other 8 hold separate-process tests and failed on the warning) and all 67 after, and the source files paired by evidence went from 10 to 11. The branch that names the extension was run with a stand-in `php` that answers the probe "not loaded". The GitHub runner on which the extension was first found missing under the scan path was not re-run.

- **A unit listing that fails prints the runner's output once.** When the listing of test units failed, the scan printed what the runner said on the `units:` line, again on the `selection:` line, and a third time inside the summary line, each copy as many lines long as the runner's output. The `units:` line still prints it in full; the two lines after it now give the reason in one line, the same sentence the record keeps.

## [v1.0.0-rc.34] — 2026-11-18

### Fixed
Expand Down
13 changes: 11 additions & 2 deletions cmd/corral/certify_repo.go
Original file line number Diff line number Diff line change
Expand Up @@ -1509,13 +1509,22 @@ func runCertifyRepo(args []string, stdout, stderr io.Writer) int {
}
selectionNote := ""
if ex != nil {
selectionNote = ex.selection.Said()
// The pass's Note, not Said(): the "selection:" or "units:" line
// above already printed what the terminal has to add (a failed
// listing's output, an error), and Said() here printed a runner's
// whole output a second and a third time (PHPUnit 9: its banner
// and "Unknown option" on the units line, the grading line and
// this one). The summary repeats the reason, once, in one line.
selectionNote = ex.selection.Note
if selectionNote == "" {
selectionNote = ex.selection.Said()
}
// For a language the unit pass serves, ITS reason is why candidacy
// is pairing-only; the selection note ("no selector for …") is
// about grading.
switch {
case ex.units.Note != "":
selectionNote = ex.units.Said()
selectionNote = ex.units.Note
case ex.unitLandingNote != "":
selectionNote = ex.unitLandingNote
case ex.units.Ran && !evidenceCandidacyOK:
Expand Down
9 changes: 6 additions & 3 deletions cmd/corral/certify_repo_unit_evidence_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -226,11 +226,14 @@ func TestUnitPassDisclosesACommandWithNoUnitForm(t *testing.T) {
root := unitEvidenceTSFixture(t)
s := runUnitEvidenceScan(t, root, "--", "npm", "test")

const note = "typescript: `npm test` has no one-unit form corral can list and run, so no unit evidence"
if !strings.Contains(s, "units: "+note) {
const said = "typescript: `npm test` has no one-unit form corral can list and run, so no unit evidence"
const note = "typescript: the unit pass could not list units: its command has no one-unit form corral can list and run"
if !strings.Contains(s, "units: "+said) {
t.Errorf("missing the units line with the pass's own note:\n%s", s)
}
if !strings.Contains(s, "pairing-only candidacy (no selection evidence: "+note+")") {
// The summary gives the reason in the one line the record keeps
// (UnitEvidence.Note); what the terminal adds is on the units line, once.
if !strings.Contains(s, "pairing-only candidacy (no selection evidence: "+note+")") || strings.Count(s, said) != 1 {
t.Errorf("the summary must say candidacy is pairing-only, and why:\n%s", s)
}
if !strings.Contains(s, "evidence-paired 0") || strings.Contains(s, "paired by evidence") {
Expand Down
4 changes: 3 additions & 1 deletion cmd/corral/certify_repo_units.go
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,9 @@ func printUnitGrading(w io.Writer, ex *localExecutor, langName string) {
}
fmt.Fprintln(w, line+")")
case !ev.Ran:
fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", ex.unitsNotRunSaid(langName))
// The reason alone: the units line above printed what the pass
// said in full (printUnitPass), a failed listing's output included.
fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", ex.unitsNotRunReason(langName))
case reposcan.NoRunnerRow(ex.unitMap) != "":
fmt.Fprintf(w, " selection: grading by the WHOLE suite — %s\n", reposcan.NoRunnerRow(ex.unitMap))
default:
Expand Down
2 changes: 1 addition & 1 deletion docs/corral/language-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,6 @@ A language has a runner capability when any of its runners does; this is which.
| `node --test` (default) | `node-test` | javascript, typescript | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | node has no collect-only mode, so listing RUNS the test file once on the unmutated code; verified on node 22.23 and 24.22: listing, one-test, one-file and several-file runs, and salvage, on both (24.22.2 measured 2026-11-10). Through a package script, listing, one-test runs and salvage need `--test` to be the script's last word, since node reads options only before the first file |
| `rspec` (default) | `rspec` | ruby | `--fail-fast` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on rspec-core 3.13 (one-test runs need 3.12 or later) |
| `minitest` | `minitest` | ruby | | | | | | | ✓ | ✓ | | minitest has no list mode and no full-name selector; rspec is the Ruby runner that lists and runs one test; no unit evidence: no listing command; Ruby falls back to name pairing under minitest |
| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs (`--list-test-files`) verified on PHPUnit 11.5; other versions were not run |
| `phpunit` (default) | `phpunit` | php | `--stop-on-failure` | | | ✓ | ✓ | ✓ | ✓ | ✓ | | listing, one-test, one-file and several-file runs verified on PHPUnit 11.5; the test-file listing (`--list-tests`, with PHP naming each test class's file) also on 9.6, 10.5 and 12.5, where one-file runs were run as well; PHPUnit 9 and 10 ignore `--group` and `--exclude-group` when listing, so a command carrying one gets no test-file listing there |
| `cargo nextest` | `cargo-nextest` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | one-binary and several-binary runs (one `-E 'binary_id(=...)'` each) and their `list` listing verified on cargo-nextest 0.9.148; a command with its own filterset or a partition has no unit form |
| `cargo test` (default) | `cargo-test` | rust | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | libtest has no retries; flaky tests are named when the output is nextest's (a wrapper that runs nextest under another command); one-target runs (`-p <package>` with `--lib`, `--test` or `--bin`, in place of the command's own package and target selection) verified on cargo 1.90; a run of several targets runs each named target in every named package, since cargo selects targets across the packages it is given (measured on cargo 1.90); doc tests, examples and benches are no unit |
45 changes: 28 additions & 17 deletions internal/lang/php.go
Original file line number Diff line number Diff line change
Expand Up @@ -656,25 +656,36 @@ func (phpPlugin) CoverageCmd(testCmd []string) (cmd []string, ok bool) {
if len(testCmd) == 0 || !phpIsRunner(testCmd) {
return nil, false
}
// `extension=pcov` is written even though the machine may already load it,
// and that redundancy is the fix for a real failure. Overriding
// PHP_INI_SCAN_DIR replaces the scan path, and although a LEADING COLON is
// documented to mean "the default directory, then this one" — and does
// exactly that locally — on a GitHub runner the default was not preserved:
// auto_prepend_file loaded from this file while pcov did not load at all,
// so \pcov\start was undefined inside the instrumented run even though a
// plain `php -r` reported the extension present. Naming the extension here
// makes the injected ini self-sufficient instead of dependent on which
// directories survived the override.
// `extension=pcov` is written only when, under this same scan path, PHP
// does not already have pcov; both halves of that are fixes for real
// failures.
//
// Loading it twice is safe: PHP warns "Module already loaded" and carries
// on, and an unresolvable name warns "Unable to load dynamic library" and
// carries on. Both land on stderr, where the suite's own output already
// goes, and both leave the prepend's function_exists guard to make the
// actual decision. Neither is fatal, which is the property that matters —
// see phpCoveragePrepend for why nothing here may kill the audited suite.
// Written when missing: overriding PHP_INI_SCAN_DIR changes the scan
// path, and although a LEADING COLON is documented to mean "the default
// directory, then this one" — and does exactly that locally — on a
// GitHub runner the default was not preserved: auto_prepend_file loaded
// from this file while pcov did not load at all, so \pcov\start was
// undefined inside the instrumented run even though a plain `php -r`
// reported the extension present. So the probe runs under the override,
// not plainly, and naming the extension when it reports none makes the
// injected ini self-sufficient.
//
// Not written when present: this used to name the extension always, on
// the reasoning that a second load only warns. The warning (`PHP
// Warning: Module "pcov" is already loaded`) goes to stderr at every
// PHP start-up, and PHPUnit fails a test it runs in a separate process
// when that process writes to stderr. On ramsey/uuid (PHPUnit 9.6.39,
// PHP 8.5.4, 2026-11-18) eight test files failed under coverage and
// passed without it, so they gave no unit evidence.
//
// The probe runs before auto_prepend_file is written, so it never loads
// the prepend. A probe that cannot run at all (no `php` word on PATH: a
// suite started through an absolute path or a wrapper) names the
// extension, which is what every run did before.
setup := `mkdir -p "$d/cov" && cat > "$d/corral_prepend.php" <<'CORRAL_PHP_EOF'` + "\n" + phpCoveragePrepend + "CORRAL_PHP_EOF\n" +
`printf 'extension=pcov\npcov.enabled=1\nauto_prepend_file=%s\n' "$d/corral_prepend.php" > "$d/zz-corral.ini"` + "\n"
`printf 'pcov.enabled=1\n' > "$d/zz-corral.ini"` + "\n" +
`PHP_INI_SCAN_DIR=":$d" php -r 'exit(extension_loaded("pcov") ? 0 : 1);' >/dev/null 2>&1 || printf 'extension=pcov\n' >> "$d/zz-corral.ini"` + "\n" +
`printf 'auto_prepend_file=%s\n' "$d/corral_prepend.php" >> "$d/zz-corral.ini"` + "\n"
// The LEADING COLON keeps the default scan directory — without it pcov
// itself, and every other extension the suite relies on, stops loading.
env := `PHP_INI_SCAN_DIR=":$d" CORRAL_COV_DIR="$d/cov"`
Expand Down
97 changes: 97 additions & 0 deletions internal/lang/php_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
package lang

import (
"bytes"
"os"
"os/exec"
"path/filepath"
Expand Down Expand Up @@ -216,3 +217,99 @@ spl_autoload_register(function ($c) {
func TestPHPCoverageCountsFromRealRun(t *testing.T) {
assertFixtureCounts(t, capturedCounts(t, "php"), "lib/Calc.php", "lib/Dead.php", "lib/Unloaded.php", "lib/Shape.php")
}

// The coverage run loads pcov ONCE. The injected ini used to name
// `extension=pcov` whether or not the machine already loaded it, and a
// second load makes PHP print `PHP Warning: Module "pcov" is already loaded`
// on stderr at every start-up. PHPUnit fails a test it runs in a separate
// process when that process writes to stderr, so on ramsey/uuid (2026-11-18,
// PHPUnit 9.6.39, PHP 8.5.4) eight test files failed under coverage and
// passed without it, and gave no unit evidence. The run still measures: the
// extension is named when, under the same ini scan path, PHP does not have
// it.
func TestPHPCoverageLoadsPcovOnce(t *testing.T) {
if _, err := phpInterpreter(nil); err != nil {
t.Skipf("SKIP php coverage: no php on PATH: %v", err)
}
if out, err := exec.Command("php", "-r", `exit(extension_loaded("pcov") ? 0 : 1);`).CombinedOutput(); err != nil {
t.Skipf("SKIP php coverage: this php does not load pcov: %v %s", err, out)
}
dir := t.TempDir()
for rel, body := range map[string]string{
"lib/Calc.php": "<?php\nfunction add(int $a, int $b): int {\n return $a + $b;\n}\nfunction unused(): int {\n return 0;\n}\n",
"main.php": "<?php\nrequire __DIR__ . '/lib/Calc.php';\nexit(add(1, 2) === 3 ? 0 : 1);\n",
} {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, rel)), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, rel), []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
cmd, ok := phpPlugin{}.CoverageCmd([]string{"php", "main.php"})
if !ok {
t.Fatal("no coverage command")
}
var stdout, stderr bytes.Buffer
x := exec.Command(cmd[0], cmd[1:]...) // #nosec G204 -- the plugin's own coverage command, in a temp dir
x.Dir = dir
x.Stdout, x.Stderr = &stdout, &stderr
if err := x.Run(); err != nil {
t.Fatalf("coverage run: %v\n%s", err, stderr.String())
}
if strings.Contains(stderr.String(), "already loaded") {
t.Errorf("the coverage run loaded pcov a second time:\n%s", stderr.String())
}
root, err := filepath.EvalSymlinks(dir)
if err != nil {
t.Fatal(err)
}
counts, err := phpPlugin{}.CoverageCounts(stdout.String(), root)
if err != nil {
t.Fatalf("CoverageCounts: %v\n%s", err, stdout.String())
}
if c := counts["lib/Calc.php"]; c.Executed == 0 || c.Executed >= c.Executable {
t.Errorf("lib/Calc.php: one function ran and one did not, got %+v (all: %v)", c, counts)
}
}

// The other half of TestPHPCoverageLoadsPcovOnce: when the probe says PHP
// has no pcov under the injected scan path, the extension IS named. The
// probe is answered "no" here by a `php` shim that passes every other call
// to the real php; this machine does load pcov, so the line the wrapper
// then writes shows as PHP's "already loaded" warning, which is the proof
// it was written.
func TestPHPCoverageNamesPcovWhenTheProbeFindsNone(t *testing.T) {
real, err := exec.LookPath("php")
if err != nil {
t.Skipf("SKIP php coverage: no php on PATH: %v", err)
}
if out, err := exec.Command(real, "-r", `exit(extension_loaded("pcov") ? 0 : 1);`).CombinedOutput(); err != nil {
t.Skipf("SKIP php coverage: this php does not load pcov: %v %s", err, out)
}
dir := t.TempDir()
shim := filepath.Join(dir, "bin")
if err := os.MkdirAll(shim, 0o750); err != nil {
t.Fatal(err)
}
script := "#!/bin/sh\ncase \"$*\" in *extension_loaded*) exit 1;; esac\nexec " + real + " \"$@\"\n"
if err := os.WriteFile(filepath.Join(shim, "php"), []byte(script), 0o700); err != nil { // #nosec G306 -- an executable shim in a temp dir
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "main.php"), []byte("<?php\nexit(0);\n"), 0o600); err != nil {
t.Fatal(err)
}
cmd, ok := phpPlugin{}.CoverageCmd([]string{"php", "main.php"})
if !ok {
t.Fatal("no coverage command")
}
var stderr bytes.Buffer
x := exec.Command(cmd[0], cmd[1:]...) // #nosec G204 -- the plugin's own coverage command, in a temp dir
x.Dir = dir
x.Env = append(os.Environ(), "PATH="+shim+string(os.PathListSeparator)+os.Getenv("PATH"))
x.Stderr = &stderr
_ = x.Run()
if !strings.Contains(stderr.String(), `Module "pcov" is already loaded`) {
t.Errorf("a probe that finds no pcov must name the extension; stderr:\n%s", stderr.String())
}
}
Loading
Loading