Skip to content

Swift coverage without the native build system, alias refusal, 6.4.2 run counts, CI tarball cache - #564

Merged
pdbethke merged 7 commits into
mainfrom
feat/swift-followups
Nov 21, 2026
Merged

pdbethke merged 7 commits into
mainfrom
feat/swift-followups

Conversation

@pdbethke

Copy link
Copy Markdown
Owner

Four follow-ups to Swift support (#563).

What changes

  • Swift coverage no longer depends on --build-system native. Swift 6.4.2 marks native deprecated, and under swiftbuild SwiftPM's own llvm-cov export left out every test product but one. Swift now merges its raw profiles and exports them itself, through the merge-and-export step Rust's coverage already used (llvmCovMergeExportShell, one mechanism for both). The products exported are exactly those whose GNU build ID this run's profiles name, so a product an earlier build left in .build/corral-coverage is not read. A profile whose binary is not found, a profile naming no build ID, a missing tool, and any non-zero run are disclosed as unavailable.
  • Both aliases of --show-codecov-path are refused (--show-code-coverage-path, --show-coverage-path), from one list.
  • The Swift test helper and the capture script sum run counts across summaries. Swift 6.4.2 prints one XCTest and one swift-testing summary per test product; keeping the last one undercounted.
  • CI caches the pinned Swift tarball (actions/cache, key derived from the provisioning script's own pins). The signature is checked on every run, cached or not, before anything is unpacked. deploy.yml and self-audit.yml share one pinned-Swift check script.

What was run (2026-11-20, Docker, network off)

  • On Swift 6.3.2 and 6.4.2, under native and swiftbuild: the coverage fixture reduced to the committed report; a rerun in the same build directory after removing a test target, and after renaming a source file, gave the report of a fresh build directory.
  • TestCapturedCoverageReportsMatchALiveRun/swift, TestEveryCoverageCmdReportsTheSuiteExit/swift, TestCanaryCodeIsInvalidSourceInEveryLanguage/swift and TestSwiftCoverageReadsOnlyThisRunsProducts passed on 6.3.2, on 6.4.2, and on 6.4.2 with gawk as awk.
  • Rust's generated coverage script differs from main's only in the quoting of one message; its live coverage tests passed before and after.
  • Provisioning in clean ubuntu:24.04 and ubuntu:26.04 containers: cache miss and cache hit installed and passed the check; a cached tarball with bytes overwritten failed the signature check with nothing unpacked.
  • Full go test ./..., gofmt, vet, the security and licensing gates and the four --check gates pass locally.

Not run

  • macOS (reading the code, coverage is disclosed as unavailable there), aarch64 provisioning, and a certify --repo on a real Swift repository with the new coverage path.
  • Neither workflow change had run in CI before this PR; actionlint was not available.

Behaviour to know

  • Swift coverage now needs readelf (binutils) on PATH and the toolchain's llvm-profdata and llvm-cov beside the resolved swift; otherwise it is disclosed as unavailable and the suite is not run for coverage.
  • An executable target no test runs writes no profile, so its files are absent from the report, as they were from SwiftPM's own export.

🤖 Generated with Claude Code

pdbethke and others added 7 commits November 20, 2026 11:43
…ct with the mechanism Rust's coverage uses, instead of reading the export SwiftPM writes, because that export covers one test product under swiftbuild (Swift 6.4's default build system) and the pin to --build-system native that worked around it rests on a build system Swift 6.4.2 says is deprecated. The merge, the -summary-only export and the reduction are now one shared function, llvmCovMergeExportShell, which Rust calls with its own facts (its llvm-tools, its profile directory, the executables cargo --no-run lists) and Swift with its own: the llvm-profdata, llvm-cov and llvm-objdump beside the swift the command runs with its links followed, the profiles in the directory of the file --show-codecov-path names (an earlier run's removed first), and as objects the files of that directory's parent that carry an __llvm_covmap section. Measured 2026-11-20 in Docker on swift:6.3.2-noble and swift:6.4.2-resolute, each under native and under swiftbuild, on the coverage fixture and on a copy with a library target only the XCTest target links: all four gave the committed report for the fixture, and for the copy the same counts plus the library (3 of 3) and the second XCTest test, where SwiftPM's own export under swiftbuild left the XCTest target and the library out (6.4.2) or held no project file (6.3.2). So the coverage run no longer adds --build-system native or refuses another build system. With an executable target no test links, its main.swift is reported with no line run (0 of 1) where SwiftPM's export left it out; it reads as never run either way. A run that does not pass is disclosed as unavailable with its exit status, because a test that trapped left no profile for its process (one of two under native, six of eight under swiftbuild, both versions) and its exit status was the same 1 as a failed assertion's; a build that failed wrote no profile. The capture script makes swift-codecov.json the same way and it was recaptured on 6.3.2 (a 6.4.2 capture in a scratch copy gave the same four files and line counts); the captured report did not change, and TestCapturedCoverageReportsMatchALiveRun, TestEveryCoverageCmdReportsTheSuiteExit and TestCanaryCodeIsInvalidSourceInEveryLanguage passed on a scratch copy in corral-swift-t9 (6.3.2), corral-swift-642-go (6.4.2) and its gawk variant. Rust's live coverage tests passed on the host with its llvm-tools, unchanged. Only Linux was measured; on macOS a test product is a bundle directory, which the object listing does not take, and what it reports there was not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…her of its aliases, --show-code-coverage-path or --show-coverage-path, as well as by that name, because each makes swift test print the path and run no test, so the coverage run would measure nothing while the alias went unrefused. swift test --help on Swift 6.3.2 (swift:6.3.2-noble) and 6.4.2 (swift:6.4.2-resolute) lists the two as aliases of --show-codecov-path, and on 2026-11-20 each of the three printed the same codecov path on the coverage fixture and ran no test, on both versions. The three names are one list, swiftCodecovPathOptions, which both the refusal and the script's own question use; the test refuses each by its literal name and every member of the list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts, in the test helper swiftRan and the capture script's swiftran alike, because Swift 6.4.2 under swiftbuild runs each test product on its own and prints one XCTest and one swift-testing summary per product, and reading the last of each undercounted: a --swift-only capture through swift:6.4.2-resolute on 2026-11-20, in a scratch copy of the repository, failed five of its checks (5 where 8 ran for the two-target run, 0 where 3 ran for the one-target run, 0 where 1 ran for the one XCTest test, 1 where 2 ran for two filters, 2 where 3 ran for the authored test), and the same tests over those captures failed the same way. The count is now every swift-testing "Test run with N tests" plus, per XCTest run, the "Executed N tests" that closes its outermost suite, found by nesting depth rather than by the suite's name ('All tests' or 'Selected tests'), since the nested suites repeat their part of the total. With it the same capture passed every check on 6.4.2 and on 6.3.2 (swift:6.3.2-noble), the helper's tests passed over the 6.4.2 captures in the scratch copy and over the committed 6.3.2 ones, and the script's awk gave the helper's counts on eight captures from each version under mawk and gawk. The 6.4.2 captures are not committed; the new test reads two committed 6.3.2 runs back to back, which is the shape a 6.4.2 run has. Production parsers do not count from summaries and are unchanged. Over the 6.4.2 captures TestSwiftDollarAnchoredFilterRunsNothingAndPasses still fails on its string checks, because a 6.4.2 --filter that matches nothing prints no summary at all, as AGENTS.md already records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipt derives from its own pins, while still checking its signature on every run, and run the pinned-Swift check in the self-audit too, from one shared script, because every validate run downloaded the same 1.1 GB tarball and the self-audit ran the real-toolchain Swift tests with nothing holding them to the pinned Swift. The pins (version and dependencies per Ubuntu release, tarball name per architecture) move into one function, swift_pin, read both by the install and by `ci-provision-test-toolchains.sh --swift-cache-key`, which prints the key (corral-ci-swift-<version>-ubuntu<release>-<arch>) and the cache directory for actions/cache (v6.1.0, pinned to its commit), so the workflows carry no copy of a version. Only the tarball is cached: the signature and the keyring are fetched fresh every run, a cached tarball is checked exactly as a download is before anything is unpacked, a failure names the cache entry to delete and installs nothing, and only a tarball that passed the check is moved into the cache. Run on 2026-11-20 in clean ubuntu:26.04 and ubuntu:24.04 containers as root with sudo shimmed, on the script's own pin section and Swift block: with no cache directory (26.04), on a cache miss and on a cache hit (both releases) the install passed the shared check; a cached tarball with six bytes overwritten failed the signature check on both releases, unpacked nothing, linked nothing, and the check then failed. The check that was inline in deploy.yml is now scripts/ci-check-pinned-swift.sh, called by deploy.yml and self-audit.yml; it now reports an absent swift as absent where ubuntu:26.04's `readlink -f ""` made it "/", and it failed in containers with no swift, with an image's own /usr/bin Swift, and with no pin. Neither workflow change has run in CI, and actionlint was not installed here, so the workflows were checked only as YAML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s profiles name, instead of every file in SwiftPM's products directory that carries a coverage map, because .build/corral-coverage outlives a run and under swiftbuild a removed test target's product stays there with the coverage map of the code it was built from, so a rerun in the same build directory read the removed target's file and a removed function's lines and said nothing. Measured 2026-11-20 on swift:6.3.2-noble and swift:6.4.2-resolute, under native and swiftbuild, on the coverage fixture plus a test target ExtraTests, run once and run again in the same build directory after ExtraTests was removed and Calc.swift lost sub: taking every covmap product, swiftbuild read Calc.swift at 3 of 6 lines and ExtraTests.swift at 0 of 5, where a fresh build directory gives 3 of 3 and no ExtraTests row; every current product had been relinked, so the stale lines came from the leftover ExtraTests.so and ExtraTests-test-runner alone; native has one product, relinked, and nothing was stale. Each raw profile carries the GNU build ID of the binary that wrote it (llvm-profdata show --binary-ids), and the products whose build ID (readelf -n) is among them were exactly the current ones that ran, with the fresh directory's report all four ways. The two other options measured lost: an unchanged rerun rewrote no product on either build system, so "newer than the run" cannot pick them; deleting the products directory's files first restored every current one and not the stale, but relinks every product on every run (0.3 s against 0.7 s native, 0.5-0.7 s against 1.3-1.7 s swiftbuild, on the fixture) and deletes files SwiftPM owns. Naming products from the manifest would need each build system's file names. Every build ID the profiles name must be found, and profiles naming none are refused, so a layout change is disclosed as unavailable, not read short. readelf is binutils' (a Swift Linux dependency per swift.org), taken from PATH since an ELF note has no LLVM format version; llvm-objdump is no longer used. TestSwiftCoverageReadsOnlyThisRunsProducts reproduces the scenario against the real toolchain under both build systems; it failed on the previous code under swiftbuild on 6.4.2 and passes on 6.3.2, 6.4.2 and 6.4.2 with gawk, as do the three real-toolchain tests; the library and executable variants gave the doc comment's reports on both versions. The capture of swift-codecov.json no longer keeps a copy of these steps: llvmCovMergeExportShell now stops at the export, CoverageCmd and the new lang.SwiftCoverageExportCmd are one builder with two endings (reduce, or print the raw export), and scripts/capture-runner-fixtures.sh runs the latter's script, printed by scripts/swiftcoverageexport, where SWIFT runs, so an empty --show-codecov-path is refused by the plugin's own check before anything is removed; a 6.3.2 recapture through it wrote the committed file byte for byte. The doc comment, CHANGELOG, AGENTS.md and both PROVENANCE files now say what holds and what was run, the stricter tool lookup, what the code does on macOS (unavailable at a check; not run), and describe the 6.3.2 test image by what it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… back on the provisioning step, because the Swift cache steps added before it had left the comment sitting above them, where it described a step it was not attached to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…that reads each profile's build IDs called the unavailable notice from inside a pipe: the notice's exit ended only the pipe's subshell and its text went into the list of build IDs, so the script went on and reported a binary missing from SwiftPM's products directory, naming the notice as the build ID. The outcome was still unavailable, with a false reason. The loop now records the failure and the script discloses it after the loop; a new test with a stand-in llvm-profdata that exits 3 failed before this and names the failed show now. readelf is run with LC_ALL=C, because the script reads its English "Build ID:" line and binutils translates that text. The capture script no longer writes a coverage capture over the committed one after one of its own checks failed, and a test's exemption for llvm-objdump, which the script stopped calling, is gone. Run on 2026-11-20 on scratch copies: the three real-toolchain Swift tests and the leftover-product test passed on Swift 6.4.2 (with gawk as awk) and on 6.3.2.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@pdbethke
pdbethke merged commit a19a761 into main Nov 21, 2026
8 checks passed
@pdbethke
pdbethke deleted the feat/swift-followups branch November 21, 2026 00:44
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant