Repository navigation
Swift coverage without the native build system, alias refusal, 6.4.2 run counts, CI tarball cache - #564
Merged
Conversation
…ct with the mechanism Rust's coverage uses, instead of reading the export SwiftPM writes, because that export covers one test product under swiftbuild (Swift 6.4's default build system) and the pin to --build-system native that worked around it rests on a build system Swift 6.4.2 says is deprecated. The merge, the -summary-only export and the reduction are now one shared function, llvmCovMergeExportShell, which Rust calls with its own facts (its llvm-tools, its profile directory, the executables cargo --no-run lists) and Swift with its own: the llvm-profdata, llvm-cov and llvm-objdump beside the swift the command runs with its links followed, the profiles in the directory of the file --show-codecov-path names (an earlier run's removed first), and as objects the files of that directory's parent that carry an __llvm_covmap section. Measured 2026-11-20 in Docker on swift:6.3.2-noble and swift:6.4.2-resolute, each under native and under swiftbuild, on the coverage fixture and on a copy with a library target only the XCTest target links: all four gave the committed report for the fixture, and for the copy the same counts plus the library (3 of 3) and the second XCTest test, where SwiftPM's own export under swiftbuild left the XCTest target and the library out (6.4.2) or held no project file (6.3.2). So the coverage run no longer adds --build-system native or refuses another build system. With an executable target no test links, its main.swift is reported with no line run (0 of 1) where SwiftPM's export left it out; it reads as never run either way. A run that does not pass is disclosed as unavailable with its exit status, because a test that trapped left no profile for its process (one of two under native, six of eight under swiftbuild, both versions) and its exit status was the same 1 as a failed assertion's; a build that failed wrote no profile. The capture script makes swift-codecov.json the same way and it was recaptured on 6.3.2 (a 6.4.2 capture in a scratch copy gave the same four files and line counts); the captured report did not change, and TestCapturedCoverageReportsMatchALiveRun, TestEveryCoverageCmdReportsTheSuiteExit and TestCanaryCodeIsInvalidSourceInEveryLanguage passed on a scratch copy in corral-swift-t9 (6.3.2), corral-swift-642-go (6.4.2) and its gawk variant. Rust's live coverage tests passed on the host with its llvm-tools, unchanged. Only Linux was measured; on macOS a test product is a bundle directory, which the object listing does not take, and what it reports there was not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…her of its aliases, --show-code-coverage-path or --show-coverage-path, as well as by that name, because each makes swift test print the path and run no test, so the coverage run would measure nothing while the alias went unrefused. swift test --help on Swift 6.3.2 (swift:6.3.2-noble) and 6.4.2 (swift:6.4.2-resolute) lists the two as aliases of --show-codecov-path, and on 2026-11-20 each of the three printed the same codecov path on the coverage fixture and ran no test, on both versions. The three names are one list, swiftCodecovPathOptions, which both the refusal and the script's own question use; the test refuses each by its literal name and every member of the list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts, in the test helper swiftRan and the capture script's swiftran alike, because Swift 6.4.2 under swiftbuild runs each test product on its own and prints one XCTest and one swift-testing summary per product, and reading the last of each undercounted: a --swift-only capture through swift:6.4.2-resolute on 2026-11-20, in a scratch copy of the repository, failed five of its checks (5 where 8 ran for the two-target run, 0 where 3 ran for the one-target run, 0 where 1 ran for the one XCTest test, 1 where 2 ran for two filters, 2 where 3 ran for the authored test), and the same tests over those captures failed the same way. The count is now every swift-testing "Test run with N tests" plus, per XCTest run, the "Executed N tests" that closes its outermost suite, found by nesting depth rather than by the suite's name ('All tests' or 'Selected tests'), since the nested suites repeat their part of the total. With it the same capture passed every check on 6.4.2 and on 6.3.2 (swift:6.3.2-noble), the helper's tests passed over the 6.4.2 captures in the scratch copy and over the committed 6.3.2 ones, and the script's awk gave the helper's counts on eight captures from each version under mawk and gawk. The 6.4.2 captures are not committed; the new test reads two committed 6.3.2 runs back to back, which is the shape a 6.4.2 run has. Production parsers do not count from summaries and are unchanged. Over the 6.4.2 captures TestSwiftDollarAnchoredFilterRunsNothingAndPasses still fails on its string checks, because a 6.4.2 --filter that matches nothing prints no summary at all, as AGENTS.md already records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipt derives from its own pins, while still checking its signature on every run, and run the pinned-Swift check in the self-audit too, from one shared script, because every validate run downloaded the same 1.1 GB tarball and the self-audit ran the real-toolchain Swift tests with nothing holding them to the pinned Swift. The pins (version and dependencies per Ubuntu release, tarball name per architecture) move into one function, swift_pin, read both by the install and by `ci-provision-test-toolchains.sh --swift-cache-key`, which prints the key (corral-ci-swift-<version>-ubuntu<release>-<arch>) and the cache directory for actions/cache (v6.1.0, pinned to its commit), so the workflows carry no copy of a version. Only the tarball is cached: the signature and the keyring are fetched fresh every run, a cached tarball is checked exactly as a download is before anything is unpacked, a failure names the cache entry to delete and installs nothing, and only a tarball that passed the check is moved into the cache. Run on 2026-11-20 in clean ubuntu:26.04 and ubuntu:24.04 containers as root with sudo shimmed, on the script's own pin section and Swift block: with no cache directory (26.04), on a cache miss and on a cache hit (both releases) the install passed the shared check; a cached tarball with six bytes overwritten failed the signature check on both releases, unpacked nothing, linked nothing, and the check then failed. The check that was inline in deploy.yml is now scripts/ci-check-pinned-swift.sh, called by deploy.yml and self-audit.yml; it now reports an absent swift as absent where ubuntu:26.04's `readlink -f ""` made it "/", and it failed in containers with no swift, with an image's own /usr/bin Swift, and with no pin. Neither workflow change has run in CI, and actionlint was not installed here, so the workflows were checked only as YAML. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s profiles name, instead of every file in SwiftPM's products directory that carries a coverage map, because .build/corral-coverage outlives a run and under swiftbuild a removed test target's product stays there with the coverage map of the code it was built from, so a rerun in the same build directory read the removed target's file and a removed function's lines and said nothing. Measured 2026-11-20 on swift:6.3.2-noble and swift:6.4.2-resolute, under native and swiftbuild, on the coverage fixture plus a test target ExtraTests, run once and run again in the same build directory after ExtraTests was removed and Calc.swift lost sub: taking every covmap product, swiftbuild read Calc.swift at 3 of 6 lines and ExtraTests.swift at 0 of 5, where a fresh build directory gives 3 of 3 and no ExtraTests row; every current product had been relinked, so the stale lines came from the leftover ExtraTests.so and ExtraTests-test-runner alone; native has one product, relinked, and nothing was stale. Each raw profile carries the GNU build ID of the binary that wrote it (llvm-profdata show --binary-ids), and the products whose build ID (readelf -n) is among them were exactly the current ones that ran, with the fresh directory's report all four ways. The two other options measured lost: an unchanged rerun rewrote no product on either build system, so "newer than the run" cannot pick them; deleting the products directory's files first restored every current one and not the stale, but relinks every product on every run (0.3 s against 0.7 s native, 0.5-0.7 s against 1.3-1.7 s swiftbuild, on the fixture) and deletes files SwiftPM owns. Naming products from the manifest would need each build system's file names. Every build ID the profiles name must be found, and profiles naming none are refused, so a layout change is disclosed as unavailable, not read short. readelf is binutils' (a Swift Linux dependency per swift.org), taken from PATH since an ELF note has no LLVM format version; llvm-objdump is no longer used. TestSwiftCoverageReadsOnlyThisRunsProducts reproduces the scenario against the real toolchain under both build systems; it failed on the previous code under swiftbuild on 6.4.2 and passes on 6.3.2, 6.4.2 and 6.4.2 with gawk, as do the three real-toolchain tests; the library and executable variants gave the doc comment's reports on both versions. The capture of swift-codecov.json no longer keeps a copy of these steps: llvmCovMergeExportShell now stops at the export, CoverageCmd and the new lang.SwiftCoverageExportCmd are one builder with two endings (reduce, or print the raw export), and scripts/capture-runner-fixtures.sh runs the latter's script, printed by scripts/swiftcoverageexport, where SWIFT runs, so an empty --show-codecov-path is refused by the plugin's own check before anything is removed; a 6.3.2 recapture through it wrote the committed file byte for byte. The doc comment, CHANGELOG, AGENTS.md and both PROVENANCE files now say what holds and what was run, the stricter tool lookup, what the code does on macOS (unavailable at a check; not run), and describe the 6.3.2 test image by what it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… back on the provisioning step, because the Swift cache steps added before it had left the comment sitting above them, where it described a step it was not attached to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…that reads each profile's build IDs called the unavailable notice from inside a pipe: the notice's exit ended only the pipe's subshell and its text went into the list of build IDs, so the script went on and reported a binary missing from SwiftPM's products directory, naming the notice as the build ID. The outcome was still unavailable, with a false reason. The loop now records the failure and the script discloses it after the loop; a new test with a stand-in llvm-profdata that exits 3 failed before this and names the failed show now. readelf is run with LC_ALL=C, because the script reads its English "Build ID:" line and binutils translates that text. The capture script no longer writes a coverage capture over the committed one after one of its own checks failed, and a test's exemption for llvm-objdump, which the script stopped calling, is gone. Run on 2026-11-20 on scratch copies: the three real-toolchain Swift tests and the leftover-product test passed on Swift 6.4.2 (with gawk as awk) and on 6.3.2. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four follow-ups to Swift support (#563).
What changes
--build-system native. Swift 6.4.2 marksnativedeprecated, and underswiftbuildSwiftPM's own llvm-cov export left out every test product but one. Swift now merges its raw profiles and exports them itself, through the merge-and-export step Rust's coverage already used (llvmCovMergeExportShell, one mechanism for both). The products exported are exactly those whose GNU build ID this run's profiles name, so a product an earlier build left in.build/corral-coverageis not read. A profile whose binary is not found, a profile naming no build ID, a missing tool, and any non-zero run are disclosed as unavailable.--show-codecov-pathare refused (--show-code-coverage-path,--show-coverage-path), from one list.actions/cache, key derived from the provisioning script's own pins). The signature is checked on every run, cached or not, before anything is unpacked.deploy.ymlandself-audit.ymlshare one pinned-Swift check script.What was run (2026-11-20, Docker, network off)
nativeandswiftbuild: the coverage fixture reduced to the committed report; a rerun in the same build directory after removing a test target, and after renaming a source file, gave the report of a fresh build directory.TestCapturedCoverageReportsMatchALiveRun/swift,TestEveryCoverageCmdReportsTheSuiteExit/swift,TestCanaryCodeIsInvalidSourceInEveryLanguage/swiftandTestSwiftCoverageReadsOnlyThisRunsProductspassed on 6.3.2, on 6.4.2, and on 6.4.2 with gawk as awk.ubuntu:24.04andubuntu:26.04containers: cache miss and cache hit installed and passed the check; a cached tarball with bytes overwritten failed the signature check with nothing unpacked.go test ./..., gofmt, vet, the security and licensing gates and the four--checkgates pass locally.Not run
certify --repoon a real Swift repository with the new coverage path.actionlintwas not available.Behaviour to know
readelf(binutils) on PATH and the toolchain'sllvm-profdataandllvm-covbeside the resolvedswift; otherwise it is disclosed as unavailable and the suite is not run for coverage.🤖 Generated with Claude Code