Raise the engine coverage gate to 88% and add a mutation harness - #139
Merged
Merged
Conversation
Engine line coverage 87.07% -> 88.96% (check.sh reads 88.66% because it
builds with -warnings-as-errors, which shifts region attribution; both
figures are deterministic, verified stable across repeat runs).
MIN_COVERAGE 80 -> 88.
The gains come from extracting pure logic out of syscall wrappers rather
than faking the OS:
- KeyInjector.cmdVEvents() split from postCmdV(). Building the events
needs no Accessibility trust; only .post fires a live Cmd-V into
whatever has focus. The untestable half was holding the testable half
at 0%, so the Cmd flag on both events and the kVK_ANSI_V keycode --
what makes the paste a paste -- went unasserted.
- FocusCapture.isElectronBundle(_:) split from isElectronApp(_:),
applying the pattern isBrowserBundleID already used in that file. The
Electron true arm -- what keeps VS Code and Slack on the paste path
instead of falling back to copy-only -- had no coverage at all; it is
now tested against a fixture bundle.
- waitUntilFrontmost / accessibilityTrusted: read-only probes that are
safe for a test process to call, and nothing did.
- DictationLog's conditional keyterms/turns encode had no
positive-direction test, so nothing pinned the on-disk contract; the
existing test only asserted the keys are omitted when empty.
Deliberately left alone: activate() and the two CGEvent.post calls steal
focus and inject keystrokes, APIKeyStore reaches the real Keychain, and
the FocusCapture AX reads need a live focused text field. check.sh now
records why ~91% is the practical ceiling, so the next person doesn't
chase it by mocking the OS.
scripts/mutate.sh is new and opt-in -- deliberately not wired into
check.sh, because a run is minutes and survivors need judgement. It
flips one operator, re-runs the suite, and reports whether anything
noticed. Coverage says a line ran; this says whether it is asserted. It
found two real gaps in fully-covered code:
- BlurtError's Equatable compares domain && code, and the existing test
varied both fields at once, so it could not tell && from ||. The
engine asserts error identity through this ==, so a loose operator
would have quietly weakened other tests rather than failed here.
- APIKeyDisplay.rendersIdentifier had its .notConnected arm unpinned;
every existing negative assertion targeted .connected(nil).
Both are now killed, and the default target set scores 100% (66/66
viable mutants). Harness notes: mutants that hang the suite are bounded
by a polled deadline (GNU timeout is absent on macOS) and counted as
killed-by-timeout; mutants that fail to compile are counted separately
so they cannot inflate the score; comments and string literals are
skipped so a mutant planted in prose cannot report as a finding; and
`// mutate-ok:` exempts an equivalent mutant, matching the existing
`# portable-ok:` convention in check-portability.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Engine line coverage 87.07% → 88.96% (
check.shreads 88.63% because it builds with-warnings-as-errors, which shifts region attribution; both figures are deterministic, verified stable across repeat runs).MIN_COVERAGE80 → 88.Plus
scripts/mutate.sh, an opt-in mutation-testing harness — deliberately not wired intocheck.sh.Coverage: extract pure logic, don't fake the OS
Every gain here comes from splitting a testable decision out of a syscall wrapper, the way
MicCapture+Meteralready was:KeyInjector.cmdVEvents()split frompostCmdV(). Building the events needs no Accessibility trust; only.postfires a live ⌘V into whatever has focus. The untestable half was holding the testable half at 0%, so the ⌘ flag on both events and thekVK_ANSI_Vkeycode — what makes the paste a paste rather than typing a "v" — went unasserted.FocusCapture.isElectronBundle(_:)split fromisElectronApp(_:), applying the patternisBrowserBundleIDalready used in that same file. The Electrontruearm — what keeps VS Code and Slack on the paste path instead of falling back to copy-only — had no coverage at all; it's now tested against a fixture bundle in a temp dir.waitUntilFrontmost/accessibilityTrusted— read-only probes, safe for a test process to call, and nothing did.DictationLog's conditionalkeyterms/turnsencode had no positive-direction test. The existing one only asserted the keys are omitted when empty, so nothing pinned the on-disk contract in the direction that matters.Deliberately left uncovered:
activate()and the twoCGEvent.postcalls (steal focus / inject keystrokes),APIKeyStore(reaches the real Keychain), and theFocusCaptureAX reads (need a live focused text field).check.shnow records why ~91% is the practical ceiling, so the next person doesn't chase the last points by mocking the OS.scripts/mutate.shCoverage says a line ran. This says whether it's asserted. It flips one operator, re-runs the suite, and reports whether anything noticed. It found two real gaps in fully-covered code:
BlurtError'sEquatablecomparesdomain && code, and the existing test varied both fields at once — so it couldn't tell&&from||(under||its unequal pair is stillfalse || false). This one compounds: the engine asserts error identity through this==, so a loose operator wouldn't have failed here, it would have quietly weakened every#expect(phase == .failed(.sttFailed(…)))elsewhere.APIKeyDisplay.rendersIdentifierhad its.notConnectedarm unpinned; every existing negative assertion targeted.connected(nil).Both now killed. The default target set scores 100% (66/66 viable mutants).
Harness design notes, each of which is the difference between a real number and a flattering one:
timeoutis absent on macOS); counted as killed-by-timeout and reported separately. One mutant genuinely does this.xctestxctestre-parents into its own process group, so a group kill can't reach it; the orphan holds.buildand makes the next mutant fail for an unrelated reason — a silent false kill. Swept by absolute bundle path.// mutate-ok: <reason>exempts a line, matching the existing# portable-ok:convention incheck-portability.sh. Used once, forresolvingAgainstBaseURLon an already-absolute URL.It stays out of
check.shbecause a run is minutes and survivors need judgement — a required gate that reports unactionable failures is one people learn to skip.Scope
The 100% mutation score covers the 18 curated pure-logic files in the default target list, not the engine as a whole. Pointing it at
FocusCapture.swiftorPermissionsChecker.swiftwould report near-total survival by construction, since nothing exercises those lines — hence the curated list, and the script says so.Verification
scripts/check.sh→ exit 0 (556 → 573 tests, coverage gate, TSan + ASan, xcodegen drift, app build, all linters, periphery clean). Re-run after rebasing ontob7513b0, since#138rewrotecheck.shand addedcheck-invariants.sh— a clean textual rebase isn't evidence of green.🤖 Generated with Claude Code