Skip to content

Raise the engine coverage gate to 88% and add a mutation harness - #139

Merged
alexkroman merged 1 commit into
mainfrom
tests/coverage-gate-88-and-mutation-harness
Aug 13, 2026
Merged

Raise the engine coverage gate to 88% and add a mutation harness#139
alexkroman merged 1 commit into
mainfrom
tests/coverage-gate-88-and-mutation-harness

Conversation

@alexkroman

Copy link
Copy Markdown
Collaborator

What

Engine line coverage 87.07% → 88.96% (check.sh reads 88.63% because it builds with -warnings-as-errors, which shifts region attribution; both figures are deterministic, verified stable across repeat runs). MIN_COVERAGE 80 → 88.

Plus scripts/mutate.sh, an opt-in mutation-testing harness — deliberately not wired into check.sh.

Coverage: extract pure logic, don't fake the OS

Every gain here comes from splitting a testable decision out of a syscall wrapper, the way MicCapture+Meter already was:

  • KeyInjector.cmdVEvents() split from postCmdV(). Building the events needs no Accessibility trust; only .post fires a live ⌘V into whatever has focus. The untestable half was holding the testable half at 0%, so the ⌘ flag on both events and the kVK_ANSI_V keycode — what makes the paste a paste rather than typing a "v" — went unasserted.
  • FocusCapture.isElectronBundle(_:) split from isElectronApp(_:), applying the pattern isBrowserBundleID already used in that same file. The Electron true arm — what keeps VS Code and Slack on the paste path instead of falling back to copy-only — had no coverage at all; it's now tested against a fixture bundle in a temp dir.
  • waitUntilFrontmost / accessibilityTrusted — read-only probes, safe for a test process to call, and nothing did.
  • DictationLog's conditional keyterms/turns encode had no positive-direction test. The existing one only asserted the keys are omitted when empty, so nothing pinned the on-disk contract in the direction that matters.

Deliberately left uncovered: activate() and the two CGEvent.post calls (steal focus / inject keystrokes), APIKeyStore (reaches the real Keychain), and the FocusCapture AX reads (need a live focused text field). check.sh now records why ~91% is the practical ceiling, so the next person doesn't chase the last points by mocking the OS.

scripts/mutate.sh

Coverage says a line ran. This says whether it's asserted. It flips one operator, re-runs the suite, and reports whether anything noticed. It found two real gaps in fully-covered code:

  • BlurtError's Equatable compares domain && code, and the existing test varied both fields at once — so it couldn't tell && from || (under || its unequal pair is still false || false). This one compounds: the engine asserts error identity through this ==, so a loose operator wouldn't have failed here, it would have quietly weakened every #expect(phase == .failed(.sttFailed(…))) elsewhere.
  • APIKeyDisplay.rendersIdentifier had its .notConnected arm unpinned; every existing negative assertion targeted .connected(nil).

Both now killed. The default target set scores 100% (66/66 viable mutants).

Harness design notes, each of which is the difference between a real number and a flattering one:

Concern Handling
A mutant hangs the suite Polled deadline (GNU timeout is absent on macOS); counted as killed-by-timeout and reported separately. One mutant genuinely does this.
Orphaned xctest xctest re-parents into its own process group, so a group kill can't reach it; the orphan holds .build and makes the next mutant fail for an unrelated reason — a silent false kill. Swept by absolute bundle path.
Mutant doesn't compile Counted separately, never as killed — a build error is the compiler objecting, not a test.
Mutants in comments/strings Scanner tracks string and comment state; this repo is comment-dense and a mutant planted in prose can never be killed, so it would report as a finding while proving nothing.
Equivalent mutants // mutate-ok: <reason> exempts a line, matching the existing # portable-ok: convention in check-portability.sh. Used once, for resolvingAgainstBaseURL on an already-absolute URL.

It stays out of check.sh because a run is minutes and survivors need judgement — a required gate that reports unactionable failures is one people learn to skip.

Scope

The 100% mutation score covers the 18 curated pure-logic files in the default target list, not the engine as a whole. Pointing it at FocusCapture.swift or PermissionsChecker.swift would report near-total survival by construction, since nothing exercises those lines — hence the curated list, and the script says so.

Verification

scripts/check.sh → exit 0 (556 → 573 tests, coverage gate, TSan + ASan, xcodegen drift, app build, all linters, periphery clean). Re-run after rebasing onto b7513b0, since #138 rewrote check.sh and added check-invariants.sh — a clean textual rebase isn't evidence of green.

🤖 Generated with Claude Code

Engine line coverage 87.07% -> 88.96% (check.sh reads 88.66% because it
builds with -warnings-as-errors, which shifts region attribution; both
figures are deterministic, verified stable across repeat runs).
MIN_COVERAGE 80 -> 88.

The gains come from extracting pure logic out of syscall wrappers rather
than faking the OS:

  - KeyInjector.cmdVEvents() split from postCmdV(). Building the events
    needs no Accessibility trust; only .post fires a live Cmd-V into
    whatever has focus. The untestable half was holding the testable half
    at 0%, so the Cmd flag on both events and the kVK_ANSI_V keycode --
    what makes the paste a paste -- went unasserted.
  - FocusCapture.isElectronBundle(_:) split from isElectronApp(_:),
    applying the pattern isBrowserBundleID already used in that file. The
    Electron true arm -- what keeps VS Code and Slack on the paste path
    instead of falling back to copy-only -- had no coverage at all; it is
    now tested against a fixture bundle.
  - waitUntilFrontmost / accessibilityTrusted: read-only probes that are
    safe for a test process to call, and nothing did.
  - DictationLog's conditional keyterms/turns encode had no
    positive-direction test, so nothing pinned the on-disk contract; the
    existing test only asserted the keys are omitted when empty.

Deliberately left alone: activate() and the two CGEvent.post calls steal
focus and inject keystrokes, APIKeyStore reaches the real Keychain, and
the FocusCapture AX reads need a live focused text field. check.sh now
records why ~91% is the practical ceiling, so the next person doesn't
chase it by mocking the OS.

scripts/mutate.sh is new and opt-in -- deliberately not wired into
check.sh, because a run is minutes and survivors need judgement. It
flips one operator, re-runs the suite, and reports whether anything
noticed. Coverage says a line ran; this says whether it is asserted. It
found two real gaps in fully-covered code:

  - BlurtError's Equatable compares domain && code, and the existing test
    varied both fields at once, so it could not tell && from ||. The
    engine asserts error identity through this ==, so a loose operator
    would have quietly weakened other tests rather than failed here.
  - APIKeyDisplay.rendersIdentifier had its .notConnected arm unpinned;
    every existing negative assertion targeted .connected(nil).

Both are now killed, and the default target set scores 100% (66/66
viable mutants). Harness notes: mutants that hang the suite are bounded
by a polled deadline (GNU timeout is absent on macOS) and counted as
killed-by-timeout; mutants that fail to compile are counted separately
so they cannot inflate the score; comments and string literals are
skipped so a mutant planted in prose cannot report as a finding; and
`// mutate-ok:` exempts an equivalent mutant, matching the existing
`# portable-ok:` convention in check-portability.sh.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alexkroman
alexkroman enabled auto-merge August 13, 2026 20:11
@alexkroman
alexkroman added this pull request to the merge queue Aug 13, 2026
Merged via the queue into main with commit 35c308c Aug 13, 2026
10 checks passed
@alexkroman
alexkroman deleted the tests/coverage-gate-88-and-mutation-harness branch August 13, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants