Skip to content

test(apple,vega): give two runner/Vega probes assertions that can lose - #2939

Merged
thymikee merged 1 commit into
mainfrom
test/runner-and-vega-probes-get-real-assertions
Sep 25, 2026
Merged

thymikee merged 1 commit into
mainfrom
test/runner-and-vega-probes-get-real-assertions

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

Two tests passed for every production bug of the kind they claim to catch. Both now lose for a
specific one, and I ran the mutant against both the old and the new body.

Test Why it could not fail Mutant planted Old body New body
runner-io.test.ts — "a tail whose file disappears stops on its own instead of throwing" onOutput: () => {} threw every chunk away; the body had zero assertions, so the "stops on its own" half of its own name was never observed drain()'s catch { stop(); } → swallow and keep polling (runner-io.ts:211), i.e. precisely the give-up-then-resume bug the in-code comment forbids 8 passed 1 failed | 7 passed
inventory.test.ts — "Vega inventory … accepts only its virtual device" expected value was parseVegaDeviceList(...), a production export — the parser was its own oracle, so both sides moved together kind: 'emulator' → 'physical' in parseVegaDeviceList (inventory.ts:26) 2 passed 1 failed | 1 passed

parseVegaDeviceList is the more interesting half: that oracle call was the function's only
reference outside inventory.ts
, so its kind, booted, target, id and display-name format
were pinned nowhere in the repo. The literal expectation is now that parser's first real test, and it
still carries the discovery claim — the mocked device list output contains a physical : tv row
which must not reach the result.

Dropped after a second read, rather than shipped as noise: runner-sequence.test.ts's unasserted
validateRunnerSequenceSteps(...) accept-paths (the cap case and the low-durationMs case). An
unasserted call still fails when it throws, and cap + 1 rejection is pinned two tests above the cap
case, so the boundary is genuinely closed. Wrapping them in assert.doesNotThrow would have been
cosmetic.

Validation

Tested at 3d11a8ff.

  • --project apple-runner packages/platform-apple/src/runner/__tests__/runner-io.test.ts → 8 passed
  • --project unit-core packages/platform-vega/src/inventory.test.ts → 2 passed
  • pnpm check:affected --run — all runnable checks passed
  • oxlint . --deny-warnings, repo-wide oxfmt, tsc -b packages/platform-vega packages/platform-apple
  • Every mutant reverted; git status clean of production changes before committing

Touches no production file, so no device lane is selected; check:production-exports reports no new
finding for the now test-unreferenced parseVegaDeviceList, which stays exported.

Two tests passed for every possible production bug. Both now fail for a specific one,
and both were checked against the mutant they are meant to catch.

- `runner-io.test.ts` "a tail whose file disappears stops on its own instead of
  throwing" discarded every chunk (`onOutput: () => {}`) and asserted nothing, so the
  sentence the test is named after was never observed. It now collects the chunks and
  requires the tail to still be silent after the log reappears. Mutating `drain()`'s
  `catch { stop(); }` to swallow and keep polling — i.e. exactly the give-up-then-
  resume bug the in-code comment forbids — reddens this test
  (`1 failed | 7 passed`) while the old body reports `8 passed`.

- `inventory.test.ts` "Vega inventory ... accepts only its virtual device" compared
  discovery output against `parseVegaDeviceList(...)`, a production export, so the
  parser was its own expected value and both sides moved together. It now spells the
  device out. Flipping `kind: 'emulator'` to `'physical'` reddens it
  (`1 failed | 1 passed`) where the oracle reported `2 passed`. That literal is also
  the first real test of `parseVegaDeviceList`: the oracle call was its only reference
  outside `inventory.ts`, so `kind`, `booted`, `target`, and the display-name format
  were pinned nowhere.

Dropped from this change after a second read: `runner-sequence.test.ts`'s unasserted
`validateRunnerSequenceSteps(...)` accept-paths. A throw fails those tests, and the
cap+1 rejection is pinned two tests above the cap case, so the boundary is closed.
@github-actions

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.81 MB 4.81 MB 0 B
Package (unpacked) 4.81 MB 4.81 MB 0 B
Package (download) 1.44 MB 1.44 MB +9 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.4 ms 26.5 ms +0.0 ms
CLI --help 79.7 ms 79.0 ms -0.8 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 3d11a8f. This is ready for human review. Checks are green (17/17). Not blocking: the 3-line comment above the literal in https://github.com/callstack/agent-device/blob/3d11a8f/packages/platform-vega/src/inventory.test.ts#L30 explains why the test no longer compares against parseVegaDeviceList(...), but that's history the PR description can carry, not a constraint the literal needs to state, so feel free to drop it or keep it.

I traced the mutation claims statically against runner-io.ts drain()/schedule()/stop() and against parseVegaDeviceList in inventory.ts rather than running the mutated builds, and I only read the PR body's claim about runner-sequence.test.ts (dropped assertions on the cap/low-durationMs accept paths) for context, since it's outside this diff.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 24, 2026
@thymikee
thymikee merged commit 479be84 into main Sep 25, 2026
17 checks passed
@thymikee
thymikee deleted the test/runner-and-vega-probes-get-real-assertions branch September 25, 2026 05:43
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-25 05:43 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant