Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/develop-lithe/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ before handoff.

| Change | Minimum relevant validation |
| --- | --- |
| Test code or test infrastructure | `./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh`, then the affected platform timing harness from `write-stable-tests` |
| Swift application or tests | `./scripts/test-macos.sh` |
| Core, Services, Views, or composition boundaries | `./scripts/verify-service-boundaries.sh` |
| Shared application behavior or JSON fixtures | `./scripts/verify-shared-contracts.sh` |
Expand Down
96 changes: 96 additions & 0 deletions .agents/skills/write-stable-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
name: write-stable-tests
description: Write and review deterministic, bounded Lithe tests for macOS Swift, Windows TypeScript, and Rust. Use whenever creating, modifying, or reviewing test code or test infrastructure, especially concurrency, timers, polling, subprocess, watcher, lifecycle, or cancellation tests that could hang CI.
---

# Write Stable Tests

Apply this Skill after `develop-lithe`. Its purpose is to make a broken test
fail locally with a useful diagnostic instead of waiting for a CI job timeout.

## Toolchain boundaries

- macOS uses Swift, `zsh`, and Node.js. It does not require Bun, `npm`, or a
JavaScript package installation for stability checks, timing, JUnit output,
or HTML report generation.
- Windows Rust scopes use Node.js plus Cargo. Windows `Frontend` additionally
uses the repository's Bun toolchain because the product tests run under Bun;
this dependency is isolated to that scope.

## Read the platform guidance

- For Swift or macOS tests, read [references/macos-swift.md](references/macos-swift.md).
- For TypeScript, Tauri Rust, or shared Rust tests exercised by Windows, read
[references/windows-and-rust.md](references/windows-and-rust.md).
- Read both references when a shared contract or cross-platform behavior changes.
- For HTML/JUnit output, performance budgets, or CI artifacts, read
[references/test-reporting.md](references/test-reporting.md).

## Preserve these invariants

- Every wait has an explicit local deadline. The CI step timeout is never the
first mechanism capable of terminating a stuck test.
- Do not use real-time sleeps to synchronize state. Inject a clock, scheduler,
event, continuation, channel, or controllable test double.
- Do not move a blocking wait into a detached task merely to make an async test
compile. Blocking a cooperative executor or main/UI thread is forbidden.
- Every spawned task, timer, process, thread, continuation, stream, and gate has
one owner and a cleanup path that runs after assertion failures as well as
success. Prefer `defer` or the framework's teardown mechanism.
- A concurrency test describes and controls its event order: operation starts,
reaches the synchronization point, is released or cancelled, and terminates.
- Polling is a last resort. It must use a monotonic deadline, produce a useful
timeout diagnostic, and poll an observable boundary rather than private state.
- Unit tests do not depend on real network services, installed developer tools,
machine speed, personal paths, or wall-clock time. Put unavoidable external
dependencies in an explicitly identified integration test.
- Assert observable behavior. Do not weaken production behavior, expose private
state solely for a test, or delete a test to satisfy the stability gate.

## Required workflow

1. Read the changed behavior, implementation, and nearby tests. Identify every
asynchronous boundary and resource whose completion the test will await.
2. Choose deterministic synchronization before writing assertions. For a race
regression, write the intended event sequence explicitly in the test or its
test-double names.
3. Run the fast static gate before the test suite:

```bash
./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh
```

On Windows use:

```powershell
./.agents/skills/write-stable-tests/scripts/verify-test-stability.ps1
```

4. Run the platform timing harness. A changed test is not verified until its
individual duration appears in the generated HTML and JUnit reports:

```bash
./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh -- --filter '<focused-test>'
./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope Frontend
./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope WindowsRust
```

5. Run the broader affected validation required by `develop-lithe`. Report the
HTML report path, slowest changed tests, exact commands, and any suite that
could not run on the current platform. Open
`.artifacts/test-stability/index.html` to review failures, module health, and
performance warnings before handoff.

## Exceptions

Do not add a scanner exception merely to make the gate pass. If a real-time or
blocking primitive is unavoidable at a native synchronous boundary, keep it
off the cooperative executor, add a short local timeout, guarantee cleanup, and
place this annotation immediately above the relevant line:

```text
test-stability: allow(<rule-id>) reason: <why deterministic synchronization is impossible>
```

The reason must describe the architectural constraint, not restate the code.
New exceptions require explicit mention in the handoff.
39 changes: 39 additions & 0 deletions .agents/skills/write-stable-tests/references/macos-swift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# macOS and Swift Test Stability

The macOS harness requires only Swift, zsh, and Node.js. Bun is not installed,
loaded, or invoked anywhere in this path.

## Preferred synchronization

- Prefer actor-owned state, checked continuations, `AsyncStream`, injected
clocks, and explicit callbacks over polling or sleeping.
- Keep `@MainActor` tests fully asynchronous. Never call a semaphore, condition
variable, blocking file read, or process wait from the main actor.
- When a synchronous production protocol forces a blocking test double, run it
only on the production-owned worker thread. Use `DispatchSemaphore.wait(timeout:)`
on both sides of the gate, propagate timeout state into an assertion, and
release the gate from `defer`.
- Cancellation tests must retain the task, trigger cancellation explicitly,
await termination, and verify owned resources were released.

## Prohibited coordination

- Bare `DispatchSemaphore.wait()` or condition waits without a deadline.
- `Thread.sleep`, `usleep`, or `Task.sleep` used to give background work time to
run. A zero-duration yield is still inferior to an observable event.
- `Task.detached { blockingWait() }.value`; this hides blocking and can exhaust
executor threads.
- `Process.waitUntilExit()` without a watchdog that terminates the process tree.
- `while` polling loops without a monotonic deadline and timeout diagnostic.

## Timing and verification

Use `./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh`. It forces serial execution so the
currently running test is unambiguous, records every Swift Testing/XCTest case,
warns about slow cases, and terminates the suite when one case exceeds its local
budget. Reports are written below `.artifacts/test-stability/`. Each run
produces JSON and raw logs for diagnosis, JUnit XML for CI tooling, and a
self-contained HTML report for module and performance review.

Run a focused test while iterating, then run the affected target or complete
lane. Do not claim that a test was timed if it is absent from the report.
42 changes: 42 additions & 0 deletions .agents/skills/write-stable-tests/references/test-reporting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Test Stability Reports

## Outputs

Every timing runner writes these artifacts below `.artifacts/test-stability/`:

- `index.html` is the current runner dashboard. CI regenerates it as a combined
dashboard after every selected lane in the job finishes.
- `<lane>.html` is a self-contained report for one Swift, Bun, or Rust lane.
- `<lane>.junit.xml` is the standard CI and IDE interchange report.
- `<lane>.json` is the normalized machine-readable timing source.
- `<lane>.log`, when available, contains the raw runner output.

The CI workflows upload the complete directory even when a test step fails.
Download the `test-stability-*` artifact and open `index.html`; it does not need
a server or external assets.

## Interpretation

The dashboard separates correctness from performance:

- `failed`, `error`, `timeout`, and `incomplete` are stability failures.
- A duration at or above `maxMs` exceeds the hard performance budget and is
emitted as a JUnit failure even when the underlying assertion passed.
- A duration at or above `warnMs` but below `maxMs` is a performance warning.
It remains a passing JUnit case but appears in the optimization queue.
- Faster passing cases are healthy. Skipped cases are reported separately.

Module health is derived from Swift Testing suites, Bun JUnit classes, and Rust
crate/module paths. Do not infer a production bottleneck from one slow test.
First determine whether the cost is test setup, external I/O, subprocess work,
or the behavior under test, then optimize the owning boundary.

## Regenerating HTML

The timing runners regenerate their own reports automatically. To intentionally
rebuild a combined HTML and JUnit view from every JSON file in the directory,
run:

```bash
node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Windows, TypeScript, and Rust Test Stability

Bun is a Windows Frontend test-runtime dependency only. The report generator,
static verifier, Swift runner, and Rust runner are Node.js-only.

## TypeScript and Bun

- Inject timer functions or a scheduler and advance them manually. The
`ManualTimer` pattern in the nearby Windows tests is preferred for debounce,
cooldown, retry, and delayed-work behavior.
- Use deferred promises only when the test controls every resolution path.
Release or reject pending deferred work in teardown after an assertion fails.
- Flush a known microtask boundary with resolved promises when necessary. Do
not use real `setTimeout` calls to wait for application state.
- Never leave intervals, listeners, subscriptions, workers, or mocked native
operations active after a test.

The Windows timing harness passes an explicit timeout to Bun and reads the
JUnit duration for every executed test case.

## Rust

- Prefer channels, barriers, and injected clocks to `thread::sleep`. Channel
receives used for coordination require `recv_timeout` or an equivalent
bounded operation.
- A thread may be joined only after a bounded signal proves that it reached a
terminating path. Keep cleanup capable of releasing all barriers and killing
child processes.
- Test subprocesses require a watchdog and process-tree termination. Reading to
EOF or calling `wait` is not a timeout strategy.
- Avoid shared global environment mutation. If unavoidable, serialize access
with an owned guard and restore the previous value in teardown.

`./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope WindowsRust` and `-Scope SharedRust`
compile the selected Cargo tests once, enumerate the produced test binaries,
then run every test case individually with a process-level timeout and duration
report. One suite deadline covers compilation, enumeration, every test process,
and the clean cache retry; a timeout writes the completed records before the
runner exits. This isolation makes the exact hanging test visible.

Every Bun and Rust lane writes JUnit XML plus a self-contained HTML dashboard
below `.artifacts/test-stability/`. The dashboard groups Rust cases by crate
module and Bun cases by their JUnit class or suite.

## Windows verification

Run the PowerShell harness in a real Windows environment. A macOS boundary
check does not verify Bun timers, Windows process termination, or native Rust
test execution. When using the Parallels guest, establish the user toolchain
paths before invoking the script as required by `debug-windows-on-parallels`.
Loading
Loading