diff --git a/.agents/skills/develop-lithe/SKILL.md b/.agents/skills/develop-lithe/SKILL.md index 29fab397d..a033288fa 100644 --- a/.agents/skills/develop-lithe/SKILL.md +++ b/.agents/skills/develop-lithe/SKILL.md @@ -180,6 +180,7 @@ before handoff. | Change | Minimum relevant validation | | --- | --- | +| Test code or test infrastructure | `./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh`, then the affected platform timing harness from `write-stable-tests` | | Swift application or tests | `./scripts/test-macos.sh` | | Core, Services, Views, or composition boundaries | `./scripts/verify-service-boundaries.sh` | | Shared application behavior or JSON fixtures | `./scripts/verify-shared-contracts.sh` | diff --git a/.agents/skills/write-stable-tests/SKILL.md b/.agents/skills/write-stable-tests/SKILL.md new file mode 100644 index 000000000..964df8788 --- /dev/null +++ b/.agents/skills/write-stable-tests/SKILL.md @@ -0,0 +1,96 @@ +--- +name: write-stable-tests +description: Write and review deterministic, bounded Lithe tests for macOS Swift, Windows TypeScript, and Rust. Use whenever creating, modifying, or reviewing test code or test infrastructure, especially concurrency, timers, polling, subprocess, watcher, lifecycle, or cancellation tests that could hang CI. +--- + +# Write Stable Tests + +Apply this Skill after `develop-lithe`. Its purpose is to make a broken test +fail locally with a useful diagnostic instead of waiting for a CI job timeout. + +## Toolchain boundaries + +- macOS uses Swift, `zsh`, and Node.js. It does not require Bun, `npm`, or a + JavaScript package installation for stability checks, timing, JUnit output, + or HTML report generation. +- Windows Rust scopes use Node.js plus Cargo. Windows `Frontend` additionally + uses the repository's Bun toolchain because the product tests run under Bun; + this dependency is isolated to that scope. + +## Read the platform guidance + +- For Swift or macOS tests, read [references/macos-swift.md](references/macos-swift.md). +- For TypeScript, Tauri Rust, or shared Rust tests exercised by Windows, read + [references/windows-and-rust.md](references/windows-and-rust.md). +- Read both references when a shared contract or cross-platform behavior changes. +- For HTML/JUnit output, performance budgets, or CI artifacts, read + [references/test-reporting.md](references/test-reporting.md). + +## Preserve these invariants + +- Every wait has an explicit local deadline. The CI step timeout is never the + first mechanism capable of terminating a stuck test. +- Do not use real-time sleeps to synchronize state. Inject a clock, scheduler, + event, continuation, channel, or controllable test double. +- Do not move a blocking wait into a detached task merely to make an async test + compile. Blocking a cooperative executor or main/UI thread is forbidden. +- Every spawned task, timer, process, thread, continuation, stream, and gate has + one owner and a cleanup path that runs after assertion failures as well as + success. Prefer `defer` or the framework's teardown mechanism. +- A concurrency test describes and controls its event order: operation starts, + reaches the synchronization point, is released or cancelled, and terminates. +- Polling is a last resort. It must use a monotonic deadline, produce a useful + timeout diagnostic, and poll an observable boundary rather than private state. +- Unit tests do not depend on real network services, installed developer tools, + machine speed, personal paths, or wall-clock time. Put unavoidable external + dependencies in an explicitly identified integration test. +- Assert observable behavior. Do not weaken production behavior, expose private + state solely for a test, or delete a test to satisfy the stability gate. + +## Required workflow + +1. Read the changed behavior, implementation, and nearby tests. Identify every + asynchronous boundary and resource whose completion the test will await. +2. Choose deterministic synchronization before writing assertions. For a race + regression, write the intended event sequence explicitly in the test or its + test-double names. +3. Run the fast static gate before the test suite: + + ```bash + ./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh + ``` + + On Windows use: + + ```powershell + ./.agents/skills/write-stable-tests/scripts/verify-test-stability.ps1 + ``` + +4. Run the platform timing harness. A changed test is not verified until its + individual duration appears in the generated HTML and JUnit reports: + + ```bash + ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh -- --filter '' + ./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope Frontend + ./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope WindowsRust + ``` + +5. Run the broader affected validation required by `develop-lithe`. Report the + HTML report path, slowest changed tests, exact commands, and any suite that + could not run on the current platform. Open + `.artifacts/test-stability/index.html` to review failures, module health, and + performance warnings before handoff. + +## Exceptions + +Do not add a scanner exception merely to make the gate pass. If a real-time or +blocking primitive is unavoidable at a native synchronous boundary, keep it +off the cooperative executor, add a short local timeout, guarantee cleanup, and +place this annotation immediately above the relevant line: + +```text +test-stability: allow() reason: +``` + +The reason must describe the architectural constraint, not restate the code. +New exceptions require explicit mention in the handoff. diff --git a/.agents/skills/write-stable-tests/references/macos-swift.md b/.agents/skills/write-stable-tests/references/macos-swift.md new file mode 100644 index 000000000..83eeb2dba --- /dev/null +++ b/.agents/skills/write-stable-tests/references/macos-swift.md @@ -0,0 +1,39 @@ +# macOS and Swift Test Stability + +The macOS harness requires only Swift, zsh, and Node.js. Bun is not installed, +loaded, or invoked anywhere in this path. + +## Preferred synchronization + +- Prefer actor-owned state, checked continuations, `AsyncStream`, injected + clocks, and explicit callbacks over polling or sleeping. +- Keep `@MainActor` tests fully asynchronous. Never call a semaphore, condition + variable, blocking file read, or process wait from the main actor. +- When a synchronous production protocol forces a blocking test double, run it + only on the production-owned worker thread. Use `DispatchSemaphore.wait(timeout:)` + on both sides of the gate, propagate timeout state into an assertion, and + release the gate from `defer`. +- Cancellation tests must retain the task, trigger cancellation explicitly, + await termination, and verify owned resources were released. + +## Prohibited coordination + +- Bare `DispatchSemaphore.wait()` or condition waits without a deadline. +- `Thread.sleep`, `usleep`, or `Task.sleep` used to give background work time to + run. A zero-duration yield is still inferior to an observable event. +- `Task.detached { blockingWait() }.value`; this hides blocking and can exhaust + executor threads. +- `Process.waitUntilExit()` without a watchdog that terminates the process tree. +- `while` polling loops without a monotonic deadline and timeout diagnostic. + +## Timing and verification + +Use `./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh`. It forces serial execution so the +currently running test is unambiguous, records every Swift Testing/XCTest case, +warns about slow cases, and terminates the suite when one case exceeds its local +budget. Reports are written below `.artifacts/test-stability/`. Each run +produces JSON and raw logs for diagnosis, JUnit XML for CI tooling, and a +self-contained HTML report for module and performance review. + +Run a focused test while iterating, then run the affected target or complete +lane. Do not claim that a test was timed if it is absent from the report. diff --git a/.agents/skills/write-stable-tests/references/test-reporting.md b/.agents/skills/write-stable-tests/references/test-reporting.md new file mode 100644 index 000000000..1ce3dcb04 --- /dev/null +++ b/.agents/skills/write-stable-tests/references/test-reporting.md @@ -0,0 +1,42 @@ +# Test Stability Reports + +## Outputs + +Every timing runner writes these artifacts below `.artifacts/test-stability/`: + +- `index.html` is the current runner dashboard. CI regenerates it as a combined + dashboard after every selected lane in the job finishes. +- `.html` is a self-contained report for one Swift, Bun, or Rust lane. +- `.junit.xml` is the standard CI and IDE interchange report. +- `.json` is the normalized machine-readable timing source. +- `.log`, when available, contains the raw runner output. + +The CI workflows upload the complete directory even when a test step fails. +Download the `test-stability-*` artifact and open `index.html`; it does not need +a server or external assets. + +## Interpretation + +The dashboard separates correctness from performance: + +- `failed`, `error`, `timeout`, and `incomplete` are stability failures. +- A duration at or above `maxMs` exceeds the hard performance budget and is + emitted as a JUnit failure even when the underlying assertion passed. +- A duration at or above `warnMs` but below `maxMs` is a performance warning. + It remains a passing JUnit case but appears in the optimization queue. +- Faster passing cases are healthy. Skipped cases are reported separately. + +Module health is derived from Swift Testing suites, Bun JUnit classes, and Rust +crate/module paths. Do not infer a production bottleneck from one slow test. +First determine whether the cost is test setup, external I/O, subprocess work, +or the behavior under test, then optimize the owning boundary. + +## Regenerating HTML + +The timing runners regenerate their own reports automatically. To intentionally +rebuild a combined HTML and JUnit view from every JSON file in the directory, +run: + +```bash +node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs +``` diff --git a/.agents/skills/write-stable-tests/references/windows-and-rust.md b/.agents/skills/write-stable-tests/references/windows-and-rust.md new file mode 100644 index 000000000..60c8e165b --- /dev/null +++ b/.agents/skills/write-stable-tests/references/windows-and-rust.md @@ -0,0 +1,50 @@ +# Windows, TypeScript, and Rust Test Stability + +Bun is a Windows Frontend test-runtime dependency only. The report generator, +static verifier, Swift runner, and Rust runner are Node.js-only. + +## TypeScript and Bun + +- Inject timer functions or a scheduler and advance them manually. The + `ManualTimer` pattern in the nearby Windows tests is preferred for debounce, + cooldown, retry, and delayed-work behavior. +- Use deferred promises only when the test controls every resolution path. + Release or reject pending deferred work in teardown after an assertion fails. +- Flush a known microtask boundary with resolved promises when necessary. Do + not use real `setTimeout` calls to wait for application state. +- Never leave intervals, listeners, subscriptions, workers, or mocked native + operations active after a test. + +The Windows timing harness passes an explicit timeout to Bun and reads the +JUnit duration for every executed test case. + +## Rust + +- Prefer channels, barriers, and injected clocks to `thread::sleep`. Channel + receives used for coordination require `recv_timeout` or an equivalent + bounded operation. +- A thread may be joined only after a bounded signal proves that it reached a + terminating path. Keep cleanup capable of releasing all barriers and killing + child processes. +- Test subprocesses require a watchdog and process-tree termination. Reading to + EOF or calling `wait` is not a timeout strategy. +- Avoid shared global environment mutation. If unavoidable, serialize access + with an owned guard and restore the previous value in teardown. + +`./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope WindowsRust` and `-Scope SharedRust` +compile the selected Cargo tests once, enumerate the produced test binaries, +then run every test case individually with a process-level timeout and duration +report. One suite deadline covers compilation, enumeration, every test process, +and the clean cache retry; a timeout writes the completed records before the +runner exits. This isolation makes the exact hanging test visible. + +Every Bun and Rust lane writes JUnit XML plus a self-contained HTML dashboard +below `.artifacts/test-stability/`. The dashboard groups Rust cases by crate +module and Bun cases by their JUnit class or suite. + +## Windows verification + +Run the PowerShell harness in a real Windows environment. A macOS boundary +check does not verify Bun timers, Windows process termination, or native Rust +test execution. When using the Parallels guest, establish the user toolchain +paths before invoking the script as required by `debug-windows-on-parallels`. diff --git a/.agents/skills/write-stable-tests/scripts/generate-test-report.mjs b/.agents/skills/write-stable-tests/scripts/generate-test-report.mjs new file mode 100755 index 000000000..95884429b --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/generate-test-report.mjs @@ -0,0 +1,363 @@ +#!/usr/bin/env node + +import { + existsSync, + mkdirSync, + readFileSync, + readdirSync, + writeFileSync, +} from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const SCRIPT_DIRECTORY = path.dirname(fileURLToPath(import.meta.url)); +const REPOSITORY_ROOT = path.resolve(SCRIPT_DIRECTORY, "../../../.."); +const FAILURE_STATUSES = new Set(["failed", "error", "timeout", "incomplete"]); + +function escapeHTML(value) { + return String(value ?? "") + .replaceAll("&", "&") + .replaceAll("<", "<") + .replaceAll(">", ">") + .replaceAll('"', """) + .replaceAll("'", "'"); +} + +function escapeXML(value) { + return escapeHTML(value); +} + +function formatDuration(milliseconds) { + if (!Number.isFinite(milliseconds)) return "—"; + if (milliseconds < 1000) return `${Math.round(milliseconds)} ms`; + if (milliseconds < 60000) return `${(milliseconds / 1000).toFixed(2)} s`; + return `${(milliseconds / 60000).toFixed(1)} min`; +} + +function reportLabel(report) { + if (report.runner === "swift") return "macOS · Swift"; + if (report.runner === "bun") return "Windows · Frontend"; + if (report.runner === "rust") { + return report.package ? `Shared Rust · ${report.package}` : "Windows · Rust"; + } + return String(report.runner ?? "Unknown runner"); +} + +function rustSuite(test) { + const parts = String(test.name ?? "").split("::"); + const testsIndex = parts.indexOf("tests"); + const moduleParts = testsIndex > 0 ? parts.slice(0, testsIndex) : parts.slice(0, -1); + const module = moduleParts.slice(0, 3).join("::"); + return module ? `${test.target} / ${module}` : String(test.target ?? "Rust tests"); +} + +function inferredSuite(report, test) { + if (test.suite) return String(test.suite); + if (test.target) return rustSuite(test); + const combinedName = String(test.name ?? ""); + const separator = combinedName.lastIndexOf(" / "); + if (separator > 0) return combinedName.slice(0, separator); + return `${report.runner ?? "test"} / uncategorized`; +} + +function normalizedEntries(reportEntries) { + return reportEntries.flatMap(({ source, report }) => { + const warnMs = Number(report.warnMs ?? 1000); + const maxMs = Number(report.maxMs ?? 15000); + return report.tests.map((test) => { + const durationMs = Number(test.durationMs ?? 0); + const status = String(test.status ?? "unknown").toLowerCase(); + const failed = FAILURE_STATUSES.has(status); + const overBudget = durationMs >= maxMs; + const slow = durationMs >= warnMs; + return { + source, + runner: report.runner, + reportLabel: reportLabel(report), + suite: inferredSuite(report, test), + name: String(test.name ?? "unknown test"), + status, + durationMs, + warnMs, + maxMs, + failed, + overBudget, + slow, + details: String(test.details ?? ""), + }; + }); + }); +} + +function summarize(tests) { + return { + total: tests.length, + passed: tests.filter((test) => test.status === "passed" && !test.overBudget).length, + failed: tests.filter((test) => test.failed).length, + skipped: tests.filter((test) => test.status === "skipped").length, + overBudget: tests.filter((test) => test.overBudget).length, + issues: tests.filter((test) => test.failed || test.overBudget).length, + slow: tests.filter((test) => test.slow && !test.overBudget).length, + durationMs: tests.reduce((total, test) => total + test.durationMs, 0), + }; +} + +function statusPresentation(test) { + if (["timeout", "incomplete"].includes(test.status)) return ["critical", test.status]; + if (test.failed) return ["critical", "failed"]; + if (test.overBudget) return ["critical", "over budget"]; + if (test.slow) return ["warning", "slow"]; + if (test.status === "skipped") return ["muted", "skipped"]; + return ["success", "passed"]; +} + +function moduleRows(tests) { + const modules = new Map(); + for (const test of tests) { + const key = `${test.reportLabel}\u0000${test.suite}`; + const module = modules.get(key) ?? { + reportLabel: test.reportLabel, + suite: test.suite, + tests: [], + }; + module.tests.push(test); + modules.set(key, module); + } + return [...modules.values()] + .map((module) => ({ ...module, summary: summarize(module.tests) })) + .sort((left, right) => { + return right.summary.issues - left.summary.issues + || right.summary.durationMs - left.summary.durationMs; + }); +} + +function summaryCard(label, value, tone, hint) { + return `
+ ${escapeHTML(label)} + ${escapeHTML(value)} + ${escapeHTML(hint)} +
`; +} + +function testTableRows(tests) { + return tests.map((test) => { + const [tone, label] = statusPresentation(test); + const ratio = test.warnMs > 0 ? Math.round((test.durationMs / test.warnMs) * 100) : 0; + return ` + ${escapeHTML(label)} + ${escapeHTML(test.name)}${ + test.details ? `
错误详情
${escapeHTML(test.details)}
` : "" + } + ${escapeHTML(test.suite)} + ${escapeHTML(test.reportLabel)} + ${escapeHTML(formatDuration(test.durationMs))} + ${ratio}% + `; + }).join("\n"); +} + +function performanceRows(tests) { + return tests + .filter((test) => test.slow || test.failed) + .sort((left, right) => { + const leftPriority = Number(left.failed || left.overBudget); + const rightPriority = Number(right.failed || right.overBudget); + return rightPriority - leftPriority || right.durationMs - left.durationMs; + }) + .map((test) => { + const [tone, label] = statusPresentation(test); + const threshold = test.warnMs > 0 ? test.durationMs / test.warnMs : 0; + const recommendation = test.failed + ? "先检查失败日志与资源清理路径" + : test.overBudget + ? "已越过硬预算，应拆分外部 I/O、进程或高成本初始化" + : "超过预警线，检查可复用 setup 与不必要的集成边界"; + return ` + ${escapeHTML(label)} + ${escapeHTML(test.name)}${escapeHTML(test.suite)} + ${escapeHTML(formatDuration(test.durationMs))} + ${threshold.toFixed(1)}× + ${escapeHTML(recommendation)} + `; + }).join("\n"); +} + +function sourceLinks(reportEntries) { + return reportEntries.map(({ source }) => { + const stem = source.replace(/\.json$/i, ""); + return `
  • ${escapeHTML(stem)}JSON · JUnit XML${ + existsSync(path.join(path.dirname(reportEntries[0].path), `${stem}.log`)) + ? ` · 原始日志` + : "" + }
    • `; + }).join("\n"); +} + +export function renderHTML(reportEntries, title = "Lithe Test Stability Report") { + const tests = normalizedEntries(reportEntries); + const summary = summarize(tests); + const modules = moduleRows(tests); + const issueCount = summary.issues; + const overallTone = issueCount > 0 ? "critical" : summary.slow > 0 ? "warning" : "success"; + const overallLabel = issueCount > 0 ? "需要处理" : summary.slow > 0 ? "通过，有性能预警" : "全部通过"; + const generatedAt = new Date().toISOString(); + const moduleHTML = modules.map((module) => { + const issues = module.summary.issues; + const tone = issues > 0 ? "critical" : module.summary.slow > 0 ? "warning" : "success"; + return ` + ${escapeHTML(module.suite)}${escapeHTML(module.reportLabel)} + ${module.summary.total} + ${issues} + ${module.summary.slow} + ${escapeHTML(formatDuration(module.summary.durationMs))} + `; + }).join("\n"); + + return ` + + + + + ${escapeHTML(title)} + + + +
    +

    ${escapeHTML(title)}

    生成时间 ${escapeHTML(generatedAt)} · 慢测试以各 runner 的 warn/max 预算判定

    ${overallLabel}
    +
    + ${summaryCard("测试总数", summary.total, "", "全部 runner")} + ${summaryCard("正常通过", summary.passed, "success", "未触发性能预警")} + ${summaryCard("失败 / 超时", summary.failed, summary.failed ? "critical" : "", "功能或稳定性问题")} + ${summaryCard("超过硬预算", summary.overBudget, summary.overBudget ? "critical" : "", "会导致测试门禁失败")} + ${summaryCard("性能预警", summary.slow, summary.slow ? "warning" : "", "超过 warn，尚未超过 max")} + ${summaryCard("累计测试耗时", formatDuration(summary.durationMs), "", "逐测试耗时之和")} +
    +

    模块健康度

    ${moduleHTML || ''}
    模块 / Suite测试问题慢测试耗时
    没有测试数据
    +

    问题与性能优化队列

    ${performanceRows(tests) || ''}
    等级测试耗时预警线比例建议
    没有失败、超时或性能预警
    +

    全部测试

    ${testTableRows(tests)}
    状态测试模块Runner耗时预警线比例
    +

    原始报告

      ${sourceLinks(reportEntries)}
    +
    + + +\n`; +} + +export function renderJUnitXML({ source, report }) { + const tests = normalizedEntries([{ source, report }]); + const errorStatuses = new Set(["error", "timeout", "incomplete"]); + const failures = tests.filter( + (test) => !errorStatuses.has(test.status) && (test.status === "failed" || test.overBudget), + ).length; + const errors = tests.filter((test) => errorStatuses.has(test.status)).length; + const skipped = tests.filter((test) => test.status === "skipped").length; + const totalSeconds = tests.reduce((total, test) => total + test.durationMs, 0) / 1000; + const cases = tests.map((test) => { + let outcome = ""; + if (["error", "timeout", "incomplete"].includes(test.status)) { + outcome = ``; + } else if (test.status === "failed" || test.overBudget) { + const message = test.overBudget + ? `Performance budget exceeded: ${test.durationMs}ms >= ${test.maxMs}ms` + : test.details || "Test failed"; + outcome = `${escapeXML(test.details)}`; + } else if (test.status === "skipped") { + outcome = ""; + } + return ` ${outcome}`; + }).join("\n"); + return ` + + +${cases} + +\n`; +} + +function loadReport(reportPath) { + const report = JSON.parse(readFileSync(reportPath, "utf8")); + if (!report.runner || !Array.isArray(report.tests)) { + throw new Error(`Unsupported test report: ${reportPath}`); + } + return { path: reportPath, source: path.basename(reportPath), report }; +} + +function reportPathsIn(directory) { + if (!existsSync(directory)) return []; + return readdirSync(directory) + .filter((name) => name.endsWith(".json")) + .map((name) => path.join(directory, name)) + .sort(); +} + +export function writeTestReportArtifacts(reportPath) { + const directory = path.dirname(reportPath); + const entry = loadReport(reportPath); + const stem = entry.path.replace(/\.json$/i, ""); + writeFileSync(`${stem}.junit.xml`, renderJUnitXML(entry)); + writeFileSync(`${stem}.html`, renderHTML([entry], `${reportLabel(entry.report)} Test Report`)); + const indexPath = path.join(directory, "index.html"); + // A runner must not silently merge stale local reports into the current + // result. CI performs an explicit directory aggregation after all lanes. + writeFileSync(indexPath, renderHTML([entry])); + console.log(`HTML test report: ${indexPath}`); + return indexPath; +} + +function parseArguments(arguments_) { + const options = { + inputDirectory: path.join(REPOSITORY_ROOT, ".artifacts/test-stability"), + output: null, + title: "Lithe Test Stability Report", + }; + for (let index = 0; index < arguments_.length; index += 1) { + const argument = arguments_[index]; + if (argument === "--input-dir") options.inputDirectory = path.resolve(arguments_[++index]); + else if (argument === "--output") options.output = path.resolve(arguments_[++index]); + else if (argument === "--title") options.title = arguments_[++index]; + else throw new Error(`Unknown argument: ${argument}`); + } + options.output ??= path.join(options.inputDirectory, "index.html"); + return options; +} + +function main() { + try { + const options = parseArguments(process.argv.slice(2)); + const entries = reportPathsIn(options.inputDirectory).map(loadReport); + if (entries.length === 0) throw new Error(`No JSON test reports found in ${options.inputDirectory}.`); + mkdirSync(path.dirname(options.output), { recursive: true }); + for (const entry of entries) { + writeFileSync(entry.path.replace(/\.json$/i, ".junit.xml"), renderJUnitXML(entry)); + } + writeFileSync(options.output, renderHTML(entries, options.title)); + console.log(`HTML test report: ${options.output}`); + } catch (error) { + console.error(`Test report generation failed: ${error.message}`); + process.exitCode = 1; + } +} + +if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) main(); diff --git a/.agents/skills/write-stable-tests/scripts/parse-junit-cases.mjs b/.agents/skills/write-stable-tests/scripts/parse-junit-cases.mjs new file mode 100644 index 000000000..b54a193ea --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/parse-junit-cases.mjs @@ -0,0 +1,38 @@ +export function decodeXML(value) { + return value + .replaceAll(""", '"') + .replaceAll("'", "'") + .replaceAll("<", "<") + .replaceAll(">", ">") + .replaceAll("&", "&"); +} + +export function parseJUnitCases(xml) { + const cases = []; + const pattern = /]*?)(?:\/>|>[\s\S]*?<\/testcase>)/g; + for (const match of xml.matchAll(pattern)) { + const attributes = match[1]; + const name = attributes.match(/\bname="([^"]*)"/)?.[1] ?? "unknown"; + const className = attributes.match(/\bclassname="([^"]*)"/)?.[1] ?? ""; + const seconds = Number(attributes.match(/\btime="([0-9.]+)"/)?.[1] ?? 0); + const body = match[0]; + const status = body.includes("]*>([\s\S]*?)<\/(?:failure|error)>/); + const details = detailsMatch + ? decodeXML(detailsMatch[1].replace(//g, "$1").replace(/<[^>]+>/g, "").trim()) + : ""; + cases.push({ + name: decodeXML(className ? `${className} / ${name}` : name), + status, + durationMs: Math.round(seconds * 1000), + ...(details ? { details } : {}), + }); + } + return cases; +} diff --git a/.agents/skills/write-stable-tests/scripts/run-bun-tests-with-timing.mjs b/.agents/skills/write-stable-tests/scripts/run-bun-tests-with-timing.mjs new file mode 100755 index 000000000..9438913c7 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/run-bun-tests-with-timing.mjs @@ -0,0 +1,115 @@ +#!/usr/bin/env node + +import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { writeTestReportArtifacts } from "./generate-test-report.mjs"; +import { parseJUnitCases } from "./parse-junit-cases.mjs"; +import { positiveInteger, runProcess } from "./test-timing-lib.mjs"; + +const SCRIPT_DIRECTORY = path.dirname(fileURLToPath(import.meta.url)); +const REPOSITORY_ROOT = path.resolve(SCRIPT_DIRECTORY, "../../../.."); + +function parseArguments(arguments_) { + const separator = arguments_.indexOf("--"); + const options = { + workingDirectory: path.join(REPOSITORY_ROOT, "windows/tauri"), + warnMs: 1000, + maxMs: 15000, + suiteTimeoutMs: 600000, + report: path.join(REPOSITORY_ROOT, ".artifacts/test-stability/windows-frontend.json"), + testArguments: separator >= 0 ? arguments_.slice(separator + 1) : [], + }; + const limit = separator >= 0 ? separator : arguments_.length; + for (let index = 0; index < limit; index += 1) { + const argument = arguments_[index]; + if (argument === "--working-directory") options.workingDirectory = path.resolve(arguments_[++index]); + else if (argument === "--warn-ms") options.warnMs = positiveInteger(arguments_[++index], "--warn-ms"); + else if (argument === "--max-ms") options.maxMs = positiveInteger(arguments_[++index], "--max-ms"); + else if (argument === "--suite-timeout-ms") { + options.suiteTimeoutMs = positiveInteger(arguments_[++index], "--suite-timeout-ms"); + } else if (argument === "--report") options.report = path.resolve(arguments_[++index]); + else throw new Error(`Unknown argument: ${argument}`); + } + if (options.warnMs >= options.maxMs) throw new Error("--warn-ms must be lower than --max-ms."); + return options; +} + +export { parseJUnitCases } from "./parse-junit-cases.mjs"; + +function writeReport(options, result, tests) { + const report = { + schemaVersion: 1, + runner: "bun", + warnMs: options.warnMs, + maxMs: options.maxMs, + suiteTimeoutMs: options.suiteTimeoutMs, + processDurationMs: Math.round(result.durationMs), + tests, + }; + writeFileSync(options.report, `${JSON.stringify(report, null, 2)}\n`); + writeTestReportArtifacts(options.report); +} + +export async function run(options, { runProcessImpl = runProcess } = {}) { + mkdirSync(path.dirname(options.report), { recursive: true }); + const junitPath = options.report.replace(/\.json$/i, ".junit.xml"); + rmSync(junitPath, { force: true }); + const arguments_ = [ + "test", + ...options.testArguments, + "--timeout", + String(options.maxMs), + "--reporter=junit", + `--reporter-outfile=${junitPath}`, + ]; + const result = await runProcessImpl({ + command: "bun", + args: arguments_, + cwd: options.workingDirectory, + timeoutMs: options.suiteTimeoutMs, + streamStdout: true, + streamStderr: true, + }); + let tests = []; + try { + tests = parseJUnitCases(readFileSync(junitPath, "utf8")); + } catch (error) { + if (!result.timedOut) { + throw new Error(`Bun did not produce a readable JUnit report: ${error.message}`); + } + } + if (result.timedOut) { + tests.push({ + name: "Bun test suite timeout", + suite: "Bun test runner", + status: "timeout", + durationMs: options.suiteTimeoutMs, + details: `Bun test suite exceeded the shared ${options.suiteTimeoutMs}ms deadline.`, + }); + } + writeReport(options, result, tests); + console.log(`Recorded ${tests.length} Bun test duration(s) in ${options.report}`); + for (const test of tests + .filter((value) => value.durationMs >= options.warnMs) + .sort((left, right) => right.durationMs - left.durationMs) + .slice(0, 10)) { + console.log(`SLOW ${test.durationMs}ms ${test.name}`); + } + if (result.timedOut) throw new Error(`Bun test suite exceeded ${options.suiteTimeoutMs}ms.`); + if (tests.length === 0) throw new Error("The Bun runner did not report any individual test durations."); + if (result.code !== 0) throw new Error(`Bun test command exited with code ${result.code}.`); + const overBudget = tests.filter((test) => test.durationMs >= options.maxMs || test.status === "failed"); + if (overBudget.length > 0) throw new Error(`${overBudget.length} Bun test(s) failed or exceeded the local budget.`); +} + +async function main() { + try { + await run(parseArguments(process.argv.slice(2))); + } catch (error) { + console.error(`Bun test timing failed: ${error.message}`); + process.exitCode = 1; + } +} + +if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) await main(); diff --git a/.agents/skills/write-stable-tests/scripts/run-rust-tests-with-timing.mjs b/.agents/skills/write-stable-tests/scripts/run-rust-tests-with-timing.mjs new file mode 100755 index 000000000..9266adedb --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/run-rust-tests-with-timing.mjs @@ -0,0 +1,316 @@ +#!/usr/bin/env node + +import { appendFileSync, mkdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { writeTestReportArtifacts } from "./generate-test-report.mjs"; +import { positiveInteger, runProcess } from "./test-timing-lib.mjs"; + +const SCRIPT_DIRECTORY = path.dirname(fileURLToPath(import.meta.url)); +const REPOSITORY_ROOT = path.resolve(SCRIPT_DIRECTORY, "../../../.."); + +class SuiteTimeoutError extends Error { + constructor(stage) { + super(`Rust test suite exceeded its deadline during ${stage}.`); + this.stage = stage; + } +} + +function parseArguments(arguments_) { + const options = { + manifest: null, + package: null, + warnMs: 1000, + maxMs: 15000, + buildTimeoutMs: 1200000, + suiteTimeoutMs: 1200000, + report: null, + keepGoing: false, + }; + for (let index = 0; index < arguments_.length; index += 1) { + const argument = arguments_[index]; + if (argument === "--manifest") options.manifest = arguments_[++index]; + else if (argument === "--package") options.package = arguments_[++index]; + else if (argument === "--warn-ms") options.warnMs = positiveInteger(arguments_[++index], "--warn-ms"); + else if (argument === "--max-ms") options.maxMs = positiveInteger(arguments_[++index], "--max-ms"); + else if (argument === "--build-timeout-ms") { + options.buildTimeoutMs = positiveInteger(arguments_[++index], "--build-timeout-ms"); + } else if (argument === "--suite-timeout-ms") { + options.suiteTimeoutMs = positiveInteger(arguments_[++index], "--suite-timeout-ms"); + } else if (argument === "--report") options.report = path.resolve(arguments_[++index]); + else if (argument === "--keep-going") options.keepGoing = true; + else throw new Error(`Unknown argument: ${argument}`); + } + if (!options.manifest) throw new Error("--manifest is required."); + options.manifest = path.resolve(REPOSITORY_ROOT, options.manifest); + options.report ??= path.join(REPOSITORY_ROOT, ".artifacts/test-stability/rust-tests.json"); + if (options.warnMs >= options.maxMs) throw new Error("--warn-ms must be lower than --max-ms."); + return options; +} + +function artifactFromLine(line) { + try { + const value = JSON.parse(line); + if (value.reason !== "compiler-artifact" || !value.profile?.test || !value.executable) return null; + return { + executable: value.executable, + manifestPath: value.manifest_path, + target: value.target?.name ?? path.basename(value.executable), + }; + } catch { + return null; + } +} + +function compilerMessageFromLine(line) { + try { + const value = JSON.parse(line); + + if ( + value.reason !== "compiler-message" || + typeof value.message?.rendered !== "string" + ) { + return null; + } + + const rendered = value.message.rendered.trim(); + + return rendered.length > 0 ? rendered : null; + } catch { + return null; + } +} + +export async function run( + options, + { runProcessImpl = runProcess, now = () => performance.now() } = {}, +) { + mkdirSync(path.dirname(options.report), { recursive: true }); + const logPath = options.report.replace(/\.json$/i, ".log"); + writeFileSync(logPath, ""); + const artifacts = new Map(); + const records = []; + const compilerMessages = []; + const suiteStartedAt = now(); + const suiteDeadline = suiteStartedAt + options.suiteTimeoutMs; + let buildDurationMs = 0; + let suiteTimedOutStage = null; + let reportWritten = false; + + const remainingSuiteMilliseconds = () => Math.floor(suiteDeadline - now()); + const timeoutBudget = (stage, stageTimeoutMs) => { + const remainingMs = remainingSuiteMilliseconds(); + if (remainingMs <= 0) throw new SuiteTimeoutError(stage); + return { + timeoutMs: Math.max(1, Math.min(stageTimeoutMs, remainingMs)), + limitedBySuite: remainingMs <= stageTimeoutMs, + }; + }; + const recordSuiteTimeout = (stage, details) => { + suiteTimedOutStage = stage; + records.push({ + target: options.package ?? "rust-suite", + name: `suite deadline during ${stage}`, + status: "timeout", + durationMs: Math.max(0, Math.round(now() - suiteStartedAt)), + details, + }); + }; + const writeReport = () => { + if (reportWritten) return; + const report = { + schemaVersion: 1, + runner: "rust", + manifest: options.manifest, + package: options.package, + warnMs: options.warnMs, + maxMs: options.maxMs, + suiteTimeoutMs: options.suiteTimeoutMs, + buildDurationMs, + suite: { + timedOut: suiteTimedOutStage !== null, + stage: suiteTimedOutStage, + durationMs: Math.max(0, Math.round(now() - suiteStartedAt)), + }, + tests: records, + }; + writeFileSync(options.report, `${JSON.stringify(report, null, 2)}\n`); + writeTestReportArtifacts(options.report); + reportWritten = true; + }; + + try { + const cargoArguments = [ + "test", + "--manifest-path", + options.manifest, + "--no-run", + "--message-format=json", + ]; + if (options.package) cargoArguments.push("--package", options.package); + + console.log(`Compiling Rust tests from ${options.manifest}`); + const buildBudget = timeoutBudget("compilation", options.buildTimeoutMs); + const build = await runProcessImpl({ + command: "cargo", + args: cargoArguments, + cwd: REPOSITORY_ROOT, + timeoutMs: buildBudget.timeoutMs, + onStdoutLine: (line) => { + const artifact = artifactFromLine(line); + if (artifact) artifacts.set(artifact.executable, artifact); + + const compilerMessage = compilerMessageFromLine(line); + if (compilerMessage) compilerMessages.push(compilerMessage); + }, + streamStderr: true, + }); + buildDurationMs = Math.round(build.durationMs); + + const compilerOutput = compilerMessages.join("\n").trim(); + + if (compilerOutput) { + appendFileSync(logPath, `${compilerOutput}\n`); + } + + appendFileSync(logPath, build.stderr); + if (build.timedOut && buildBudget.limitedBySuite) throw new SuiteTimeoutError("compilation"); + if (build.timedOut) throw new Error(`Cargo test compilation exceeded ${options.buildTimeoutMs}ms.`); + if (build.code !== 0) { + const details = [ + compilerOutput, + build.stderr.trim(), + ] + .filter(Boolean) + .join("\n") + .slice(0, 8000); + + records.push({ + target: options.package ?? "rust-suite", + name: "Cargo test compilation", + status: "failed", + durationMs: buildDurationMs, + ...(details ? { details } : {}), + }); + + writeReport(); + + throw new Error( + `Cargo test compilation exited with code ${build.code}.`, + ); + } + if (artifacts.size === 0) throw new Error("Cargo did not produce any test executables."); + + let shouldStop = false; + for (const artifact of artifacts.values()) { + const cwd = path.dirname(artifact.manifestPath); + const enumerationTimeoutMs = Math.min(Math.max(options.maxMs, 5000), 10000); + const enumerationBudget = timeoutBudget( + `test enumeration for ${artifact.target}`, + enumerationTimeoutMs, + ); + const listed = await runProcessImpl({ + command: artifact.executable, + args: ["--list", "--color", "never"], + cwd, + timeoutMs: enumerationBudget.timeoutMs, + }); + if (listed.timedOut && enumerationBudget.limitedBySuite) { + throw new SuiteTimeoutError(`test enumeration for ${artifact.target}`); + } + if (listed.timedOut || listed.code !== 0) { + throw new Error( + `Could not enumerate Rust tests in ${artifact.target}.\n${listed.stdout}${listed.stderr}`, + ); + } + const tests = listed.stdout + .split(/\r?\n/) + .map((line) => line.match(/^(.*): test$/)?.[1]) + .filter(Boolean); + + for (const testName of tests) { + console.log(`RUN ${artifact.target}::${testName}`); + const testBudget = timeoutBudget(`test ${artifact.target}::${testName}`, options.maxMs); + const result = await runProcessImpl({ + command: artifact.executable, + args: ["--exact", testName, "--test-threads", "1", "--color", "never"], + cwd, + timeoutMs: testBudget.timeoutMs, + }); + const suiteDeadlineReached = result.timedOut && testBudget.limitedBySuite; + const status = result.timedOut + ? "timeout" + : result.code === 0 && /running 0 tests/.test(result.stdout) + ? "skipped" + : result.code === 0 + ? "passed" + : "failed"; + const durationMs = Math.round(result.durationMs); + records.push({ + target: artifact.target, + name: testName, + status, + durationMs, + ...(suiteDeadlineReached + ? { details: `The shared suite deadline expired while this test was running.\n${result.stdout}${result.stderr}`.trim().slice(0, 8000) } + : !["passed", "skipped"].includes(status) + ? { details: `${result.stdout}${result.stderr}`.trim().slice(0, 8000) } + : {}), + }); + appendFileSync( + logPath, + `\n=== ${artifact.target}::${testName} (${status}, ${durationMs}ms) ===\n${result.stdout}${result.stderr}`, + ); + const statusLabel = status === "passed" ? "PASS" : status === "skipped" ? "SKIP" : "FAIL"; + console.log(`${statusLabel} ${durationMs}ms ${artifact.target}::${testName}`); + if (suiteDeadlineReached) { + suiteTimedOutStage = `test ${artifact.target}::${testName}`; + shouldStop = true; + break; + } + if (!["passed", "skipped"].includes(status) && !options.keepGoing) { + shouldStop = true; + break; + } + } + if (shouldStop) break; + } + + writeReport(); + console.log(`Recorded ${records.length} Rust test duration(s) in ${options.report}`); + for (const record of records + .filter((value) => value.durationMs >= options.warnMs) + .sort((left, right) => right.durationMs - left.durationMs) + .slice(0, 10)) { + console.log(`SLOW ${record.durationMs}ms ${record.target}::${record.name}`); + } + if (suiteTimedOutStage) { + throw new Error( + `Rust test suite exceeded ${options.suiteTimeoutMs}ms during ${suiteTimedOutStage}.`, + ); + } + if (records.length === 0) throw new Error("No Rust tests were enumerated."); + const failures = records.filter((record) => !["passed", "skipped"].includes(record.status)); + if (failures.length > 0) { + throw new Error(`${failures.length} Rust test(s) failed or exceeded ${options.maxMs}ms.`); + } + } catch (error) { + if (error instanceof SuiteTimeoutError) { + recordSuiteTimeout(error.stage, error.message); + writeReport(); + throw new Error(`Rust test suite exceeded ${options.suiteTimeoutMs}ms during ${error.stage}.`); + } + throw error; + } +} + +async function main() { + try { + await run(parseArguments(process.argv.slice(2))); + } catch (error) { + console.error(`Rust test timing failed: ${error.message}`); + process.exitCode = 1; + } +} + +if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) await main(); diff --git a/.agents/skills/write-stable-tests/scripts/run-swift-tests-with-timing.mjs b/.agents/skills/write-stable-tests/scripts/run-swift-tests-with-timing.mjs new file mode 100755 index 000000000..a0e1b7cde --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/run-swift-tests-with-timing.mjs @@ -0,0 +1,228 @@ +#!/usr/bin/env node + +import { mkdirSync, writeFileSync, createWriteStream } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { writeTestReportArtifacts } from "./generate-test-report.mjs"; +import { positiveInteger, runProcess } from "./test-timing-lib.mjs"; + +const SCRIPT_DIRECTORY = path.dirname(fileURLToPath(import.meta.url)); +const REPOSITORY_ROOT = path.resolve(SCRIPT_DIRECTORY, "../../../.."); + +export function parseSwiftTimingLine(line) { + const plain = line.replace(/\u001B\[[0-9;]*m/g, ""); + const swiftStart = plain.match(/(?:^|\s)Test (?!run\b|case\b)(.+?) started\.$/); + if (swiftStart) { + return { + name: swiftStart[1], + event: "started", + durationMs: null, + caseCount: null, + }; + } + const swiftFinish = plain.match( + /(?:^|\s)Test (?!run\b|case\b)(.+?)(?: with (\d+) test cases)? (passed|failed|skipped) after ([0-9.]+) seconds\.$/, + ); + if (swiftFinish) { + return { + name: swiftFinish[1], + event: swiftFinish[3], + durationMs: Number(swiftFinish[4]) * 1000, + caseCount: swiftFinish[2] ? Number(swiftFinish[2]) : null, + }; + } + const xctestStart = plain.match(/^Test Case '(.+)' started\.$/); + if (xctestStart) { + return { name: xctestStart[1], event: "started", durationMs: null, caseCount: null }; + } + const xctestFinish = plain.match(/^Test Case '(.+)' (passed|failed) \(([0-9.]+) seconds\)\.$/); + if (xctestFinish) { + return { + name: xctestFinish[1], + event: xctestFinish[2], + durationMs: Number(xctestFinish[3]) * 1000, + caseCount: null, + }; + } + return null; +} + +export function parseSwiftSuiteLine(line) { + const plain = line.replace(/\u001B\[[0-9;]*m/g, ""); + const event = plain.match(/(?:^|\s)Suite (.+?) (started|passed|failed|skipped)(?: after [0-9.]+ seconds)?\.$/); + if (!event) return null; + return { + name: event[1].replace(/^(["'])(.*)\1$/, "$2"), + event: event[2], + }; +} + +function parseArguments(arguments_) { + const separator = arguments_.indexOf("--"); + if (separator < 0 || separator === arguments_.length - 1) { + throw new Error("Provide the test command after --."); + } + const options = { + warnMs: 1000, + maxMs: 15000, + suiteTimeoutMs: 600000, + report: path.join(REPOSITORY_ROOT, ".artifacts/test-stability/macos-swift.json"), + command: arguments_[separator + 1], + commandArguments: arguments_.slice(separator + 2), + }; + for (let index = 0; index < separator; index += 1) { + const argument = arguments_[index]; + if (argument === "--warn-ms") options.warnMs = positiveInteger(arguments_[++index], "--warn-ms"); + else if (argument === "--max-ms") options.maxMs = positiveInteger(arguments_[++index], "--max-ms"); + else if (argument === "--suite-timeout-ms") { + options.suiteTimeoutMs = positiveInteger(arguments_[++index], "--suite-timeout-ms"); + } else if (argument === "--report") options.report = path.resolve(arguments_[++index]); + else throw new Error(`Unknown argument: ${argument}`); + } + if (options.warnMs >= options.maxMs) throw new Error("--warn-ms must be lower than --max-ms."); + return options; +} + +export async function run(options, { runProcessImpl = runProcess } = {}) { + mkdirSync(path.dirname(options.report), { recursive: true }); + const logPath = options.report.replace(/\.json$/i, ".log"); + const log = createWriteStream(logPath, { flags: "w" }); + const active = new Map(); + const records = []; + let currentSuite = null; + let timedOutTest = null; + let testTimer = null; + let terminateChild = () => {}; + + const recordLine = (line, stream) => { + log.write(`${stream}: ${line}\n`); + const suiteEvent = parseSwiftSuiteLine(line); + if (suiteEvent?.event === "started") currentSuite = suiteEvent.name; + else if (suiteEvent && suiteEvent.name === currentSuite) currentSuite = null; + const event = parseSwiftTimingLine(line); + if (!event) return; + if (event.event === "started") { + const startedAt = performance.now(); + active.set(event.name, { startedAt, suite: currentSuite }); + if (testTimer) clearTimeout(testTimer); + testTimer = setTimeout(() => { + timedOutTest = { name: event.name, suite: currentSuite }; + terminateChild(); + }, options.maxMs); + return; + } + + const activeTest = active.get(event.name); + active.delete(event.name); + if (testTimer) { + clearTimeout(testTimer); + testTimer = null; + } + records.push({ + name: event.name, + ...(activeTest?.suite ? { suite: activeTest.suite } : {}), + status: event.event, + durationMs: Math.round( + event.durationMs ?? (activeTest ? performance.now() - activeTest.startedAt : 0), + ), + ...(event.caseCount ? { caseCount: event.caseCount } : {}), + }); + }; + + const startedAt = new Date().toISOString(); + const childPromise = runProcessImpl({ + command: options.command, + args: options.commandArguments, + cwd: REPOSITORY_ROOT, + timeoutMs: options.suiteTimeoutMs, + onSpawn: ({ terminate }) => { + terminateChild = terminate; + }, + onStdoutLine: (line) => recordLine(line, "stdout"), + onStderrLine: (line) => recordLine(line, "stderr"), + streamStdout: true, + streamStderr: true, + }); + + const result = await childPromise; + if (testTimer) clearTimeout(testTimer); + await new Promise((resolve, reject) => { + log.once("error", reject); + log.end(resolve); + }); + + if (timedOutTest && !records.some((record) => record.name === timedOutTest.name)) { + records.push({ + name: timedOutTest.name, + ...(timedOutTest.suite ? { suite: timedOutTest.suite } : {}), + status: "timeout", + durationMs: options.maxMs, + }); + } + for (const [name, activeTest] of active) { + if (!records.some((record) => record.name === name)) { + records.push({ + name, + ...(activeTest.suite ? { suite: activeTest.suite } : {}), + status: result.timedOut ? "timeout" : "incomplete", + durationMs: options.maxMs, + }); + } + } + if (result.timedOut && !records.some((record) => record.status === "timeout")) { + records.push({ + name: "Swift test suite timeout", + suite: "Swift test runner", + status: "timeout", + durationMs: options.suiteTimeoutMs, + details: `Swift test suite exceeded the shared ${options.suiteTimeoutMs}ms deadline before reporting a test duration.`, + }); + } + + const slow = records.filter((record) => record.durationMs >= options.warnMs); + const overBudget = records.filter( + (record) => record.durationMs >= options.maxMs || ["timeout", "incomplete"].includes(record.status), + ); + const report = { + schemaVersion: 1, + runner: "swift", + startedAt, + command: [options.command, ...options.commandArguments], + warnMs: options.warnMs, + maxMs: options.maxMs, + suiteTimeoutMs: options.suiteTimeoutMs, + process: { + exitCode: result.code, + signal: result.signal, + timedOut: result.timedOut, + durationMs: Math.round(result.durationMs), + }, + tests: records, + }; + writeFileSync(options.report, `${JSON.stringify(report, null, 2)}\n`); + writeTestReportArtifacts(options.report); + + console.log(`\nRecorded ${records.length} Swift test duration(s) in ${options.report}`); + for (const record of [...slow].sort((left, right) => right.durationMs - left.durationMs).slice(0, 10)) { + console.log(`SLOW ${record.durationMs}ms ${record.name}`); + } + + if (timedOutTest) { + throw new Error(`Swift test exceeded ${options.maxMs}ms: ${timedOutTest.name}`); + } + if (result.timedOut) throw new Error(`Swift test suite exceeded ${options.suiteTimeoutMs}ms.`); + if (records.length === 0) throw new Error("The Swift runner did not report any individual test durations."); + if (overBudget.length > 0) throw new Error(`${overBudget.length} Swift test(s) exceeded the local budget.`); + if (result.code !== 0) throw new Error(`Swift test command exited with code ${result.code}.`); +} + +async function main() { + try { + await run(parseArguments(process.argv.slice(2))); + } catch (error) { + console.error(`Test timing failed: ${error.message}`); + process.exitCode = 1; + } +} + +if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) await main(); diff --git a/.agents/skills/write-stable-tests/scripts/test-stability-macos.sh b/.agents/skills/write-stable-tests/scripts/test-stability-macos.sh new file mode 100755 index 000000000..d4a9e4cc6 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/test-stability-macos.sh @@ -0,0 +1,62 @@ +#!/bin/zsh +set -euo pipefail + +SCRIPT_DIR="${0:A:h}" +ROOT_DIR="$(cd -- "$SCRIPT_DIR/../../../.." && pwd)" +WARN_SECONDS=1 +MAX_SECONDS=15 +SUITE_TIMEOUT_SECONDS=600 +REPORT="$ROOT_DIR/.artifacts/test-stability/macos-swift.json" +SWIFT_ARGS=() + +while (( $# > 0 )); do + case "$1" in + --warn-seconds) + WARN_SECONDS="$2" + shift 2 + ;; + --max-seconds) + MAX_SECONDS="$2" + shift 2 + ;; + --suite-timeout-seconds) + SUITE_TIMEOUT_SECONDS="$2" + shift 2 + ;; + --report) + REPORT="$2" + shift 2 + ;; + --) + shift + SWIFT_ARGS=("$@") + break + ;; + *) + print -u2 -- "Unknown argument: $1" + exit 2 + ;; + esac +done + +for command in node swift; do + if ! command -v "$command" >/dev/null 2>&1; then + print -u2 -- "macOS test stability requires $command; Bun is not required." + exit 127 + fi +done + +for argument in "${SWIFT_ARGS[@]}"; do + if [[ "$argument" == "--parallel" ]]; then + print -u2 -- "--parallel is not allowed: per-test watchdog attribution requires serial execution." + exit 2 + fi +done + +"$SCRIPT_DIR/verify-test-stability.sh" +node "$SCRIPT_DIR/run-swift-tests-with-timing.mjs" \ + --warn-ms "$(( WARN_SECONDS * 1000 ))" \ + --max-ms "$(( MAX_SECONDS * 1000 ))" \ + --suite-timeout-ms "$(( SUITE_TIMEOUT_SECONDS * 1000 ))" \ + --report "$REPORT" \ + -- "$ROOT_DIR/scripts/test-macos.sh" --no-parallel "${SWIFT_ARGS[@]}" diff --git a/.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 b/.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 new file mode 100644 index 000000000..46189daab --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 @@ -0,0 +1,140 @@ +[CmdletBinding()] +param( + [ValidateSet("All", "Frontend", "WindowsRust", "SharedRust")] + [string]$Scope = "All", + [ValidateRange(1, 3600)] + [int]$WarnSeconds = 1, + [ValidateRange(1, 3600)] + [int]$MaxSeconds = 15, + [ValidateRange(1, 7200)] + [int]$SuiteTimeoutSeconds = 1200, + [string[]]$FrontendTestPath = @() +) + +$ErrorActionPreference = "Stop" +if ($WarnSeconds -ge $MaxSeconds) { throw "WarnSeconds must be lower than MaxSeconds." } + +if (-not (Get-Command node -ErrorAction SilentlyContinue)) { + throw "The Windows test stability harness requires Node.js." +} +if ($Scope -in @("All", "Frontend") -and -not (Get-Command bun -ErrorAction SilentlyContinue)) { + throw "Bun is required only for the Windows Frontend scope. Use WindowsRust or SharedRust without Bun." +} + +$root = [System.IO.Path]::GetFullPath((Join-Path $PSScriptRoot "../../../..")) +$reportRoot = Join-Path $root ".artifacts/test-stability" +New-Item -ItemType Directory -Path $reportRoot -Force | Out-Null + +& (Join-Path $PSScriptRoot "verify-test-stability.ps1") +if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + +$warnMilliseconds = $WarnSeconds * 1000 +$maxMilliseconds = $MaxSeconds * 1000 +$suiteTimeoutMilliseconds = $SuiteTimeoutSeconds * 1000 + +function Invoke-TimedRustTests { + param( + [Parameter(Mandatory)] + [string]$Manifest, + [string]$Package, + [Parameter(Mandatory)] + [string]$TargetDirectory, + [Parameter(Mandatory)] + [string]$Report + ) + + $suiteTimer = [System.Diagnostics.Stopwatch]::StartNew() + function Get-RemainingRustSuiteMilliseconds { + $remainingMilliseconds = [Math]::Floor( + $suiteTimeoutMilliseconds - $suiteTimer.Elapsed.TotalMilliseconds + ) + if ($remainingMilliseconds -le 0) { + throw "Rust test suite exceeded the shared $SuiteTimeoutSeconds second deadline." + } + return $remainingMilliseconds + } + + function New-RustTimingArguments { + $remainingMilliseconds = Get-RemainingRustSuiteMilliseconds + $arguments = @( + (Join-Path $PSScriptRoot "run-rust-tests-with-timing.mjs"), + "--manifest", $Manifest, + "--warn-ms", $warnMilliseconds, + "--max-ms", $maxMilliseconds, + "--build-timeout-ms", $remainingMilliseconds, + "--suite-timeout-ms", $remainingMilliseconds, + "--report", $Report + ) + if (-not [string]::IsNullOrWhiteSpace($Package)) { + $arguments += @("--package", $Package) + } + return $arguments + } + + if (Test-Path -LiteralPath $Report) { Remove-Item -Force -LiteralPath $Report } + $arguments = New-RustTimingArguments + & node @arguments + if ($LASTEXITCODE -eq 0) { return } + if (Test-Path -LiteralPath $Report) { + $suiteTimedOut = $false + try { + $timingReport = Get-Content -LiteralPath $Report -Raw | ConvertFrom-Json + $suiteTimedOut = $null -ne $timingReport.suite -and $timingReport.suite.timedOut -eq $true + } catch { + Write-Warning "Could not inspect the Rust timing report after failure: $($_.Exception.Message)" + } + if ($suiteTimedOut) { + throw "Rust test suite exceeded the shared $SuiteTimeoutSeconds second deadline." + } + } + if ($env:LITHE_CARGO_BUILD_CACHE_RESTORED -ne "true") { + throw "Rust timing tests failed." + } + + $null = Get-RemainingRustSuiteMilliseconds + $target = [System.IO.Path]::GetFullPath((Join-Path $root $TargetDirectory)) + $trimCharacters = [char[]]@( + [System.IO.Path]::DirectorySeparatorChar, + [System.IO.Path]::AltDirectorySeparatorChar + ) + $repositoryPrefix = [System.IO.Path]::GetFullPath($root).TrimEnd($trimCharacters) + + [System.IO.Path]::DirectorySeparatorChar + if (-not $target.StartsWith($repositoryPrefix, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Cargo target directory must stay inside the repository: $target" + } + Write-Warning "Rust timing failed after restoring cached build outputs. Clearing $TargetDirectory and retrying once." + if (Test-Path -LiteralPath $target) { Remove-Item -Recurse -Force -LiteralPath $target } + if (Test-Path -LiteralPath $Report) { Remove-Item -Force -LiteralPath $Report } + $arguments = New-RustTimingArguments + & node @arguments + if ($LASTEXITCODE -ne 0) { throw "Rust timing tests failed after a clean retry." } +} + +if ($Scope -in @("All", "Frontend")) { + $arguments = @( + (Join-Path $PSScriptRoot "run-bun-tests-with-timing.mjs"), + "--working-directory", (Join-Path $root "windows/tauri"), + "--warn-ms", $warnMilliseconds, + "--max-ms", $maxMilliseconds, + "--suite-timeout-ms", $suiteTimeoutMilliseconds, + "--report", (Join-Path $reportRoot "windows-frontend.json"), + "--" + ) + $FrontendTestPath + & node @arguments + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } +} + +if ($Scope -in @("All", "WindowsRust")) { + Invoke-TimedRustTests ` + -Manifest "windows/tauri/src-tauri/Cargo.toml" ` + -TargetDirectory "windows/tauri/src-tauri/target" ` + -Report (Join-Path $reportRoot "windows-rust.json") +} + +if ($Scope -in @("All", "SharedRust")) { + Invoke-TimedRustTests ` + -Manifest "rust/Cargo.toml" ` + -Package "lithe-core" ` + -TargetDirectory "rust/target" ` + -Report (Join-Path $reportRoot "shared-rust.json") +} diff --git a/.agents/skills/write-stable-tests/scripts/test-timing-lib.mjs b/.agents/skills/write-stable-tests/scripts/test-timing-lib.mjs new file mode 100755 index 000000000..5a3e79416 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/test-timing-lib.mjs @@ -0,0 +1,173 @@ +import { spawn, spawnSync } from "node:child_process"; +import { StringDecoder } from "node:string_decoder"; + +function delay(milliseconds) { + return new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +function processGroupIsRunning(processID) { + try { + process.kill(-processID, 0); + return true; + } catch (error) { + return error?.code === "EPERM"; + } +} + +function signalProcessGroup(child, signal) { + try { + process.kill(-child.pid, signal); + return true; + } catch { + try { + return child.kill(signal); + } catch { + return false; + } + } +} + +async function waitForProcessGroupExit(processID, timeoutMs, pollIntervalMs) { + const deadline = performance.now() + timeoutMs; + while (processGroupIsRunning(processID) && performance.now() < deadline) { + await delay(pollIntervalMs); + } + return !processGroupIsRunning(processID); +} + +export async function terminateProcessTree( + child, + { + gracePeriodMs = 1000, + forcedTerminationTimeoutMs = 1000, + pollIntervalMs = 20, + } = {}, +) { + if (!child.pid) return true; + if (process.platform === "win32") { + spawnSync("taskkill", ["/PID", String(child.pid), "/T", "/F"], { + stdio: "ignore", + windowsHide: true, + }); + return true; + } + + signalProcessGroup(child, "SIGTERM"); + if (await waitForProcessGroupExit(child.pid, gracePeriodMs, pollIntervalMs)) return true; + signalProcessGroup(child, "SIGKILL"); + return waitForProcessGroupExit(child.pid, forcedTerminationTimeoutMs, pollIntervalMs); +} + +function lineCollector(callback) { + const decoder = new StringDecoder("utf8"); + let pending = ""; + return { + push(chunk) { + pending += decoder.write(chunk); + const lines = pending.split(/\r?\n/); + pending = lines.pop() ?? ""; + for (const line of lines) callback(line); + }, + finish() { + pending += decoder.end(); + if (pending) callback(pending); + pending = ""; + }, + }; +} + +export function runProcess({ + command, + args = [], + cwd, + env = process.env, + timeoutMs, + onStdoutLine = () => {}, + onStderrLine = () => {}, + onSpawn = () => {}, + streamStdout = false, + streamStderr = false, + terminationGraceMs = 1000, + forcedTerminationTimeoutMs = 1000, + terminationPollIntervalMs = 20, +}) { + return new Promise((resolve, reject) => { + const startedAt = performance.now(); + const child = spawn(command, args, { + cwd, + env, + detached: process.platform !== "win32", + windowsHide: true, + stdio: ["ignore", "pipe", "pipe"], + }); + const stdoutChunks = []; + const stderrChunks = []; + const stdoutLines = lineCollector(onStdoutLine); + const stderrLines = lineCollector(onStderrLine); + let timedOut = false; + let timeout; + let terminationPromise = null; + let settled = false; + + const terminate = () => { + terminationPromise ??= terminateProcessTree(child, { + gracePeriodMs: terminationGraceMs, + forcedTerminationTimeoutMs, + pollIntervalMs: terminationPollIntervalMs, + }); + return terminationPromise; + }; + + onSpawn({ + pid: child.pid, + terminate, + }); + + if (timeoutMs > 0) { + timeout = setTimeout(() => { + timedOut = true; + void terminate(); + }, timeoutMs); + } + + child.stdout.on("data", (chunk) => { + stdoutChunks.push(chunk); + stdoutLines.push(chunk); + if (streamStdout) process.stdout.write(chunk); + }); + child.stderr.on("data", (chunk) => { + stderrChunks.push(chunk); + stderrLines.push(chunk); + if (streamStderr) process.stderr.write(chunk); + }); + child.once("error", (error) => { + if (settled) return; + settled = true; + if (timeout) clearTimeout(timeout); + reject(error); + }); + child.once("close", async (code, signal) => { + if (settled) return; + settled = true; + if (timeout) clearTimeout(timeout); + const terminationConfirmed = terminationPromise ? await terminationPromise : true; + stdoutLines.finish(); + stderrLines.finish(); + resolve({ + code, + signal, + timedOut, + terminationConfirmed, + durationMs: performance.now() - startedAt, + stdout: Buffer.concat(stdoutChunks).toString("utf8"), + stderr: Buffer.concat(stderrChunks).toString("utf8"), + }); + }); + }); +} + +export function positiveInteger(value, label) { + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed <= 0) throw new Error(`${label} must be a positive integer.`); + return parsed; +} diff --git a/.agents/skills/write-stable-tests/scripts/test-verify-test-stability.mjs b/.agents/skills/write-stable-tests/scripts/test-verify-test-stability.mjs new file mode 100755 index 000000000..19d8f39ba --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/test-verify-test-stability.mjs @@ -0,0 +1,559 @@ +#!/usr/bin/env node + +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { parseAddedLines, scanFile } from "./verify-test-stability.mjs"; +import { parseJUnitCases } from "./parse-junit-cases.mjs"; +import { run as runBunTestsWithTiming } from "./run-bun-tests-with-timing.mjs"; +import { run as runRustTestsWithTiming } from "./run-rust-tests-with-timing.mjs"; +import { + parseSwiftSuiteLine, + parseSwiftTimingLine, + run as runSwiftTestsWithTiming, +} from "./run-swift-tests-with-timing.mjs"; +import { runProcess } from "./test-timing-lib.mjs"; + +const swiftViolations = scanFile( + "macos/Tests/LitheTests/BlockingTests.swift", + `import Testing +private let gate = DispatchSemaphore(value: 0) +gate.wait() +gate.wait(timeout: .distantFuture) +gate.wait(timeout: DispatchTime.distantFuture) +gate.wait(timeout: .now() + .seconds(1)) +// test-stability: allow(swift-real-sleep) reason: verifies a native synchronous timeout boundary +try await Task.sleep(for: .milliseconds(1)) +`, +); +assert.deepEqual( + swiftViolations.map((violation) => violation.rule), + ["swift-unbounded-wait", "swift-unbounded-wait", "swift-unbounded-wait"], +); + +const detachedViolations = scanFile( + "macos/Tests/LitheTests/DetachedBlockingTests.swift", + `Task.detached { + operations + .waitUntilBlocked() +} +`, +); +assert.deepEqual(detachedViolations.map((violation) => violation.rule), ["swift-detached-blocking"]); +assert.equal(detachedViolations[0].line, 3); + +const multilineWaitContent = `gate.wait( + timeout: .distantFuture +) +`; + +const multilineWaitViolations = scanFile( + "macos/Tests/LitheTests/MultilineWaitTests.swift", + multilineWaitContent, +); + +assert.deepEqual( + multilineWaitViolations.map((violation) => violation.rule), + ["swift-unbounded-wait"], +); + +assert.equal(multilineWaitViolations[0].line, 1); + +const selectedMultilineWaitViolations = scanFile( + "macos/Tests/LitheTests/MultilineWaitTests.swift", + multilineWaitContent, + new Set([2]), +); + +assert.deepEqual( + selectedMultilineWaitViolations.map((violation) => violation.rule), + ["swift-unbounded-wait"], +); + +assert.equal(selectedMultilineWaitViolations[0].line, 2); + +assert.match( + selectedMultilineWaitViolations[0].source, + /distantFuture/, +); + +const bareMultilineWaitViolations = scanFile( + "macos/Tests/LitheTests/MultilineWaitTests.swift", + `gate.wait( +) +`, +); + +assert.deepEqual( + bareMultilineWaitViolations.map((violation) => violation.rule), + ["swift-unbounded-wait"], +); + +const finiteMultilineWaitViolations = scanFile( + "macos/Tests/LitheTests/MultilineWaitTests.swift", + `gate.wait( + timeout: .now() + .seconds(timeoutSeconds()) +) +`, +); + +assert.deepEqual(finiteMultilineWaitViolations, []); + +const typescriptViolations = scanFile( + "windows/tauri/src/example.test.ts", + `test("bad timer", async () => { + await new Promise((resolve) => setTimeout(resolve, 10)); +}); +`, +); +assert.deepEqual(typescriptViolations.map((violation) => violation.rule), ["typescript-real-timer"]); + +const rustViolations = scanFile( + "rust/lithe-core/src/example.rs", + `use std::thread; +fn production_backoff() { thread::sleep(Duration::from_millis(1)); } +#[cfg(test)] +mod tests { + #[test] + fn blocks() { + thread::sleep(Duration::from_millis(1)); + } +} +`, +); +assert.deepEqual(rustViolations.map((violation) => violation.rule), ["rust-real-sleep"]); +assert.equal(rustViolations[0].line, 7); + +const diff = `diff --git a/macos/Tests/LitheTests/Example.swift b/macos/Tests/LitheTests/Example.swift +--- a/macos/Tests/LitheTests/Example.swift ++++ b/macos/Tests/LitheTests/Example.swift +@@ -2,0 +3,2 @@ ++let gate = DispatchSemaphore(value: 0) ++gate.wait() +`; +const added = parseAddedLines(diff); +assert.deepEqual([...added.get("macos/Tests/LitheTests/Example.swift")], [3, 4]); + +const selectedViolations = scanFile( + "macos/Tests/LitheTests/Example.swift", + "let unchanged = true\nThread.sleep(forTimeInterval: 1)\nlet added = true\ngate.wait()\n", + new Set([3, 4]), +); +assert.deepEqual(selectedViolations.map((violation) => violation.rule), ["swift-unbounded-wait"]); + +assert.deepEqual( + parseSwiftTimingLine("✔ Test deterministicGate() passed after 0.024 seconds."), + { name: "deterministicGate()", event: "passed", durationMs: 24, caseCount: null }, +); +assert.deepEqual( + parseSwiftTimingLine("Test Case '-[LitheTests.Legacy testValue]' failed (1.250 seconds)."), + { name: "-[LitheTests.Legacy testValue]", event: "failed", durationMs: 1250, caseCount: null }, +); +assert.equal( + parseSwiftTimingLine("◇ Test case passing 1 argument value → 1 to parameterized(_:) started."), + null, +); +assert.deepEqual( + parseSwiftTimingLine("✔ Test parameterized(_:) with 4 test cases passed after 0.125 seconds."), + { name: "parameterized(_:)", event: "passed", durationMs: 125, caseCount: 4 }, +); +assert.deepEqual( + parseSwiftSuiteLine('◇ Suite "App localization" started.'), + { name: "App localization", event: "started" }, +); + +assert.deepEqual( + parseJUnitCases( + '', + ), + [ + { name: "scheduler / fires & clears", status: "passed", durationMs: 125 }, + { name: "skips", status: "skipped", durationMs: 0 }, + ], +); + +const bunTimeoutRoot = mkdtempSync(path.join(os.tmpdir(), "lithe-test-stability-bun-timeout-")); +try { + const reportPath = path.join(bunTimeoutRoot, "bun-timeout.json"); + await assert.rejects( + runBunTestsWithTiming( + { + workingDirectory: bunTimeoutRoot, + warnMs: 50, + maxMs: 200, + suiteTimeoutMs: 500, + report: reportPath, + testArguments: [], + }, + { + runProcessImpl: async () => ({ + code: null, + signal: "SIGTERM", + timedOut: true, + terminationConfirmed: true, + durationMs: 500, + stdout: "", + stderr: "", + }), + }, + ), + /Bun test suite exceeded 500ms/, + ); + const timeoutReport = JSON.parse(readFileSync(reportPath, "utf8")); + assert.deepEqual( + timeoutReport.tests.map(({ name, status }) => ({ name, status })), + [{ name: "Bun test suite timeout", status: "timeout" }], + ); + assert.match(readFileSync(reportPath.replace(/\.json$/, ".junit.xml"), "utf8"), /errors="1"/); + assert.ok(existsSync(reportPath.replace(/\.json$/, ".html"))); +} finally { + rmSync(bunTimeoutRoot, { recursive: true, force: true }); +} + +const swiftTimeoutRoot = mkdtempSync(path.join(os.tmpdir(), "lithe-test-stability-swift-timeout-")); +try { + const reportPath = path.join(swiftTimeoutRoot, "swift-timeout.json"); + await assert.rejects( + runSwiftTestsWithTiming( + { + warnMs: 50, + maxMs: 200, + suiteTimeoutMs: 500, + report: reportPath, + command: "swift", + commandArguments: ["test"], + }, + { + runProcessImpl: async ({ onSpawn }) => { + onSpawn({ terminate: async () => true }); + return { + code: null, + signal: "SIGTERM", + timedOut: true, + terminationConfirmed: true, + durationMs: 500, + stdout: "", + stderr: "", + }; + }, + }, + ), + /Swift test suite exceeded 500ms/, + ); + const timeoutReport = JSON.parse(readFileSync(reportPath, "utf8")); + assert.deepEqual( + timeoutReport.tests.map(({ name, status }) => ({ name, status })), + [{ name: "Swift test suite timeout", status: "timeout" }], + ); + assert.match(readFileSync(reportPath.replace(/\.json$/, ".junit.xml"), "utf8"), /errors="1"/); + assert.ok(existsSync(reportPath.replace(/\.json$/, ".html"))); +} finally { + rmSync(swiftTimeoutRoot, { recursive: true, force: true }); +} + +const rustCompileFailureRoot = mkdtempSync( + path.join( + os.tmpdir(), + "lithe-test-stability-rust-compile-failure-", + ), +); + +try { + const reportPath = path.join( + rustCompileFailureRoot, + "rust-compile-failure.json", + ); + + await assert.rejects( + runRustTestsWithTiming( + { + manifest: path.join( + rustCompileFailureRoot, + "Cargo.toml", + ), + package: null, + warnMs: 50, + maxMs: 200, + buildTimeoutMs: 1000, + suiteTimeoutMs: 2000, + report: reportPath, + keepGoing: false, + }, + { + runProcessImpl: async ({ + onStdoutLine = () => {}, + }) => { + onStdoutLine( + JSON.stringify({ + reason: "compiler-message", + message: { + rendered: + "error[E0425]: cannot find value `missing` in this scope\n", + }, + }), + ); + + return { + code: 101, + signal: null, + timedOut: false, + terminationConfirmed: true, + durationMs: 12, + stdout: "", + stderr: "", + }; + }, + }, + ), + /Cargo test compilation exited with code 101/, + ); + + const compileFailureReport = JSON.parse( + readFileSync(reportPath, "utf8"), + ); + + assert.deepEqual( + compileFailureReport.tests.map( + ({ name, status }) => ({ name, status }), + ), + [ + { + name: "Cargo test compilation", + status: "failed", + }, + ], + ); + + assert.match( + compileFailureReport.tests[0].details, + /E0425/, + ); + + assert.match( + readFileSync( + reportPath.replace(/\.json$/, ".log"), + "utf8", + ), + /E0425/, + ); + + assert.match( + readFileSync( + reportPath.replace(/\.json$/, ".junit.xml"), + "utf8", + ), + /failures="1"/, + ); + + assert.ok( + existsSync( + reportPath.replace(/\.json$/, ".html"), + ), + ); +} finally { + rmSync(rustCompileFailureRoot, { + recursive: true, + force: true, + }); +} + +if (process.platform !== "win32") { + let rootPID = null; + let descendantPID = null; + const descendantSource = "process.on('SIGTERM', () => {}); setInterval(() => {}, 1000);"; + const rootSource = ` + const { spawn } = require("node:child_process"); + const descendant = spawn(process.execPath, ["-e", ${JSON.stringify(descendantSource)}], { + stdio: "ignore", + }); + console.log(descendant.pid); + setInterval(() => {}, 1000); + `; + try { + const result = await runProcess({ + command: process.execPath, + args: ["-e", rootSource], + timeoutMs: 100, + terminationGraceMs: 100, + forcedTerminationTimeoutMs: 1000, + terminationPollIntervalMs: 10, + onSpawn: ({ pid }) => { + rootPID = pid; + }, + onStdoutLine: (line) => { + descendantPID = Number(line); + }, + }); + assert.equal(result.timedOut, true); + assert.equal(result.terminationConfirmed, true); + assert.ok(Number.isInteger(descendantPID) && descendantPID > 0); + assert.throws(() => process.kill(descendantPID, 0), { code: "ESRCH" }); + } finally { + for (const pid of [descendantPID, rootPID]) { + if (!Number.isInteger(pid) || pid <= 0) continue; + try { + process.kill(pid === rootPID ? -pid : pid, "SIGKILL"); + } catch { + // The expected path already removed the process tree. + } + } + } +} + +const fixtureRoot = mkdtempSync(path.join(os.tmpdir(), "lithe-test-stability-rust-")); +try { + mkdirSync(path.join(fixtureRoot, "src")); + writeFileSync( + path.join(fixtureRoot, "Cargo.toml"), + '[package]\nname = "timing-fixture"\nversion = "0.1.0"\nedition = "2021"\n', + ); + writeFileSync( + path.join(fixtureRoot, "src/lib.rs"), + `#[cfg(test)] +mod tests { + #[test] + fn a_quick() { assert_eq!(2 + 2, 4); } + + #[test] + fn z_hangs() { std::thread::sleep(std::time::Duration::from_secs(5)); } +} +`, + ); + const reportPath = path.join(fixtureRoot, "timing.json"); + const runnerPath = path.join(path.dirname(fileURLToPath(import.meta.url)), "run-rust-tests-with-timing.mjs"); + const result = spawnSync( + process.execPath, + [ + runnerPath, + "--manifest", path.join(fixtureRoot, "Cargo.toml"), + "--warn-ms", "50", + "--max-ms", "200", + "--build-timeout-ms", "30000", + "--suite-timeout-ms", "30000", + "--report", reportPath, + ], + { encoding: "utf8", timeout: 60000 }, + ); + assert.notEqual(result.status, 0, "the Rust timing runner must reject a hanging test"); + assert.ok( + existsSync(reportPath), + `the Rust timing runner did not write a report\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); + const timingReport = JSON.parse(readFileSync(reportPath, "utf8")); + assert.deepEqual( + timingReport.tests.map(({ name, status }) => ({ name, status })), + [ + { name: "tests::a_quick", status: "passed" }, + { name: "tests::z_hangs", status: "timeout" }, + ], + ); + const html = readFileSync(path.join(fixtureRoot, "timing.html"), "utf8"); + const junit = readFileSync(path.join(fixtureRoot, "timing.junit.xml"), "utf8"); + assert.match(html, /问题与性能优化队列/); + assert.match(html, /tests::z_hangs/); + assert.match(html, /over budget|timeout/); + assert.match(junit, /errors="1"/); + assert.match(junit, /tests::z_hangs/); + assert.ok(existsSync(path.join(fixtureRoot, "index.html"))); +} finally { + rmSync(fixtureRoot, { recursive: true, force: true }); +} + +const deadlineFixtureRoot = mkdtempSync(path.join(os.tmpdir(), "lithe-test-stability-deadline-")); +try { + const reportPath = path.join(deadlineFixtureRoot, "deadline.json"); + const manifestPath = path.join(deadlineFixtureRoot, "Cargo.toml"); + const executablePath = path.join(deadlineFixtureRoot, "fake-tests"); + let currentTime = 0; + let invocation = 0; + const runProcessImpl = async ({ args, onStdoutLine = () => {}, timeoutMs }) => { + invocation += 1; + if (invocation === 1) { + assert.equal(timeoutMs, 1000); + currentTime = 400; + onStdoutLine(JSON.stringify({ + reason: "compiler-artifact", + profile: { test: true }, + executable: executablePath, + manifest_path: manifestPath, + target: { name: "deadline-fixture" }, + })); + return { code: 0, signal: null, timedOut: false, durationMs: 400, stdout: "", stderr: "" }; + } + if (args[0] === "--list") { + assert.equal(timeoutMs, 600); + currentTime = 500; + return { + code: 0, + signal: null, + timedOut: false, + durationMs: 100, + stdout: "tests::first: test\ntests::second: test\n", + stderr: "", + }; + } + if (args[1] === "tests::first") { + assert.equal(timeoutMs, 500); + currentTime = 800; + return { + code: 0, + signal: null, + timedOut: false, + durationMs: 300, + stdout: "running 1 test\ntest tests::first ... ok\n", + stderr: "", + }; + } + assert.deepEqual(args.slice(0, 2), ["--exact", "tests::second"]); + assert.equal(timeoutMs, 200); + currentTime = 1000; + return { + code: null, + signal: "SIGTERM", + timedOut: true, + durationMs: 200, + stdout: "running 1 test\n", + stderr: "", + }; + }; + + await assert.rejects( + runRustTestsWithTiming( + { + manifest: manifestPath, + package: null, + warnMs: 50, + maxMs: 500, + buildTimeoutMs: 5000, + suiteTimeoutMs: 1000, + report: reportPath, + keepGoing: false, + }, + { runProcessImpl, now: () => currentTime }, + ), + /Rust test suite exceeded 1000ms during test deadline-fixture::tests::second/, + ); + const deadlineReport = JSON.parse(readFileSync(reportPath, "utf8")); + assert.deepEqual(deadlineReport.suite, { + timedOut: true, + stage: "test deadline-fixture::tests::second", + durationMs: 1000, + }); + assert.deepEqual( + deadlineReport.tests.map(({ name, status }) => ({ name, status })), + [ + { name: "tests::first", status: "passed" }, + { name: "tests::second", status: "timeout" }, + ], + ); + assert.match(deadlineReport.tests[1].details, /shared suite deadline expired/); + assert.ok(existsSync(path.join(deadlineFixtureRoot, "deadline.html"))); + assert.ok(existsSync(path.join(deadlineFixtureRoot, "deadline.junit.xml"))); +} finally { + rmSync(deadlineFixtureRoot, { recursive: true, force: true }); +} + +console.log("Test stability verifier tests passed."); diff --git a/.agents/skills/write-stable-tests/scripts/verify-test-stability.mjs b/.agents/skills/write-stable-tests/scripts/verify-test-stability.mjs new file mode 100755 index 000000000..7a644f565 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/verify-test-stability.mjs @@ -0,0 +1,492 @@ +#!/usr/bin/env node + +import { execFileSync } from "node:child_process"; +import { readFileSync, readdirSync, statSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const SCRIPT_DIRECTORY = path.dirname(fileURLToPath(import.meta.url)); +const REPOSITORY_ROOT = path.resolve(SCRIPT_DIRECTORY, "../../../.."); + +const RULES = { + swift: [ + { + id: "swift-unbounded-wait", + pattern: /\.wait\s*\(\s*\)|\.wait\s*\([^)]*\.\s*distantFuture\b[^)]*\)/, + message: "Use a finite timeout and assert its result; bare waits and distantFuture can hang CI.", + }, + { + id: "swift-real-sleep", + pattern: /\b(?:Task|Thread)\.sleep\s*\(|\busleep\s*\(/, + message: "Use an injected clock, event, or continuation instead of real-time sleep.", + }, + { + id: "swift-process-wait", + pattern: /\.waitUntilExit\s*\(/, + message: "Wait for subprocesses through a watchdog that can terminate the process tree.", + }, + { + id: "swift-run-loop-wait", + pattern: /RunLoop\.current\.run\s*\(/, + message: "Do not spin a run loop to synchronize a test; await an observable event.", + }, + ], + typescript: [ + { + id: "typescript-real-timer", + pattern: /\b(?:globalThis\.|window\.)?set(?:Timeout|Interval)\s*\(/, + message: "Inject a manual timer or scheduler instead of using a real timer in a test.", + }, + { + id: "typescript-atomic-wait", + pattern: /\bAtomics\.wait\s*\(/, + message: "Atomics.wait blocks the test worker; use a deferred promise with owned cleanup.", + }, + { + id: "typescript-infinite-loop", + pattern: /\bwhile\s*\(\s*true\s*\)/, + message: "Test polling loops require an explicit deadline and timeout diagnostic.", + }, + ], + rust: [ + { + id: "rust-real-sleep", + pattern: /\b(?:std::)?thread::sleep\s*\(/, + message: "Use a channel, barrier, or injected clock instead of sleeping to coordinate a test.", + }, + { + id: "rust-unbounded-receive", + pattern: /\.recv\(\s*\)/, + message: "Use recv_timeout or another bounded receive in test synchronization.", + }, + ], +}; + +function normalizePath(filePath) { + return filePath.split(path.sep).join("/").replace(/^\.\//, ""); +} + +function languageFor(filePath) { + const normalized = normalizePath(filePath); + if ( + normalized.endsWith(".swift") && + (normalized.includes("/Tests/") || normalized.startsWith("macos/Tests/")) + ) { + return "swift"; + } + if (/\.(?:test|spec)\.tsx?$/.test(normalized)) return "typescript"; + if (normalized.endsWith(".rs")) return "rust"; + return null; +} + +function belongsToPlatform(filePath, platform) { + if (platform === "all") return true; + const normalized = normalizePath(filePath); + if (platform === "macos") { + return ( + normalized.startsWith("macos/") || + normalized.startsWith("Plugins/mac/") || + normalized.startsWith("rust/") + ); + } + return normalized.startsWith("windows/") || normalized.startsWith("rust/lithe-core/"); +} + +function stripStringsAndLineComments(line) { + let result = ""; + let quote = null; + let escaped = false; + for (let index = 0; index < line.length; index += 1) { + const character = line[index]; + const next = line[index + 1]; + if (quote) { + if (escaped) escaped = false; + else if (character === "\\") escaped = true; + else if (character === quote) quote = null; + result += " "; + continue; + } + if (character === "/" && next === "/") break; + if (character === '"' || character === "'") { + quote = character; + result += " "; + continue; + } + result += character; + } + return result; +} + +function rustTestLineMask(lines, filePath) { + const normalized = normalizePath(filePath); + if (normalized.includes("/tests/") || /(?:^|\/)tests\.rs$/.test(normalized)) { + return lines.map(() => true); + } + + const mask = lines.map(() => false); + let depth = 0; + let pendingTestModule = false; + let pendingTestFunction = false; + const activeDepths = []; + + for (let index = 0; index < lines.length; index += 1) { + const structural = stripStringsAndLineComments(lines[index]); + if (/^\s*#\s*\[\s*cfg\s*\(\s*test\s*\)\s*\]/.test(structural)) { + pendingTestModule = true; + } + if (/^\s*#\s*\[\s*(?:[A-Za-z_][\w:]*::)?test(?:\s*\([^\]]*\))?\s*\]/.test(structural)) { + pendingTestFunction = true; + } + + const startsModule = pendingTestModule && /\bmod\s+[A-Za-z_]\w*\s*\{/.test(structural); + const startsFunction = pendingTestFunction && /\bfn\s+[A-Za-z_]\w*[^;]*\{/.test(structural); + const startsTestRegion = startsModule || startsFunction; + if (activeDepths.length > 0 || pendingTestFunction || startsTestRegion) mask[index] = true; + + const opens = (structural.match(/\{/g) ?? []).length; + const closes = (structural.match(/\}/g) ?? []).length; + if (startsTestRegion) activeDepths.push(depth + 1); + depth += opens - closes; + while (activeDepths.length > 0 && depth < activeDepths[activeDepths.length - 1]) { + activeDepths.pop(); + } + + if (startsModule) pendingTestModule = false; + if (startsFunction) pendingTestFunction = false; + } + + return mask; +} + +function exceptionReason(lines, lineIndex, ruleID) { + const candidates = [lines[lineIndex], lines[lineIndex - 1]].filter(Boolean); + const escapedRule = ruleID.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const pattern = new RegExp( + `test-stability:\\s*allow\\(${escapedRule}\\)\\s*reason:\\s*(.+)$`, + ); + for (const candidate of candidates) { + const match = candidate.match(pattern); + if (match) return match[1].trim(); + } + return null; +} + +function swiftUnboundedWaitViolations(lines, selected) { + const violations = []; + const rule = RULES.swift.find((candidate) => candidate.id === "swift-unbounded-wait"); + + for (let index = 0; index < lines.length; index += 1) { + const firstLine = stripStringsAndLineComments(lines[index]); + let searchOffset = 0; + + while (searchOffset < firstLine.length) { + const match = firstLine.slice(searchOffset).match(/\.wait\s*\(/); + if (!match) break; + + const waitColumn = searchOffset + match.index; + let depth = 0; + let callStarted = false; + let callClosed = false; + let callEnd = index; + let callText = ""; + + callLines: + for (let callIndex = index; callIndex < lines.length; callIndex += 1) { + const structural = stripStringsAndLineComments(lines[callIndex]); + const columnStart = callIndex === index ? waitColumn : 0; + + for (let column = columnStart; column < structural.length; column += 1) { + const character = structural[column]; + callText += character; + if (character === "(") { + depth += 1; + callStarted = true; + } else if (character === ")" && callStarted) { + depth -= 1; + + if (depth === 0) { + callEnd = callIndex; + callClosed = true; + break callLines; + } + } + } + + callEnd = callIndex; + callText += "\n"; + } + + if (!callStarted || !callClosed) { + break; + } + + const openParen = callText.indexOf("("); + const closeParen = callText.lastIndexOf(")"); + const argumentsText = callText.slice(openParen + 1, closeParen); + + const isUnbounded = + argumentsText.trim().length === 0 || + /\.\s*distantFuture\b/.test(argumentsText); + + if (isUnbounded) { + const selectedLines = selected + ? [...selected].filter( + (lineNumber) => + lineNumber >= index + 1 && + lineNumber <= callEnd + 1, + ) + : []; + + if (!selected || selectedLines.length > 0) { + const reportLine = selected + ? Math.min(...selectedLines) - 1 + : index; + + const reason = exceptionReason(lines, index, rule.id); + + if (!reason || reason.length < 16) { + violations.push({ + file: null, + line: reportLine + 1, + rule: rule.id, + message: reason + ? `Exception reason is too short. ${rule.message}` + : rule.message, + source: lines[reportLine].trim(), + }); + } + } + } + + searchOffset = waitColumn + match[0].length; + } + } + + return violations; +} + +function swiftDetachedBlockingViolations(lines, selected) { + const violations = []; + for (let index = 0; index < lines.length; index += 1) { + const firstLine = stripStringsAndLineComments(lines[index]); + const detachedIndex = firstLine.search(/\bTask\.detached\b/); + if (detachedIndex < 0) continue; + + let depth = 0; + let blockStarted = false; + let blockEnd = index; + let waitLine = null; + for (let blockIndex = index; blockIndex < lines.length; blockIndex += 1) { + let structural = stripStringsAndLineComments(lines[blockIndex]); + if (blockIndex === index) structural = structural.slice(detachedIndex); + if (!blockStarted) { + const openingBrace = structural.indexOf("{"); + if (openingBrace < 0) continue; + structural = structural.slice(openingBrace); + blockStarted = true; + } + if (waitLine === null && /\bwait\w*\s*\(/.test(structural)) waitLine = blockIndex; + depth += (structural.match(/\{/g) ?? []).length; + depth -= (structural.match(/\}/g) ?? []).length; + blockEnd = blockIndex; + if (depth <= 0) break; + } + if (!blockStarted || waitLine === null) continue; + + const selectedLines = selected + ? [...selected].filter((lineNumber) => lineNumber >= index + 1 && lineNumber <= blockEnd + 1) + : []; + if (selected && selectedLines.length === 0) continue; + const reportLine = selected?.has(waitLine + 1) + ? waitLine + : selected + ? Math.min(...selectedLines) - 1 + : waitLine; + const reason = exceptionReason(lines, waitLine, "swift-detached-blocking"); + if (reason && reason.length >= 16) continue; + violations.push({ + file: null, + line: reportLine + 1, + rule: "swift-detached-blocking", + message: reason + ? "Exception reason is too short. Do not hide a blocking wait in Task.detached; use bounded asynchronous signaling." + : "Do not hide a blocking wait in Task.detached; use bounded asynchronous signaling.", + source: lines[reportLine].trim(), + }); + index = blockEnd; + } + return violations; +} + +export function scanFile(filePath, content, selectedLineNumbers = null, platform = "all") { + const normalized = normalizePath(filePath); + const language = languageFor(normalized); + if (!language || !belongsToPlatform(normalized, platform)) return []; + + const lines = content.split(/\r?\n/); + const rustMask = language === "rust" ? rustTestLineMask(lines, normalized) : null; + const selected = selectedLineNumbers ? new Set(selectedLineNumbers) : null; + const violations = []; + + for (let index = 0; index < lines.length; index += 1) { + const lineNumber = index + 1; + if (selected && !selected.has(lineNumber)) continue; + if (rustMask && !rustMask[index]) continue; + + for (const rule of RULES[language]) { + if (language === "swift" && rule.id === "swift-unbounded-wait") { + continue; + } + + if (!rule.pattern.test(lines[index])) continue; + const reason = exceptionReason(lines, index, rule.id); + if (reason && reason.length >= 16) continue; + violations.push({ + file: normalized, + line: lineNumber, + rule: rule.id, + message: reason + ? `Exception reason is too short. ${rule.message}` + : rule.message, + source: lines[index].trim(), + }); + } + } + if (language === "swift") { + for (const violation of swiftUnboundedWaitViolations(lines, selected)) { + violations.push({ ...violation, file: normalized }); + } + + for (const violation of swiftDetachedBlockingViolations(lines, selected)) { + violations.push({ ...violation, file: normalized }); + } + } + return violations; +} + +export function parseAddedLines(diff) { + const files = new Map(); + let currentPath = null; + let newLine = 0; + for (const line of diff.split(/\r?\n/)) { + if (line.startsWith("+++ ")) { + const value = line.slice(4); + currentPath = value === "/dev/null" ? null : value.replace(/^b\//, ""); + continue; + } + const hunk = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/); + if (hunk) { + newLine = Number(hunk[1]); + continue; + } + if (!currentPath || line.startsWith("\\ No newline")) continue; + if (line.startsWith("+") && !line.startsWith("+++")) { + if (!files.has(currentPath)) files.set(currentPath, new Set()); + files.get(currentPath).add(newLine); + newLine += 1; + } else if (!line.startsWith("-")) { + newLine += 1; + } + } + return files; +} + +function walk(directory, result = []) { + if (!statSync(directory).isDirectory()) return result; + for (const entry of readdirSync(directory, { withFileTypes: true })) { + if ([".git", ".build", ".artifacts", "target", "node_modules"].includes(entry.name)) continue; + const absolute = path.join(directory, entry.name); + if (entry.isDirectory()) walk(absolute, result); + else result.push(normalizePath(path.relative(REPOSITORY_ROOT, absolute))); + } + return result; +} + +function git(...arguments_) { + return execFileSync("git", ["-c", `safe.directory=${REPOSITORY_ROOT}`, ...arguments_], { + cwd: REPOSITORY_ROOT, + encoding: "utf8", + maxBuffer: 32 * 1024 * 1024, + }); +} + +function parseArguments(arguments_) { + const options = { all: false, platform: "all", base: null, head: "HEAD" }; + for (let index = 0; index < arguments_.length; index += 1) { + const argument = arguments_[index]; + if (argument === "--all") options.all = true; + else if (argument === "--platform") options.platform = arguments_[++index]; + else if (argument === "--base") options.base = arguments_[++index]; + else if (argument === "--head") options.head = arguments_[++index]; + else if (argument === "--help") options.help = true; + else throw new Error(`Unknown argument: ${argument}`); + } + if (!['all', 'macos', 'windows'].includes(options.platform)) { + throw new Error(`Unsupported platform: ${options.platform}`); + } + return options; +} + +function changedFiles(options) { + const diffArguments = ["-c", "core.quotePath=false", "diff", "--unified=0", "--no-color"]; + if (options.base) diffArguments.push(options.base, options.head); + else diffArguments.push("HEAD"); + diffArguments.push("--"); + const changed = parseAddedLines(git(...diffArguments)); + + if (!options.base) { + const untracked = git("ls-files", "--others", "--exclude-standard").split(/\r?\n/).filter(Boolean); + for (const file of untracked) changed.set(normalizePath(file), null); + } + return changed; +} + +export function run(options) { + const candidates = options.all + ? new Map(walk(REPOSITORY_ROOT).map((file) => [file, null])) + : changedFiles(options); + const violations = []; + for (const [file, selectedLines] of candidates) { + if (!languageFor(file) || !belongsToPlatform(file, options.platform)) continue; + const absolute = path.resolve(REPOSITORY_ROOT, file); + let content; + try { + content = readFileSync(absolute, "utf8"); + } catch { + continue; + } + violations.push(...scanFile(file, content, selectedLines, options.platform)); + } + return violations; +} + +function main() { + let options; + try { + options = parseArguments(process.argv.slice(2)); + } catch (error) { + console.error(error.message); + process.exitCode = 2; + return; + } + if (options.help) { + console.log("Usage: verify-test-stability.mjs [--all] [--platform all|macos|windows] [--base REV --head REV]"); + return; + } + + const violations = run(options); + if (violations.length === 0) { + console.log(`Test stability check passed (${options.all ? "full tree" : "added lines"}, ${options.platform}).`); + return; + } + + console.error(`Test stability check found ${violations.length} blocking issue(s):`); + for (const violation of violations) { + console.error(`${violation.file}:${violation.line}: [${violation.rule}] ${violation.message}`); + console.error(` ${violation.source}`); + } + console.error("Use deterministic synchronization. Exceptions require a bounded implementation and a reasoned test-stability annotation."); + process.exitCode = 1; +} + +if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) main(); diff --git a/.agents/skills/write-stable-tests/scripts/verify-test-stability.ps1 b/.agents/skills/write-stable-tests/scripts/verify-test-stability.ps1 new file mode 100644 index 000000000..8fc2eb291 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/verify-test-stability.ps1 @@ -0,0 +1,22 @@ +[CmdletBinding()] +param( + [ValidateSet("windows", "macos", "all")] + [string]$Platform = "windows", + [string]$BaseRevision, + [string]$HeadRevision = "HEAD", + [switch]$All +) + +$ErrorActionPreference = "Stop" +$arguments = @( + (Join-Path $PSScriptRoot "verify-test-stability.mjs"), + "--platform", + $Platform +) +if ($All) { $arguments += "--all" } +if (-not [string]::IsNullOrWhiteSpace($BaseRevision)) { + $arguments += @("--base", $BaseRevision, "--head", $HeadRevision) +} + +& node @arguments +if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/.agents/skills/write-stable-tests/scripts/verify-test-stability.sh b/.agents/skills/write-stable-tests/scripts/verify-test-stability.sh new file mode 100755 index 000000000..6d6ae4d74 --- /dev/null +++ b/.agents/skills/write-stable-tests/scripts/verify-test-stability.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec node "$SCRIPT_DIR/verify-test-stability.mjs" --platform macos "$@" diff --git a/.github/workflows/ci-database.yml b/.github/workflows/ci-database.yml index 4ce2961e2..64783834f 100644 --- a/.github/workflows/ci-database.yml +++ b/.github/workflows/ci-database.yml @@ -104,7 +104,20 @@ jobs: - name: Run database-focused Swift tests timeout-minutes: 12 - run: ./scripts/test-macos.sh --filter '(LitheDatabaseModuleTests|LitheTests\.LitheCoreLogicTests/database)' + run: ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh --report .artifacts/test-stability/macos-database.json --suite-timeout-seconds 660 -- --filter '(LitheDatabaseModuleTests|LitheTests\.LitheCoreLogicTests/database)' + + - name: Generate combined database test report + if: always() + run: node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs + + - name: Upload database test reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-stability-database + path: .artifacts/test-stability/ + if-no-files-found: ignore + retention-days: 14 rust-database-tests: name: Rust database tests diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index 29e7458e7..7557be2fe 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -84,6 +84,13 @@ jobs: shell: bash run: git diff --check "$BASE_SHA" "$GITHUB_SHA" + - name: Reject newly added blocking test patterns + if: steps.base.outputs.manual == 'false' + env: + BASE_SHA: ${{ steps.base.outputs.base_sha }} + shell: bash + run: ./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh --platform all --base "$BASE_SHA" --head "$GITHUB_SHA" + - name: Set up Rust for comment verification if: steps.classify.outputs.rust_core == 'false' && steps.classify.outputs.rust_comments == 'true' uses: dtolnay/rust-toolchain@stable @@ -144,7 +151,9 @@ jobs: - name: Run Swift unit tests timeout-minutes: 12 run: | - ./scripts/test-macos.sh \ + ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh \ + --report .artifacts/test-stability/macos-unit.json \ + --suite-timeout-seconds 660 -- \ --skip WebKitIntegrationTests \ --skip GitStatusObservationTests \ --skip '(LitheDatabaseModuleTests|LitheTests\.LitheCoreLogicTests/database)' \ @@ -160,10 +169,25 @@ jobs: - name: Run WebKit lifecycle integration tests timeout-minutes: 3 run: | - ./scripts/test-macos.sh \ + ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh \ + --report .artifacts/test-stability/macos-webkit.json \ + --max-seconds 60 --suite-timeout-seconds 150 -- \ --skip-build \ --filter WebKitIntegrationTests + - name: Generate combined Swift test report + if: always() + run: node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs + + - name: Upload Swift test reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-stability-macos + path: .artifacts/test-stability/ + if-no-files-found: ignore + retention-days: 14 + rust-core-tests: name: Rust Core and Swift bridge tests needs: changes diff --git a/.github/workflows/ci-plugins.yml b/.github/workflows/ci-plugins.yml index fafe6a396..740fcf99a 100644 --- a/.github/workflows/ci-plugins.yml +++ b/.github/workflows/ci-plugins.yml @@ -98,16 +98,29 @@ jobs: - name: Run plugin-focused Swift tests timeout-minutes: 12 - run: ./scripts/test-macos.sh --filter '([Pp]lugin|LitheGoSupportModuleTests|LinuxDo)' + run: ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh --report .artifacts/test-stability/macos-plugins.json --suite-timeout-seconds 660 -- --filter '([Pp]lugin|LitheGoSupportModuleTests|LinuxDo)' - name: Run plugin WebKit lifecycle integration tests timeout-minutes: 3 - run: ./scripts/test-macos.sh --skip-build --filter WebKitIntegrationTests + run: ./.agents/skills/write-stable-tests/scripts/test-stability-macos.sh --report .artifacts/test-stability/macos-plugin-webkit.json --max-seconds 60 --suite-timeout-seconds 150 -- --skip-build --filter WebKitIntegrationTests - name: Build and verify official plugin packages timeout-minutes: 10 run: ./scripts/verify-official-plugins.sh + - name: Generate combined plugin test report + if: always() + run: node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs + + - name: Upload plugin test reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-stability-plugins + path: .artifacts/test-stability/ + if-no-files-found: ignore + retention-days: 14 + gate: name: Plugin CI gate needs: diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index 4a1a1ae59..591321cc8 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -73,6 +73,13 @@ jobs: ./scripts/classify-ci-changes.sh "$BASE_SHA" "$GITHUB_SHA" >> "$GITHUB_OUTPUT" + - name: Reject newly added Windows blocking test patterns + if: steps.base.outputs.manual == 'false' + env: + BASE_SHA: ${{ steps.base.outputs.base_sha }} + shell: bash + run: ./.agents/skills/write-stable-tests/scripts/verify-test-stability.sh --platform windows --base "$BASE_SHA" --head "$GITHUB_SHA" + - name: Summarize selected lanes shell: bash env: @@ -223,7 +230,6 @@ jobs: - name: Run Windows frontend tests if: needs.changes.outputs.windows == 'true' - working-directory: windows/tauri shell: pwsh run: | bun test src/platform @@ -235,20 +241,28 @@ jobs: - name: Test shared Rust Core if: needs.changes.outputs.rust_core == 'true' shell: pwsh - run: | - ./scripts/invoke-cargo-with-cache-fallback.ps1 ` - -TargetDirectory "rust/target" ` - -FailureMessage "Shared Rust Core tests failed" ` - -CargoArguments @("test", "--manifest-path", "rust/Cargo.toml", "-p", "lithe-core") + timeout-minutes: 20 + run: ./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope SharedRust -SuiteTimeoutSeconds 1080 - name: Test Windows Rust host if: needs.changes.outputs.windows_rust == 'true' shell: pwsh - run: | - ./scripts/invoke-cargo-with-cache-fallback.ps1 ` - -TargetDirectory "windows/tauri/src-tauri/target" ` - -FailureMessage "Windows Rust host tests failed" ` - -CargoArguments @("test", "--manifest-path", "windows/tauri/src-tauri/Cargo.toml") + timeout-minutes: 20 + run: ./.agents/skills/write-stable-tests/scripts/test-stability-windows.ps1 -Scope WindowsRust -SuiteTimeoutSeconds 1080 + + - name: Generate combined Windows test report + if: always() + shell: pwsh + run: node .agents/skills/write-stable-tests/scripts/generate-test-report.mjs + + - name: Upload Windows test reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-stability-windows + path: .artifacts/test-stability/ + if-no-files-found: ignore + retention-days: 14 gate: name: Windows CI gate diff --git a/AGENTS.md b/AGENTS.md index a8b951e24..ee21a47ff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,6 +5,11 @@ at `.agents/skills/develop-lithe/SKILL.md`. That Skill is the single source of truth for AI coding and verification rules, including the required Rust Core comment standard. +If a task creates, modifies, or reviews test code or test infrastructure, +additionally load `.agents/skills/write-stable-tests/SKILL.md` before +proceeding. That Skill defines the mandatory bounded-wait, deterministic-time, +cleanup, and per-test timing rules for both macOS and Windows. + If the task involves building, running, diagnosing, or transferring files to the Windows product through a Parallels guest VM, additionally load `.agents/skills/debug-windows-on-parallels/SKILL.md` before proceeding. diff --git a/scripts/classify-ci-changes.sh b/scripts/classify-ci-changes.sh index 142222a45..d3a213123 100755 --- a/scripts/classify-ci-changes.sh +++ b/scripts/classify-ci-changes.sh @@ -112,6 +112,16 @@ while IFS=$'\t' read -r status first_path _; do rust_core=true macos_release=true ;; + .agents/skills/write-stable-tests/scripts/*) + # Test policy and timing infrastructure can select or reject every + # product test lane, but does not affect release packaging. + swift=true + plugins=true + swift_database=true + rust_core=true + windows=true + windows_rust=true + ;; .github/*|docs/*|.agents/*|.idea/*|.gitignore|license) ;; package.swift|package.resolved) diff --git a/scripts/test-classify-ci-changes.sh b/scripts/test-classify-ci-changes.sh index 61c8f1127..df2cb7b7a 100755 --- a/scripts/test-classify-ci-changes.sh +++ b/scripts/test-classify-ci-changes.sh @@ -137,6 +137,10 @@ modify_shared_fixture() { printf '%s\n' '{"operation":"updated"}' > shared/fixtu modify_windows_frontend() { printf '%s\n' 'export const value = 2;' > windows/tauri/src/value.ts; } modify_windows_rust() { printf '%s\n' 'fn main() { println!("updated"); }' > windows/tauri/src-tauri/src/main.rs; } modify_download_cache_validator() { printf '%s\n' 'console.log("updated");' > scripts/verify-download-cache.mjs; } +modify_test_stability_runner() { + mkdir -p .agents/skills/write-stable-tests/scripts + printf '%s\n' '#!/bin/zsh' 'print -- test-stability' > .agents/skills/write-stable-tests/scripts/test-stability-macos.sh +} modify_macos_cache_action() { mkdir -p .github/actions/prepare-macos-dependency-cache printf '%s\n' 'name: updated' > .github/actions/prepare-macos-dependency-cache/action.yml @@ -239,6 +243,9 @@ assert_classification windows-rust \ assert_classification download-cache-validator \ "$(classification true true true true false true true true false false)" \ modify_download_cache_validator +assert_classification test-stability-runner \ + "$(classification true true true true false false true true false false)" \ + modify_test_stability_runner assert_classification macos-cache-action \ "$(classification true true true true false true false false false false)" \ modify_macos_cache_action