diff --git a/plugins/dotnet-test/skills/assertion-quality/SKILL.md b/plugins/dotnet-test/skills/assertion-quality/SKILL.md index 4cf816ebe2..2eac1adb83 100644 --- a/plugins/dotnet-test/skills/assertion-quality/SKILL.md +++ b/plugins/dotnet-test/skills/assertion-quality/SKILL.md @@ -1,6 +1,6 @@ --- name: assertion-quality -description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / expect(x).toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication (use test-anti-patterns), fixing assertions." +description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit), Python (pytest), TS/JS (Jest/Vitest), Java, Go, Ruby, Rust, Swift, Kotlin, PowerShell (Pester), C++. DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication, or a general severity-ranked anti-pattern audit even when focused on self-referential / tautological assertions and not asking for assertion-diversity metrics (use test-anti-patterns); fixing assertions." license: MIT --- diff --git a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md index 128f2bdec4..2193c48e29 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md +++ b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md @@ -10,8 +10,12 @@ description: > blocking coverage, coverage gap, CRAP scores, risk hotspots, where to add tests, coverage analysis, coverage report. DO NOT USE FOR: targeted single-method CRAP analysis (use crap-score), - writing tests, running tests without coverage, or troubleshooting test - execution (use run-tests). + auditing test code for the "coverage-touching" anti-pattern (tests that + execute / call code but assert nothing, inflating coverage without + verifying behavior) — that is a test-code quality audit, use + test-anti-patterns; writing tests; running tests without coverage, or + troubleshooting test execution (use run-tests). This skill requires or + produces coverage (Cobertura) and CRAP metrics. license: MIT --- diff --git a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md index 7cdd67d891..a3c90b6df6 100644 --- a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md +++ b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md @@ -1,22 +1,22 @@ --- name: test-anti-patterns description: > - Audits existing test code in any language for anti-patterns and quality - issues — produces a severity-ranked report (Critical / Warning / Info) - with concrete code-level fixes. Polyglot: .NET (MSTest/xUnit/NUnit/ - TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), - Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin - (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). - INVOKE when asked to audit, review, rank, or find problems in existing - tests — "audit my tests", "test smell audit", "rank by severity", tests - that pass but verify nothing, no/missing assertions, swallowed - exceptions, always-true / self-comparing / tautological assertions, - broad exception types, flakiness (sleep/Date.now/time.sleep), ordering - dependency, shared global state, duplicated tests, magic values, - missing await on async assertions. - DO NOT USE FOR: writing new tests (use code-testing-agent, or - writing-mstest-tests for MSTest); running tests (use run-tests); - framework migration. + Audits an existing test file or suite in any language for anti-patterns + and quality issues — produces a severity-ranked report + (Critical/Warning/Info). INVOKE whenever asked to audit or review tests, + find what's wrong with a suite, judge whether tests are any good, or + check tests for: tests that pass but verify nothing, no/missing + assertions, swallowed exceptions, self-comparing/self-referential/ + tautological assertions (output==input on round-trip/identity ops), + coverage-touching tests (every method called but nothing verified), + broad exceptions, flaky or order-dependent tests (Thread.Sleep, + DateTime.Now, time.sleep, shared state, reflection coupling), + duplicated tests, magic values — in .NET, Python/pytest, TS/Jest, Java, + Go, Ruby or C++. DO NOT USE FOR: writing new tests (use + code-testing-agent); running tests (use run-tests); migration; + assertion-diversity metrics (use assertion-quality); coverage/CRAP + metrics (use coverage-analysis); the testsmells.org academic catalog + (use test-smell-detection). license: MIT --- diff --git a/tests/dotnet-test/assertion-quality/eval.vally.yaml b/tests/dotnet-test/assertion-quality/eval.vally.yaml index a253b6e836..7e740d3581 100644 --- a/tests/dotnet-test/assertion-quality/eval.vally.yaml +++ b/tests/dotnet-test/assertion-quality/eval.vally.yaml @@ -97,15 +97,15 @@ stimuli: - If any gaps were identified, they were presented as minor enhancement opportunities rather than problems - name: Identify self-referential assertions in identity and round-trip tests prompt: | - I want an assertion-quality analysis of my Config tests. They all pass + I want an assertion quality analysis of my Config tests. They all pass but I'm not sure they'd catch real bugs. Analyze the assertion quality and depth in `ConfigService.Tests/ConfigTests.cs` — specifically identify self-referential or tautological assertions where the test asserts the output equals the input on identity / round-trip - operations. Report assertion-quality metrics (how many tests are + operations. Report assertion quality metrics (how many tests are self-referential, which ones), explain why each flagged test fails to verify real behavior, and recommend concrete improvements. Do not - edit the files — this is a read-only assertion-quality review. + edit the files — this is a read-only assertion quality review. environment: files: - src: fixtures/self-referential/ConfigService.Tests/ConfigTests.cs diff --git a/tests/dotnet-test/assertion-quality/eval.yaml b/tests/dotnet-test/assertion-quality/eval.yaml index 32d32e2b37..03197452da 100644 --- a/tests/dotnet-test/assertion-quality/eval.yaml +++ b/tests/dotnet-test/assertion-quality/eval.yaml @@ -97,15 +97,15 @@ scenarios: - name: "Identify self-referential assertions in identity and round-trip tests" prompt: | - I want an assertion-quality analysis of my Config tests. They all pass + I want an assertion quality analysis of my Config tests. They all pass but I'm not sure they'd catch real bugs. Analyze the assertion quality and depth in `ConfigService.Tests/ConfigTests.cs` — specifically identify self-referential or tautological assertions where the test asserts the output equals the input on identity / round-trip - operations. Report assertion-quality metrics (how many tests are + operations. Report assertion quality metrics (how many tests are self-referential, which ones), explain why each flagged test fails to verify real behavior, and recommend concrete improvements. Do not - edit the files — this is a read-only assertion-quality review. + edit the files — this is a read-only assertion quality review. setup: files: - path: "ConfigService.Tests/ConfigService.Tests.csproj" @@ -166,9 +166,9 @@ scenarios: prompt: | Audit the assertion quality and depth of `tests/order.test.ts`. The production code is `src/order.ts`. This is a Jest/TypeScript suite, - not .NET — apply the same assertion-quality catalog using - Jest equivalents (`toBeDefined`, `toBeTruthy`, `not.toBeNull`, - `toBe`, `.resolves`, `toThrow`). + not .NET — apply the same assertion quality catalog using + Jest equivalents (`toBeDefined()`, `toBeTruthy()`, `not.toBeNull()`, + `toBe()`, `.resolves`, `toThrow()`). Specifically check for: shallow / trivial-only assertions, always-true assertions, missing `await` on async assertion chains (a silent-pass diff --git a/tests/dotnet-test/test-anti-patterns/eval.vally.yaml b/tests/dotnet-test/test-anti-patterns/eval.vally.yaml index 276657c36a..8b09752d4f 100644 --- a/tests/dotnet-test/test-anti-patterns/eval.vally.yaml +++ b/tests/dotnet-test/test-anti-patterns/eval.vally.yaml @@ -10,10 +10,8 @@ stimuli: not actually verifying anything. Audit the test class in `UserRepo.Tests/UserRepositoryTests.cs` for .NET test anti-patterns (use the production code at `UserRepo/UserRepository.cs` for context). - Look for the full anti-pattern catalog: missing/no assertions, - swallowed exceptions, always-true / self-comparing assertions, weak - negative assertions, broad exception types, unused fields, and - anything else that could let bugs sneak through. + I want to know which tests look like they pass but wouldn't actually + catch a regression. Give me a severity-ranked list (Critical / Warning / Info) with each finding named (e.g. "swallowed exception", "self-comparing assertion") @@ -64,11 +62,9 @@ stimuli: prompt: | I have a test suite that sometimes passes and sometimes fails depending on execution order. Audit `FlakyCoupled.Tests/OrderServiceTests.cs` for - .NET test anti-patterns. Specifically look for these named patterns: - flakiness indicators (Thread.Sleep, DateTime.Now, real network), - test ordering dependency, shared/static state across tests, - reflection-based access to private members (implementation coupling), - god tests that bundle many assertions, and broad exception types. + .NET test anti-patterns. Help me figure out why it's order-dependent + and what else about these tests is fragile, non-deterministic, or + coupled to implementation details. Give me a severity-ranked list (Critical / Warning / Info) with each finding named (e.g. "shared static state", "Thread.Sleep flakiness", diff --git a/tests/dotnet-test/test-anti-patterns/eval.yaml b/tests/dotnet-test/test-anti-patterns/eval.yaml index b64e22ba31..079e20e999 100644 --- a/tests/dotnet-test/test-anti-patterns/eval.yaml +++ b/tests/dotnet-test/test-anti-patterns/eval.yaml @@ -5,10 +5,8 @@ scenarios: not actually verifying anything. Audit the test class in `UserRepo.Tests/UserRepositoryTests.cs` for .NET test anti-patterns (use the production code at `UserRepo/UserRepository.cs` for context). - Look for the full anti-pattern catalog: missing/no assertions, - swallowed exceptions, always-true / self-comparing assertions, weak - negative assertions, broad exception types, unused fields, and - anything else that could let bugs sneak through. + I want to know which tests look like they pass but wouldn't actually + catch a regression. Give me a severity-ranked list (Critical / Warning / Info) with each finding named (e.g. "swallowed exception", "self-comparing assertion") @@ -47,11 +45,9 @@ scenarios: prompt: | I have a test suite that sometimes passes and sometimes fails depending on execution order. Audit `FlakyCoupled.Tests/OrderServiceTests.cs` for - .NET test anti-patterns. Specifically look for these named patterns: - flakiness indicators (Thread.Sleep, DateTime.Now, real network), - test ordering dependency, shared/static state across tests, - reflection-based access to private members (implementation coupling), - god tests that bundle many assertions, and broad exception types. + .NET test anti-patterns. Help me figure out why it's order-dependent + and what else about these tests is fragile, non-deterministic, or + coupled to implementation details. Give me a severity-ranked list (Critical / Warning / Info) with each finding named (e.g. "shared static state", "Thread.Sleep flakiness",