Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/dotnet-test/skills/assertion-quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: assertion-quality
description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / expect(x).toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication (use test-anti-patterns), fixing assertions."
description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit), Python (pytest), TS/JS (Jest/Vitest), Java, Go, Ruby, Rust, Swift, Kotlin, PowerShell (Pester), C++. DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication, or a general severity-ranked anti-pattern audit even when focused on self-referential / tautological assertions and not asking for assertion-diversity metrics (use test-anti-patterns); fixing assertions."
license: MIT
---

Expand Down
8 changes: 6 additions & 2 deletions plugins/dotnet-test/skills/coverage-analysis/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,12 @@ description: >
blocking coverage, coverage gap, CRAP scores, risk hotspots, where to add
tests, coverage analysis, coverage report.
DO NOT USE FOR: targeted single-method CRAP analysis (use crap-score),
writing tests, running tests without coverage, or troubleshooting test
execution (use run-tests).
auditing test code for the "coverage-touching" anti-pattern (tests that
execute / call code but assert nothing, inflating coverage without
verifying behavior) — that is a test-code quality audit, use
test-anti-patterns; writing tests; running tests without coverage, or
troubleshooting test execution (use run-tests). This skill requires or
produces coverage (Cobertura) and CRAP metrics.
license: MIT
---

Expand Down
32 changes: 16 additions & 16 deletions plugins/dotnet-test/skills/test-anti-patterns/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
---
name: test-anti-patterns
description: >
Audits existing test code in any language for anti-patterns and quality
issues — produces a severity-ranked report (Critical / Warning / Info)
with concrete code-level fixes. Polyglot: .NET (MSTest/xUnit/NUnit/
TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test),
Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin
(JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2).
INVOKE when asked to audit, review, rank, or find problems in existing
tests — "audit my tests", "test smell audit", "rank by severity", tests
that pass but verify nothing, no/missing assertions, swallowed
exceptions, always-true / self-comparing / tautological assertions,
broad exception types, flakiness (sleep/Date.now/time.sleep), ordering
dependency, shared global state, duplicated tests, magic values,
missing await on async assertions.
DO NOT USE FOR: writing new tests (use code-testing-agent, or
writing-mstest-tests for MSTest); running tests (use run-tests);
framework migration.
Audits an existing test file or suite in any language for anti-patterns
and quality issues — produces a severity-ranked report
(Critical/Warning/Info). INVOKE whenever asked to audit or review tests,
find what's wrong with a suite, judge whether tests are any good, or
check tests for: tests that pass but verify nothing, no/missing
assertions, swallowed exceptions, self-comparing/self-referential/
tautological assertions (output==input on round-trip/identity ops),
coverage-touching tests (every method called but nothing verified),
broad exceptions, flaky or order-dependent tests (Thread.Sleep,
DateTime.Now, time.sleep, shared state, reflection coupling),
duplicated tests, magic values — in .NET, Python/pytest, TS/Jest, Java,
Go, Ruby or C++. DO NOT USE FOR: writing new tests (use
code-testing-agent); running tests (use run-tests); migration;
assertion-diversity metrics (use assertion-quality); coverage/CRAP
metrics (use coverage-analysis); the testsmells.org academic catalog
(use test-smell-detection).
license: MIT
---

Expand Down
6 changes: 3 additions & 3 deletions tests/dotnet-test/assertion-quality/eval.vally.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,15 +97,15 @@ stimuli:
- If any gaps were identified, they were presented as minor enhancement opportunities rather than problems
- name: Identify self-referential assertions in identity and round-trip tests
prompt: |
I want an assertion-quality analysis of my Config tests. They all pass
I want an assertion quality analysis of my Config tests. They all pass
but I'm not sure they'd catch real bugs. Analyze the assertion quality
and depth in `ConfigService.Tests/ConfigTests.cs` — specifically
identify self-referential or tautological assertions where the test
asserts the output equals the input on identity / round-trip
operations. Report assertion-quality metrics (how many tests are
operations. Report assertion quality metrics (how many tests are
self-referential, which ones), explain why each flagged test fails to
verify real behavior, and recommend concrete improvements. Do not
edit the files — this is a read-only assertion-quality review.
edit the files — this is a read-only assertion quality review.
environment:
files:
- src: fixtures/self-referential/ConfigService.Tests/ConfigTests.cs
Expand Down
12 changes: 6 additions & 6 deletions tests/dotnet-test/assertion-quality/eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,15 +97,15 @@ scenarios:

- name: "Identify self-referential assertions in identity and round-trip tests"
prompt: |
I want an assertion-quality analysis of my Config tests. They all pass
I want an assertion quality analysis of my Config tests. They all pass
but I'm not sure they'd catch real bugs. Analyze the assertion quality
and depth in `ConfigService.Tests/ConfigTests.cs` — specifically
identify self-referential or tautological assertions where the test
asserts the output equals the input on identity / round-trip
operations. Report assertion-quality metrics (how many tests are
operations. Report assertion quality metrics (how many tests are
self-referential, which ones), explain why each flagged test fails to
verify real behavior, and recommend concrete improvements. Do not
edit the files — this is a read-only assertion-quality review.
edit the files — this is a read-only assertion quality review.
setup:
files:
- path: "ConfigService.Tests/ConfigService.Tests.csproj"
Expand Down Expand Up @@ -166,9 +166,9 @@ scenarios:
prompt: |
Audit the assertion quality and depth of `tests/order.test.ts`. The
production code is `src/order.ts`. This is a Jest/TypeScript suite,
not .NET — apply the same assertion-quality catalog using
Jest equivalents (`toBeDefined`, `toBeTruthy`, `not.toBeNull`,
`toBe`, `.resolves`, `toThrow`).
not .NET — apply the same assertion quality catalog using
Jest equivalents (`toBeDefined()`, `toBeTruthy()`, `not.toBeNull()`,
`toBe()`, `.resolves`, `toThrow()`).
Comment thread
Evangelink marked this conversation as resolved.

Specifically check for: shallow / trivial-only assertions, always-true
assertions, missing `await` on async assertion chains (a silent-pass
Expand Down
14 changes: 5 additions & 9 deletions tests/dotnet-test/test-anti-patterns/eval.vally.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ stimuli:
not actually verifying anything. Audit the test class in
`UserRepo.Tests/UserRepositoryTests.cs` for .NET test anti-patterns
(use the production code at `UserRepo/UserRepository.cs` for context).
Look for the full anti-pattern catalog: missing/no assertions,
swallowed exceptions, always-true / self-comparing assertions, weak
negative assertions, broad exception types, unused fields, and
anything else that could let bugs sneak through.
I want to know which tests look like they pass but wouldn't actually
catch a regression.

Give me a severity-ranked list (Critical / Warning / Info) with each
finding named (e.g. "swallowed exception", "self-comparing assertion")
Expand Down Expand Up @@ -64,11 +62,9 @@ stimuli:
prompt: |
I have a test suite that sometimes passes and sometimes fails depending
on execution order. Audit `FlakyCoupled.Tests/OrderServiceTests.cs` for
.NET test anti-patterns. Specifically look for these named patterns:
flakiness indicators (Thread.Sleep, DateTime.Now, real network),
test ordering dependency, shared/static state across tests,
reflection-based access to private members (implementation coupling),
god tests that bundle many assertions, and broad exception types.
.NET test anti-patterns. Help me figure out why it's order-dependent
and what else about these tests is fragile, non-deterministic, or
coupled to implementation details.

Give me a severity-ranked list (Critical / Warning / Info) with each
finding named (e.g. "shared static state", "Thread.Sleep flakiness",
Expand Down
14 changes: 5 additions & 9 deletions tests/dotnet-test/test-anti-patterns/eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,8 @@ scenarios:
not actually verifying anything. Audit the test class in
`UserRepo.Tests/UserRepositoryTests.cs` for .NET test anti-patterns
(use the production code at `UserRepo/UserRepository.cs` for context).
Look for the full anti-pattern catalog: missing/no assertions,
swallowed exceptions, always-true / self-comparing assertions, weak
negative assertions, broad exception types, unused fields, and
anything else that could let bugs sneak through.
I want to know which tests look like they pass but wouldn't actually
catch a regression.

Give me a severity-ranked list (Critical / Warning / Info) with each
finding named (e.g. "swallowed exception", "self-comparing assertion")
Expand Down Expand Up @@ -47,11 +45,9 @@ scenarios:
prompt: |
I have a test suite that sometimes passes and sometimes fails depending
on execution order. Audit `FlakyCoupled.Tests/OrderServiceTests.cs` for
.NET test anti-patterns. Specifically look for these named patterns:
flakiness indicators (Thread.Sleep, DateTime.Now, real network),
test ordering dependency, shared/static state across tests,
reflection-based access to private members (implementation coupling),
god tests that bundle many assertions, and broad exception types.
.NET test anti-patterns. Help me figure out why it's order-dependent
and what else about these tests is fragile, non-deterministic, or
coupled to implementation details.

Give me a severity-ranked list (Critical / Warning / Info) with each
finding named (e.g. "shared static state", "Thread.Sleep flakiness",
Expand Down
Loading