DRAFT - [QUALITY REVIEW SKILL] Test Quality Review

quality-review-cases/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: quality-review-cases
+description: Perform comprehensive quality assessment of test suites. Detect automation and manual test cases, analyze each by quality criteria, and generate a detailed Quality Summary Review with health score and improvement plan.
+license: MIT
+metadata:
+  author: Testomat.io
+  version: 1.0.0
+---
+
+# QUALITY-REVIEW-CASES SKILL: What I do
+
+This skill performs a comprehensive quality assessment of test suites, providing a quantitative health score, detailed findings, and a prioritized improvement plan.
+
+## When to Use
+
+**Trigger this skill whenever user mentions:**
+- Test quality assessment, test health check, test audit
+- "How good are my tests?", "test quality review", "review test suite quality", "test audit", "test health check", or similar.
+- Test case review, test case audit, analyze test quality
+- Any variation of: test suite health, test quality score.
+
+---
+
+## Workflow: Quality Review of Test Cases
+
+### Step 1: Detect Automation Tests
+
+Identify automation tests in the project:
+- Supported frameworks: `Playwright`, `CodeceptJS`, `Jest`, `Mocha`, `WebdriverIO`, `Cypress`, `Robot Framework`, etc.
+- Detect test file like: `test/*.spec.ts`, `tests/**/*.js`, `cypress/e2e/**/*.cy.js`, etc.
+
+> If framework is unclear, inspect repository structure, dependencies to determine the framework.
+
+**Output:**
+- List of automation test files with their paths.
+- Detected framework(s) & Programming language(s).
+
+### Step 2: Detect Manual Test Cases
+
+Identify manual test cases in `.md` format:
+- Scan for markdown files: `**/*.md` (excluding README, CHANGELOG, docs)
+    - Common locations: `tests/manual/`, `test-cases/`, `docs/test-cases/`, `**/test-cases/**/*.md`, etc.
+- Look for test case patterns in markdown (step-by-step format, expected results).
+
+**Output:**
+- List of manual test case files.
+- Test case count per file.
+
+### Step 3.1: Analyze Automation Tests (if their exist)
+
+Evaluate each automation test using criteria from `./references/QUALITY_CRITERIA.md`:
+
+**Automation Criteria (see QUALITY_CRITERIA.md):**
+- Test Logic Clarity (max 3 pts).
+- Code Maintainability & Structure (max 3 pts).
+- Code Smells & Stability (max 2 pts).
+- Assertion Quality & Coverage (max 2 pts).
+
+**Analysis Rules:**
+- Evaluate only code clarity, maintainability, stability, and assertion quality.
+- Do NOT assess functional correctness or rewrite code.
+- Score each criterion and calculate weighted average.
+- Identify specific issues for "Top Tests For Improvements".
+
+### Step 3.2: Analyze Manual Test Cases (if their exist)
+
+Evaluate each manual test using criteria from `./references/QUALITY_CRITERIA.md`:
+
+**Manual Test Criteria (see QUALITY_CRITERIA.md):**
+- Title Clarity (max 2 pts).
+- Steps Defined (max 2 pts).
+- Expected Results (max 2 pts).
+- Viability (max 2 pts).
+- Unambiguity (max 2 pts).
+
+**Analysis Rules:**
+- Parse markdown structure to identify test steps and expected results.
+- Score each criterion per test case.
+- Calculate aggregate scores per dimension.
+
+### Step 4: Generate Quality Report Overview
+
+Compile all findings into a structured Quality Report using the template from `./references/REPORT_TEMPLATES.md`.
+
+### Step 5: Present Quality Review Summary
+
+Deliver the final Quality Review summary to the user with:
+- Executive summary of test suite health.
+- Key findings and critical issues.
+- Prioritized improvement recommendations.
+- Overall health score with star rating.
+
+> Based on the Quality Review Summary, determine and propose concrete next actions.
+
+**Next Steps:**
+1. Improve test cases:
+  * Apply fixes based on identified quality issues and recommendations.
+  * Incorporate any user-provided comments or notes (if available).
+  * If `improve-cases` skill is available, invoke it with:
+    - The prioritized list of issues.
+    - Target test cases or suites.
+    - Expected improvements (e.g., coverage, readability, stability).
+
+2. Clarify scope (if needed):
+  * Ask the user for confirmation before making large-scale or breaking changes.
+  * Request missing context if recommendations cannot be applied directly.
+
+---
+
+## Quality Scoring Methodology
+
+See `./references/REPORT_TEMPLATES.md` for detailed scoring methodology and health rating thresholds.
+
+---
+
+## Error Handling
+
+### Recoverable Issues
+
+* **No test files found**
+  - Ask user to confirm test file locations.
+  - Suggest common test directory patterns.
+
+* **Unclear test framework**
+  - Inspect package.json, config files
+  - Ask user to specify framework
+
+* **Markdown parsing issues**
+  - Handle non-standard markdown formats gracefully.
+  - Flag unparseable tests for manual review.
+
+### Blocking Issues
+
+* **Empty project with no tests**
+  - Report that no tests were found.
+
+* **Insufficient access to test files**
+  - Report access denied.
+  - Request proper permissions.
+
+---
+
+## Example Outputs
+
+### Basic Quality Review
+
+```
+User: Review quality of my test suite
+Agent: Detecting test cases...
+Agent: Found 45 automation tests (Playwright) and 12 manual test cases (.md)
+Agent: Analyzing automation tests...
+Agent: Analyzing manual test cases...
+Agent: Generating Quality Report...
+
+# Test Quality Report Overview
+
+**Date:** 2026-03-17
+**Scope:** /project/tests/*
+**Key Metrics:**
+    - Total test cases: 57
+    - Automated: 45 (79%)
+    - Manual only: 12 (21%)
+**Overall Score:** 7.2 / 10.0 ⭐⭐⭐⭐
+
+[Full report presented]
+```
+
+### Mixed Test Suite Review
+
+```
+User: Audit my e2e tests quality
+Agent: Scanning for test files...
+Agent: Found Playwright (32 tests), Manual .md (28 cases)
+Agent: Analyzing...
+
+Quality Review Complete:
+- Automation Tests: 7.5/10 (Good)
+- Manual Tests: 6.8/10 (Fair)
+- Overall: 7.2/10 ⭐⭐⭐⭐
+
+Top Improvements:
+1. Add expected results to 8 manual tests
+2. Reduce hardcoded waits in 5 automation tests
+3. Standardize test naming convention
+```
+
+---
+
+## References
+
+| Description | File |
+|-------------|------|
+| Quality Criteria Definitions | ./references/QUALITY_CRITERIA.md |
+| Report Templates & Scoring Methodology | ./references/REPORT_TEMPLATES.md |

quality-review-cases/references/QUALITY_CRITERIA.md

@@ -0,0 +1,177 @@
+# Quality Criteria Definitions
+
+This document defines the quality assessment criteria for evaluating both automation and manual test cases.
+
+---
+
+## Automation Test Criteria
+
+Evaluate each automation test using these criteria. Each criterion has a max point value and defined scoring levels.
+
+### 1. Test Logic Clarity
+
+- **Key:** Test Logic Clarity
+- **Max Points:** 3
+- **Description:** How clearly the test expresses its purpose and flow. Are test scenarios easy to interpret without deep digging into the code?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 3 | ✅ Logical and easy to follow | Test flow is clear, purpose is obvious |
+| 2 | 🔶 Some unclear or ambiguous steps | Some parts require investigation to understand |
+| 1 | ❌ Confusing or hard-to-understand flow | Purpose or flow is difficult to discern |
+
+---
+
+### 2. Code Maintainability & Structure
+
+- **Key:** Code Maintainability & Structure
+- **Max Points:** 3
+- **Description:** How clean, organized, and maintainable the test code is — focusing on structure, naming, and reuse.
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 3 | ✅ Well-structured and easy to maintain | Clean code, good naming, reusable patterns |
+| 2 | 🔶 Some duplication or inconsistent structure | Minor issues with organization or naming |
+| 1 | ❌ Hard to modify or poorly organized | Significant maintainability concerns |
+
+---
+
+### 3. Code Smells & Stability
+
+- **Key:** Code Smells & Stability
+- **Max Points:** 2
+- **Description:** Whether the code contains fragile waits, hardcoded data, or patterns that could cause instability.
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Stable and free of code smells | No fragile patterns detected |
+| 1 | 🔶 Some fragile waits or hardcoded data | Minor stability concerns |
+| 0 | ❌ Brittle code or poor error handling | Significant stability issues |
+
+---
+
+### 4. Assertion Quality & Coverage
+
+- **Key:** Assertion Quality & Coverage
+- **Max Points:** 2
+- **Description:** How effectively assertions verify expected behavior and validate outcomes.
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Clear and meaningful assertions | Assertions properly validate outcomes |
+| 1 | 🔶 Some redundant or vague checks | Some unclear or redundant assertions |
+| 0 | ❌ Missing or irrelevant validations | Missing or ineffective assertions |
+
+---
+
+## Manual Test Criteria
+
+Evaluate each manual test case using these criteria. Each criterion has a max point value and defined scoring levels.
+
+### 1. Title Clarity
+
+- **Key:** Title Clarity
+- **Max Points:** 2
+- **Description:** Evaluates if the test case title is specific, clear, and meaningful, or if it is vague/missing.
+- **Context:** Does the title accurately reflect the purpose and scope of the test case and suite?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Clear & meaningful | Title is specific and descriptive |
+| 1 | ⚠️ Partially unclear or incomplete | Title exists but lacks clarity |
+| 0 | ❌ Missing or meaningless | No title or meaningless title |
+
+---
+
+### 2. Steps Defined
+
+- **Key:** Steps Defined
+- **Max Points:** 2
+- **Description:** Evaluates whether the test steps are clear, sequential, and logically structured.
+- **Context:** Do the steps clearly describe how to execute the test and align with the test objective?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Clear, complete, and logical | Steps are well-defined and sequential |
+| 1 | ⚠️ Partially incomplete or unclear | Some steps missing or unclear |
+| 0 | ❌ Missing or insufficient | No steps or insufficient detail |
+
+---
+
+### 3. Expected Results
+
+- **Key:** Expected Results
+- **Max Points:** 2
+- **Description:** Checks if the expected results are explicitly stated, measurable, and relevant to the steps.
+- **Context:** Do the expected results align with the described steps and reflect the intended outcome?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Clear & measurable | Results are explicit and measurable |
+| 1 | ⚠️ Partially unclear or incomplete | Some results stated but unclear |
+| 0 | ❌ Missing or vague | No expected results or very vague |
+
+---
+
+### 4. Viability
+
+- **Key:** Viability
+- **Max Points:** 2
+- **Description:** Assesses whether the test case is executable, coherent, contextually relevant, and free from gaps or contradictions.
+- **Context:** Is the test case logically consistent, relevant to the suite, and practically executable without blocking issues?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Fully viable and coherent | Test can be executed as designed |
+| 1 | ⚠️ Partially viable with minor gaps | Some issues but generally executable |
+| 0 | ❌ Not viable, incoherent, or impossible | Cannot be executed as designed |
+
+---
+
+### 5. Unambiguity
+
+- **Key:** Unambiguity
+- **Max Points:** 2
+- **Description:** Evaluates whether test steps and expected results use precise, testable, and unambiguous language, avoiding generic or subjective terms.
+- **Context:** Are the steps and expected results written in a way that any tester would interpret and execute them the same way?
+
+**Scoring Levels:**
+
+| Points | Label | Description |
+|--------|-------|-------------|
+| 2 | ✅ Unambiguous and testable | Language is precise and testable |
+| 1 | ⚠️ Some ambiguity or generic wording | Some terms are vague |
+| 0 | ❌ Ambiguous, subjective, or open to interpretation | Multiple interpretations possible |
+
+---
+
+## Analysis Rules
+
+### For Automation Tests
+
+- Evaluate only code clarity, maintainability, stability, and assertion quality
+- Do NOT assess functional correctness or rewrite code
+- Score each criterion and calculate weighted average
+- Identify specific issues for "Top Tests For Improvements"
+
+### For Manual Test Cases
+
+- Parse markdown structure to identify test steps and expected results
+- Score each criterion per test case
+- Calculate aggregate scores per dimension