From a2d756f36896bf682208768f7555eec21d328a00 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Tue, 17 Mar 2026 11:39:43 +0200 Subject: [PATCH 1/4] [QUALITY REVIEW SKILL] New quality-review-cases QA skill --- seed-skills/quality-review-cases/SKILL.md | 179 ++++++++++++++++++ .../references/QUALITY_CRITERIA.md | 177 +++++++++++++++++ .../references/REPORT_TEMPLATES.md | 108 +++++++++++ 3 files changed, 464 insertions(+) create mode 100644 seed-skills/quality-review-cases/SKILL.md create mode 100644 seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md create mode 100644 seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md diff --git a/seed-skills/quality-review-cases/SKILL.md b/seed-skills/quality-review-cases/SKILL.md new file mode 100644 index 0000000..60cd39b --- /dev/null +++ b/seed-skills/quality-review-cases/SKILL.md @@ -0,0 +1,179 @@ +--- +name: quality-review-cases +description: Perform comprehensive quality assessment of test suites. Detect automation and manual test cases, analyze each by quality criteria, and generate a detailed Quality Summary Review with health score and improvement plan. +license: MIT +metadata: + author: Testomat.io + version: 1.0.0 +--- + +# QUALITY-REVIEW-CASES SKILL: What I do + +This skill performs a comprehensive quality assessment of test suites, providing a quantitative health score, detailed findings, and a prioritized improvement plan. + +## When to Use + +Trigger this skill when user wants: +- Overall assessment of their test suite quality. +- Quality gate check before a release. +- Answers to "how good are my tests?" or "provide test quality review", etc. +- Test case quality review, test audit, or test health check. + +--- + +## Workflow: Quality Review of Test Cases + +### Step 1: Detect Automation Tests + +Identify automation tests in the project: +- Supported frameworks: `Playwright`, `CodeceptJS`, `Jest`, `Mocha`, `WebdriverIO`, `Cypress`, `Robot Framework`, etc. +- Detect test file like: `test/*.spec.ts`, `tests/**/*.js`, `cypress/e2e/**/*.cy.js`, etc. + +> If framework is unclear, inspect repository structure, dependencies to determine the framework. + +**Output:** +- List of automation test files with their paths. +- Detected framework(s) & Programming language(s). + +### Step 2: Detect Manual Test Cases + +Identify manual test cases in `.md` format: +- Scan for markdown files: `**/*.md` (excluding README, CHANGELOG, docs) + - Common locations: `tests/manual/`, `test-cases/`, `docs/test-cases/`, `**/test-cases/**/*.md`, etc. +- Look for test case patterns in markdown (step-by-step format, expected results). + +**Output:** +- List of manual test case files. +- Test case count per file. + +### Step 3.1: Analyze Automation Tests (if their exist) + +Evaluate each automation test using criteria from `./references/QUALITY_CRITERIA.md`: + +**Automation Criteria (see QUALITY_CRITERIA.md):** +- Test Logic Clarity (max 3 pts). +- Code Maintainability & Structure (max 3 pts). +- Code Smells & Stability (max 2 pts). +- Assertion Quality & Coverage (max 2 pts). + +**Analysis Rules:** +- Evaluate only code clarity, maintainability, stability, and assertion quality. +- Do NOT assess functional correctness or rewrite code. +- Score each criterion and calculate weighted average. +- Identify specific issues for "Top Tests For Improvements". + +### Step 3.2: Analyze Manual Test Cases (if their exist) + +Evaluate each manual test using criteria from `./references/QUALITY_CRITERIA.md`: + +**Manual Test Criteria (see QUALITY_CRITERIA.md):** +- Title Clarity (max 2 pts). +- Steps Defined (max 2 pts). +- Expected Results (max 2 pts). +- Viability (max 2 pts). +- Unambiguity (max 2 pts). + +**Analysis Rules:** +- Parse markdown structure to identify test steps and expected results. +- Score each criterion per test case. +- Calculate aggregate scores per dimension. + +### Step 4: Generate Quality Report Overview + +Compile all findings into a structured Quality Report using the template from `./references/REPORT_TEMPLATES.md`. + +### Step 6: Present Quality Review Summary + +Deliver the final Quality Review summary to the user with: +- Executive summary of test suite health. +- Key findings and critical issues. +- Prioritized improvement recommendations. +- Overall health score with star rating. + +--- + +## Quality Scoring Methodology + +See `./references/REPORT_TEMPLATES.md` for detailed scoring methodology and health rating thresholds. + +--- + +## Error Handling + +### Recoverable Issues + +* **No test files found** + - Ask user to confirm test file locations. + - Suggest common test directory patterns. + +* **Unclear test framework** + - Inspect package.json, config files + - Ask user to specify framework + +* **Markdown parsing issues** + - Handle non-standard markdown formats gracefully. + - Flag unparseable tests for manual review. + +### Blocking Issues + +* **Empty project with no tests** + - Report that no tests were found. + +* **Insufficient access to test files** + - Report access denied. + - Request proper permissions. + +--- + +## Example Outputs + +### Basic Quality Review + +``` +User: Review quality of my test suite +Agent: Detecting test cases... +Agent: Found 45 automation tests (Playwright) and 12 manual test cases (.md) +Agent: Analyzing automation tests... +Agent: Analyzing manual test cases... +Agent: Generating Quality Report... + +# Test Quality Report Overview + +**Date:** 2026-03-17 +**Scope:** /project/tests/* +**Key Metrics:** + - Total test cases: 57 + - Automated: 45 (79%) + - Manual only: 12 (21%) +**Overall Score:** 7.2 / 10.0 ⭐⭐⭐⭐ + +[Full report presented] +``` + +### Mixed Test Suite Review + +``` +User: Audit my e2e tests quality +Agent: Scanning for test files... +Agent: Found Playwright (32 tests), Manual .md (28 cases) +Agent: Analyzing... + +Quality Review Complete: +- Automation Tests: 7.5/10 (Good) +- Manual Tests: 6.8/10 (Fair) +- Overall: 7.2/10 ⭐⭐⭐⭐ + +Top Improvements: +1. Add expected results to 8 manual tests +2. Reduce hardcoded waits in 5 automation tests +3. Standardize test naming convention +``` + +--- + +## References + +| Description | File | +|-------------|------| +| Quality Criteria Definitions | ./references/QUALITY_CRITERIA.md | +| Report Templates & Scoring Methodology | ./references/REPORT_TEMPLATES.md | diff --git a/seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md b/seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md new file mode 100644 index 0000000..182454a --- /dev/null +++ b/seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md @@ -0,0 +1,177 @@ +# Quality Criteria Definitions + +This document defines the quality assessment criteria for evaluating both automation and manual test cases. + +--- + +## Automation Test Criteria + +Evaluate each automation test using these criteria. Each criterion has a max point value and defined scoring levels. + +### 1. Test Logic Clarity + +- **Key:** Test Logic Clarity +- **Max Points:** 3 +- **Description:** How clearly the test expresses its purpose and flow. Are test scenarios easy to interpret without deep digging into the code? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 3 | ✅ Logical and easy to follow | Test flow is clear, purpose is obvious | +| 2 | 🔶 Some unclear or ambiguous steps | Some parts require investigation to understand | +| 1 | ❌ Confusing or hard-to-understand flow | Purpose or flow is difficult to discern | + +--- + +### 2. Code Maintainability & Structure + +- **Key:** Code Maintainability & Structure +- **Max Points:** 3 +- **Description:** How clean, organized, and maintainable the test code is — focusing on structure, naming, and reuse. + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 3 | ✅ Well-structured and easy to maintain | Clean code, good naming, reusable patterns | +| 2 | 🔶 Some duplication or inconsistent structure | Minor issues with organization or naming | +| 1 | ❌ Hard to modify or poorly organized | Significant maintainability concerns | + +--- + +### 3. Code Smells & Stability + +- **Key:** Code Smells & Stability +- **Max Points:** 2 +- **Description:** Whether the code contains fragile waits, hardcoded data, or patterns that could cause instability. + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Stable and free of code smells | No fragile patterns detected | +| 1 | 🔶 Some fragile waits or hardcoded data | Minor stability concerns | +| 0 | ❌ Brittle code or poor error handling | Significant stability issues | + +--- + +### 4. Assertion Quality & Coverage + +- **Key:** Assertion Quality & Coverage +- **Max Points:** 2 +- **Description:** How effectively assertions verify expected behavior and validate outcomes. + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Clear and meaningful assertions | Assertions properly validate outcomes | +| 1 | 🔶 Some redundant or vague checks | Some unclear or redundant assertions | +| 0 | ❌ Missing or irrelevant validations | Missing or ineffective assertions | + +--- + +## Manual Test Criteria + +Evaluate each manual test case using these criteria. Each criterion has a max point value and defined scoring levels. + +### 1. Title Clarity + +- **Key:** Title Clarity +- **Max Points:** 2 +- **Description:** Evaluates if the test case title is specific, clear, and meaningful, or if it is vague/missing. +- **Context:** Does the title accurately reflect the purpose and scope of the test case and suite? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Clear & meaningful | Title is specific and descriptive | +| 1 | ⚠️ Partially unclear or incomplete | Title exists but lacks clarity | +| 0 | ❌ Missing or meaningless | No title or meaningless title | + +--- + +### 2. Steps Defined + +- **Key:** Steps Defined +- **Max Points:** 2 +- **Description:** Evaluates whether the test steps are clear, sequential, and logically structured. +- **Context:** Do the steps clearly describe how to execute the test and align with the test objective? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Clear, complete, and logical | Steps are well-defined and sequential | +| 1 | ⚠️ Partially incomplete or unclear | Some steps missing or unclear | +| 0 | ❌ Missing or insufficient | No steps or insufficient detail | + +--- + +### 3. Expected Results + +- **Key:** Expected Results +- **Max Points:** 2 +- **Description:** Checks if the expected results are explicitly stated, measurable, and relevant to the steps. +- **Context:** Do the expected results align with the described steps and reflect the intended outcome? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Clear & measurable | Results are explicit and measurable | +| 1 | ⚠️ Partially unclear or incomplete | Some results stated but unclear | +| 0 | ❌ Missing or vague | No expected results or very vague | + +--- + +### 4. Viability + +- **Key:** Viability +- **Max Points:** 2 +- **Description:** Assesses whether the test case is executable, coherent, contextually relevant, and free from gaps or contradictions. +- **Context:** Is the test case logically consistent, relevant to the suite, and practically executable without blocking issues? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Fully viable and coherent | Test can be executed as designed | +| 1 | ⚠️ Partially viable with minor gaps | Some issues but generally executable | +| 0 | ❌ Not viable, incoherent, or impossible | Cannot be executed as designed | + +--- + +### 5. Unambiguity + +- **Key:** Unambiguity +- **Max Points:** 2 +- **Description:** Evaluates whether test steps and expected results use precise, testable, and unambiguous language, avoiding generic or subjective terms. +- **Context:** Are the steps and expected results written in a way that any tester would interpret and execute them the same way? + +**Scoring Levels:** + +| Points | Label | Description | +|--------|-------|-------------| +| 2 | ✅ Unambiguous and testable | Language is precise and testable | +| 1 | ⚠️ Some ambiguity or generic wording | Some terms are vague | +| 0 | ❌ Ambiguous, subjective, or open to interpretation | Multiple interpretations possible | + +--- + +## Analysis Rules + +### For Automation Tests + +- Evaluate only code clarity, maintainability, stability, and assertion quality +- Do NOT assess functional correctness or rewrite code +- Score each criterion and calculate weighted average +- Identify specific issues for "Top Tests For Improvements" + +### For Manual Test Cases + +- Parse markdown structure to identify test steps and expected results +- Score each criterion per test case +- Calculate aggregate scores per dimension \ No newline at end of file diff --git a/seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md b/seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md new file mode 100644 index 0000000..2ef922a --- /dev/null +++ b/seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md @@ -0,0 +1,108 @@ +# Report Templates + +This document contains templates for generating Quality Review reports. + +--- + +## Quality Report Overview Template + +Use this template to compile all findings into a structured Quality Report. + +```md +# Test Quality Report Overview + +**Date:** [YYYY-MM-DD] +**Scope:** [Path to test files] +**Key Metrics:** + - Total test cases: [N] + - Automated: [N] ([X]%) + - Manual only: [N] ([X]%) + - Tests with expected results on all steps: [N] ([X]%) + - Tests with tags: [N] ([X]%) +**Overall Score:** [X.X] / 10.0 ⭐⭐⭐ + +## Manual Key Quality Dimensions + +| Dimension | Score | Status | +|-----------|-------|--------| +| Title Clarity | X.X/10 | ... | +| Steps Defined | X.X/10 | ... | +| Expected Results | X.X/10 | ... | +| Viability | X.X/10 | ... | +| Unambiguity | X.X/10 | ... | + +## Automation Key Quality Dimensions + +| Dimension | Score | Status | +|-----------|-------|--------| +| Test Logic Clarity | X/10 | ... | +| Code Maintainability | X/10 | ... | +| Code Smells & Stability | X/10 | ... | +| Assertion Quality | X/10 | ... | + +## Top 5 Tests For Improvements + +### 1. 🔴 [Test Name] [Path] +**Impact:** [High/Medium/Low] — [Description of issue] +**Recommendation:** [Specific action to improve] + +### 2. 🟡 [Test Name] [Path] +... + +## Improvement Roadmap + +### Stage 1 (Critical) +- [ ] [Action item 1] +- [ ] [Action item 2] + +### Stage 2 (Important) +- [ ] [Action item 1] +- [ ] [Action item 2] + +### Stage 3 (Nice to Have) +- [ ] [Action item 1] +``` + +--- + +## Scoring Methodology + +### Overall Score Calculation + +**Automation Tests Score (0-10):** +``` +(Logic Score / 3 * 25) + (Structure Score / 3 * 25) + (Smell Score / 2 * 25) + (Coverage Score / 2 * 25) +``` + +**Manual Tests Score (0-10):** +``` +(Title Score / 2 * 20) + (Steps Score / 2 * 20) + (Results Score / 2 * 20) + (Viable Score / 2 * 20) + (Clarity Score / 2 * 20) +``` + +**Combined Score:** +``` +(Automation Count * Auto Score + Manual Count * Manual Score) / Total Count +``` + +### Health Rating Thresholds + +| Score Range | Rating | +|-------------|--------| +| 8.0 - 10.0 | ⭐⭐⭐⭐⭐ Excellent | +| 6.0 - 7.9 | ⭐⭐⭐⭐ Good | +| 4.0 - 5.9 | ⭐⭐⭐ Fair | +| 2.0 - 3.9 | ⭐⭐ Poor | +| 0.0 - 1.9 | ⭐ Critical | + +--- + +## Quick Commands Reference + +| Action | Command | +|--------|---------| +| Full Review | Use quality-review-cases skill on `/tests` directory | +| Quick Scan | Review only automation tests | +| Manual Only | Review only `.md` test cases | +| Summary | Generate executive summary only | + +--- \ No newline at end of file From 276370440374b3107b6545378e3fdc154d1bb3d7 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Tue, 17 Mar 2026 11:47:51 +0200 Subject: [PATCH 2/4] [QUALITY REVIEW SKILL] small trigger updates --- seed-skills/quality-review-cases/SKILL.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/seed-skills/quality-review-cases/SKILL.md b/seed-skills/quality-review-cases/SKILL.md index 60cd39b..072043b 100644 --- a/seed-skills/quality-review-cases/SKILL.md +++ b/seed-skills/quality-review-cases/SKILL.md @@ -13,11 +13,11 @@ This skill performs a comprehensive quality assessment of test suites, providing ## When to Use -Trigger this skill when user wants: -- Overall assessment of their test suite quality. -- Quality gate check before a release. -- Answers to "how good are my tests?" or "provide test quality review", etc. -- Test case quality review, test audit, or test health check. +**Trigger this skill whenever user mentions:** +- Test quality assessment, test health check, test audit +- "How good are my tests?", "test quality review", "review test suite quality", "test audit", "test health check", or similar. +- Test case review, test case audit, analyze test quality +- Any variation of: test suite health, test quality score. --- @@ -82,7 +82,7 @@ Evaluate each manual test using criteria from `./references/QUALITY_CRITERIA.md` Compile all findings into a structured Quality Report using the template from `./references/REPORT_TEMPLATES.md`. -### Step 6: Present Quality Review Summary +### Step 5: Present Quality Review Summary Deliver the final Quality Review summary to the user with: - Executive summary of test suite health. From dc8084d8e456fc9605acca376280332b4d180a39 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Tue, 17 Mar 2026 13:13:02 +0200 Subject: [PATCH 3/4] [QUALITY REVIEW SKILL] move skill folder to root --- .../quality-review-cases => quality-review-cases}/SKILL.md | 0 .../references/QUALITY_CRITERIA.md | 0 .../references/REPORT_TEMPLATES.md | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {seed-skills/quality-review-cases => quality-review-cases}/SKILL.md (100%) rename {seed-skills/quality-review-cases => quality-review-cases}/references/QUALITY_CRITERIA.md (100%) rename {seed-skills/quality-review-cases => quality-review-cases}/references/REPORT_TEMPLATES.md (100%) diff --git a/seed-skills/quality-review-cases/SKILL.md b/quality-review-cases/SKILL.md similarity index 100% rename from seed-skills/quality-review-cases/SKILL.md rename to quality-review-cases/SKILL.md diff --git a/seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md b/quality-review-cases/references/QUALITY_CRITERIA.md similarity index 100% rename from seed-skills/quality-review-cases/references/QUALITY_CRITERIA.md rename to quality-review-cases/references/QUALITY_CRITERIA.md diff --git a/seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md b/quality-review-cases/references/REPORT_TEMPLATES.md similarity index 100% rename from seed-skills/quality-review-cases/references/REPORT_TEMPLATES.md rename to quality-review-cases/references/REPORT_TEMPLATES.md From 71269efe07f8b6806abd25be4801a7461d7fcbc1 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Tue, 17 Mar 2026 15:15:16 +0200 Subject: [PATCH 4/4] [QUALITY REVIEW SKILL] Next step instructions. --- quality-review-cases/SKILL.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/quality-review-cases/SKILL.md b/quality-review-cases/SKILL.md index 072043b..70d455c 100644 --- a/quality-review-cases/SKILL.md +++ b/quality-review-cases/SKILL.md @@ -90,6 +90,21 @@ Deliver the final Quality Review summary to the user with: - Prioritized improvement recommendations. - Overall health score with star rating. +> Based on the Quality Review Summary, determine and propose concrete next actions. + +**Next Steps:** +1. Improve test cases: + * Apply fixes based on identified quality issues and recommendations. + * Incorporate any user-provided comments or notes (if available). + * If `improve-cases` skill is available, invoke it with: + - The prioritized list of issues. + - Target test cases or suites. + - Expected improvements (e.g., coverage, readability, stability). + +2. Clarify scope (if needed): + * Ask the user for confirmation before making large-scale or breaking changes. + * Request missing context if recommendations cannot be applied directly. + --- ## Quality Scoring Methodology