From 2b0b50d79e6500342c30a76b303fd4db66bd1edb Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Mon, 1 Jun 2026 12:30:57 +0300 Subject: [PATCH 1/5] [PR REQUIREMENTS] A new skill to check pr-requirements for futher testing --- skills/pr-requirements-check/SKILL.md | 429 ++++++++++++++++++ .../references/summary-example.md | 56 +++ 2 files changed, 485 insertions(+) create mode 100644 skills/pr-requirements-check/SKILL.md create mode 100644 skills/pr-requirements-check/references/summary-example.md diff --git a/skills/pr-requirements-check/SKILL.md b/skills/pr-requirements-check/SKILL.md new file mode 100644 index 0000000..6f62745 --- /dev/null +++ b/skills/pr-requirements-check/SKILL.md @@ -0,0 +1,429 @@ +--- +name: pr-requirements-check +description: Analyze a pull request's context — title, description, comments, linked Jira/GitHub issues, attached images — to understand the original intent, verify scope against the linked ticket, surface ambiguities and edge cases, and produce a structured requirements summary with testable acceptance criteria saved as a markdown file. Use this skill when the user wants to understand WHAT a PR is supposed to do (not just what code changed), check whether the PR matches its linked ticket/task, find gaps or missing requirements, or generate acceptance criteria from PR/ticket context rather than from code. +license: MIT +metadata: + author: Testomat.io + version: 1.0.0 +--- + +# PR-REQUIREMENTS-CHECK SKILL: What I Do + +This skill reviews a pull request's **context** - the human-written parts: title, description, comments, linked tickets, attached images - to extract the original requirements, verify scope, and produce a requirements summary that downstream skills (`generate-cases`) can consume. + +**Core Workflow:** +- Read the PR's human-written context (title, description, comments, images, linked tickets). +- Resolve linked issues via Jira MCP / GitHub MCP / `gh` CLI — read original ticket description, acceptance criteria, comments. +- Verify whether the PR scope matches the original ticket scope. +- Surface ambiguities (unclear requirements, missing edge cases, open questions in comments). +- Save a structured requirements summary `.md` to **available cahce (if exist - `.testclaw/`)** or root folder. + +**Full PR knowledge by available skills** + +It is the **complement** of `pr-diff`: + +| Skill | Input | Output | Focus | +|-------|-------|--------|-------| +| `pr-diff` | git diff, changed files | AC from code changes | "what the code does" | +| `pr-requirements-check` | PR title, description, comments, linked Jira/GitHub issue | requirements `.md` with scope verification, ambiguities, edge cases, AC | "what the PR is supposed to do" | + +> Use them together for a full picture. Use `pr-requirements-check` alone when the question is "is this PR doing the right thing?" rather than "what does this PR do?". + +--- + +## When to Use + +Trigger this skill when the user wants to: +- **Understand the original intent** of a PR from its description and linked ticket, not just the code. +- **Verify scope** — does the code in this PR actually address what the linked Jira/GitHub issue asked for? +- **Find ambiguities or missing requirements** in the PR description before generating test cases. +- **Generate acceptance criteria from ticket context** (especially when the PR description is thin but the ticket is detailed). +- **Review a PR before review/merge** from a requirements/completeness angle, not a code-quality angle. + +Do **not** trigger this skill when the user only wants to know what code changed — that is `pr-diff`. Trigger it when the user's words include things like "requirements", "ticket", "Jira", "issue", "scope", "what is this PR supposed to do", "does this match the ticket", "missing acceptance criteria", "ambiguities in this PR". + +--- + +## Workflow: PR Requirements Check + +### Step 1: Detect PR Context + +Identify which PR to analyze and where it lives. + +#### Determine PR Identifier + +Accept any of these inputs (in priority order): +1. A GitHub PR number / URL passed in by the user (e.g. `#1234`, `https://github.com/org/repo/pull/1234`). + +2. The PR for the current branch — auto-detect: + +```bash +gh pr view --json number,title,baseRefName,headRefName,url +``` +3. A user-provided ticket key (Jira `PRJ-123`, GitHub Issue `#123`) without a PR — in that case the skill still runs but pivots to "analyze ticket, then check if any open PR references it". + +If no PR is detectable, ask the user: + +``` +❓ No pull request detected. Provide one of: +1. PR number or URL (e.g. https://github.com/org/repo/pull/1234) +2. A branch name (I will find its open PR) +3. A ticket key only (e.g. PRJ-123) — I will analyze the ticket and any linked PR +``` + +#### Detect Working Directory Convention + +Before pulling or writing anything, check if the project uses the `.testclaw/` convention (see [References](#references)). +--- + +### Step 2: Read PR Human-Written Context + +Collect everything the developer wrote about the PR — **not** the code itself. + +#### 2.1 Title, Description, Comments + +Fetch via GitHub MCP if available, else `gh` CLI: +```bash +gh pr view {PR_NUMBER} --json title,body,comments,labels,milestone,assignees +gh pr view {PR_NUMBER} --comments +``` + +Extract from the response: +- **Title** — short summary, the developer's own framing. +- **Body** — the PR description. Look for: summary, motivation, screenshots, "How to test", "Acceptance Criteria", "Fixes #N" / "Closes PRJ-123", checkbox lists, embedded images (`![](...)`). +- **Comments** — review feedback, scope changes, design decisions, edge cases the reviewer raised. +- **Labels, milestone, assignees** — signal priority and ownership. + +#### 2.2 Attached Images and Diagrams + +If the PR body contains image URLs (Markdown `![](...)` or HTML ``), read them — they often show UI changes, expected error states, or flow diagrams that text does not capture. Prefer a vision-capable agent or `fetch` + describe. + +#### 2.3 Linked Tickets + +Parse the PR body and branch name for ticket references. Common patterns: + +| Pattern | Source | Example | +|---------|--------|---------| +| `Fixes #N`, `Closes #N`, `Resolves #N` | GitHub Issues | `Closes` | +| `JIRA: PRJ-123`, `[PRJ-123]`, `PRJ-123 in body` | Jira | `Implements PRJ-4567` | +| Branch name with ticket key | Jira | `feature/PRJ-789-user-export` | +| Linear / Asana / ClickUp patterns | various | `LIN-123`, `ASA-456` | + +Collect **all** detected ticket references. Do not stop at the first one. + +--- + +### Step 3: Resolve Linked Tickets + +Fetch the **full** original requirements from each linked ticket. This is the most important step — the PR description is often a thin summary; the ticket has the real requirements. + +#### 3.1 If a Jira MCP is Available + +Use it to fetch: +- Issue summary and description (often contains the original AC, mockups, links). +- Acceptance Criteria section (if the team uses a convention like "AC:" or a child checklist). +- Comments — product/QA may have added scope, edge cases, or "out of scope" notes. +- Linked issues — epics, related bugs, blockers. +- Attachments — additional specs or designs. + +Example of possible MCP calls (adapt to the Jira MCP actually configured): +- `get_issue({ issue_key: "PRJ-xxx", include_comments: true })` + +#### 3.2 If a GitHub MCP is Available + +Use available Mcp's tools to fetch the linked GitHub issue: +- `get_issue({ owner, repo, issue_number: xxx })` + +#### 3.3 Fallback — `gh` CLI + +```bash +gh issue view {NUMBER} --json title,body,comments,labels +gh issue view {NUMBER} --comments +``` + +For Jira without an MCP, ask the user to paste the issue content — do not silently skip it. + +#### 3.4 No MCP and No Access + +If no MCP is available and `gh` cannot reach the ticket source, note it explicitly in the output and continue with what you have. +**Do not fabricate ticket content or add incorrect information**. + +--- + +### Step 4: Verify Scope and Detect Ambiguities + +Now compare **what the PR description + code says it does** against **what the ticket says it should do**. + +#### 4.1 Scope Verification + +Build three lists: + +- **In scope** - requirements from the ticket that appear to be addressed (by PR description OR by code, see Step 5). +- **Out of scope / missing** - requirements from the ticket that have **no** corresponding change in the PR. +- **Extra / out of ticket** - changes in the PR that are **not** mentioned in the ticket. Flag these for review — they may be intentional or accidental. + +#### 4.2 Ambiguity Detection + +Surface places where the requirements are unclear. Categories: + +- **Underspecified behavior** - the ticket says "support export" but does not say which formats, what the file naming convention is, what happens with empty data, etc. +- **Unanswered comments** - questions in the PR comments or ticket comments that have no reply. +- **Conflicting requirements** - different parts of the ticket (or ticket vs. PR description) say different things. +- **Missing edge cases** - common edge cases (empty input, error path, permissions, concurrency, large data) that neither ticket nor PR address. + +#### 4.3 Edge Case Surfacing + +For each affected area, list likely edge cases the developer may have missed. These are **questions to raise**, not assertions that the code is wrong: + +- Empty / null / boundary inputs. +- Permission / role checks (does this work for admins vs. regular users?). +- Error paths and how they are surfaced to the user. +- Performance with realistic data volumes. +- Backward compatibility — does existing data still work? +- Localization / timezone / currency where relevant. + +--- + +### Step 5: Identify Affected Files (Briefly) + +This skill is **not** `pr-diff` — do not do a deep code review. But the summary needs an accurate "Affected Files" list, so: +- Prefer files **mentioned in the PR description or ticket** (the developer usually lists them). +- Cross-check against `git diff {base}...HEAD --name-only` from `pr-diff` only if the description does not enumerate them. Call out: `> File list cross-checked against git diff (pr-diff skill)`. +- Categorize each file: source code, config, test, docs, migration, dependency. This helps the reader of the summary see at a glance whether config-only or docs-only changes snuck in. + +#### When to Short-Circuit (no full analysis) + +If **every** changed file is non-source-code (`*.md`, `docs/**`, `tests/**`, configs, CI, dependency bumps, lock files), the PR does **not** describe new behavior. In that case: +- Do **not** invent acceptance criteria. +- Do **not** invoke `pr-diff` for AC extraction. +- Still produce a short summary `.md` with: + - PR title and type (`docs` | `chore` | `ci` | `deps`). + - One-sentence `Changes` describing what was updated. + - The full file list. + - An explicit note: `> No source code behavior changed. Acceptance criteria generation is not applicable for this PR.` + +**Stop after this — there is nothing else to verify or test.** + +#### When the PR Description is Empty or Missing + +If the PR has no body and the branch name does not encode intent: +- Pull the **commit messages** of the branch: + +```bash +git log {base}..HEAD --pretty=format:"%s%n%b" +``` + +- Combine commit messages + changed files + branch name into a **tentative** one-sentence `Changes` line. +- Mark it explicitly: `> PR description was empty. Summary derived from branch name, commits, and changed files only.` + +> If a linked ticket exists, prefer the ticket's title/description as the source of truth and note the empty PR body as a finding under "Ambiguities". + +--- + +### Step 6: Generate the Requirements Summary `.md` + +Compose the final output and save it to disk. + +#### Filename + +Slugify the PR title (lowercase, dashes, no special characters, max 40 chars). +Example: `Add user export to CSV` => `add-user-export-to-csv-pr-1234.md`. Include the PR number to avoid collisions. + +#### Output Template + +```md +## PR Requirements Summary + +**PR:** {title} +**Branch:** {headRefName} → {baseRefName} +**Type:** ... (feature | bugfix | refactor | deps) +**Linked Tickets:** {list of PRJ-123 or "none detected"} + +**Source of Truth:** +- PR description: {present | empty — derived from commits/branch} +- Jira/GitHub ticket: {resolved | not resolved — no MCP available} +- Most reliable source: {PR description | ticket | commits} + +**Changes:** ... (one sentence describing the overall purpose of this PR) + +**Affected Files:** +- `path/to/file.ts` — source +- `path/to/config.yml` — config +- `docs/readme.md` — docs +- ... (categorized: source | config | test | docs | migration | deps) + +**Impacted Areas:** +- item-1 ... (1-3 bullet points, only real impact areas in pr) +- ... + +**Scope Verification:** +- ✅ In scope: {ticket requirement → where in PR it is addressed} +- ⚠️ Out of scope / missing: {ticket requirement not addressed in PR} +- ➕ Extra (not in ticket): {change present in PR but not in ticket} + +**Ambiguities, Edge Cases & Open Questions:** +- {underspecified behavior / unanswered comment / conflicting requirement / missing edge case} +- {empty input / permissions / error path / performance / backward compatibility / ...} +- ... + +**Acceptance Criteria:** +- action to perform → expected result +- ... +``` + +**Summary Rules:** +- **Concise and high-level.** This summary is a requirements document, not a code review. +- **`Changes` is ONE sentence.** Not a paragraph. Capture intent, not implementation. +- **`Impacted Areas` only meaningful business/technical impact.** 1–3 bullets. Omit the section if there is nothing meaningful. +- **`Source of Truth`** is required — it tells the reader (and downstream skills) what to trust when sources disagree. +- **Scope Verification** — be honest. `⚠️ Out of scope` items are the most valuable output of this skill. +- **Acceptance Criteria** — testable and user-oriented. Written as "action → expected result" so they map directly to test steps. If no source-code behavior changed, replace this entire section with the explicit "not applicable" note described in Step 5. +- **Omit empty sections** instead of writing `None` / `N/A` — except the special case of "no source code behavior changed" which must be explicit. +- **Avoid generic statements** like "code cleanup", "minor fixes", "various improvements". Either name what was cleaned up or do not mention it. + +--- + +### Step 7: Save and Report + +1. Print the "overview summary" version in the chat so the user can read it immediately. +3. Tell the user the saved file path and explicitly mention: + - Whether scope matched the linked ticket (`✅ all in scope` / `⚠️ N items missing` / `➕ N extra changes`). + - Whether ambiguities were found and how many. + - Whether downstream skills (`generate-cases`) can consume this file as input. + +--- + +## Error Handling + +### Recovery + +Attempt recovery before failing: +- **No `gh` CLI** — check for a GitHub MCP; if neither, ask the user to paste the PR title/description manually. +- **No Jira MCP and no `gh` access to the issue** — note it, continue with PR description, ask user to paste the ticket content if they want full scope verification. +- **PR not found** — confirm the number, repo, and that the user has access. +- **Branch not pushed** — fetch the latest and retry. + +### Hard Fail + +Stop if: +- The user repeatedly refuses to provide any PR identifier or ticket content. +- A required MCP returns repeated auth/permission errors. +- The skill cannot write to corresponding folder and the user will not allow a different path. + +--- + +## User Interaction Examples + +### Example 1: Well-Documented Feature PR with Linked Jira Ticket + +**User:** `Check the requirements for PR ...` + +**Output (abbreviated):** + +```md +## PR Requirements Summary + +**PR:** Add user export to CSV +**Branch:** feature/PRJ-789-user-export → main +**Type:** feature +**Linked Tickets:** PRJ-111 + +**Source of Truth:** +- PR description: present (detailed, includes AC checklist) +- Jira ticket: resolved (PRJ-111) +- Most reliable source: PR description + ticket (consistent) + +**Changes:** Adds a CSV export action on the user list page that streams all matching users to a downloadable file. + +**Affected Files:** +- `src/pages/UserList.tsx` — source +- `src/services/ExportService.ts` — source +- `src/api/export.ts` — source +- `i18n/en.json` — config +- `tests/e2e/export.test.ts` — test + +**Impacted Areas:** +- User list page UI. +- User data export pipeline. +- Permission checks for export (admin only). + +**Scope Verification:** +- ✅ In scope: PRJ-111 AC1 (export button on list) — addressed in UserList.tsx +- ✅ In scope: PRJ-111 AC2 (admin-only) — addressed in ExportService.ts permission check +- ⚠️ Out of scope / missing: PRJ-111 AC3 (email notification when export is ready for >10k users) — no implementation found +- ➕ Extra (not in ticket): i18n string added for "Export ready" toast + +**Ambiguities, Edge Cases & Open Questions:** +- Ticket does not specify CSV column order — needs product confirmation. +- PR comment from @reviewer: "What happens for users with no email?" — unanswered. +- Empty user list (zero rows). +- User without an email address. +- Export of 50k+ users (memory/timeout). + +**Acceptance Criteria:** +- Admin clicks "Export" on user list with filters applied → CSV file downloads with matching users +- Non-admin clicks "Export" → action is hidden / returns 403 +- Export of >10k users → email notification is sent when file is ready +- Export contains columns: id, name, email, created_at → in that order + +> Saved to: `.testclaw/pr-summaries/add-user-export-to-csv-pr-1234.md` + +(If `.testclaw/pr-summaries/` - yes, follow it. If not, create `.testclaw/pr-summaries/` at the repo root and add `.testclaw/` to `.gitignore` if not already there) +``` + +### Example 2: Docs-Only PR (No Source Code Changes) + +**User:** `Check the requirements for PR ...` + +**Output (abbreviated):** + +```md +## PR Requirements Summary + +**PR:** ... — Update README with new install steps +**Branch:** docs/update-install → main +**Type:** docs +**Linked Tickets:** none detected + +**Source of Truth:** +- PR description: present (one-line) +- Jira ticket: not resolved +- Most reliable source: PR description + +**Changes:** Updates README install instructions to reflect the new Node 20 requirement and adds a Docker quick-start section. + +**Affected Files:** +- `README.md` — docs +- `docs/install.md` — docs + +**Impacted Areas:** (omitted — no business/technical impact beyond documentation) + +> No source code behavior changed. Acceptance criteria generation is not applicable for this PR. + +> Saved to project root: `update-readme-with-new-install-steps-pr-1240.md` +``` + +--- + +## References + +| Description | File | +|-------------|------| +| Filled example of the output summary | [summary-example.md](./references/summary-example.md) | +| `.testclaw/` directory convention | inherited from `project-scan` (see project-scan SKILL.md) | +| Code-focused PR analysis (companion skill) | `../pr-diff/SKILL.md` | +| Downstream: generate test cases from requirements | `../generate-cases/SKILL.md` | +| Downstream: sync generated cases to Testomat.io | `../sync-cases/SKILL.md` | + +--- + +## Quick Commands + +| Action | Command | +|--------|---------| +| Detect PR for current branch | `gh pr view --json number,title,baseRefName,headRefName,url` | +| Read PR description + comments | `gh pr view {N} --json title,body,comments,labels` | +| Read PR diff (delegate to pr-diff) | `gh pr diff {N}` | +| Read linked GitHub issue | `gh issue view {N} --json title,body,comments` | +| Read branch commits | `git log {base}..HEAD --pretty=format:"%s%n%b"` | +| Files changed | `git diff {base}...HEAD --name-only` | diff --git a/skills/pr-requirements-check/references/summary-example.md b/skills/pr-requirements-check/references/summary-example.md new file mode 100644 index 0000000..7f8cb9c --- /dev/null +++ b/skills/pr-requirements-check/references/summary-example.md @@ -0,0 +1,56 @@ +# PR Requirements Summary — Example + +This file is a **filled-in example** of the artifact produced by the `pr-requirements-check` skill and saved to corresponding file. + +--- + +## PR Requirements Summary + +**PR:** Add user export to CSV +**Branch:** feature/PRJ-789-user-export → main +**Type:** feature +**Linked Tickets:** PRJ-789 + +**Source of Truth:** +- PR description: present (detailed, includes AC checklist and screenshot of the export modal) +- Jira/GitHub ticket: resolved (PRJ-789) +- Most reliable source: PR description + ticket (consistent, no conflicts) + +**Changes:** Adds a CSV export action on the user list page that streams all currently filtered users to a downloadable file, with an email notification when exports exceed 10k rows. + +**Affected Files:** +- `src/pages/UserList.tsx` — source +- `src/services/ExportService.ts` — source +- `src/api/export.ts` — source +- `src/components/ExportModal.tsx` — source +- `i18n/en.json` — config +- `i18n/de.json` — config +- `tests/e2e/export.test.ts` — test +- `docs/features/export.md` — docs + +**Impacted Areas:** +- User list page UI (new "Export" action). +- User data export pipeline (CSV generation, streaming, email delivery). +- Permission checks for export (admin-only). + +**Scope Verification:** +- ✅ In scope: PRJ-789 AC1 (export button on the user list) — addressed in `src/pages/UserList.tsx` and `src/components/ExportModal.tsx` +- ➕ Extra (not in ticket): i18n strings for German locale (`i18n/de.json`) — ticket only mentioned English + +**Ambiguities, Edge Cases & Open Questions:** +- Ticket does not specify CSV column order — needs product confirmation. +- Concurrent exports by the same admin — no rate limiting or deduplication specified. +- Export of 50k+ users (memory/timeout) — no performance budget in the ticket. +- Localization of date/time fields in the CSV — not specified (UTC vs. user locale). +- Backward compatibility with existing scheduled exports (if any) — not addressed. + +**Acceptance Criteria:** +- Admin opens user list with filters applied and clicks "Export" → modal appears with row count estimate +- Admin clicks "Start export" with fewer than 10k matching users → CSV file downloads immediately with columns `id, name, email, created_at` in that order +- Non-admin opens user list → "Export" action is hidden (or returns 403 if accessed via API) +- Admin triggers export of >10k matching users → export is queued and an email notification is sent when the file is ready *(NOT IMPLEMENTED IN THIS PR — see Scope Verification)* +- User without an email address is included in the CSV → email field is empty string +- Export of an empty filtered list → button is disabled OR an empty CSV is downloaded with only the header row +- Exporting 50k+ users completes without server timeout or out-of-memory error +- CSV dates are formatted in the user's locale timezone (or UTC — confirm with product) +- German admin sees the export modal translated into German From 0ec24318f214590d762b55fef4330ee89b1b0aa4 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Wed, 3 Jun 2026 23:09:38 +0300 Subject: [PATCH 2/5] [PR REQUIREMENTS] Removed specific tech instruction to be more clear and reusable --- skills/pr-requirements-check/SKILL.md | 339 +++++------------- .../references/summary-example.md | 27 +- 2 files changed, 110 insertions(+), 256 deletions(-) diff --git a/skills/pr-requirements-check/SKILL.md b/skills/pr-requirements-check/SKILL.md index 6f62745..b7c6706 100644 --- a/skills/pr-requirements-check/SKILL.md +++ b/skills/pr-requirements-check/SKILL.md @@ -1,10 +1,10 @@ --- name: pr-requirements-check -description: Analyze a pull request's context — title, description, comments, linked Jira/GitHub issues, attached images — to understand the original intent, verify scope against the linked ticket, surface ambiguities and edge cases, and produce a structured requirements summary with testable acceptance criteria saved as a markdown file. Use this skill when the user wants to understand WHAT a PR is supposed to do (not just what code changed), check whether the PR matches its linked ticket/task, find gaps or missing requirements, or generate acceptance criteria from PR/ticket context rather than from code. +description: Analyze a pull request's context - title, description, comments, linked issues, attached schemas/images — to understand the original intent, verify scope against the linked ticket, surface ambiguities and edge cases, and produce a structured requirements summary with testable acceptance criteria. Use this skill when the user wants to understand WHAT a PR is supposed to do (not just what code changed), check whether the PR matches its linked ticket/task or generate acceptance criteria from PR/ticket context rather than from code. license: MIT metadata: author: Testomat.io - version: 1.0.0 + version: 1.1.0 --- # PR-REQUIREMENTS-CHECK SKILL: What I Do @@ -13,10 +13,10 @@ This skill reviews a pull request's **context** - the human-written parts: title **Core Workflow:** - Read the PR's human-written context (title, description, comments, images, linked tickets). -- Resolve linked issues via Jira MCP / GitHub MCP / `gh` CLI — read original ticket description, acceptance criteria, comments. +- Resolve linked issues via enabled MCP's or CLI (like `gh`): read original ticket description, acceptance criteria, comments, attachemnts. - Verify whether the PR scope matches the original ticket scope. -- Surface ambiguities (unclear requirements, missing edge cases, open questions in comments). -- Save a structured requirements summary `.md` to **available cahce (if exist - `.testclaw/`)** or root folder. +- Surface ambiguities (missing edge cases, open questions in comments). +- Save a structured requirements summary as a .md file in the user-specified folder **only if the original prompt explicitly requests** this action. **Full PR knowledge by available skills** @@ -25,7 +25,7 @@ It is the **complement** of `pr-diff`: | Skill | Input | Output | Focus | |-------|-------|--------|-------| | `pr-diff` | git diff, changed files | AC from code changes | "what the code does" | -| `pr-requirements-check` | PR title, description, comments, linked Jira/GitHub issue | requirements `.md` with scope verification, ambiguities, edge cases, AC | "what the PR is supposed to do" | +| `pr-requirements-check` | PR title, description, comments, linked issue | summary or requirements with scope verification, AC | "what the PR is supposed to do" | > Use them together for a full picture. Use `pr-requirements-check` alone when the question is "is this PR doing the right thing?" rather than "what does this PR do?". @@ -35,12 +35,11 @@ It is the **complement** of `pr-diff`: Trigger this skill when the user wants to: - **Understand the original intent** of a PR from its description and linked ticket, not just the code. -- **Verify scope** — does the code in this PR actually address what the linked Jira/GitHub issue asked for? -- **Find ambiguities or missing requirements** in the PR description before generating test cases. +- **Verify scope** — does the code in this PR actually address what the linked Jira/GitHub/etc issue asked for? +- **Find ambiguities or missing cases** in the PR description before generating test cases. - **Generate acceptance criteria from ticket context** (especially when the PR description is thin but the ticket is detailed). -- **Review a PR before review/merge** from a requirements/completeness angle, not a code-quality angle. -Do **not** trigger this skill when the user only wants to know what code changed — that is `pr-diff`. Trigger it when the user's words include things like "requirements", "ticket", "Jira", "issue", "scope", "what is this PR supposed to do", "does this match the ticket", "missing acceptance criteria", "ambiguities in this PR". +Do **not** trigger this skill when the user only wants to know what code changed - that is `pr-diff` skill. Trigger it when the user's words include things like "requirements", "ticket", "Jira", "issue", "scope", "what is this PR supposed to do", "does this match the ticket", "missing acceptance criteria", "ambiguities in this PR". --- @@ -48,203 +47,96 @@ Do **not** trigger this skill when the user only wants to know what code changed ### Step 1: Detect PR Context -Identify which PR to analyze and where it lives. +Identify the PR to analyze and its location. #### Determine PR Identifier -Accept any of these inputs (in priority order): -1. A GitHub PR number / URL passed in by the user (e.g. `#1234`, `https://github.com/org/repo/pull/1234`). +Accept, in priority order: +1. A PR number or URL from the user prompt/request (e.g. `https://github.com/org/repo/pull/1234`). +2. The open PR for the current branch (e.g. `gh pr view --json number,title,baseRefName,headRefName,url`) +3. A ticket key only (Jira `PRJ-123`, Linear, etc) - pivot to "analyze the ticket, then check for any open PR referencing it". -2. The PR for the current branch — auto-detect: +> If none are detectable, ask the user for a PR number/URL, branch name, or ticket key to continue work. -```bash -gh pr view --json number,title,baseRefName,headRefName,url -``` -3. A user-provided ticket key (Jira `PRJ-123`, GitHub Issue `#123`) without a PR — in that case the skill still runs but pivots to "analyze ticket, then check if any open PR references it". - -If no PR is detectable, ask the user: - -``` -❓ No pull request detected. Provide one of: -1. PR number or URL (e.g. https://github.com/org/repo/pull/1234) -2. A branch name (I will find its open PR) -3. A ticket key only (e.g. PRJ-123) — I will analyze the ticket and any linked PR -``` - -#### Detect Working Directory Convention - -Before pulling or writing anything, check if the project uses the `.testclaw/` convention (see [References](#references)). --- -### Step 2: Read PR Human-Written Context - -Collect everything the developer wrote about the PR — **not** the code itself. - -#### 2.1 Title, Description, Comments - -Fetch via GitHub MCP if available, else `gh` CLI: -```bash -gh pr view {PR_NUMBER} --json title,body,comments,labels,milestone,assignees -gh pr view {PR_NUMBER} --comments -``` - -Extract from the response: -- **Title** — short summary, the developer's own framing. -- **Body** — the PR description. Look for: summary, motivation, screenshots, "How to test", "Acceptance Criteria", "Fixes #N" / "Closes PRJ-123", checkbox lists, embedded images (`![](...)`). -- **Comments** — review feedback, scope changes, design decisions, edge cases the reviewer raised. -- **Labels, milestone, assignees** — signal priority and ownership. - -#### 2.2 Attached Images and Diagrams - -If the PR body contains image URLs (Markdown `![](...)` or HTML ``), read them — they often show UI changes, expected error states, or flow diagrams that text does not capture. Prefer a vision-capable agent or `fetch` + describe. - -#### 2.3 Linked Tickets +### Step 2: Read PR Context -Parse the PR body and branch name for ticket references. Common patterns: +Collect everything the developer/team members wrote — not the code itself. Detect available MCP/CLI tools and use them to pull: +- **PR metadata** (title, description, comments, labels, milestone, assignees) - via Jira/Linear/GitHub MCP if enabled, else `gh` CLI tools to extract PR "title", "body", "comments", "labels", "attachments", etc. +- **Embedded images/diagrams** in the body — read them (vision-capable agent or `fetch` + describe) when present; they often capture UI changes, error states, or flows that text misses. +- **Linked tickets** — parse the body and branch name for ticket references (table below). -| Pattern | Source | Example | -|---------|--------|---------| -| `Fixes #N`, `Closes #N`, `Resolves #N` | GitHub Issues | `Closes` | -| `JIRA: PRJ-123`, `[PRJ-123]`, `PRJ-123 in body` | Jira | `Implements PRJ-4567` | -| Branch name with ticket key | Jira | `feature/PRJ-789-user-export` | -| Linear / Asana / ClickUp patterns | various | `LIN-123`, `ASA-456` | +> Identify and collect all references available to you from the provided context and accessible sources. -Collect **all** detected ticket references. Do not stop at the first one. +**Do not fabricate ticket content or add incorrect information**. --- ### Step 3: Resolve Linked Tickets -Fetch the **full** original requirements from each linked ticket. This is the most important step — the PR description is often a thin summary; the ticket has the real requirements. - -#### 3.1 If a Jira MCP is Available - -Use it to fetch: -- Issue summary and description (often contains the original AC, mockups, links). -- Acceptance Criteria section (if the team uses a convention like "AC:" or a child checklist). -- Comments — product/QA may have added scope, edge cases, or "out of scope" notes. -- Linked issues — epics, related bugs, blockers. -- Attachments — additional specs or designs. - -Example of possible MCP calls (adapt to the Jira MCP actually configured): -- `get_issue({ issue_key: "PRJ-xxx", include_comments: true })` - -#### 3.2 If a GitHub MCP is Available - -Use available Mcp's tools to fetch the linked GitHub issue: -- `get_issue({ owner, repo, issue_number: xxx })` - -#### 3.3 Fallback — `gh` CLI +Fetch the full original requirements from each linked ticket — the PR description is often a thin summary; the ticket has the real requirements. -```bash -gh issue view {NUMBER} --json title,body,comments,labels -gh issue view {NUMBER} --comments -``` - -For Jira without an MCP, ask the user to paste the issue content — do not silently skip it. +Use the first available tool: -#### 3.4 No MCP and No Access - -If no MCP is available and `gh` cannot reach the ticket source, note it explicitly in the output and continue with what you have. -**Do not fabricate ticket content or add incorrect information**. +1. **Jira/Linear MCP** if enabled — fetch issue summary, description, AC, comments, linked issues, attachments. +2. **GitHub MCP** if enabled — fetch the linked issue. +3. **`gh` CLI** fallback — `gh issue view {N} --json title,body,comments,labels`. For Jira/Linear without an MCP, ask the user to paste the issue content. +4. **No access** — note the gap explicitly in the output and continue with what you have. --- ### Step 4: Verify Scope and Detect Ambiguities -Now compare **what the PR description + code says it does** against **what the ticket says it should do**. - -#### 4.1 Scope Verification - -Build three lists: - -- **In scope** - requirements from the ticket that appear to be addressed (by PR description OR by code, see Step 5). -- **Out of scope / missing** - requirements from the ticket that have **no** corresponding change in the PR. -- **Extra / out of ticket** - changes in the PR that are **not** mentioned in the ticket. Flag these for review — they may be intentional or accidental. +Compare what the PR does against what the ticket asked for, and surface anything unclear. These are **questions to raise**, not assertions that the code is wrong. -#### 4.2 Ambiguity Detection +**Scope Verification** — build three lists: +- **In scope** — ticket requirements addressed by the PR (description or code, see Step 5). +- **Out of scope / missing** — ticket requirements with no corresponding PR change. These are the most valuable output of this skill. +- **Extra / out of ticket** — PR changes not mentioned in the ticket. Flag for review (intentional or accidental?). -Surface places where the requirements are unclear. Categories: - -- **Underspecified behavior** - the ticket says "support export" but does not say which formats, what the file naming convention is, what happens with empty data, etc. -- **Unanswered comments** - questions in the PR comments or ticket comments that have no reply. -- **Conflicting requirements** - different parts of the ticket (or ticket vs. PR description) say different things. -- **Missing edge cases** - common edge cases (empty input, error path, permissions, concurrency, large data) that neither ticket nor PR address. - -#### 4.3 Edge Case Surfacing - -For each affected area, list likely edge cases the developer may have missed. These are **questions to raise**, not assertions that the code is wrong: - -- Empty / null / boundary inputs. -- Permission / role checks (does this work for admins vs. regular users?). -- Error paths and how they are surfaced to the user. -- Performance with realistic data volumes. -- Backward compatibility — does existing data still work? -- Localization / timezone / currency where relevant. +**Ambiguities & Edge Cases:** +- **Underspecified behavior** — ticket says "support export" but not format, naming, empty-data handling, etc. +- **Unanswered comments** — questions in PR or ticket comments with no reply. +- **Conflicting requirements** — different parts of the ticket (or ticket vs. PR description) disagree. +- **Missing edge cases** — empty/null inputs, permissions/roles, backward compatibility, localization/timezone/currency. --- -### Step 5: Identify Affected Files (Briefly) +### Step 5: Identify Affected Files & Edge Cases -This skill is **not** `pr-diff` — do not do a deep code review. But the summary needs an accurate "Affected Files" list, so: -- Prefer files **mentioned in the PR description or ticket** (the developer usually lists them). -- Cross-check against `git diff {base}...HEAD --name-only` from `pr-diff` only if the description does not enumerate them. Call out: `> File list cross-checked against git diff (pr-diff skill)`. -- Categorize each file: source code, config, test, docs, migration, dependency. This helps the reader of the summary see at a glance whether config-only or docs-only changes snuck in. +This skill is not `pr-diff` — do not do a deep code review. Prefer files **mentioned in the PR description or ticket**; cross-check with `git diff {base}...HEAD --name-only` only if the description doesn't enumerate them. -#### When to Short-Circuit (no full analysis) +**Short-circuit (no source behavior):** if every changed file is non-source-code (`*.md`, `docs/**`, `tests/**`, configs, CI, deps, lock files), do not invent AC and do not invoke `pr-diff`. Produce a short summary with: PR title, type (`docs`/`chore`/`ci`/`deps`), one-sentence `Changes`, full file list. +- Post explicit note `> No source code behavior changed. Acceptance criteria generation is not applicable for this PR.` Stop after this. -If **every** changed file is non-source-code (`*.md`, `docs/**`, `tests/**`, configs, CI, dependency bumps, lock files), the PR does **not** describe new behavior. In that case: -- Do **not** invent acceptance criteria. -- Do **not** invoke `pr-diff` for AC extraction. -- Still produce a short summary `.md` with: - - PR title and type (`docs` | `chore` | `ci` | `deps`). - - One-sentence `Changes` describing what was updated. - - The full file list. - - An explicit note: `> No source code behavior changed. Acceptance criteria generation is not applicable for this PR.` - -**Stop after this — there is nothing else to verify or test.** - -#### When the PR Description is Empty or Missing - -If the PR has no body and the branch name does not encode intent: -- Pull the **commit messages** of the branch: - -```bash -git log {base}..HEAD --pretty=format:"%s%n%b" -``` - -- Combine commit messages + changed files + branch name into a **tentative** one-sentence `Changes` line. -- Mark it explicitly: `> PR description was empty. Summary derived from branch name, commits, and changed files only.` - -> If a linked ticket exists, prefer the ticket's title/description as the source of truth and note the empty PR body as a finding under "Ambiguities". +**Empty PR description:** if there's no body and the branch name doesn't encode intent, pull `git log {base}..HEAD --pretty=format:"%s%n%b"`, combine with changed files and branch name into a tentative one-sentence `Changes`, and mark `> PR description was empty. Summary derived from branch name, commits, and changed files only.` +- If a linked ticket exists, prefer the ticket as source of truth and note the empty body under "Ambiguities". --- ### Step 6: Generate the Requirements Summary `.md` -Compose the final output and save it to disk. - -#### Filename +Compose the final output and save it to disk only when the original prompt includes an instruction to save it. -Slugify the PR title (lowercase, dashes, no special characters, max 40 chars). -Example: `Add user export to CSV` => `add-user-export-to-csv-pr-1234.md`. Include the PR number to avoid collisions. +**Filename:** slugify the PR title (lowercase, dashes, no special chars, max 40 chars) and append the PR number. Example: `Add user export to CSV` -> `add-user-export-to-csv-pr-1234.md`. -#### Output Template +**Output Template:** ```md ## PR Requirements Summary **PR:** {title} **Branch:** {headRefName} → {baseRefName} -**Type:** ... (feature | bugfix | refactor | deps) -**Linked Tickets:** {list of PRJ-123 or "none detected"} +**Type:** {feature | bugfix | refactor | deps} +**Linked Tickets:** {list or "none detected"} **Source of Truth:** - PR description: {present | empty — derived from commits/branch} - Jira/GitHub ticket: {resolved | not resolved — no MCP available} - Most reliable source: {PR description | ticket | commits} -**Changes:** ... (one sentence describing the overall purpose of this PR) +**Changes:** ... (one sentence — intent, not implementation) **Affected Files:** - `path/to/file.ts` — source @@ -253,8 +145,7 @@ Example: `Add user export to CSV` => `add-user-export-to-csv-pr-1234.md`. Includ - ... (categorized: source | config | test | docs | migration | deps) **Impacted Areas:** -- item-1 ... (1-3 bullet points, only real impact areas in pr) -- ... +- ... (1–3 bullets, only meaningful business/technical impact; omit if empty) **Scope Verification:** - ✅ In scope: {ticket requirement → where in PR it is addressed} @@ -262,34 +153,24 @@ Example: `Add user export to CSV` => `add-user-export-to-csv-pr-1234.md`. Includ - ➕ Extra (not in ticket): {change present in PR but not in ticket} **Ambiguities, Edge Cases & Open Questions:** -- {underspecified behavior / unanswered comment / conflicting requirement / missing edge case} -- {empty input / permissions / error path / performance / backward compatibility / ...} +- ... (underspecified behavior / unanswered comment / conflicting requirement / missing edge case) +- ... (empty input / permissions / error path / performance / backward compatibility / ...) - ... **Acceptance Criteria:** -- action to perform → expected result +- {action to perform} → {expected result} - ... ``` **Summary Rules:** -- **Concise and high-level.** This summary is a requirements document, not a code review. -- **`Changes` is ONE sentence.** Not a paragraph. Capture intent, not implementation. -- **`Impacted Areas` only meaningful business/technical impact.** 1–3 bullets. Omit the section if there is nothing meaningful. -- **`Source of Truth`** is required — it tells the reader (and downstream skills) what to trust when sources disagree. -- **Scope Verification** — be honest. `⚠️ Out of scope` items are the most valuable output of this skill. -- **Acceptance Criteria** — testable and user-oriented. Written as "action → expected result" so they map directly to test steps. If no source-code behavior changed, replace this entire section with the explicit "not applicable" note described in Step 5. -- **Omit empty sections** instead of writing `None` / `N/A` — except the special case of "no source code behavior changed" which must be explicit. -- **Avoid generic statements** like "code cleanup", "minor fixes", "various improvements". Either name what was cleaned up or do not mention it. - ---- - -### Step 7: Save and Report - -1. Print the "overview summary" version in the chat so the user can read it immediately. -3. Tell the user the saved file path and explicitly mention: - - Whether scope matched the linked ticket (`✅ all in scope` / `⚠️ N items missing` / `➕ N extra changes`). - - Whether ambiguities were found and how many. - - Whether downstream skills (`generate-cases`) can consume this file as input. +- Concise, high-level, requirements-not-implementation — this is a requirements document, not a code review. +- `Changes` is **one sentence**. +- `Impacted Areas` — 1–3 meaningful bullets; omit the section if empty. +- `Source of Truth` is required — tells downstream skills what to trust when sources disagree. +- Scope Verification — be honest; `⚠️ Out of scope` items are the most valuable output of this skill. +- Acceptance Criteria — testable, user-oriented, written as "action → expected result". If no source-code behavior changed, replace this section with the explicit "not applicable" note from Step 5. +- Omit empty sections instead of writing `None`/`N/A` — except the "no source code changed" note which must be explicit. +- Avoid generic statements like "code cleanup", "minor fixes", "various improvements" — name what changed or omit it. --- @@ -297,28 +178,24 @@ Example: `Add user export to CSV` => `add-user-export-to-csv-pr-1234.md`. Includ ### Recovery -Attempt recovery before failing: -- **No `gh` CLI** — check for a GitHub MCP; if neither, ask the user to paste the PR title/description manually. -- **No Jira MCP and no `gh` access to the issue** — note it, continue with PR description, ask user to paste the ticket content if they want full scope verification. -- **PR not found** — confirm the number, repo, and that the user has access. -- **Branch not pushed** — fetch the latest and retry. +- **No MCP / `gh` access to the ticket** — note it, continue with the PR description, ask the user to paste the ticket if they want full scope verification. +- **PR not found** — confirm the number, repo, and access. +- **Branch not pushed** — `git fetch` and retry. -### Hard Fail +### Hard Fail (Stop immediately) -Stop if: -- The user repeatedly refuses to provide any PR identifier or ticket content. -- A required MCP returns repeated auth/permission errors. -- The skill cannot write to corresponding folder and the user will not allow a different path. +Stop execution if: +- the user repeatedly refuses to provide a PR identifier or ticket content. +- required MCP returns repeated auth/permission errors. +- skill cannot write to the output folder and the user won't allow a different path. --- ## User Interaction Examples -### Example 1: Well-Documented Feature PR with Linked Jira Ticket +### Example 1: Feature PR with Linked Jira Ticket -**User:** `Check the requirements for PR ...` - -**Output (abbreviated):** +**User:** "Check the requirements for PR ticket PRJ-1234 and save to cache `.testclaw/pr-summaries/` folder." ```md ## PR Requirements Summary @@ -326,83 +203,72 @@ Stop if: **PR:** Add user export to CSV **Branch:** feature/PRJ-789-user-export → main **Type:** feature -**Linked Tickets:** PRJ-111 +**Linked Tickets:** PRJ-789 **Source of Truth:** -- PR description: present (detailed, includes AC checklist) -- Jira ticket: resolved (PRJ-111) +- PR description: present (includes AC checklist + screenshot) +- Jira/GitHub ticket: resolved (PRJ-789) - Most reliable source: PR description + ticket (consistent) -**Changes:** Adds a CSV export action on the user list page that streams all matching users to a downloadable file. +**Changes:** Adds a CSV export action on the user list page that streams filtered users to a downloadable file, with email notification when exports exceed 10k rows. **Affected Files:** - `src/pages/UserList.tsx` — source - `src/services/ExportService.ts` — source -- `src/api/export.ts` — source - `i18n/en.json` — config - `tests/e2e/export.test.ts` — test **Impacted Areas:** -- User list page UI. -- User data export pipeline. -- Permission checks for export (admin only). +- User list page UI (new "Export" action). +- User data export pipeline (CSV streaming, email delivery). +- Permission checks for export (admin-only). **Scope Verification:** -- ✅ In scope: PRJ-111 AC1 (export button on list) — addressed in UserList.tsx -- ✅ In scope: PRJ-111 AC2 (admin-only) — addressed in ExportService.ts permission check -- ⚠️ Out of scope / missing: PRJ-111 AC3 (email notification when export is ready for >10k users) — no implementation found -- ➕ Extra (not in ticket): i18n string added for "Export ready" toast +- ✅ In scope: PRJ-789 AC1 (export button on list) — addressed in `UserList.tsx` +- ✅ In scope: PRJ-789 AC2 (admin-only) — addressed in `ExportService.ts` +- ⚠️ Out of scope / missing: PRJ-789 AC3 (email notification for >10k exports) — no implementation +- ➕ Extra (not in ticket): German locale strings **Ambiguities, Edge Cases & Open Questions:** -- Ticket does not specify CSV column order — needs product confirmation. -- PR comment from @reviewer: "What happens for users with no email?" — unanswered. -- Empty user list (zero rows). -- User without an email address. -- Export of 50k+ users (memory/timeout). +- CSV column order not specified — needs product confirmation. +- Empty user list, users without email, 50k+ user export (memory/timeout). **Acceptance Criteria:** -- Admin clicks "Export" on user list with filters applied → CSV file downloads with matching users -- Non-admin clicks "Export" → action is hidden / returns 403 -- Export of >10k users → email notification is sent when file is ready -- Export contains columns: id, name, email, created_at → in that order - -> Saved to: `.testclaw/pr-summaries/add-user-export-to-csv-pr-1234.md` - -(If `.testclaw/pr-summaries/` - yes, follow it. If not, create `.testclaw/pr-summaries/` at the repo root and add `.testclaw/` to `.gitignore` if not already there) +- Admin opens user list with filters and clicks "Export" → modal with row count +- Admin starts export with <10k rows → CSV downloads with `id, name, email, created_at` columns +- Non-admin opens user list → "Export" action is hidden (or 403 via API) ``` -### Example 2: Docs-Only PR (No Source Code Changes) +> Saved to: `.testclaw/pr-summaries/add-user-export-to-csv-pr-1234.md` -**User:** `Check the requirements for PR ...` +### Example 2: Docs-Only PR (Step 5 short-circuit) -**Output (abbreviated):** +When Step 5's short-circuit rule triggers, produce a minimal summary — no AC, no scope verification: ```md ## PR Requirements Summary -**PR:** ... — Update README with new install steps +**PR:** Update README with new install steps **Branch:** docs/update-install → main **Type:** docs **Linked Tickets:** none detected **Source of Truth:** - PR description: present (one-line) -- Jira ticket: not resolved +- Jira/GitHub ticket: not resolved - Most reliable source: PR description -**Changes:** Updates README install instructions to reflect the new Node 20 requirement and adds a Docker quick-start section. +**Changes:** Updates README install instructions for the new Node 20 requirement and adds a Docker quick-start section. **Affected Files:** - `README.md` — docs - `docs/install.md` — docs -**Impacted Areas:** (omitted — no business/technical impact beyond documentation) - > No source code behavior changed. Acceptance criteria generation is not applicable for this PR. - -> Saved to project root: `update-readme-with-new-install-steps-pr-1240.md` ``` +> Not Saved as there is no user's explicit request or instruction. + --- ## References @@ -414,16 +280,3 @@ Stop if: | Code-focused PR analysis (companion skill) | `../pr-diff/SKILL.md` | | Downstream: generate test cases from requirements | `../generate-cases/SKILL.md` | | Downstream: sync generated cases to Testomat.io | `../sync-cases/SKILL.md` | - ---- - -## Quick Commands - -| Action | Command | -|--------|---------| -| Detect PR for current branch | `gh pr view --json number,title,baseRefName,headRefName,url` | -| Read PR description + comments | `gh pr view {N} --json title,body,comments,labels` | -| Read PR diff (delegate to pr-diff) | `gh pr diff {N}` | -| Read linked GitHub issue | `gh issue view {N} --json title,body,comments` | -| Read branch commits | `git log {base}..HEAD --pretty=format:"%s%n%b"` | -| Files changed | `git diff {base}...HEAD --name-only` | diff --git a/skills/pr-requirements-check/references/summary-example.md b/skills/pr-requirements-check/references/summary-example.md index 7f8cb9c..14234e2 100644 --- a/skills/pr-requirements-check/references/summary-example.md +++ b/skills/pr-requirements-check/references/summary-example.md @@ -1,6 +1,6 @@ # PR Requirements Summary — Example -This file is a **filled-in example** of the artifact produced by the `pr-requirements-check` skill and saved to corresponding file. +A filled-in example of the artifact produced by the `pr-requirements-check` skill. --- @@ -35,22 +35,23 @@ This file is a **filled-in example** of the artifact produced by the `pr-require **Scope Verification:** - ✅ In scope: PRJ-789 AC1 (export button on the user list) — addressed in `src/pages/UserList.tsx` and `src/components/ExportModal.tsx` +- ✅ In scope: PRJ-789 AC2 (admin-only) — addressed in `ExportService.ts` permission check +- ⚠️ Out of scope / missing: PRJ-789 AC3 (email notification for >10k exports) — no implementation found - ➕ Extra (not in ticket): i18n strings for German locale (`i18n/de.json`) — ticket only mentioned English **Ambiguities, Edge Cases & Open Questions:** -- Ticket does not specify CSV column order — needs product confirmation. +- CSV column order not specified — needs product confirmation. - Concurrent exports by the same admin — no rate limiting or deduplication specified. -- Export of 50k+ users (memory/timeout) — no performance budget in the ticket. -- Localization of date/time fields in the CSV — not specified (UTC vs. user locale). +- 50k+ user export (memory/timeout) — no performance budget in the ticket. +- CSV date/time localization (UTC vs. user locale) — not specified. - Backward compatibility with existing scheduled exports (if any) — not addressed. **Acceptance Criteria:** -- Admin opens user list with filters applied and clicks "Export" → modal appears with row count estimate -- Admin clicks "Start export" with fewer than 10k matching users → CSV file downloads immediately with columns `id, name, email, created_at` in that order -- Non-admin opens user list → "Export" action is hidden (or returns 403 if accessed via API) -- Admin triggers export of >10k matching users → export is queued and an email notification is sent when the file is ready *(NOT IMPLEMENTED IN THIS PR — see Scope Verification)* -- User without an email address is included in the CSV → email field is empty string -- Export of an empty filtered list → button is disabled OR an empty CSV is downloaded with only the header row -- Exporting 50k+ users completes without server timeout or out-of-memory error -- CSV dates are formatted in the user's locale timezone (or UTC — confirm with product) -- German admin sees the export modal translated into German +- Admin opens user list with filters and clicks "Export" → modal appears with row count estimate +- Admin starts export with <10k matching users → CSV downloads with columns `id, name, email, created_at` in that order +- Non-admin opens user list → "Export" action is hidden (or returns 403 via API) +- Admin triggers >10k user export → export is queued, email sent when ready *(NOT IMPLEMENTED — see Scope Verification)* +- User without email is included → email field is empty string in CSV +- Empty filtered list → button disabled OR empty CSV with header only +- 50k+ user export completes without server timeout or OOM +- German admin sees the export modal translated From 133eb0e98dec90bfa0de819ddd5f3f32be4d240a Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Fri, 5 Jun 2026 23:17:50 +0300 Subject: [PATCH 3/5] [PR REQUIREMENTS] Rename to qa-pr-requirements-analyzer skill --- .../test-management/skills/qa-pr-requirements-analyzer | 1 + .../SKILL.md | 8 ++++---- .../references/summary-example.md | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) create mode 120000 plugins/test-management/skills/qa-pr-requirements-analyzer rename skills/{pr-requirements-check => qa-pr-requirements-analyzer}/SKILL.md (96%) rename skills/{pr-requirements-check => qa-pr-requirements-analyzer}/references/summary-example.md (96%) diff --git a/plugins/test-management/skills/qa-pr-requirements-analyzer b/plugins/test-management/skills/qa-pr-requirements-analyzer new file mode 120000 index 0000000..e50ee7a --- /dev/null +++ b/plugins/test-management/skills/qa-pr-requirements-analyzer @@ -0,0 +1 @@ +../../../skills/qa-pr-requirements-analyzer \ No newline at end of file diff --git a/skills/pr-requirements-check/SKILL.md b/skills/qa-pr-requirements-analyzer/SKILL.md similarity index 96% rename from skills/pr-requirements-check/SKILL.md rename to skills/qa-pr-requirements-analyzer/SKILL.md index b7c6706..d6aeda5 100644 --- a/skills/pr-requirements-check/SKILL.md +++ b/skills/qa-pr-requirements-analyzer/SKILL.md @@ -1,5 +1,5 @@ --- -name: pr-requirements-check +name: qa-pr-requirements-analyzer description: Analyze a pull request's context - title, description, comments, linked issues, attached schemas/images — to understand the original intent, verify scope against the linked ticket, surface ambiguities and edge cases, and produce a structured requirements summary with testable acceptance criteria. Use this skill when the user wants to understand WHAT a PR is supposed to do (not just what code changed), check whether the PR matches its linked ticket/task or generate acceptance criteria from PR/ticket context rather than from code. license: MIT metadata: @@ -7,7 +7,7 @@ metadata: version: 1.1.0 --- -# PR-REQUIREMENTS-CHECK SKILL: What I Do +# QA-PR-REQUIREMENTS-ANALYZER SKILL: What I Do This skill reviews a pull request's **context** - the human-written parts: title, description, comments, linked tickets, attached images - to extract the original requirements, verify scope, and produce a requirements summary that downstream skills (`generate-cases`) can consume. @@ -25,9 +25,9 @@ It is the **complement** of `pr-diff`: | Skill | Input | Output | Focus | |-------|-------|--------|-------| | `pr-diff` | git diff, changed files | AC from code changes | "what the code does" | -| `pr-requirements-check` | PR title, description, comments, linked issue | summary or requirements with scope verification, AC | "what the PR is supposed to do" | +| `qa-pr-requirements-analyzer` | PR title, description, comments, linked issue | summary or requirements with scope verification, AC | "what the PR is supposed to do" | -> Use them together for a full picture. Use `pr-requirements-check` alone when the question is "is this PR doing the right thing?" rather than "what does this PR do?". +> Use them together for a full picture. Use `qa-pr-requirements-analyzer` alone when the question is "is this PR doing the right thing?" rather than "what does this PR do?". --- diff --git a/skills/pr-requirements-check/references/summary-example.md b/skills/qa-pr-requirements-analyzer/references/summary-example.md similarity index 96% rename from skills/pr-requirements-check/references/summary-example.md rename to skills/qa-pr-requirements-analyzer/references/summary-example.md index 14234e2..b3f8312 100644 --- a/skills/pr-requirements-check/references/summary-example.md +++ b/skills/qa-pr-requirements-analyzer/references/summary-example.md @@ -1,6 +1,6 @@ # PR Requirements Summary — Example -A filled-in example of the artifact produced by the `pr-requirements-check` skill. +A filled-in example of the artifact produced by the `qa-pr-requirements-analyzer` skill. --- From eed03d23f3280df58085377e98d8ecf4e0c7b3c2 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Sun, 14 Jun 2026 14:18:37 +0300 Subject: [PATCH 4/5] [PR REQUIREMENTS] Skill text updates after review --- skills/qa-pr-requirements-analyzer/SKILL.md | 143 ++++-------------- .../references/summary-example.md | 19 +-- 2 files changed, 41 insertions(+), 121 deletions(-) diff --git a/skills/qa-pr-requirements-analyzer/SKILL.md b/skills/qa-pr-requirements-analyzer/SKILL.md index d6aeda5..f9ad737 100644 --- a/skills/qa-pr-requirements-analyzer/SKILL.md +++ b/skills/qa-pr-requirements-analyzer/SKILL.md @@ -9,7 +9,7 @@ metadata: # QA-PR-REQUIREMENTS-ANALYZER SKILL: What I Do -This skill reviews a pull request's **context** - the human-written parts: title, description, comments, linked tickets, attached images - to extract the original requirements, verify scope, and produce a requirements summary that downstream skills (`generate-cases`) can consume. +This skill reviews a pull request's **context** - the human-written parts: title, description, comments, linked tickets, attached images - to extract the original requirements, verify scope, and produce a requirements summary that downstream skills (`qa-write-test-cases`) can consume. **Core Workflow:** - Read the PR's human-written context (title, description, comments, images, linked tickets). @@ -20,11 +20,11 @@ This skill reviews a pull request's **context** - the human-written parts: title **Full PR knowledge by available skills** -It is the **complement** of `pr-diff`: +It is the **complement** of `pull-request-diff-analyzer`: | Skill | Input | Output | Focus | |-------|-------|--------|-------| -| `pr-diff` | git diff, changed files | AC from code changes | "what the code does" | +| `pull-request-diff-analyzer` | git diff, changed files | AC from code changes | "what the code does" | | `qa-pr-requirements-analyzer` | PR title, description, comments, linked issue | summary or requirements with scope verification, AC | "what the PR is supposed to do" | > Use them together for a full picture. Use `qa-pr-requirements-analyzer` alone when the question is "is this PR doing the right thing?" rather than "what does this PR do?". @@ -39,7 +39,7 @@ Trigger this skill when the user wants to: - **Find ambiguities or missing cases** in the PR description before generating test cases. - **Generate acceptance criteria from ticket context** (especially when the PR description is thin but the ticket is detailed). -Do **not** trigger this skill when the user only wants to know what code changed - that is `pr-diff` skill. Trigger it when the user's words include things like "requirements", "ticket", "Jira", "issue", "scope", "what is this PR supposed to do", "does this match the ticket", "missing acceptance criteria", "ambiguities in this PR". +Do **not** trigger this skill when the user only wants to know what code changed - that is `pull-request-diff-analyzer` skill. Trigger it when the user's words include things like "requirements", "ticket", "Jira", "issue", "scope", "what is this PR supposed to do", "does this match the ticket", "missing acceptance criteria", "ambiguities in this PR". --- @@ -92,22 +92,26 @@ Compare what the PR does against what the ticket asked for, and surface anything **Scope Verification** — build three lists: - **In scope** — ticket requirements addressed by the PR (description or code, see Step 5). -- **Out of scope / missing** — ticket requirements with no corresponding PR change. These are the most valuable output of this skill. +- **Out of scope** — ticket requirements with no corresponding PR change. These are the most valuable output of this skill. - **Extra / out of ticket** — PR changes not mentioned in the ticket. Flag for review (intentional or accidental?). -**Ambiguities & Edge Cases:** -- **Underspecified behavior** — ticket says "support export" but not format, naming, empty-data handling, etc. -- **Unanswered comments** — questions in PR or ticket comments with no reply. -- **Conflicting requirements** — different parts of the ticket (or ticket vs. PR description) disagree. -- **Missing edge cases** — empty/null inputs, permissions/roles, backward compatibility, localization/timezone/currency. +**Ambiguities, Edge Cases & Open Questions** - surface only issues that are supported by the available context. Do not invent new requirements or assume the product should handle a scenario unless the ticket, PR description, comments, or surrounding context suggest it is relevant. + +These are possible questions for clarification: +- **Underspecified behavior** — the ticket describes a capability but omits important details (e.g. export is required, but format, naming, permissions, or empty-state behavior are not defined). +- **Unanswered comments** — questions in PR or ticket comments with no visible resolution. +- **Conflicting requirements** — different parts of the ticket, comments, or PR description disagree. +- **Potentially relevant edge cases** — scenarios that appear related to the stated requirements but whose expected behavior is not specified. Present these as questions, not requirements. + - Never report an edge case solely because it is a common software concern. + - Only raise an edge case when the ticket, PR description, affected domain, or discussion suggests it may be relevant. --- ### Step 5: Identify Affected Files & Edge Cases -This skill is not `pr-diff` — do not do a deep code review. Prefer files **mentioned in the PR description or ticket**; cross-check with `git diff {base}...HEAD --name-only` only if the description doesn't enumerate them. +This skill is not `pull-request-diff-analyzer` — do not do a deep code review. Prefer files **mentioned in the PR description or ticket**; cross-check with `git diff {base}...HEAD --name-only` only if the description doesn't enumerate them. -**Short-circuit (no source behavior):** if every changed file is non-source-code (`*.md`, `docs/**`, `tests/**`, configs, CI, deps, lock files), do not invent AC and do not invoke `pr-diff`. Produce a short summary with: PR title, type (`docs`/`chore`/`ci`/`deps`), one-sentence `Changes`, full file list. +**Short-circuit (no source behavior):** if every changed file is non-source-code (`*.md`, `docs/**`, `tests/**`, configs, CI, deps, lock files), do not invent AC and do not invoke `pull-request-diff-analyzer`. Produce a short summary with: PR title, type (`docs`/`chore`/`ci`/`deps`), one-sentence `Changes`, full file list. - Post explicit note `> No source code behavior changed. Acceptance criteria generation is not applicable for this PR.` Stop after this. **Empty PR description:** if there's no body and the branch name doesn't encode intent, pull `git log {base}..HEAD --pretty=format:"%s%n%b"`, combine with changed files and branch name into a tentative one-sentence `Changes`, and mark `> PR description was empty. Summary derived from branch name, commits, and changed files only.` @@ -129,27 +133,27 @@ Compose the final output and save it to disk only when the original prompt inclu **PR:** {title} **Branch:** {headRefName} → {baseRefName} **Type:** {feature | bugfix | refactor | deps} -**Linked Tickets:** {list or "none detected"} +**Linked Tickets:** ... (only of exist) **Source of Truth:** - PR description: {present | empty — derived from commits/branch} - Jira/GitHub ticket: {resolved | not resolved — no MCP available} - Most reliable source: {PR description | ticket | commits} -**Changes:** ... (one sentence — intent, not implementation) +**Changes Overview:** ... (1-2 sentences — intent, not implementation) + +**Impacted Areas:** +- Payment processing (6 source files) +- Billing configuration (2 config files) +- ... (bullet list, only meaningful business/technical impact; omit if empty) -**Affected Files:** +**High Level Key files (up to 5):** - `path/to/file.ts` — source -- `path/to/config.yml` — config -- `docs/readme.md` — docs - ... (categorized: source | config | test | docs | migration | deps) -**Impacted Areas:** -- ... (1–3 bullets, only meaningful business/technical impact; omit if empty) - **Scope Verification:** - ✅ In scope: {ticket requirement → where in PR it is addressed} -- ⚠️ Out of scope / missing: {ticket requirement not addressed in PR} +- ⚠️ Out of scope: {ticket requirement not addressed in PR} - ➕ Extra (not in ticket): {change present in PR but not in ticket} **Ambiguities, Edge Cases & Open Questions:** @@ -164,11 +168,10 @@ Compose the final output and save it to disk only when the original prompt inclu **Summary Rules:** - Concise, high-level, requirements-not-implementation — this is a requirements document, not a code review. -- `Changes` is **one sentence**. -- `Impacted Areas` — 1–3 meaningful bullets; omit the section if empty. +- `Changes Overview` is **1-2 sentence**. +- `Impacted Areas` — meaningful bullets; omit the section if empty. - `Source of Truth` is required — tells downstream skills what to trust when sources disagree. -- Scope Verification — be honest; `⚠️ Out of scope` items are the most valuable output of this skill. -- Acceptance Criteria — testable, user-oriented, written as "action → expected result". If no source-code behavior changed, replace this section with the explicit "not applicable" note from Step 5. +- `Acceptance Criteria` — testable, user-oriented, written as "action → expected result". If no source-code behavior changed, replace this section with the explicit "not applicable" note from Step 5. - Omit empty sections instead of writing `None`/`N/A` — except the "no source code changed" note which must be explicit. - Avoid generic statements like "code cleanup", "minor fixes", "various improvements" — name what changed or omit it. @@ -179,104 +182,24 @@ Compose the final output and save it to disk only when the original prompt inclu ### Recovery - **No MCP / `gh` access to the ticket** — note it, continue with the PR description, ask the user to paste the ticket if they want full scope verification. -- **PR not found** — confirm the number, repo, and access. -- **Branch not pushed** — `git fetch` and retry. +- **PR not found** — the referenced pull request cannot be located. Confirm the PR number, repository, and access permissions. If the PR is private or in a different repository, ask the user to provide the correct identifier or relevant context directly. +- **Branch not available remotely** — the referenced branch cannot be found on any remote. Confirm whether the branch has been pushed. If it exists only locally, ask the user to push it or provide the PR details/context directly. ### Hard Fail (Stop immediately) Stop execution if: - the user repeatedly refuses to provide a PR identifier or ticket content. -- required MCP returns repeated auth/permission errors. +- third part services returns repeated auth/permission errors. - skill cannot write to the output folder and the user won't allow a different path. --- -## User Interaction Examples - -### Example 1: Feature PR with Linked Jira Ticket - -**User:** "Check the requirements for PR ticket PRJ-1234 and save to cache `.testclaw/pr-summaries/` folder." - -```md -## PR Requirements Summary - -**PR:** Add user export to CSV -**Branch:** feature/PRJ-789-user-export → main -**Type:** feature -**Linked Tickets:** PRJ-789 - -**Source of Truth:** -- PR description: present (includes AC checklist + screenshot) -- Jira/GitHub ticket: resolved (PRJ-789) -- Most reliable source: PR description + ticket (consistent) - -**Changes:** Adds a CSV export action on the user list page that streams filtered users to a downloadable file, with email notification when exports exceed 10k rows. - -**Affected Files:** -- `src/pages/UserList.tsx` — source -- `src/services/ExportService.ts` — source -- `i18n/en.json` — config -- `tests/e2e/export.test.ts` — test - -**Impacted Areas:** -- User list page UI (new "Export" action). -- User data export pipeline (CSV streaming, email delivery). -- Permission checks for export (admin-only). - -**Scope Verification:** -- ✅ In scope: PRJ-789 AC1 (export button on list) — addressed in `UserList.tsx` -- ✅ In scope: PRJ-789 AC2 (admin-only) — addressed in `ExportService.ts` -- ⚠️ Out of scope / missing: PRJ-789 AC3 (email notification for >10k exports) — no implementation -- ➕ Extra (not in ticket): German locale strings - -**Ambiguities, Edge Cases & Open Questions:** -- CSV column order not specified — needs product confirmation. -- Empty user list, users without email, 50k+ user export (memory/timeout). - -**Acceptance Criteria:** -- Admin opens user list with filters and clicks "Export" → modal with row count -- Admin starts export with <10k rows → CSV downloads with `id, name, email, created_at` columns -- Non-admin opens user list → "Export" action is hidden (or 403 via API) -``` - -> Saved to: `.testclaw/pr-summaries/add-user-export-to-csv-pr-1234.md` - -### Example 2: Docs-Only PR (Step 5 short-circuit) - -When Step 5's short-circuit rule triggers, produce a minimal summary — no AC, no scope verification: - -```md -## PR Requirements Summary - -**PR:** Update README with new install steps -**Branch:** docs/update-install → main -**Type:** docs -**Linked Tickets:** none detected - -**Source of Truth:** -- PR description: present (one-line) -- Jira/GitHub ticket: not resolved -- Most reliable source: PR description - -**Changes:** Updates README install instructions for the new Node 20 requirement and adds a Docker quick-start section. - -**Affected Files:** -- `README.md` — docs -- `docs/install.md` — docs - -> No source code behavior changed. Acceptance criteria generation is not applicable for this PR. -``` - -> Not Saved as there is no user's explicit request or instruction. - ---- - ## References | Description | File | |-------------|------| | Filled example of the output summary | [summary-example.md](./references/summary-example.md) | | `.testclaw/` directory convention | inherited from `project-scan` (see project-scan SKILL.md) | -| Code-focused PR analysis (companion skill) | `../pr-diff/SKILL.md` | -| Downstream: generate test cases from requirements | `../generate-cases/SKILL.md` | +| Code-focused PR analysis (companion skill) | `../pull-request-diff-analyzer/SKILL.md` | +| Downstream: generate test cases from requirements | `../qa-write-test-cases/SKILL.md` | | Downstream: sync generated cases to Testomat.io | `../sync-cases/SKILL.md` | diff --git a/skills/qa-pr-requirements-analyzer/references/summary-example.md b/skills/qa-pr-requirements-analyzer/references/summary-example.md index b3f8312..98fe0f4 100644 --- a/skills/qa-pr-requirements-analyzer/references/summary-example.md +++ b/skills/qa-pr-requirements-analyzer/references/summary-example.md @@ -16,27 +16,24 @@ A filled-in example of the artifact produced by the `qa-pr-requirements-analyzer - Jira/GitHub ticket: resolved (PRJ-789) - Most reliable source: PR description + ticket (consistent, no conflicts) -**Changes:** Adds a CSV export action on the user list page that streams all currently filtered users to a downloadable file, with an email notification when exports exceed 10k rows. +**Changes Overview:** Adds a CSV export action on the user list page that streams all currently filtered users to a downloadable file, with an email notification when exports exceed 10k rows. -**Affected Files:** +**Impacted Areas:** +- User list page UI (new "Export" action). +- User data export pipeline (CSV generation, streaming, email delivery). +- Permission checks for export (admin-only). + +**High Level Key files (up to 5):** - `src/pages/UserList.tsx` — source - `src/services/ExportService.ts` — source - `src/api/export.ts` — source - `src/components/ExportModal.tsx` — source - `i18n/en.json` — config -- `i18n/de.json` — config -- `tests/e2e/export.test.ts` — test -- `docs/features/export.md` — docs - -**Impacted Areas:** -- User list page UI (new "Export" action). -- User data export pipeline (CSV generation, streaming, email delivery). -- Permission checks for export (admin-only). **Scope Verification:** - ✅ In scope: PRJ-789 AC1 (export button on the user list) — addressed in `src/pages/UserList.tsx` and `src/components/ExportModal.tsx` - ✅ In scope: PRJ-789 AC2 (admin-only) — addressed in `ExportService.ts` permission check -- ⚠️ Out of scope / missing: PRJ-789 AC3 (email notification for >10k exports) — no implementation found +- ⚠️ Out of scope: PRJ-789 AC3 (email notification for >10k exports) — no implementation found - ➕ Extra (not in ticket): i18n strings for German locale (`i18n/de.json`) — ticket only mentioned English **Ambiguities, Edge Cases & Open Questions:** From 495a68e46542bc2535d67f34e103ec235ed69138 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Sun, 14 Jun 2026 14:26:45 +0300 Subject: [PATCH 5/5] [PR REQUIREMENTS] Typo fix --- skills/qa-pr-requirements-analyzer/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/qa-pr-requirements-analyzer/SKILL.md b/skills/qa-pr-requirements-analyzer/SKILL.md index f9ad737..51839ec 100644 --- a/skills/qa-pr-requirements-analyzer/SKILL.md +++ b/skills/qa-pr-requirements-analyzer/SKILL.md @@ -93,7 +93,7 @@ Compare what the PR does against what the ticket asked for, and surface anything **Scope Verification** — build three lists: - **In scope** — ticket requirements addressed by the PR (description or code, see Step 5). - **Out of scope** — ticket requirements with no corresponding PR change. These are the most valuable output of this skill. -- **Extra / out of ticket** — PR changes not mentioned in the ticket. Flag for review (intentional or accidental?). +- **Extra (not in ticket)** — PR changes not mentioned in the ticket. Flag for review (intentional or accidental?). **Ambiguities, Edge Cases & Open Questions** - surface only issues that are supported by the available context. Do not invent new requirements or assume the product should handle a scenario unless the ticket, PR description, comments, or surrounding context suggest it is relevant.