diff --git a/README.md b/README.md index 5f9baf9..78e5638 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,7 @@ For other ways of installation (Claude Code plugin, Codex, Cursor etc.) see [ins | `improve-test-cases` | Analyze and improve existing markdown test cases for clarity | | `detect-duplicate-test-cases` | Find duplicate, near-duplicate, and overlapping test cases | | `sync-test-cases-with-tms` | Synchronize Markdown test scenarios between local project and Testomat.io | -| `qa-manual-tests-to-code-coverage` | Map manual test cases to source files; generate `coverage.manual.yml` for affected-only runs | -| `e2e-test-coverage-mapping` | Map e2e tests to source files; generate `coverage.e2e.yml` to run only the tests affected by a diff | +| `qa-test-coverage-map` | Map manual and automated tests to source files; generate a per-project `coverage.*.yml` for affected-only runs | | `senior-qa-testing-workflow` | Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS | | `scan-automation-project` | Scan project source code to inventory languages, frameworks, and existing tests | diff --git a/plugins/test-management/skills/e2e-test-coverage-mapping b/plugins/test-management/skills/e2e-test-coverage-mapping deleted file mode 120000 index 05311a6..0000000 --- a/plugins/test-management/skills/e2e-test-coverage-mapping +++ /dev/null @@ -1 +0,0 @@ -../../../skills/e2e-test-coverage-mapping \ No newline at end of file diff --git a/plugins/test-management/skills/qa-manual-tests-to-code-coverage b/plugins/test-management/skills/qa-manual-tests-to-code-coverage deleted file mode 120000 index a9a7df2..0000000 --- a/plugins/test-management/skills/qa-manual-tests-to-code-coverage +++ /dev/null @@ -1 +0,0 @@ -../../../skills/qa-manual-tests-to-code-coverage \ No newline at end of file diff --git a/plugins/test-management/skills/qa-test-coverage-map b/plugins/test-management/skills/qa-test-coverage-map new file mode 120000 index 0000000..ac1c194 --- /dev/null +++ b/plugins/test-management/skills/qa-test-coverage-map @@ -0,0 +1 @@ +../../../skills/qa-test-coverage-map \ No newline at end of file diff --git a/plugins/test-management/skills/setup-pr-testing b/plugins/test-management/skills/setup-pr-testing new file mode 120000 index 0000000..82ae913 --- /dev/null +++ b/plugins/test-management/skills/setup-pr-testing @@ -0,0 +1 @@ +../../../skills/setup-pr-testing \ No newline at end of file diff --git a/skills/e2e-test-coverage-mapping/SKILL.md b/skills/e2e-test-coverage-mapping/SKILL.md deleted file mode 100644 index 5c74e0b..0000000 --- a/skills/e2e-test-coverage-mapping/SKILL.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -name: e2e-test-coverage-mapping -description: Map automated end-to-end tests (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium) to source code files and produce a `coverage.e2e.yml` mapping consumed by `@testomatio/reporter --filter "coverage:..."`. Use this skill when the user wants to run only the e2e tests affected by a code change, generate an e2e coverage file, or build a traceability matrix between automated tests and source code. -license: MIT -metadata: - author: Testomat.io - version: 1.0.0 ---- - -# AUTOMATION-COVERAGE SKILL: What I do - -This skill analyzes **automated e2e tests** and the project source code, then produces `coverage.e2e.yml` — a mapping from source files (or globs) to test identifiers (suite IDs, test IDs, tags) embedded in the test code. The mapping is consumed by `@testomatio/reporter run "" --filter "coverage:file=coverage.e2e.yml,diff="` to run only the e2e tests affected by the diff. - -## When to Use - -Trigger this skill when the user wants to: -- Map automated e2e tests to the source code they exercise. -- Generate `coverage.e2e.yml` (or a similarly named file) for use with `@testomatio/reporter`. -- Run only the e2e tests affected by a change instead of the full suite. -- Build a code → e2e-test traceability matrix. -- Speed up CI by limiting the e2e run to tests affected by the PR diff. -- Phrases: "e2e coverage", "automated test coverage", "map tests to code", "affected e2e tests", "selective e2e run", "generate coverage.e2e.yml". - ---- - -## CRITICAL CONSTRAINTS - -This skill works **only with automated e2e tests** (Playwright, Cypress, WebdriverIO, Puppeteer, CodeceptJS, Appium, etc.). - -- **DO NOT** process unit tests. -- **DO NOT** process manual markdown test cases (use `qa-manual-tests-to-code-coverage` instead - if already exists). -- **DO NOT** suggest creating new tests. -- **Only touch two files in this repo.** It may write `coverage.e2e.yml` (or the path the user gave) and add one `.testclaw/` line to `.gitignore` if it is missing. Nothing else — never a source or test file. If the e2e tests live in another repo, clone it into the gitignored `.testclaw/e2e-tests/`, never into a tracked folder (see Step 1). -- **Don't write scripts. Never use Python.** Read test files with your file tool; pull out IDs and tags with `grep` (Step 3). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper (symlinked from `qa-manual-tests-to-code-coverage`): `npx js-yaml coverage.e2e.yml | node scripts/check-coverage.mjs` (Step 6). That's the only script. If you ever need more than a `grep`, a one-line `node -e '…'` is the limit — never `python`, never a parser of your own. - ---- - -## Workflow: Build E2E Coverage Map - -### Step 1: Discover the project (delegate to `scan-automation-project`) - -Run the **`scan-automation-project`** skill to inventory the project. Use its output as the source of truth for framework detection, the e2e test set, and the high-level codebase overview — do not duplicate those scans here. - -From the `scan-automation-project` result, capture: -- **Frameworks** — automation framework(s) in use (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium, Mocha, Jest…). See [E2E Frameworks](./references/E2E_FRAMEWORKS.md) for detection signals if `scan-automation-project` is ambiguous. -- **Automated Tests** — the list of e2e test files (the input to Step 3). -- **Project Overview** — languages, complexity (the framing for Step 5). - -If `scan-automation-project` reports **no automated tests** (it checks `.testclaw/e2e-tests/` too, so this means nothing was cloned before), or no e2e framework is detected: -- ❓ Ask the user to either: - 1. Give the path to the e2e tests (then re-run `scan-automation-project` there). - 2. Give the git URL of the e2e tests repo → `git clone .testclaw/e2e-tests`, add `.testclaw/` to `.gitignore` if missing, and re-run `scan-automation-project` against `.testclaw/e2e-tests`. - 3. Stop. - -Never clone the tests into a tracked folder in this repo. - -If two frameworks are detected (say Jest and Playwright), ask which one is the e2e framework — coverage filtering runs per runner. - -### Step 2: Verify Testomatio IDs are present - -Coverage filtering relies on `@S` / `@T` identifiers embedded in the test code: - -```javascript -describe('user settings @S92321384', () => { - it('updates avatar @Ta011dfa3', () => { ... }); -}); -``` - -```javascript -test('login @smoke @T6f8e9174', async ({ page }) => { ... }); -``` - -If most files have no `@S` / `@T` markers, stop and instruct the user to populate them first by running: - -```bash -npx check-tests@latest "" --update-ids -``` - -(see the `qa-e2e-tests-reporting` skill for the full per-framework command). Without IDs in the source, the reporter cannot select tests by coverage. - -### Step 3: Extract test information - -For each test file extract: - -- **Suite IDs** — `@S` + 8 chars in `describe` / `context` / `Feature` blocks. -- **Test IDs** — `@T` + 8 chars in `it` / `test` / `Scenario` blocks. -- **Tags** — other `@word` markers (`@smoke`, `@jira-123`, `@regression`). -- **What is exercised** — page objects imported, routes hit, fixtures used — used to reason about which source files each test covers. - -**How to extract.** Read the test files, or `grep` for the IDs and tags in test/suite names. Run these in whichever directory holds the tests (`tests/e2e`, or the cache `.testclaw/e2e-tests`): - -```bash -grep -rnoE '@[ST][0-9a-f]{8}' # suite/test IDs (+ which file/line) -grep -rnoE '@[A-Za-z0-9_-]+' | sort -u # every @token, tags included -``` - -Don't write a parser. Never use `python`. If there are no `@S`/`@T` IDs, add them first with `npx check-tests@latest "" --update-ids` (see Step 2). - -### Step 4: Explore the source codebase - -Use the **Project Overview** from Step 1's `scan-automation-project` result (languages, frameworks, complexity) as the starting frame, then identify business code the tests exercise. Skip: - -- Test code itself. -- Dependency / build / vendor folders. -- Framework configs (`playwright.config.*`, `cypress.config.*`, `wdio.conf.*`, `codecept.conf.*`, lock files). - -**Templates and views are source code — map them too.** E2E tests drive the rendered UI, so a change to a template breaks or alters the page a test asserts on. Treat view/template files as first-class mappable source alongside controllers, models, and components — do **not** skip them because they aren't `.js`/`.ts`/`.py`. Cover at least: - -- HTML & component templates: `.html`, `.htm`, `.vue`, `.svelte`, Angular `*.component.html`. -- Logic-in-markup engines: `.hbs`/`.handlebars`, `.ejs`, `.pug`/`.jade`, `.mustache`, `.liquid`. -- Server-side views: `.erb`, `.haml`, `.slim` (Rails); `.blade.php`, `.twig` (PHP); `.j2`/`.jinja`/`.jinja2` (Python); `.cshtml`/`.razor` (.NET); `.jsp` (Java). - -Map a template the same way as code: to the suite/test/tag whose e2e tests render and assert against that view (e.g. `app/views/sessions/new.html.erb` → the login suite). - -❓ If the project structure is large or ambiguous (`scan-automation-project` reports `large` / `very-large` complexity), ask which directories to focus on or to exclude. - -### Step 5: Choose the best mapping per source file - -For each candidate source file, pick **one** strategy: - -**A) Map to Suite (`@S...`)** — when most tests in a suite relate to the file. - -```yaml -app/models/user.rb: - - "@S92321384" # Suite: user settings e2e tests -``` - -**B) Map to Test (`@T...`)** — when only one test matches the file. - -```yaml -app/controllers/sessions_controller.rb: - - "@Ta011dfa3" # Test: login with valid credentials -``` - -**C) Map to Tag (`@tag`)** — when a tag groups tests across suites. - -```yaml -tag:@smoke: - - "@Ta011dfa3" - - "@Tb022dfa4" -``` - -**Rules:** -- Prefer specific file paths when tests target a single file. -- Prefer globs (`app/services/jira/**`) when tests cover an entire subtree. -- Prefer Suite mapping when most of a suite relates — terser, survives test additions. -- Use Test mapping when only one test in a large suite is relevant. -- Use Tag mapping for cross-cutting concerns. -- Avoid empty entries. -- Add `#` comments next to each identifier explaining the mapping. - -See [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md) for the full YAML grammar. - -### Step 6: Save and validate the coverage file - -Write the YAML to the resolved output path (default `coverage.e2e.yml` in the project root). If the user supplied a different path => use it. - -**Check it** — one command, run from the project root: - -```bash -npx js-yaml coverage.e2e.yml | node scripts/check-coverage.mjs -``` - -`npx js-yaml` parses the file (and fails loudly if the YAML is malformed, so a broken file never reaches the script). `check-coverage.mjs` then flags any key whose path is missing on disk, any key with no identifiers, and prints every `@S…` / `@T…` / tag it references. Check that list against the IDs you extracted in Step 3 — only you know which are real. Don't re-parse the test files; use the set you already have. Never use `python`. - -> The keys in the coverage file are paths to source files in this repo, never `.testclaw/...` paths. The cache holds the cloned tests; the coverage file points at the code they exercise. - -Then show the produced YAML to the user. - -### Step 7: Show next steps - -Tell the user how to use the file with `@testomatio/reporter`. The runner command **must** be passed in — `--filter` generates `--grep` that the runner consumes: - -```bash -# Playwright -npx @testomatio/reporter run "npx playwright test" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" - -# Cypress -npx @testomatio/reporter run "npx cypress run" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" - -# WebdriverIO -npx @testomatio/reporter run "npx wdio" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" - -# CodeceptJS -npx @testomatio/reporter run "npx codeceptjs run" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" -``` - -Recommend committing `coverage.e2e.yml` to the repository so CI uses the same mapping. Provide a GitHub Actions snippet on request — see [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md) for the reporter contract. - -### Step 8: Final summary - -Report: -- Framework detected. -- Number of test files scanned. -- Number of tests with `@S` / `@T` IDs. -- Number of source files mapped. -- Output file path. - ---- - -## References - -| Description | File | -| ---------------------------- | --------------------------------------------- | -| Coverage YAML format | ./references/COVERAGE_FILE_FORMAT.md | -| E2E framework detection | ./references/E2E_FRAMEWORKS.md | - -## Bundled script - -`scripts/check-coverage.mjs` (~25 lines, zero deps; symlinked from `qa-manual-tests-to-code-coverage`) — reads the parsed coverage map on stdin, flags keys whose path is missing on disk and keys with no identifiers, and lists every `@S`/`@T`/tag the file references. Feed it via `js-yaml`, from the project root: - -```bash -npx js-yaml coverage.e2e.yml | node scripts/check-coverage.mjs -``` - -It's the only script — everything else is `grep`, your file tool, or `npx js-yaml`. Don't rewrite it in `python`. - ---- - -## Error Handling - -### Recovery - -- **No e2e tests found (cache included)** → ask for the path, or a git URL to clone into the gitignored `.testclaw/e2e-tests/`. -- **Tests have no `@S`/`@T` IDs** → instruct the user to run `npx check-tests "" --update-ids` (cross-link `qa-e2e-tests-reporting`). -- **Ambiguous source layout** → ask which directories are application code. - -### Hard Fail (stop immediately) - -- Cannot create or write the output file. -- User refuses to provide a tests directory or to clone a repo. -- The agent is asked to modify files other than the output file (refuse — see CRITICAL CONSTRAINTS). - ---- - -## Examples - -**Playwright project, default output:** -``` -Use e2e-test-coverage-mapping skill to map our Playwright tests to source code -``` - -**Custom directory + output:** -``` -Use e2e-test-coverage-mapping skill, tests in e2e/playwright, output to ops/coverage.e2e.yml -``` - -**Full workflow (CI):** -1. Run `qa-e2e-tests-reporting` to install `@testomatio/reporter` and import tests via `check-tests --update-ids` (so suite/test IDs land in the source). -2. Run `e2e-test-coverage-mapping` — internally delegates to `scan-automation-project` for framework detection and inventory, then produces `coverage.e2e.yml`. -3. Commit `coverage.e2e.yml`. -4. In CI, run `npx @testomatio/reporter run "npx playwright test" --filter "coverage:file=coverage.e2e.yml,diff=origin/main"` — only the affected tests execute. - ---- - -## Quick Commands - -| Action | Command | -| ----------------------------------- | ------------------------------------------------------------------------------------------------------ | -| Populate IDs in test source | `npx check-tests@latest "" --update-ids` | -| Run affected Playwright tests | `npx @testomatio/reporter run "npx playwright test" --filter "coverage:file=coverage.e2e.yml,diff=main"` | -| Run affected Cypress tests | `npx @testomatio/reporter run "npx cypress run" --filter "coverage:file=coverage.e2e.yml,diff=main"` | -| Run affected CodeceptJS tests | `npx @testomatio/reporter run "npx codeceptjs run" --filter "coverage:file=coverage.e2e.yml,diff=main"` | diff --git a/skills/e2e-test-coverage-mapping/references/COVERAGE_FILE_FORMAT.md b/skills/e2e-test-coverage-mapping/references/COVERAGE_FILE_FORMAT.md deleted file mode 120000 index c9e33ff..0000000 --- a/skills/e2e-test-coverage-mapping/references/COVERAGE_FILE_FORMAT.md +++ /dev/null @@ -1 +0,0 @@ -../../qa-manual-tests-to-code-coverage/references/COVERAGE_FILE_FORMAT.md \ No newline at end of file diff --git a/skills/e2e-test-coverage-mapping/scripts/check-coverage.mjs b/skills/e2e-test-coverage-mapping/scripts/check-coverage.mjs deleted file mode 120000 index 69e7a14..0000000 --- a/skills/e2e-test-coverage-mapping/scripts/check-coverage.mjs +++ /dev/null @@ -1 +0,0 @@ -../../qa-manual-tests-to-code-coverage/scripts/check-coverage.mjs \ No newline at end of file diff --git a/skills/qa-manual-tests-to-code-coverage/SKILL.md b/skills/qa-manual-tests-to-code-coverage/SKILL.md deleted file mode 100644 index 49f3268..0000000 --- a/skills/qa-manual-tests-to-code-coverage/SKILL.md +++ /dev/null @@ -1,253 +0,0 @@ ---- -name: qa-manual-tests-to-code-coverage -description: Map manual test cases (markdown) to source code files and produce a `coverage.manual.yml` mapping consumed by `@testomatio/reporter --filter "coverage:..."`. Use this skill when the user wants to run only the manual tests affected by a code change, generate a manual coverage file, build a traceability matrix between manual cases and source code, or set up change-aware regression for manual QA. -license: MIT -metadata: - author: Testomat.io - version: 1.0.0 ---- - -# MANUAL-COVERAGE SKILL: What I do - -This skill analyzes **manual test cases in markdown format** and the project source code, then produces `coverage.manual.yml` — a mapping from source files (or globs) to manual test identifiers (suite IDs, test IDs, tags). The mapping is consumed by `@testomatio/reporter run --filter "coverage:file=coverage.manual.yml,diff="` to create manual runs in Testomat.io that contain only the cases affected by the diff. - -## When to Use - -Trigger this skill when the user wants to: -- Map manual test cases to the source code that implements them. -- Generate `coverage.manual.yml` (or a similarly named file) for use with `@testomatio/reporter`. -- Run only manual regression tests relevant to a code change instead of the full suite. -- Build a code → manual-test traceability matrix. -- Find dead manual tests (tests with no matching source) or coverage gaps (source with no manual tests). -- Phrases: "manual test coverage", "map manual cases to code", "affected manual tests", "selective regression for manual tests", "generate coverage.manual.yml". - ---- - -## CRITICAL CONSTRAINTS - -This skill works **only with manual tests in markdown format**. - -- **DO NOT** process unit, functional, or e2e test files. -- **DO NOT** suggest creating new automated tests. -- **Only touch two files in this repo.** This skill runs inside the user's source-code repo. It may write `coverage.manual.yml` (or the path the user gave) and add one `.testclaw/` line to `.gitignore` if it is missing. Nothing else — never a source file. Cases pulled from Testomat.io go into the gitignored `.testclaw/manual-tests/`, never into a tracked folder (see Step 1). -- **Don't write scripts. Never use Python.** Read `.test.md` files with your file tool; pull out IDs and tags with `grep` (Step 2). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper: `npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs` (Step 5). If more checks needed use custom script with `npx js-yaml`. - -If automated test files (e.g. e2e test, unit, api) are encountered while exploring, ignore them and continue with the manual markdown set. - ---- - -## Workflow: Build Manual Coverage Map - -### Step 1: Discover the project (delegate to `scan-automation-project`) - -Run the **`scan-automation-project`** skill to inventory the project. Use its output as the source of truth for both the manual test set and the high-level codebase overview — do not duplicate that scan here. - -From the `scan-automation-project` result, capture: -- **Manual Tests** — the list of `.test.md` files and their suite/test titles (the input to Step 2). -- **Project Overview** — languages, frameworks, complexity (the framing for Step 3). - - -If `scan-automation-project` reports **no manual tests** (it checks the cache too, so this means nothing was pulled before): -- ❓ Ask the user how to proceed: - 1. Pull cases from Testomat.io — have **`sync-test-cases-with-tms`** pull into the gitignored cache: `npx check-tests pull -d .testclaw/manual-tests`, then add `.testclaw/` to `.gitignore` if missing. Re-run `scan-automation-project` and continue. - 2. Point to a directory the scan missed (then re-run `scan-automation-project` there). - 3. Stop. - -Never pull cases into a tracked folder. Don't repeat `sync-test-cases-with-tms` pull logic here. - -### Step 2: Extract test information - -Each manual test markdown file follows the format described in [Classical Tests Markdown Format](../qa-write-test-cases/references/test-case-format.md) (canonical reference, owned by `qa-write-test-cases`). For every file extract: - -- **Suite ID** — `@S` + 8 chars, found in the `` block. -- **Test IDs** — `@T` + 8 chars, found in `` blocks. -- **Tags** — `@tag` markers in suite/test titles and `tags:` lines inside the metadata blocks. -- **Context** — suite title, test titles, steps, and expected results — used to reason about which source files implement each behavior. - -**How to extract.** Read the `.test.md` files — the metadata blocks are short. For a quick overview, `grep` instead of a parser. Run these in whichever directory holds the cases (`manual-tests/`, or the cache `.testclaw/manual-tests/`): - -```bash -grep -rnE 'id:[[:space:]]*@S' # suite IDs (+ which file) -grep -rnE 'id:[[:space:]]*@T' # test IDs -grep -rnE '^tags:' # tags from metadata blocks -grep -rhoE '@[A-Za-z0-9_-]+' | sort -u # every @token (titles included) -``` - -Don't write a markdown parser. Never use `python`. - -If a file has no `@S` / `@T` IDs, the user hasn't pushed it to Testomat.io yet. Ask whether to push first via `sync-test-cases-with-tms`, or skip those files — a mapping without IDs is useless to the reporter. - -### Step 3: Explore the codebase - -Use the **Project Overview** from Step 1's `scan-automation-project` result (languages, frameworks, complexity) as the starting frame, then identify business code that implements the behaviors described by the manual tests. Skip everything not part of the application — `scan-automation-project` already excludes most of this, but reinforce when reading source: - -- Test code: `*.test.*`, `*.spec.*`, `*_test.*`, `test_*.*`, `*.cy.*`, `__tests__/`, `__specs__/`, `tests/`, `spec/`, `specs/`. -- Markdown manual test directories themselves. -- Dependency / build / vendor folders. -- Framework configs and lock files. - -Work with the project structure (controllers, models, services, components, pages, routes, etc.) — not the testing infrastructure. - -**Templates and views are source code — map them too.** A manual case walks through the rendered UI, so a change to a template alters the screen the tester checks. Treat view/template files as first-class mappable source alongside controllers, models, and components — do **not** skip them because they aren't `.js`/`.ts`/`.py`. Cover at least: - -- HTML & component templates: `.html`, `.htm`, `.vue`, `.svelte`, Angular `*.component.html`. -- Logic-in-markup engines: `.hbs`/`.handlebars`, `.ejs`, `.pug`/`.jade`, `.mustache`, `.liquid`. -- Server-side views: `.erb`, `.haml`, `.slim` (Rails); `.blade.php`, `.twig` (PHP); `.j2`/`.jinja`/`.jinja2` (Python); `.cshtml`/`.razor` (.NET); `.jsp` (Java). - -Map a template the same way as code: to the suite/test/tag whose manual cases exercise that screen (e.g. `app/views/sessions/new.html.erb` → the login suite). - -❓ If the project structure is ambiguous or large (`scan-automation-project` reports `large` / `very-large` complexity), ask the user which directories to focus on or to exclude. - -### Step 4: Choose the best mapping per source file - -For each candidate source file, pick **one** mapping strategy based on which option gives the cleanest, most stable selection: - -**A) Map to Suite (`@S...`)** — when most tests in a suite relate to the file. - -```yaml -app/models/user.rb: - - "@S816410d6" # Suite: User permissions -``` - -**B) Map to Test (`@T...`)** — when only one specific test matches the file. - -```yaml -app/controllers/sessions_controller.rb: - - "@T6f8e9174" # Test: User is blocked after 5 failed login attempts -``` - -**C) Map to Tag (`@tag`)** — when a tag groups tests across multiple suites that all relate to the file. - -```yaml -app/services/jira_service.rb: - - "@jira" # Tag: All JIRA integration manual tests -``` - -**Decision guidance:** -- Prefer Suite mapping over listing many individual Test IDs from the same suite. -- Prefer Test mapping when only one test in a large suite is relevant. -- Prefer Tag mapping when the relevant tests live across several suites. -- Globs (`app/services/jira/**`) are valid file keys when a whole subtree maps to the same identifiers. -- Avoid empty entries. -- Add `#` comments next to each identifier explaining the mapping. - -See [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md) for the full YAML grammar. - -### Step 5: Save and validate the coverage file - -Write the YAML to the resolved output path (default `coverage.manual.yml`). If the user supplied a different path => use it. -(keep `#` comments next to each ID so future readers can audit the mapping without opening Testomat.io). - -**Check it** — one command, run from the project root: - -```bash -npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs -``` - -`npx js-yaml` parses the file (and fails loudly if the YAML is malformed, so a broken file never reaches the script). `check-coverage.mjs` then flags any key whose path is missing on disk, any key with no identifiers, and prints every `@S…` / `@T…` / tag it references. Check that list against the IDs you extracted in Step 2 — only you know which are real. Don't re-parse the markdown; use the set you already have. Never use `python`. - -> The keys in the coverage file are paths to source files in this repo, never `.testclaw/...` paths. The cache holds the test cases; the coverage file points at the code they cover. - -Then show the produced YAML to the user. - -### Step 6: Show next steps - -Tell the user how to use the file with `@testomatio/reporter`: - -```bash -# Create a pending manual run that includes only cases affected by the diff vs main -npx @testomatio/reporter run \ - --kind manual \ - --filter "coverage:file=coverage.manual.yml,diff=main" -``` - -For batched regression cycles, recommend grouping runs: - -```bash -TESTOMATIO_RUNGROUP="Regression 911" npx @testomatio/reporter run \ - --kind manual \ - --filter "coverage:file=coverage.manual.yml,diff=main" -``` - -Recommend committing `coverage.manual.yml` to the repository so CI and teammates use the same mapping. - -If you pulled the cases, tell the user they stay in the gitignored `.testclaw/manual-tests/`. A re-run or a follow-up question reuses them, and nothing was committed. Don't delete the cache or move it into a tracked folder. - -### Step 7: Suggest follow-ups - -Once the file is saved, propose any of: - -- Scan the source for **coverage gaps** (features without manual tests). On approval, propose new manual cases (delegate to `qa-write-test-cases`). -- Scan for **dead tests** (manual tests whose features no longer exist in source). -- Answer questions like "do we have manual tests for X?" from the cached cases in `.testclaw/manual-tests/`. -- If the user wants to *edit* cases, they can edit them right in `.testclaw/manual-tests/` and push back with `sync-test-cases-with-tms` — gitignored doesn't mean read-only. - ---- - -## References - -| Description | File | -| ---------------------------- | ---------------------------------------------------------- | -| Coverage YAML format | ./references/COVERAGE_FILE_FORMAT.md | -| Manual test markdown format | ../qa-write-test-cases/references/test-case-format.md | - -## Bundled script - -`scripts/check-coverage.mjs` (~25 lines, zero deps) — reads the parsed coverage map on stdin, flags keys whose path is missing on disk and keys with no identifiers, and lists every `@S`/`@T`/tag the file references. Feed it via `js-yaml`, from the project root: - -```bash -npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs -``` - -It's the only script — everything else is `grep`, your file tool, or `npx js-yaml`. Don't rewrite it in `python`. - ---- - -## Error Handling - -### Recovery - -- **No manual tests found (cache included)** → have `sync-test-cases-with-tms` pull into `.testclaw/manual-tests/`, or ask the user for a directory the scan missed. -- **Markdown files with no `@S`/`@T` IDs** → ask whether to push first via `sync-test-cases-with-tms`, or skip those files. -- **Ambiguous source layout** → ask the user which directories are application code. - -### Hard Fail (stop immediately) - -- Cannot create or write the output file. -- User refuses to provide a tests directory or to pull from Testomat.io. -- The agent is asked to modify files other than the output file (refuse — see CRITICAL CONSTRAINTS). - ---- - -## Examples - -**Build the mapping in a source repo (cases pulled if needed):** -``` -Use qa-manual-tests-to-code-coverage skill to build coverage.manual.yml for our manual cases -``` -If there are no local `.test.md` files, it pulls them into the gitignored `.testclaw/manual-tests/` and works from there. - -**Cases already local:** -``` -Use qa-manual-tests-to-code-coverage skill for the cases in manual-tests/ -``` - -**With a custom output path:** -``` -Use qa-manual-tests-to-code-coverage skill, output to ops/coverage.qa.yml -``` - -**Full workflow (source repo):** -1. `qa-manual-tests-to-code-coverage` runs `scan-automation-project`. No local cases, so it has `sync-test-cases-with-tms` pull into `.testclaw/manual-tests/` (gitignored) and re-runs `scan-automation-project`. -2. It maps source files to suite/test/tag IDs and writes `coverage.manual.yml`. The only tracked changes are that file and one line in `.gitignore`. -3. `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` creates a pending run with only the affected cases. - ---- - -## Quick Commands - -| Action | Command | -| -------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| Pull cases into the gitignored cache | `npx check-tests pull -d .testclaw/manual-tests` | -| Create affected manual run | `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` | -| Group runs | `TESTOMATIO_RUNGROUP="Regression 911" npx @testomatio/reporter run --kind manual --filter "..."` | diff --git a/skills/qa-test-coverage-map/SKILL.md b/skills/qa-test-coverage-map/SKILL.md new file mode 100644 index 0000000..f6c2b0e --- /dev/null +++ b/skills/qa-test-coverage-map/SKILL.md @@ -0,0 +1,208 @@ +--- +name: qa-test-coverage-map +description: Map tests to the source code they cover and generate a per-project coverage map (`coverage..yml`) consumed by `@testomatio/reporter --filter "coverage:..."`. Handles manual markdown test cases and automated e2e tests (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium) in one file. Use this skill when the user wants to run only the tests affected by a code change, generate a coverage file (any coverage*.yml), build a code-to-test traceability matrix, or set up change-aware regression — for manual QA, automated suites, or both. Replaces the separate qa-manual-tests-to-code-coverage and e2e-test-coverage-mapping skills. +license: MIT +metadata: + author: Testomat.io + version: 2.0.0 +--- + +# QA-TEST-COVERAGE-MAP SKILL: What I do + +I analyze a project's tests — manual cases in markdown, automated e2e tests, or +both — and produce a **coverage map**: YAML mapping source files (or globs) to +the Testomat.io identifiers of the tests that cover them (`@S` suite IDs, `@T` +test IDs, `@tag`s). `@testomatio/reporter` consumes the map to count, create, +or run only the tests affected by a git diff: + +```bash +npx @testomatio/reporter run --filter "coverage:file=,diff=main" ... +``` + +Manual and automated tests living in one project go into **one map**. Files are +split per *project*, never per test kind. + +## File naming is the contract + +One file per Testomat.io project, in the repo root. The name tells every +consumer — CI, the `setup-pr-testing` skill — what kinds of tests are inside +without opening the file: + +| Project's tests | File | +| ------------------ | ----------------------------------- | +| manual + automated | `coverage..yml` | +| manual only | `coverage..manual.yml` | +| automated e2e only | `coverage..e2e.yml` | + +`` is a kebab-case identifier of the Testomat.io project (e.g. +`billing-app`). Most repos serve one project and get one file; a monorepo or a +split manual/e2e setup may serve several projects — one file each. Legacy +`coverage.manual.yml` / `coverage.e2e.yml` files (no slug) are single-kind maps +from the old scheme; offer to migrate them when you touch them. + +## Constraints + +- **Write only the coverage file(s)**, plus one `.testclaw/` line in + `.gitignore` if missing. Never modify source or test files. +- **Pulled or cloned material lives in the gitignored `.testclaw/`**: + manual cases pulled from Testomat.io → `.testclaw/manual-tests/`, an + external e2e repo → `.testclaw/e2e-tests/`. Never into a tracked + folder. Tests already in the repo are used where they are. +- **Only tests carrying Testomat.io IDs can be mapped.** A map without real + `@S`/`@T` IDs is useless to the reporter. +- **Unit and integration tests are out of scope** — map manual cases and e2e + tests only. +- **No ad-hoc scripts, never Python.** File reads, `grep`, `npx js-yaml`, and + the bundled `scripts/check-coverage.mjs` cover everything this skill needs. + +## Workflow + +### 1. Discover (delegate to `scan-automation-project`) + +Run **`scan-automation-project`**. It reports which test kinds exist (manual `.test.md` +files, automated e2e frameworks), the languages, and the project shape — that +determines the file name(s) and everything after. + +If a kind the user expects is missing locally: + +- manual cases → have `sync-test-cases-with-tms` pull them: + `npx check-tests pull -d .testclaw/manual-tests` +- e2e tests in another repo → `git clone .testclaw/e2e-tests` +- otherwise ask for the directory, or stop. + +If several automated frameworks are detected, ask which one is the e2e suite. + +### 2. Pick the file name(s) + +Confirm which Testomat.io project the tests belong to and its slug (ask if you +cannot tell). Choose the file name from the table above based on what Step 1 +found. Multiple projects → split the tests by project and build one map each. +If the user supplied an output path, use it. + +### 3. Extract test identifiers + +- **Manual cases** carry IDs in metadata comments — + ``, ``, plus `tags:` + lines ([format reference](../qa-write-test-cases/references/test-case-format.md)). +- **Automated tests** carry IDs in test titles — + `describe('user settings @S92321384')`, `it('updates avatar @Ta011dfa3')` + (per-framework syntax: [E2E_FRAMEWORKS.md](./references/E2E_FRAMEWORKS.md)). + +`grep` is enough — don't write a parser: + +```bash +grep -rnoE '@[ST][0-9a-f]{8}' # suite/test IDs +grep -rhoE '@[A-Za-z0-9_-]+' | sort -u # every @token, tags included +``` + +Also read the tests themselves — titles, steps, page objects, routes — to +understand which source files each one exercises. + +**Missing IDs?** The tests haven't been synced with Testomat.io yet. Manual +cases: push via `sync-test-cases-with-tms` first. Automated tests: +`npx check-tests@latest "" --update-ids` (see +`qa-e2e-tests-reporting`). Don't map tests without IDs. + +### 4. Map source files to identifiers + +Explore the business code — controllers, models, services, components, routes. +Skip test code, dependencies, build output, and framework configs. + +**Templates and views are source too** (`.html`, `.vue`, `.svelte`, `.erb`, +`.blade.php`, `.twig`, `.cshtml`, …): a template change alters the screen a +test checks, so map them like any other file. + +Per source file or glob, pick the identifier that gives the most stable +selection: + +- **Suite `@S...`** — most tests of a suite relate to the file (preferred, + survives test additions). +- **Test `@T...`** — only one test in a large suite is relevant. +- **Tag `@tag`** — the related tests are spread across suites. + +In a mixed project, manual and automated identifiers sit side by side under +the same key: + +```yaml +app/controllers/sessions_controller.rb: + - "@S816410d6" # Suite: Login (manual) + - "@Ta011dfa3" # Test: login with valid credentials (e2e) +``` + +Use globs (`app/services/jira/**`) for whole subtrees, annotate every +identifier with a `#` comment, avoid empty entries. Full grammar: +[COVERAGE_FILE_FORMAT.md](./references/COVERAGE_FILE_FORMAT.md). + +If the codebase is large or ambiguous, ask which directories to focus on. + +### 5. Validate + +```bash +npx js-yaml coverage..yml | node scripts/check-coverage.mjs +``` + +`js-yaml` fails loudly on malformed YAML; the checker flags keys whose path is +missing on disk and keys with no identifiers, and lists every referenced ID — +cross-check that list against the set you extracted in Step 3. Map keys are +repo paths, never `.testclaw/...` paths. Show the final YAML to the +user. + +### 6. Hand off + +Show how the map is used — the suffix decides the `--kind` flag: + +```bash +# manual-only map → pending manual run with the affected cases +npx @testomatio/reporter run --kind manual \ + --filter "coverage:file=coverage..manual.yml,diff=main" + +# mixed map → one run holding affected manual cases and automated tests +npx @testomatio/reporter run --kind mixed \ + --filter "coverage:file=coverage..yml,diff=main" + +# automated-only map → wrap the runner, no --kind needed +npx @testomatio/reporter run "npx playwright test" \ + --filter "coverage:file=coverage..e2e.yml,diff=main" +``` + +Recommend committing the map so CI and teammates share one mapping, and +**`setup-pr-testing`** to wire it into CI (PR comments with affected counts, +post-merge regression runs). If cases were pulled, they stay in the gitignored +cache for reuse — don't delete or commit them. + +Useful follow-ups: scan for coverage gaps (source with no tests → delegate to +`qa-write-test-cases`) or dead tests (tests whose feature no longer exists). + +## References + +| Description | File | +| --------------------------- | ------------------------------------------------ | +| Coverage YAML format | ./references/COVERAGE_FILE_FORMAT.md | +| E2E framework detection | ./references/E2E_FRAMEWORKS.md | +| Manual test markdown format | ../qa-write-test-cases/references/test-case-format.md | + +## Bundled script + +`scripts/check-coverage.mjs` (~25 lines, zero deps) — sanity-checks a parsed +map from stdin. It is the only script; everything else is `grep`, file tools, +or `npx js-yaml`. Don't rewrite it, don't add others, never use `python`. + +## Examples + +**Mixed project:** "Build a coverage map for our manual cases and Playwright +tests" → one `coverage.billing-app.yml` mapping source files to both kinds of +identifiers. + +**Manual-only, nothing local:** cases are pulled into +`.testclaw/manual-tests/`, mapped, and written to +`coverage.billing-app.manual.yml`. Tracked changes: that file and one +`.gitignore` line. + +**Two projects in a monorepo:** `apps/shop` and `apps/admin` sync to different +Testomat.io projects → `coverage.shop.yml` + `coverage.admin.yml`. + +## Related skills + +`scan-automation-project` (mandatory first), `sync-test-cases-with-tms` (pull/push manual cases), +`qa-e2e-tests-reporting` (reporter install + `--update-ids`), `qa-write-test-cases` (author +missing cases), `setup-pr-testing` (consume the map in CI). diff --git a/skills/qa-manual-tests-to-code-coverage/references/COVERAGE_FILE_FORMAT.md b/skills/qa-test-coverage-map/references/COVERAGE_FILE_FORMAT.md similarity index 54% rename from skills/qa-manual-tests-to-code-coverage/references/COVERAGE_FILE_FORMAT.md rename to skills/qa-test-coverage-map/references/COVERAGE_FILE_FORMAT.md index 4e321b8..9f1ca43 100644 --- a/skills/qa-manual-tests-to-code-coverage/references/COVERAGE_FILE_FORMAT.md +++ b/skills/qa-test-coverage-map/references/COVERAGE_FILE_FORMAT.md @@ -1,10 +1,14 @@ # Coverage File Format -`coverage.manual.yml` and `coverage.e2e.yml` share the same grammar. Both are read by `@testomatio/reporter run --filter "coverage:file=,diff="`. +All coverage maps share one grammar, whatever their name — +`coverage..yml` (manual + automated), `coverage..manual.yml` +(manual only), `coverage..e2e.yml` (automated only). All are read by +`@testomatio/reporter --filter "coverage:file=,diff="`. ## Top-level shape -A YAML map. Keys are file paths or globs relative to the repository root. Values are lists of identifiers — Suite IDs, Test IDs, or tags. +A YAML map. Keys are file paths or globs relative to the repository root. +Values are lists of identifiers — Suite IDs, Test IDs, or tags. ```yaml : @@ -29,7 +33,13 @@ describe('user settings @S92321384', () => { }); ``` -They are populated by `npx check-tests "" --update-ids` after the tests are imported into Testomat.io. +They are populated by `npx check-tests "" --update-ids` after +the tests are imported into Testomat.io. + +For manual cases, IDs live in the markdown metadata blocks +(``, ``). In a mixed map +both kinds sit under the same key — the reporter doesn't care where an ID came +from. ## File keys @@ -62,23 +72,27 @@ app/models/user.rb: ## Reporter usage -The runner command **must** be the first positional argument — `--filter` generates a `--grep` that the runner consumes. +The map's suffix decides the `--kind` flag: ```bash -# Playwright -npx @testomatio/reporter run "npx playwright test" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" +# manual-only map — create a pending manual run +npx @testomatio/reporter run --kind manual \ + --filter "coverage:file=coverage..manual.yml,diff=main" -# Cypress -npx @testomatio/reporter run "npx cypress run" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" +# mixed map — one run holding manual cases and automated tests +npx @testomatio/reporter run --kind mixed \ + --filter "coverage:file=coverage..yml,diff=main" -# WebdriverIO / Mocha / Jest / CodeceptJS — pass the corresponding runner command -npx @testomatio/reporter run "npx codeceptjs run" \ - --filter "coverage:file=coverage.e2e.yml,diff=main" +# automated-only map — wrap the runner, no --kind +# (--filter generates a --grep that the runner consumes) +npx @testomatio/reporter run "npx playwright test" \ + --filter "coverage:file=coverage..e2e.yml,diff=main" ``` -The `diff` value must be a stable branch (`main`, `origin/main`) the reporter can run `git diff` against. +For automated execution, the runner command (`npx playwright test`, +`npx cypress run`, `npx codeceptjs run`, `npx wdio`, …) is the first positional +argument. The `diff` value must be a stable ref (`main`, `origin/main`) the +reporter can run `git diff` against. ## GitHub Actions example @@ -104,7 +118,7 @@ jobs: TESTOMATIO: ${{ secrets.TESTOMATIO }} run: | npx @testomatio/reporter run "npx playwright test" \ - --filter "coverage:file=coverage.e2e.yml,diff=origin/main" + --filter "coverage:file=coverage.shop.e2e.yml,diff=origin/main" ``` ## Checking the coverage file @@ -112,15 +126,22 @@ jobs: One command, run from the project root — no parser of your own: ```bash -npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs # or coverage.e2e.yml +npx js-yaml coverage..yml | node scripts/check-coverage.mjs ``` -`npx js-yaml` parses the YAML (it fails loudly on a malformed file, so a broken one never reaches the script). `scripts/check-coverage.mjs` (~25 lines, zero deps; ships with both `qa-manual-tests-to-code-coverage` and `e2e-test-coverage-mapping`, the latter as a symlink) reads that parsed map on stdin, flags any key whose path is missing on disk, flags any key with no identifiers, lists every `@S…` / `@T…` / tag the file references, and exits non-zero on a problem. +`npx js-yaml` parses the YAML (it fails loudly on a malformed file, so a broken +one never reaches the script). `scripts/check-coverage.mjs` (~25 lines, zero +deps; ships with `qa-test-coverage-map`) reads that parsed map on stdin, flags any key +whose path is missing on disk, flags any key with no identifiers, lists every +`@S…` / `@T…` / tag the file references, and exits non-zero on a problem. -The script can't tell which identifiers are real — check the listed ones against the set you extracted earlier in the workflow. Don't re-parse the test set, don't write your own YAML parser, and never use `python`. +The script can't tell which identifiers are real — check the listed ones +against the set you extracted earlier in the workflow. Don't re-parse the test +set, don't write your own YAML parser, and never use `python`. ## Authoring tips +- One file per Testomat.io project; the name says what's inside. - Prefer Suite IDs when most of a suite relates to a file. - Use Test IDs only when one test in a large suite is relevant. - Use tags (`@smoke`, `@billing`, `@jira-…`) for cross-cutting concerns. diff --git a/skills/e2e-test-coverage-mapping/references/E2E_FRAMEWORKS.md b/skills/qa-test-coverage-map/references/E2E_FRAMEWORKS.md similarity index 89% rename from skills/e2e-test-coverage-mapping/references/E2E_FRAMEWORKS.md rename to skills/qa-test-coverage-map/references/E2E_FRAMEWORKS.md index c9869dc..7f307fd 100644 --- a/skills/e2e-test-coverage-mapping/references/E2E_FRAMEWORKS.md +++ b/skills/qa-test-coverage-map/references/E2E_FRAMEWORKS.md @@ -57,9 +57,9 @@ npx check-tests@latest CodeceptJS "**/*_test.js" --update-ids npx check-tests@latest WebdriverIO "**/*.{test,e2e}.js" --update-ids ``` -`check-tests` rewrites the test files in place, inserting the IDs assigned by Testomat.io. Commit the changes before running `e2e-test-coverage-mapping`. +`check-tests` rewrites the test files in place, inserting the IDs assigned by Testomat.io. Commit the changes before running `qa-test-coverage-map`. ## Related skills - `qa-e2e-tests-reporting` — install `@testomatio/reporter` and import tests via `check-tests`. -- `sync-test-cases-with-tms` — pull/push manual cases (the qa-manual-tests-to-code-coverage counterpart). See its [Testomat.io CLI reference](../../sync-test-cases-with-tms/references/TESTOMATIO_CLI.md) for the full `check-tests` command set, including `--update-ids`. +- `sync-test-cases-with-tms` — pull/push manual cases. See its [Testomat.io CLI reference](../../sync-test-cases-with-tms/references/TESTOMATIO_CLI.md) for the full `check-tests` command set, including `--update-ids`. diff --git a/skills/qa-manual-tests-to-code-coverage/scripts/check-coverage.mjs b/skills/qa-test-coverage-map/scripts/check-coverage.mjs similarity index 91% rename from skills/qa-manual-tests-to-code-coverage/scripts/check-coverage.mjs rename to skills/qa-test-coverage-map/scripts/check-coverage.mjs index 0d9134c..267923b 100755 --- a/skills/qa-manual-tests-to-code-coverage/scripts/check-coverage.mjs +++ b/skills/qa-test-coverage-map/scripts/check-coverage.mjs @@ -1,10 +1,10 @@ #!/usr/bin/env node -// Sanity-check a coverage.*.yml mapping (manual or e2e — same format). +// Sanity-check a coverage*.yml mapping (manual, e2e, or mixed — same format). // // `js-yaml` does the YAML parsing; this script just checks the result. // Run it from the project root and pipe the parsed file in: // -// npx js-yaml coverage.manual.yml | node check-coverage.mjs +// npx js-yaml coverage..yml | node check-coverage.mjs // // (`npx js-yaml` prints JSON, and fails loudly if the YAML is malformed, // so a broken file never reaches this script.) diff --git a/skills/senior-qa-testing-workflow/SKILL.md b/skills/senior-qa-testing-workflow/SKILL.md index cefd3a3..d405bcd 100644 --- a/skills/senior-qa-testing-workflow/SKILL.md +++ b/skills/senior-qa-testing-workflow/SKILL.md @@ -39,8 +39,7 @@ The skill orchestrates these specialized capabilities: | **improve-test-cases** | Improve existing test cases quality | | **sync-test-cases-with-tms** | Upload test cases to Testomat.io TMS | | **qa-e2e-tests-reporting** | Add Testomat.io reporter to your automation project | -| **qa-manual-tests-to-code-coverage** | Map manual test cases to source files (`coverage.manual.yml`) | -| **e2e-test-coverage-mapping** | Map automated e2e tests to source files (`coverage.e2e.yml`) | +| **qa-test-coverage-map** | Map manual and automated tests to source files (per-project `coverage.*.yml`) | | **scan-automation-project** | Scan project source code to inventory languages, frameworks, and existing tests | diff --git a/skills/setup-pr-testing/SKILL.md b/skills/setup-pr-testing/SKILL.md new file mode 100644 index 0000000..73159ba --- /dev/null +++ b/skills/setup-pr-testing/SKILL.md @@ -0,0 +1,294 @@ +--- +name: setup-pr-testing +description: Set up change-aware PR regression testing — wire CI so each pull request posts a comment listing how many manual and automated tests its diff affects, and so that after the PR is merged and deployed only those affected tests are actually run, via Testomat.io coverage maps and @testomatio/reporter. Use this skill when the user wants PR-triggered regression, "comment affected test counts on PRs", "run only affected tests after merge/deploy", selective manual runs per PR, post-deploy e2e, a coverage-driven CI pipeline, weekly/grouped regression runs in a rungroup, launching automated regression on Testomat.io CI via `--remote`, or to connect coverage*.yml maps to their CI. CI-agnostic — adapts to whatever CI the project already uses (GitHub Actions, GitLab CI, Jenkins, Bitbucket, CircleCI, etc.). Trigger it even if the user only says "make PRs comment affected tests" or "run the right tests after merge". +license: MIT +metadata: + author: Testomat.io + version: 3.0.0 +--- + +# SETUP-PR-TESTING SKILL: What I do + +I wire a project's CI so a pull request's change drives testing in **three +phases** — a cheap notice while the PR is open, runs created once it is merged, +execution once it is deployed. Nothing heavy happens until the change is +actually going somewhere. + +> **GOAL: a working pipeline inside the project's own CI system.** That +> committed CI configuration is the one and only finished result. +> **I run locally to author it — I am never part of CI.** Do not run tests, +> create runs, or call `@testomatio/reporter` to "see it work"; every command +> below is something the pipeline executes later. + +``` +PR opened ──▶ NOTICE ONLY — comment the affected test counts on the PR + reporter run --filter-list per coverage map → counts + posted via the CI's native PR-comment API + no runs created · never blocks the PR + +PR merged ──▶ One regression run per coverage map (created, NOT executed) + coverage..manual.yml → reporter start --kind manual + coverage..yml → reporter start --kind mixed (shared run) + coverage..e2e.yml → reporter start (shared run) + titled by the merge commit · in a rungroup + manual cases sit pending for testers · automated tests wait for deploy + +deploy done ──▶ Launch automated execution on Testomat.io CI + reporter run --remote (runs containing automated tests) + TESTOMATIO_RUN=$RUN_ID · TESTOMATIO_SHARED_RUN=1 · same shared title +``` + +## Coverage maps drive everything + +A coverage map maps source files → test identifiers; the reporter filters it by +the PR diff so only impacted tests are counted, prepared, and run. **One map +per Testomat.io project, named by content** — created by the `qa-test-coverage-map` +skill, never hand-written here: + +| Map | Contains | On merge | After deploy | +| ---------------------------- | ------------------ | ----------------------------------------------------------- | ------------------------------------- | +| `coverage..manual.yml` | manual only | `start --kind manual` — run is complete, testers pick it up | nothing | +| `coverage..yml` | manual + automated | `start --kind mixed`, prepared as a shared run | launch automated tests via `--remote` | +| `coverage..e2e.yml` | automated only | `start` (no `--kind`), prepared as a shared run | launch via `--remote` | + +The suffix is the whole contract: it tells CI which `--kind` to pass and +whether a deploy-time execute phase exists. Legacy `coverage.manual.yml` / +`coverage.e2e.yml` (no slug) mean manual-only / automated-only. A repo serving +several Testomat.io projects has several maps — repeat the per-map commands +for each. + +## Method, not snippets + +The valuable knowledge here is the three-phase model, the reporter command +contract ([references/REPORTER_CONTRACT.md](references/REPORTER_CONTRACT.md)), +and the decisions to confirm with the user. Translating a trigger into a +specific CI's YAML/Groovy is not — you already know how every CI expresses "on +PR open", "on merge", "after deploy", "post a PR comment", and "don't fail the +pipeline". Write that config yourself for the CI in front of you; never bake +per-CI workflow files into this skill. + +## When to use + +- "Comment how many tests a PR affects." +- "Run only the affected tests after a PR is merged and deployed." +- "Create a pending manual regression run per merged PR." +- "Launch affected e2e on Testomat.io CI after deploy" / "use `--remote`". +- "Hook our coverage*.yml into CI." + +--- + +## CRITICAL CONSTRAINTS + +- **The deliverable is committed CI config — never execute the reporter + yourself.** +- **Discovery first.** Delegate to `scan-automation-project` before writing anything. +- **Never assume or hardcode the CI system.** Read the repo; if unclear, ask. +- **No coverage map → no regression.** Missing maps are created via the + `qa-test-coverage-map` skill, never hand-written; filtering is never skipped. +- **PR open is a notice only.** Counts in a comment; no runs; never blocks the + PR. +- **Runs are created on merge, executed after deploy.** Manual-only runs are + complete at creation; runs containing automated tests are prepared as shared + runs (`TESTOMATIO_SHARED_RUN=1`, title from the merge commit, + `TESTOMATIO_SHARED_RUN_TIMEOUT` above the merge→deploy gap — the 20-minute + default is usually too short). +- **Automated execution goes through `reporter run --remote `** — a + Testomat.io CI profile (Settings → CI), never by reaching into another + repo's pipeline. The prepare and execute steps MUST share the same + `TESTOMATIO_TITLE` so they converge on one run. +- **The execute step never fails the deploy/release pipeline.** It is + observation, not a gate — isolate it as a non-failing job. +- **Every run gets a meaningful title and a rungroup.** `TESTOMATIO_TITLE` + from the merge commit; `TESTOMATIO_RUNGROUP_TITLE` per the user's grouping + strategy. +- **Only touch CI config** (and coverage maps if delegated). Never source or + test files. + +--- + +## Workflow + +### Step 1 — Discover (delegate to `scan-automation-project`) + +Capture: is there an e2e framework (unit/integration don't count), are there +manual `.test.md` cases, do automated tests live here or elsewhere. This tells +you which maps to expect and which phases apply. + +### Step 2 — Identify the CI + +Read the repo's CI config files. Several CIs or none → ask which one runs PRs. + +### Step 3 — Locate the coverage maps + +Look for `coverage*.yml` in the repo root. The filename tells you what each +map contains (table above). Validate that a map still resolves before wiring +CI to it (the `qa-test-coverage-map` skill bundles a checker). If the map for a test +kind the user wants is missing → propose creating it and delegate to +**`qa-test-coverage-map`**; the phases for that kind cannot be wired without it. + +### Step 4 — Ask the unknowns + +Read the CI files first so you don't ask what's already answered: + +1. **What triggers a deploy** that should launch automated execution — merge + to main, release event, tag, push to a deploy branch, or manual? +2. **How does the deploy signal completion?** A concrete observable event — + a job finishing, a pipeline event, a status check, a health endpoint. +3. **How long is merge → deploy-complete?** Sets + `TESTOMATIO_SHARED_RUN_TIMEOUT` comfortably above it. +4. **Rungroup strategy** — week / day / milestone / release / submodule. +5. **Diff base** — the PR's target branch is the natural base; for post-deploy + ranges see the caveat in REPORTER_CONTRACT.md. + +The PR comment needs only the maps and the base (Q5). Manual-only runs need +only the rungroup (Q4). Q1–Q3 matter only when a map contains automated tests. + +### Step 5 — Confirm how automated execution launches + +`reporter run --remote ` dispatches a **Testomat.io CI profile** +(Settings → CI) that owns the runner, browsers, environment URLs, and secrets — +whether the e2e suite lives in this repo or a dedicated one. No profile yet → +that is a prerequisite for the user; wire the execute step so it can be enabled +once the profile exists. + +**Run inline instead of `--remote`** when the suite must run in this pipeline: +mobile (simulators, signing), API tests against a server this repo spins up, or +an existing same-repo e2e job that already works. The execute step then wraps +the runner directly on the prepared run. + +**No e2e suite anywhere** → wire only the comment and manual phases; explain +the automated phase needs a suite first. Never fabricate an e2e job. + +### Step 6 — Wire the phases into the CI + +Write the jobs in the CI's own syntax. The skill-specific parts are the +reporter commands and env vars; triggers, secrets, and the PR-comment call are +ordinary CI config. + +**(a) PR opened → counts comment (notice only).** + +For each map, a `--filter-list` dry run lists the matching IDs without running +or creating anything: + +```bash +npx @testomatio/reporter run \ + --filter-list "coverage:file=,diff=" --format ids +``` + +Count the IDs (empty output = 0) and post one comment via the CI's native +PR/MR comment API, e.g. `0 automated tests, 10 manual tests are affected by +this PR`. Per-kind numbers come from `.manual`/`.e2e` maps; a mixed map yields +one combined count (`12 tests (manual + automated)`). Update one existing +comment rather than re-posting. Never fail the PR check. + +**(b) PR merged → create one run per map.** + +Required env on every call: `TESTOMATIO`, `TESTOMATIO_TITLE` (merge commit, +e.g. `report for commit `), `TESTOMATIO_RUNGROUP_TITLE`. + +Manual-only map — created pending, done: + +```bash +npx @testomatio/reporter start --kind manual \ + --filter "coverage:file=coverage..manual.yml,diff=" +``` + +Map containing automated tests — prepared as a shared run, **not executed**. +Mixed maps add `--kind mixed`; e2e-only maps take no `--kind`: + +```bash +RUN_ID=$(TESTOMATIO_SHARED_RUN=1 \ +TESTOMATIO_TITLE="report for commit " \ +TESTOMATIO_SHARED_RUN_TIMEOUT= \ +npx @testomatio/reporter start --kind mixed \ + --filter "coverage:file=coverage..yml,diff=" --format id) +``` + +`--format id` prints only the run id, so `RUN_ID` captures it cleanly — carry +it to the execute step (pipeline output/artifact). In a mixed run the manual +cases are immediately pending for testers; the automated part waits for +deploy. + +**(c) Deploy done → launch automated execution.** + +For each run prepared in (b), triggered by the deploy-completion signal — +never by PR open: + +```bash +TESTOMATIO_RUN=$RUN_ID \ +TESTOMATIO_SHARED_RUN=1 \ +TESTOMATIO_TITLE="report for commit " \ +npx @testomatio/reporter run --remote +``` + +With no fresh `--filter`, Testomat.io reuses the run's stored scope — only the +affected automated tests run on the CI profile and report back into the same +run. If the deploy pipeline can't carry `RUN_ID`, the identical shared title +(within the timeout) matches the prepared run instead. Keep this job +non-failing and off the release's critical path. + +Inline exception (Step 5) — wrap the runner instead of `--remote`: + +```bash +TESTOMATIO_RUN=$RUN_ID npx @testomatio/reporter run "" \ + --filter "coverage:file=,diff=" +``` + +State the secrets to provision: `TESTOMATIO` on jobs that talk to Testomat.io, +and a token able to post PR/MR comments (the CI's built-in token usually +suffices). The CI profile for `--remote` is configured in Testomat.io, not as +a repo secret. + +### Step 7 — Summarize and hand off + +Report: the CI targeted and files written; which phases are wired per map and +which were skipped (missing map / no e2e suite / no CI profile); the comment +format and its maps; title scheme and rungroup; the shared-run timeout and the +deploy duration it covers; how the execute step finds the prepared run +(`RUN_ID` or shared title); secrets and prerequisites still needed; assumptions +to confirm. Recommend committing the coverage maps. + +--- + +## Examples + +**Example 1 — one project, manual + automated (`coverage.shop.yml`)** +"Comment how many tests each PR touches, then after merge+deploy run the +affected e2e and give testers their cases." → Ask rungroup, deploy signal, +deploy duration. PR open: a non-blocking job posts the combined affected +count. On merge: one `start --kind mixed` shared run titled +`report for commit ` — manual cases pending immediately, `RUN_ID` +captured. On deploy-complete: a non-failing job runs +`reporter run --remote ` with that `RUN_ID`, executing the affected +automated tests into the same run. + +**Example 2 — no coverage maps yet** +"Set up selective regression on our PRs." → Find no `coverage*.yml`; explain +nothing can be counted or filtered without a map; delegate to `qa-test-coverage-map`; +only then wire CI. + +**Example 3 — manual-only project** +scan-automation-project finds `.test.md` cases and no e2e framework → wire the PR comment +(manual count) and the on-merge `start --kind manual` run from +`coverage..manual.yml`. No deploy phase at all; explain the automated +phase needs an e2e suite first. Never fabricate an e2e job. + +**Example 4 — no CI profile for `--remote`** +Confirm none exists; explain `--remote` dispatches a configured profile +(Testomat.io → Settings → CI) and cannot work without one. Wire phases (a) and +(b) now and the (c) execute step ready to enable once the profile exists; +offer the inline-runner exception meanwhile. + +--- + +## References + +| Description | File | +| -------------------------------------- | ------------------------------- | +| Reporter command contract & `--remote` | references/REPORTER_CONTRACT.md | + +## Related skills + +`scan-automation-project` (mandatory first), `qa-test-coverage-map` (creates the maps this skill +consumes), `qa-e2e-tests-reporting` (install the reporter if the project has no +Testomat.io integration yet). diff --git a/skills/setup-pr-testing/evals/evals.json b/skills/setup-pr-testing/evals/evals.json new file mode 100644 index 0000000..218d18f --- /dev/null +++ b/skills/setup-pr-testing/evals/evals.json @@ -0,0 +1,33 @@ +{ + "skill_name": "setup-pr-testing", + "evals": [ + { + "id": 0, + "name": "github-remote-profile-happy-path", + "prompt": "The project is at FIXTURE/acme-api. On each PR I want a comment showing how many tests the change affects. After the PR is merged and the staging deploy finishes, run only the affected e2e tests. The e2e suite runs through our Testomat.io CI profile 'staging-e2e'. Deploys happen on merge to main via the existing 'Deploy Staging' workflow.", + "expected_output": "Runs scan-automation-project first; detects GitHub Actions (does not assume); reuses the existing per-project coverage map(s) (coverage..yml / coverage..manual.yml / coverage..e2e.yml) and reads the test kinds from the filename suffix. Phase 1: a non-blocking PR-opened job computes affected counts with reporter run --filter-list per map and posts/updates a single PR comment with the counts; creates no runs. Phase 2 (on merge): one reporter start per map with --kind derived from the suffix (--kind manual for manual-only, --kind mixed for a mixed map, no --kind for e2e-only); runs containing automated tests are prepared as shared runs with TESTOMATIO_SHARED_RUN=1, TESTOMATIO_TITLE set to the merge commit, and TESTOMATIO_SHARED_RUN_TIMEOUT above the deploy duration, capturing RUN_ID without executing anything. Phase 3 (after Deploy Staging succeeds): a non-failing job runs reporter run --remote staging-e2e with TESTOMATIO_RUN=$RUN_ID + the same shared title, launching the affected automated tests on the CI profile. Sets meaningful title (merge commit) and rungroup; isolates the execute job from the release; does NOT dispatch a foreign repo directly.", + "files": [] + }, + { + "id": 1, + "name": "gitlab-missing-coverage-maps", + "prompt": "The project is at FIXTURE/shop-web. I want each MR to comment how many tests it affects, and after merge+deploy run only the affected tests. I don't think we have any coverage files set up yet.", + "expected_output": "Runs scan-automation-project; detects GitLab CI (not GitHub); finds no coverage*.yml; explains both the affected-counts comment and the regression runs cannot work without a coverage map and proposes/delegates to the qa-test-coverage-map skill to create the per-project map (coverage..yml, or the .manual/.e2e variant matching the test kinds) before wiring .gitlab-ci.yml; asks how merges deploy, how deploy completion is observable, and roughly how long deploys take (for the shared-run timeout); notes the PR-open comment needs no deploy answers.", + "files": [] + }, + { + "id": 2, + "name": "no-e2e-suite-refuse-to-fabricate", + "prompt": "The project is at FIXTURE/lib-utils. Make our pull requests run the affected end-to-end tests automatically.", + "expected_output": "Runs scan-automation-project; finds only Jest unit tests, no e2e framework anywhere; does NOT fabricate an e2e job, pipeline, or --remote launch; explains the automated phase needs an e2e suite first (points at test authoring / qa-test-coverage-map once a suite exists); offers the PR-open comment (manual count only) and an on-merge pending manual run via start --kind manual from the manual coverage map if manual cases exist; asks which CI is used since none is configured.", + "files": [] + }, + { + "id": 3, + "name": "no-ci-profile-for-remote", + "prompt": "The project is at FIXTURE/acme-api. Use --remote to run our affected e2e tests after deploy, and comment the affected counts on each PR. We use GitHub Actions and deploy on tag v*.", + "expected_output": "Runs scan-automation-project; detects GitHub Actions; confirms whether a Testomat.io CI profile exists for --remote and, finding none, explains --remote dispatches a configured profile and cannot work without one, pointing the user to Testomat.io Settings > CI to create it. Still wires phase 1 (PR-open counts comment per coverage map) and phase 2 (one run per map on merge, --kind by filename suffix, runs containing automated tests prepared as shared runs titled by commit). Wires the phase-3 reporter run --remote execute step keyed off the tag/deploy-complete signal as non-failing, to be enabled once the profile exists; offers the inline-runner exception if the e2e suite can run in this pipeline instead.", + "files": [] + } + ] +} diff --git a/skills/setup-pr-testing/references/REPORTER_CONTRACT.md b/skills/setup-pr-testing/references/REPORTER_CONTRACT.md new file mode 100644 index 0000000..8ca56b9 --- /dev/null +++ b/skills/setup-pr-testing/references/REPORTER_CONTRACT.md @@ -0,0 +1,203 @@ +# Reporter Contract: counts, prepared runs & `--remote` + +How `@testomatio/reporter` consumes a coverage map, counts affected tests for +the PR comment, prepares runs without executing them, and launches a prepared +run on a Testomat.io CI profile. This is CI-independent — the CI config wraps +these commands. + +## 1. The coverage filter + +A coverage map maps source globs → test/suite IDs/tags. There is one map per +Testomat.io project, and its suffix says which kinds of tests it selects: +`coverage..yml` (manual + automated), `coverage..manual.yml` +(manual only), `coverage..e2e.yml` (automated only). Legacy +`coverage.manual.yml` / `coverage.e2e.yml` are single-kind maps. The reporter +resolves *changed files* via git, maps them through the YAML, and selects the +matching tests. + +``` +--filter "coverage:file=,diff=" +``` + +- `file=` — path to the coverage map. **May be absolute.** It is read with + `fs`, independent of the working directory. +- `diff=` — git ref to diff against. Defaults to `master` if omitted. The + reporter runs `git diff --name-only` **in `process.cwd()`** (no `-C`). + So the reporter must be launched from inside the repo whose changes you want + to detect. Confirmed behavior, not configurable. + +The whole model rests on this: the repo that holds the coverage map + git +history is where the affected selection is computed — for the comment counts +and for scoping the prepared run alike. + +### The `--kind` rule + +The map's suffix decides the `--kind` flag on `start`/`run`: + +| Map | `--kind` | +| ---------------------------- | --------------- | +| `coverage..manual.yml` | `--kind manual` | +| `coverage..yml` | `--kind mixed` | +| `coverage..e2e.yml` | *(no flag)* | + +## 2. Phase 1 — affected-counts comment (`--filter-list`) + +`--filter-list` computes the affected tests **without executing or creating a +run** — exactly what the PR-open notice needs. Pair it with `--format` to get a +clean, parseable list on stdout (the banner is suppressed and logs go to +stderr). Run it once per coverage map: + +```bash +npx @testomatio/reporter run \ + --filter-list "coverage:file=,diff=$BASE" --format ids +``` + +`--format` values: `ids` (comma-separated, default), `newline` (one per line — +easy to `wc -l`), `json`, `grep` (`(@Sxxxx|@Tyyyy|...)`). + +- Count the entries per map; empty output = `0`. +- Assemble one line: per-kind counts when `.manual`/`.e2e` maps exist (`0 + automated tests, 10 manual tests are affected by this PR`); a mixed map + yields one combined count (`12 tests (manual + automated) are affected`). + Post it via the CI's PR/MR comment API (GitHub / GitLab / Bitbucket); prefer + updating one existing comment over re-posting. +- Needs `TESTOMATIO` (API key) for the `coverage:` resolution. Creates nothing, + must never fail the PR check. + +## 3. Phase 2 — create the regression runs on merge + +One run per coverage map, with `--kind` per the rule in §1. + +### 3a. Manual-only map — run created pending, complete at creation + +No test runner; the reporter creates a manual run containing only the affected +cases for testers to pick up. Nothing happens at deploy time: + +```bash +npx @testomatio/reporter start --kind manual \ + --filter "coverage:file=coverage..manual.yml,diff=$BASE" +``` + +- Required env: `TESTOMATIO`, `TESTOMATIO_TITLE` (merge commit, e.g. `report + for commit `), `TESTOMATIO_RUNGROUP_TITLE` (the rungroup bucket; + supports `/` nesting). `TESTOMATIO_ENV` optional. + +### 3b. Map containing automated tests — prepared as a shared run, not executed + +`start` creates the run scoped to the affected tests and returns its id +**without running anything**. Created as a *shared* run so the later execute +step — and any parallel executors — converge on this one run by title. A mixed +map adds `--kind mixed` (its manual cases are immediately pending for +testers); an e2e-only map takes no `--kind`: + +```bash +RUN_ID=$(TESTOMATIO_SHARED_RUN=1 \ + TESTOMATIO_TITLE="report for commit $SHA" \ + TESTOMATIO_SHARED_RUN_TIMEOUT=$DEPLOY_MINUTES \ + npx @testomatio/reporter start --kind mixed \ + --filter "coverage:file=coverage..yml,diff=$BASE" --format id) +``` + +- `--filter` scopes the prepared run to the affected tests; that scope is + stored on the run and reused at launch time (§4). +- **Output:** `--format id` makes `start` print only the run id to stdout + (banner and logs go to stderr), so `RUN_ID=$(...)` captures just the id. +- Required env: `TESTOMATIO`, `TESTOMATIO_TITLE`, `TESTOMATIO_RUNGROUP_TITLE`. + +### Shared-run env vars (the convergence mechanism) + +- `TESTOMATIO_SHARED_RUN=1` — report/launch into the run matching + `TESTOMATIO_TITLE` instead of creating a new one. All parallel reporters with + the same title land in the same run. +- `TESTOMATIO_TITLE` — the match key. The prepare step (§3b) and the execute + step (§4) **must** use the identical title. +- `TESTOMATIO_SHARED_RUN_TIMEOUT` — minutes the title stays matchable, **default + 20**. After it elapses a new run is created instead. Set it above the typical + merge→deploy duration, or the execute step won't attach to the prepared run. + Example: `TESTOMATIO_SHARED_RUN_TIMEOUT=120` for a 2-hour window. + +## 4. Phase 3 — launch the prepared run on CI (`reporter run --remote`) + +`reporter run --remote ` asks Testomat.io to dispatch the project's CI +profile (configured under **Settings → CI**) for an already-prepared run, +instead of executing tests locally. This replaces the old cross-repo dispatch: +the CI profile owns the runner, browsers, environment URLs and secrets — the +reporter just triggers it. Applies to every run prepared in §3b (mixed and +e2e-only); manual-only runs have no launch phase: + +```bash +TESTOMATIO_RUN=$RUN_ID \ +TESTOMATIO_SHARED_RUN=1 \ +TESTOMATIO_TITLE="report for commit $SHA" \ +npx @testomatio/reporter run --remote +``` + +- `TESTOMATIO_RUN=$RUN_ID` points the launch at the run prepared in §3b. With no + fresh `--filter`, Testomat.io greps that run's **own stored scope**, so only + the affected automated tests run — no need to recompute the diff at deploy + time. +- The CI profile name must exist on the project, otherwise the call fails with + `CI launch failed: No settings for `. `--remote` cannot be combined + with `--filter-list`. +- Testomat.io passes the run id into the dispatched workflow, so the tests + running there report back into the same prepared run. +- On success the CLI prints the launched profile and run URL, then exits 0; the + run transitions as the CI reports results. + +**Decoupled deploy (no `RUN_ID` to hand over).** If the deploy pipeline can't +carry `RUN_ID`, drop it and let the shared-run **title** match the prepared run +— keep `TESTOMATIO_SHARED_RUN=1` and the identical `TESTOMATIO_TITLE`, and make +sure the shared-run timeout (§3b) is still open. Carrying `RUN_ID` is the more +direct link when the same pipeline does merge→deploy; matching by title is the +fallback for a separate deploy pipeline. + +**Optional CI overrides.** `--remote-param key=value` (repeatable) forwards +values into the CI profile config at launch (e.g. `--remote-param branch=main`). +The same launch config can be set via env: `TESTOMATIO_CI_PROFILE` (= the +profile) and `TESTOMATIO_CI_PARAMS` (= comma-separated `key=value` overrides). + +## 5. Inline exception — execute in this pipeline (no CI profile) + +When the automated suite runs in this pipeline (mobile, API/contract, or an +existing same-repo job) rather than via a CI profile, launch the prepared run +by wrapping the runner directly after deploy: + +```bash +TESTOMATIO_RUN=$RUN_ID npx @testomatio/reporter run "" \ + --filter "coverage:file=,diff=$BASE" +``` + +Examples of ``: `npx playwright test`, `npx cypress run`, +`npx codeceptjs run`, `npx wdio`. The reporter injects the framework-appropriate +grep for the matched IDs and reports into `$RUN_ID`. + +## 6. Diff base caveats + +- **Comment + on-merge runs**: `diff=` (e.g. `main`) — the + natural "what this PR changes". +- **If you recompute at deploy time**: the meaningful diff is *what was just + deployed*. On a squash/rebase merge that is one commit, so `diff=~1` + works. With **multi-commit** pushes onto the deploy branch, `~1` under-selects + — diff against the previously-deployed SHA (persist it between deploys). + Preferably avoid recomputing: the prepared run already carries its scope, so + the `--remote` launch reuses it (§4) and no deploy-time diff is needed. + Surface this to the user rather than hard-coding silently. + +## 7. Required environment + +- `TESTOMATIO` — Testomat.io API key (every phase that talks to Testomat.io). +- `TESTOMATIO_TITLE` — run title; derive from the merge commit. The automated + prepare (§3b) and execute (§4) steps MUST use the same value. +- `TESTOMATIO_RUNGROUP_TITLE` — rungroup for the created runs. Supports `/` + nesting. +- `TESTOMATIO_SHARED_RUN` — `1` to converge on the title-matched run (automated + prepare + execute). +- `TESTOMATIO_SHARED_RUN_TIMEOUT` — minutes the shared title stays matchable + (default 20); set above the merge→deploy duration. +- `TESTOMATIO_RUN` — the prepared run id; set on the execute step to launch that + specific run. +- `TESTOMATIO_CI_PROFILE` / `TESTOMATIO_CI_PARAMS` — env equivalents of + `--remote` / `--remote-param`. +- `TESTOMATIO_ENV` (optional) — target environment label. +- A Testomat.io **CI profile** (Settings → CI) is required for `--remote`; it is + configured in Testomat.io, not as a repo secret.