From 1adf2e0a226b23a996d79d3299f527711f13b31e Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Thu, 28 May 2026 11:22:57 +0300 Subject: [PATCH 1/2] [CACHE FOLDER] Use autodect folder option for manual-tests --- skills/sync-cases/SKILL.md | 70 +++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 19 deletions(-) diff --git a/skills/sync-cases/SKILL.md b/skills/sync-cases/SKILL.md index 10558eb..6574753 100644 --- a/skills/sync-cases/SKILL.md +++ b/skills/sync-cases/SKILL.md @@ -1,14 +1,14 @@ --- name: sync-cases -description: Synchronize test scenarios and cases between a local project and Testomat.io. Use this skill whenever the user wants to pull/export/download tests from Testomat.io; or push/import/sync new or updated test cases back to the TMS in corresponding `*.test.md` format. Supports custom directories, markdown test format and advanced import/export workflows. +description: Synchronize test scenarios and cases between a local project and Testomat.io. Use this skill whenever the user wants to pull/export/download tests from Testomat.io; or push/import/sync new or updated test cases back to the TMS in corresponding `*.test.md` format. Supports custom directories, markdown test format and advanced import/export workflows. Automatically detects project type and stores test cases in `manual-tests/` (for simple/test projects) or `.testclaw-context/manual-tests/` (for projects with source code). inputs: testDir: - description: "Where to put pulled tests (default: `.testclaw-context/manual-tests` — the gitignored cache). Only override with a tracked path if the repo already keeps its `*.test.md` files there." + description: "Override auto-detected test directory. By default: `manual-tests/` for simple projects, `.testclaw-context/manual-tests/` for big projects with additional backend, frontend logic" required: false license: MIT metadata: author: Testomat.io - version: 1.0.0 + version: 1.1.0 --- # SYNC-CASES SKILL: What I do @@ -57,7 +57,29 @@ Ask the user to obtain it from Testomat.io project: - Navigate to **Settings → Project → Project Reporting API key** _( Project path example by "project-id": `https://app.testomat.io/projects//settings/project` )_ -### Step 2: Pull or Push Operations +### Step 2: Detect Project Type and Choose Test Directory + +Before pulling or pushing, detect the project type to determine the correct test case location. + +#### Test Directory Rules + +| Project Type | Test Cases Location | Gitignored? | +| ------------ | ------------------- | ----------- | +| **Manual-only** (empty project, no backend, frontend logic) OR **E2E/test project** (`tests/`, `playwright/`, `cypress/`, `e2e/`) | `manual-tests/` | No | +| **Has additional source code** (has backend, frontend logic) or **big project** (monorepo, multiple test types) | `.testclaw-context/manual-tests/` | **Yes** — add to `.gitignore` if missing | +| **Folder from the user's prompt/request** | `testDir` | No | + +#### Detection Logic + +1. Has backend, frontend, extra logic code? => Use `.testclaw-context/manual-tests/` +2. Has `tests/`, `playwright/`, `cypress/`, `e2e/`, `tests-e2e/`? => Use `manual-tests/` +3. Otherwise => Default to `manual-tests/` + +> **Override:** If `testDir` input is provided or user specifies `-d ` in command, use that path instead. + +--- + +### Step 3: Pull or Push Operations #### Always use latest version of check-tests @@ -76,10 +98,12 @@ Download/Retrieves test scenarios from Testomat.io and saves them as Markdown fi - Refactor test cases offline. - Export only some specific suites by id. -**Where to pull:** into the gitignored cache `.testclaw-context/manual-tests/`, and add `.testclaw-context/` to the project `.gitignore` if it is not there yet. Pulled cases must never land in a tracked folder (`manual-tests/`, etc.) — that pollutes the repo. You can still edit them there and push back; being gitignored doesn't stop that. (If the repo *already* keeps its `*.test.md` files in a tracked folder, you don't need to pull at all — work with them where they are, or pass `-d ` for an in-place refresh.) This matches `project-scan`, which pulls the *code* into `.testclaw-context/code/` when it runs inside a manual-tests repo. +**Where to pull:** Determined by [Step 2: Detect Project Type](#step-2-detect-project-type-and-choose-test-directory). The target directory is created if it doesn't exist. **Pre-Pull:** -- Create `.testclaw-context/manual-tests/` if it doesn't exist; ensure `.testclaw-context/` is gitignored. +1. Run detection (if `testDir` not provided) +2. Create target directory if it doesn't exist +3. If target is `.testclaw-context/` → ensure it's gitignored **Command:** ```bash @@ -97,13 +121,16 @@ Optional variant - **pull by specific suite-ids** **Pull Examples:** ```bash -# Default — pull into the gitignored cache -npx check-tests pull -d .testclaw-context/manual-tests +# Auto-detected location +npx check-tests pull -# Repo that already keeps its test cases tracked — refresh them in place +# Manual-only or E2E project → manual-tests/ npx check-tests pull -d manual-tests -# Pull specific suites by IDs +# Big project with extra source code → gitignored cache +npx check-tests pull -d .testclaw-context/manual-tests + +# Specific suite IDs npx check-tests pull --suite-ids "@S12345678,@S87654321" ``` @@ -156,16 +183,22 @@ npx check-tests push [-d ] [--files ] **IMPORTANT:** When the files to push are known (e.g. just produced by `generate-cases` / `improve-test-cases`), pass them explicitly via `--files` (alias `-f`). Without `--files` the CLI falls back to the default glob `**/*.test.md`, which may pick up unrelated files. Quote glob patterns. Paths resolve relative to `--dir`. -**Examples:** +**Push Examples:** ```bash -# Specific files (preferred when known — e.g. just produced by generate-cases) +# Push from manual-tests/ +npx check-tests push -d manual-tests + +# Push from gitignored cache +npx check-tests push -d .testclaw-context/manual-tests + +# Specific files from cache folder (preferred when known — e.g. just produced by generate-cases) npx check-tests push --files .testclaw-context/manual-tests/login.test.md .testclaw-context/manual-tests/checkout.test.md # Custom glob npx check-tests push --files ".testclaw-context/manual-tests/**/*.test.md" -# Default glob (**/*.test.md) under -d — cases you pulled and edited in the cache -npx check-tests push -d .testclaw-context/manual-tests +# Specific files not from cache .testclaw-context folder +npx check-tests push --files login.test.md checkout.test.md ``` **Important constraints:** @@ -177,8 +210,6 @@ npx check-tests push -d .testclaw-context/manual-tests #### Labels Handling (Intent-Based) -Use `TESTOMATIO_LABELS` in sync/push **only if the user explicitly requests to set or override labels** in their query. - **Triggers:** * "push tests with labels smoke". * "import tests to TMS and set label=regression": @@ -193,7 +224,7 @@ After completing sync operations, output a short log-style summary: ``` Sync Complete: - Action: pull/push -- Directory: .testclaw-context/manual-tests +- Directory: - Tests synced: 15 - Status: Success ``` @@ -231,7 +262,7 @@ Stop execution if: ## Examples -**Pull tests** (lands in the gitignored `.testclaw-context/manual-tests/`): +**Pull tests:** ``` Use sync-cases skill to pull tests from Testomat.io ``` @@ -252,7 +283,8 @@ Use sync-cases to push tests to Testomat.io | Action | Command | | --------------- | -------------------------------------------------------------------- | -| Pull | `npx check-tests pull -d manual-tests` (the default; gitignored) | +| Pull | `npx check-tests pull -d ` | +| Push | `npx check-tests push -d ` | | Push (files) | `npx check-tests push --files ` | | Push (glob) | `npx check-tests push --files "/**/*.test.md"` | | Push (default) | `npx check-tests push -d ` (glob: `**/*.test.md`) | From 480ef564cb3c2ce526ce4807c7866d7b4c01ef85 Mon Sep 17 00:00:00 2001 From: Vitaliy Mykhailiuk AQA Date: Fri, 29 May 2026 11:05:50 +0300 Subject: [PATCH 2/2] [CACHE FOLDER] Autodetect folder for manual & auto coverage skill, project-scan skill --- skills/automation-coverage/SKILL.md | 6 +++++- skills/manual-coverage/SKILL.md | 23 ++++++++++++++++------- skills/project-scan/SKILL.md | 27 +++++++++++++-------------- 3 files changed, 34 insertions(+), 22 deletions(-) diff --git a/skills/automation-coverage/SKILL.md b/skills/automation-coverage/SKILL.md index f35fbf6..16d4d17 100644 --- a/skills/automation-coverage/SKILL.md +++ b/skills/automation-coverage/SKILL.md @@ -30,7 +30,11 @@ This skill works **only with automated e2e tests** (Playwright, Cypress, Webdriv - **DO NOT** process unit tests. - **DO NOT** process manual markdown test cases (use `manual-coverage` instead - if already exists). - **DO NOT** suggest creating new tests. -- **Only touch two files in this repo.** It may write `coverage.e2e.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source or test file. If the e2e tests live in another repo, clone it into the gitignored `.testclaw-context/e2e-tests/`, never into a tracked folder (see Step 1). +- **Only touch two files in this repo.** It may write `coverage.e2e.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source or test file. If the e2e tests live in another repo, clone it into: + - **Big project with source code** → gitignored `.testclaw-context/e2e-tests/` + - **Test-only project** (e2e repo without `src/`) → tracked folder (e.g., `e2e-tests/`) + +Never clone into a tracked folder in a source-code repo. - **Don't write scripts. Never use Python.** Read test files with your file tool; pull out IDs and tags with `grep` (Step 3). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper (symlinked from `manual-coverage`): `npx js-yaml coverage.e2e.yml | node scripts/check-coverage.mjs` (Step 6). That's the only script. If you ever need more than a `grep`, a one-line `node -e '…'` is the limit — never `python`, never a parser of your own. --- diff --git a/skills/manual-coverage/SKILL.md b/skills/manual-coverage/SKILL.md index 7144b7b..6208b01 100644 --- a/skills/manual-coverage/SKILL.md +++ b/skills/manual-coverage/SKILL.md @@ -29,7 +29,12 @@ This skill works **only with manual tests in markdown format**. - **DO NOT** process unit, functional, or e2e test files. - **DO NOT** suggest creating new automated tests. -- **Only touch two files in this repo.** This skill runs inside the user's source-code repo. It may write `coverage.manual.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source file. Cases pulled from Testomat.io go into the gitignored `.testclaw-context/manual-tests/`, never into a tracked folder (see Step 1). +- **Only touch two files in this repo.** This skill runs inside the user's source-code repo. It may write `coverage.manual.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source file. Cases pulled from Testomat.io go into: + - **Big project with source code** → gitignored `.testclaw-context/manual-tests/` + - **Test-only project** (no `src/`) → tracked `manual-tests/` + +Never pull cases into a tracked folder in a source-code repo. + - **Don't write scripts. Never use Python.** Read `.test.md` files with your file tool; pull out IDs and tags with `grep` (Step 2). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper: `npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs` (Step 5). If more checks needed use custom script with `npx js-yaml`. If automated test files (e.g. e2e test, unit, api) are encountered while exploring, ignore them and continue with the manual markdown set. @@ -49,11 +54,14 @@ From the `project-scan` result, capture: If `project-scan` reports **no manual tests** (it checks the cache too, so this means nothing was pulled before): - ❓ Ask the user how to proceed: - 1. Pull cases from Testomat.io — have **`sync-cases`** pull into the gitignored cache: `npx check-tests pull -d .testclaw-context/manual-tests`, then add `.testclaw-context/` to `.gitignore` if missing. Re-run `project-scan` and continue. + 1. Pull cases from Testomat.io — have **`sync-cases`** pull into the appropriate location: + - **Source-code project** (has `src/`) → gitignored `.testclaw-context/manual-tests/` + - **Test-only project** (no `src/`) → tracked `manual-tests/` + Then add `.testclaw-context/` to `.gitignore` if needed. Re-run `project-scan` and continue. 2. Point to a directory the scan missed (then re-run `project-scan` there). 3. Stop. -Never pull cases into a tracked folder. Don't repeat `sync-cases` pull logic here. +Never pull cases into a tracked folder in a source-code repo. Don't repeat `sync-cases` pull logic here. ### Step 2: Extract test information @@ -207,7 +215,7 @@ It's the only script — everything else is `grep`, your file tool, or `npx js-y ### Recovery -- **No manual tests found (cache included)** → have `sync-cases` pull into `.testclaw-context/manual-tests/`, or ask the user for a directory the scan missed. +- **No manual tests found (cache included)** → have `sync-cases` pull (auto-detects location based on project type), or ask the user for a directory the scan missed. - **Markdown files with no `@S`/`@T` IDs** → ask whether to push first via `sync-cases`, or skip those files. - **Ambiguous source layout** → ask the user which directories are application code. @@ -225,7 +233,7 @@ It's the only script — everything else is `grep`, your file tool, or `npx js-y ``` Use manual-coverage skill to build coverage.manual.yml for our manual cases ``` -If there are no local `.test.md` files, it pulls them into the gitignored `.testclaw-context/manual-tests/` and works from there. +If there are no local `.test.md` files, it pulls them into the appropriate location (auto-detected: `.testclaw-context/manual-tests/` for source-code projects, `manual-tests/` for test-only projects) and works from there. **Cases already local:** ``` @@ -238,7 +246,7 @@ Use manual-coverage skill, output to ops/coverage.qa.yml ``` **Full workflow (source repo):** -1. `manual-coverage` runs `project-scan`. No local cases, so it has `sync-cases` pull into `.testclaw-context/manual-tests/` (gitignored) and re-runs `project-scan`. +1. `manual-coverage` runs `project-scan`. No local cases, so it has `sync-cases` pull (auto-detects location) and re-runs `project-scan`. 2. It maps source files to suite/test/tag IDs and writes `coverage.manual.yml`. The only tracked changes are that file and one line in `.gitignore`. 3. `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` creates a pending run with only the affected cases. @@ -248,6 +256,7 @@ Use manual-coverage skill, output to ops/coverage.qa.yml | Action | Command | | -------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| Pull cases into the gitignored cache | `npx check-tests pull -d .testclaw-context/manual-tests` | +| Pull cases into cache (source-code project) | `npx check-tests pull -d .testclaw-context/manual-tests` (auto-detected) | +| Pull cases into tracked folder (test-only project) | `npx check-tests pull -d manual-tests` (auto-detected) | | Create affected manual run | `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` | | Group runs | `TESTOMATIO_RUNGROUP="Regression 911" npx @testomatio/reporter run --kind manual --filter "..."` | diff --git a/skills/project-scan/SKILL.md b/skills/project-scan/SKILL.md index 4b87e4e..9cdc2dd 100644 --- a/skills/project-scan/SKILL.md +++ b/skills/project-scan/SKILL.md @@ -56,25 +56,24 @@ Where is the source code? - If user provides `Git repo` or `another folder` variant => clone or symlink it into `.testclaw-context/code/` (see the rule below). No-op if it's already there. -#### Rule for pulled data — `.testclaw-context/` +#### Rule for pulled data — test directory detection -**This applies to every skill, not just `project-scan`.** +**When you need to pull external data** (manual test cases, app code, e2e tests from another repo) into this project: -1. First, see what's already in the current folder. -2. If you need data that isn't here — manual test cases, the application code, e2e tests, a repo to clone — pull it into `.testclaw-context/` (gitignored). **Never** pull it into a regular folder like `manual-tests/` or `automated-tests/`; that pollutes the repo. -3. If the data is already in the repo, use it where it is — you don't need `.testclaw-context/` for that. +| This project has… | Store pulled data in… | Gitignored? | +| ------------------ | ---------------------- | ----------- | +| **Source code** (`src/` folder) or **big project** | `.testclaw-context/…` | **Yes** | +| **Only test infrastructure** (e2e dirs like `tests/`, `playwright/`, `cypress/` but no `src/`) | `manual-tests/` or `e2e-tests/` | No | +| **Empty / manual-only** | `manual-tests/` | No | -| When you run inside… | What you'd pull in… | Goes in… | -| ----------------------- | -------------------- | --------------------------------- | -| a source-code repo | manual test cases | `.testclaw-context/manual-tests/` | -| a manual-tests repo | the application code | `.testclaw-context/code/` | -| (e2e tests in own repo) | the e2e tests | `.testclaw-context/e2e-tests/` | +**Detection logic:** +1. Has `src/` folder or is a monorepo? → Use `.testclaw-context/` +2. Has e2e test dirs (`tests/`, `playwright/`, `cypress/`, `e2e/`)? → Use tracked folder (`manual-tests/`, `e2e-tests/`) +3. Otherwise → Default to `manual-tests/` -Any skill that creates `.testclaw-context/...` must also add `.testclaw-context/` to the project's `.gitignore` — but only if it is not there yet. Never add it twice. +> **Override:** If user specifies a path, use that instead. -(The rule is about *pulled* data. Output a skill *produces* and the user wants — a `coverage.*.yml`, new `*.test.md` files — goes in the repo as normal.) - -On a source repo with no manual tests, `project-scan` changes no tracked file: it only creates the gitignored `.testclaw-context/` directory and, if needed, adds one line to `.gitignore`. +**Only touch `.testclaw-context/` and tracked folders — never pollute repo with cache in wrong location.** ---