Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion skills/automation-coverage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,11 @@ This skill works **only with automated e2e tests** (Playwright, Cypress, Webdriv
- **DO NOT** process unit tests.
- **DO NOT** process manual markdown test cases (use `manual-coverage` instead - if already exists).
- **DO NOT** suggest creating new tests.
- **Only touch two files in this repo.** It may write `coverage.e2e.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source or test file. If the e2e tests live in another repo, clone it into the gitignored `.testclaw-context/e2e-tests/`, never into a tracked folder (see Step 1).
- **Only touch two files in this repo.** It may write `coverage.e2e.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source or test file. If the e2e tests live in another repo, clone it into:
- **Big project with source code** → gitignored `.testclaw-context/e2e-tests/`
- **Test-only project** (e2e repo without `src/`) → tracked folder (e.g., `e2e-tests/`)

Never clone into a tracked folder in a source-code repo.
- **Don't write scripts. Never use Python.** Read test files with your file tool; pull out IDs and tags with `grep` (Step 3). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper (symlinked from `manual-coverage`): `npx js-yaml coverage.e2e.yml | node scripts/check-coverage.mjs` (Step 6). That's the only script. If you ever need more than a `grep`, a one-line `node -e '…'` is the limit — never `python`, never a parser of your own.

---
Expand Down
23 changes: 16 additions & 7 deletions skills/manual-coverage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,12 @@ This skill works **only with manual tests in markdown format**.

- **DO NOT** process unit, functional, or e2e test files.
- **DO NOT** suggest creating new automated tests.
- **Only touch two files in this repo.** This skill runs inside the user's source-code repo. It may write `coverage.manual.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source file. Cases pulled from Testomat.io go into the gitignored `.testclaw-context/manual-tests/`, never into a tracked folder (see Step 1).
- **Only touch two files in this repo.** This skill runs inside the user's source-code repo. It may write `coverage.manual.yml` (or the path the user gave) and add one `.testclaw-context/` line to `.gitignore` if it is missing. Nothing else — never a source file. Cases pulled from Testomat.io go into:
- **Big project with source code** → gitignored `.testclaw-context/manual-tests/`
- **Test-only project** (no `src/`) → tracked `manual-tests/`

Never pull cases into a tracked folder in a source-code repo.

- **Don't write scripts. Never use Python.** Read `.test.md` files with your file tool; pull out IDs and tags with `grep` (Step 2). To check the finished coverage file, pipe it through `js-yaml` into the one tiny bundled helper: `npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs` (Step 5). If more checks needed use custom script with `npx js-yaml`.

If automated test files (e.g. e2e test, unit, api) are encountered while exploring, ignore them and continue with the manual markdown set.
Expand All @@ -49,11 +54,14 @@ From the `project-scan` result, capture:

If `project-scan` reports **no manual tests** (it checks the cache too, so this means nothing was pulled before):
- ❓ Ask the user how to proceed:
1. Pull cases from Testomat.io — have **`sync-cases`** pull into the gitignored cache: `npx check-tests pull -d .testclaw-context/manual-tests`, then add `.testclaw-context/` to `.gitignore` if missing. Re-run `project-scan` and continue.
1. Pull cases from Testomat.io — have **`sync-cases`** pull into the appropriate location:
- **Source-code project** (has `src/`) → gitignored `.testclaw-context/manual-tests/`
- **Test-only project** (no `src/`) → tracked `manual-tests/`
Then add `.testclaw-context/` to `.gitignore` if needed. Re-run `project-scan` and continue.
2. Point to a directory the scan missed (then re-run `project-scan` there).
3. Stop.

Never pull cases into a tracked folder. Don't repeat `sync-cases` pull logic here.
Never pull cases into a tracked folder in a source-code repo. Don't repeat `sync-cases` pull logic here.

### Step 2: Extract test information

Expand Down Expand Up @@ -207,7 +215,7 @@ It's the only script — everything else is `grep`, your file tool, or `npx js-y

### Recovery

- **No manual tests found (cache included)** → have `sync-cases` pull into `.testclaw-context/manual-tests/`, or ask the user for a directory the scan missed.
- **No manual tests found (cache included)** → have `sync-cases` pull (auto-detects location based on project type), or ask the user for a directory the scan missed.
- **Markdown files with no `@S`/`@T` IDs** → ask whether to push first via `sync-cases`, or skip those files.
- **Ambiguous source layout** → ask the user which directories are application code.

Expand All @@ -225,7 +233,7 @@ It's the only script — everything else is `grep`, your file tool, or `npx js-y
```
Use manual-coverage skill to build coverage.manual.yml for our manual cases
```
If there are no local `.test.md` files, it pulls them into the gitignored `.testclaw-context/manual-tests/` and works from there.
If there are no local `.test.md` files, it pulls them into the appropriate location (auto-detected: `.testclaw-context/manual-tests/` for source-code projects, `manual-tests/` for test-only projects) and works from there.

**Cases already local:**
```
Expand All @@ -238,7 +246,7 @@ Use manual-coverage skill, output to ops/coverage.qa.yml
```

**Full workflow (source repo):**
1. `manual-coverage` runs `project-scan`. No local cases, so it has `sync-cases` pull into `.testclaw-context/manual-tests/` (gitignored) and re-runs `project-scan`.
1. `manual-coverage` runs `project-scan`. No local cases, so it has `sync-cases` pull (auto-detects location) and re-runs `project-scan`.
2. It maps source files to suite/test/tag IDs and writes `coverage.manual.yml`. The only tracked changes are that file and one line in `.gitignore`.
3. `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` creates a pending run with only the affected cases.

Expand All @@ -248,6 +256,7 @@ Use manual-coverage skill, output to ops/coverage.qa.yml

| Action | Command |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Pull cases into the gitignored cache | `npx check-tests pull -d .testclaw-context/manual-tests` |
| Pull cases into cache (source-code project) | `npx check-tests pull -d .testclaw-context/manual-tests` (auto-detected) |
| Pull cases into tracked folder (test-only project) | `npx check-tests pull -d manual-tests` (auto-detected) |
| Create affected manual run | `npx @testomatio/reporter run --kind manual --filter "coverage:file=coverage.manual.yml,diff=main"` |
| Group runs | `TESTOMATIO_RUNGROUP="Regression 911" npx @testomatio/reporter run --kind manual --filter "..."` |
27 changes: 13 additions & 14 deletions skills/project-scan/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,25 +56,24 @@ Where is the source code?

- If user provides `Git repo` or `another folder` variant => clone or symlink it into `.testclaw-context/code/` (see the rule below). No-op if it's already there.

#### Rule for pulled data — `.testclaw-context/`
#### Rule for pulled data — test directory detection

**This applies to every skill, not just `project-scan`.**
**When you need to pull external data** (manual test cases, app code, e2e tests from another repo) into this project:

1. First, see what's already in the current folder.
2. If you need data that isn't here — manual test cases, the application code, e2e tests, a repo to clone — pull it into `.testclaw-context/` (gitignored). **Never** pull it into a regular folder like `manual-tests/` or `automated-tests/`; that pollutes the repo.
3. If the data is already in the repo, use it where it is — you don't need `.testclaw-context/` for that.
| This project has… | Store pulled data in… | Gitignored? |
| ------------------ | ---------------------- | ----------- |
| **Source code** (`src/` folder) or **big project** | `.testclaw-context/…` | **Yes** |
| **Only test infrastructure** (e2e dirs like `tests/`, `playwright/`, `cypress/` but no `src/`) | `manual-tests/` or `e2e-tests/` | No |
| **Empty / manual-only** | `manual-tests/` | No |

| When you run inside… | What you'd pull in… | Goes in… |
| ----------------------- | -------------------- | --------------------------------- |
| a source-code repo | manual test cases | `.testclaw-context/manual-tests/` |
| a manual-tests repo | the application code | `.testclaw-context/code/` |
| (e2e tests in own repo) | the e2e tests | `.testclaw-context/e2e-tests/` |
**Detection logic:**
1. Has `src/` folder or is a monorepo? → Use `.testclaw-context/`
2. Has e2e test dirs (`tests/`, `playwright/`, `cypress/`, `e2e/`)? → Use tracked folder (`manual-tests/`, `e2e-tests/`)
3. Otherwise → Default to `manual-tests/`

Any skill that creates `.testclaw-context/...` must also add `.testclaw-context/` to the project's `.gitignore` — but only if it is not there yet. Never add it twice.
> **Override:** If user specifies a path, use that instead.

(The rule is about *pulled* data. Output a skill *produces* and the user wants — a `coverage.*.yml`, new `*.test.md` files — goes in the repo as normal.)

On a source repo with no manual tests, `project-scan` changes no tracked file: it only creates the gitignored `.testclaw-context/` directory and, if needed, adds one line to `.gitignore`.
**Only touch `.testclaw-context/` and tracked folders — never pollute repo with cache in wrong location.**

---

Expand Down
70 changes: 51 additions & 19 deletions skills/sync-cases/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
name: sync-cases
description: Synchronize test scenarios and cases between a local project and Testomat.io. Use this skill whenever the user wants to pull/export/download tests from Testomat.io; or push/import/sync new or updated test cases back to the TMS in corresponding `*.test.md` format. Supports custom directories, markdown test format and advanced import/export workflows.
description: Synchronize test scenarios and cases between a local project and Testomat.io. Use this skill whenever the user wants to pull/export/download tests from Testomat.io; or push/import/sync new or updated test cases back to the TMS in corresponding `*.test.md` format. Supports custom directories, markdown test format and advanced import/export workflows. Automatically detects project type and stores test cases in `manual-tests/` (for simple/test projects) or `.testclaw-context/manual-tests/` (for projects with source code).
inputs:
testDir:
description: "Where to put pulled tests (default: `.testclaw-context/manual-tests` — the gitignored cache). Only override with a tracked path if the repo already keeps its `*.test.md` files there."
description: "Override auto-detected test directory. By default: `manual-tests/` for simple projects, `.testclaw-context/manual-tests/` for big projects with additional backend, frontend logic"
required: false
license: MIT
metadata:
author: Testomat.io
version: 1.0.0
version: 1.1.0
---

# SYNC-CASES SKILL: What I do
Expand Down Expand Up @@ -57,7 +57,29 @@ Ask the user to obtain it from Testomat.io project:
- Navigate to **Settings → Project → Project Reporting API key**
_( Project path example by "project-id": `https://app.testomat.io/projects/<project-id>/settings/project` )_

### Step 2: Pull or Push Operations
### Step 2: Detect Project Type and Choose Test Directory

Before pulling or pushing, detect the project type to determine the correct test case location.

#### Test Directory Rules

| Project Type | Test Cases Location | Gitignored? |
| ------------ | ------------------- | ----------- |
| **Manual-only** (empty project, no backend, frontend logic) OR **E2E/test project** (`tests/`, `playwright/`, `cypress/`, `e2e/`) | `manual-tests/` | No |
| **Has additional source code** (has backend, frontend logic) or **big project** (monorepo, multiple test types) | `.testclaw-context/manual-tests/` | **Yes** — add to `.gitignore` if missing |
| **Folder from the user's prompt/request** | `testDir` | No |

#### Detection Logic

1. Has backend, frontend, extra logic code? => Use `.testclaw-context/manual-tests/`
2. Has `tests/`, `playwright/`, `cypress/`, `e2e/`, `tests-e2e/`? => Use `manual-tests/`
3. Otherwise => Default to `manual-tests/`

> **Override:** If `testDir` input is provided or user specifies `-d <path>` in command, use that path instead.

---

### Step 3: Pull or Push Operations

#### Always use latest version of check-tests

Expand All @@ -76,10 +98,12 @@ Download/Retrieves test scenarios from Testomat.io and saves them as Markdown fi
- Refactor test cases offline.
- Export only some specific suites by id.

**Where to pull:** into the gitignored cache `.testclaw-context/manual-tests/`, and add `.testclaw-context/` to the project `.gitignore` if it is not there yet. Pulled cases must never land in a tracked folder (`manual-tests/`, etc.) — that pollutes the repo. You can still edit them there and push back; being gitignored doesn't stop that. (If the repo *already* keeps its `*.test.md` files in a tracked folder, you don't need to pull at all — work with them where they are, or pass `-d <that folder>` for an in-place refresh.) This matches `project-scan`, which pulls the *code* into `.testclaw-context/code/` when it runs inside a manual-tests repo.
**Where to pull:** Determined by [Step 2: Detect Project Type](#step-2-detect-project-type-and-choose-test-directory). The target directory is created if it doesn't exist.

**Pre-Pull:**
- Create `.testclaw-context/manual-tests/` if it doesn't exist; ensure `.testclaw-context/` is gitignored.
1. Run detection (if `testDir` not provided)
2. Create target directory if it doesn't exist
3. If target is `.testclaw-context/` → ensure it's gitignored

**Command:**
```bash
Expand All @@ -97,13 +121,16 @@ Optional variant - **pull by specific suite-ids**

**Pull Examples:**
```bash
# Default — pull into the gitignored cache
npx check-tests pull -d .testclaw-context/manual-tests
# Auto-detected location
npx check-tests pull

# Repo that already keeps its test cases tracked — refresh them in place
# Manual-only or E2E project → manual-tests/
npx check-tests pull -d manual-tests

# Pull specific suites by IDs
# Big project with extra source code → gitignored cache
npx check-tests pull -d .testclaw-context/manual-tests

# Specific suite IDs
npx check-tests pull --suite-ids "@S12345678,@S87654321"
```

Expand Down Expand Up @@ -156,16 +183,22 @@ npx check-tests push [-d <directory>] [--files <files...>]

**IMPORTANT:** When the files to push are known (e.g. just produced by `generate-cases` / `improve-test-cases`), pass them explicitly via `--files` (alias `-f`). Without `--files` the CLI falls back to the default glob `**/*.test.md`, which may pick up unrelated files. Quote glob patterns. Paths resolve relative to `--dir`.

**Examples:**
**Push Examples:**
```bash
# Specific files (preferred when known — e.g. just produced by generate-cases)
# Push from manual-tests/
npx check-tests push -d manual-tests

# Push from gitignored cache
npx check-tests push -d .testclaw-context/manual-tests

# Specific files from cache folder (preferred when known — e.g. just produced by generate-cases)
npx check-tests push --files .testclaw-context/manual-tests/login.test.md .testclaw-context/manual-tests/checkout.test.md

# Custom glob
npx check-tests push --files ".testclaw-context/manual-tests/**/*.test.md"

# Default glob (**/*.test.md) under -d — cases you pulled and edited in the cache
npx check-tests push -d .testclaw-context/manual-tests
# Specific files not from cache .testclaw-context folder
npx check-tests push --files login.test.md checkout.test.md
```

**Important constraints:**
Expand All @@ -177,8 +210,6 @@ npx check-tests push -d .testclaw-context/manual-tests

#### Labels Handling (Intent-Based)

Use `TESTOMATIO_LABELS` in sync/push **only if the user explicitly requests to set or override labels** in their query.

**Triggers:**
* "push tests with labels smoke".
* "import tests to TMS and set label=regression":
Expand All @@ -193,7 +224,7 @@ After completing sync operations, output a short log-style summary:
```
Sync Complete:
- Action: pull/push
- Directory: .testclaw-context/manual-tests
- Directory: <auto-detected path>
- Tests synced: 15
- Status: Success
```
Expand Down Expand Up @@ -231,7 +262,7 @@ Stop execution if:

## Examples

**Pull tests** (lands in the gitignored `.testclaw-context/manual-tests/`):
**Pull tests:**
```
Use sync-cases skill to pull tests from Testomat.io
```
Expand All @@ -252,7 +283,8 @@ Use sync-cases to push tests to Testomat.io

| Action | Command |
| --------------- | -------------------------------------------------------------------- |
| Pull | `npx check-tests pull -d manual-tests` (the default; gitignored) |
| Pull | `npx check-tests pull -d <dir>` |
| Push | `npx check-tests push -d <dir>` |
| Push (files) | `npx check-tests push --files <file1.test.md> <file2.test.md>` |
| Push (glob) | `npx check-tests push --files "<dir>/**/*.test.md"` |
| Push (default) | `npx check-tests push -d <directory>` (glob: `**/*.test.md`) |