diff --git a/collections.yaml b/collections.yaml index 183429d..2f9a259 100644 --- a/collections.yaml +++ b/collections.yaml @@ -37,6 +37,12 @@ common-terraform: - src: assets/instructions/terraform/plans.instructions.md dest: .github/instructions/terraform-plans.instructions.md +common-spread: + description: "Spread integration task runner skills and guidance" + items: + - src: skills/running-spread + dest: .github/skills/running-spread/ + common-documentation: description: "Documentation standards and review skills" items: diff --git a/skills/running-spread/SKILL.md b/skills/running-spread/SKILL.md new file mode 100644 index 0000000..11dd270 --- /dev/null +++ b/skills/running-spread/SKILL.md @@ -0,0 +1,48 @@ +--- +name: running-spread +description: >- + Discovers, selects, executes, and debugs Spread integration test tasks + in repositories with spread.yaml configurations. Use when running spread + tests, inspecting spread.yaml matrix dimensions (backends, systems, suites, + tasks, variants), troubleshooting spread failures, or managing spread + server lifecycle. +compatibility: spread yq +allowed-tools: spread yq find grep env ls cat +--- + +# Using Spread + +## Scope + +This skill covers discovering, executing, and debugging Spread tasks. +It does NOT cover writing new `task.yaml` files, modifying `spread.yaml` configuration, writing Spread backend plugins, configuring cloud credentials, or setting up local virtualization providers (Multipass/LXD/QEMU). + +--- + +## How to Use This Skill + +This skill acts as a router. When you need to perform a task, read the corresponding reference guide: + +1. **Discover the test matrix** → Read [test-matrix-and-selectors.md](references/test-matrix-and-selectors.md) to inspect backends, systems, suites, tasks, variants, and constraints using `yq` and `find`. +2. **Execute and iterate** → Read [command-reference.md](references/command-reference.md) for CLI flags (e.g., `-reuse`, `-resend`, `-abend`, `-workers`, `-repeat`). +3. **Diagnose failures** → Read [failure-diagnosis-and-troubleshooting.md](references/failure-diagnosis-and-troubleshooting.md) for the error classification matrix (infrastructure vs hooks vs timeouts). +4. **Manage servers** → Read [server-management-and-cleanup.md](references/server-management-and-cleanup.md) for lifecycle control (`-reuse -discard`, `-gc`, recovering PIDs). +5. **Collect diagnostic output** → Read [logs-and-artifacts.md](references/logs-and-artifacts.md) for parsing `-json` results, inspecting `-logs`, and collecting `-artifacts`. + +--- + +## Critical Invariant Rules + +> [!IMPORTANT] +> **Always run `spread -list` before executing tasks.** +> Before triggering any execution, you MUST run `spread -list ` to inspect the complete set of resolved task runs on `stdout`. This ensures you only provision instances and execute the exact intended set of tasks, backends, and systems, avoiding unintended blanket matrix runs. + +> [!IMPORTANT] +> **Trailing Slash on Suites**: A suite name **must always end with a trailing slash (`/`)** both in `spread.yaml` and in selector expressions. Omitting the trailing slash causes Spread to treat the string as a task path instead of a suite. + +> [!WARNING] +> If `spread` exits with a non-zero exit code, classify the failure before retrying. +> Read [failure-diagnosis-and-troubleshooting.md](references/failure-diagnosis-and-troubleshooting.md) and follow the Quick Failure Classification Matrix. + +> [!CAUTION] +> **Interactive Flags**: Do NOT use `-debug`, `-shell`, `-shell-before`, or `-shell-after`. These spawn interactive TTY sessions that hang agent execution indefinitely. They are for human developers only. diff --git a/skills/running-spread/references/command-reference.md b/skills/running-spread/references/command-reference.md new file mode 100644 index 0000000..601461b --- /dev/null +++ b/skills/running-spread/references/command-reference.md @@ -0,0 +1,123 @@ +# Spread CLI Command Reference + +This reference covers the command-line flags, runtime options, debugging helpers, and practical workflows for executing tasks with [Spread](https://github.com/canonical/spread). + +> [!TIP] +> For a detailed explanation of the matrix hierarchy (Backends, Systems, Suites, Tasks, Variants), programmatic matrix discovery with `yq`/`find`, and full selector grammar, refer to [test-matrix-and-selectors.md](test-matrix-and-selectors.md). + +--- + +## 1. Command-Line Flags + +> [!NOTE] +> For flags taking arguments, Spread accepts both space-separated (`-flag `) and equal-separated (`-flag=`) syntax. + +| Flag | Description | Typical Use Case | +|---|---|---| +| `-list` | List all matched jobs on stdout without executing them | Previewing task selection before execution | +| `-json=` | Save structured machine-readable task results (JSON) | Automated result parsing by agents or CI | +| `-logs=` | Save generated execution and communication logs to a directory | Archiving full task logs for diagnosis | +| `-reuse` | Keep allocated instances alive for subsequent runs | Accelerating local iterative execution | +| `-discard` | Discard/destroy any running reusable instances | Cleaning up environment after execution | +| `-resend` | Resend project content to reused servers | Syncing updated source files to reused instance | +| `-repeat ` | Repeat each selected task `N` times | Flake hunting and stability verification | +| `-order` | Execute tasks in exact declared order without shuffling | Preventing task randomization | +| `-seed ` | Seed for random task permutation | Reproducing order-dependent failures | +| `-abend` | Stop immediately on first error without restoring/cleaning up | Preserving failure state on the backend instance | +| `-artifacts=` | Directory to save collected artifacts | Fetching logs, crash dumps, and outputs | +| `-perf` | Show timestamps and task execution durations | Performance benchmarking and timing analysis | +| `-v` / `-vv` | Verbose / debug logging output from Spread | Troubleshooting Spread allocation/setup | +| `-workers ` | Number of concurrent workers per system | Tuning task parallelism | +| `-restore` | Run only the restore scripts | Cleaning up partially executed suites | +| `-gc` | Discard allocated servers no longer in use | Purging orphaned backend instances | +| `-reuse-pid ` | Select servers reused by a specific Spread process | Recovering from a crashed Spread run | + +--- + +## 2. Practical Workflow Recipes + +### 1. Previewing Task Selection (`-list`) + +For detailed selector syntax examples across backends, systems, suites, tasks, and variants, see [test-matrix-and-selectors.md](test-matrix-and-selectors.md): + +```bash +# List all jobs in the project matrix +spread -list + +# List all tasks in a specific suite (suite must end with '/') +spread -list tests/spread/commands/ + +# List matching tasks on OpenStack for Ubuntu 24.04 +spread -list openstack:ubuntu-24.04-64: +``` + +### 2. Fast Local Iteration & Server Management (`-reuse`, `-resend`, `-discard`) + +When developing or modifying a task, avoid re-provisioning instances each run. For complete server lifecycle and garbage-collection guides, see [server-management-and-cleanup.md](server-management-and-cleanup.md): + +```bash +# First run: provisions the backend instance and keeps it alive +spread -reuse multipass:ubuntu-24.04-64:tests/spread/commands/version + +# Subsequent runs: syncs modified files and reuses the running machine +spread -reuse -resend multipass:ubuntu-24.04-64:tests/spread/commands/version + +# When finished testing, discard the reused instance +spread -reuse -discard +``` + +### 3. Preserving Failure State (`-abend`) + +Stop execution on the first error and prevent restoration scripts from altering the machine state. For error classification and troubleshooting steps, see [failure-diagnosis-and-troubleshooting.md](failure-diagnosis-and-troubleshooting.md): + +```bash +# Halt on first failure and preserve remote machine state for inspection +spread -abend openstack:ubuntu-24.04-64:tests/spread/commands/version +``` + +### 4. Hunting Flaky Tasks (`-repeat`, `-seed`, `-order`) + +By default, Spread shuffles task order within a suite to uncover hidden order dependencies: + +```bash +# Repeat the task 10 consecutive times to verify determinism +spread -repeat 10 multipass:ubuntu-24.04-64:tests/spread/commands/version + +# Execute tasks in sequential/declared order without shuffling +spread -order tests/spread/commands/ + +# Reproduce a specific randomized execution order using its seed +spread -seed 42 tests/spread/commands/ +``` + +### 5. Controlling Concurrency on Local Host (`-workers`) + +When running on local backends (Multipass, LXD, QEMU), limit worker parallelism to avoid CPU contention or memory exhaustion: + +```bash +# Run with a single worker to avoid host overloading +spread -workers 1 multipass:ubuntu-24.04-64:tests/spread/commands/ +``` + +### 6. Collecting Artifacts (`-artifacts`) + +Download logs and output files declared under `artifacts:` in `task.yaml`. For layout details and inspection procedures, see [logs-and-artifacts.md](logs-and-artifacts.md): + +```bash +spread -artifacts=./test-artifacts multipass:ubuntu-24.04-64:tests/spread/commands/version +``` + +### 7. Exporting Machine-Readable Results (`-json` & `-logs`) + +Save structured task execution results and logs for automated parsing. For schema details and diagnostic workflows, see [logs-and-artifacts.md](logs-and-artifacts.md): + +```bash +# Run tasks and export structured JSON summary and log files +spread -json=./spread-results -logs=./spread-logs multipass:ubuntu-24.04-64:tests/spread/commands/version +``` + +--- + +## 3. Interactive Flags (Human Operators Only) + +These flags (`-debug`, `-shell`, `-shell-before`, `-shell-after`) are strictly for human debugging and spawn interactive PTY shells. As noted in the main skill router, agents must never invoke them. diff --git a/skills/running-spread/references/failure-diagnosis-and-troubleshooting.md b/skills/running-spread/references/failure-diagnosis-and-troubleshooting.md new file mode 100644 index 0000000..187b979 --- /dev/null +++ b/skills/running-spread/references/failure-diagnosis-and-troubleshooting.md @@ -0,0 +1,131 @@ +# Spread Failure Diagnosis & Troubleshooting Guide + +When a Spread run reports an error or exit failure, agents must distinguish between **Selector Syntax Errors**, **Backend Infrastructure Failures**, **Setup/Hook Failures**, **Timeouts**, and genuine **Task Execution Failures** inside `execute:`. + +--- + +## 1. Quick Failure Classification Matrix + +| Failure Category | When It Happens | Typical Error Symptoms | Primary Action | +|---|---|---|---| +| **1. Selector Error** | Pre-execution | `error: nothing matches provider filter`
`error: invalid filter string` | Fix selector syntax (check trailing `/` for suites, verify system names in `spread.yaml`). | +| **2. Infrastructure / Allocation** | Machine provisioning | Quota exceeded, SSH timeout, backend API error, `wait-timeout` reached. | Run `spread -gc`, check cloud credentials/proxies, or execute locally on `multipass:`/`lxd:`. | +| **3. Lifecycle Hook Error** | Machine setup/teardown | Non-zero exit code during `prepare`, `prepare-each`, or `restore`. | Fix dependency installation, snap channels, or network proxy in suite `prepare:`. | +| **4. Task Execution Failure** | Task execution | Non-zero exit code during task `execute:`. | Debug the application code or task script assertions. | +| **5. Execution Timeout** | Long-running task | Task terminated after exceeding `kill-timeout` or `warn-timeout`. | Check for deadlocks, infinite loops, or increase `kill-timeout` in `task.yaml`. | + +--- + +## 2. Diagnosing Each Failure Type + +### 1. Selector & Filter Errors + +Spread aborts immediately before communicating with any backend: + +- **Missing trailing slash on suite**: + ```bash + $ spread -list tests/spread/commands + error: nothing matches provider filter + ``` + *Fix*: Add trailing slash -> `spread -list tests/spread/commands/` +- **Invalid colon syntax**: + ```bash + $ spread -list ::tests/spread/commands/... + error: invalid filter string: "::" + ``` + *Fix*: Avoid consecutive colons -> `spread -list tests/spread/commands/...` + +--- + +### 2. Backend Infrastructure & Allocation Errors + +Infrastructure failures occur when Spread cannot allocate, boot, or establish SSH communication with the backend instance. + +- **Symptoms**: + - `cannot allocate server on backend : ...` + - `timeout waiting for SSH connectivity` + - Cloud API 401/403 (unauthorized) or quota exceeded. +- **Diagnostic Procedure**: + 1. **Verify Host Environment & Credentials**: When executing on cloud backends (e.g. OpenStack), ensure authentication credentials are exported in the host shell: + ```bash + env | grep -E '^(OS_|LXD_)' + ``` + 2. **Inspect full communication logs**: + ```bash + spread -logs=./spread-logs + ``` + 3. **Run garbage collection** to purge leaked or orphaned instances: + ```bash + spread -gc : + ``` + 4. **Verify on a local backend**: Test if the same task reproduces on a local virtualization backend (e.g. `multipass:` or `lxd:`). + +--- + +### 3. Lifecycle Hook Failures (`prepare` & `restore`) + +Spread executes hooks in a defined lifecycle before running the task itself: + +`suite.prepare` -> `suite.prepare-each` -> `task.prepare` -> `task.execute` + +- **Symptoms**: + - Failure occurs before the `execute:` script runs. + - Apt package installation fails, snap refresh fails, or Juju bootstrap errors out. +- **Distinguishing from Task Execution Bugs**: + - Hook errors indicate that the execution environment was not ready or external dependencies failed to download, not that the code executed by the task is broken. +- **Diagnostic Procedure**: + - Check `./spread-logs/` to examine the failing hook script. + - If intermittent network flakes occur during package installation, verify proxy settings in `spread.yaml`. + +--- + +### 4. Task Execution Failures (`execute:`) + +A task execution failure occurs when the commands inside the task's `execute:` block return a non-zero exit code. + +- **Symptoms**: + - Output shows application traceback, assertion failure, or non-zero exit from the task script. + - Spread triggers `suite.debug-each` (if defined) to collect diagnostics. +- **Diagnostic Procedure**: + 1. **Preserve the instance**: Re-run with `-abend` and `-reuse` so Spread aborts on failure without running restoration scripts and keeps the machine running: + ```bash + spread -abend -reuse -artifacts=./test-artifacts + ``` + 2. **Inspect artifacts**: Read output logs and reports collected in `./test-artifacts/`. + 3. **Iterative re-execution**: Use `-reuse -resend` to test code fixes against the running instance: + ```bash + spread -reuse -resend + ``` + +--- + +### 5. Task Timeouts (`kill-timeout` & `warn-timeout`) + +Tasks can define timeout limits in `task.yaml` (e.g., `kill-timeout: 30m`). + +- **Symptoms**: + - Spread terminates the task process with SIGKILL and marks the job as timed out. +- **Diagnostic Procedure**: + - Run with `-perf` to view timestamps for each output line: + ```bash + spread -perf + ``` + - Identify whether the task hung on a blocking prompt, network request, or infinite loop. + +--- + +## 3. Recommended Troubleshooting Decision Flow + +```text +Spread Execution Result + ├── Exited immediately with syntax error? + │ └── Selector Error: Verify system/suite syntax (check trailing /). + ├── Failed during VM/container allocation or SSH connection? + │ └── Infrastructure Error: Run spread -gc, check quota/credentials. + ├── Failed during prepare / prepare-each hook? + │ └── Environment/Hook Error: Check proxy, package repository, or snap channels. + ├── Failed during execute: script? + │ └── Task Execution Failure: Re-run with -abend -reuse -artifacts=./test-artifacts to inspect failure state. + └── Task killed unexpectedly? + └── Timeout: Run with -perf to locate hanging command. +``` diff --git a/skills/running-spread/references/logs-and-artifacts.md b/skills/running-spread/references/logs-and-artifacts.md new file mode 100644 index 0000000..c6bcbbf --- /dev/null +++ b/skills/running-spread/references/logs-and-artifacts.md @@ -0,0 +1,135 @@ +# Spread Logs, Artifacts & Diagnostic Output + +When debugging Spread task runs, agents cannot rely on interactive shells. Instead, Spread provides built-in mechanisms to export **Artifacts**, **Execution Logs**, and **Machine-Readable JSON Results**. + +--- + +## 1. Artifacts Collection (`-artifacts`) + +Artifacts are files or directories generated on the task instance (e.g., application logs, core dumps, execution reports) that Spread pulls back to the host upon task completion or failure. + +### Checking Declared Artifacts in `task.yaml` + +To check if a task exports artifacts, inspect the `artifacts:` key in its `task.yaml`: + +```bash +yq '.artifacts // []' tests/spread/commands/version/task.yaml +``` + +### Pulling Artifacts with `-artifacts` + +Pass `-artifacts=` to specify the destination directory on the host when running Spread: + +```bash +spread -artifacts=./test-artifacts multipass:ubuntu-24.04-64:tests/spread/commands/version +``` + +### Artifact Directory Layout on Host + +Spread organizes downloaded artifacts into a folder named after the full job descriptor: + +```text +test-artifacts/ +└── multipass.ubuntu-24.04-64.tests_spread_commands_version/ + ├── app.log + ├── syslog + └── crash-reports/ +``` + +### Inspecting Artifacts + +List and inspect downloaded artifact files: + +```bash +# Locate all downloaded artifact files +find ./test-artifacts/ -type f +``` + +--- + +## 2. Execution & Communication Logs (`-logs`) + +The `-logs` flag stores the complete raw output and backend communication transcripts for each job. This is particularly useful for debugging backend provisioning issues, SSH timeouts, and suite-level `prepare`/`restore` hook failures. + +### Exporting Logs + +```bash +spread -logs=./spread-logs openstack:ubuntu-24.04-64:tests/spread/commands/ +``` + +### Log File Layout on Host + +Spread generates individual log files for each job in the matrix: + +```text +spread-logs/ +├── openstack.ubuntu-24.04-64.tests_spread_commands_git-build-root.log +└── openstack.ubuntu-24.04-64.tests_spread_commands_version.log +``` + +--- + +## 3. Structured JSON Results (`-json`) + +The `-json` flag produces machine-readable task execution summaries containing job status, execution timings, and error traces. This enables agents and CI systems to parse task outcomes programmatically. + +### Exporting JSON Results + +```bash +spread -json=./spread-results multipass:ubuntu-24.04-64:tests/spread/commands/... +``` + +### JSON Results Structure + +The output directory contains a `summary.json` file detailing the executed jobs. A single job entry contains: + +- `job`: Full job identifier. +- `status`: Outcome (`passed`, `failed`, `aborted`). +- `duration`: Execution duration in seconds. +- `error`: Error messages or non-zero exit codes if the task failed. + +**Example output:** + +```json +[ + { + "job": "multipass:ubuntu-24.04-64:tests/spread/commands/version", + "status": "passed", + "duration": 42.3, + "error": null + } +] +``` + +--- + +## 4. Console Verbosity & Timing Flags + +| Flag | Purpose | Description | +|---|---|---| +| `-v` | Verbose output | Displays detailed step-by-step progress during execution. | +| `-vv` | Debug output | Displays low-level debug messages, including backend API calls and SSH handshakes. | +| `-perf` | Timestamps | Prepends datetime timestamps to task script output to identify bottlenecks and hanging steps. | + +--- + +## 5. Recommended Agent Diagnostic Workflow + +When investigating task failures, use this combined non-interactive recipe: + +```bash +# Run with failure preservation, structured JSON output, logs, and artifacts +spread -abend \ + -reuse \ + -artifacts=./test-artifacts \ + -logs=./spread-logs \ + -json=./spread-results \ + openstack:ubuntu-24.04-64:tests/spread/commands/version +``` + +### Diagnostic Steps: + +1. **Check JSON summary**: Confirm whether failure occurred in task `execute:`, a suite hook (`prepare`), or backend allocation. +2. **Review task artifacts**: Check `./test-artifacts/` for application logs. +3. **Inspect full logs**: Read `./spread-logs/` for backend or hook execution logs. +4. **Preserved instance**: Because `-abend` was used, the machine remains alive on the backend for targeted follow-up re-runs with `-reuse`. diff --git a/skills/running-spread/references/server-management-and-cleanup.md b/skills/running-spread/references/server-management-and-cleanup.md new file mode 100644 index 0000000..2cdb540 --- /dev/null +++ b/skills/running-spread/references/server-management-and-cleanup.md @@ -0,0 +1,97 @@ +# Spread Server Management & Resource Cleanup Guide + +When running tasks across virtualization (LXD, Multipass, QEMU) and cloud backends (OpenStack, Google Cloud), Spread allocates virtual machines and containers. This guide explains how to manage active server instances, reuse them for fast iteration, and cleanly discard or garbage-collect resources. + +--- + +## 1. The Spread Server Lifecycle + +By default, Spread provisions a fresh instance for a task run and automatically destroys it upon task completion. However, provisioning instances repeatedly introduces significant latency. Spread provides controls to keep servers alive across runs and clean them up when done. + +| Action | Command / Flag | Purpose | +|---|---|---| +| **Keep server alive** | `-reuse` | Leaves backend instances running for subsequent task runs. | +| **Resend source files** | `-resend` | Synchronizes modified local files into the running reused instance. | +| **Discard reused servers** | `-discard` | Shuts down and deletes active reused servers. | +| **Garbage collection** | `-gc` | Reclaims orphaned or leaked backend resources (VMs, volumes, containers). | +| **Recover from crash** | `-reuse-pid ` | Reconnects to servers left behind by a crashed Spread process. | +| **Run restore scripts** | `-restore` | Runs only the suite and task `restore` scripts on the instance. | + +--- + +## 2. Iterative Development with `-reuse` and `-resend` + +### Step 1: Provision and Keep Alive + +Run the task with `-reuse`. Spread allocates the machine, runs the task, and keeps the server running: + +```bash +spread -reuse multipass:ubuntu-24.04-64:tests/spread/commands/version +``` + +### Step 2: Re-run with Updated Code + +After editing task files or source code in the repository, use `-reuse -resend` to push the updated files to the existing server without re-provisioning: + +```bash +spread -reuse -resend multipass:ubuntu-24.04-64:tests/spread/commands/version +``` + +--- + +## 3. Discarding Reused Servers (`-reuse -discard`) + +When execution is complete, always release backend resources by discarding the reused servers. Combining `-reuse` and `-discard` explicitly instructs Spread to target the active reused server pool and tear it down: + +```bash +spread -reuse -discard +``` + +> [!TIP] +> You can also discard reused servers for a specific backend or selector: +> +> ```bash +> spread -reuse -discard multipass: +> spread -reuse -discard openstack:ubuntu-24.04-64: +> ``` + +--- + +## 4. Backend Garbage Collection (`-gc`) + +If tasks are abruptly terminated or cloud resources leak, Spread can inspect the backend provider and remove orphaned VMs, dangling storage volumes, and unused security groups: + +```bash +spread -gc +``` + +To run garbage collection for a specific backend: + +```bash +spread -gc openstack: +spread -gc multipass: +``` + +--- + +## 5. Recovering from Crashed Processes (`-reuse-pid`) + +When Spread runs with `-reuse`, it tags the provisioned server with its process ID (PID). If the Spread process crashes or is killed (`SIGKILL`, terminal disconnect): + +1. Identify the PID of the previous Spread process from logs or history. +2. Reconnect to and discard the orphaned instances: + +```bash +# Clean up servers left by process 12345 +spread -reuse-pid 12345 -discard +``` + +--- + +## 6. Recommended Agent Cleanup Checklist + +To avoid leaking instances or exhausting cloud quotas, agents should follow this cleanup procedure: + +1. **Active Iteration**: Use `spread -reuse ...` and `spread -reuse -resend ...` while modifying and re-running code. +2. **Post-Execution Cleanup**: Always execute `spread -reuse -discard` once task execution concludes. +3. **Quota/Allocation Errors**: If Spread reports backend allocation errors or resource exhaustion, run `spread -gc` to purge orphaned resources before retrying. diff --git a/skills/running-spread/references/test-matrix-and-selectors.md b/skills/running-spread/references/test-matrix-and-selectors.md new file mode 100644 index 0000000..0031975 --- /dev/null +++ b/skills/running-spread/references/test-matrix-and-selectors.md @@ -0,0 +1,308 @@ +# Spread Task Matrix Hierarchy & Selector Guide + +Spread organizes execution environments and task suites into a multi-dimensional matrix. This guide teaches agents how to inspect `spread.yaml` using `yq` and `find` to discover available **Backends**, **Systems**, and **Suites**, how tasks and variants are structured, and how to select them using Spread's selector syntax. + +--- + +## 1. Inspecting `spread.yaml` with `yq` & `find` + +Rather than parsing large `spread.yaml` files manually, agents should use the following commands to inspect matrix components. + +### Discover Backends + +List all virtualization/cloud backends defined in the repository: + +```bash +yq '.backends | keys | .[]' spread.yaml +``` + +*Example Output:* + +```text +openstack +multipass +lxd +``` + +--- + +### Discover Systems + +#### 1. List systems configured for a specific backend + +```bash +# Replace with the target backend (e.g. openstack, multipass, lxd) +yq '.backends..systems[] | (keys | .[0]) // .' spread.yaml +``` + +*Example Output:* + +```text +ubuntu-20.04-64 +ubuntu-22.04-64 +ubuntu-24.04-64 +``` + +#### 2. List all unique systems across all backends + +```bash +yq '[.backends.[].systems[] | (keys | .[0]) // .] | unique | .[]' spread.yaml +``` + +--- + +### Discover Suites + +#### 1. List all suites in the project + +```bash +yq '.suites | keys | .[]' spread.yaml +``` + +*Example Output:* + +```text +tests/spread/commands/ +tests/spread/dependencies/ +tests/spread/smoketests/ +``` + +> [!NOTE] +> Suite names in `spread.yaml` and selectors must end with a trailing slash (`/`). + +#### 2. List suites along with their summaries + +```bash +yq '.suites | to_entries | .[] | .key + "\t(" + (.value.summary // "no summary") + ")"' spread.yaml +``` + +#### 3. List manual suites (suites excluded from default matrix runs) + +```bash +yq '.suites | to_entries | .[] | select(.value.manual == true) | .key' spread.yaml +``` + +#### 4. Check system constraints for a specific suite + +```bash +yq '.suites."tests/spread/dependencies/".systems[]' spread.yaml +``` + +--- + +### Discover Tasks and Variants + +- **Tasks**: Use `find` with `-printf '%h\n'` to print the directory paths containing `task.yaml` without invoking subprocesses: + + ```bash + # List all task directories inside a suite + find tests/spread/commands/ -name "task.yaml" -printf '%h\n' + + # Or list all task directories in the entire repository + find . -name "task.yaml" -printf '%h\n' + ``` + +- **Manual Tasks**: Tasks can have `manual: true` in their `task.yaml`. Check if a task is marked manual: + + ```bash + yq '.manual // false' tests/spread/commands/version/task.yaml + ``` + +- **Variants**: Read the task's `task.yaml` file to extract variants defined under `environment:` or `variants:`: + + ```bash + # Find variants defined via environment keys containing '/' + yq '.environment | keys | .[] | select(. == "*/*")' tests/spread/commands/init-extensions/task.yaml + + # Or find variants defined explicitly in a variants block + yq '.variants | keys | .[]' tests/spread/commands/init-extensions/task.yaml + ``` + + > [!TIP] + > Tasks may use either format. If one `yq` command returns empty, try the other to ensure no variants are missed. + +--- + +### Manual Suites & Tasks (`manual: true`) + +Spread supports marking both entire suites and individual tasks as **manual**: + +1. **Manual Suites (`suites..manual: true` in `spread.yaml`)**: + - **Behavior**: Excluded from default runs (e.g. running `spread -list` with no arguments or running broad wildcard selectors). + - **Execution**: To run tasks in a manual suite, the suite (or task within it) **must be explicitly specified** by name: + ```bash + spread -list docs/howto/code/ + spread docs/howto/code/ + ``` + +2. **Manual Tasks (`manual: true` in `task.yaml`)**: + - **Behavior**: Excluded when running the parent suite (e.g. `spread tests/spread/commands/` will skip any task inside `commands/` that has `manual: true`). + - **Execution**: To run a manual task, you **must target the specific task explicitly**: + ```bash + spread -list tests/spread/commands/expensive-task + spread tests/spread/commands/expensive-task + ``` + +--- + +### System & Backend Constraints + +Suites and tasks can restrict which systems and backends they run on using `systems:` and `backends:` lists: + +- **Inclusion list**: Only matching systems/backends will execute (e.g. `systems: [ubuntu-22.04-64, ubuntu-24.04-64]`). +- **Exclusion syntax (`-` prefix)**: Specific systems can be excluded from the matrix (e.g. `systems: [-ubuntu-18.04-64]`). + +#### 1. Checking Suite Constraints + +```bash +# Check if a suite restricts systems +yq '.suites."tests/spread/dependencies/".systems // []' spread.yaml + +# Check if a suite restricts backends +yq '.suites."tests/spread/dependencies/".backends // []' spread.yaml +``` + +#### 2. Checking Task Constraints + +```bash +# Check if a specific task restricts systems +yq '.systems // []' tests/spread/commands/version/task.yaml + +# Check if a specific task restricts backends +yq '.backends // []' tests/spread/commands/version/task.yaml +``` + +> [!NOTE] +> If `spread -list ` returns `error: nothing matches provider filter`, check whether suite or task constraints excluded your target system or backend. + +--- + +## 2. The Matrix Hierarchy Summary + +```text +Backend (e.g., lxd, openstack, multipass) +└── System (e.g., ubuntu-24.04-64, ubuntu-22.04) + └── Suite (e.g., tests/spread/commands/) + └── Task (e.g., tests/spread/commands/version) + └── Variant (e.g., flask, django) +``` + +| Dimension | Description | Defined In | How to Select | +|---|---|---|---| +| **Backend** | Virtualization / cloud infrastructure provider. | `backends:` in `spread.yaml` | `:` (e.g. `openstack:`) | +| **System** | Target OS image or machine spec. | `backends..systems` | `::` or `ubuntu-24.04-64:` | +| **Suite** | Group of tasks with shared lifecycle hooks (`prepare`, `restore`, etc.). | `suites:` in `spread.yaml` (**must end in `/`**) | `/` or `/...` | +| **Task** | Executable task containing `task.yaml`. | Directory containing `task.yaml` | `/` (no trailing slash) | +| **Variant** | Parameter variation of a task. | `environment: KEY/variant: val` in `task.yaml` | `:` or `:` | + +--- + +## 3. Selector Syntax & Grammar + +The complete selector syntax is: + +```text +[]:[]:[][:] +``` + +### Syntax Rules + +1. **Colons (`:`)**: Delimit matrix dimensions (`backend:system:task-path:variant`). +2. **No Consecutive Colons (`::`)**: Consecutive colons (like `::`) are invalid in Spread. To omit intermediate dimensions, use shorthand names or wildcard `...` syntax. +3. **Paths Use Forward Slashes (`/`)**: Paths are filesystem directory paths relative to the project root. +4. **Suites vs. Tasks**: + - `tests/spread/commands/` → Targets the entire suite (must end with `/` or `...`). + - `tests/spread/commands/version` → Targets the specific task. + +--- + +## 4. Selector Matching Examples (using `spread -list`) + +```text +::[:] +``` + +> [!NOTE] +> If `spread -list` returns successfully but produces **zero lines of output**, it means your selector matched tasks, but all matched tasks were marked `manual: true`. You must target them explicitly by name. + +### Examples + +#### Selecting by Backend + +```bash +# Preview all jobs configured on OpenStack +spread -list openstack: + +# Preview all jobs configured on Multipass +spread -list multipass: +``` + +#### Selecting by System + +```bash +# Preview all jobs on Ubuntu 24.04 across all backends +spread -list :ubuntu-24.04-64: +# Or shorthand: +spread -list ubuntu-24.04-64: + +# Preview all jobs on Ubuntu 24.04 on OpenStack only +spread -list openstack:ubuntu-24.04-64: +``` + +#### Selecting by Suite (Must End with `/`) + +```bash +# Preview an entire suite across all backends & systems (trailing '/' is required) +spread -list tests/spread/commands/ +# Or with wildcard: +spread -list tests/spread/commands/... + +# Preview a suite on a specific backend and system +spread -list openstack:ubuntu-24.04-64:tests/spread/commands/ +``` + +#### Selecting by Task + +```bash +# Preview a single task across all backends and systems +spread -list tests/spread/commands/version + +# Preview a single task on a specific backend and system +spread -list openstack:ubuntu-24.04-64:tests/spread/commands/version +``` + +#### Selecting by Variant + +```bash +# Preview a specific variant of a task +spread -list tests/spread/commands/init-extensions:flask + +# Preview a specific variant on a specific backend and system +spread -list openstack:ubuntu-22.04-64:tests/spread/commands/init-extensions:flask + +# Preview all tasks that define the 'flask' variant across the project +spread -list :flask +``` + +#### Wildcard Matching (`...`) + +```bash +# Match all Ubuntu systems on OpenStack +spread -list openstack:ubuntu-...: + +# Match all tasks containing 'smoke' in any suite +spread -list ...smoke... + +# Match all tasks under a directory structure +spread -list openstack:...:tests/spread/... +``` + +#### Multiple Selectors (Union) + +```bash +# Select a single task and an entire separate suite together +spread -list tests/spread/commands/version tests/spread/smoketests/ + +# Select two systems +spread -list :ubuntu-22.04-64: :ubuntu-24.04-64: +```