feat(skills): add running-spread skill

collections.yaml

@@ -37,6 +37,12 @@ common-terraform:
     - src: assets/instructions/terraform/plans.instructions.md
       dest: .github/instructions/terraform-plans.instructions.md
 
+common-spread:
+  description: "Spread integration task runner skills and guidance"
+  items:
+    - src: skills/running-spread
+      dest: .github/skills/running-spread/
+
 common-documentation:
   description: "Documentation standards and review skills"
   items:

skills/running-spread/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: running-spread
+description: >-
+  Discovers, selects, executes, and debugs Spread integration test tasks
+  in repositories with spread.yaml configurations. Use when running spread
+  tests, inspecting spread.yaml matrix dimensions (backends, systems, suites,
+  tasks, variants), troubleshooting spread failures, or managing spread
+  server lifecycle.
+compatibility: spread yq
+allowed-tools: spread yq find grep env ls cat
+---
+
+# Using Spread
+
+## Scope
+
+This skill covers discovering, executing, and debugging Spread tasks.
+It does NOT cover writing new `task.yaml` files, modifying `spread.yaml` configuration, writing Spread backend plugins, configuring cloud credentials, or setting up local virtualization providers (Multipass/LXD/QEMU).
+
+---
+
+## How to Use This Skill
+
+This skill acts as a router. When you need to perform a task, read the corresponding reference guide:
+
+1. **Discover the test matrix** → Read [test-matrix-and-selectors.md](references/test-matrix-and-selectors.md) to inspect backends, systems, suites, tasks, variants, and constraints using `yq` and `find`.
+2. **Execute and iterate** → Read [command-reference.md](references/command-reference.md) for CLI flags (e.g., `-reuse`, `-resend`, `-abend`, `-workers`, `-repeat`).
+3. **Diagnose failures** → Read [failure-diagnosis-and-troubleshooting.md](references/failure-diagnosis-and-troubleshooting.md) for the error classification matrix (infrastructure vs hooks vs timeouts).
+4. **Manage servers** → Read [server-management-and-cleanup.md](references/server-management-and-cleanup.md) for lifecycle control (`-reuse -discard`, `-gc`, recovering PIDs).
+5. **Collect diagnostic output** → Read [logs-and-artifacts.md](references/logs-and-artifacts.md) for parsing `-json` results, inspecting `-logs`, and collecting `-artifacts`.
+
+---
+
+## Critical Invariant Rules
+
+> [!IMPORTANT]
+> **Always run `spread -list` before executing tasks.**
+> Before triggering any execution, you MUST run `spread -list <selector...>` to inspect the complete set of resolved task runs on `stdout`. This ensures you only provision instances and execute the exact intended set of tasks, backends, and systems, avoiding unintended blanket matrix runs.
+
+> [!IMPORTANT]
+> **Trailing Slash on Suites**: A suite name **must always end with a trailing slash (`/`)** both in `spread.yaml` and in selector expressions. Omitting the trailing slash causes Spread to treat the string as a task path instead of a suite.
+
+> [!WARNING]
+> If `spread` exits with a non-zero exit code, classify the failure before retrying.
+> Read [failure-diagnosis-and-troubleshooting.md](references/failure-diagnosis-and-troubleshooting.md) and follow the Quick Failure Classification Matrix.
+
+> [!CAUTION]
+> **Interactive Flags**: Do NOT use `-debug`, `-shell`, `-shell-before`, or `-shell-after`. These spawn interactive TTY sessions that hang agent execution indefinitely. They are for human developers only.

skills/running-spread/references/command-reference.md

@@ -0,0 +1,123 @@
+# Spread CLI Command Reference
+
+This reference covers the command-line flags, runtime options, debugging helpers, and practical workflows for executing tasks with [Spread](https://github.com/canonical/spread).
+
+> [!TIP]
+> For a detailed explanation of the matrix hierarchy (Backends, Systems, Suites, Tasks, Variants), programmatic matrix discovery with `yq`/`find`, and full selector grammar, refer to [test-matrix-and-selectors.md](test-matrix-and-selectors.md).
+
+---
+
+## 1. Command-Line Flags
+
+> [!NOTE]
+> For flags taking arguments, Spread accepts both space-separated (`-flag <val>`) and equal-separated (`-flag=<val>`) syntax.
+
+| Flag | Description | Typical Use Case |
+|---|---|---|
+| `-list` | List all matched jobs on stdout without executing them | Previewing task selection before execution |
+| `-json=<DIR>` | Save structured machine-readable task results (JSON) | Automated result parsing by agents or CI |
+| `-logs=<DIR>` | Save generated execution and communication logs to a directory | Archiving full task logs for diagnosis |
+| `-reuse` | Keep allocated instances alive for subsequent runs | Accelerating local iterative execution |
+| `-discard` | Discard/destroy any running reusable instances | Cleaning up environment after execution |
+| `-resend` | Resend project content to reused servers | Syncing updated source files to reused instance |
+| `-repeat <N>` | Repeat each selected task `N` times | Flake hunting and stability verification |
+| `-order` | Execute tasks in exact declared order without shuffling | Preventing task randomization |
+| `-seed <N>` | Seed for random task permutation | Reproducing order-dependent failures |
+| `-abend` | Stop immediately on first error without restoring/cleaning up | Preserving failure state on the backend instance |
+| `-artifacts=<DIR>` | Directory to save collected artifacts | Fetching logs, crash dumps, and outputs |
+| `-perf` | Show timestamps and task execution durations | Performance benchmarking and timing analysis |
+| `-v` / `-vv` | Verbose / debug logging output from Spread | Troubleshooting Spread allocation/setup |
+| `-workers <N>` | Number of concurrent workers per system | Tuning task parallelism |
+| `-restore` | Run only the restore scripts | Cleaning up partially executed suites |
+| `-gc` | Discard allocated servers no longer in use | Purging orphaned backend instances |
+| `-reuse-pid <PID>` | Select servers reused by a specific Spread process | Recovering from a crashed Spread run |
+
+---
+
+## 2. Practical Workflow Recipes
+
+### 1. Previewing Task Selection (`-list`)
+
+For detailed selector syntax examples across backends, systems, suites, tasks, and variants, see [test-matrix-and-selectors.md](test-matrix-and-selectors.md):
+
+```bash
+# List all jobs in the project matrix
+spread -list
+
+# List all tasks in a specific suite (suite must end with '/')
+spread -list tests/spread/commands/
+
+# List matching tasks on OpenStack for Ubuntu 24.04
+spread -list openstack:ubuntu-24.04-64:
+```
+
+### 2. Fast Local Iteration & Server Management (`-reuse`, `-resend`, `-discard`)
+
+When developing or modifying a task, avoid re-provisioning instances each run. For complete server lifecycle and garbage-collection guides, see [server-management-and-cleanup.md](server-management-and-cleanup.md):
+
+```bash
+# First run: provisions the backend instance and keeps it alive
+spread -reuse multipass:ubuntu-24.04-64:tests/spread/commands/version
+
+# Subsequent runs: syncs modified files and reuses the running machine
+spread -reuse -resend multipass:ubuntu-24.04-64:tests/spread/commands/version
+
+# When finished testing, discard the reused instance
+spread -reuse -discard
+```
+
+### 3. Preserving Failure State (`-abend`)
+
+Stop execution on the first error and prevent restoration scripts from altering the machine state. For error classification and troubleshooting steps, see [failure-diagnosis-and-troubleshooting.md](failure-diagnosis-and-troubleshooting.md):
+
+```bash
+# Halt on first failure and preserve remote machine state for inspection
+spread -abend openstack:ubuntu-24.04-64:tests/spread/commands/version
+```
+
+### 4. Hunting Flaky Tasks (`-repeat`, `-seed`, `-order`)
+
+By default, Spread shuffles task order within a suite to uncover hidden order dependencies:
+
+```bash
+# Repeat the task 10 consecutive times to verify determinism
+spread -repeat 10 multipass:ubuntu-24.04-64:tests/spread/commands/version
+
+# Execute tasks in sequential/declared order without shuffling
+spread -order tests/spread/commands/
+
+# Reproduce a specific randomized execution order using its seed
+spread -seed 42 tests/spread/commands/
+```
+
+### 5. Controlling Concurrency on Local Host (`-workers`)
+
+When running on local backends (Multipass, LXD, QEMU), limit worker parallelism to avoid CPU contention or memory exhaustion:
+
+```bash
+# Run with a single worker to avoid host overloading
+spread -workers 1 multipass:ubuntu-24.04-64:tests/spread/commands/
+```
+
+### 6. Collecting Artifacts (`-artifacts`)
+
+Download logs and output files declared under `artifacts:` in `task.yaml`. For layout details and inspection procedures, see [logs-and-artifacts.md](logs-and-artifacts.md):
+
+```bash
+spread -artifacts=./test-artifacts multipass:ubuntu-24.04-64:tests/spread/commands/version
+```
+
+### 7. Exporting Machine-Readable Results (`-json` & `-logs`)
+
+Save structured task execution results and logs for automated parsing. For schema details and diagnostic workflows, see [logs-and-artifacts.md](logs-and-artifacts.md):
+
+```bash
+# Run tasks and export structured JSON summary and log files
+spread -json=./spread-results -logs=./spread-logs multipass:ubuntu-24.04-64:tests/spread/commands/version
+```
+
+---
+
+## 3. Interactive Flags (Human Operators Only)
+
+These flags (`-debug`, `-shell`, `-shell-before`, `-shell-after`) are strictly for human debugging and spawn interactive PTY shells. As noted in the main skill router, agents must never invoke them.

skills/running-spread/references/failure-diagnosis-and-troubleshooting.md

@@ -0,0 +1,131 @@
+# Spread Failure Diagnosis & Troubleshooting Guide
+
+When a Spread run reports an error or exit failure, agents must distinguish between **Selector Syntax Errors**, **Backend Infrastructure Failures**, **Setup/Hook Failures**, **Timeouts**, and genuine **Task Execution Failures** inside `execute:`.
+
+---
+
+## 1. Quick Failure Classification Matrix
+
+| Failure Category | When It Happens | Typical Error Symptoms | Primary Action |
+|---|---|---|---|
+| **1. Selector Error** | Pre-execution | `error: nothing matches provider filter`<br>`error: invalid filter string` | Fix selector syntax (check trailing `/` for suites, verify system names in `spread.yaml`). |
+| **2. Infrastructure / Allocation** | Machine provisioning | Quota exceeded, SSH timeout, backend API error, `wait-timeout` reached. | Run `spread -gc`, check cloud credentials/proxies, or execute locally on `multipass:`/`lxd:`. |
+| **3. Lifecycle Hook Error** | Machine setup/teardown | Non-zero exit code during `prepare`, `prepare-each`, or `restore`. | Fix dependency installation, snap channels, or network proxy in suite `prepare:`. |
+| **4. Task Execution Failure** | Task execution | Non-zero exit code during task `execute:`. | Debug the application code or task script assertions. |
+| **5. Execution Timeout** | Long-running task | Task terminated after exceeding `kill-timeout` or `warn-timeout`. | Check for deadlocks, infinite loops, or increase `kill-timeout` in `task.yaml`. |
+
+---
+
+## 2. Diagnosing Each Failure Type
+
+### 1. Selector & Filter Errors
+
+Spread aborts immediately before communicating with any backend:
+
+- **Missing trailing slash on suite**:
+  ```bash
+  $ spread -list tests/spread/commands
+  error: nothing matches provider filter
+  ```
+  *Fix*: Add trailing slash -> `spread -list tests/spread/commands/`
+- **Invalid colon syntax**:
+  ```bash
+  $ spread -list ::tests/spread/commands/...
+  error: invalid filter string: "::"
+  ```
+  *Fix*: Avoid consecutive colons -> `spread -list tests/spread/commands/...`
+
+---
+
+### 2. Backend Infrastructure & Allocation Errors
+
+Infrastructure failures occur when Spread cannot allocate, boot, or establish SSH communication with the backend instance.
+
+- **Symptoms**:
+  - `cannot allocate server on backend <backend>: ...`
+  - `timeout waiting for SSH connectivity`
+  - Cloud API 401/403 (unauthorized) or quota exceeded.
+- **Diagnostic Procedure**:
+  1. **Verify Host Environment & Credentials**: When executing on cloud backends (e.g. OpenStack), ensure authentication credentials are exported in the host shell:
+     ```bash
+     env | grep -E '^(OS_|LXD_)'
+     ```
+  2. **Inspect full communication logs**:
+     ```bash
+     spread -logs=./spread-logs <selector>
+     ```
+  3. **Run garbage collection** to purge leaked or orphaned instances:
+     ```bash
+     spread -gc <backend>:
+     ```
+  4. **Verify on a local backend**: Test if the same task reproduces on a local virtualization backend (e.g. `multipass:` or `lxd:`).
+
+---
+
+### 3. Lifecycle Hook Failures (`prepare` & `restore`)
+
+Spread executes hooks in a defined lifecycle before running the task itself:
+
+`suite.prepare` -> `suite.prepare-each` -> `task.prepare` -> `task.execute`
+
+- **Symptoms**:
+  - Failure occurs before the `execute:` script runs.
+  - Apt package installation fails, snap refresh fails, or Juju bootstrap errors out.
+- **Distinguishing from Task Execution Bugs**:
+  - Hook errors indicate that the execution environment was not ready or external dependencies failed to download, not that the code executed by the task is broken.
+- **Diagnostic Procedure**:
+  - Check `./spread-logs/` to examine the failing hook script.
+  - If intermittent network flakes occur during package installation, verify proxy settings in `spread.yaml`.
+
+---
+
+### 4. Task Execution Failures (`execute:`)
+
+A task execution failure occurs when the commands inside the task's `execute:` block return a non-zero exit code.
+
+- **Symptoms**:
+  - Output shows application traceback, assertion failure, or non-zero exit from the task script.
+  - Spread triggers `suite.debug-each` (if defined) to collect diagnostics.
+- **Diagnostic Procedure**:
+  1. **Preserve the instance**: Re-run with `-abend` and `-reuse` so Spread aborts on failure without running restoration scripts and keeps the machine running:
+     ```bash
+     spread -abend -reuse -artifacts=./test-artifacts <selector>
+     ```
+  2. **Inspect artifacts**: Read output logs and reports collected in `./test-artifacts/`.
+  3. **Iterative re-execution**: Use `-reuse -resend` to test code fixes against the running instance:
+     ```bash
+     spread -reuse -resend <selector>
+     ```
+
+---
+
+### 5. Task Timeouts (`kill-timeout` & `warn-timeout`)
+
+Tasks can define timeout limits in `task.yaml` (e.g., `kill-timeout: 30m`).
+
+- **Symptoms**:
+  - Spread terminates the task process with SIGKILL and marks the job as timed out.
+- **Diagnostic Procedure**:
+  - Run with `-perf` to view timestamps for each output line:
+     ```bash
+     spread -perf <selector>
+     ```
+  - Identify whether the task hung on a blocking prompt, network request, or infinite loop.
+
+---
+
+## 3. Recommended Troubleshooting Decision Flow
+
+```text
+Spread Execution Result
+ ├── Exited immediately with syntax error?
+ │    └── Selector Error: Verify system/suite syntax (check trailing /).
+ ├── Failed during VM/container allocation or SSH connection?
+ │    └── Infrastructure Error: Run spread -gc, check quota/credentials.
+ ├── Failed during prepare / prepare-each hook?
+ │    └── Environment/Hook Error: Check proxy, package repository, or snap channels.
+ ├── Failed during execute: script?
+ │    └── Task Execution Failure: Re-run with -abend -reuse -artifacts=./test-artifacts to inspect failure state.
+ └── Task killed unexpectedly?
+      └── Timeout: Run with -perf to locate hanging command.
+```