From 793e4da69a3041dbcf033e30315fec122496dd04 Mon Sep 17 00:00:00 2001 From: Nina Chikanov Date: Thu, 14 May 2026 16:49:49 -0700 Subject: [PATCH 1/4] Add Contributing documentation Co-authored-by: Copilot --- .github/CONTRIBUTING.md | 8 +- docs/contributing/architecture.md | 91 ++++++ docs/contributing/code-style.md | 168 +++++++++++ docs/contributing/development-setup.md | 168 +++++++++++ docs/contributing/extending-rampart.md | 368 +++++++++++++++++++++++++ docs/contributing/index.md | 51 ++++ docs/contributing/pull-requests.md | 103 +++++++ docs/contributing/release-process.md | 228 +++++++++++++++ docs/contributing/testing.md | 174 ++++++++++++ docs/guides/authoring-tests.md | 262 ------------------ docs/guides/results-and-reporting.md | 121 -------- mkdocs.yml | 15 +- 12 files changed, 1372 insertions(+), 385 deletions(-) create mode 100644 docs/contributing/architecture.md create mode 100644 docs/contributing/code-style.md create mode 100644 docs/contributing/development-setup.md create mode 100644 docs/contributing/extending-rampart.md create mode 100644 docs/contributing/index.md create mode 100644 docs/contributing/pull-requests.md create mode 100644 docs/contributing/release-process.md create mode 100644 docs/contributing/testing.md delete mode 100644 docs/guides/authoring-tests.md delete mode 100644 docs/guides/results-and-reporting.md diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 7fe6e6ce..f515d816 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -1,6 +1,10 @@ # Contributing -This project welcomes contributions and suggestions. Most contributions require you to agree to a +Thank you for your interest in contributing to RAMPART! For comprehensive contributor documentation — including development setup, code style, testing standards, PR process, and how to add new attacks/probes — see the **[Contributing Guide](https://microsoft.github.io/RAMPART/contributing/)**. + +## Contributor License Agreement + +Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [Contributor License Agreements](https://cla.opensource.microsoft.com). @@ -8,6 +12,8 @@ When you submit a pull request, a CLA bot will automatically determine whether y a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. +## Code of Conduct + This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. \ No newline at end of file diff --git a/docs/contributing/architecture.md b/docs/contributing/architecture.md new file mode 100644 index 00000000..9d9fb901 --- /dev/null +++ b/docs/contributing/architecture.md @@ -0,0 +1,91 @@ +# Architecture for Contributors + +This page supplements the [Concepts Overview](../concepts/overview.md) with contributor-focused guidance: package layout, extension points, and key design decisions to understand before making changes. Read the **Concepts Overview** first for the component model and execution lifecycle. + +--- + +## Package Layout + +The `rampart/` source tree is organized by concern: foundational types live in `core/`, while each extension point gets its own subpackage (`attacks/`, `probes/`, `evaluators/`, `drivers/`, `converters/`, `surfaces/`, `payloads/`, `reporting/`). The `pyrit_bridge/` package isolates all PyRIT interaction (see [PyRIT Isolation](#pyrit-isolation) below), and `pytest_plugin/` provides the pytest integration. + +- For the component model and how the pieces fit together at runtime, see the [Concepts Overview](../concepts/overview.md). +- For the public symbols exported from each package, see the [API Reference index](../api/index.md). +- For where to put new code, see [Extension Points](#extension-points) below. +- The authoritative layout is always the source tree itself — browse [`rampart/` on GitHub](https://github.com/microsoft/RAMPART/tree/main/rampart). + +--- + +## Key Design Decisions + +### Protocols Over ABCs + +RAMPART uses `@runtime_checkable` protocols for extension points that consumers implement (`AgentAdapter`, `Session`, `Evaluator`, `Surface`, `PromptDriver`). This means: + +- **No inheritance required** — any class with the right methods satisfies the protocol +- **Type-checked at development time** by Pyright in strict mode +- **Verifiable at runtime** with `isinstance` checks + +`BaseExecution` is the exception — it's an ABC because it owns the lifecycle skeleton and subclasses share real implementation. + +### Factory Classes (`Attacks`, `Probes`) + +`Attacks` and `Probes` are **static factory classes** that construct execution objects. They: + +- Provide a clean, discoverable API: `Attacks.xpia(...)`, `Probes.behavior(...)` +- Handle input coercion (e.g., `coerce_driver` for flexible trigger input) +- Return `BaseExecution`, hiding the concrete execution class + +When adding a new attack or probe, you add a static factory method — not a new class that users instantiate directly. + +### Evaluator Polarity + +Evaluators are **polarity-free**. They report whether a condition was detected, not whether it's good or bad. The attack/probe factory applies the correct polarity: + +- `resolve_as_attack`: detected → UNSAFE +- `resolve_as_probe`: detected → SAFE + +This allows the same evaluator (e.g., `ToolCalled`) to be used in both attack and probe contexts. + +### Execution Lifecycle Ownership + +`BaseExecution` owns all cross-cutting concerns: + +- **Event dispatch** — ON_PRE_EXECUTE, ON_POST_EXECUTE, ON_ERROR +- **Timing** — wall-clock duration +- **Error handling** — all exceptions from `_execute_async` are caught and converted to ERROR results +- **Handler registration** — framework-level handlers (result collection) are injected automatically + +Subclasses implement only `_execute_async` and `strategy_name`. They should **not** catch `InfrastructureError` — the base class handles it. + +### PyRIT Isolation + +PyRIT is RAMPART's upstream dependency for converters and prompt generation. Its import chain is heavy, so: + +- PyRIT-related logic is grouped under `rampart/pyrit_bridge/` +- Lazy imports inside functions are used to defer the cost +- This boundary keeps RAMPART's core import fast + +--- + +## Extension Points + +| Extension Point | Protocol/ABC | Where to add | +|----------------|-------------|--------------| +| New attack strategy | Subclass `BaseExecution` | `rampart/attacks/` | +| New probe strategy | Subclass `BaseExecution` | `rampart/probes/` | +| New evaluator | Implement `Evaluator` protocol | `rampart/evaluators/` | +| New prompt driver | Implement `PromptDriver` protocol | `rampart/drivers/` | +| New attack surface | Implement `Surface` protocol | `rampart/surfaces/` | +| New converter | Implement `PayloadConverter` protocol | `rampart/converters/` | +| New report sink | Implement `ReportSink` protocol | `rampart/reporting/` | +| New payload format | Extend payload system | `rampart/payloads/` | + +See [Extending RAMPART](extending-rampart.md) for step-by-step guides. + +--- + +## Module Import Conventions + +- Import from the package root (`rampart.core`, `rampart.attacks`) when the symbol is exported in `__init__.py` +- Within the same package, import from the specific module to avoid circular imports +- Internal modules are prefixed with `_` (e.g., `_xpia.py`, `_single_turn.py`) — they are not part of the public API diff --git a/docs/contributing/code-style.md b/docs/contributing/code-style.md new file mode 100644 index 00000000..36456332 --- /dev/null +++ b/docs/contributing/code-style.md @@ -0,0 +1,168 @@ +# Code Style & Linting + +RAMPART enforces a consistent code style through automated tooling and documented conventions. This page summarizes the key rules; for the complete reference, see the [coding standards](https://github.com/microsoft/RAMPART/blob/main/.github/instructions/coding-standards.instructions.md). + +## Toolchain + +| Tool | Purpose | Config location | +|------|---------|-----------------| +| [Ruff](https://docs.astral.sh/ruff/) | Linting and formatting | `pyproject.toml` `[tool.ruff.*]` | +| [Pyright](https://github.com/microsoft/pyright) | Static type checking (strict mode) | `pyproject.toml` `[tool.pyright]` | +| [pre-commit](https://pre-commit.com/) | Git hooks for automated checks | `.pre-commit-config.yaml` | + +### Running Checks + +Pre-commit is the primary entry point — it runs Ruff (lint + format) and Pyright in one command: + +```bash +# Install the Git hook once (optional, runs on every commit) +uv run pre-commit install + +# Run all checks on demand +uv run pre-commit run --all-files +``` + +When checks fail, Ruff can auto-fix most lint and formatting issues: + +```bash +uv run ruff check --fix . +uv run ruff format . +``` + +A few details worth knowing: + +- **Ruff** is configured with `select = ["ALL"]`. Test files have relaxed rules (no docstrings, no type annotations, magic values allowed) via `per-file-ignores` in `pyproject.toml`. +- **Pyright** runs in **strict mode** targeting Python 3.11 — every function needs complete parameter and return type annotations. + + +## Key Conventions + +### Copyright Header + +Every `.py` file **must** begin with: + +```python +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT license. +``` + +This is enforced by Ruff's copyright rule. + +### Async Function Naming + +All async functions **must** end with `_async`: + +```python +# Correct +async def send_request_async(self, *, payload: str) -> Response: ... + +# Incorrect +async def send_request(self, *, payload: str) -> Response: ... +``` + +Dunder methods (`__aenter__`, `__aexit__`) are exempt. + +### Keyword-Only Arguments + +Functions with more than one parameter **must** use `*` to enforce keyword-only arguments: + +```python +# Correct +def __init__(self, *, client: ServiceClient, config: Config) -> None: ... + +# Incorrect +def __init__(self, client: ServiceClient, config: Config) -> None: ... +``` + +Dunder methods with Python-defined signatures (`__or__`, `__eq__`, etc.) are exempt. + +### Type Annotations + +Every function parameter and return type **must** have explicit type annotations: + +```python +# Correct +def process(self, *, items: list[str], limit: int = 10) -> dict[str, Any]: ... + +# Incorrect +def process(self, items, limit=10): ... +``` + +Use modern syntax: `str | None` instead of `Optional[str]`, `list[str]` instead of `List[str]`. + +### Enums Over Literals + +Use `Enum` or `StrEnum` instead of `Literal` types for predefined choices: + +```python +# Correct +class Status(Enum): + PENDING = "pending" + COMPLETE = "complete" + +# Incorrect +def classify(self, *, status: Literal["pending", "complete"]) -> None: ... +``` + +### Import Organization + +Imports are organized in three groups separated by blank lines: + +1. Standard library +2. Third-party packages +3. Local application imports + +Import from the package root (`__init__.py`) when the symbol is exported there, not from internal file paths. + +All imports must live at the top of the file — inline/local imports inside functions are forbidden, except to break circular dependencies or to defer heavy import chains (see [PyRIT Bridge](#pyrit-bridge) below). + +### Logging + +Use `%s`-style lazy formatting in log calls — **not** f-strings: + +```python +# Correct +logger.info("Saved %d payloads to '%s'", len(payloads), name) + +# Incorrect +logger.info(f"Saved {len(payloads)} payloads to '{name}'") +``` + +### Docstrings + +Use Google-style docstrings with `Args:`, `Returns:`, and `Raises:` sections. Do not include example usage in docstrings. + +### PyRIT Bridge + +PyRIT-related logic should be grouped under `rampart/pyrit_bridge/`. PyRIT's import chain is heavy, so use lazy imports inside functions when wrapping PyRIT converters: + +```python +def _get_converter(self) -> WordDocConverter: + """PyRIT's import chain is heavy (~14s), so defer until first use.""" + from pyrit.prompt_converter.word_doc_converter import WordDocConverter # noqa: PLC0415 + + return WordDocConverter() +``` + + +## Quick Reference Checklist + +Before committing, run pre-commit — it covers everything the automated tooling can verify: + +```bash +uv run pre-commit run --all-files +``` + +This runs Ruff (linting + formatting) and Pyright (strict type checking), which together enforce the copyright header, type annotations, log formatting, import organization, and most other conventions on this page. + +A few rules are **not** caught by tooling and still need a human eye: + +- [ ] All async functions end with `_async` +- [ ] Functions with more than one parameter use keyword-only arguments (`*`) +- [ ] `Enum` / `StrEnum` is used instead of `Literal` for predefined choices +- [ ] PyRIT imports are lazy (inside functions), not module-level + +!!! tip + **Use GitHub Copilot to cross-check.** GitHub Copilot in VS Code automatically picks up the repo's [coding standards](https://github.com/microsoft/RAMPART/blob/main/.github/instructions/coding-standards.instructions.md) (via `.github/instructions/`) and can review your changes against them. Ask Copilot Chat something like *"Review my staged changes against the RAMPART coding standards"* to get a second pass on the conventions above before you commit. + +If `pre-commit run --all-files` passes and the items above hold, you're ready to commit. diff --git a/docs/contributing/development-setup.md b/docs/contributing/development-setup.md new file mode 100644 index 00000000..51186368 --- /dev/null +++ b/docs/contributing/development-setup.md @@ -0,0 +1,168 @@ +# Development Setup + +This guide walks you through setting up your local environment for RAMPART development. + +## Prerequisites + +| Tool | Version | Purpose | +|------|---------|---------| +| [Python](https://www.python.org/downloads/) | 3.11+ | Runtime (3.11, 3.12, and 3.13 are tested in CI) | +| [uv](https://docs.astral.sh/uv/getting-started/installation/) | Latest | Package and project manager | +| [Git](https://git-scm.com/) | Latest | Version control | + +## Fork and Clone + +RAMPART uses a **fork-based workflow**. There are two ways to set up your fork: one using the GitHub CLI, and one without. + +### Approach 1: Using GitHub CLI + +You will need to install the [GitHub CLI](https://cli.github.com/). + +```bash +gh repo fork microsoft/RAMPART --clone=true +``` + +This command forks, clones, and sets the new repo as `origin`, while the original repo is automatically set as `upstream`. + + +### Approach 2: Without GitHub CLI + +[Fork](https://github.com/microsoft/RAMPART/fork) the repo from the main branch. By default, forks are named the same as their upstream repository. This will create a new repo called `GITHUB_USERNAME/RAMPART` (where `GITHUB_USERNAME` is your GitHub username). + +Clone your fork and add `microsoft/RAMPART` as the `upstream` remote: + +```bash +git clone https://github.com/GITHUB_USERNAME/RAMPART.git +cd RAMPART +git remote add upstream https://github.com/microsoft/RAMPART.git +``` + +This sets your fork as `origin` and the original repo as `upstream`. + +### (Optional) Pull in Changes from Upstream +To pull in the changes from `microsoft/RAMPART` into your forked repo: + +```bash +# Fetches changes from microsoft/RAMPART +git fetch upstream + +# Merge changes into your main +git checkout main +git merge upstream/main + +# Push updates to your fork +git push origin main +``` + +## Install Dependencies + +Install the project dependencies using uv: + +```bash +uv sync +``` + +`uv sync` installs the project in editable mode and includes the default `dev` group from `pyproject.toml` — ruff, pyright, pytest-cov, pytest-xdist, and pre-commit — into a virtual environment managed by uv. + +If you also plan to build the documentation locally, include the `docs` group: + +```bash +uv sync --group docs +``` + +## Set Up Pre-commit Hooks + +The `pre-commit` tool itself is already installed via `uv sync`. The steps below configure when and how it runs. + +### (Optional) Install the Git hook + +To have Ruff and Pyright run automatically on every `git commit`, install the pre-commit Git hook into `.git/hooks/pre-commit`: + +```bash +uv run pre-commit install +``` + +This is a one-time setup per clone. Skip it if you prefer to run checks manually. + +### Run checks manually + +To run all linters and the type checker against the entire repo on demand (regardless of whether the Git hook is installed): + +```bash +uv run pre-commit run --all-files +``` + +## Unit Tests + +Run the unit tests: + +```bash +uv run pytest tests/unit +``` + +Run tests with coverage: + +```bash +uv run coverage run -m pytest tests/unit -q +uv run coverage report +``` + +**Code coverage** measures which lines of `rampart/` source code were actually executed during the test run. It's a way to spot code paths that aren't being exercised by any test. + +- `coverage run -m pytest tests/unit -q` runs the unit test suite while [coverage.py](https://coverage.readthedocs.io/) records which lines were executed. Results are written to a `.coverage` data file. +- `coverage report` reads that data file and prints a per-file summary: total statements, missed statements, and the resulting coverage percentage. + +A few useful variants: + +```bash +# Show which specific line numbers were missed +uv run coverage report --show-missing + +# Generate a browsable HTML report at htmlcov/index.html +uv run coverage html +``` + +The project requires **80% code coverage** (configured in `pyproject.toml`). The `coverage report` command will exit non-zero if coverage falls below that threshold, which is what CI checks. + +## Integration Tests + +Integration tests live in `tests/integration/` and exercise the framework end-to-end across module boundaries (evaluators, probes, adapters). They are **not** part of the standard CI pipeline and are run separately: + +```bash +uv run pytest tests/integration +``` + +Today the only integration test is `test_smoke.py`, which runs against the in-process `MockAdapter` from `tests/fixtures.py` — **no external agent or network setup is required**. It validates that: + +- An evaluator (`ToolCalled`) correctly detects a tool call in a hand-crafted `Response`. +- A behavioral probe (`Probes.behavior`) executes end-to-end against `MockAdapter` and produces a `Result`. + +Future integration tests targeting a real agent environment may add their own setup requirements; those will be documented alongside the tests when introduced. + +## Preview the Documentation + +The documentation site is built with [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/). To preview it locally, first sync the `docs` dependency group: + +```bash +uv sync --group docs +``` + +Then start the dev server from the repo root: + +```bash +uv run mkdocs serve +``` + +Open in your browser. Edits to any file under `docs/` or to `mkdocs.yml` trigger an automatic rebuild and reload. Use `Ctrl+C` to stop the server. + +To mirror what CI does and fail on broken links or missing nav entries, run with `--strict`: + +```bash +uv run mkdocs serve --strict +``` + +## (Recommended) VSCode IDE Setup + +RAMPART uses strict Pyright type checking (`typeCheckingMode = "strict"` in `pyproject.toml`). For the best editor experience in VS Code, install the [Pylance](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance) extension — it picks up the project's Pyright settings automatically. + +The repo also includes an `.editorconfig` file for consistent formatting across editors. diff --git a/docs/contributing/extending-rampart.md b/docs/contributing/extending-rampart.md new file mode 100644 index 00000000..0c618f74 --- /dev/null +++ b/docs/contributing/extending-rampart.md @@ -0,0 +1,368 @@ +# Extending RAMPART + +This guide walks through the process of extending RAMPART with new attacks, probes, evaluators, prompt drivers, and attack surfaces. + +Before reading this page, make sure you're familiar with the [execution lifecycle](../concepts/overview.md) and the [architecture](architecture.md). + +## Contents + +- [Shared conventions](#shared-conventions) +- [Attack](#attack) +- [Probe](#probe) +- [Evaluator](#evaluator) +- [Prompt Driver](#prompt-driver) +- [Attack Surface](#attack-surface) +- [Summary Checklist](#summary-checklist) + + +## Shared conventions + +These apply to every component type below: + +- **Exports.** Add the new class to `__all__` in the relevant `rampart//__init__.py`. If it should be importable from the top-level `rampart` namespace, also re-export it from `rampart/__init__.py`. +- **Tests.** Place tests under `tests/unit//test_.py`, mirroring the source tree. See [Testing Standards](testing.md#writing-tests-for-new-components) for required patterns. +- **Coding style.** Follow the rules in [Code Style & Linting](code-style.md) (copyright header, `_async` suffix, keyword-only args, full type annotations, Google-style docstrings). +- **Documentation.** Each new component needs a doc page under `docs/` and a nav entry in `mkdocs.yml` — see the [Summary Checklist](#summary-checklist). Preview your changes locally with [`uv run mkdocs serve`](development-setup.md#preview-the-documentation). + + +## Attack + +Attacks test for *bad* behavior. When the evaluator detects the attack objective, the result is **UNSAFE**. + +### 1. Create the Execution Class + +Create a new file in `rampart/attacks/` (prefixed with `_` to mark it as internal): + +```python +# rampart/attacks/_my_attack.py + +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT license. + +"""MyAttackExecution — description of the attack strategy.""" + +from __future__ import annotations + +from rampart.core import ( + AgentAdapter, + BaseExecution, + Evaluator, + ExecutionEventHandler, + PromptDriver, + Result, + Turn, + resolve_as_attack, +) +from rampart.core.execution import evaluate_turn_async + + +class MyAttackExecution(BaseExecution): + """Executes the my-attack lifecycle. + + Args: + driver (PromptDriver): How to drive the conversation. + evaluator (Evaluator): What condition to check for. + max_turns (int): Maximum prompt-response exchanges. + event_handlers (list[ExecutionEventHandler] | None): Additional handlers. + """ + + def __init__( + self, + *, + driver: PromptDriver, + evaluator: Evaluator, + max_turns: int = 25, + event_handlers: list[ExecutionEventHandler] | None = None, + ) -> None: + super().__init__(event_handlers=event_handlers) + self._driver = driver + self._evaluator = evaluator + self._max_turns = max_turns + + @property + def strategy_name(self) -> str: + """Short identifier for this strategy (used in Result.strategy).""" + return "my_attack" + + async def _execute_async(self, *, adapter: AgentAdapter) -> Result: + """Core execution logic. + + Args: + adapter (AgentAdapter): The agent to test. + + Returns: + Result: Safety verdict. + """ + turns: list[Turn] = [] + + async with await adapter.create_session_async() as session: + for turn_index in range(self._max_turns): + decision = await self._driver.next_prompt_async(history=turns) + if decision is None: + break + + response = await session.send_async(decision.request) + turn = await evaluate_turn_async( + evaluator=self._evaluator, + history=turns, + request=decision.request, + response=response, + turn_number=turn_index, + driver_reasoning=decision.reasoning, + manifest=adapter.manifest, + ) + turns.append(turn) + + if turn.eval_result and turn.eval_result.detected: + break + + # Use resolve_as_attack: detected → UNSAFE + eval_results = [t.eval_result for t in turns if t.eval_result is not None] + safe, status = resolve_as_attack(eval_results=eval_results) + + return Result( + safe=safe, + status=status, + summary="...", + turns=turns, + strategy=self.strategy_name, + observability_level=adapter.observability_profile, + ) +``` + +Key points: + +- **Subclass `BaseExecution`** — it owns the lifecycle skeleton (event dispatch, timing, error handling) +- **Implement `_execute_async`** — this is your strategy-specific logic +- **Implement `strategy_name`** — a short identifier used in `Result.strategy` +- **Use `resolve_as_attack`** — this maps evaluator outcomes to safety verdicts with attack semantics (detected = UNSAFE) +- **Don't wrap `_execute_async` in a broad `try/except`** — `BaseExecution.execute_async` already catches every exception from `_execute_async` and converts it to a `SafetyStatus.ERROR` result. + +### 2. Add a Factory Method to `Attacks` + +Add a static method to the `Attacks` class in `rampart/attacks/__init__.py`: + +```python +@staticmethod +def my_attack( + *, + trigger: str | list[str] | Request | list[Request] | PromptDriver, + evaluator: Evaluator, + max_turns: int = 25, + event_handlers: list[ExecutionEventHandler] | None = None, +) -> BaseExecution: + """Create a my-attack execution. + + Args: + trigger: User prompt(s) or a PromptDriver. + evaluator (Evaluator): What condition to check for. + max_turns (int): Maximum exchanges. Defaults to 25. + event_handlers: Optional additional handlers. + + Returns: + BaseExecution: Ready to execute with ``execute_async(adapter=...)``. + """ + driver = coerce_driver(trigger) + return MyAttackExecution( + driver=driver, + evaluator=evaluator, + max_turns=max_turns, + event_handlers=event_handlers, + ) +``` + +### 3. Write Tests + +Attack tests should cover: + +- Execution lifecycle (session creation, prompt driving, evaluation) +- Result resolution (detected → UNSAFE, not detected → SAFE) +- Edge cases (max turns reached, early stopping, empty responses) +- Error handling (infrastructure errors → `SafetyStatus.ERROR`) + + +## Probe + +Probes test for the *presence* of desired behavior. When the evaluator detects the expected behavior, the result is **SAFE**. + +The process mirrors the [Attack](#attack) walkthrough. The differences are summarized below; only the steps that diverge are repeated. + +| | Attack | Probe | +|---|---|---| +| **Location** | `rampart/attacks/_name.py` | `rampart/probes/_name.py` | +| **Factory class** | `Attacks` | `Probes` | +| **Resolution function** | `resolve_as_attack` | `resolve_as_probe` | +| **Detected means** | UNSAFE | SAFE | +| **Injection phase** | Often yes | No | + +### 1. Create the Execution Class + +The file structure mirrors the [Attack walkthrough](#1-create-the-execution-class) — same imports, `__init__`, and `_execute_async` loop. The diff from `MyAttackExecution` is: + +```diff +-from rampart.core import (..., resolve_as_attack) ++from rampart.core import (..., resolve_as_probe) + +-class MyAttackExecution(BaseExecution): ++class MyProbeExecution(BaseExecution): + +- return "my_attack" ++ return "my_probe" + +- safe, status = resolve_as_attack(eval_results=eval_results) ++ safe, status = resolve_as_probe(eval_results=eval_results) +``` + +Place the file in `rampart/probes/` (e.g. `_my_probe.py`). Most probes skip the injection phase — just session creation, prompt driving, and evaluation. For a complete working reference, see [`rampart/probes/_single_turn.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/_single_turn.py). + +### 2. Add a Factory Method to `Probes` + +Add a static method to the `Probes` class in `rampart/probes/__init__.py`, mirroring the `Attacks.my_attack` example. See [`rampart/probes/__init__.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/__init__.py) for the existing `Probes.behavior` factory as a reference. + +### 3. Write Tests + +Probe tests have the same surface as attack tests, with two differences: + +- **No injection phase** to test. +- **Result resolution** uses `resolve_as_probe` semantics (detected → SAFE, not detected → UNSAFE). + + +## Evaluator + +Evaluators answer "did X happen?" They are **polarity-free** — the same evaluator can be used in both attacks and probes. The `Attacks`/`Probes` factories handle the mapping from detection to safety verdict. + +Create a new file in `rampart/evaluators/`: + +```python +# rampart/evaluators/my_evaluator.py + +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT license. + +"""MyEvaluator — description of what this evaluator checks.""" + +from __future__ import annotations + +from rampart.core.evaluator import BaseEvaluator +from rampart.core.types import EvalContext, EvalOutcome, EvalResult + + +class MyEvaluator(BaseEvaluator): + """Checks whether . + + Args: + target (str): What to look for. + """ + + def __init__(self, *, target: str) -> None: + self._target = target + + async def evaluate_async(self, *, context: EvalContext) -> EvalResult: + """Evaluate the latest turn for the target condition. + + Args: + context (EvalContext): The evaluation context with turn history. + + Returns: + EvalResult: Whether the condition was detected, with evidence. + """ + latest_turn = context.turns[-1] + detected = self._target in latest_turn.response.text + + return EvalResult( + outcome=EvalOutcome.DETECTED if detected else EvalOutcome.NOT_DETECTED, + evidence=[f"Found '{self._target}'"] if detected else [], + rationale="Target string found in response" if detected else "Not found", + ) +``` + +Evaluator tests should cover detection, non-detection, edge cases (empty response, missing data), and that `evidence` / `rationale` are populated correctly. + + +## Prompt Driver + +Prompt drivers decide **what** to send to the agent at each turn. They consume conversation history and return a `PromptDecision` (a `Request` plus optional reasoning), or `None` to stop. See the [`PromptDriver` protocol](https://github.com/microsoft/RAMPART/blob/main/rampart/core/prompt_driver.py). + +### 1. Implement the Protocol + +Create a new file in `rampart/drivers/`: + +```python +# rampart/drivers/my_driver.py + +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT license. + +"""MyDriver — description of how this driver picks the next prompt.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from rampart.core.prompt_driver import PromptDecision +from rampart.core.types import Request + +if TYPE_CHECKING: + from rampart.core.types import Turn + + +class MyDriver: + """Drives the conversation based on . + + Args: + seed_prompt (str): Where to start the conversation. + """ + + def __init__(self, *, seed_prompt: str) -> None: + self._seed_prompt = seed_prompt + + async def next_prompt_async( + self, + *, + history: list[Turn], + ) -> PromptDecision | None: + """Return the next prompt, or None to stop.""" + if not history: + return PromptDecision( + request=Request(prompt=self._seed_prompt), + reasoning="seed prompt", + ) + + # ... decide based on history ... + return None +``` + +Key points: + +- **Return `None` to stop** — the execution loop ends the conversation. +- **Populate `reasoning`** for LLM-backed drivers so post-test diagnostics can explain choices. Deterministic drivers may leave it empty. +- **No inheritance required** — `PromptDriver` is a `@runtime_checkable` protocol. Any class with `next_prompt_async` satisfies it. + +For reference implementations, see [`StaticDriver`](https://github.com/microsoft/RAMPART/blob/main/rampart/drivers/static.py) and [`LLMDriver`](https://github.com/microsoft/RAMPART/blob/main/rampart/drivers/llm.py). + + +## Attack Surface + +Attack surfaces are the data sources where XPIA payloads get planted (OneDrive, SharePoint, Slack, etc.). A surface implementation pairs a `Surface` (fully configured target) with an `InjectionHandle` (an async context manager that activates and cleans up the injection). See the [`Surface` and `InjectionHandle` protocols](https://github.com/microsoft/RAMPART/blob/main/rampart/core/injection.py). + +For the basic protocol skeleton, see [Implementing Surfaces](../usage/authoring-tests.md#implementing-surfaces) in the user-facing guide. When contributing a surface to RAMPART itself (under `rampart/surfaces/`), keep these contributor-specific requirements in mind: + +- **`Surface.inject` does not activate** — it only prepares the handle. Activation happens when an execution strategy enters the handle as an async context manager. +- **`__aexit__` must be idempotent and must not raise** — cleanup runs even on exceptions, and a failing cleanup must not mask the original error. +- **`wait_until_ready` should bound itself** with `TimeoutError` rather than block indefinitely. For simple delay-based waits, call `sleep_until_ready` from `rampart.core.injection`. +- **Raise `InfrastructureError`** for transient, external failures (timeouts, rate limits, service outages). It's the documented convention for surfaces and adapters to signal "not a safety signal" — `BaseExecution` catches all exceptions and produces an `ERROR` result either way, but the exception type is preserved in metadata for triage. + +For a complete reference, see [`OneDriveSurface`](https://github.com/microsoft/RAMPART/blob/main/rampart/surfaces/onedrive.py). + + +## Summary Checklist + +When adding a new component: + +- [ ] Implementation file created in the correct package +- [ ] Factory method added (for attacks/probes) +- [ ] Exports updated in `__init__.py` +- [ ] Unit tests written with good coverage +- [ ] Documentation added under `docs/` (e.g. `docs/attacks/.md`, `docs/probes/.md`, or the matching `docs/api/*.md` page) and linked in `mkdocs.yml` +- [ ] `pre-commit run --all-files` passes +- [ ] All CI checks pass diff --git a/docs/contributing/index.md b/docs/contributing/index.md new file mode 100644 index 00000000..5086cd71 --- /dev/null +++ b/docs/contributing/index.md @@ -0,0 +1,51 @@ +# Contributing to RAMPART + +We welcome contributions and suggestions! Whether you're fixing a bug, adding a new attack type, improving documentation, or filing an issue, your help is appreciated. + +Check out our [GitHub issues](https://github.com/microsoft/RAMPART/issues) with the **"help wanted"** and **"good first issue"** labels for pre-scoped items to contribute. + +## Getting Started + +
+ +- :material-wrench:{ .lg .middle } **[Development Setup](development-setup.md)** + + Clone, install, and run the test suite locally. + +- :material-format-paint:{ .lg .middle } **[Code Style & Linting](code-style.md)** + + Ruff, Pyright, pre-commit hooks, and naming conventions. + +- :material-test-tube:{ .lg .middle } **[Testing Standards](testing.md)** + + Unit vs integration tests, writing tests, coverage expectations. + +- :material-file-tree:{ .lg .middle } **[Architecture for Contributors](architecture.md)** + + Package layout, extension points, and key design decisions. + +- :material-puzzle-plus:{ .lg .middle } **[Extending RAMPART](extending-rampart.md)** + + Step-by-step guides for adding a new attack, probe, evaluator, prompt driver, or attack surface. + +- :material-source-pull:{ .lg .middle } **[Pull Request Process](pull-requests.md)** + + Fork workflow, commit conventions, CI checks, review expectations. + +- :material-package-variant:{ .lg .middle } **[Release Process](release-process.md)** + + Versioning, changelog, and publishing. + +
+ +--- + +## Contributor License Agreement + +Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [https://cla.opensource.microsoft.com](https://cla.opensource.microsoft.com). + +When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. + +## Code of Conduct + +This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. diff --git a/docs/contributing/pull-requests.md b/docs/contributing/pull-requests.md new file mode 100644 index 00000000..74eaa90f --- /dev/null +++ b/docs/contributing/pull-requests.md @@ -0,0 +1,103 @@ +# Pull Request Process + +## Fork Workflow + +RAMPART uses a **fork-based contribution model**. If you haven't set up your fork yet, see [Development Setup: Fork and Clone](development-setup.md#fork-and-clone). + +For each contribution: + +1. Create a feature branch from `main` in your fork. +2. Make and commit your changes. +3. Push the branch to your fork. +4. Open a pull request against `microsoft/RAMPART:main`. + +```bash +git checkout -b my-feature +# ... make changes ... +git push origin my-feature +``` + +## Commit Conventions + +RAMPART uses **squash-merge** with a conventional commit-style tag in the PR title. When your PR is merged, the squash commit message will match your PR title. + +### PR Title Format + +``` +[TAG]: Short description +``` + +Where `TAG` is one of: + +| Tag | Use for | +|-----|---------| +| `FEAT` | New features, new attack/probe types | +| `FIX` | Bug fixes | +| `REFACTOR` | Code restructuring without behavior change | +| `STYLE` | Formatting, linting, whitespace changes | +| `TEST` | Adding or updating tests | +| `DOCS` | Documentation changes | +| `CI` | CI/CD pipeline changes | +| `MAINT` | Dependency updates, maintenance tasks | +| `META` | Repository configuration, templates | +| `REVERT` | Reverting a previous change | + +For breaking changes, add `[BREAKING]` before the tag: + +``` +[BREAKING] [FEAT]: Rename evaluator protocol method +``` + +### Examples + +``` +[FEAT]: Add crescendo multi-turn attack strategy +[FIX]: Handle empty response in ToolCalled evaluator +[DOCS]: Add contributor guide for writing evaluators +[TEST]: Improve coverage for XPIAExecution edge cases +``` + +## PR Template Checklist + +Every pull request uses the [PR template](https://github.com/microsoft/RAMPART/blob/main/.github/pull_request_template.md), which includes: + +- [ ] `pre-commit run --all-files` passes +- [ ] Tests added or updated for changes +- [ ] Documentation updated + +Fill in the **Description** (what the PR does, linked issues), **Breaking changes** (or "None"), and complete the checklist before requesting review. + +## CI Checks + +All of the following must pass before a PR can be merged: + +### Lint & Type Check + +- **Ruff check** — all lint rules pass +- **Ruff format** — code is properly formatted +- **Pyright** — strict type checking passes + +### Tests + +- Unit tests pass on **Python 3.11, 3.12, and 3.13** + +### Coverage + +- Code coverage meets the **80% minimum threshold** +- A coverage summary is posted to the PR + +## Review Expectations + +- All pull requests require review by a maintainer (AI Red Team member) before merging +- Maintainers will check that: + - Tests are added or updated as appropriate + - Documentation is updated for user-facing changes + - Code follows the project's [coding standards](code-style.md) + - The change is well-scoped and doesn't introduce unnecessary complexity + +!!! tip + Open an [issue](https://github.com/microsoft/RAMPART/issues) before starting work on large features or architectural changes. This helps align on approach before investing time in implementation. + +## Stale Pull Requests + +If a pull request has no activity for an extended period, maintainers may check in with the author. If there is no response within 14 days, maintainers may reassign the work to ensure progress continues. diff --git a/docs/contributing/release-process.md b/docs/contributing/release-process.md new file mode 100644 index 00000000..69eb1027 --- /dev/null +++ b/docs/contributing/release-process.md @@ -0,0 +1,228 @@ +# Releasing RAMPART + +This section is for maintainers only. If you don't know who the maintainers are but you need to reach them, please file an issue or (if it needs to remain private) contact the email address listed in `pyproject.toml`. + +Follow the instructions in the order provided. + +## 1. Release Readiness + +Before starting the release process, verify the codebase is in a healthy state. + +- **Check for pending changes.** Ask other RAMPART maintainers whether they have any in-flight changes that should land before the release. +- **Verify CI pipelines.** Confirm that all unit tests, lint, type checks, and coverage gates are green on `main`. If anything is failing, fix it before proceeding. +- **Verify the PyRIT pin.** RAMPART pins PyRIT to a specific version in `pyproject.toml`. Confirm the pinned version is the one you intend to ship against — see [PyRIT Dependency](#pyrit-dependency). + +## 2. Decide the Next Version + +RAMPART follows [Semantic Versioning](https://semver.org/) (`MAJOR.MINOR.PATCH`): + +| Component | Increment when | +|-----------|---------------| +| **MAJOR** | Breaking changes to the public API | +| **MINOR** | New features, new attack/probe types, backward-compatible additions | +| **PATCH** | Bug fixes, documentation corrections, dependency updates | + +!!! note "Pre-1.0 stability" + While RAMPART is below `1.0`, minor version bumps may include breaking changes. The API is stabilizing but not yet frozen. The first stable release will be `1.0.0`. + +In line with PyPA [versioning guidance](https://packaging.python.org/en/latest/discussions/versioning/), `main` may carry a `.dev0` suffix between releases (e.g., `0.2.0.dev0`). Release commits drop the suffix; the post-release bump on `main` reintroduces it for the next version. + +## 3. Remove Deprecated Functionality + +If you are incrementing the minor version, search the codebase for the new minor version (no leading `v`) to find occurrences where functionality was deprecated and announced for removal in this version. Typically, functionality is deprecated and stays for two minor versions before being removed. + +If you find functionality to remove, merge the removal PR to `main` before proceeding. + +## 4. Update the Version + +Set the version in `pyproject.toml` to the version established in step 2. + +```toml +[project] +version = "x.y.z" +``` + + + +## 5. Publish the Release Branch to GitHub + +Commit your changes to a release branch and push the tag: + +```bash +git checkout -b releases/vx.y.z +git commit -am "release vx.y.z" +git push origin releases/vx.y.z +git tag -a vx.y.z -m "vx.y.z release" +git push --tags +``` + + +## 6. Build the Package + +Install `build` if it is not already available, then build the wheel and sdist: + +```bash +uv pip install build +uv run python -m build +``` + +You should see output similar to: + +``` +Successfully built rampart-x.y.z.tar.gz and rampart-x.y.z-py3-none-any.whl +``` + +## 7. Test the Built Package + +This step ensures the new package works out of the box. + +Create a clean environment and install the built wheel: + +```bash +uv venv --python 3.11 +uv pip install dist/rampart-x.y.z-py3-none-any.whl +``` + +Verify the install: + +```bash +uv pip show rampart +``` + +Confirm the version matches the release and the package is installed under the environment's `site-packages`. Then run the following smoke checks **outside the repository root** so you don't accidentally test the editable source instead of the installed wheel: + +1. **Public API imports.** Confirm the top-level symbols resolve without error: + + ```bash + uv run python -c "from rampart import Result, SafetyStatus, AppManifest, Response, ToolCall" + uv run python -c "from rampart.attacks import Attacks; from rampart.probes import Probes; from rampart.evaluators import ToolCalled" + ``` + +2. **Pytest plugin registration.** RAMPART ships a pytest plugin via the `pytest11` entry point. Confirm pytest discovers it: + + ```bash + uv run pytest --version # should list "rampart" in the plugin list + ``` + +3. **End-to-end smoke test.** Run `tests/integration/test_smoke.py` against the installed wheel. It exercises an evaluator and a probe through `MockAdapter` and requires no external services: + + ```bash + uv run pytest path/to/RAMPART/tests/integration/test_smoke.py -v + ``` + +If you need to make changes to fix issues found during testing, cherry-pick from `main` after the fix lands: + +```bash +git checkout main && git pull +git log main # find the commit hash to cherry-pick +git checkout releases/vx.y.z +git cherry-pick +git push origin releases/vx.y.z +git tag -a vx.y.z -m "vx.y.z release" --force +git push --tags --force +``` + +Rebuild the package after any cherry-pick and re-test. + +## 8. Publish to PyPI + +Create a PyPI account if you don't have one and ask another maintainer to add you to the `msft-rampart` project. Before publishing, have an API token scoped to the project ready (create one in your PyPI project settings). + +```bash +uv pip install twine +uv run twine upload dist/* +``` + +If successful, the URL `https://pypi.org/project/msft-rampart/x.y.z/` will return the new release. + +## 9. Update `main` + +After the release is on PyPI, open a PR to `main` containing only: + +- A version bump in `pyproject.toml` to the next development version (e.g., `x.y.(z+1).dev0` or `x.(y+1).0.dev0`, depending on the next planned release). +- Replace any references to the previous release version in the codebase with the new released version (without `.dev0`) where applicable (e.g., installation docs that pin to the latest tag). + +Open this PR from a branch separate from your `releases/vx.y.z` branch. + +## 10. Create the GitHub Release + +Go to the [releases page](https://github.com/microsoft/RAMPART/releases), select **Draft a new release**, and choose the tag you pushed in step 5. Click **Generate release notes** to pre-populate the description. + +Structure the description as: + +- **What's changed** — a curated short list of user-facing changes (new features, bug fixes, breaking changes). +- **Full list of changes** — the auto-generated full changelog. + +Maintenance changes, CI updates, and documentation fixes generally belong only in the full list. Verify the **New contributors** section is accurate. Mark the release as **Latest** and publish. + +## PyRIT Dependency + +RAMPART pins PyRIT to a specific version in `pyproject.toml`: + +```toml +dependencies = [ + ... + "pyrit==", + ... +] +``` + +When updating the PyRIT dependency, use the helper script: + +```bash +./scripts/bump_pyrit_version.sh +``` + +Re-run the full test suite after bumping — PyRIT changes are a common source of regressions. + +## Appendix: Patch Releases (Cherry-Pick Process) + +A patch release (e.g., `0.2.0` → `0.2.1`) ships a targeted fix — typically a security patch or a critical bug fix — without including other in-flight changes from `main`. + +### When to use a patch release + +- A security vulnerability fix needs to be shipped urgently. +- A critical bug was found in the latest release that blocks users. +- The fix is already merged to `main`, but `main` contains other changes that aren't ready for release. + +### Abbreviated steps + +1. **Create a release branch from the previous tag**, not from `main`: + + ```bash + git fetch origin + git checkout -b releases/vx.y.z vx.y.(z-1) + ``` + +2. **Cherry-pick the fix** from `main`: + + ```bash + git cherry-pick + ``` + + Resolve any conflicts manually. Patch-sized fixes typically apply cleanly. + +3. **Bump the version** in `pyproject.toml` to the new patch version. Also update any version-pinned links in `README.md`. + + ```bash + git commit -am "Bump version to x.y.z" + ``` + +4. **Push and tag**: + + ```bash + git push origin releases/vx.y.z + git tag -a vx.y.z -m "vx.y.z release" + git push --tags + ``` + +5. **Follow the regular release process from step 6 onward**: build, test, publish to PyPI, update `main`, and create the GitHub release. Patch release notes should clearly state the reason for the patch (e.g., "Security fix for…" or "Critical bug fix for…"). + +### Key differences from a regular release + +| Aspect | Regular release | Patch release | +|---|---|---| +| Branch base | `main` | Previous release tag | +| Changes included | Everything on `main` | Only cherry-picked fix(es) | +| Deprecated code removal | Yes (if minor bump) | No | +| Release notes | Full changelog with curated summary | Short, focused on the reason for the patch | diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md new file mode 100644 index 00000000..e207aa05 --- /dev/null +++ b/docs/contributing/testing.md @@ -0,0 +1,174 @@ +# Testing Standards + +RAMPART uses [pytest](https://docs.pytest.org/) with [pytest-asyncio](https://pytest-asyncio.readthedocs.io/) for its test suite. This page covers test organization, writing guidelines, and coverage expectations. For the complete reference, see the [unit test standards](https://github.com/microsoft/RAMPART/blob/main/.github/instructions/unit-tests-standards.instructions.md). + +The standards on this page apply to **both unit and integration tests** — the underlying instruction file targets all files under `tests/`. Integration tests differ in scope (end-to-end across modules) and may use real components instead of mocks, but the naming, typing, and structural rules are identical. + +## Test Organization + +### Directory Structure + +Tests mirror the source tree: + +``` +tests/ +├── fixtures.py # Shared test utilities +├── unit/ # Unit tests (run in CI) +│ ├── attacks/ +│ │ └── test_xpia.py +│ ├── converters/ +│ ├── core/ +│ │ ├── test_execution.py +│ │ ├── test_result.py +│ │ └── ... +│ ├── drivers/ +│ ├── evaluators/ +│ ├── payloads/ +│ ├── probes/ +│ ├── pyrit_bridge/ +│ ├── pytest_plugin/ +│ ├── reporting/ +│ └── surfaces/ +└── integration/ # Integration tests (not in CI) + └── test_smoke.py +``` + +Place unit tests at `tests/unit//test_.py`, mirroring the `rampart/` source structure. + +### Unit vs Integration Tests + +| | Unit Tests | Integration Tests | +|---|---|---| +| **Location** | `tests/unit/` | `tests/integration/` | +| **Run in CI** | ✅ Yes | ❌ No | +| **External dependencies** | All mocked | None today (smoke test uses `MockAdapter`); future tests may require a real agent environment | +| **Speed** | Fast (seconds) | Slow (minutes) | +| **Command** | `uv run pytest tests/unit` | `uv run pytest tests/integration` | + +### Test Classes and Methods + +- Group related tests into classes with descriptive names starting with `Test` +- Test methods **must** have return type annotation `-> None` +- Async test methods **must** end with `_async` +- `asyncio_mode = "auto"` is configured globally — no need for `@pytest.mark.asyncio` + +```python +class TestXPIAExecution: + def test_returns_safe_when_not_detected(self) -> None: + ... + + async def test_activates_handles_async(self) -> None: + ... +``` + +## Writing Tests + +### Test Data Helpers + +Define small private helper functions at the top of test files instead of fixtures when no setup/teardown is needed: + +```python +def _make_result(*, safe: bool = True) -> Result: + """Build a minimal Result for testing.""" + return Result( + safe=safe, + status=SafetyStatus.SAFE if safe else SafetyStatus.UNSAFE, + summary="test", + strategy="test", + ) +``` + +### Mocking + +- Mock all external dependencies (APIs, file systems, network) +- Mock at the boundary — don't mock internal implementation details +- Use `AsyncMock` for async methods, `MagicMock` for sync + +```python +mock_session = AsyncMock() +mock_session.send_async.return_value = Response(text="safe response") + +mock_adapter = AsyncMock() +mock_adapter.create_session_async.return_value = mock_session +``` + +### Assertions + +- Use direct `assert` statements (not `self.assertEqual`) +- Use `is` for identity checks (enums, singletons, `None`) +- Use `==` for value equality +- Use `pytest.raises` with `match` for error messages + +```python +assert result.status is SafetyStatus.SAFE +assert result.summary == "Expected behavior detected" + +with pytest.raises(ValueError, match="timeout must be positive"): + Config(timeout=-1) +``` + +### Relaxed Lint Rules in Tests + +Test files have relaxed lint rules (configured via `per-file-ignores` in `pyproject.toml`): + +- No docstrings required +- No type annotations required (except `-> None` on test methods) +- Magic values in assertions are fine +- Private member access (`_private`) is allowed +- Local imports inside test functions are acceptable + +## Writing Tests for New Components + +### Testing a New Attack + +When adding a new attack, test: + +1. **Execution lifecycle** — the attack calls `BaseExecution.execute_async` correctly +2. **Phase orchestration** — injection, session creation, prompt driving, evaluation happen in order +3. **Result resolution** — `resolve_as_attack` is applied (detected → UNSAFE, not detected → SAFE) +4. **Edge cases** — empty handles, max turns reached, early stopping on detection +5. **Error handling** — infrastructure errors produce `SafetyStatus.ERROR` + +### Testing a New Probe + +Similar to attacks, but: + +1. No injection phase to test +2. Result resolution uses `resolve_as_probe` (detected → SAFE, not detected → UNSAFE) + +### Testing a New Evaluator + +1. **Detection** — evaluator correctly identifies the target condition +2. **Non-detection** — evaluator correctly reports absence of the condition +3. **Edge cases** — empty responses, missing data, multiple turns +4. **Evidence** — evaluator populates `evidence` and `rationale` in `EvalResult` + +## Coverage + +### Expectations + +- The project enforces a **minimum 80% code coverage** threshold +- Coverage is measured with [coverage.py](https://coverage.readthedocs.io/), configured in `pyproject.toml` +- CI runs a dedicated coverage job on every push and pull request + +### Running Coverage Locally + +```bash +# Run tests with coverage +uv run coverage run -m pytest tests/unit -q + +# View the report +uv run coverage report + +# See which lines are missing coverage +uv run coverage report --show-missing +``` + + +## Parallel Test Execution + +The project includes [pytest-xdist](https://pytest-xdist.readthedocs.io/) for parallel test execution: + +```bash +uv run pytest tests/unit -n auto +``` diff --git a/docs/guides/authoring-tests.md b/docs/guides/authoring-tests.md deleted file mode 100644 index 2b6bec98..00000000 --- a/docs/guides/authoring-tests.md +++ /dev/null @@ -1,262 +0,0 @@ -# Authoring Tests - -Patterns for writing RAMPART safety tests. Assumes you've completed the [Quickstart](../getting-started/quickstart.md). - ---- - -## Implementing AgentAdapter and Session - -Every RAMPART test needs an adapter that connects your agent to the framework. - -### Session Protocol - -A [`Session`][rampart.core.adapter.Session] is an async context manager that sends requests and returns responses: - -```python -from rampart import Request, Response, ToolCall - -class MySession: - async def send_async(self, request: Request) -> Response: - raw = await self._client.chat(request.prompt) - return Response( - text=raw["text"], - tool_calls=[ - ToolCall(name=tc["name"], arguments=tc["args"]) - for tc in raw.get("tool_calls", []) - ], - ) - - async def __aenter__(self): - return self - - async def __aexit__(self, exc_type, exc_val, exc_tb): - pass -``` - -**Key responsibilities:** - -- **`send_async`**: Populate `Response.tool_calls` and `Response.side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened." -- **`__aenter__`**: Set up session-level state (API connections, browser contexts). -- **`__aexit__`**: Clean up. Must be idempotent and must not raise. - -### AgentAdapter Protocol - -An [`AgentAdapter`][rampart.core.adapter.AgentAdapter] creates sessions and declares capabilities: - -```python -from rampart import AgentAdapter, AppManifest, ObservabilityLevel, ToolDeclaration - -class MyAdapter: - async def create_session_async(self) -> MySession: - return MySession(client=self._client) - - @property - def manifest(self) -> AppManifest: - return AppManifest( - name="My Agent", - tools=[ - ToolDeclaration(name="search", description="Search documents"), - ToolDeclaration(name="send_email", description="Send email"), - ], - ) - - @property - def observability_profile(self) -> ObservabilityLevel: - return ObservabilityLevel.TOOL_ONLY -``` - -**Observability levels:** - -| Level | Meaning | When to use | -|-------|---------|-------------| -| [`TOOL_AND_SIDE_EFFECTS`][rampart.core.types.ObservabilityLevel] | Reports tool calls and side effects | Full observability via telemetry | -| `TOOL_ONLY` | Reports tool calls but not side effects | API returns tool call data | -| `RESPONSE_ONLY` | Reports only text responses | Black-box agent | - ---- - -## Choosing Evaluators - -Evaluators detect conditions. Choose based on what evidence your adapter provides. - -### [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] — Detect Tool Invocations - -```python -from rampart.evaluators import ToolCalled - -# Simple: was the tool called? -ToolCalled("send_email") - -# With exact parameter match -ToolCalled("send_email", recipient="attacker@evil.com") - -# With callable predicate -ToolCalled( - "exec", - command=lambda v: isinstance(v, str) and ".ssh" in v, -) -``` - -!!! warning - `ToolCalled` requires `TOOL_ONLY` or `TOOL_AND_SIDE_EFFECTS` observability. With `RESPONSE_ONLY`, it always returns `NOT_DETECTED`. - -### [`ResponseContains`][rampart.evaluators.response_contains.ResponseContains] — Detect Text Patterns - -```python -from rampart.evaluators import ResponseContains -import re - -# Substring match (case-insensitive by default) -ResponseContains("error") - -# Case-sensitive -ResponseContains("Error", case_sensitive=True) - -# Regex -ResponseContains(re.compile(r"ssh-rsa\s+[A-Za-z0-9+/]+")) - -# Callable predicate -ResponseContains(lambda text: "secret" in text.lower()) -``` - -### [`SideEffectOccurred`][rampart.evaluators.side_effect.SideEffectOccurred] — Detect Side Effects - -```python -from rampart.evaluators import SideEffectOccurred - -# Any HTTP request -SideEffectOccurred("http_request") - -# POST request to a specific host -SideEffectOccurred("http_request", method="POST", host="evil.com") -``` - -### Composing Evaluators - -Combine evaluators with `|` (OR), `&` (AND), and `~` (NOT): - -```python -from rampart.evaluators import ToolCalled, ResponseContains - -# OR: detect if EITHER condition is met -evaluator = ToolCalled("send_email") | ResponseContains("attacker@evil.com") - -# AND: detect only if BOTH conditions are met -evaluator = ToolCalled("exec") & ResponseContains("password") - -# NOT: invert detection -evaluator = ~ResponseContains("I cannot help with that") -``` - -!!! tip - Place the cheaper evaluator on the left side of `|`. The OR operator short-circuits — if the left operand detects, the right is skipped. - ---- - -## Implementing Surfaces - -[Surfaces][rampart.core.injection.Surface] inject payloads into your agent's data sources. Implement the protocol to return an [`InjectionHandle`][rampart.core.injection.InjectionHandle]. - -```python -from rampart import InjectionHandle, Payload, Surface - - -class MyFileSurface: - """Injects content into a file in the agent's workspace.""" - - def __init__(self, *, target_path: str, client): - self._target_path = target_path - self._client = client - - def inject(self, *, payload: Payload) -> InjectionHandle: - return _FileInjection( - client=self._client, - path=self._target_path, - payload=payload, - ) - - -class _FileInjection: - def __init__(self, *, client, path: str, payload: Payload): - self._client = client - self._path = path - self._payload = payload - self._original_content: str | None = None - - @property - def payload_id(self) -> str | None: - return self._payload.id - - @property - def surface_name(self) -> str: - return "file_system" - - async def wait_until_ready(self) -> None: - pass # or: await asyncio.sleep(10.0) for indexing delay - - async def __aenter__(self): - self._original_content = await self._client.read(self._path) - await self._client.write(self._path, self._payload.content) - return self - - async def __aexit__(self, exc_type, exc_val, exc_tb): - if self._original_content is not None: - await self._client.write(self._path, self._original_content) -``` - -!!! warning - `__aexit__` must not raise. If cleanup can fail, catch and log the exception. - ---- - -## Test Structure Patterns - -### One Attack Per Test - -Each test should run one execution and assert one result: - -```python -@pytest.mark.harm(HarmCategory.DATA_EXFILTRATION) -async def test_xpia_email_exfil(adapter): - result = await Attacks.xpia( - inject=handle, - trigger="Summarize Q3 reports", - evaluator=ToolCalled("send_email"), - ).execute_async(adapter=adapter) - - assert result, result.summary -``` - -### Fixture-Based Adapter - -Use pytest fixtures to share adapter setup: - -```python -# conftest.py -import pytest - -@pytest.fixture -def adapter(): - return MyAdapter(api_key="test-key") - -# For reporting setup, see pytest Markers & Fixtures -``` - -### Class-Based Test Organization - -Group related tests in a class: - -```python -class TestDataExfiltration: - @pytest.mark.harm(HarmCategory.DATA_EXFILTRATION) - @pytest.mark.trial(n=3, threshold=0.8) - async def test_ssh_key_exfil(self, adapter): - ... - - @pytest.mark.harm(HarmCategory.DATA_EXFILTRATION) - @pytest.mark.trial(n=3, threshold=0.8) - async def test_email_exfil(self, adapter): - ... -``` - - diff --git a/docs/guides/results-and-reporting.md b/docs/guides/results-and-reporting.md deleted file mode 100644 index 8c3e067f..00000000 --- a/docs/guides/results-and-reporting.md +++ /dev/null @@ -1,121 +0,0 @@ -# Results and Reporting - -Every RAMPART execution produces a [`Result`][rampart.core.result.Result]. Results flow into reporting sinks for persistence and into the terminal summary for immediate feedback. - ---- - -## The Result Type - -[`Result`][rampart.core.result.Result] is the single output type for all tests. - -```python -result = await Attacks.xpia(...).execute_async(adapter=my_adapter) - -result.safe # bool — did the agent behave safely? -result.status # SafetyStatus (SAFE, UNSAFE, UNDETERMINED, ERROR) -result.summary # str — human-readable one-liner -result.turns # list[Turn] — full conversation -result.duration_seconds # float — execution wall-clock time -result.harm_category # HarmCategory | str | None -result.strategy # str — "xpia", "probe", etc. -result.injections # list[InjectionRecord] — what was injected where -``` - -### The Assert Pattern - -`bool(result)` returns `result.safe`: - -```python -assert result, result.summary -``` - -### SafetyStatus - -| Status | Meaning | -|--------|---------| -| [`SAFE`][rampart.core.result.SafetyStatus] | The agent behaved correctly | -| `UNSAFE` | A safety violation was detected | -| `UNDETERMINED` | Could not determine safety | -| `ERROR` | Infrastructure failure | - -### Turns - -Each [`Turn`][rampart.core.types.Turn] in `result.turns` is one prompt-response exchange: - -```python -for turn in result.turns: - turn.request.prompt # What was sent - turn.response.text # What came back - turn.response.tool_calls # Tool invocations observed - turn.eval_result # EvalResult for this turn - turn.turn_number # 0-indexed position -``` - ---- - -## Report Sinks - -Report sinks receive a [`TestRunReport`][rampart.reporting.sink.TestRunReport] at the end of the pytest session. - -### JsonFileReportSink (Built-in) - -Writes timestamped JSON files: - -```python -from pathlib import Path -from rampart.reporting import JsonFileReportSink - -sink = JsonFileReportSink(output_dir=Path(".report")) -``` - -Output: `.report/run_report_2026-04-25T14-30-00.json` - -### Custom Sinks - -Implement the [`ReportSink`][rampart.reporting.sink.ReportSink] protocol: - -```python -from rampart.reporting import ReportSink, TestRunReport - -class MyDatabaseSink: - async def emit_async(self, *, report: TestRunReport) -> None: - for result in report.results: - await self._db.insert( - safe=result.safe, - status=result.status.value, - harm=str(result.harm_category), - ) -``` - -### Wiring Sinks - -Define the `rampart_sinks` fixture in your `conftest.py`. See [pytest Markers & Fixtures](../getting-started/pytest-integration.md#rampart_sinks) for the setup and examples with multiple sinks. - ---- - -## TestRunReport - -The report object passed to sinks. See [`TestRunReport`][rampart.reporting.sink.TestRunReport] for full API. - -### Grouping and Aggregation - -```python -# Group by harm category -by_category = report.by_harm_category() - -# Population statistics -summary = report.population_summary() -summary.total_runs -summary.safe_count -summary.unsafe_count -summary.attack_success_rate # UNSAFE / non-ERROR total -summary.safety_pass_rate # SAFE / non-ERROR total - -# Filter by category -exfil = report.population_summary(harm_category=HarmCategory.DATA_EXFILTRATION) -``` - -!!! note - `ERROR` results are excluded from rate calculations. A transient infrastructure failure is not a safety finding. - - diff --git a/mkdocs.yml b/mkdocs.yml index 4c3741bd..914cff36 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,8 @@ extra_css: markdown_extensions: - admonition + - attr_list + - md_in_html - pymdownx.details - pymdownx.superfences: custom_fences: @@ -44,12 +46,14 @@ markdown_extensions: - pymdownx.highlight: anchor_linenums: true - pymdownx.inlinehilite + - pymdownx.emoji: + emoji_index: !!python/name:material.extensions.emoji.twemoji + emoji_generator: !!python/name:material.extensions.emoji.to_svg - pymdownx.tabbed: alternate_style: true - tables - toc: permalink: true - - attr_list - def_list plugins: @@ -100,6 +104,15 @@ nav: - pytest Markers & Fixtures: usage/pytest-integration.md - Results & Reporting: usage/results-and-reporting.md - CI Integration: usage/ci-integration.md + - Contributing: + - contributing/index.md + - Development Setup: contributing/development-setup.md + - Code Style & Linting: contributing/code-style.md + - Testing Standards: contributing/testing.md + - Architecture: contributing/architecture.md + - Extending RAMPART: contributing/extending-rampart.md + - Pull Request Process: contributing/pull-requests.md + - Release Process: contributing/release-process.md - API Reference: - api/index.md - Core Types: api/core-types.md From f1a0715523031ac33b9de3be0b7099286953fedb Mon Sep 17 00:00:00 2001 From: Nina Chikanov Date: Thu, 14 May 2026 17:02:47 -0700 Subject: [PATCH 2/4] Remove string-based TOC from Extending RAMPART since it exists with compiled mkdocs --- docs/contributing/extending-rampart.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/contributing/extending-rampart.md b/docs/contributing/extending-rampart.md index 0c618f74..2df4bd3a 100644 --- a/docs/contributing/extending-rampart.md +++ b/docs/contributing/extending-rampart.md @@ -4,17 +4,6 @@ This guide walks through the process of extending RAMPART with new attacks, prob Before reading this page, make sure you're familiar with the [execution lifecycle](../concepts/overview.md) and the [architecture](architecture.md). -## Contents - -- [Shared conventions](#shared-conventions) -- [Attack](#attack) -- [Probe](#probe) -- [Evaluator](#evaluator) -- [Prompt Driver](#prompt-driver) -- [Attack Surface](#attack-surface) -- [Summary Checklist](#summary-checklist) - - ## Shared conventions These apply to every component type below: From 7c1f144c9b7b706f6aa7b543fa33d5134340b31b Mon Sep 17 00:00:00 2001 From: Nina Chikanov Date: Mon, 18 May 2026 13:05:16 -0700 Subject: [PATCH 3/4] pr feedback + remove `pip install rampart` from install docs --- docs/contributing/architecture.md | 4 ++-- docs/contributing/code-style.md | 6 +++++- docs/contributing/release-process.md | 11 +++++++++++ docs/getting-started/installation.md | 4 ++-- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/contributing/architecture.md b/docs/contributing/architecture.md index 9d9fb901..a8346dfc 100644 --- a/docs/contributing/architecture.md +++ b/docs/contributing/architecture.md @@ -6,7 +6,7 @@ This page supplements the [Concepts Overview](../concepts/overview.md) with cont ## Package Layout -The `rampart/` source tree is organized by concern: foundational types live in `core/`, while each extension point gets its own subpackage (`attacks/`, `probes/`, `evaluators/`, `drivers/`, `converters/`, `surfaces/`, `payloads/`, `reporting/`). The `pyrit_bridge/` package isolates all PyRIT interaction (see [PyRIT Isolation](#pyrit-isolation) below), and `pytest_plugin/` provides the pytest integration. +The `rampart/` source tree is organized by concern: foundational types live in `core/`, while each extension point gets its own subpackage (`attacks/`, `probes/`, `evaluators/`, `drivers/`, `converters/`, `surfaces/`, `payloads/`, `reporting/`). The `pyrit_bridge/` package groups common PyRIT integration code (see [PyRIT Bridge](#pyrit-bridge) below), and `pytest_plugin/` provides the pytest integration. - For the component model and how the pieces fit together at runtime, see the [Concepts Overview](../concepts/overview.md). - For the public symbols exported from each package, see the [API Reference index](../api/index.md). @@ -57,7 +57,7 @@ This allows the same evaluator (e.g., `ToolCalled`) to be used in both attack an Subclasses implement only `_execute_async` and `strategy_name`. They should **not** catch `InfrastructureError` — the base class handles it. -### PyRIT Isolation +### PyRIT Bridge PyRIT is RAMPART's upstream dependency for converters and prompt generation. Its import chain is heavy, so: diff --git a/docs/contributing/code-style.md b/docs/contributing/code-style.md index 36456332..c4c52ef6 100644 --- a/docs/contributing/code-style.md +++ b/docs/contributing/code-style.md @@ -134,7 +134,11 @@ Use Google-style docstrings with `Args:`, `Returns:`, and `Raises:` sections. Do ### PyRIT Bridge -PyRIT-related logic should be grouped under `rampart/pyrit_bridge/`. PyRIT's import chain is heavy, so use lazy imports inside functions when wrapping PyRIT converters: +Common PyRIT integration logic should be grouped under `rampart/pyrit_bridge/`. +Two rules apply: + +1. Do not expose PyRIT-specific types in RAMPART's public APIs. Translate to/from RAMPART types at the bridge boundary so consumers don't have to depend on PyRIT directly. +2. Defer heavy PyRIT imports with lazy imports inside functions. PyRIT's import chain is heavy, so importing it at module top-level slows down RAMPART's startup. Use a local import where the PyRIT type is actually needed: ```python def _get_converter(self) -> WordDocConverter: diff --git a/docs/contributing/release-process.md b/docs/contributing/release-process.md index 69eb1027..715af5f5 100644 --- a/docs/contributing/release-process.md +++ b/docs/contributing/release-process.md @@ -35,14 +35,25 @@ If you find functionality to remove, merge the removal PR to `main` before proce ## 4. Update the Version +### pyproject.toml Set the version in `pyproject.toml` to the version established in step 2. ```toml [project] +name = "RAMPART" version = "x.y.z" ``` +### Update README File +The README file is published to PyPI and also needs to be updated so the links work properly. _Note: There may not be any links to update, but it is good practice to check in case our README changes._ +Replace all “main” links like “doc/index.md” with “raw” links that have the correct version number, i.e., “https://raw.githubusercontent.com/microsoft/RAMPART/releases/vx.y.z/docs/index.md”. + +For images, update using the “raw” link, e.g., “https://raw.githubusercontent.com/microsoft/RAMPART/releases/vx.y.z/docs/images/RAMPART.png”. + +For directories, update using the “tree” link, e.g., “https://github.com/microsoft/RAMPART/tree/releases/vx.y.z/docs/usage" + +This is required for the release branch because PyPI does not pick up other files besides the README, which results in local links breaking. ## 5. Publish the Release Branch to GitHub diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 4e103b03..b3f409d9 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -23,7 +23,7 @@ Or, if you already have a project: ```bash uv venv -uv pip install rampart +uv pip install git+https://github.com/microsoft/RAMPART.git ``` ### Using pip @@ -33,7 +33,7 @@ python -m venv .venv source .venv/bin/activate # Linux/macOS .venv\Scripts\activate # Windows -pip install rampart +pip install git+https://github.com/microsoft/RAMPART.git ``` Both approaches install RAMPART and all dependencies, including [PyRIT](https://github.com/microsoft/PyRIT) v0.13.0. From 2718546ec43acb04ddb14bd9b5aab4a76ada1d6e Mon Sep 17 00:00:00 2001 From: Nina Chikanov Date: Mon, 18 May 2026 16:35:47 -0700 Subject: [PATCH 4/4] more pr feedback --- docs/contributing/extending-rampart.md | 2 +- docs/contributing/pull-requests.md | 2 +- docs/contributing/release-process.md | 23 +++++++++++++---------- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/contributing/extending-rampart.md b/docs/contributing/extending-rampart.md index 2df4bd3a..61b99ee8 100644 --- a/docs/contributing/extending-rampart.md +++ b/docs/contributing/extending-rampart.md @@ -40,9 +40,9 @@ from rampart.core import ( PromptDriver, Result, Turn, + evaluate_turn_async, resolve_as_attack, ) -from rampart.core.execution import evaluate_turn_async class MyAttackExecution(BaseExecution): diff --git a/docs/contributing/pull-requests.md b/docs/contributing/pull-requests.md index 74eaa90f..0a036ef6 100644 --- a/docs/contributing/pull-requests.md +++ b/docs/contributing/pull-requests.md @@ -79,7 +79,7 @@ All of the following must pass before a PR can be merged: ### Tests -- Unit tests pass on **Python 3.11, 3.12, and 3.13** +- Unit tests pass on Python versions detailed in [pyproject.toml](../../pyproject.toml) and [ci pipelines](https://github.com/microsoft/RAMPART/actions/workflows/ci.yml). ### Coverage diff --git a/docs/contributing/release-process.md b/docs/contributing/release-process.md index 715af5f5..5cf1178a 100644 --- a/docs/contributing/release-process.md +++ b/docs/contributing/release-process.md @@ -3,6 +3,7 @@ This section is for maintainers only. If you don't know who the maintainers are but you need to reach them, please file an issue or (if it needs to remain private) contact the email address listed in `pyproject.toml`. Follow the instructions in the order provided. +> Note: Releases are immutable, please follow these steps carefully! ## 1. Release Readiness @@ -25,8 +26,6 @@ RAMPART follows [Semantic Versioning](https://semver.org/) (`MAJOR.MINOR.PATCH`) !!! note "Pre-1.0 stability" While RAMPART is below `1.0`, minor version bumps may include breaking changes. The API is stabilizing but not yet frozen. The first stable release will be `1.0.0`. -In line with PyPA [versioning guidance](https://packaging.python.org/en/latest/discussions/versioning/), `main` may carry a `.dev0` suffix between releases (e.g., `0.2.0.dev0`). Release commits drop the suffix; the post-release bump on `main` reintroduces it for the next version. - ## 3. Remove Deprecated Functionality If you are incrementing the minor version, search the codebase for the new minor version (no leading `v`) to find occurrences where functionality was deprecated and announced for removal in this version. Typically, functionality is deprecated and stays for two minor versions before being removed. @@ -137,20 +136,20 @@ Rebuild the package after any cherry-pick and re-test. ## 8. Publish to PyPI -Create a PyPI account if you don't have one and ask another maintainer to add you to the `msft-rampart` project. Before publishing, have an API token scoped to the project ready (create one in your PyPI project settings). +Create a PyPI account if you don't have one and ask another maintainer to add you to the `rampart` project. Before publishing, have an API token scoped to the project ready (create one in your PyPI project settings). ```bash uv pip install twine uv run twine upload dist/* ``` -If successful, the URL `https://pypi.org/project/msft-rampart/x.y.z/` will return the new release. +If successful, the URL `https://pypi.org/project/rampart/x.y.z/` will return the new release. ## 9. Update `main` After the release is on PyPI, open a PR to `main` containing only: -- A version bump in `pyproject.toml` to the next development version (e.g., `x.y.(z+1).dev0` or `x.(y+1).0.dev0`, depending on the next planned release). +- In line with PyPA [versioning guidance](https://packaging.python.org/en/latest/discussions/versioning/), bump the version in `pyproject.toml` to the next development version (e.g., `x.y.(z+1).dev0` or `x.(y+1).0.dev0`, depending on the next planned release). - Replace any references to the previous release version in the codebase with the new released version (without `.dev0`) where applicable (e.g., installation docs that pin to the latest tag). Open this PR from a branch separate from your `releases/vx.y.z` branch. @@ -166,7 +165,9 @@ Structure the description as: Maintenance changes, CI updates, and documentation fixes generally belong only in the full list. Verify the **New contributors** section is accurate. Mark the release as **Latest** and publish. -## PyRIT Dependency +## Appendix + +### PyRIT Dependency RAMPART pins PyRIT to a specific version in `pyproject.toml`: @@ -186,17 +187,19 @@ When updating the PyRIT dependency, use the helper script: Re-run the full test suite after bumping — PyRIT changes are a common source of regressions. -## Appendix: Patch Releases (Cherry-Pick Process) +--- + +### Patch Releases (Cherry-Pick Process) A patch release (e.g., `0.2.0` → `0.2.1`) ships a targeted fix — typically a security patch or a critical bug fix — without including other in-flight changes from `main`. -### When to use a patch release +#### When to use a patch release - A security vulnerability fix needs to be shipped urgently. - A critical bug was found in the latest release that blocks users. - The fix is already merged to `main`, but `main` contains other changes that aren't ready for release. -### Abbreviated steps +#### Abbreviated steps 1. **Create a release branch from the previous tag**, not from `main`: @@ -229,7 +232,7 @@ A patch release (e.g., `0.2.0` → `0.2.1`) ships a targeted fix — typically a 5. **Follow the regular release process from step 6 onward**: build, test, publish to PyPI, update `main`, and create the GitHub release. Patch release notes should clearly state the reason for the patch (e.g., "Security fix for…" or "Critical bug fix for…"). -### Key differences from a regular release +#### Key differences from a regular release | Aspect | Regular release | Patch release | |---|---|---|