Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
persist-credentials: false

- name: Set up Python
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<p align="center">
<img src="docs/images/RAMPART.png" alt="RAMPART Logo" width="400"/>
<img src="docs/images/RAMPART.svg" alt="RAMPART Logo" width="300"/>
</p>

<h1 align="center">RAMPART</h1>
Expand Down
13 changes: 13 additions & 0 deletions docs/404.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Page not found

Sorry, the page you're looking for doesn't exist.

It may have been moved, renamed, or removed. Try one of the following:

- Use the search bar above to find what you're looking for.
- Head back to the [home page](index.md).
- Browse the [getting started guide](getting-started/index.md).
- Check the [API reference](api/index.md).

If you think this is a broken link in the documentation, please
[open an issue](https://github.com/microsoft/RAMPART/issues/new).
10 changes: 10 additions & 0 deletions docs/_includes/abbreviations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
*[XPIA]: Cross-Prompt Injection Attack — an attack where adversarial content reaches the agent indirectly via a data source (file, email, web page) rather than directly through the user's prompt.
*[PyRIT]: Python Risk Identification Tool — Microsoft's open-source AI red-teaming library that RAMPART builds on.
*[CVE]: Common Vulnerabilities and Exposures — public catalog of disclosed security vulnerabilities.
*[OG]: Open Graph — protocol that defines how URLs render as rich previews on social media and chat platforms.
*[OIDC]: OpenID Connect — identity layer on top of OAuth 2.0 used by GitHub Actions for keyless authentication.
*[OWASP]: Open Worldwide Application Security Project — community producing security guidance and the OWASP Top 10.
*[SemVer]: Semantic Versioning — `MAJOR.MINOR.PATCH` versioning scheme.
*[PyPA]: Python Packaging Authority — working group maintaining Python packaging standards and tooling.
*[gRPC]: Google Remote Procedure Call — high-performance RPC framework based on HTTP/2 and Protocol Buffers.
*[SUT]: System Under Test — the agent or component being evaluated by a RAMPART test.
6 changes: 6 additions & 0 deletions docs/_overrides/main.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{% extends "base.html" %}

{% block announce %}
<strong>RAMPART is in alpha</strong> &mdash; APIs may change before <code>1.0</code>.
See the <a href="{{ 'contributing/release-process/' | url }}">release process</a> for stability details.
{% endblock %}
5 changes: 5 additions & 0 deletions docs/api/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
---
description: API reference for RAMPART, auto-generated from source docstrings. Covers core types, protocols, attacks, probes, evaluators, drivers, payloads, surfaces, converters, reporting, and the pytest plugin.
---

# API Reference

API reference organized by RAMPART's package layout. Each page documents the public symbols in one module, auto-generated from source docstrings.


| Page | Contents |
|------|----------|
| [Core Types](core-types.md) | `Payload`, `Request`, `Response`, `Turn`, `Result`, `SafetyStatus`, `HarmCategory`, and more |
Expand Down
4 changes: 4 additions & 0 deletions docs/attacks/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Built-in RAMPART attacks, including XPIA (Cross-Prompt Injection Attack), with patterns for surface-based and inline injection.
---

# Attacks

For background on what attacks are and how they work, see [Concepts: Attacks](../concepts/attacks.md).
Expand Down
4 changes: 4 additions & 0 deletions docs/concepts/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Mental model for RAMPART — attacks vs probes, evaluators, the execution lifecycle, and how RAMPART builds on PyRIT.
---

# Concepts

Core ideas behind RAMPART. Read these before writing tests.
Expand Down
4 changes: 4 additions & 0 deletions docs/contributing/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to contribute to RAMPART — development setup, code style, testing standards, architecture, extension points, and the release process.
---

# Contributing to RAMPART

We welcome contributions and suggestions! Whether you're fixing a bug, adding a new attack type, improving documentation, or filing an issue, your help is appreciated.
Expand Down
6 changes: 3 additions & 3 deletions docs/contributing/release-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ Follow the instructions in the order provided.

Before starting the release process, verify the codebase is in a healthy state.

- **Check for pending changes.** Ask other RAMPART maintainers whether they have any in-flight changes that should land before the release.
- **Verify CI pipelines.** Confirm that all unit tests, lint, type checks, and coverage gates are green on `main`. If anything is failing, fix it before proceeding.
- **Verify the PyRIT pin.** RAMPART pins PyRIT to a specific version in `pyproject.toml`. Confirm the pinned version is the one you intend to ship against — see [PyRIT Dependency](#pyrit-dependency).
- [ ] **Check for pending changes.** Ask other RAMPART maintainers whether they have any in-flight changes that should land before the release.
- [ ] **Verify CI pipelines.** Confirm that all unit tests, lint, type checks, and coverage gates are green on `main`. If anything is failing, fix it before proceeding.
- [ ] **Verify the PyRIT pin.** RAMPART pins PyRIT to a specific version in `pyproject.toml`. Confirm the pinned version is the one you intend to ship against — see [PyRIT Dependency](#pyrit-dependency).

## 2. Decide the Next Version

Expand Down
4 changes: 4 additions & 0 deletions docs/getting-started/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Install RAMPART, run your first safety test, and learn how the pytest plugin fits into your existing test suite.
---

# Getting Started

| Page | Description |
Expand Down
22 changes: 15 additions & 7 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,26 @@ Or, if you already have a project:

```bash
uv venv
uv pip install git+https://github.com/microsoft/RAMPART.git
uv pip install rampart
```

### Using pip

```bash
python -m venv .venv
source .venv/bin/activate # Linux/macOS
.venv\Scripts\activate # Windows
=== "Linux / macOS"

pip install git+https://github.com/microsoft/RAMPART.git
```
```bash
python -m venv .venv
source .venv/bin/activate
pip install rampart
```

=== "Windows (PowerShell)"

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install rampart
```

Both approaches install RAMPART and all dependencies, including [PyRIT](https://github.com/microsoft/PyRIT) v0.13.0.

Expand Down
24 changes: 15 additions & 9 deletions docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Follow the [Installation](installation.md) guide, then return here.

Your adapter bridges RAMPART and your agent. Implement two protocols: [`AgentAdapter`][rampart.core.adapter.AgentAdapter] (factory + metadata) and [`Session`][rampart.core.adapter.Session] (interaction).

```python
```python linenums="1"
# my_agent/adapter.py

from rampart import (
Expand All @@ -32,7 +32,7 @@ class MyAgentSession:
def __init__(self, api_client):
self._client = api_client

async def send_async(self, request: Request) -> Response:
async def send_async(self, request: Request) -> Response: # (1)!
# Replace this with your agent's actual API call.
# This could be an OpenAI client, an HTTP request,
# a gRPC call, a Playwright browser session — whatever
Expand All @@ -41,16 +41,16 @@ class MyAgentSession:

return Response(
text=raw_response["text"],
tool_calls=[
tool_calls=[ # (2)!
ToolCall(name=tc["name"], arguments=tc["args"])
for tc in raw_response.get("tool_calls", [])
],
)

async def __aenter__(self):
async def __aenter__(self): # (3)!
return self

async def __aexit__(self, exc_type, exc_val, exc_tb):
async def __aexit__(self, exc_type, exc_val, exc_tb): # (4)!
pass


Expand All @@ -60,19 +60,25 @@ class MyAgentAdapter:
def __init__(self, api_client):
self._client = api_client

async def create_session_async(self) -> MyAgentSession:
async def create_session_async(self) -> MyAgentSession: # (5)!
return MyAgentSession(api_client=self._client)

@property
def manifest(self) -> AppManifest:
def manifest(self) -> AppManifest: # (6)!
return AppManifest(name="My Agent")

@property
def observability_profile(self) -> ObservabilityLevel:
def observability_profile(self) -> ObservabilityLevel: # (7)!
return ObservabilityLevel.TOOL_ONLY
```

The adapter needs three things: `create_session_async()` to create sessions, `manifest` to identify your agent, and `observability_profile` to declare what the adapter can observe. See [Writing Tests](../usage/authoring-tests.md) for the full details on manifests, tool declarations, and observability levels.
1. **Send a request, return a response.** Populate `tool_calls` and `side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened."
2. **Tool calls go here.** The evaluator [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] only fires if these are reported, so don't skip them when your agent supports tools.
3. **Set up session-level state.** API connections, browser contexts, anything that lives for one interaction.
4. **Clean up.** Must be idempotent and must not raise — RAMPART always calls this, even after errors.
5. **Create a fresh session per execution.** RAMPART manages the lifecycle; you just hand over a new `Session`.
6. **Identify the agent.** Add `tools=[...]` and `data_sources=[...]` here once you wire those up.
7. **Declare what you can observe.** Affects which evaluators are reliable — see [Writing Tests](../usage/authoring-tests.md).

---

Expand Down
4 changes: 4 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Definitions for terms used throughout the RAMPART documentation — adapter, attack, probe, evaluator, surface, payload, and more.
---

# Glossary

Terms used throughout the RAMPART documentation.
Expand Down
513 changes: 513 additions & 0 deletions docs/images/RAMPART.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 5 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
<div style="text-align: center; margin-bottom: 1.5em;">
<img src="images/RAMPART.png" alt="RAMPART" style="max-width: 400px;" />
</div>
---
description: RAMPART is a pytest-native safety testing framework for agentic AI applications. Write attack and probe tests, orchestrate the interaction, and report results in CI.
---

![RAMPART](images/RAMPART.svg){ .hero-logo }

# RAMPART Documentation

Expand Down
4 changes: 4 additions & 0 deletions docs/probes/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Built-in RAMPART probes that verify expected agent behavior, including the behavioral probe.
---

# Probes

Available probe types.
Expand Down
4 changes: 4 additions & 0 deletions docs/stylesheets/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,7 @@
margin-left: auto;
margin-right: auto;
}

.md-typeset img.hero-logo {
max-width: 50%;
}
63 changes: 32 additions & 31 deletions docs/usage/authoring-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ Every RAMPART test needs an adapter that connects your agent to the framework.

A [`Session`][rampart.core.adapter.Session] is an async context manager that sends requests and returns responses:

```python
```python linenums="1"
from rampart import Request, Response, ToolCall

class MySession:
async def send_async(self, request: Request) -> Response:
async def send_async(self, request: Request) -> Response: # (1)!
raw = await self._client.chat(request.prompt)
return Response(
text=raw["text"],
Expand All @@ -26,18 +26,16 @@ class MySession:
],
)

async def __aenter__(self):
async def __aenter__(self): # (2)!
return self

async def __aexit__(self, exc_type, exc_val, exc_tb):
async def __aexit__(self, exc_type, exc_val, exc_tb): # (3)!
pass
```

**Key responsibilities:**

- **`send_async`**: Populate `Response.tool_calls` and `Response.side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened."
- **`__aenter__`**: Set up session-level state (API connections, browser contexts).
- **`__aexit__`**: Clean up. Must be idempotent and must not raise.
1. Populate `Response.tool_calls` and `Response.side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened."
2. Set up session-level state (API connections, browser contexts).
3. Clean up. Must be idempotent and must not raise.

### AgentAdapter Protocol

Expand Down Expand Up @@ -174,35 +172,38 @@ class MyFileSurface:
path=self._target_path,
payload=payload,
)
```

??? note "`_FileInjection` reference implementation"

class _FileInjection:
def __init__(self, *, client, path: str, payload: Payload):
self._client = client
self._path = path
self._payload = payload
self._original_content: str | None = None
```python linenums="1"
class _FileInjection:
def __init__(self, *, client, path: str, payload: Payload):
self._client = client
self._path = path
self._payload = payload
self._original_content: str | None = None

@property
def payload_id(self) -> str | None:
return self._payload.id
@property
def payload_id(self) -> str | None:
return self._payload.id

@property
def surface_name(self) -> str:
return "file_system"
@property
def surface_name(self) -> str:
return "file_system"

async def wait_until_ready(self) -> None:
pass # or: await asyncio.sleep(10.0) for indexing delay
async def wait_until_ready(self) -> None:
pass # or: await asyncio.sleep(10.0) for indexing delay

async def __aenter__(self):
self._original_content = await self._client.read(self._path)
await self._client.write(self._path, self._payload.content)
return self
async def __aenter__(self):
self._original_content = await self._client.read(self._path)
await self._client.write(self._path, self._payload.content)
return self

async def __aexit__(self, exc_type, exc_val, exc_tb):
if self._original_content is not None:
await self._client.write(self._path, self._original_content)
```
async def __aexit__(self, exc_type, exc_val, exc_tb):
if self._original_content is not None:
await self._client.write(self._path, self._original_content)
```

!!! warning
`__aexit__` must not raise. If cleanup can fail, catch and log the exception.
Expand Down
20 changes: 19 additions & 1 deletion docs/usage/ci-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,25 @@ The JSON file contains aggregate statistics and per-result data that CI dashboar

## Environment Variables

RAMPART itself does not read environment variables. Your adapter and test configuration typically do:
RAMPART itself does not read environment variables. Your adapter and test configuration typically do. Setting them locally for ad-hoc runs:

=== "Linux / macOS"

```bash
export AGENT_API_KEY="..."
export AGENT_ENDPOINT="https://..."
pytest tests/
```

=== "Windows (PowerShell)"

```powershell
$env:AGENT_API_KEY = "..."
$env:AGENT_ENDPOINT = "https://..."
pytest tests/
```

Then consume them in your adapter and configuration:

```python
import os
Expand Down
4 changes: 4 additions & 0 deletions docs/usage/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Patterns for authoring RAMPART tests — adapters, manifests, evaluators, surfaces, pytest markers, reporting, and CI integration.
---

# Usage

| Page | Description |
Expand Down
Loading
Loading