Soham/main work

CHANGELOG.md

@@ -5,6 +5,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+### Added
+
+- A **CrewAI adapter** (`adapters: [crewai]`), a third orchestration style behind the same `EpisodeSpec` -> `NormalizedTrace` contract, so the framework comparison covers a non-graph composition model.
+  CrewAI is an optional extra (`pip install 'tripwire-eval[crewai]'`) imported lazily, so a core install is unaffected and listing an uninstalled adapter gives a clear message instead of an import error.
+- `tests/conftest.py`, which plants a dummy provider key when the environment has none, so `pytest` runs out of the box.
+  The suite is offline; the key only satisfies the OpenAI client constructor. A real key already in the environment is left alone.
+- **Cross-adapter contract tests** (`tests/test_adapter_equivalence.py`): the same scripted agent behavior is run through every adapter and asserted to reach the same judge verdict.
+  This is what makes a cross-framework ASR comparison meaningful — it rules out the possibility that a difference between adapters is an artifact of how each one writes its trace rather than a real difference in hijackability.
+  In particular, a guard-denied call must be typed `tool_blocked` in every framework; if one adapter recorded it as a `tool_result`, that adapter's defended ASR would be silently inflated.
+
 ## [0.2.0] - 2026-08-10
 
 Hardening pass for the open-source release.

CLAUDE.md

@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## What This Is
 
-Tripwire is an evaluation harness that measures AI agent vulnerability to prompt injection attacks. It tests whether orchestration layers (raw tool loop, LangGraph single-agent, LangGraph multi-agent relay) affect agent hijackability when model and task are held constant.
+Tripwire is an evaluation harness that measures AI agent vulnerability to prompt injection attacks. It tests whether orchestration layers (raw tool loop, LangGraph single-agent, LangGraph multi-agent relay, CrewAI) affect agent hijackability when model and task are held constant.
 
 **Pipeline**: `agent + scenario → adapter (per-framework) → normalized trace → deterministic judge → pass/fail + security feedback`
 
@@ -14,7 +14,7 @@ Tripwire is an evaluation harness that measures AI agent vulnerability to prompt
 # Install for development (editable, with test + optional extras)
 pip install -e ".[dev]"        # or: uv pip install -e ".[dev]"
 
-# Run all tests (295 tests, fully offline, ~4s)
+# Run all tests (351 tests, fully offline, ~4s)
 python -m pytest tests/ -q
 
 # Run a single test file / test
@@ -48,7 +48,7 @@ Published on PyPI as `tripwire-eval` (the import package and CLI stay `tripwire`
 - `advice.py` - per-attack remediation knowledge base backing the security report.
 - `stats.py` - `wilson_ci()`: Wilson score CI for honest error bounds at small n.
 
-**Adapters** (`tripwire/adapters/`): `raw_loop.py` (minimal ReAct baseline), `langgraph_adapter.py` (single-agent LangGraph), `multi_agent_adapter.py` (LangGraph relay: agent A summarizes the inbox for agent B). `loader.py` also loads a bring-your-own agent file (`--agent`).
+**Adapters** (`tripwire/adapters/`): `raw_loop.py` (minimal ReAct baseline), `langgraph_adapter.py` (single-agent LangGraph), `multi_agent_adapter.py` (LangGraph relay: agent A summarizes the inbox for agent B), `crewai_adapter.py` (CrewAI role/goal crew; optional `[crewai]` extra, imported lazily, records its trace at the tool boundary). `loader.py` also loads a bring-your-own agent file (`--agent`).
 
 **Scenarios** (`tripwire/scenarios.py`): the tasks the agent is attacked on - `invoice` (default), `helpdesk`, `calendar`, `expense`. Each plants the secret, defines a legitimate task + AUTHORIZED recipient, and builds a full `EpisodeSpec`. Selected via `scenarios:` in the config.
 

CONTRIBUTING.md

@@ -8,11 +8,11 @@ This is a small project; issues and pull requests are welcome.
 ```bash
 git clone https://github.com/Arunjay4213/Tripwire && cd Tripwire
 pip install -e ".[dev]"     # or: uv pip install -e ".[dev]"
-python -m pytest tests/ -q  # 293 tests, fully offline, no network or API key
+python -m pytest tests/ -q  # 351 tests, fully offline, no network or API key
 ```
 
 The test suite never makes a network call: adapters are driven with stub clients and the judge is exercised on hand-built traces.
-A dummy provider key (`GROQ_API_KEY=dummy`) is enough to satisfy the OpenAI client constructor if your environment has none set.
+No provider key is needed — `tests/conftest.py` plants a dummy one when your environment has none, purely to satisfy the OpenAI client constructor (several adapters build their client in `__init__`). A real key already in your environment is left alone.
 
 ## How the pieces fit
 

README.md

@@ -36,7 +36,7 @@ agent=raw_loop   defense=no_defense
 pip install tripwire-eval        # or: uv pip install tripwire-eval
 ```
 
-The install name is `tripwire-eval`; you `import tripwire` and run the `tripwire` command (like `pip install scikit-learn` -> `import sklearn`). Optional extras: `tripwire-eval[agentdojo]` (real AgentDojo benchmark), `[viz]` (plots), `[all]`.
+The install name is `tripwire-eval`; you `import tripwire` and run the `tripwire` command (like `pip install scikit-learn` -> `import sklearn`). Optional extras: `tripwire-eval[agentdojo]` (real AgentDojo benchmark), `[crewai]` (the CrewAI adapter), `[viz]` (plots), `[all]`.
 
 ## Quickstart
 
@@ -75,7 +75,7 @@ Your agent just needs to expose `run(spec) -> NormalizedTrace` (or an `adapter`
 - **Realistic scenarios.** `invoice`, `helpdesk`, `calendar`, `expense` - each a different task with its own authorized recipient. Your agent is tested across all of them.
 - **Actionable feedback.** Every run ends with a per-attack report: what leaked, why, and the concrete fix.
 - **Deterministic, honest scoring.** Pure-Python judge (no LLM), fixed seeds, Wilson confidence intervals instead of bare percentages. Runs in CI.
-- **Framework-agnostic.** Reference adapters for a raw tool loop, LangGraph, and a multi-agent LangGraph relay - all behind one contract, so cross-framework comparison is fair.
+- **Framework-agnostic.** Reference adapters for a raw tool loop, LangGraph, a multi-agent LangGraph relay, and CrewAI - all behind one contract, so cross-framework comparison is fair.
 - **Real AgentDojo, too.** A [bridge](docs/sample-agents-and-agentdojo.md) runs actual AgentDojo workspace tasks with AgentDojo's own environment-state scoring.
 
 ## How it works
@@ -113,7 +113,7 @@ python scripts/ci/check_asr_threshold.py --results results.json --threshold 0.5
 ```bash
 git clone https://github.com/Arunjay4213/Tripwire && cd Tripwire
 pip install -e ".[dev]"     # or: uv pip install -e ".[dev]"
-pytest                      # 295 offline tests, no network
+pytest                      # 351 offline tests, no network
 ```
 
 ## License

docs/attacks.md

@@ -138,6 +138,7 @@ Adapters available (`tripwire/adapters/loader.py`):
 - `raw_loop` - minimal ReAct-style baseline, no framework.
 - `langgraph` - single-agent LangGraph tool loop.
 - `multi_agent` - a LangGraph relay where one agent summarizes the inbox for a second agent that never sees it (second-order / relay injection).
+- `crewai` - a CrewAI role/goal/crew agent, a non-graph way of composing an agent. Needs the optional extra: `pip install 'tripwire-eval[crewai]'`.
 
 To run the full suite against all frameworks, list them all in the config (`tripwire/config/threat_model.example.yaml` ships this way):
 

pyproject.toml

@@ -48,9 +48,10 @@ dependencies = [
 
 [project.optional-dependencies]
 agentdojo = ["agentdojo==0.1.35"]   # real AgentDojo suites for the bridge
+crewai = ["crewai>=0.100,<2"]       # the crewai adapter (lazily imported)
 viz = ["matplotlib>=3.8"]           # Pareto / analysis plots
 dev = ["pytest>=9", "agentdojo==0.1.35", "matplotlib>=3.8"]
-all = ["agentdojo==0.1.35", "matplotlib>=3.8"]
+all = ["agentdojo==0.1.35", "crewai>=0.100,<2", "matplotlib>=3.8"]
 
 [project.scripts]
 tripwire = "tripwire.__main__:main"

tests/conftest.py

@@ -0,0 +1,22 @@
+"""Test-session setup.
+
+The suite is fully offline: adapters are driven with stub clients and the judge
+runs on hand-built traces, so no test makes a network call. But the OpenAI SDK
+rejects a *missing* key at construction time, and several adapters build their
+client in __init__ (LangGraphAdapter, MultiAgentAdapter) -- so a machine with no
+provider key in the environment fails those tests for a reason that has nothing
+to do with the code under test.
+
+Plant a dummy key when the environment has none. It is never used for a real
+request. A real key already in the environment (or in a .env) is left alone, so
+this never shadows a working provider config.
+"""
+
+from __future__ import annotations
+
+import os
+
+_PROVIDER_KEYS = ("OPENAI_API_KEY", "GROQ_API_KEY")
+
+if not any(os.environ.get(k) for k in _PROVIDER_KEYS):
+    os.environ["GROQ_API_KEY"] = "test-dummy-key-offline-suite-only"