AgentDojo-X

Cross-framework AI agent red-teaming benchmark. Measures whether orchestration framework choice explains more variance in adversarial attack success than the underlying LLM does.

Companion papers in this research series: Paper A, Paper B, Paper C — all published on Figshare, forming a self-contained series on refusal-detector reliability that this benchmark's dataset also draws on. See docs/README.md for the full series index.

Research artifact for "Cross-Framework Portability of Agentic AI Security: A Controlled, Payload-Verified Evaluation" (Paper D in this project's research series; target venue: USENIX Security '27, Cycle 1). Published on Figshare (preprint, CC BY 4.0): https://doi.org/10.6084/m9.figshare.33110642. Extends prior art from Nguyen & Husain (arXiv:2512.14860), whose 2-framework binary refusal-rate comparison motivated this project's formal variance-decomposition approach across 6 frameworks.

Status: private, pre-publication. Full 7,020-trial matrix complete (6 frameworks × 6 models × 5 attack families — see Trial matrix below). Recovered from a mid-run network outage, retried, and independently verified twice: exact row counts, (framework, model, prompt_id, seed) key parity between results/full_run_7020.jsonl and its .raw.jsonl sibling, and a full error inventory — see verification_report.md. The paper is published (Figshare, DOI above); this repo remains private pending a review of raw-response content in results/full_run_7020.raw.jsonl for public release, not because the paper itself is unpublished.

---

What this measures

Two related metrics, both defined formally in docs/CFPS_SPEC.md:

• Per-model CFPS (Cross-Framework Portability Score) — coefficient of variation of a model's attack-success rate across all 6 frameworks. Low = the model's vulnerability is stable regardless of orchestration framework. High = the same model becomes meaningfully more or less exploitable purely based on which framework wraps it.
• Framework Variance Share — an ANOVA-style variance decomposition isolating how much of total attack-success variance is attributable to framework choice vs. model choice vs. attack family. This is the metric that directly answers the core research question, and is the primary contribution beyond Nguyen & Husain's descriptive 2-framework comparison.

Trial matrix

Tier	Models	Frameworks	Attack families	Seeds	Trials
Cheap (full matrix — feeds the primary variance decomposition)	3	6	5	10	5,400
Frontier (confirmatory spot-check, not pooled into the ANOVA)	3	6	5	3	1,620
Total	6	6	5	—	7,020

Models (all IDs live-verified against each provider's API, not assumed):

Tier	Provider	Model	Model ID
Cheap	Anthropic	Claude Haiku 4.5	claude-haiku-4-5-20251001
Cheap	OpenAI	GPT-5.4 nano	gpt-5.4-nano
Cheap	Google	Gemini 3.1 Flash-Lite	gemini-3.1-flash-lite
Frontier	Anthropic	Claude Opus 4.8	claude-opus-4-8
Frontier	OpenAI	GPT-5.5	gpt-5.5
Frontier	Google	Gemini 3.5 Flash	gemini-3.5-flash

(gpt-5.5-pro was the original frontier-tier OpenAI candidate; swapped for gpt-5.5 during Phase 1 integration testing after confirming gpt-5.5-pro is Responses-API-only and rejects Chat Completions calls, which several framework adapters in this matrix depend on.)

Frameworks (6 execution paths, via safelabs-eval's adapters): generic HTTP, LangChain, CrewAI, AutoGen (ag2), LlamaIndex, OpenAI Agents SDK.

Attack families (5 families, each spanning 2 OWASP ASI categories, 6 prompts per family — 30 prompts total, reused directly from safelabs-eval's prompt library):

Family	ASI categories
Injection & Output Handling	ASI01 + ASI02
Agency & Scope	ASI03 + ASI09
Resource & Tool Abuse	ASI04 + ASI05
Trust & Privacy	ASI06 + ASI07
Drift & Misinformation	ASI08 + ASI10

Dependency on safelabs-eval

This project reuses safelabs-eval's agent adapters, prompt library, and scoring detectors directly rather than reimplementing them. Trial verdicts computed under safelabs-eval < 0.2.1 reflect detectors with known refusal-recognition gaps (Unicode punctuation mismatch, narrow refusal vocabulary) — fixed in safelabs-eval 0.2.1. Because raw model responses are preserved separately from computed verdicts (see Usage below), the fix for any affected data is to re-score the saved raw responses against the current detector, not re-collect via fresh API calls — re-collection was never actually required in practice. results/full_run_7020.jsonl itself needs no action here: its stored verdicts were confirmed scored under the fixed detector directly (a 270-cell spot-check against an independent current-HEAD re-scoring matched exactly, 0 discrepancies). This note applies only to older cached data that predates the fix and hasn't been re-scored since.

Setup

git clone https://github.com/AgentSafeLabs/agentdojo-x.git
cd agentdojo-x
python -m venv .venv && source .venv/bin/activate
pip install -e .
cp .env.example .env   # add ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY

Usage

# Small validation run (36 trials, ~$0.01, one model per provider)
python -m agentdojo_x.cli --dry-run

# Full matrix (7,020 trials, est. $10-20)
python -m agentdojo_x.cli --full-run

# Resume an interrupted run — pass the same flag again; automatically
# skips any (framework, model, prompt_id, seed) key already present
python -m agentdojo_x.cli --full-run

Each mode writes incrementally to its own default path — --dry-run to results/trials.jsonl, --staging to results/staging.jsonl, --full-run to results/full_run_7020.jsonl — overridable with --output.

None of these are gitignored. .gitignore only excludes local tooling artifacts (.venv/, .env, __pycache__/, build byproducts); every run's output — dry-run, staging, and the full matrix — is committed directly to the repo for provenance and reproducibility, not treated as disposable working data. Staging and full-run outputs additionally get a .raw.jsonl sibling (prompt, response, and metadata only — no computed verdict; see results/full_run_7020.raw.jsonl), so verdicts can be re-scored later against a different detector version without re-running any API calls — see "Dependency on safelabs-eval" above for why that matters, and Results provenance below for how it's verified.

Results provenance and recovery tooling

• verification_report.md — certifies the current state of results/full_run_7020.jsonl and its .raw.jsonl sibling: exact row counts (7,020/7,020), unique-key parity between the two files, confirmation that no network-outage artifacts remain from the mid-run recovery, and the full inventory of the 140 rows with no real model output (103 content-policy blocks, 21 reasoning-token exhaustions, 14 timeouts, 2 other — documented, not a bug; full breakdown in docs/CFPS_SPEC.md §7.1). Read this before treating the full-run data as final for any downstream analysis.
• results/archive/ — point-in-time backups of full_run_7020{,.raw}.jsonl taken before each retry/recovery pass, plus retry-run logs (e.g. retry_batch64.log). Kept, not deleted, so that any results-file mutation is always preceded by a recoverable snapshot.
• agentdojo_x/scripts/retry_failed_cells.py — reusable tooling for selectively re-attempting specific failed trial cells (identify by error category, back up, delete, regenerate) without re-running the full matrix. Built for the network-outage recovery; kept as general-purpose tooling for any future partial failure rather than a one-off script, deliberately separate from cli.py's run-forward modes since it mutates an existing results file rather than producing a new one.

License

Apache 2.0 — see LICENSE.