Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/TRIAGE_RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
OWASP GenAI Crosswalk
File : docs/TRIAGE_RULES.md
Purpose : Where an incoming item goes — incident, benchmark catalogue, or noted and closed
Version : 1.0.0 — 2026-09-18
Version : 1.1.0 — 2026-10-09
License : CC BY-SA 4.0
-->

Expand Down Expand Up @@ -51,6 +51,11 @@ this repository sets, and not a reproduction of anyone's results. Benchmarks are
published with their own harness and licence, and a pass mark for someone else's
benchmark is theirs to set.

A defense paper that also introduces a **named** benchmark gets a row for the benchmark;
the defense itself is noted. The row names the authors' own defense evaluated on it.
A defense paper with no named benchmark is noted and closed. (Maintainer ruling,
2026-10-09, on #194 and #198.)

## CVEs

A published CVE in GenAI tooling is an incident record when it names an affected version
Expand Down
6 changes: 5 additions & 1 deletion evals/EXTERNAL_BENCHMARKS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!--
OWASP GenAI Crosswalk
Document : External Benchmarks — catalogue of published evaluation corpora
Version : 1.1.0 — 2026-09-30
Version : 1.2.0 — 2026-10-09
License : CC BY-SA 4.0
-->

Expand Down Expand Up @@ -45,6 +45,9 @@ as a profile in this repository.**
| **AgentLSD** | **Adversarial task contamination** of AI security agents — deceptive artifacts in the environment, including non-instructional evidence "such as fake results and decoy endpoints", rather than injected instructions alone | Paired clean and trap-augmented runs on CTF challenges with deterministic trap generation; 6 models on 11 web CTF challenges | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2609.19140](https://arxiv.org/abs/2609.19140) · release: [Golim/agent-lsd](https://github.com/Golim/agent-lsd) (MIT) |
| **AgentXploit-Bench** | **End-to-end exploitability of AI-agent systems** in authorized white-box pre-deployment auditing: attacks must act through the task-defined attacker interface and be confirmed by an external verifier | 72 reproducible vulnerabilities across 12 open-source AI-agent systems and frameworks | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution · LLM01 Prompt Injection | [arXiv:2609.31318](https://arxiv.org/abs/2609.31318) · release: [lwd17/AgentXploit](https://github.com/lwd17/AgentXploit) (README states Apache-2.0; no licence file in the repository) |
| **PrivDrift** | **User-secret leakage under topic drift**: whether secrets a user disclosed earlier in an active conversation remain recoverable after the dialogue moves on, under persuasion-based probing | 1,000 controlled multi-turn dialogues with seeded secrets, content-dense drift turns and standardized extraction probes; 3 LLMs with extended context windows | LLM02 Sensitive Information Disclosure · DSGAI11 Cross-Context Conversation Bleed | [arXiv:2609.30094](https://arxiv.org/abs/2609.30094) · not released: the paper states the authors "plan to release" the generation code, probes, scripts and a sanitized subset |
| **EvoRiskBench** | **Runtime security risks in workspace agents** — models combined with execution harnesses that perform stateful, multi-step tasks on external resources — organised around the authors' EP-Path-EF framework, which "links an initial risk entry point to a one-hop technical effect through an agent-mediated risk path"; outcomes are verified independently from runtime traces and environment states | 450 adversarial tasks across six scenarios; nine entry-point and five effect categories; 9 model-harness configurations (3 models × Claude Code, Codex and OpenClaw) | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution | [arXiv:2610.03153](https://arxiv.org/abs/2610.03153) · not released: the paper states the cases and platform "will be released after completion of artifact safety and reproducibility checks" |
| **PAGE** (Persona-Aware Guardrail Evaluation) | **Function-specific guardrails** for customer-facing agents: whether a guardrail keeps an agent within its intended functionality, across "benign, adversarial, and out-of-domain interactions on both user and agent turns" | Not stated in the abstract | LLM01 Prompt Injection · LLM03 Excessive Agency | [arXiv:2610.03434](https://arxiv.org/abs/2610.03434) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside, and used to evaluate, the authors' own guardrail (Persona Guardrail), which the paper states is deployed in production. |
| **STEER-Bench** | **Branch steering** against computer-use agents: whether crafted untrusted data can "coerce a CUA down a hazardous, pre-approved branch without injecting explicit instructions", including agents built on the Dual-LLM planner/quarantine pattern | 101 tasks across 9 domains; the authors report 94.4% attack success against standard and 89.5% against vanilla Dual-LLM agents | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2610.03089](https://arxiv.org/abs/2610.03089) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside the authors' own defense (COBRA), which they report reduces attack success on it to 0%. |

## Why these and not others

Expand All @@ -56,3 +59,4 @@ Routing — incident, catalogue, or noted and closed — is defined once in [`..
|---|---|
| 2026-09-18 | Created with LongPIBench, GenIaC-SecBench and TIER, from the watcher triage of issues #47, #55 and #70. |
| 2026-09-30 | Added APort Vault, ClashBench, AgentLSD, AgentXploit-Bench and PrivDrift, from the watcher triage of issues #125, #137, #144, #160 and #171; added the release and licence note. |
| 2026-10-09 | Added EvoRiskBench, PAGE and STEER-Bench, from the watcher triage of issues #196, #194 and #198. PAGE and STEER-Bench are introduced by defense papers; each row names the authors' own defense evaluated on it. |
Loading