[Paper Suggestion] AgentLeak: A Full-Stack Benchmark for Privacy Leakage in Multi-Agent LLM Systems — fits Privacy section (arXiv 2602.11510)

[yagobski]

Paper Suggestion: AgentLeak

Hi! I'd like to suggest adding AgentLeak to the Privacy section (under Vulnerabilities) — it benchmarks privacy leakage across the full multi-agent LLM pipeline.

📄 Paper

"AgentLeak: A Full-Stack Benchmark for Privacy Leakage in Multi-Agent LLM Systems"

• arXiv: https://arxiv.org/abs/2602.11510
• GitHub: https://github.com/Privatris/AgentLeak
• Project page: https://privatris.github.io/AgentLeak/
• Authors: Faouzi El Yagoubi, Godwin Badu-Marfo, Ranwa Al Mallah (Polytechnique Montreal)
• Date: 2025-02

🔍 Why it fits

AgentLeak directly addresses LLM privacy vulnerabilities in multi-agent settings — a growing deployment paradigm not covered by existing privacy benchmarks:

• 7 leakage channels measured simultaneously: tool calls, inter-agent messages, RAG queries, code execution, API calls, final outputs, and reasoning traces
• 68.8% of leakage occurs through inter-agent communication — undetectable by output-only auditing
• 41.7% of leakage is missed by output-only evaluation (the current standard approach)
• Covers AutoGen, LangGraph, CrewAI; evaluates GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 70B
• Introduces PII propagation tracking across agent boundaries

This fills a clear gap in the Privacy section: existing entries focus on single-agent or extraction attacks; AgentLeak addresses privacy leakage in multi-agent orchestration.

Thanks for maintaining this resource!