Does RAG-Bench cover the cybersecurity leg of Article 15?

[apundhir]

No. RAG-Bench covers accuracy and robustness — measuring faithfulness, retrieval precision, agentic metrics, and adversarial-passage robustness.

The cybersecurity leg of EU AI Act Article 15 (prompt injection resistance, jailbreak defence, model integrity) requires a runtime AI security control such as AgentShield.

Pair the two for full Article 15 coverage.

— Full FAQ at https://aiexponent.com/docs/rag-benchmarking

[musaabhasan]

I would separate Article 15 coverage into two layers: benchmark evidence and operating control evidence.

RAG-Bench can support the benchmark-evidence side well: retrieval precision, faithfulness, robustness against adversarial passages, and traceable measurement of answer quality. That is useful for Article 15 because robustness and accuracy are not only model qualities; they are system qualities.

For the cybersecurity leg, I would not treat a runtime guard as a full substitute for benchmark coverage. I would add a companion security evaluation pack with cases such as:

• indirect prompt injection embedded in retrieved documents,
• retrieval of unauthorized context across tenants, roles, or courses,
• malicious corpus poisoning that shifts answers over time,
• tool-call escalation from retrieved text,
• sensitive-data exfiltration through citations, summaries, or follow-up prompts,
• logging and evidence checks showing why a response was allowed or blocked.

Then the runtime control can become the operational enforcement layer, while the benchmark demonstrates repeatable pre-release evidence. In practice, the stronger Article 15 story is: benchmark the RAG system under security-relevant scenarios, enforce controls at runtime, and retain enough audit evidence to reconstruct failures.

So my answer would be: RAG-Bench covers part of the robustness story, but cybersecurity coverage should be explicit rather than implied. Pairing it with a runtime security guard is sensible, but the benchmark itself should still include security-specific RAG scenarios if it is being used as Article 15 evidence.