Skip to content

Add six MCP abuse-case scenarios with passing and failing traces - #177

Open
astrogilda wants to merge 5 commits into
OWASP:mainfrom
astrogilda-forks:scenario/mcp-abuse-cases
Open

astrogilda wants to merge 5 commits into
OWASP:mainfrom
astrogilda-forks:scenario/mcp-abuse-cases

Conversation

@astrogilda

@astrogilda astrogilda commented Sep 16, 2026 •

Copy link
Copy Markdown

Fixes #

Describe the changes you have made in this PR -

No linked issue. This pull request adds MCP abuse, context-injection, approval-bypass and unsafe-tool-execution cases, at least one of each.

This adds six MCP abuse cases at trace level, each with a trace that trips it: two in schema categories no bundled scenario used (context_injection, unauthorized_outbound_action), and four on the MCP surface that the bundled MCP scenarios do not cover.

Each failing trace runs in the regression_demo gate, following examples/traces/README.md: a scenario that cannot fail is no regression test.

The approval_required assertion matches its action against observed tool names. MCP calls are recorded as mcp/<server_id>/<tool>, so a bare action name returns not_run even when an agent called it.

The cases exercise the evidence vectors at probityai/agent-evidence-vectors, cited in each scope note.

Edited 2026-10-09: the approval_required docs line and its two tests are in this PR, and the vectors link points at the current owner.


Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code or documentation ?

  • No, I wrote all the code myself
  • Yes, I used AI assistance

If you used AI assistance, briefly describe:

  • Tool(s) used: Claude Code.
  • Parts of the contribution that were AI-assisted: scenario YAML, trace fixtures and prose, from my own threat model of MCP servers and attack cases I wrote.
  • How you reviewed the output: I read each scenario against schemas/scenario.schema.json. I checked each fails on its violating trace, and passes or returns not_run on its clean one.
  • Tests or checks I ran: the commands below, on a clean checkout.
ruff check src tests
mypy
pytest --cov=src/agent_harness --cov-fail-under=80
# plus the whole security-regression job, with its three regression_demo gates

Checklist before requesting a review

  • I have added proper PR title and linked to the issue. Title yes; issue no, because no open issue covers this.
  • I have performed a self-review of my code
  • I can explain the purpose of every documentation, function, class, and logic block I added
  • I understand why my changes work and have tested them thoroughly
  • I have considered potential edge cases and how my code handles them
  • If it is a core feature, I have added thorough tests
  • My code follows the project's style guidelines and conventions
  • I added an entry under [Unreleased] in CHANGELOG.md (or this PR does not need one — pure refactor, internal tests, or CI-only change)

Adds six trace-level scenarios on the MCP protocol surface, each with a
passing trace fixture under examples/traces/<category>/ so
tests/test_scenario_pass_fixtures.py covers them.

Two fill categories the schema declares but no bundled scenario used:
context_injection and unauthorized_outbound_action. The other four are
MCP trust-boundary, approval-bypass and unsafe-tool-execution cases that
do not overlap the three MCP scenarios already bundled.

Two also exercise the mcp_prompt_get and mcp_tool_result event types
that docs/trace-format.md normalizes and no scenario used.
Each of the six scenarios gets a trace that trips it, run in the
security-regression job and written to regression_demo/, so the existing
gate fails if any of them stops detecting its own abuse case. A scenario
that cannot fail is not a regression test.
Each of the six scope notes now points at the conformance vector suite
the evidence shapes are published in, so a reader checking whether a
case models a real failure has the vectors to check it against.
@astrogilda

Copy link
Copy Markdown
Author

Hey @mertsatilmaz, the tests run on this pull request is waiting for approval. I ran its three jobs against the merge commit, and lint, tests and the security-regression job all pass. Each of the six new MCP scenarios returns fail against its failing trace, as that job requires. Would you approve the run when you have a moment?

The conformance-vector repository moved to a new owner. The old URL only resolves through a redirect, so link the current path.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
approval_required matches action exactly against observed tool names, and MCP calls are recorded as mcp/<server_id>/<tool>. A bare tool name therefore returns not_run even when the agent made the call. Document the rule, cross-reference it from the trace format, and pin both results with unit tests.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant