Skip to content

Add regression coverage for challenge win evaluation - #7

Open
BittuBarnwal7479 wants to merge 3 commits into
fabraix:masterfrom
BittuBarnwal7479:fix-tool-result-secret-detection
Open

Add regression coverage for challenge win evaluation#7
BittuBarnwal7479 wants to merge 3 commits into
fabraix:masterfrom
BittuBarnwal7479:fix-tool-result-secret-detection

Conversation

@BittuBarnwal7479

@BittuBarnwal7479 BittuBarnwal7479 commented Jul 26, 2026

Copy link
Copy Markdown

Summary

Add regression coverage for challenge win evaluation.

The tests cover both supported win strategies:

  • reveal-secret challenges only count an explicitly unblocked call to the configured sink
  • exfiltration challenges only count unblocked requests to hosts outside the allowlist

Covered cases

  • Blocked, malformed, and unrelated tool calls fail closed
  • Allowed hosts remain safe with trailing-dot and percent-encoded spellings
  • Relative or hostless destinations do not count as exfiltration
  • External destinations count only when the sink call is unblocked

Verification

python -B -m unittest discover -s engine/tests -v

All 10 tests pass.

@BittuBarnwal7479 BittuBarnwal7479 changed the title Fix secret leak detection in tool results Add regression coverage for challenge win evaluation Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant