Add SecretPatternFilter — block real credential formats, not just keywords - #73
Merged
Conversation
The bundled SensitiveDataFilter matches the literal words "secret", "password" and "api_key", so a live sk-... or AKIA... key passed straight through while ordinary prose about a password was blocked. Someone wrapping @shield_tool for credential exfiltration -- the most common reason to reach for this -- was not protected against it. SecretPatternFilter matches the shape of a credential instead of the vocabulary around it: Anthropic, OpenAI, Stripe, AWS, GitHub, Google and Slack key formats, JWTs, and PEM private-key headers. It is opt-in, so the bare @shield_tool default is unchanged. The block reason names the format and never quotes the matched text. Blocks are logged at WARNING, and a reason carrying the credential would leak the thing the rule exists to contain. Extracts the container walk shared with SensitiveDataFilter into _iter_strings so the two cannot drift. Fixes #70
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SensitiveDataFiltermatches the literal wordssecret,password, andapi_key. That means a livesk-ant-…orAKIA…key passed straight through the bundled default, while ordinary prose containing "password" got blocked. Credential exfiltration is the most common reason someone reaches for@shield_tool, and it was the case the default didn't cover.SecretPatternFiltermatches the shape of a credential rather than the vocabulary around it:Covers Anthropic, OpenAI, Stripe, AWS, GitHub (classic + fine-grained), Google, and Slack key formats, JWTs, and PEM private-key headers. Extend with
extra_patterns={...}, or replace the table withpatterns={...}.Opt-in — the bare
@shield_tooldefault is unchanged, so this is not a breaking change (as scoped in #70).Two things worth a reviewer's attention
WARNINGwith the reason attached; a reason echoing the key would leak the exact thing the rule exists to contain. It names the format and where it was found, nothing else. There's a test pinning this.\banchors on thesk-patterns. Without it,sk-[A-Za-z0-9_-]{20,}matches inside ordinary hyphenated words —task-oriented-approach-for-agentsandrisk-management-…both tripped as OpenAI keys during development. Caught before commit; regression tests added.Also extracts the container walk shared with
SensitiveDataFilterinto_iter_stringsso the two can't drift. Behavior ofSensitiveDataFilteris unchanged (its existing tests all pass untouched).Limits are documented rather than papered over, in the same spirit as the rest of the README: listed formats only, and it matches shape rather than validity — a placeholder or revoked key in the right shape blocks like a live one.
Fixes #70
Test plan
uv run ruff check ./uv run ruff format --check ./uv run mypy --strict src/modelfuzzall cleanuv run pytest -q— 155 passed (43 new), covering: every bundled format; the motivating gap in both directions (a real keySensitiveDataFiltermisses; prose it blocks that this one correctly allows); specific-beats-broad ordering (sk-ant-…reports as Anthropic, not OpenAI); the reason never echoing the secret; false-positive guards forsk-inside hyphenated words; the full container walk (nested dicts, dict keys, list/tuple/set/frozenset, bytes/bytearray, cycles); the documented custom-object limit; and all three config modes including an explicitly emptypatterns={}AGENTS.mdrule 4 (assistants previously had a standing instruction that ModelFuzz cannot detect credentials — now corrected, with the caveats to pass on)