feat(ais_core): implement Secret-Redaction-Filter (#472, Issue #1d) - #479
Merged
Conversation
- ais_core/secret_filter.py: full implementation with 21 compiled regex patterns covering GitHub PATs (classic + fine-grained), OpenAI legacy + project, Anthropic, AWS Access Key ID + Secret assignment, Google API Key + OAuth, Slack tokens + webhooks, Stripe live secret + public, Discord bot tokens, JWT, Bearer tokens, PEM private key headers, env-var assignments of SECRET/TOKEN/KEY/PASSWORD/PASS/API_KEY/AUTH, plus Twilio SIDs, Mailgun keys, SendGrid keys. - Combined alternation regex (_COMBINED_PATTERN) used by redact_secrets for performance (single re.sub pass over the input instead of 21). - redact_secrets(text) -> str; redact_dict(data) -> dict; redact_list(data) -> list. Recursive redaction of nested dicts, lists, tuples. Non-string scalars pass through unchanged. - Pure: no I/O, no env-var reads at import, no global state mutation, no logging side-effects. - Type-hints + docstrings throughout. Tests added (42 total, was 11 smoke): - TestPatternCatalog: >=20 patterns registered (21 actual). - TestRedactSecretsBasic: empty, no-secret, marker, type-error. - TestPositivePatterns: one positive test per pattern (21 tests). - TestRedactDict: flat, nested, list-of-strings, non-string values, tuple, type-error. - TestRedactList: strings, nested, type-error. - TestFalsePositives: 15 safe samples + short-value rejection. - TestPerformance: 1 MB redact under 500 ms (CI budget; soft-target 100 ms, local reality 200-300 ms documented in test docstring). - TestIdempotence: redact(redact(x)) == redact(x). Tests: 42 OK in 0.267s. Refs #472
SaJaToGu
force-pushed
the
ais-fix-472-secret-filter
branch
from
June 28, 2026 12:22
0d667f0 to
d125adc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #472 (Issue #1d — Wave 1b, Secret-Redaction-Filter)
Summary
Replaces the placeholder stub in
ais_core/secret_filter.pywith a fullpattern-based secret-detection implementation, plus comprehensive tests.
What's in this PR
ais_core/secret_filter.py21 compiled regex patterns in
SECRET_PATTERNScovering GitHub PATs(classic + fine-grained), OpenAI legacy + project, Anthropic, AWS
Access Key ID + Secret assignment, Google API Key + OAuth, Slack
tokens + webhooks, Stripe live secret + public, Discord bot tokens,
JWT, Bearer tokens, PEM private key headers, env-var assignments of
SECRET/TOKEN/KEY/PASSWORD/PASS/API_KEY/AUTH, plus Twilio SIDs,
Mailgun keys, SendGrid keys.
_COMBINED_PATTERN— single alternation regex built at import timefrom all
SECRET_PATTERNS, used byredact_secretsfor performance(one
re.subpass over the input instead of 21).redact_secrets(text) -> str— replaces all matches with[REDACTED].Pure function, no I/O, no logging side-effects.
redact_dict(data) -> dictandredact_list(data) -> list—recursive deep-copy redaction. Strings get
redact_secrets, nesteddicts/lists/tuples recurse, non-string scalars (int, float, bool,
None) pass through unchanged.
Pattern-compile flags are passed as
re.IGNORECASE | re.MULTILINEargs (not inline
(?imx)) so the pattern strings remain compatiblewith the combined alternation regex (Python's
rerejects globalflags in the middle of an expression).
Tests (42, was 11 smoke)
TestPatternCatalog: >=20 patterns registered (21 actual),all instances of
re.Pattern.TestRedactSecretsBasic: empty, no-secret passthrough,[REDACTED]marker, type-error on non-string.TestPositivePatterns: one positive test per registered pattern(21 patterns = 21 tests).
TestRedactDict: flat, nested, list-of-strings, non-stringpassthrough, tuple-in-list, type-error.
TestRedactList: strings, nested, type-error.TestFalsePositives: 15 safe samples (timestamps, branches,PR/Issue refs, normal log lines, file paths, URLs) + short-value
rejection.
TestPerformance: 1 MB redact under 500 ms CI budget. Testdocstring documents the soft-target/realistic-local-runtime/CI-
budget distinction (100 ms ideal / 200-300 ms local / 500 ms CI).
TestIdempotence:redact(redact(x)) == redact(x).Note on test samples
Slack-token + Slack-webhook positive tests use
secrets.token_hex(...)to construct fake token values at runtime.The literal strings are never present in source, which both
documents the test intent ("a real-looking fake token") and avoids
GitHub push-protection false positives when the literal patterns
match GitHub's own secret-format scanner.
Performance
The combined alternation regex on Python stdlib
retakes ~230 msfor 1 MB of safe text locally. Per #472 review (commit 654+), we
agreed:
regexpackage for 0.10.0 Issue #1d — Secret-Redaction-Filter pure helper #472.on shared runners).
high-confidence prefixes (
ghp_,sk-,AKIA,xoxb-,AIza)that run first, and slow-path (env-var, generic) only on demand.
This would bring common-case latency well under 100 ms without an
external dependency.
Tests
python -m unittest tests.test_ais_core.test_secret_filter -v→ 42 tests, 0.277s, all OK.
LOC
ais_core/secret_filter.pytests/test_ais_core/test_secret_filter.pyOut of scope (intentional)
ais_core.json_contract.success_envelope/error_envelope(planned for 0.10.0 GA, separate issue).referenced
reports/runs/**/worker.logfrom the last 30 days,but those are gitignored per repo convention; we use a curated
safe-samples list in
TestFalsePositivesinstead).Parent issue
Part of #468 (Release 0.10.0 — AIS Tooling Interface).