Redact credentials in urllib3's DEBUG request logs - #209
Merged
Merged
Conversation
urllib3 logs every request line, query string included, at DEBUG, so running with DEBUG logging still wrote the Census API key into logs after #207. morpc.req now attaches a filter to the urllib3.connectionpool logger that redacts key, token, and api_key in its records. The request details stay in the log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #207 (#206).
Problem
After #207, morpc's own logs no longer contain the Census API key. With logging at DEBUG, though, urllib3 (used by
requests) logs every request line, query string included, e.g."GET /data/2024/acs/acs5?get=...&key=<key> HTTP/1.1" 200, so the key still reached the logs.Change
morpc.reqattaches alogging.Filterto theurllib3.connectionpoollogger. The filter runsredact()over each record's message and string arguments. It's installed whenmorpc.reqis imported, so it applies whether or notmorpc.logs.config_logsis used. The request details stay in DEBUG output with credentials replaced byREDACTED, so they're still useful for debugging; urllib3 isn't silenced.Testing
test_urllib3_debug_request_log_is_redacted: it logs a request line the way urllib3 does and asserts the host is kept and the key is redacted. It failed before the change.test_req(12),test_frictionless,test_gpkg,test_morpc,test_release, andtest_geocodeall pass.test_utils: the same 4 datetime tests fail onmain.CensusAPIfetches (ACS 5-year 2024 places, 2017070): 281 urllib3 request lines logged, and the key doesn't appear anywhere in the log.🤖 Generated with Claude Code