OPS-LOG-001A add ingestion diagnostics boundaries#642
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8dd7902dd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _SENSITIVE_QUERY_KEYS = frozenset(_SENSITIVE_ASSIGNMENT_KEYS) | ||
| _URL_PATTERN = re.compile(r"(?P<url>[a-z][a-z0-9+.-]*://[^\s'\"<>]+)", re.I) | ||
| _ASSIGNMENT_PATTERN = re.compile( | ||
| r"(?i)(?P<prefix>\b(?:password|passwd|pwd|token|secret|key|dsn|connection_string)\b\s*[:=]\s*)" |
There was a problem hiding this comment.
Redact suffixed token and key assignments
When runtime diagnostics contain common credential names such as api_key=... or access_token=..., this new boundary leaves the values intact because both the query-key set and assignment regex only match exact key/token words. The helpers this replaces in the ingestion runner redacted those cases by matching key=/token= within the assignment name, and other diagnostics in this repo already classify api_key/access-key variants as sensitive, so these secrets can now be written to stdout or --summary-output unredacted. Please include the known suffixed key/token forms before emitting sanitized diagnostics.
Useful? React with 👍 / 👎.
Motivation
run-ingestionand ensure failures are sanitized.Description
src/carbonfactor_parser/diagnostics/redaction.pyexposingredact_sensitive_textandsrc/carbonfactor_parser/diagnostics/ingestion_runtime_events.pyexposingbuild_configured_cycle_summary_payloadandbuild_configured_runner_summary_payloadto build sanitized JSON-ready summaries.--summary-output <path>tocarbonops-parser run-ingestionand implemented_write_ingestion_summary_outputto write sanitized JSON summaries; the default text stdout behavior is unchanged when the option is omitted._redact_sensitive_texthelpers inconfigured_cycle_runner,production_e2e_year_orchestrator, andpostgresql_source_family_repositoryto the centralizedredact_sensitive_textto ensure consistent sanitization of emitted messages.tests/test_diagnostics_runtime_output.pyfor redaction and payload contents,tests/test_run_ingestion_summary_cli.pyfor CLI summary writing and default behavior, and updates totests/test_postgresql_source_family_repository.pyto validate repository error redaction.Testing
python -m pytest tests/test_diagnostics_runtime_output.py tests/test_run_ingestion_summary_cli.py tests/test_postgresql_source_family_repository.py tests/test_configured_cycle_runner.py, which passed (27 passed, 1 skipped).python -m pytest, which passed (2101 passed, 8 skipped).git diff --checkand ensured no generated artifacts were added and the change preserves existing CLI exit behavior when--summary-outputis omitted.Codex Task