Skip to content

OPS-LOG-001A add ingestion diagnostics boundaries#642

Merged
ktalpay merged 1 commit into
developfrom
codex/add-centralized-redaction-and-runtime-output
Jun 2, 2026
Merged

OPS-LOG-001A add ingestion diagnostics boundaries#642
ktalpay merged 1 commit into
developfrom
codex/add-centralized-redaction-and-runtime-output

Conversation

@ktalpay

@ktalpay ktalpay commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Centralize and standardize redaction logic so all runtime diagnostics pass through a single safe boundary instead of duplicated helpers.
  • Provide a lightweight, machine-readable ingestion run summary boundary so orchestration can emit sanitized structured output for downstream consumers.
  • Preserve existing human-readable stdout behavior while adding an opt-in JSON summary output for run-ingestion and ensure failures are sanitized.

Description

  • Added a new diagnostics package with src/carbonfactor_parser/diagnostics/redaction.py exposing redact_sensitive_text and src/carbonfactor_parser/diagnostics/ingestion_runtime_events.py exposing build_configured_cycle_summary_payload and build_configured_runner_summary_payload to build sanitized JSON-ready summaries.
  • Wired an optional CLI flag --summary-output <path> to carbonops-parser run-ingestion and implemented _write_ingestion_summary_output to write sanitized JSON summaries; the default text stdout behavior is unchanged when the option is omitted.
  • Replaced/delegated the local _redact_sensitive_text helpers in configured_cycle_runner, production_e2e_year_orchestrator, and postgresql_source_family_repository to the centralized redact_sensitive_text to ensure consistent sanitization of emitted messages.
  • Added focused tests: tests/test_diagnostics_runtime_output.py for redaction and payload contents, tests/test_run_ingestion_summary_cli.py for CLI summary writing and default behavior, and updates to tests/test_postgresql_source_family_repository.py to validate repository error redaction.

Testing

  • Ran focused tests with python -m pytest tests/test_diagnostics_runtime_output.py tests/test_run_ingestion_summary_cli.py tests/test_postgresql_source_family_repository.py tests/test_configured_cycle_runner.py, which passed (27 passed, 1 skipped).
  • Ran full test suite with python -m pytest, which passed (2101 passed, 8 skipped).
  • Performed git diff --check and ensured no generated artifacts were added and the change preserves existing CLI exit behavior when --summary-output is omitted.

Codex Task

@ktalpay
ktalpay merged commit e6fa3a5 into develop Jun 2, 2026
2 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8dd7902dd8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +25 to +28
_SENSITIVE_QUERY_KEYS = frozenset(_SENSITIVE_ASSIGNMENT_KEYS)
_URL_PATTERN = re.compile(r"(?P<url>[a-z][a-z0-9+.-]*://[^\s'\"<>]+)", re.I)
_ASSIGNMENT_PATTERN = re.compile(
r"(?i)(?P<prefix>\b(?:password|passwd|pwd|token|secret|key|dsn|connection_string)\b\s*[:=]\s*)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Redact suffixed token and key assignments

When runtime diagnostics contain common credential names such as api_key=... or access_token=..., this new boundary leaves the values intact because both the query-key set and assignment regex only match exact key/token words. The helpers this replaces in the ingestion runner redacted those cases by matching key=/token= within the assignment name, and other diagnostics in this repo already classify api_key/access-key variants as sensitive, so these secrets can now be written to stdout or --summary-output unredacted. Please include the known suffixed key/token forms before emitting sanitized diagnostics.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant