Skip to content

Split source-family ingestion-run and source-document upserts into helpers#652

Merged
ktalpay merged 1 commit into
developfrom
codex/extract-source-document-and-ingestion-run-upsert-helpers
Jun 2, 2026
Merged

Split source-family ingestion-run and source-document upserts into helpers#652
ktalpay merged 1 commit into
developfrom
codex/extract-source-document-and-ingestion-run-upsert-helpers

Conversation

@ktalpay

@ktalpay ktalpay commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Reduce responsibilities of the PostgreSQL source-family repository by extracting the source-document and ingestion-run upsert logic into a small focused module to make the repository orchestration cleaner and easier to test.
  • Preserve existing runtime behavior, SQL text, UUID usage, parameter ordering, and conflict/fallback semantics while not touching transaction, fetch, or error redaction helpers.

Description

  • Added src/carbonfactor_parser/persistence/postgresql_source_family_upserts.py exposing ensure_ingestion_run(connection, master: SourceFamilyMasterRecord) -> None and ensure_source_document(connection, master: SourceFamilyMasterRecord) -> None and a local _execute helper; exports added via __all__.
  • Moved the ingestion-run and source-document INSERT/ON CONFLICT upsert logic into the new helpers without changing SQL text, parameter order, DO NOTHING behavior, or UUID helper usage (ingestion_run_uuid, source_document_uuid) and continued use of source_family_postgresql_value and checksum/URI fallbacks.
  • Updated src/carbonfactor_parser/persistence/postgresql_source_family_repository.py to call ensure_ingestion_run(self._connection, master) and ensure_source_document(self._connection, master) in place of the previously inline helper methods, leaving transaction/execute/fetch/_redact helpers in the repository module.
  • Added focused tests in tests/test_postgresql_source_family_repository.py that validate the extracted helpers target the correct SQL, use deterministic UUIDs and source_family_postgresql_value, preserve artifact/source-document fallbacks and checksum fallback ("checksum-unavailable"), and retain ON CONFLICT ... DO NOTHING semantics.

Testing

  • Ran python -m pytest tests/test_postgresql_source_family_repository.py and observed all tests in that file passed (21 passed, 1 skipped).
  • Ran the targeted repository/ingestion-run tests python -m pytest tests/test_configured_cycle_runner.py tests/test_ingestion_run_history_mapping.py tests/test_ingestion_run_history_repository.py tests/test_ingestion_run_history_reader.py and observed all tests passed (50 passed).
  • Ran the full test suite with python -m pytest and observed success (2162 passed, 8 skipped).
  • Ran static checks with python -m ruff check src/carbonfactor_parser/persistence/postgresql_source_family_repository.py src/carbonfactor_parser/persistence/postgresql_source_family_upserts.py tests/test_postgresql_source_family_repository.py and git diff --check and observed no issues.

Codex Task

@ktalpay
ktalpay merged commit d95fd91 into develop Jun 2, 2026
2 checks passed
@ktalpay
ktalpay deleted the codex/extract-source-document-and-ingestion-run-upsert-helpers branch June 2, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant