Skip to content

Develop to main#655

Merged
ktalpay merged 286 commits into
mainfrom
develop
Jun 18, 2026
Merged

Develop to main#655
ktalpay merged 286 commits into
mainfrom
develop

Conversation

@ktalpay

@ktalpay ktalpay commented Jun 18, 2026

Copy link
Copy Markdown
Owner

Summary

Briefly describe the task and outcome.

Changes

Validation

  • Tests/checks run and listed:
  • git diff --check passed

Scope Guard Checklist

  • No confidential data
  • No production/compliance/legal claims
  • No source-specific ingestion unless requested
  • No parser/database/scheduler/downloader coupling unless requested
  • No external dependencies unless requested
  • One commit only
  • Tests/checks run and listed

Deferred Items

Reviewer Notes

futureopsltd and others added 30 commits May 11, 2026 14:50
…pository-contract

[DN-043] [DN-043] .NET Parser run repository contract
…-parser-run-repository-contract

[PT-043] [PT-043] Parity review for Parser run repository contract
…ily-master-detail-repository-contract

[PY-044] [PY-044] Python Source-family master/detail repository contract
…-master-detail-repository-contract

DN-044: .NET Source-family master/detail repository contract
…retry-and-recovery-visibility

[OPS-030] [OPS-030] Add PR creation retry and recovery visibility
…-source-family-master-detail-repository-contrac

[PT-044] [PT-044] Parity review for Source-family master/detail repository contract
…scovery-runtime-boundary

[DN-045] [DN-045] .NET GHG source discovery runtime boundary
…label-state-transitions-in-workflows

[OPS-031] [OPS-031] Fix task status label state transitions in workflows
…wnload-execution

[DN-046] [DN-046] .NET GHG source download execution
…discovery-runtime-boundary

[DN-047] [DN-047] .NET DEFRA source discovery runtime boundary
…-defra-source-discovery-runtime-boundary

[PT-047] [PT-047] Parity review for DEFRA source discovery runtime boundary
…-download-execution-parity

[OPS-032] [OPS-032] Align GHG source download execution parity
…-ghg-source-download-execution

[PT-046] [PT-046] Parity review for GHG source download execution
…download-execution

[DN-048] [DN-048] .NET DEFRA source download execution
ktalpay added 28 commits June 2, 2026 10:14
…and-runtime-output

OPS-LOG-001A add ingestion diagnostics boundaries
…-duplication-and-redaction

Fix diagnostics issue de-duplication and expand redaction key coverage
…istory-schema

Add parser ingestion run-history schema and PostgreSQL repository contract
…history-persistence

Wire configured runner run history persistence
…istory-read-repository

Add ingestion run history reader boundary
…rtifact-transport

Split configured cycle config loading and artifact transport
…tion-module

Split configured cycle dependency composition
…history-helpers

Split configured cycle summary and history helpers
…builders

OPS-REF-002A: split PostgreSQL source-family stable ID and SQL builders
…and-json-helpers

OPS-REF-002B: Split PostgreSQL source-family parameter mapping
…d-ingestion-run-upsert-helpers

Split source-family ingestion-run and source-document upserts into helpers
…on-helpers

OPS-REF-002D: split PostgreSQL execution helpers
…cumentation

OPS-POLISH-001: Polish public repository status docs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5bb6ab9bd9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

def is_success(self) -> bool:
"""Return whether the insert summary has no failed rows."""

return self.failed == 0 and not self.status.startswith("failed")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat no-record inserts as unsuccessful

When a parser returns a valid but empty batch, PostgreSQLSourceFamilyRuntimeRepository reports status='no_records', attempted=0, and failed=0; this predicate treats that as success, so _run_source_family advances record_ingested_year and future cycles skip that year even though no factor rows were persisted. Please require actual attempted rows/records, or treat no_records as a failed/no-op insert, before marking the source year completed.

Useful? React with 👍 / 👎.

Comment on lines +279 to +282
status = (
ParsedFactorPersistenceStatus.DECLARED
if repository_result.status is SourceFamilyRepositoryPersistStatus.DECLARED
else ParsedFactorPersistenceStatus.FAILED_VALIDATION

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve database-failure status from repositories

When repository.persist_source_family_records returns SourceFamilyRepositoryPersistStatus.FAILED_DATABASE (for example after a PostgreSQL insert error), this collapses every non-DECLARED result into FAILED_VALIDATION. Direct callers of persist_parsed_factor_records then see final_status='failed_validation' for an infrastructure/database failure and cannot distinguish it from bad input data, so retry/error handling will be misleading; preserve or expose the database-failure status here.

Useful? React with 👍 / 👎.

@ktalpay
ktalpay merged commit eb044ee into main Jun 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants