Skip to content

Add parser ingestion run-history schema and PostgreSQL repository contract#644

Merged
ktalpay merged 2 commits into
developfrom
codex/add-parser-ingestion-run-history-schema
Jun 2, 2026
Merged

Add parser ingestion run-history schema and PostgreSQL repository contract#644
ktalpay merged 2 commits into
developfrom
codex/add-parser-ingestion-run-history-schema

Conversation

@ktalpay

@ktalpay ktalpay commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Provide a persistent, idempotent run-history surface for parser ingestion observability (runs, per-source results, and operational issues) so downstream services can query parser execution history.
  • Keep the parser global (non-tenant) and enforce centralized redaction and JSON-safety for any persisted text/metadata to avoid storing credentials or DSNs.
  • Introduce a small repository boundary so persistence can be wired later without changing run-time parser behavior.

Description

  • Add immutable Python data contracts, validation, sanitation, and redaction for run history: ParserIngestionRunRecord, ParserIngestionSourceResultRecord, ParserIngestionIssueRecord, ParserIngestionRunHistoryCommand, ParserIngestionRunHistoryPersistResult, ParserIngestionRunHistoryIssue, and the ParserIngestionRunHistoryRepository protocol (src/carbonfactor_parser/persistence/ingestion_run_history.py).
  • Add a PostgreSQL repository implementation that validates input, sanitizes/redacts text and metadata, upserts parser_ingestion_runs by run_id, upserts parser_ingestion_source_results by the natural key (run_id, source_family, target_year), and inserts issues idempotently using deterministic stable UUIDs derived from sanitized issue identity (src/carbonfactor_parser/persistence/postgresql_ingestion_run_history_repository.py).
  • Add Phase-1 schema catalog entries and SQL rendering for three new tables: parser_ingestion_runs, parser_ingestion_source_results, and parser_ingestion_issues, including defaults, foreign keys, unique constraint and indexes; extend DDL renderer to render DEFAULT clauses and emit additive IF NOT EXISTS statements for idempotent bootstrap (src/carbonfactor_parser/persistence/postgresql_schema_catalog.py, postgresql_schema_ddl.py, postgresql_ddl_renderer.py, and postgresql_runtime_schema_bootstrap.py).
  • Add focused tests and fixtures for the new contracts and schema rendering and update bootstrap/planner tests and DDL snapshot; tests validate contract, validation rules, redaction behavior, idempotency contract (stable UUIDs), and additive SQL rendering (tests/test_ingestion_run_history_repository.py, tests/test_postgresql_phase1_schema_contract.py, updated DDL/planner/catalog tests and snapshot fixture).

Testing

  • Ran focused repository and schema tests: python -m pytest tests/test_ingestion_run_history_repository.py tests/test_postgresql_phase1_schema_contract.py — passed.
  • Ran related persistence & runtime tests (diagnostics, configured runner, and source-family repository): python -m pytest tests/test_diagnostics_runtime_output.py tests/test_configured_cycle_runner.py tests/test_postgresql_source_family_repository.py — passed (one opt-in PostgreSQL integration test remains skipped unless opted-in).
  • Ran full test suite: python -m pytest — passed with opt-in integration tests skipped (final run: 2122 passed, 8 skipped).
  • Performed git diff --check as part of validation — no whitespace or check failures reported.

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3adb989bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

%s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s, %s, %s::jsonb, NOW(), NOW()
)
ON CONFLICT (run_id, source_family, target_year) DO UPDATE SET

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make null-year source upserts hit an idempotent conflict target

When a source result has target_year=None (validation permits it and the schema declares the column nullable), PostgreSQL will not treat the nullable (run_id, source_family, target_year) unique key as a conflict for repeated inserts. Re-persisting the same null-year result therefore bypasses this ON CONFLICT target and collides with the deterministic primary key instead, causing the repository to return FAILED_DATABASE rather than being idempotent.

Useful? React with 👍 / 👎.

Comment on lines +270 to +271
record.code,
sanitized_message,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include field name in the stable issue identity

If one command reports two validation issues with the same run/source/year/stage/code/message but different field_name values, both records generate the same parser_ingestion_issue_id; the second insert then hits ON CONFLICT ... DO NOTHING and is silently dropped. Since field_name is part of the issue record and persisted column, the deterministic identity needs to include it (or another differentiator) to avoid losing per-field issues.

Useful? React with 👍 / 👎.

@ktalpay
ktalpay merged commit ff5bb43 into develop Jun 2, 2026
1 check passed
@ktalpay
ktalpay deleted the codex/add-parser-ingestion-run-history-schema branch June 2, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant