Skip to content

Make preserved artifact backfills idempotent - #679

Merged
nicho1ab merged 1 commit into
mainfrom
issue-475-preserved-artifact-idempotence
Aug 9, 2026
Merged

nicho1ab merged 1 commit into
mainfrom
issue-475-preserved-artifact-idempotence

Conversation

@nicho1ab

@nicho1ab nicho1ab commented Aug 8, 2026 •

Copy link
Copy Markdown
Owner

Pull Request Evidence

Governing issue and intended outcome

  • Governing issue: Make preserved-artifact backfills safely repeatable #475. The issue remains open after merge until its separate deployed production-style acceptance gate passes.
  • Intended outcome: Eliminate representation-only writes during repeated preserved-artifact backfills while preserving legitimate governed source changes, provenance, stable identities, import scope, audit history, and reviewer-created state.

Implementation scope

  • Major files or components changed: src/ccld_complaints/hosted_app/seeded_import.py, tests/unit/test_hosted_ccld_backfill.py, and tests/unit/test_hosted_seeded_corpus_import.py.
  • The importer compares and persists only the final projection for each stable source_record_key while retaining established deterministic flattened-record order and final-write semantics.
  • Unique source-derived records remain present.
  • Important behavior intentionally unchanged: canonical schema, migrations, data contract, live retrieval, reviewer-created state, source identity, source hashes, provenance, import scope, and genuine governed change detection.
  • Issue Fix preserved-artifact backfill failures for 27 facilities #476 hard-failure remediation is out of scope.
  • Reviewer UI regression contracts: RT-RC-004 and RT-RC-006 are affected only as protected invariants. The change preserves stable source/facility identity, valid source-document relationships, fixture integrity, reviewer-created-state separation, and audit relationships; no reviewer-facing presentation or interaction changes.

Root cause

Multiple preserved source documents can project the same stable complaint, allegation, or event identity. The importer previously compared and wrote every sequential projection. Intermediate projections temporarily changed source-document and governed-field representation before the final projection restored the same deterministic final state, so equivalent replay still reported updates. Unique source-document and extraction-audit records were not the cause.

Acceptance-criteria evidence

Acceptance criterion Evidence and result
Equivalent normalized inputs produce equivalent stored representations PASS - duplicate projections are reduced to the final projection for each exact stable source-record key before comparison/persistence.
Existing final-write behavior is preserved PASS - no sorting or alternate precedence was introduced; the final occurrence in established flattened-record order remains authoritative.
Multi-document facility replay is covered PASS - regression uses two preserved complaint reports projecting the same complaint/allegation/event identities.
Immediate equivalent dry run performs no update PASS - regression asserts updated == 0 and unchanged == 1.
Second equivalent apply performs no source-derived write PASS - regression asserts zero updates and unchanged persisted source-row snapshot.
Genuine governed source changes remain detectable PASS - Substantiated to Inconclusive produces one update and governed conflict provenance.
Conflict traceability remains deterministic PASS - retained conflict traceability is asserted across repeat processing.
Stable identities and source hashes are preserved PASS - regression snapshots stable identities and source-document hashes before and after replay.
Import scope is preserved PASS - source-record/import-batch relationships are asserted unchanged.
Reviewer-created state and audit history are preserved PASS - reviewer/audit snapshot remains unchanged through apply and repeat.

Validation and failure classification

Validation Result Failure classification, if applicable
Ruff PASS in CI none
mypy PASS in CI - 91 source files none
Full pytest suite PASS in CI - 2038 passed, 9 skipped none
Documentation check PASS in CI none
Required workflow contract PASS in CI none
New focused regressions PASS locally - 2 passed none
Backfill/import/CLI plus isolated PostgreSQL regression group PASS locally - 22 passed, 1 skipped explicit local PostgreSQL test configuration unavailable
Artifact/reload/retrieval import paths PASS locally - 53 passed none
git diff --check PASS none
Initial PR evidence verification FAIL PR evidence-body formatting/contract defect only; implementation validation passed
  • Implementation-caused test failures: None.
  • Pre-existing failures relied upon for passage: None.
  • Environmental failures: one locally configured PostgreSQL case could not run because the explicit local test-database configuration was unavailable; CI did not report an implementation test failure.
  • Tests intentionally not run: None relevant after CI completed the full repository test suite successfully.

UI and accessibility evidence (when applicable)

Not applicable - no hosted UI, reviewer presentation, interaction, accessibility behavior, or visual output changed.

Reviewer-facing redesign artifact classification (when applicable)

Not applicable - no reviewer-facing redesign occurred.

Documentation, assumptions, and remaining risks

  • Documentation impact: No documentation was changed. A pre-existing documentation mismatch regarding the later canonical-complaint-observations operation was observed and remains outside this focused correction.
  • Assumptions and limitations: The correction preserves the existing contract that one current persisted row exists per stable source_record_key and preserves established final-projection ordering semantics; the retained production 29-facility list was not available locally and was not fabricated.
  • Remaining risks or follow-up: Issue Make preserved-artifact backfills safely repeatable #475 remains open after merge. Deployment and governed QNAP acceptance must still prove the retained 29-facility replay and final 312-facility repeat dry run with updated=0, unchanged=312, warnings=0, and failed=0.

Governed-boundary review

Governed boundary Status Specific explanation or evidence
Schemas and migrations No change No schema, migration, or canonical persistence shape changed.
Ingestion and source-connector contracts Authorized change Seeded/preserved source-derived import behavior now removes intermediate duplicate projections for the same exact stable source-record key before comparison and persistence; no connector retrieval contract changed.
Security and privacy No change No authentication, authorization, secrets, privacy boundary, or private-data handling changed.
Production data and correction behavior Authorized change The correction changes preserved-artifact backfill/import comparison behavior but no production data was accessed or mutated during implementation; production acceptance remains separately governed.
Deployment and infrastructure No change No deployment, container, QNAP, Cloudflare, or infrastructure behavior changed.
Repository governance No change No governance document, workflow policy, branch policy, or development-loop rule changed.
Required GitHub workflows and checks No change No required workflow or required check changed or was weakened.
Tests or checks weakened to obtain passage Authorized change Tests were modified and added to strengthen idempotence and genuine-change regression coverage; no test, assertion, or required check was removed, skipped, relaxed, or weakened to obtain passage.

Required GitHub checks

  • validate
  • docs-check
  • fixtures
  • security

@nicho1ab
nicho1ab merged commit 6e78a28 into main Aug 9, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant