feat(phase1-b): civiccore integration (env.py + 14 guarded migrations + 3 gate tests)

[scottconverse]

Draft. Phase 1 Part B per ADR-0003.

Completes the civiccore extraction by wiring records' env.py to civiccore's migration runner, guarding shared-schema operations across the 14 records migrations that target civiccore-owned tables, and landing the three migration-gate tests that are the Phase 1 release authority.

What landed (commits ba6ed45 + 36fd9e6, 23 files, +723/−90)

Wiring — backend/alembic/env.py

do_run_migrations() invokes civiccore.migrations.runner.upgrade_to_head() in a subprocess before records' own chain. Subprocess isolation is required because alembic's process-global context proxy cannot be safely nested inside another active env.py — civiccore's EnvironmentContext.__exit__ would clear records' active proxy state, causing AttributeError: 'NoneType' object has no attribute 'configure' on records' next line. Civiccore opens its own SYNC psycopg2 connection from DATABASE_URL inside the subprocess.

fileConfig(config.config_file_name, disable_existing_loggers=False) — see Audit resolution below for why this flag is load-bearing.

14-migration guard pass

Every shared-schema operation (create_table, add_column, alter_column, create_index, create_foreign_key, create_unique_constraint, create_check_constraint) in 001, 002, 003, 004, 006, 011, 012, 013, 014, 015, 016, 017, 018, 019, 787207afc66a now uses civiccore's idempotent_* helpers. Records-only operations on records-only tables stay unguarded (per ADR-0003). Migration 004's raw-SQL ALTER TABLE document_chunks ADD COLUMN content_tsvector and the two CREATE INDEX calls on document_chunks use native PostgreSQL IF NOT EXISTS.

Three migration gates — backend/tests/test_civiccore_migration_gates.py

• Gate 1 (fresh install): empty DB → alembic upgrade head → all 16 shared + 15 records-only tables present, alembic_version=019_encrypt_connection_config, alembic_version_civiccore=civiccore_0001_baseline_v1.
• Gate 2 (upgrade from v1.2.x): stamped-at-019 DB → upgrade → records head unchanged, civiccore baseline now stamped.
• Gate 3 (reapplication idempotency): run upgrade head twice → second run emits no "Running upgrade" lines, both heads unchanged.

Test helper uses programmatic alembic (alembic 1.18 CLI is broken — no longer auto-discovers alembic.ini), and mutates app.config.settings.database_url in place since the Pydantic singleton is captured at module-import time and doesn't see per-test DATABASE_URL changes. Both are test-harness-only adaptations; records production behavior is unchanged.

Honest caveat — Gate 2 fixture (still applies)

Gate 2 stamps alembic_version=019 on an otherwise empty DB rather than loading a real v1.2.x schema dump. This proves the stamp contract (records head preserved, civiccore stamped) but does not exercise schema-diff fidelity against an actual v1.2.x deployment. Gates 1 and 3 do the heavy schema-construction lifting. Audit recommendation: do not oversell Gate 2 as a full historical-upgrade reproduction.

Dockerfile

Added git to build-time apt install so pip can fetch the civiccore git+https dependency. Drops when civiccore 0.1.0 publishes to PyPI.

pyproject

Added civiccore @ git+https://github.com/CivicSuite/civiccore.git@e7c5570 to dependencies. Will switch to a versioned wheel before records v1.3.0 releases. integration pytest marker now registered alongside portal_mode.

Docs (36fd9e6)

CHANGELOG.md — Unreleased entry covers the civiccore dep, env.py wiring, the guarded migrations, and the three new gate tests. README.md — new "Phase 1 migration layer" subsection covers the install requirement, the git build dep, and the two-layer migration order.

Audit resolution — commit 36fd9e6

Audit on ba6ed45 flagged three blockers (TEST-002 / DOC-003 / TEST-003). All three resolved in one commit; full backend suite now green in a single pass.

TEST-002 (Blocker) — test_structured_log_on_fetch_failure was failing in the full suite (AssertionError: Expected 'Record fetch failed' log message) but passing in isolation. Root cause: backend/alembic/env.py called fileConfig(config.config_file_name) with the default disable_existing_loggers=True. Phase 1 Part B's gate tests run alembic in-process via programmatic command.upgrade, so env.py executes in the pytest parent process. fileConfig set .disabled=True on every pre-existing app.* logger, including app.ingestion.sync_runner. That state survives across tests. pytest's caplog.at_level adjusts .level and bypasses logging.disable() but does not reset the .disabled attribute, so subsequent log emissions were silently dropped and caplog.records stayed empty. Order-dependent because alphabetically test_circuit_breaker runs first and pre-imports sync_runner, so the logger exists in the registry by the time gate tests' fileConfig runs. Fix: pass disable_existing_loggers=False. This is the standard posture for alembic env.py invoked from a Python process that owns its own logger hierarchy.

DOC-003 (Major) — CHANGELOG.md Unreleased section + README.md "Phase 1 migration layer" subsection now document the new civiccore dep, the git build requirement, and the two-layer migration order. Links to ADR-0003 for the gate contract.

TEST-003 (Minor) — backend/pyproject.toml registers the integration pytest marker, eliminating the PytestUnknownMarkWarning on every gate test run.

Verification (local, raw output, single pass)

$ docker compose run --rm api python -m pytest tests/test_civiccore_migration_gates.py -v
tests/test_civiccore_migration_gates.py::test_gate1_fresh_install PASSED
tests/test_civiccore_migration_gates.py::test_gate2_upgrade_from_v1_2 PASSED
tests/test_civiccore_migration_gates.py::test_gate3_reapplication_idempotent PASSED
======================== 3 passed, 7 warnings in 3.73s =========================

$ docker compose run --rm api python -m pytest tests --tb=long -q
620 passed, 145 warnings in 759.34s (0:12:39)

620 = 617 baseline + 3 new gates. Single pass, no flakes, no order-dependent failures.

Civiccore commits consumed (all on CivicSuite/civiccore main)

• 48ff94c — alembic.ini in package_data (pip wasn't shipping it)
• 8c7db0d — split multi-statement DDL for asyncpg compat
• ac16b22 — runner opens own connection (un-nest from caller's alembic context)
• e7c5570 — 4 new idempotent_create_* guards (index, foreign_key, unique_constraint, check_constraint)

ADR-0003 merge criteria

• Gate 1 (fresh install) passes
• Gate 2 (upgrade from v1.2.x) passes — with the documented caveat above
• Gate 3 (reapplication idempotency) passes
• Full records pytest suite green in a single pass (620 passed)
• Records production code unchanged in semantic meaning (env.py adds 6 lines + subprocess wrapper + the disable_existing_loggers=False flag; migration files add idempotency guards but no schema change)
• Scott's explicit review + approval

Related: civicrecords-ai inventory PR, civiccore PR #1 (Part A, merged), ADR-0003.

🤖 Generated with Claude Code