Document safe pre-backup recovery when PostgreSQL connections are exhausted #675

Goal

Document the narrow, human-operated recovery sequence required when the currently deployed pre-fix application exhausts PostgreSQL client connections and thereby prevents the mandatory pre-activation PostgreSQL dump required by the QNAP release deployment runbook.

Production evidence

During the #640 deployment gate on 2026-08-08:

• PostgreSQL remained running and healthy with restart count 0.
• PostgreSQL max_connections was 100.
• pg_stat_activity showed 93 idle application sessions from the app container plus 1 application session idle in transaction.
• Only one hosted-app Python process was running.
• PostgreSQL repeatedly logged FATAL: sorry, too many clients already.
• The runbook-required pre-activation pg_dump could not obtain a connection.
• The deployment was stopped before release extraction/activation.
• Issue Fix hosted PostgreSQL connection leakage blocking deployment #673 corrected the application connection lifecycle and merged as PR Fix hosted PostgreSQL connection lifecycle #674, but that fix is not yet deployed, so the currently running pre-fix app can still hold the leaked sessions.

Scope

• Update only the authoritative QNAP release deployment runbook and the narrowest directly related documentation/test contract if required.
• Define the exact human-only recovery condition: the running pre-fix app is verified as the source of exhausted client slots, PostgreSQL itself remains healthy, and the mandatory dump cannot connect.
• Define the bounded recovery sequence that permits stopping only the app service before the dump solely to release application-owned client sessions, while PostgreSQL remains running and its volume untouched.
• Require re-verification that PostgreSQL is still running/healthy immediately after the app stop and before the dump.
• Require the non-empty PostgreSQL dump to succeed before any release extraction, active-directory rename, application rebuild/recreation, or other activation step.
• Preserve the existing rule that PostgreSQL must never be recreated or its volume replaced for an application-only release.
• Preserve the existing application rollback model and all post-deployment verification gates.

Hard limits

• Documentation/governance correction only; no application implementation changes.
• No QNAP, SSH, container, database, deployment, rollback, restore, production-data, or Cloudflare operation by an agent.
• Do not authorize restarting or recreating PostgreSQL as the recovery mechanism.
• Do not authorize raising max_connections.
• Do not authorize routine pg_terminate_backend use.
• Do not weaken or remove the mandatory non-empty PostgreSQL backup gate.
• Do not begin Make preserved-artifact backfills safely repeatable #475, Fix preserved-artifact backfill failures for 27 facilities #476, Show complaint deficiencies and plans of correction #483, Activate and reconcile the accepted statewide facility snapshot #651, Prove resumable and idempotent statewide complaint retrieval for 2024-present #652, Execute statewide complaint population for 2024-present #653, Enrich facility gaps and reconcile report metadata after statewide population #654, Automate scheduled CCLD data refresh and recovery #478, or other statewide work.
• Epic: Bring hosted Compare Facilities to stakeholder-review readiness #640 remains blocked until this runbook correction is merged, the corrected application is deployed, and hosted owner acceptance completes.

Acceptance

• The authoritative runbook explicitly describes the exceptional pre-backup recovery condition and its narrow authorization boundary.
• The recovery sequence stops only app, never postgres.
• PostgreSQL health/running state is rechecked after stopping app and before backup.
• A verified non-empty PostgreSQL dump remains mandatory before any release activation step.
• Normal deployments retain the existing sequence; the exception is not presented as the default path.
• No application code, schema, migration, production data, Compose configuration, or infrastructure setting changes are introduced.
• Focused documentation/governance validation passes.
• After merge, the human operator resumes Epic: Bring hosted Compare Facilities to stakeholder-review readiness #640 deployment from authoritative main and proves the dump succeeds before activation.