P2: Run production release acceptance after business flows are accepted

[tomqwu]

Current Priority Decision

The Church/Basketball business milestone was accepted in #289 on 2026-09-15. This issue is the final release/pilot acceptance aggregator, not the next implementation package.

Execution lane: phase: platform-later. Status: waiting for #265, #267, #268 and #284 plus owner pilot decisions.

Prepared

PR #339 merged the guarded make test-staging path. It requires an HTTPS target, exact expected release SHA, specific approval reference and explicit remote opt-in, then exercises every discovered Church/Basketball playbook plus a secure browser session. Exact head 965fede3d99871679ab4a65430edc079d343cc48 passed 2,019 tests with 21 documented skips, including all 73 Playwright tests, plus local artifact and source/image security validation. This is reusable acceptance tooling, not a staging or pilot result.

Remaining

After the authorized staging artifact, monitoring receipt, recovery/retention evidence and reconciled assurance matrix exist, freeze one candidate. Run the complete weekly Church and Basketball journeys on that built artifact, execute the approved load/recovery/rollback/device scope, update release notes/screenshots/risks, and obtain explicit owner pilot go/no-go. The owner must name the pilot cohort, support owner, data policy and go/no-go authority.

Business acceptance and local green tests do not authorize deployment or constitute production acceptance. All tests and code review remain local; GitHub Actions does not run tests, and Ollama is not a code-review provider.

---

Current Implementation Handoff

Prepared 2026-09-13 for a lower-cost builder at source 21a4a804aa57580451edded04736b1f51aef7e48.
No CI checks. All implementation validation and code review run locally.
Use the shared builder contract and this issue's work packages; no xhigh model or automatic model upgrade is required. This is a detailed recommendation, not a claim that a smaller model cannot make mistakes or that tests have passed.

Risk/review focus: High: production go/no-go and acceptance evidence.
Start condition: WAITING ON #265/#267/#268/#284 AND OWNER PILOT DECISIONS. Do not infer release approval from business acceptance.

Source of Truth and Current State

• docs/ROADMAP.md
• docs/TESTING.md
• docs/playbooks/church.md
• docs/playbooks/basketball.md
• docs/playbooks/validation.md
• docs/RUNBOOK.md
• tests/performance/test_load.py

This is an evidence-consuming release issue, not an instruction to implement every dependency in one giant PR. Source test passes do not establish a PostgreSQL artifact, real delivery or device readiness.

This handoff supersedes stale implementation statements in the background below. Preserve existing successful behavior and tests. Recheck the current branch before editing; the baseline is a source pointer, not permission to discard newer changes.

Dependencies and Ownership

• Current execution prerequisite: owner reactivates this deferred track after P1: Accept complete Church and Basketball day-to-day business flows #289. Original provider/platform dependencies are retained as future acceptance, not blockers for the current business flow.
• Coordinate, but do not wait for entire tickets to close merely to inspect/write isolated tests: P2: Regenerate the mobile API client and validate native release #191, P3: Defer paid billing and SMS product acceptance #270.
• The final commands below require P1: Repair local test entry points, coverage and isolation #281's safe local environment or an equivalently verified isolated setup. A missing dependency blocks only the slice that needs it; document that state.
• Shared source/ORM/migration changes are serialized per the master roadmap. Do not start concurrent edits to the same ownership area.

Recommended Decisions

1. Use one release matrix with rows for source validation, artifact digest, migrated PostgreSQL, browser scenarios, load, alerts, restore/rollback, delivery and optional native/paid scope. Each row needs an actual result or explicit blocker/accepted exclusion.
2. Freeze the tested source/image identity. Re-run affected acceptance when a dependency changes; never borrow results from a different commit/configuration.
3. Agree workload, hardware and thresholds before running load. A suggested starting profile is 20 concurrent users for 30 minutes, but it is not a verified capacity promise.
4. Recommend an invited web/API pilot first; optional paid/SMS/native remain disabled. The operator and owner decide go/no-go, not an agent marking checkboxes.

Small Work Packages

Each item is one reviewable slice, not permission for one giant PR. Add the failing regression first; finish code, tests and affected docs for that slice together. Leave this issue open until all packages and original acceptance criteria are satisfied or explicitly revised by the owner.

• 271.1: Fill owner/cohort/region/data/support/rollback decisions and dependency links; mark every unmet item blocked.
• 271.2: Run repaired local load tools against a disposable authorized release-equivalent target, then complete both full weekly domain journeys on the built artifact.
• 271.3: Perform authorized alert, recovery and rollback drills; capture actual message/device evidence only for approved in-scope surfaces.
• 271.4: Attach release notes, current screenshots/docs, residual-risk ledger and pilot observations; request owner go/no-go and close only after the accepted exit criteria.

Required Regression Cases

These are specifications for tests to add/retain, not claimed execution results. Each new negative case must assert unchanged unauthorized state and zero forbidden side effects.

• T271-01: Any unmet P0/core acceptance or mismatched SHA/digest -> no-go.
• T271-02: A failed load/restore/alert/browser drill -> issue remains open with reproduction, no averaging away failures.
• T271-03: A new release revision invalidates affected evidence and triggers an explicit rerun.
• T271-04: Excluded billing/SMS/native feature is visibly unavailable and cannot accidentally charge/send through a hidden route.

Local Commands and Evidence

Existing targeted commands (paths checked against the audit source; run only after the stated safe preflight):

make test-all
poetry run pytest tests/api/test_domain_playbooks.py -q
poetry run pytest tests/e2e/test_domain_playbooks.py -q

Also run the shared formatting/lint/touched-type/unit/full-suite and local review protocol from #252 for the final pushed revision. Add new targeted tests to these commands when implemented. Run API and browser tiers in separate processes. Native, PostgreSQL, image, provider and operator drills require their explicit environment; an unavailable tool/target is blocked/not run, never a pass.

Schema and Compatibility

No schema changes in this coordination issue. Migration readiness and rollback compatibility must come from #260/#265/#268.

Stop Conditions

Missing operator/cohort/target/approval or physical/provider evidence blocks the associated release stage. Do not deploy, purge, charge, message customers or publish stores from an issue plan alone.

After two failed focused repair attempts without new diagnostic evidence, stop the affected package and post the exact failure, commands, suspected boundary and needed decision. Do not silently broaden scope, weaken tests or upgrade models. A fresh local reviewer checks: Review the evidence ledger for actual outcomes, identity/config matches, unexecuted scope and owner-approved residual risks.

Completion Receipt

• Work-package and regression IDs above map to changed files and actual results.
• Commands, versions, dates, pass/fail/skip/not-run counts, logs/screenshots and tested head/base SHAs are linked.
• A separate local review records findings and resolution; self-review is labeled if used and is not misrepresented as independent review.
• Affected docs/README/playbooks/screenshots and dependency/roadmap status are reconciled, not left as unnamed follow-ups.
• If implementation is authorized through PR/merge, GitHub reports mergeable and the shared local-evidence requirements are met; reviewer agents never merge.
• No hosted CI check, status attestation, Ollama reviewer, live provider action, deployment, real-data purge or store submission was introduced by implication.

Copyable Builder Prompt

First read this issue's Current Priority Decision and #289. Run only its NOW slice.
If this issue is deferred, report that state instead of starting the older package list.
Implement the next ready work package in tomqwu/SignUpFlow issue #271.
Read its Current Implementation Handoff and #252 Builder Handoff Contract first.
Inspect current source and preserve newer/unrelated changes. Start with the
package's failing regression, then complete code, local tests, local review and
affected docs/assets together. Do not skip acceptance or invent passing evidence.
No CI checks or Ollama code review. Do not deploy, activate providers, purge real
data or submit to stores. Stop and report unmet prerequisites or policy decisions.
Record the package/test IDs and exact reviewed/tested source SHAs before claiming done.

---

Earlier Audit and Acceptance Context

Current policy (2026-09-13)

No CI checks. Everything is validated locally. This includes code review,
formatting, lint, type checks, migrations, all test tiers, security scans,
artifact checks and mobile validation. Do not add hosted jobs, required CI
statuses, synthetic success checks or an Ollama reviewer. GitHub is for source,
PRs, issues and publication, not validation.

Record commands, environment, results, limitations and reviewed head/base SHAs.
Builders merge only with completed local evidence and GitHub mergeability;
reviewer agents never merge. Real staging/provider/device acceptance remains
required where applicable, driven by authorized local operator tools.
Historical evidence and older comments do not override this policy.

Progress reconciliation (2026-09-13)

PR #275 reconciled testing docs and router-registration summaries. Broader security/readiness claims and the operations runbook still require reconciliation; do not assume those were certified.

Parent roadmap: #252

Priority: P1, blocks core production pilot. Phase: D. Suggested owner: Release owner plus QA/product. Original estimate (superseded; re-estimate remaining work): 4-6 engineering days plus proposed seven-day pilot.

Historical audit evidence (recheck against current source)

The repository has extensive API/web/browser workflows and prior completed marathon issues, but current audit checks found important behavior outside that coverage. tests/performance/test_load.py exists and is not run by current CI. docs/LAUNCH_ROADMAP.md includes stale completion percentages, pricing/provider assumptions and already-completed tasks. AGENTS/CLAUDE describe disabled surfaces that are now mounted. Open ICS PR #251 and existing mobile issue #191 are still unresolved.

Source baseline: GitHub main 214e3f3f17a582d5f9b2063be6872ea2b1d25714, audited 2026-09-09. Infrastructure/provider claims marked unverified require actual staging evidence.

Implementation plan

1. Publish a concise release matrix: API/web core, invitation/reset email and inbox included; optional paid/SMS/mobile explicitly excluded until their gates pass. Reconcile README, AGENTS, CLAUDE, Copilot instructions and runbook with registered routes and the local-only validation policy.
2. Have the owner name the pilot cohort, support/release operators, deployment region and data policy. Estimate dates only after critical issues have owners and remaining effort is re-evaluated.
3. Run the release artifact on PostgreSQL under production settings: org creation/invitation, real emailed reset, availability/recurrence, constraint-aware solve, compare/publish/rollback, claim/decline/swap, notification preferences, calendar and supported exports.
4. Resolve PR Sprint 4 follow-up — implement ICS format for /solutions/{id}/export #251 by normal review after export authorization is fixed; do not count the open PR as shipped ICS support. Verify every advertised export or mark it unavailable.
5. Define workload and initial targets before benchmarking. Suggested pilot test: 20 concurrent users, representative maximum tenant dataset, 30-minute sustained mixed traffic, bounded solver runs, zero tenant leaks/overfilled shifts, HTTP error rate <1%, ordinary reads p95 <500ms on the named staging hardware; tune targets from measured customer needs.
6. Run browser tests with security controls enabled, accessibility/keyboard/mobile-width checks, interrupted-network recovery and a release rollback/restore drill.
7. Operate a small invited pilot for a proposed seven stable days with metrics and support feedback. The release owner makes and records the go/no-go; unresolved P0/P1 core blockers prohibit general availability.

Acceptance criteria

• Every in-scope journey has evidence tied to the release SHA, artifact digest and production-like configuration.
• Security fixes have independent local review; builders refuse merging without complete local evidence and GitHub mergeability.
• Load, alert delivery, queue recovery, backup restore and rollback meet agreed targets with recorded results.
• Core pilot issues are closed based on evidence, and optional feature status is clear to users.
• Release notes, operator/support runbook, accepted-risk register and owner go/no-go are linked here.
• General availability occurs only after the stable pilot exit criteria are met.

Dependencies

#253, #254, #255, #256, #257, #258, #259, #260, #261, #262, #263, #264, #265, #266, #267, #268, #269

Validation

Run make test-unit and make test-all, contract/web/e2e suites, production PostgreSQL tests, artifact/security checks and the measured staging load profile. Browser/device outcomes must be real runs, not inferred from HTML/unit checks. Preserve logs/reports and record tests not run.

Whole-repository audit scope (2026-09-13)

Baseline: 21a4a804aa57580451edded04736b1f51aef7e48. This addendum assigns full-scope follow-through; it is not a new test pass or production sign-off. No CI checks; all review and validation runs locally.

Consume the full audit backlog, not only the previous short documentation follow-up: #277 documentation/spec ledger; #278 README/examples/site; #279 Church/Basketball coverage; #280 current screenshots; #281 complete local validation; #282 agent lifecycle; #283 safe tools; #284 security/ops claims; #285 whole-web usability; #286 publication safety. Include all six weeks, secondary events, accepted/swapped/repaired final rosters, DST, role/team policies and failure recovery in the release matrix. Existing performance tests hard-code localhost:8000 and create data, so repair isolation under #281 before measured load. Native/provider tracks remain optional; source/unit green is not installed artifact, PostgreSQL, physical-device or operator evidence. No production-ready wording or issue closure without the linked acceptance results.

Keep evidence and disposition synchronized with master roadmap #252 and documentation ledger #277. Close only after the remaining acceptance criteria have linked local results; a planning/audit note is not completion.

[tomqwu]

Local release-preparation receipt and current stop boundary

Local release-oriented gaps discovered during the repository-wide audit are now repaired:

• PR Add tenant-scoped solution ICS export #333 merged tenant-scoped solution ICS export and deterministic OpenAPI/Dart client regeneration.
• PR Correct stale API, environment, and mobile guidance #334 marked removed REST/CLI and environment-runtime guidance historical, redirected readers to maintained sources, and corrected the mobile invitation smoke boundary.
• Exact tested head e6bc24b6a0df4b585f3ed797e90ec22435619872: make test-all 1,985 passed / 21 skipped / 0 failed / 0 errors, including 73 Playwright.
• SHA-bound report: test-artifacts/local-validation/20260915T183125.899040Z-48293-4928b80a/report.json.
• make test-docs: 188 manual / 138 generated; make validate-screenshots: passed.

#271 is not complete and should remain open. Its remaining acceptance requires:

1. P1: Accept complete Church and Basketball day-to-day business flows #289 owner business acceptance.
2. Owner decisions for pilot cohort, region/data handling, support, rollback, and go/no-go.
3. An explicitly authorized release-equivalent target, built artifact, and load workflow.
4. Actual alert receipt, recovery/retention, rollback, and named-device evidence.
5. Release notes, residual-risk review, pilot evidence, and owner production decision.

No deployment, live provider call, paid activation, production data operation, hosted CI, or Ollama code review was authorized or performed.

[tomqwu]

271.1 completion receipt

PR #335 merged the canonical release acceptance matrix to main as 1f13b51b0fb79e6e831df0047ada4cf962568d9f.

Completed:

• Recorded every unresolved owner decision as BLOCKED_OWNER.
• Defined candidate scope, with billing and paid SMS excluded and native release separately gated.
• Separated local source, artifact, PostgreSQL/Redis, security, recovery, external delivery, target, load, alert, rollback, pilot, and go/no-go evidence.
• Marked earlier artifact/database evidence INVALIDATED for a future candidate instead of borrowing results across revisions.
• Added exact execution order, invalidation/no-go rules, and safe local commands.
• Documented the current performance suite as fixed loopback smoke coverage, not sustained release load evidence.

Exact validation:

• Reviewed/tested head: 3d487503ef2cbfa91c61d129be692e4ddbd6ac0b
• Base: 924fc0351166bebb9870fed0e44b737ff7234bb5
• make test-all: 1,985 passed / 21 skipped / 0 failed/errors, including 73 Playwright
• Report: test-artifacts/local-validation/20260915T190726.719395Z-63852-979fe9c6/report.json, tracked_tree_clean=true
• make test-unit: 738 passed / 21 skipped
• Focused docs/policy: 15 passed
• make test-docs: 189 manual / 138 generated
• Screenshot validation, Black, Ruff, diff, links, and secret-pattern review passed.
• GitHub reported PR Add a release acceptance matrix #335 MERGEABLE / CLEAN before builder merge; local self-review was used and is not represented as independent.

#271.1 is checked. #271.2 through #271.4 remain open for the candidate-bound load harness/run, authorized release-equivalent target, alert/recovery/rollback drills, release notes/pilot evidence, and owner go/no-go. No deployment, provider call, hosted CI, Ollama review, branch deletion, or store submission occurred.

[tomqwu]

Source-bound load-harness foundation receipt

PR #336 merged as 75cc94fd5d3fe5b1fcdb1a740e1d746e33db315c after GitHub reported MERGEABLE / CLEAN.

Completed locally:

• Added make test-load, a bounded profile, an owned loopback server/database/process lifecycle, fictional tenant seeding, raw request samples, threshold evaluation, and exact X-Release-SHA verification.
• Kept the legacy fixed make test-performance suite separate and classified both as opt-in scopes.
• Remote targets require explicit authorization, HTTPS, an exact expected SHA, and a release_candidate profile with an approval reference.
• External providers stay disabled and inherited provider credentials are not loaded.
• Self-review found unbounded duration/RPS/timeout and remote HTTP acceptance; failing regressions were added first and the runner now caps 3,600 seconds, 1,000 RPS, and 120-second request timeout and requires remote HTTPS. Self-review is not represented as independent review.

Exact tested head 810eb224c055f564936246bc4bebd54047274b99, base 1f13b51b0fb79e6e831df0047ada4cf962568d9f:

• make test-load: 72 requests / 0 errors / 11.98 RPS / 97.419 ms overall p95; all seven operations sampled; expected and observed SHA matched.
• Load report: test-artifacts/load-validation/20260915T195037.549970Z-73014-6d04a672/report.json; release_evidence=false.
• make test-all: 1,993 passed / 21 skipped / 0 failed / 0 errors, including 73 Playwright.
• Full report: test-artifacts/local-validation/20260915T195103.532794Z-73132-0fbb9f6b/report.json; tracked_tree_clean=true and exact source SHA matched.
• make test-unit: 744 passed / 21 skipped. make test-docs: 189 manual / 138 generated. Black, Ruff, touched mypy, diff and secret-pattern review passed.

271.2 remains unchecked. This receipt proves only the local harness foundation and bounded source smoke. It does not prove the owner-approved workload, named hardware, immutable artifact, PostgreSQL/Redis target, built-artifact Church/Basketball journeys, sustained load, staging, operator receipt, rollback, provider delivery, or production readiness. No deployment, provider call, production-data operation, hosted CI, Ollama review, branch deletion, or store submission occurred.

[tomqwu]

Tenant-safe external-email prerequisite receipt

PR #337 merged to main as cb77a9fd295880359055e8aecc96bbb9d868c19e after GitHub reported MERGEABLE / CLEAN for exact validated head beea983de9eccadb3be5091d5308de7821596eae.

Completed locally:

• scheduling messages now carry tenant and notification identity as SendGrid custom arguments;
• the signed SendGrid callback is mounted only when email is enabled and is hidden from the public schema;
• callback mutations require matching organization, notification, and provider-message identity;
• the existing provider-event ledger provides durable duplicate handling, attempt tracking, and reconciliation state;
• processing failures return a retryable error instead of acknowledging and losing the event;
• production configuration fails closed unless both the SendGrid API key and webhook verification key are present;
• worker/beat configuration and current docs/runbooks were reconciled.

Exact validation:

• make test-all: 2,006 passed / 21 skipped / 0 failed / 0 errors, including 73 Playwright;
• report: test-artifacts/local-validation/20260915T202718.968598Z-83207-b47448a0/report.json, exact source SHA and tracked_tree_clean=true;
• make test-unit: 751 passed / 21 skipped;
• make test-docs: 189 manual / 138 generated;
• Black, Ruff, touched-module mypy, diff/secret review passed;
• local self-review was used and is not represented as independent review.

This does not complete #271. No SendGrid account was contacted, no provider was enabled, and no live inbox/callback, release target, alert receipt, rollback, or owner go/no-go evidence was produced. Email remains disabled by default.

[tomqwu]

Current-main local candidate refresh

After PR #337, all available owned local release-preparation checks were rerun at exact main cb77a9fd295880359055e8aecc96bbb9d868c19e:

• production artifact: passed; digest sha256:fbf8d65e4ed7a9c63dc6230e1ed25fee81a27e71c96c008ea9e9e20fdef7c217; report test-artifacts/artifact-validation/20260915T204739.531422Z-92487-264e40f3/report.json;
• PostgreSQL 16.14: 10/10 passed; report test-artifacts/postgres-validation/2026-09-15T204813.715061-0000-92841-432c997c/report.json;
• encrypted isolated recovery: 12/12 passed; report test-artifacts/recovery-drill/20260915T204824.548972Z-93216-331f13a2/report.json;
• bounded source load: 72 requests / 0 errors / 11.979 RPS / 189.714 ms p95, all seven operations sampled and expected/observed SHA matched; report test-artifacts/load-validation/20260915T204841.523451Z-93455-efd9d101/report.json;
• source and exact image security scan: passed with 0 blocking, accepted, or reported findings; report test-artifacts/security-validation/20260915T204935.370990Z-93963-08986ad3/report.json;
• full local gate from PR Harden tenant-scoped SendGrid delivery events #337: 2,006 passed / 21 skipped / 0 failures/errors, including 73 Playwright; report test-artifacts/local-validation/20260915T202718.968598Z-83207-b47448a0/report.json.

The load report remains release_evidence=false. These are owned local checks, not #271.2 or #271.3 completion. No authorized release-equivalent target, owner-approved workload, staging HTTPS, promoted artifact, real alert recipient, provider delivery, rollback/cutover, pilot, or owner go/no-go was supplied or exercised.

[tomqwu]

Progress receipt (2026-09-15)

PR #339 is merged. The repository now has a reusable make test-staging operator path that binds acceptance evidence to an exact expected release SHA and runs all pluggable church/basketball playbooks plus a secure browser-session smoke. The command is deliberately outside make test-all and cannot contact a remote target without HTTPS, a specific approval reference, and explicit remote opt-in.

This prepares the pilot acceptance mechanism but does not satisfy the issue: no frozen candidate has been deployed to a disposable staging target, no pilot has been run, and no owner acceptance has been recorded.