P2: Add production monitoring after business-flow acceptance

[tomqwu]

Current Priority Decision

The Church/Basketball business milestone was accepted in #289 on 2026-09-15. Production monitoring is next after #265 provides an authorized staging target.

Execution lane: phase: platform-later. Status: queued; waiting for the #265 target and a named operator/alert destination.

Completed

Local health, diagnostics and bounded-error behavior remain covered by the merged application and local test suite.

Remaining

On the authorized staging candidate, trigger one controlled error, correlate request/error/notification evidence, prove receipt at a real owner-approved operator destination, and verify routine logs remain bounded and free of sensitive content. Keep provider integrations disabled unless separately authorized; do not use a fabricated receipt as acceptance evidence.

All tests and code review remain local. GitHub Actions does not run tests, and Ollama is not a code-review provider.

---

Current Implementation Handoff

Prepared 2026-09-13 for a lower-cost builder at source 21a4a804aa57580451edded04736b1f51aef7e48.
No CI checks. All implementation validation and code review run locally.
Use the shared builder contract and this issue's work packages; no xhigh model or automatic model upgrade is required. This is a detailed recommendation, not a claim that a smaller model cannot make mistakes or that tests have passed.

Risk/review focus: Medium/high: truthful probes, logging and alert evidence.
Start condition: WAITING ON #265 TARGET AND NAMED OPERATOR. Do not contact an alert destination or run a deployed failure drill without owner approval.

Source of Truth and Current State

• api/main.py
• api/logging_config.py
• api/database.py
• api/utils/request_id_middleware.py
• api/utils/request_context.py
• api/tasks/
• tests/api/test_readiness.py
• tests/unit/test_security_headers_extra.py

Startup logs SENTRY_DSN presence but that alone is not error-reporting initialization. /health and /ready both query the DB and expose exception strings on failure.

This handoff supersedes stale implementation statements in the background below. Preserve existing successful behavior and tests. Recheck the current branch before editing; the baseline is a source pointer, not permission to discard newer changes.

Dependencies and Ownership

• Current execution prerequisite: owner reactivates this deferred track after P1: Accept complete Church and Basketball day-to-day business flows #289. Original provider/platform dependencies are retained as future acceptance, not blockers for the current business flow.
• Coordinate, but do not wait for entire tickets to close merely to inspect/write isolated tests: P2: Validate production configuration after business-flow acceptance #258, P2: Build and validate the production artifact after business acceptance #265, P1: Make workflow notifications reliable; defer multiworker rollout #266, P2: Prove production recovery and retention after business acceptance #268, P2: Reconcile platform assurances with later release evidence #284.
• The final commands below require P1: Repair local test entry points, coverage and isolation #281's safe local environment or an equivalently verified isolated setup. A missing dependency blocks only the slice that needs it; document that state.
• Shared source/ORM/migration changes are serialized per the master roadmap. Do not start concurrent edits to the same ownership area.

Recommended Decisions

1. Recommend /health as process liveness and /ready as dependency readiness. Return generic public status and keep details only in redacted structured logs; use context-managed sessions so failures release resources.
2. Initialize the existing optional Sentry SDK once when configured, before serving traffic; ensure exception-catching middleware still reports unhandled errors. No DSN means explicitly disabled, not failed or secretly active.
3. Emit structured stdout logs with request/release/operation correlation; no unbounded application log files. Include solver time, HTTP errors/latency, broker/outbox age/failures and backup freshness signals.
4. Use bounded labels; do not label metrics by unbounded person/email/request IDs. Alerts need operator, environment, threshold, runbook and recovery notification.

Small Work Packages

Each item is one reviewable slice, not permission for one giant PR. Add the failing regression first; finish code, tests and affected docs for that slice together. Leave this issue open until all packages and original acceptance criteria are satisfied or explicitly revised by the owner.

• 267.1: Add failure-injection tests for sanitized health/readiness output and closed DB sessions.
• 267.2: Implement actual error-hook initialization and structured/redacted logs; remove misleading fatal/debug noise.
• 267.3: Add measured signal/alert rules and local fake-sink tests that prove both trigger and recovery.
• 267.4: Run one authorized end-to-end controlled error with the tested artifact and confirm real operator receipt; update P2: Reconcile platform assurances with later release evidence #284 only to that evidence level.

Required Regression Cases

These are specifications for tests to add/retain, not claimed execution results. Each new negative case must assert unchanged unauthorized state and zero forbidden side effects.

• T267-01: DB down -> liveness remains live, readiness 503 generic; repeated probes do not exhaust the pool.
• T267-02: Thrown request error -> one correlated error event, sanitized HTTP 500 and no leaked token/DB URL.
• T267-03: No configured reporting sink -> explicit disabled state; invalid sink -> observable delivery failure rather than success.
• T267-04: Queue/backup failure crosses test threshold -> alert received by test sink and clears on recovery.

Local Commands and Evidence

Existing targeted commands (paths checked against the audit source; run only after the stated safe preflight):

poetry run pytest tests/api/test_readiness.py tests/api/test_middleware.py -q
poetry run pytest tests/unit/test_security_headers_extra.py -q

Also run the shared formatting/lint/touched-type/unit/full-suite and local review protocol from #252 for the final pushed revision. Add new targeted tests to these commands when implemented. Run API and browser tiers in separate processes. Native, PostgreSQL, image, provider and operator drills require their explicit environment; an unavailable tool/target is blocked/not run, never a pass.

Schema and Compatibility

No schema migration expected for basic instrumentation. New persisted operational counters require a concrete retention reason, not convenience.

Stop Conditions

Real alert recipient, retention and escalation ownership must be chosen before claiming production monitoring. Do not trigger production incidents to test delivery.

After two failed focused repair attempts without new diagnostic evidence, stop the affected package and post the exact failure, commands, suspected boundary and needed decision. Do not silently broaden scope, weaken tests or upgrade models. A fresh local reviewer checks: Verify actual capture/delivery and redaction, resource cleanup, liveness/readiness semantics and bounded telemetry.

Completion Receipt

• Work-package and regression IDs above map to changed files and actual results.
• Commands, versions, dates, pass/fail/skip/not-run counts, logs/screenshots and tested head/base SHAs are linked.
• A separate local review records findings and resolution; self-review is labeled if used and is not misrepresented as independent review.
• Affected docs/README/playbooks/screenshots and dependency/roadmap status are reconciled, not left as unnamed follow-ups.
• If implementation is authorized through PR/merge, GitHub reports mergeable and the shared local-evidence requirements are met; reviewer agents never merge.
• No hosted CI check, status attestation, Ollama reviewer, live provider action, deployment, real-data purge or store submission was introduced by implication.

Copyable Builder Prompt

First read this issue's Current Priority Decision and #289. Run only its NOW slice.
If this issue is deferred, report that state instead of starting the older package list.
Implement the next ready work package in tomqwu/SignUpFlow issue #267.
Read its Current Implementation Handoff and #252 Builder Handoff Contract first.
Inspect current source and preserve newer/unrelated changes. Start with the
package's failing regression, then complete code, local tests, local review and
affected docs/assets together. Do not skip acceptance or invent passing evidence.
No CI checks or Ollama code review. Do not deploy, activate providers, purge real
data or submit to stores. Stop and report unmet prerequisites or policy decisions.
Record the package/test IDs and exact reviewed/tested source SHAs before claiming done.

---

Earlier Audit and Acceptance Context

Current policy (2026-09-13)

No CI checks. Everything is validated locally. This includes code review,
formatting, lint, type checks, migrations, all test tiers, security scans,
artifact checks and mobile validation. Do not add hosted jobs, required CI
statuses, synthetic success checks or an Ollama reviewer. GitHub is for source,
PRs, issues and publication, not validation.

Record commands, environment, results, limitations and reviewed head/base SHAs.
Builders merge only with completed local evidence and GitHub mergeability;
reviewer agents never merge. Real staging/provider/device acceptance remains
required where applicable, driven by authorized local operator tools.
Historical evidence and older comments do not override this policy.

Parent roadmap: #252

Priority: P1, blocks core production pilot. Phase: C. Suggested owner: Platform/operations. Original estimate (superseded; re-estimate remaining work): 2-4 engineering days.

Historical audit evidence (recheck against current source)

api/main.py:62 reports Sentry enabled based on a DSN but no sentry_sdk initialization was found in api/web. api/logging_config.py writes plain unrotated files; api/database.py emits fatal DEBUG messages on ordinary database access. Health/readiness expose raw exception text and both depend on the database. Existing readiness tests cover status codes, not alert delivery or recoverability.

Source baseline: GitHub main 214e3f3f17a582d5f9b2063be6872ea2b1d25714, audited 2026-09-09. Infrastructure/provider claims marked unverified require actual staging evidence.

Implementation plan

1. Initialize actual error reporting when configured and use a supported integration; ensure middleware handling still captures exceptions. Prove one controlled test error reaches the intended operator.
2. Use structured production logs with request ID, deployment SHA and safe tenant correlation. Remove noisy debug/fatal statements and redact tokens, cookies, secrets and unnecessary personal data; bound retention/storage.
3. Separate liveness from readiness and define which dependencies make an instance unavailable. Return generic public failures and internal diagnostic details only in protected logs.
4. Instrument HTTP error/latency rates, DB pool saturation, solver duration, queue age/failures, delivery failures and process memory.
5. Set initial alert thresholds from staging measurements; document on-call ownership and short runbooks for database outage, queue backlog, failed delivery and elevated auth failures.

Acceptance criteria

• A controlled application failure produces a correlated log/error event and a notification to the responsible operator.
• Routine requests do not emit fatal errors; logs have bounded storage and omit credentials/customer message bodies.
• Readiness removes an unhealthy instance without creating a liveness restart loop during DB outage.
• Alerts identify the environment, release SHA, affected dependency and runbook.

Dependencies

#258, #265, #266

Validation

Inject controlled failures in staging, verify actual alert receipt and log redaction, measure normal-load noise, and attach sanitized screenshots/log references.

Whole-repository audit scope (2026-09-13)

Baseline: 21a4a804aa57580451edded04736b1f51aef7e48. This addendum assigns full-scope follow-through; it is not a new test pass or production sign-off. No CI checks; all review and validation runs locally.

Current /health and /ready both open DB sessions and return raw exception text on failure; startup logs SENTRY_DSN presence without proving a reporting hook. Add bounded session cleanup on failure, sanitized public probes, actual error capture, request correlation and measured alert delivery. Include task/queue/solver, migration failure and storage/backup signals. #284 must not describe a log line/configured DSN as working monitoring. Record local mock checks separately from authorized real operator alert receipt.

Keep evidence and disposition synchronized with master roadmap #252 and documentation ledger #277. Close only after the remaining acceptance criteria have linked local results; a planning/audit note is not completion.

[tomqwu]

Monitoring package receipt (PR #322)

Completed locally and merged:

• 267.1: process liveness is dependency-free at /health; database readiness is isolated at /ready; container probes use readiness.
• 267.2: optional Sentry initialization runs before database startup, remains explicitly disabled without a DSN, strips PII/tracing, and logs sanitized startup, readiness, and unhandled failures.
• 267.3: bounded transition alerts cover database.readiness, notification.queue, and backup.freshness through local/fake sinks, including one-shot trigger and recovery behavior.
• Production stdout logs are structured JSON and redact credentials. Production requires an exact 40-character lowercase RELEASE_SHA.
• Alembic and PostgreSQL validation no longer reload inherited provider credentials from .env.

Evidence:

• Source head validated: 39a6dfbf7c22dc82c5e0810b00c327bdb95b9e11
• Squash merge on main: fd9e42755d54cc5a180c782eb39f4ae54e526888
• PR: Add production monitoring and sanitized probes #322
• make test-all: 1,814 passed, 21 skipped, 0 failed, 0 errors
• Local report: test-artifacts/local-validation/20260915T032756.627840Z-84373-48afbadb/report.json
• PostgreSQL 16.14: 8 passed
• PostgreSQL report: test-artifacts/postgres-validation/2026-09-15T032712.346832-0000-84104-3a31fbd1/report.json
• Production artifact: passed; image sha256:6f47ac21beacd0dacd29d9f589c0464ac30e6cf7b60f14717f146ae3926dbeed
• Artifact report: test-artifacts/artifact-validation/20260915T032724.799997Z-84180-dae2e4c6/report.json
• Static format, lint, focused type checks, contract snapshot, and local self-review passed.
• No hosted test or AI-review check was used. No Ollama code review exists by policy.
• make test-mobile was attempted but blocked by the host's unaccepted Xcode license; no mobile files changed, so it is not acceptance evidence for this package.

Still open:

• 267.4 requires an explicitly authorized staging target, a real alert recipient, a triggered delivery, and operator acknowledgement.
• Queue and backup producers remain to be wired by P1: Make workflow notifications reliable; defer multiworker rollout #266 and P2: Prove production recovery and retention after business acceptance #268.
• No external provider, staging environment, or paid feature was contacted or enabled.

This issue remains open until the original external acceptance criteria are evidenced.

[tomqwu]

Current-main monitoring no-regression receipt

PR #337 did not change the monitoring contract, and the refreshed current-main evidence remains clean:

• exact main cb77a9fd295880359055e8aecc96bbb9d868c19e built and passed the production-artifact harness with health/readiness-compatible two-replica operation;
• exact PR head beea983de9eccadb3be5091d5308de7821596eae passed make test-all with 2,006 passed / 21 skipped, including the operational-alert trigger/recovery tests;
• source and exact image security scanning passed with zero findings;
• external providers and real reporting recipients were not contacted.

Keep #267 open. No authorized environment, alert recipient, controlled staging failure, or operator acknowledgment was supplied, so 267.4 and the real-receipt criteria remain unclaimed.