No alerting is wired — every failure is discovered by a miner (submit outage + 429 bug both found this way)

[wallscaler]

Problem (100x reliability audit, CRITICAL — reliability is the #1 challenge)

Alert SPEC is complete (deploy/observability/ALERTS.md defines PAGE/WARN thresholds) but alerting is NOT wired (deploy/RELIABILITY_UPGRADE_PLAN.md:916 "[ ] P7: alerts and dashboards live" UNCHECKED). No scheduled watcher exists in the repo. This is the "429 found by a miner, not monitoring" gap, generalized — and it just happened AGAIN with the 2026-07-05 submit outage (found by miner report, not an alert).

Signals that exist in code and go NOWHERE:

• weights.refresh_health() (weights.py:1507) — a dead refresh thread is invisible until the 30-min 503 cliff.
• ratelimit.unresolved_ip_count() (ratelimit.py:56-68) — the Harden client-IP derivation: trusted-proxy guard for rate limiting on un-proxied origins #333 fail-open counter has ZERO callers; the limiter can silently un-throttle whole paths.
• verify-worker heartbeats written to DB (app.py:1654-1666) but no endpoint reads heartbeat age.
• /v1/admin/validator-health (app.py:4635) computes level=page but nothing polls it.

Fix (~1 day)

One Cloudflare Worker cron (you already run two) every 1-5 min:

• GET weights/next on all three validator URLs → assert 200 + generated_at age <5min + source != stale_fallback.
• GET validator-health → assert no level=page, weights_feed_5xx flat, unresolved_ip_count flat (once wired), worker heartbeat age <60s.
• Push failures to Discord/Telegram/email.

Converts every other finding in the audit from "a miner tells Fred" to "Fred knows first." Also: wire unresolved_ip_count() (from #333) and worker heartbeat age into validator-health so the cron can watch them.

[wallscaler]

@ai-hpc — fresh read-only telemetry, runbook-route, traffic, and outage-notifier status

Reviewed current main at exact SHA c8028af479861a61072b20fc2f93620b9c599fe7 and probed only public read-only production routes. No deployment, credentialed admin request, paid action, or chain write was performed.

Current verdict

Still open. Cathedral now has useful telemetry endpoints/counters and an alert specification in deploy/observability/ALERTS.md, but deploy/RELIABILITY_UPGRADE_PLAN.md still records P7: alerts and dashboards live as unchecked. I found no scheduled notifier implementation proving the full detect → deduplicate → deliver → acknowledge → recover lifecycle.

In other words: observability ingredients exist; live outage notification remains NOT PROVEN.

Fresh public service snapshot

At the review time:

• https://api.cathedral.computer/health/live → 200, body status ok.
• Canonical, legacy-prefixed, and read-host public weight feeds → 200.
• All three weight-feed bodies were byte-identical:
  • SHA-256: d792166c3bbeccc2cc80e1589495f74cb8b3dcb740f50cedf1eaa1e8219164c4
  • size: 3025 bytes
  • generated_at: 2026-07-27T00:31:07.634Z

That is a useful point-in-time availability/consistency PASS. It is not evidence that alert delivery works, that submit traffic is healthy, or that an outage would page anyone.

Runbook/deployment route mismatch

The documented readiness probe /health/ready returned 404 on the public API variants checked. /openapi.json also returned 404.

This is not being labelled a production outage: /health/live and the public weight feeds were healthy. It is a deployed route/runbook contract mismatch. A monitor copied literally from the current documentation could false-page continuously or block a deployment while the service is actually serving its supported paths.

Required resolution:

1. Establish one versioned public route registry as source of truth.
2. Either deploy /health/ready with a precise dependency contract or update every runbook, deployment probe, and monitor to the actual supported readiness route.
3. Add a contract test that checks documented probes against the deployed route map before release.
4. Classify 404 as route_contract_mismatch until the route registry says the endpoint must exist; do not automatically call every missing candidate path an outage.

Traffic visibility

Current submit/runtime traffic is behind admin-protected telemetry. A public read-only audit therefore cannot prove current request volume, success rate, queue age, receipt throughput, or customer impact. Status: NOT PROVEN from public surfaces.

Repository traffic is available but is not product traffic: the public cathedral repository showed 197 views / 27 unique visitors and 795 clones / 184 unique cloners over the available 14-day window through July 25. Those numbers must never be used as a proxy for validator, miner, API, or customer workload health.

The notifier/dashboard needs, at minimum:

• requests by route/status class;
• submissions accepted/rejected/retried;
• p50/p95/p99 latency;
• receipt production/verification rate and age;
• epoch/weight-feed age and digest consistency across mirrors;
• worker/validator heartbeat age;
• queue depth and oldest-item age;
• auth failures, rate limits, 5xx/network failures;
• notification delivery, deduplication, acknowledgement, and recovery state.

All labels must avoid hotkeys, bearer/API keys, prompts, workload inputs/results, raw quote bodies, or unbounded IDs.

Concrete recurrence: Distill corpus incident

The new operational issue cathedral-distill#2 is a concrete example of why this P0 remains relevant:

• a near-complete remote-teacher batch lost all in-memory rows;
• no request ledger retained exact traffic or spend;
• current retry exhaustion and auth failure are not cleanly distinguishable;
• there is no wall-clock heartbeat, spend/request ceiling, deduplicated outage lifecycle, or second-operator runbook;
• raw provider error-body fragments could contaminate notifier payloads.

Distill needs its own run-level structured events and alert thresholds, while Cathedral provides the shared delivery, secret-redaction, deduplication, acknowledgement, and recovery infrastructure.

Launch-grade notifier acceptance

• One declared route map drives docs, probes, monitors, and deployment checks.
• Scheduled alert evaluation is deployed and externally supervised.
• Alerts have stable fingerprints and opened → updated → recovered state.
• Delivery failure is itself observable; at least one fallback channel exists.
• --notify-test or equivalent proves delivery before risky runs.
• Synthetic drills prove live-route failure, stale weights, submit failure, queue stall, worker silence, and recovery.
• Alert payload redaction is tested with sentinels.
• Dashboards expose current traffic and data age without credentials or sensitive cardinality.
• A second operator can follow the committed runbook from symptom to reconciliation and closure.

Until those gates are demonstrated, the accurate status is: service availability currently PASS at the public live/weight routes; runbook route consistency FAIL; runtime traffic health NOT PROVEN publicly; automated outage notification NOT PROVEN.