Motivation
The HAR-replay e2e flakiness investigation (PR #4542/#4585) and this session's discovery that the chain-events pipeline can go fully dark (latest_indexed_block: null, latest_event_at: null — zero blocks/extrinsics for an unknown window, caught only by manually poking /health and /api/v1/blocks) both point at the same gap: nothing pages or dashboards when a live subsystem silently stops producing data. We currently rely on wrangler tail and ad hoc curl/gh run list checks — reactive, not alerting.
Scope (proposed, not yet designed)
- Blackbox freshness monitoring — an external check (BetterStack-style, or a scheduled GH Actions job) polling
/health's chain_events/operational_health/freshness fields and paging on staleness, instead of relying on someone noticing.
- Structured logging — Workers
observability.logs is already enabled (wrangler.jsonc) but nothing currently ships those logs anywhere queryable; evaluate Grafana Loki (or Cloudflare's own Logpush) for retention + search.
- Metrics/dashboards — Prometheus + Grafana (or Cloudflare Analytics Engine, which avoids running our own TSDB) for request volume, RPC proxy failover/error rate, cron job success/failure, D1 write latency.
- The specific incident that motivated this: a deployed streamer/cron silently drifting stale vs.
main (no diff-detection today) — a targeted GH Actions check comparing deployed vs. HEAD would have caught this class of bug specifically, likely before a general metrics stack would.
Non-goals (for now)
- Not committing to a specific vendor/stack yet — this issue is to scope the design, not implement blind.
- Not replacing
wrangler tail for interactive debugging.
Open questions
- Self-hosted (Prometheus/Grafana/Loki on the existing Hetzner/archive-node infra, see #the self-host program) vs. managed (BetterStack, Cloudflare-native Analytics Engine + Logpush)?
- Given current infra scale, is a full OTEL instrumentation pass justified yet, or is targeted blackbox + drift-detection sufficient for the next stage of growth?
Motivation
The HAR-replay e2e flakiness investigation (PR #4542/#4585) and this session's discovery that the chain-events pipeline can go fully dark (
latest_indexed_block: null,latest_event_at: null— zero blocks/extrinsics for an unknown window, caught only by manually poking/healthand/api/v1/blocks) both point at the same gap: nothing pages or dashboards when a live subsystem silently stops producing data. We currently rely onwrangler tailand ad hoccurl/gh run listchecks — reactive, not alerting.Scope (proposed, not yet designed)
/health'schain_events/operational_health/freshnessfields and paging on staleness, instead of relying on someone noticing.observability.logsis already enabled (wrangler.jsonc) but nothing currently ships those logs anywhere queryable; evaluate Grafana Loki (or Cloudflare's own Logpush) for retention + search.main(no diff-detection today) — a targeted GH Actions check comparing deployed vs. HEAD would have caught this class of bug specifically, likely before a general metrics stack would.Non-goals (for now)
wrangler tailfor interactive debugging.Open questions