Repository navigation
feat(monitoring): scrape the notification system services - #2165
Merged
Merged
Conversation
Add Prometheus jobs for subscription-server, dispatcher, consumers and logic-system (blockful/notification-system), a Notification System row in the Grafana dashboard (scrape status, request rate, 5xx, p99 latency) and document the NOTIF_*_ENDPOINT variables. ServiceDown already matches every non-DAO job, so the new jobs are covered without a rule change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
LeonardoVieira1630
requested review from
PedroBinotto,
alextnetto,
brunod-e and
pikonha
as code owners
September 17, 2026 18:31
|
🚅 Deployed to the anticapture-pr-2165 environment in anticapture-infra
|
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2165
September 17, 2026 18:32
Destroyed
brunod-e
approved these changes
Sep 17, 2026
…lugin Add createHttpServerMetrics / createHttpServerRequestDuration, the standard http_server_request_duration_seconds histogram with the labels the alerts and dashboard expect, and registerFastifyMetrics under @anticapture/observability/fastify, which serves /metrics and records every request. Only fastify types are imported, so the entry adds no runtime dependency (fastify is an optional peer). Fix createObservabilityProvider registering the instrumentations before the global meter provider was set: instruments were created on the API no-op meter, so the HTTP auto-instrumentation never exported http_server_duration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2165
September 17, 2026 19:33
Destroyed
Contributor
|
🔍 Vercel preview: https://anticapture-mxn6noxyz-ful.vercel.app |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
blockful's SPP2 Year 2 roadmap for the ENS DAO (forum post) promises that the notification system's uptime is measured the same way as Anticapture's, by Prometheus, and published quarterly. Today the monitoring stack only scrapes
anticapture-*jobs and the notification system only ships logs to Loki.This PR adds the notification system to the Anticapture monitoring stack:
prometheus.yml: four static jobs,notification-subscription-server,notification-dispatcher,notification-consumers,notification-logic-system, 15s interval, targets fromNOTIF_SUBSCRIPTION_SERVER_ENDPOINT,NOTIF_DISPATCHER_ENDPOINT,NOTIF_CONSUMERS_ENDPOINT,NOTIF_LOGIC_SYSTEM_ENDPOINT. No bearer: the endpoints are unauthenticated and only reachable through Railway private networking, like authful and relayer.alerts.yml: no rule change.ServiceDownmatches every job outside theanticapture-<dao>-*pattern, so thenotification-*jobs are covered as-is; the comment now says so. Because the services emithttp_server_request_duration_secondswith the same labels as Gateful (see the companion PR below), the genericHighLatencyandHighErrorRatealerts cover them too.grafana/dashboards/anticapture.json: new "Notification System" row before Traces, matching the existing rows: scrape status (same shape as General Scrape Status), request rate per service, 5xx per service and route, p99 latency per service. The General Scrape Status panel also picks the new jobs up automatically; its description was updated.README.md: new "Notification system metrics" section documenting the four env vars and which port each one needs.@anticapture/observability, plus an empty one for the infra-only files.Not included: a "notifications dispatched per channel" panel. No such metric exists yet in the notification system; a
notifications_delivered_total{channel}counter in consumers is the natural follow-up.packages/observability(minor): the request metrics the notification services need now live in the shared package instead of being copied per app.createHttpServerMetrics()/createHttpServerRequestDuration(meter): framework-agnostic definition of the standardhttp_server_request_duration_secondshistogram (name, buckets,http_request_method/http_route/http_response_status_codelabels). api, authful, gateful and user-api each define this same histogram by hand today; switching them to the helper is a follow-up.registerFastifyMetrics(server, { exporter })under the new@anticapture/observability/fastifysubpath: serves/metricsand records every request in that histogram, skipping/metricsand/health, withhttp_routeas the route pattern. Only fastify types are imported, so the entry adds no runtime dependency;fastifyis an optional peer.createObservabilityProviderregistered the OpenTelemetry instrumentations beforemetrics.setGlobalMeterProvider, so the HTTP auto-instrumentation created its instruments on the API no-op meter andhttp_server_duration/http_client_durationnever reached/metrics. The provider is now set first and passed toregisterInstrumentations. Consumers will see those two extra histograms after bumping.collectPrometheusMetricsmoved tosrc/prometheus.ts(re-exported unchanged) so the fastify entry does not pull the whole SDK; CJS output uses tsup code splitting to share one chunk betweenindex.cjsandfastify.cjs.pnpm install --frozen-lockfile.Companion PR: blockful/notification-system#276 is in draft until this package is published. It will call
registerFastifyMetricsfrom@anticapture/observability/fastifyin each service instead of shipping its own copy. Without it the scrape status panel works but the request rate, 5xx and latency panels stay empty.Steps to reproduce
Not deployed. After merge, set on the Prometheus Railway service:
Then check Status → Targets for the four
notification-*jobs and the "Notification System" row in the Anticapture dashboard.Related Issue
SPP2 Year 2 roadmap commitment (notification system uptime measured by Prometheus).
Changes
Changes to Core Features:
Additional Notes
Verification done locally:
prometheus.ymlthroughentrypoint.prometheus.shwith dummy env vars (including the DAO loop) and parsed the result as YAML: 15 jobs, the fournotification-*targets substituted correctly.promtool check configwas not run:promtoolis not installed on my machine and Docker was not running.alerts.ymlparses (18 rules).@anticapture/observability: build, typecheck, lint and 4 tests pass; gateful (182 tests) and authful (33) still pass against the changed package; a CommonJS smoke test from the notification-system worktree using the packed tarball confirmedregisterFastifyMetricsand the fixed auto-instrumentation both emit.🤖 Generated with Claude Code