Skip to content

fix(spectre): remove duplicate metrics service causing double scrape#275

Merged
devkoriel merged 2 commits intomainfrom
fix/spectre-duplicate-scrape-metricrelabelings
Feb 20, 2026
Merged

fix(spectre): remove duplicate metrics service causing double scrape#275
devkoriel merged 2 commits intomainfrom
fix/spectre-duplicate-scrape-metricrelabelings

Conversation

@devkoriel
Copy link
Contributor

Summary

  • Removes service-metrics.yaml which created a duplicate Service (spectre-metrics) targeting the same pods as the main spectre Service
  • Both services exposed port 9090 with identical selector labels, causing the ServiceMonitor to discover two endpoints and produce two identical scrape jobs (spectre and spectre-metrics)
  • Adds metrics port (9090) to default service.ports in values.yaml so the main service always exposes the metrics endpoint
  • Adds metricRelabelings support to the ServiceMonitor template
  • Bumps chart version 0.3.4 → 0.3.5

Impact

  • 38,736 duplicate series eliminatedspectre-metrics was an exact copy of spectre
  • No dashboards or alerts reference the spectre-metrics job name (all use spectre)
  • Existing deployments that override service.ports to include metrics (e.g. prod-relay-appset) are unaffected
  • Deployments using default service.ports now get the metrics port from the main service instead of the removed service-metrics service

Changes

File Change
charts/spectre/templates/service-metrics.yaml Removed — duplicate service
charts/spectre/values.yaml Add metrics: {port: 9090} to default service.ports, add metricRelabelings: []
charts/spectre/templates/servicemonitor.yaml Add metricRelabelings block (gated on non-empty)
charts/spectre/Chart.yaml Version bump 0.3.4 → 0.3.5

Downstream

  • chronicleprotocol/app-of-apps (pending): bumps to 0.3.5, adds metricRelabelings to drop libp2p_holepunch_address_outcomes_total

Test plan

  • helm template produces one Service with metrics port (not two)
  • helm template renders ServiceMonitor without metricRelabelings when empty
  • helm template renders ServiceMonitor with metricRelabelings when values set
  • Chart lints cleanly

The chart created two Services (spectre and spectre-metrics) both
exposing port 9090 with the same selector labels. The ServiceMonitor
matched both, producing two identical scrape jobs (spectre and
spectre-metrics) with ~38.7K duplicate series.

- Remove service-metrics.yaml (duplicate metrics-only service)
- Add metrics port (9090) to default service.ports in values.yaml
- Add metricRelabelings support to ServiceMonitor template
- Bump chart version 0.3.4 → 0.3.5
@devkoriel
Copy link
Contributor Author

Downstream PR: chronicleprotocol/app-of-apps#1067 (consumes chart 0.3.5, adds metricRelabelings for holepunch metric)

@devkoriel devkoriel self-assigned this Feb 11, 2026
@devkoriel devkoriel added the enhancement New feature or request label Feb 11, 2026
@WesleyCharlesBlake
Copy link
Member

just need to figure out why its failing here https://github.com/chronicleprotocol/charts/actions/runs/21892153998/job/63200110770?pr=275#step:9:50

could be issues with exposing metrics

Metrics port 9090 is not available in CI kind cluster since spectre
requires valid config/keys to start its metrics server. Only test
the listen port for connectivity.
@devkoriel
Copy link
Contributor Author

just need to figure out why its failing here https://github.com/chronicleprotocol/charts/actions/runs/21892153998/job/63200110770?pr=275#step:9:50

could be issues with exposing metrics

Fixed and will deploy!

@devkoriel devkoriel merged commit d83d5e0 into main Feb 20, 2026
2 checks passed
@devkoriel devkoriel deleted the fix/spectre-duplicate-scrape-metricrelabelings branch February 20, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants