Expose Prometheus to the Triangle Grafana, sharing Loki's credentials - #180
Merged
Conversation
The CMS dashboard is 14 Prometheus panels to 3 Loki ones, so exposing only Loki left the Triangle Grafana rendering a mostly empty dashboard. Prometheus was reachable from inside Delta only. Front it with Nginx the same way as Loki: container published to 127.0.0.1:19090, Nginx on 9090 adding basic auth. The query API is all a Grafana datasource needs, so everything else 404s -- the web UI, /metrics, and /config, which echoes the running scrape configuration. /api/v1/admin/ is denied outright: those endpoints do not exist today because neither --web.enable-admin-api nor --web.enable-lifecycle is passed, but the deny means enabling a flag later for debugging cannot silently expose destructive endpoints. Both sites now read one htpasswd file, so the Triangle Grafana uses a single credential pair for both datasources rather than two. Verified on Delta: 401 unauthenticated, 200 authenticated, 403 on the admin API, 404 on the web UI, and sum(http_requests_total) returning real data through Nginx. The Loki endpoint still answers 200 after the htpasswd move. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CMS dashboard is 14 Prometheus panels to 3 Loki ones, so exposing only Loki left the Triangle Grafana rendering a mostly empty dashboard. Prometheus was reachable from inside Delta only.
What this does
Fronts Prometheus with Nginx the same way as Loki: container published to
127.0.0.1:19090, Nginx on9090adding basic auth.Only the query API is reachable — that is everything a Grafana datasource needs. Everything else 404s: the web UI,
/metrics, and/config, which echoes the running scrape configuration./api/v1/admin/is denied outright. Those endpoints don't exist today because neither--web.enable-admin-apinor--web.enable-lifecycleis passed, but the deny means turning a flag on later for debugging can't silently expose delete-series to everything that can reach the port.Both sites now read a single
triangle-observability.htpasswd, so the Triangle Grafana uses one credential pair for both datasources instead of two.Verified on Delta
/api/v1/labels401/api/v1/labels200/api/v1/admin/tsdb/delete_series403/404/api/v1/status/buildinfo200200sum(http_requests_total)through Nginx returns765— real traffic.Note
This adds a second cleartext basic-auth endpoint on a host where
ufwis inactive. Same tradeoff as the Loki endpoint and acceptable while VPN-internal, but it doubles what wants folding into the TLS work.🤖 Generated with Claude Code