Scrape the CMS slots by container name, not through the host gateway - #179
Merged
Conversation
The Delta scrape config pointed at host.docker.internal:8081/8082, which cannot work: the CMS slots publish to 127.0.0.1 only, so Prometheus dialling the gateway address got connection refused on both targets. Verified on Delta -- both targets down with "dial tcp 172.17.0.1:8081: connect: connection refused". Attach Prometheus to the CMS project's network instead and scrape the containers on their internal port 8080. This needs no change to compose.cms.yml and adds no host exposure. The network is declared external, so this stack attaches to it and can never create or remove it; the cost is that the CMS stack must now be up before the observability stack starts. Blue now scrapes healthy. Green 404s until the next deploy rotates that slot onto an image containing /metrics, which is expected rather than a fault. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #178, found while rolling it out to Delta.
The bug
prometheus.delta.ymltargetedhost.docker.internal:8081/8082. That cannot work: the CMS slots publish to127.0.0.1only, so Prometheus dialling the gateway got:Both targets down. My assumption in #178 that the gateway could reach a loopback-published port was simply wrong.
The fix
Attach Prometheus to the CMS project's network and scrape the containers on their internal port 8080. No change to
compose.cms.yml, no new host exposure, and both slots stay individually visible.The network is declared
external, so this stack attaches to it and can never create, modify, or remove it — the CMS side stays untouchable from here. The real cost, stated in the compose file and README: an ordering dependency. The CMS stack must be up before this one, or Compose fails with a missing-network error rather than silently scraping nothing. I'd take a loud failure over a quiet one here.Verified on Delta
sum(http_requests_total)returns 126 — real traffic recorded.Green 404s because it still runs the pre-merge image without
/metrics. It resolves the next time a deploy rotates that slot; no action needed.Also rolled out
The full stack is now live on Delta: Prometheus running, both datasources provisioned with correct UIDs, "CMS Dashboard" loaded.
One snag worth recording: Grafana crash-looped on
Datasource provisioning error: data source not found, because its existing volume held a Loki datasource with an auto-generated UID and #178 pinsuid: loki. Provisioning cannot reconcile that. Fixed by recreating the volume, which was safe here since that Grafana had no user-created content. Anyone pinning a UID on an established Grafana will hit the same and should expect to delete the old datasource first.🤖 Generated with Claude Code