Grafana Cloud integration

[rootulp]

I've had some trouble getting this exporter to work with Grafana agent. Based on this article it seems like Grafana Cloud supports default integrations that make it easier to set up the configuration for a Grafana agent. I'm not sure what the process is for on-boarding a new integration but I figured I'd create an issue if maintainers of this project were aware / interested in it.

[tedder]

Does it not work or do you need help setting it up?

[gradoj]

well first step would be testing it first before trying the setup the integration. i didn't test it but example 2 is very close to what you'd want to do to get it running with miner_exporter instead of blackbox_exporter. did it not scrape properly?

[rootulp]

Does it not work or do you need help setting it up?

miner_exporter works for me. I'm able to verify that it's emitting metrics locally via:

[ec2-user@ip-172-31-52-73 ~]$ curl localhost:9825
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 385.0
python_gc_objects_collected_total{generation="1"} 0.0
python_gc_objects_collected_total{generation="2"} 0.0
<...>

However, I'd rather not run a Prometheus instance on my EC2 instance. I'm attempting to use Grafana agent to send metrics to Grafana Cloud Prometheus. My most recent grafana-agent.yml doesn't appear to send miner_exporter metrics back :

server:
  log_level: info
  http_listen_port: 12345

prometheus:
  wal_directory: /tmp/wal
  configs:
    - name: helium-validator-exporter
      scrape_configs:
        - job_name: helium-validator-exporter
          static_configs:
            - targets: ['localhost:9825']
      remote_write:
        - url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
          basic_auth:
            username: 123817
            password: <redacted>

integrations:
  agent:
    enabled: true
  node_exporter:
    enabled: true
  prometheus_remote_write:
  - url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
    basic_auth:
      username: 123817
      password: <redacted>

[gradoj]

is you url correct in the yml config? Shouldn't it be:
https://prometheus-us-central1.grafana.net/api/prom/push

What do you see at http://localhost:12345/metrics ?

[rootulp]

is you url correct in the yml config? Shouldn't it be:
https://prometheus-us-central1.grafana.net/api/prom/push

On Grafana cloud I see:

Remote Write Endpoint:
https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push

And I've verified this is correct because agent and node_exporter metrics do end up in Grafana cloud. However, node_exporter metrics don't.

What do you see at http://localhost:12345/metrics ?

I haven't been able to get the local server working so I don't see any metrics on the host.

[ec2-user@ip-172-31-52-73 ~]$ curl http://localhost:12345/metrics
curl: (7) Failed to connect to localhost port 12345: Connection refused

I appreciate your help on this! I'm working with Grafana support too.

[gradoj]

You using docker or binary for the grafana agent what is your command?

[rootulp]

You using docker or binary for the grafana agent what is your command?

Docker

docker run \
    -v /tmp/wal:/etc/agent \
    -v /home/ec2-user/grafana-agent.yml:/etc/agent/agent.yaml \
    grafana/agent:v0.14.0

[gradoj]

I've never run grafana agent but most config files out there have a scrape interval

prometheus:
  wal_directory: /tmp/wal
  global:
    scrape_interval: 30s

Also looks like they've released a version 0.15

[rootulp]

👍 thanks! Just upgraded to 0.15 and added an explicit scrape_interval config. Still no luck

[rootulp]

Happy to close this issue given it's probably not the right forum to troubleshoot my issue with Grafana agent. Once again, really appreciate all your help @tedder @gradoj

[tedder]

@rootulp I'm going to reopen, please come back if you fix it. If someone gets time to dig in we can talk about it here. You're probably not the only one who is gonna want to do this 👍

[gradoj]

1.Create account at grafana.com. Get API Key, username and push url from My Account - Prometheus
2.Install agent on same machine as miner_exporter from https://github.com/grafana/agent/releases/
3.Create agent.yaml file from below. Replacing username and password with info from your GrafanaCloud
4.Run agent with ./agent-linux-amd64 -config.file agent.yaml
5.verify metrics published locally with curl http://localhost:12345/metrics
6.Create new dashboard in Grafana Cloud and see validator_ metrics are available from the prometheus datasource

tedders dashboard 14319 hosted by grafana doesn't work in grafana cloud for some reason but the exporter and agent are working. Need to create your own dashboard for now

server:
  log_level: info
  http_listen_port: 12345
prometheus:
  wal_directory: /tmp/wal
  global:
    scrape_interval: 30s
  configs:
    - name: agent
      host_filter: false
      scrape_configs:
        - job_name: 'miner_exporter'
          metrics_path: /probe
          params:
            module: [http_2xx]  # Look for a HTTP 200 response.
          static_configs:
            - targets:
              - grafana.com    # Target to probe with http.
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9825  # The miner exporter's real hostname:port.
      remote_write:
        - url: https://prometheus-us-central1.grafana.net/api/prom/push
          basic_auth:
            username: 39xxxfromGrafanaCloud
            password: APIKeyfromGrafanaCloud
integrations:
  agent:
    enabled: true
  node_exporter:
    enabled: true

  prometheus_remote_write:
   - url: https://prometheus-us-central1.grafana.net/api/prom/push
     basic_auth:
       username: 39xxxfromGrafanaCloud
       password: APIKeyfromGrafanaCloud

[tedder]

As you've found in #21 this config has gotta be close. I think the instance names of the miner_exporter and node_exporter don't match and that's throwing it off.