added prometheus metrics to the live running host

[tehlers320]

Hello, i found your project and was looking to use it for some very long runs and would prefer the metric data to be sent live if possible. AFAIK you cant do this out of the box. I've tested a quick-n-dirty implementation of go-grpc-prometheus with config flags to turn it on/off.

Here is a sample dashboard all based on the examples here.

Thanks, let me know what you think.

[tehlers320]

Sorry my IDE did this should i revert all of them?

[bojand]

Sorry somehow missed this PR. Will take a look soon.

[bojand]

I am interested to learn more about your use case. I get the general idea and I think something like this could be useful. The issue with this specific implementation is that we would possibly (probably?) miss some scrape data? Prometheus normally scrapes metrics endpoints at some interval (ex 60s). So if for example you do a ghz run that runs less than 60s, or any amount of time over 1m, you could potentially miss almost a full scrape worth of data at the end. This may or may not matter, but can be problematic. And this is only one example. Essentially the process needs to run until all data has been scraped for this to be useful. So we almost need a different signal to stop the process somehow.

Additionally I think that package is deprecated in favour of github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus.

I think something like this is useful, but it might be better to try and build something that can be used programatically first and maybe generalized... that may inform opinions and offer some insights into practical way to provide it as part of CLI if possible.

I will think about this some more.