docs: add metrics.md operator guide - #6973
Conversation
github-actions
Bot
added
area/project
area/testing
area/buildkitd
area/docs
area/worker
labels
| | `buildkit.builds` | counter | | `status`, `error_code` | Builds completed. `status` is `success` or `failure`; `error_code` carries the gRPC status code string and is present only on failure. | | ||
| | `buildkit.builds.steps` | counter | | `kind` | Build steps observed, partitioned by `kind`: `completed`, `cached`, `total`, `warnings`. | | ||
| | `buildkit.build.duration` | histogram | `s` | `status` | Wall-clock duration of build solves, from creation to completion. Exported as an exponential (native) histogram. | | ||
| | `buildkit.workers` | gauge | | | Number of workers currently registered with the daemon. | |
There was a problem hiding this comment.
What'st the use case for this value? The number of workers is currently completely internal artifact of BuildKit codebase as there is no way to run your build in anything but a single worker. (This is a different concept from buildx multi-node builders, for example.
There was a problem hiding this comment.
Would this not expose the changing number of workers across a period of time allowing administrators observability into buildkit's workload? since administrators usually do not scale buildkit per build would it not give more insight into when it is necessary to scale buildkit? Might it not provide a metric that Kubernetes Horizontal Pod Autoscaler could be tied to via an external metric and a metric adapter?
Asking as a Kubernetes administrator with little buildkit knowledge but trying to understand how buildkit could be scaled with the current metric output. I recently enabled buildkit metrics and struggled to identify a meaningful metric which could be used to scale the service. Most of the current metric output seemed to be aimed at OTEL tracing.
There was a problem hiding this comment.
Dropping the gauge from this PR. it is now docs-only, covering the existing build-completion metrics.
@tonistiigi fair point, thanks: the worker count is an internal artifact and effectively static per daemon, so it is not a useful signal on its own.
@areilly-thrive thanks as well, a metric that moves with workload so operators know when to scale buildkitd.
- an in-flight builds gauge (currently running solves an HPA-friendly load signal)
- cache-state gauges (cache size and record counts, useful for GC policy tuning)
- build stats partitioned per client
Are there other instruments either of you would want to see? Happy to take suggestions before opening the follow-ups.
1302d60 to
d3a2eb6
Compare
asbarron
changed the title
3 participants
Adds
docs/metrics.md, an operator guide documenting the OpenTelemetry metric instruments exported by buildkitd (the build-completion metrics from #6736) and how to scrape them over Prometheus (--debugaddr/metrics) or OTLP, linked from the README.An earlier revision of this PR also added a
buildkit.workersgauge; that has been dropped after review discussion — additional instruments (in-flight builds, cache size, per-client build stats) will be proposed separately.Refs #1544