Skip to content

feat(observability): the Proxmox node's CPU, memory and storages in Grafana, with alerts, from a read-only token - #136

Open
Teg79 wants to merge 2 commits into
mainfrom
feat/node-metrics
Open

Teg79 wants to merge 2 commits into
mainfrom
feat/node-metrics

Conversation

@Teg79

@Teg79 Teg79 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Proxmox node's CPU, memory and storages in Grafana, with alerts, from an optional read-only token (#94). Nothing is installed on the node.

  • internal/credentials: proxmoxMonitorToken (URBEX_PROXMOX_MONITOR_TOKEN), user@realm!name=secret. It is user-provided, so teardown keeps it.

  • platform/stacks/observability/compose.yaml: pve-exporter (prompve/prometheus-pve-exporter:3.12.0, pinned by digest), labels platform/pve-exporter, in the compose profile pve:

    • the token's parts come from the Stack's encrypted secrets (PVE_USER, PVE_TOKEN_NAME, PVE_TOKEN_VALUE);
    • it reaches the node as pve-node, through extra_hosts with the node's IP from the Stack's environment.
  • platform/stacks/observability/prometheus.yml: a pve job (/pve?target=pve-node&cluster=1&node=1, every 60 s).

    • It finds the exporter with Docker's DNS (dns_sd_configs): without the token the container doesn't exist, the name doesn't resolve, and there is no target, rather than a down one.
    • metric_relabel_configs keep only id=~"(node|storage)/.+": never the guests, which the PVEAuditor token can see.
  • internal/bootstrap/stacks.go:

    • with the token, the observability Stack gets COMPOSE_PROFILES=pve and URBEX_PVE_NODE_IP in its Komodo environment (.env), the token's parts in its secrets, and PVE_VERIFY_SSL=false;
    • without it, the environment is written with the profile empty, so removing the token turns the exporter off at the next deploy;
    • a malformed token stops the bootstrap with a message.

    PlatformStack.Environment is passed to Komodo.

  • TLS: the exporter doesn't verify the node's certificate. It reaches the node under another name, and Proxmox's own certificate is self-signed; the token only reads. Stated in the docs.

  • internal/platform/catalog.go: pve-exporter, kind platform, on the observability LXC, outbound only: it publishes nothing on the LXC.

  • Grafana:

    • Urbex resources gets a row "Proxmox node": CPU, memory used against total, each storage's use;
    • urbex-resources.yml gets three rules, all kind: platform, service: proxmox, with node/storage labels: node memory above 90% for 10 min (warning), a storage above 85% (warning) or 95% (critical) for 5 min.
  • internal/doctor/nodemetrics.go: urbex doctor's node metrics:

    • skip ("off") without the token;
    • fail if the token is only its secret (as pveum prints it on its own line: the lab's first run hit this), if Proxmox refuses it, or if it lacks Sys.Audit/Datastore.Audit on /;
    • ok otherwise.
  • CHANGELOG.md: under Unreleased.

Companion: urbforge/urbex#104.

Why

#94: the LXCs' metrics don't show the node. On Cubotto the node peaked at 9.26 of 15.6 GiB while urbex's LXCs used about 4. urbex's pool-scoped token gets 403 on the node.

Tests I ran

  • mise run test, mise run lint: pass. New tests:
    • the Stack with the token: environment, secrets, config without the secret;
    • the Stack without the token: profile empty;
    • a malformed token;
    • SplitProxmoxToken;
    • the doctor check's five cases.
  • mise run e2e: passed, 24 steps (no token: the exporter stays off and everything works as before).
  • promtool check config on the Prometheus config; the seven new PromQL expressions (alerts and panels) parse (promtool check rules).
  • docker compose config (docker 29.9.3) with the Stack's env file: COMPOSE_PROFILES=pve adds pve-exporter with pve-node=<ip>, an empty one leaves it out. The metric names and id formats were checked in the exporter's source.
  • On the test node (lab-cubotto/nodemetrics-test.sh), with a PVEAuditor token Teg created as root:
    • urbex doctor: ok node metrics;
    • urbex bootstrap: 169 s;
    • after two scrapes, Prometheus holds only node/cubotto, storage/cubotto/local, storage/cubotto/local-zfs, storage/cubotto/wall-nfs, and no guest;
    • the node's memory, Prometheus 8,444,571,648 bytes against Proxmox's API 8,438,366,208 (of 16,729,333,760), read seconds apart;
    • a second urbex bootstrap changed nothing.
  • Not run: an alert firing on the lab (the node is at 50% memory, the storages are low), and turning the token off again.

Checklist before review

  • Linked issue, Conventional Commit title
  • Tests added (unit, e2e, lab)
  • Service catalog entry, labels urbex.kind / urbex.service
  • Secrets only in credentials and the encrypted Stack secrets
  • CHANGELOG entry, docs in the companion PR
  • No secrets, tokens or private addresses

Fixes #94

🤖 Generated with Claude Code

https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA

Teg79 and others added 2 commits November 19, 2026 20:51
…rafana, with alerts, from a read-only token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
…t the token's id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): the Proxmox nodes' total CPU, memory and storage in Grafana, with alerts

1 participant