nix-watcher

Declarative nix-os configuration for a Prometheus monitoring VM with auto-provisioned Grafana dashboards.

Features

Prometheus exporters:

• Node exporter (local + remote systems)
• Graphite exporter (TrueNAS metrics)
• PVE exporter (Proxmox VE API)
• Unpoller exporter (Unifi Controller API)

Grafana dashboards:

• Full Node Exporter
• pfSense
• TrueNAS
• Proxmox VE
• cAdvisor containers
• Unifi controllers

Network architecture:

• Dual interface setup (admin + services networks)
• Firewall disabled - managed by hypervisor

Requirements

• a local machine with nix and/or nixos-anywhere installed
  • tested on nix-os 25.05
• a remote target server
  • accessible via SSH
  • ideally booted from a nix-os live CD (tested with nix-os minimal ISO image 25.11)
  • must have internet connection
  • must have 2 network interfaces

Deployment

1. Configure monitored systems

See the docs/ folder to setup all target systems for Prometheus monitoring.

2. Prepare host-vars.nix

Configure network settings and monitoring targets by editing host-vars.nix:

{
  config.vars = {
    users.admin.name = "admin";

    network = {
      hostname = "monitoring";
      dns = "10.0.2.1";
      gateway = "10.0.2.1";

      vAdmin = {
        interface = "ens18";
        ipv4 = "10.0.1.10";
        netmask = 24;
      };

      vServices = {
        interface = "ens19";
        ipv4 = "10.0.2.10";
        netmask = 24;
      };
    };

    prometheus.targets = {
      pfsense.socket = "10.0.1.1:9100";
      node.sockets = ["10.0.2.5:9100" "10.0.2.8:9100"];
      pve.ip = "10.0.1.5";
      cadvisor.socket = "10.0.2.20:8080";
      unifi.socket = "10.0.2.24:8443";
    };
  };
}

Full list of parameters available is described in modules/host-vars-def.nix.

After initial setup, run git update-index --skip-worktree host-vars.nix to prevent local changes from being committed.

3. Prepare extra-files/

Create secrets directory structure from template. Note extra-files/ is git-ignored. The whole folder will be copied on deployment.

cp -r extra-files-template extra-files

For Proxmox VE monitoring, refer to the docs/ folder to create an API token and add to extra-files/etc/secrets/prometheus-pve-exporter.yml.

default:
  user: "prometheus@pve"
  token_name: "prometheus"
  token_value: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  verify_ssl: false

For Unifi monitoring, refer to the docs/ folder to create a Unifi monitoring user and add its password to extra-files/etc/secrets/unifi-password:

your-unifi-password-here

Make sure the file does not contain a new line character at the end:

# no '$' symbol at the end -> OK
cat -A unifi-password
# strip new line character
tr -d '\n' < unifi-password > temp && mv temp unifi-password

4. Deploy

Boot target system with a nix-os live CD.

Note the nix-os live CD won't allow SSH access until a password is set for the root user. From the live CD console:

# as default 'nixos' user
sudo -i
# set root password
passwd

Then from your local machine run the following command:

NOTE: this will wipe the disk!!!

nix run github:nix-community/nixos-anywhere -- \
  --flake .#watcher-deploy \
  --extra-files ./extra-files \
  --generate-hardware-config nixos-generate-config ./hardware-configuration.nix \
  root@<target-ip>

After installation, the remote server will reboot into the new coonfiguration.

Post-Deployment

Security

• SSH: <admin-user>@<target-ip> (password: password)
  • change password immediately
• Setup SSH keys as required
• Setup firewall rules on hypervisor as required:
  • SSH
  • Grafana on port TCP 3000
  • Prometheus on port TCP 9090
  • Prometheus Graphite exporter on port TCP 2003

Access services:

• Grafana: http://<target-ip>:3000
  • default credentials are admin/admin
  • change admin password immediately
• Prometheus: http://<target-ip>:9090

Local Development

A second flake configuration allows rebuilding directly on the nix-watcher machine.

Start by cloning the repo.

Then you need to re-generate /etc/nixos/hardware-configuration.nix (if not present):

sudo sh -c 'nixos-generate-config --show-hardware-config > /etc/nixos/hardware-configuration.nix'

Rebuild the flake with the folllowing commands:

sudo nixos-rebuild build --flake .#watcher --impure

sudo nixos-rebuild switch --flake .#watcher --impure

sudo nixos-rebuild test --flake .#watcher --impure

Shell aliases available: just-nrb, just-nrt, just-nrs

License

This repository is licensed under the GNU General Public License v3.0 (GPL-3.0). See the full license file.

Credits

The Prometheus Graphite exporter setup was adapted from Supporterino/truenas-graphite-to-prometheus. The original graphite_mapping.conf file was converted to nix syntax in prometheus/truenas_graphite_mappings.nix.

All provisioned Grafana dashboards have been used as-is or had small edits to correctly auto-deploy. See below the complete list:

• https://grafana.com/grafana/dashboards/16877-pfsense/
• https://grafana.com/grafana/dashboards/11314-unifi-poller-uap-insights-prometheus/
• https://grafana.com/grafana/dashboards/11315-unifi-poller-client-insights-prometheus/
• https://grafana.com/grafana/dashboards/1860-node-exporter-full/
• https://grafana.com/grafana/dashboards/14282-cadvisor-exporter/
• https://grafana.com/grafana/dashboards/10347-proxmox-via-prometheus/
• https://github.com/Supporterino/truenas-graphite-to-prometheus/tree/main/dashboards

To Do

• Deploy sops-nix module to manage secrets
• Manage deployment of authorized public ssh keys
• Deploy alert manager