diff --git a/README.md b/README.md index d9eaf35..e93e6dc 100644 --- a/README.md +++ b/README.md @@ -421,7 +421,8 @@ flowchart LR | **Never** | a Proxmox node shell, OPNsense, the NAS, the router's admin, Vaultwarden, Cloudflare | | **Workflow** | PR + change note (*what, where, apply, verify, undo, result*) → owner merges → a watcher script wakes the agent → it backs up, applies exactly the note, verifies, reports | | **Rules** | never change what was not asked; ask before anything big; never merge; never commit, print or ask for secrets in chat; every commit authored by the owner | -| **Cost** | the watcher and the alerts run without the AI; it only wakes for a message or a merge | +| **Cost** | the watcher and the alerts run without the AI; it only wakes for a message, a merge, or the weekly report | +| **Weekly report** | Sunday evening: a read-only script collects 7 days of metrics, logs, alert history (Grafana, Viewer token) and open findings; the agent writes state, security and suggestions; the monitoring bot sends it | | **Weak spots** | written down, not hidden: the agent VM holds every key, `main` is not yet branch-protected, prompt injection through logs | Full architecture, access table, data flow and incidents: diff --git a/docs/23-logs-and-security-monitoring.md b/docs/23-logs-and-security-monitoring.md index fedf8f5..2a1b243 100644 --- a/docs/23-logs-and-security-monitoring.md +++ b/docs/23-logs-and-security-monitoring.md @@ -57,6 +57,8 @@ from this lab, the project is called `stacks`): ```bash docker network inspect stacks_default -f '{{range .Containers}}{{.Name}} {{end}}' # -> only the website, the tunnel connector and FTP +docker network inspect stacks_monitoring -f '{{range .Containers}}{{.Name}} {{end}}' +# -> Prometheus, Grafana, Loki, Alloy, CrowdSec and the exporters docker exec stacks-webserver-1 getent hosts pve-exporter loki # -> prints nothing: the website cannot even find them by name ``` diff --git a/docs/26-ai-agent-devops.md b/docs/26-ai-agent-devops.md index fbf71cd..b040689 100644 --- a/docs/26-ai-agent-devops.md +++ b/docs/26-ai-agent-devops.md @@ -86,6 +86,7 @@ flowchart LR PT["Pterodactyl
(subuser)"] NPM["Nginx Proxy Manager
(non-admin)"] WUD["WUD
(read-only)"] + GF["Grafana
(Viewer)"] GH["GitHub
(fine-grained token)"] end subgraph no["NO access"] @@ -111,6 +112,8 @@ flowchart LR | **Pterodactyl** | non-admin user, **subuser** on the game servers; client API key limited to VM 103's address | start/stop, console, files, backups of those servers | the panel's admin area (verified: HTTP 403) | | **Nginx Proxy Manager** | own non-admin user | manage proxy hosts and redirects, view certificates | users, settings, certificate private keys | | **WUD** (image update checker) | read-only tokens | read | anything else | +| **Grafana** | a **service account with the Viewer role** and its token (added 9 October 2026 for the weekly report) | read dashboards, alert states and the alert history | change anything: dashboards, rules, users (verified: a write attempt gets HTTP 403) | +| **Prometheus, Loki** | plain HTTP inside the LAN (Loki through `docker exec` on LXC 102) | run queries | nothing; both are read-only for queries | | **GitHub** | fine-grained token on the owner's account | branches and pull requests on the owner's repositories (technically also push to `main` and merge) | change repository settings (GitHub refuses: 403). Merging and pushing to `main` it does not do, by rule; see the weak spots | | **Monitoring logins** | copies of the read-only Proxmox exporter token, the router monitoring user, the alert bot, the game exporter key | apply and repair the monitoring | nothing beyond those (deliberately narrow, see [docs/24](24-alerting.md)) | | **OPNsense, NAS, FRITZ!Box admin, Vaultwarden, Cloudflare** | **none** | | changes there are done by the owner, from a click-guide the agent writes | @@ -187,10 +190,46 @@ The model is paid per use, so the design avoids waking it for nothing: | a message from the owner | **alerts**: Grafana sends them straight to a separate Telegram bot ([docs/24](24-alerting.md)) | | the merge-watcher, *only* when a PR was merged | the merge-watcher's checks every 2 minutes (a script) | | its own heartbeat, if the heartbeat checklist has something on it (empty = skipped) | monitoring, dashboards, logs: all run without it | +| the **weekly report**, once a week (below) | | Work that needs no judgement (alerting, collecting metrics) never goes through the agent. If the agent is down, the lab and its alerts carry on. +### The weekly report + +Once a week (Sunday 18:00) an automation starts the agent in a fresh, isolated +session that may only read files, write one temporary file and run commands. +The split is the same as everywhere else: **a script collects, the model +writes.** + +```mermaid +flowchart LR + S["collector script
(read-only)"] --> P["Prometheus
7 days of metrics"] + S --> L["Loki
logins, firewall, CrowdSec"] + S --> G["Grafana API
alerts fired, firing now"] + S --> R["REPORTS.md
open findings"] + S -- "plain facts" --> A["agent
writes 3 short parts"] + A --> B["monitoring bot
-> owner's Telegram"] +``` + +- **The collector** is a Python script with no dependencies. It asks + Prometheus for uptime *of the time each machine was monitored* (nodes that are + switched off to save power are not "down"), disk growth, CPU/RAM peaks, + guests without a backup, the internet line and the game servers; Loki for SSH + logins, blocked connections and CrowdSec alerts; Grafana for the alert history; and lists the open findings. + It prints plain text and changes nothing. +- **The agent** turns that into three short parts: overall state, security, + suggestions. Suggestions are only suggestions; nothing is changed from a + report. +- **It is sent by the monitoring bot**, the same Telegram bot the alerts come + from, so everything about the lab's health arrives in one chat. The agent's + own chat bot is only the fallback if that send fails. +- **Grafana is read with a Viewer token.** Prometheus keeps no history of which + alerts fired; Grafana does. The owner created the service account and typed + the token on the Docker host (`read -rsp`), the agent moved it to its + credentials folder and deleted the copy. +- **Cost:** one run of about 13 000 tokens and half a minute, once a week. + --- ## What leaves the house diff --git a/runbooks/26-test-grafana-changes-safely.md b/runbooks/26-test-grafana-changes-safely.md index c8eafc6..1f8f9a1 100644 --- a/runbooks/26-test-grafana-changes-safely.md +++ b/runbooks/26-test-grafana-changes-safely.md @@ -30,11 +30,12 @@ query and every alert-rule query against Prometheus and Loki (read-only, GET only), tries the Hosts dashboard once per host, and checks that no panels overlap. -Loki usually has no published port, so run the check on the Docker network, -from the folder with this repository: +Loki usually has no published port, so run the check on the monitoring Docker +network (`stacks_monitoring` here: compose project `stacks` + network +`monitoring`), from the folder with this repository: ```bash -docker run --rm --network stacks_default -v "$PWD":/w:ro python:3.13-alpine \ +docker run --rm --network stacks_monitoring -v "$PWD":/w:ro python:3.13-alpine \ python -I /w/scripts/grafana/check_dashboards.py --quiet \ --prom http://prometheus:9090 --loki http://loki:3100 \ /w/compose/monitoring/grafana/dashboards/*.json \ diff --git a/runbooks/27-ai-agent-with-least-privilege.md b/runbooks/27-ai-agent-with-least-privilege.md index 0e6e70d..1c4edbb 100644 --- a/runbooks/27-ai-agent-with-least-privilege.md +++ b/runbooks/27-ai-agent-with-least-privilege.md @@ -149,6 +149,7 @@ For every app the agent should manage, an account of its own, never yours: | AMP | its own user and role | a role with only what it needs (this lab chose broad, and says so) | | Nginx Proxy Manager | a non-admin user | "manage" for proxy hosts only; certificates "view" | | anything with read-only tokens (WUD, ...) | read-only token | | +| Grafana (only if the agent should read alerts) | **service account**, role **Viewer**, one token | Viewer cannot change anything; test it: a `POST /api/folders` with the token must return 403 | Secrets go to **one folder on the agent VM, one file per system, mode 600**: @@ -256,6 +257,30 @@ Two lessons from running it: - **Check the PR state before pushing to it.** A PR merged a minute ago is gone; a push to its branch never reaches `main`. +### Optional: a weekly report + +Same idea as the watcher: a script does the collecting, the model only writes. + +1. Write a **read-only collector** that prints plain text: uptime, disk growth, + logins and blocked connections from the logs, the open findings. Test it by + hand first. +2. Grafana → *Administration → Users and access → Service accounts → Add* → + role **Viewer** → *Add service account token*. Hand the token over the + no-chat way (above). +3. Schedule it in an **isolated** session, so it does not see your chats: + +```bash +openclaw automations add \ + --name "weekly-homelab-report" \ + --cron "0 18 * * 0" --tz "Europe/Berlin" \ + --session isolated \ + --message "Run the collector, write a short report (state, security, suggestions), send it. Change nothing." \ + --announce +``` + +Limit that job to the tools it needs (read, write, exec) and let it send through +your **alert bot** if you want everything in one chat. + --- ## 10 · First test