Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,8 @@ flowchart LR
| **Never** | a Proxmox node shell, OPNsense, the NAS, the router's admin, Vaultwarden, Cloudflare |
| **Workflow** | PR + change note (*what, where, apply, verify, undo, result*) → owner merges → a watcher script wakes the agent → it backs up, applies exactly the note, verifies, reports |
| **Rules** | never change what was not asked; ask before anything big; never merge; never commit, print or ask for secrets in chat; every commit authored by the owner |
| **Cost** | the watcher and the alerts run without the AI; it only wakes for a message or a merge |
| **Cost** | the watcher and the alerts run without the AI; it only wakes for a message, a merge, or the weekly report |
| **Weekly report** | Sunday evening: a read-only script collects 7 days of metrics, logs, alert history (Grafana, Viewer token) and open findings; the agent writes state, security and suggestions; the monitoring bot sends it |
| **Weak spots** | written down, not hidden: the agent VM holds every key, `main` is not yet branch-protected, prompt injection through logs |

Full architecture, access table, data flow and incidents:
Expand Down
2 changes: 2 additions & 0 deletions docs/23-logs-and-security-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ from this lab, the project is called `stacks`):
```bash
docker network inspect stacks_default -f '{{range .Containers}}{{.Name}} {{end}}'
# -> only the website, the tunnel connector and FTP
docker network inspect stacks_monitoring -f '{{range .Containers}}{{.Name}} {{end}}'
# -> Prometheus, Grafana, Loki, Alloy, CrowdSec and the exporters
docker exec stacks-webserver-1 getent hosts pve-exporter loki
# -> prints nothing: the website cannot even find them by name
```
Expand Down
39 changes: 39 additions & 0 deletions docs/26-ai-agent-devops.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ flowchart LR
PT["Pterodactyl<br/>(subuser)"]
NPM["Nginx Proxy Manager<br/>(non-admin)"]
WUD["WUD<br/>(read-only)"]
GF["Grafana<br/>(Viewer)"]
GH["GitHub<br/>(fine-grained token)"]
end
subgraph no["NO access"]
Expand All @@ -111,6 +112,8 @@ flowchart LR
| **Pterodactyl** | non-admin user, **subuser** on the game servers; client API key limited to VM 103's address | start/stop, console, files, backups of those servers | the panel's admin area (verified: HTTP 403) |
| **Nginx Proxy Manager** | own non-admin user | manage proxy hosts and redirects, view certificates | users, settings, certificate private keys |
| **WUD** (image update checker) | read-only tokens | read | anything else |
| **Grafana** | a **service account with the Viewer role** and its token (added 9 October 2026 for the weekly report) | read dashboards, alert states and the alert history | change anything: dashboards, rules, users (verified: a write attempt gets HTTP 403) |
| **Prometheus, Loki** | plain HTTP inside the LAN (Loki through `docker exec` on LXC 102) | run queries | nothing; both are read-only for queries |
| **GitHub** | fine-grained token on the owner's account | branches and pull requests on the owner's repositories (technically also push to `main` and merge) | change repository settings (GitHub refuses: 403). Merging and pushing to `main` it does not do, by rule; see the weak spots |
| **Monitoring logins** | copies of the read-only Proxmox exporter token, the router monitoring user, the alert bot, the game exporter key | apply and repair the monitoring | nothing beyond those (deliberately narrow, see [docs/24](24-alerting.md)) |
| **OPNsense, NAS, FRITZ!Box admin, Vaultwarden, Cloudflare** | **none** | | changes there are done by the owner, from a click-guide the agent writes |
Expand Down Expand Up @@ -187,10 +190,46 @@ The model is paid per use, so the design avoids waking it for nothing:
| a message from the owner | **alerts**: Grafana sends them straight to a separate Telegram bot ([docs/24](24-alerting.md)) |
| the merge-watcher, *only* when a PR was merged | the merge-watcher's checks every 2 minutes (a script) |
| its own heartbeat, if the heartbeat checklist has something on it (empty = skipped) | monitoring, dashboards, logs: all run without it |
| the **weekly report**, once a week (below) | |

Work that needs no judgement (alerting, collecting metrics) never goes through
the agent. If the agent is down, the lab and its alerts carry on.

### The weekly report

Once a week (Sunday 18:00) an automation starts the agent in a fresh, isolated
session that may only read files, write one temporary file and run commands.
The split is the same as everywhere else: **a script collects, the model
writes.**

```mermaid
flowchart LR
S["collector script<br/>(read-only)"] --> P["Prometheus<br/>7 days of metrics"]
S --> L["Loki<br/>logins, firewall, CrowdSec"]
S --> G["Grafana API<br/>alerts fired, firing now"]
S --> R["REPORTS.md<br/>open findings"]
S -- "plain facts" --> A["agent<br/>writes 3 short parts"]
A --> B["monitoring bot<br/>-> owner's Telegram"]
```

- **The collector** is a Python script with no dependencies. It asks
Prometheus for uptime *of the time each machine was monitored* (nodes that are
switched off to save power are not "down"), disk growth, CPU/RAM peaks,
guests without a backup, the internet line and the game servers; Loki for SSH
logins, blocked connections and CrowdSec alerts; Grafana for the alert history; and lists the open findings.
It prints plain text and changes nothing.
- **The agent** turns that into three short parts: overall state, security,
suggestions. Suggestions are only suggestions; nothing is changed from a
report.
- **It is sent by the monitoring bot**, the same Telegram bot the alerts come
from, so everything about the lab's health arrives in one chat. The agent's
own chat bot is only the fallback if that send fails.
- **Grafana is read with a Viewer token.** Prometheus keeps no history of which
alerts fired; Grafana does. The owner created the service account and typed
the token on the Docker host (`read -rsp`), the agent moved it to its
credentials folder and deleted the copy.
- **Cost:** one run of about 13 000 tokens and half a minute, once a week.

---

## What leaves the house
Expand Down
7 changes: 4 additions & 3 deletions runbooks/26-test-grafana-changes-safely.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,12 @@ query and every alert-rule query against Prometheus and Loki (read-only, GET
only), tries the Hosts dashboard once per host, and checks that no panels
overlap.

Loki usually has no published port, so run the check on the Docker network,
from the folder with this repository:
Loki usually has no published port, so run the check on the monitoring Docker
network (`stacks_monitoring` here: compose project `stacks` + network
`monitoring`), from the folder with this repository:

```bash
docker run --rm --network stacks_default -v "$PWD":/w:ro python:3.13-alpine \
docker run --rm --network stacks_monitoring -v "$PWD":/w:ro python:3.13-alpine \
python -I /w/scripts/grafana/check_dashboards.py --quiet \
--prom http://prometheus:9090 --loki http://loki:3100 \
/w/compose/monitoring/grafana/dashboards/*.json \
Expand Down
25 changes: 25 additions & 0 deletions runbooks/27-ai-agent-with-least-privilege.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ For every app the agent should manage, an account of its own, never yours:
| AMP | its own user and role | a role with only what it needs (this lab chose broad, and says so) |
| Nginx Proxy Manager | a non-admin user | "manage" for proxy hosts only; certificates "view" |
| anything with read-only tokens (WUD, ...) | read-only token | |
| Grafana (only if the agent should read alerts) | **service account**, role **Viewer**, one token | Viewer cannot change anything; test it: a `POST /api/folders` with the token must return 403 |

Secrets go to **one folder on the agent VM, one file per system, mode 600**:

Expand Down Expand Up @@ -256,6 +257,30 @@ Two lessons from running it:
- **Check the PR state before pushing to it.** A PR merged a minute ago is gone;
a push to its branch never reaches `main`.

### Optional: a weekly report

Same idea as the watcher: a script does the collecting, the model only writes.

1. Write a **read-only collector** that prints plain text: uptime, disk growth,
logins and blocked connections from the logs, the open findings. Test it by
hand first.
2. Grafana → *Administration → Users and access → Service accounts → Add* →
role **Viewer** → *Add service account token*. Hand the token over the
no-chat way (above).
3. Schedule it in an **isolated** session, so it does not see your chats:

```bash
openclaw automations add \
--name "weekly-homelab-report" \
--cron "0 18 * * 0" --tz "Europe/Berlin" \
--session isolated \
--message "Run the collector, write a short report (state, security, suggestions), send it. Change nothing." \
--announce
```

Limit that job to the tools it needs (read, write, exec) and let it send through
your **alert bot** if you want everything in one chat.

---

## 10 · First test
Expand Down
Loading