From 3344cbe8f1a96f31be982cdab9b41088fa177a4f Mon Sep 17 00:00:00 2001 From: thatbagu Date: Sat, 21 Nov 2026 15:49:41 +0500 Subject: [PATCH] docs: a troubleshooting page, and what nixlab's version numbers mean - troubleshooting.md: what went wrong on a real cluster and the tool for each (a node that won't unlock or install, alerts, Argo CD and Longhorn states, backups that fail or need restoring, secrets). - updates.md: patch/minor/major and what each asks of you; every release's notes end with Upgrading; the weekly update keeps your nixlab tag. Co-Authored-By: Claude Opus 5.5 --- docs/SUMMARY.md | 1 + docs/troubleshooting.md | 39 +++++++++++++++++++++++++++++++++++++++ docs/updates.md | 12 ++++++++++++ 3 files changed, 52 insertions(+) create mode 100644 docs/troubleshooting.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 54d5991..fcd909d 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -18,3 +18,4 @@ * [Testing & CI](./testing.md) * [Adding Apps](./adding-charts.md) * [Managing Secrets](./secrets.md) +* [Troubleshooting](./troubleshooting.md) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..c2e78ee --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,39 @@ +# Troubleshooting + +What goes wrong in practice, and what to run. Each entry is something that happened on a real cluster. + +## A node + +| Symptom | What to do | +|---|---| +| An encrypted node doesn't come back after a reboot (no SSH on port 22) | Its TPM didn't unlock the disk, typically after a firmware or BIOS update. `nix run .#unlock -- ` gives it the passphrase and enrolls the TPM again. See [Encryption at rest](./encryption.md#unlocking-a-node-remotely). | +| `nix run .#install` hangs after "kexec", and the machine needs a power cycle | Some Intel mini PCs (N100 and others) hang in kexec. Boot them from a NixOS USB stick instead, or reinstall them with `nix run .#reinstall`. See [Adding nodes](./adding-nodes.md#machines-where-kexec-doesnt-work-intel-mini-pcs). | +| A node is off | `nix run .#wake -- `, with `wakeOnLan` set for it. | +| A node needs a clean reinstall | `nix run .#reinstall -- `. It works over SSH, encrypted or not, and Longhorn forgets the node's old disk afterwards. | +| The nightly upgrade failed (the `NixlabUpgradeFailed` alert) | On that node: `journalctl -u nixos-upgrade`. The node keeps running its previous system until the next successful upgrade. | + +## Apps + +| Symptom | What to do | +|---|---| +| An Argo CD app is OutOfSync or Degraded (`ArgoAppOutOfSync`, `ArgoAppDegraded`) | `kubectl -n argocd get applications` shows which one; its status message in the Argo CD UI says why. Argo CD keeps retrying failed syncs by itself. | +| The `longhorn` app is OutOfSync | That's expected after a change to it: Longhorn is manual-sync, so a push can't touch storage unattended. Press **Sync** on it. | +| Old pods in `Error` or `Completed` next to running ones | They're left over from a node restart. Masters delete them hourly. | +| A volume is Degraded in Longhorn after a node came back | Longhorn rebuilds the missing replica by itself. After a *reinstall*, `reinstall` already made it forget the node's old disk. | + +## Backups + +| Symptom | What to do | +|---|---| +| `AppBackupOutOfSync` or `BackupNotRecent` | A scheduled backup was missed, often because the target (R2, S3, NFS) wasn't reachable. The next run catches up on its own once it is; `sudo nixlab-backup-now /` runs one now. | +| A volume's backup fails with "repository is already locked" | An interrupted restore left its lock behind. Restores made with nixlab ≥ 1.24.2 clean this up automatically. For an older one, set the ReplicationSource's `spec.restic.unlock` to any new value, then run `sudo nixlab-backup-now /`. | +| Lost a volume | `sudo nixlab-restore /`. To restore everything as it was at one moment: `sudo nixlab-restore-all --as-of