Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@
* [Testing & CI](./testing.md)
* [Adding Apps](./adding-charts.md)
* [Managing Secrets](./secrets.md)
* [Troubleshooting](./troubleshooting.md)
39 changes: 39 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Troubleshooting

What goes wrong in practice, and what to run. Each entry is something that happened on a real cluster.

## A node

| Symptom | What to do |
|---|---|
| An encrypted node doesn't come back after a reboot (no SSH on port 22) | Its TPM didn't unlock the disk, typically after a firmware or BIOS update. `nix run .#unlock -- <node>` gives it the passphrase and enrolls the TPM again. See [Encryption at rest](./encryption.md#unlocking-a-node-remotely). |
| `nix run .#install` hangs after "kexec", and the machine needs a power cycle | Some Intel mini PCs (N100 and others) hang in kexec. Boot them from a NixOS USB stick instead, or reinstall them with `nix run .#reinstall`. See [Adding nodes](./adding-nodes.md#machines-where-kexec-doesnt-work-intel-mini-pcs). |
| A node is off | `nix run .#wake -- <node>`, with `wakeOnLan` set for it. |
| A node needs a clean reinstall | `nix run .#reinstall -- <node>`. It works over SSH, encrypted or not, and Longhorn forgets the node's old disk afterwards. |
| The nightly upgrade failed (the `NixlabUpgradeFailed` alert) | On that node: `journalctl -u nixos-upgrade`. The node keeps running its previous system until the next successful upgrade. |

## Apps

| Symptom | What to do |
|---|---|
| An Argo CD app is OutOfSync or Degraded (`ArgoAppOutOfSync`, `ArgoAppDegraded`) | `kubectl -n argocd get applications` shows which one; its status message in the Argo CD UI says why. Argo CD keeps retrying failed syncs by itself. |
| The `longhorn` app is OutOfSync | That's expected after a change to it: Longhorn is manual-sync, so a push can't touch storage unattended. Press **Sync** on it. |
| Old pods in `Error` or `Completed` next to running ones | They're left over from a node restart. Masters delete them hourly. |
| A volume is Degraded in Longhorn after a node came back | Longhorn rebuilds the missing replica by itself. After a *reinstall*, `reinstall` already made it forget the node's old disk. |

## Backups

| Symptom | What to do |
|---|---|
| `AppBackupOutOfSync` or `BackupNotRecent` | A scheduled backup was missed, often because the target (R2, S3, NFS) wasn't reachable. The next run catches up on its own once it is; `sudo nixlab-backup-now <namespace>/<claim>` runs one now. |
| A volume's backup fails with "repository is already locked" | An interrupted restore left its lock behind. Restores made with nixlab ≥ 1.24.2 clean this up automatically. For an older one, set the ReplicationSource's `spec.restic.unlock` to any new value, then run `sudo nixlab-backup-now <namespace>/<claim>`. |
| Lost a volume | `sudo nixlab-restore <namespace>/<claim>`. To restore everything as it was at one moment: `sudo nixlab-restore-all --as-of <time>`. See [Backups](./backups.md#restoring-one-app). |
| Lost the whole cluster, or rebuilding it on purpose | `nix run .#reinstall -- <master> --new-cluster`: it backs up, reinstalls, lets the workers join and restores every volume. See [Backups](./backups.md#everything-into-a-new-cluster). |

## Secrets

| Symptom | What to do |
|---|---|
| `nix run .#install` says `secrets.yaml` has no `disk_encryption_passphrase` | You turned on `encrypt` after `init`. Run `nix run .#init` again: it adds what's missing. |
| The age key leaked, or you want a new one | `nix run .#rekey`, then commit and push, then `nix run .#rekey -- --finish`. See [Managing secrets](./secrets.md#re-keying-replacing-the-age-key). |
| The age key is lost | Without it `secrets.yaml` can't be decrypted, and neither can the backups whose passwords are in it. That's why `init` asks you to keep an offline copy. |
12 changes: 12 additions & 0 deletions docs/updates.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,18 @@ Check a node: `systemctl status nixos-upgrade` and `journalctl -u nixos-upgrade`

**The nodes must be able to read your repository.** A public repository just works. For a private one, give the nodes a read-only token: add `access-tokens = github.com=<token>` to `nix.extraOptions` through a SOPS secret, or turn `autoUpgrade.enable` off and deploy with `colmena apply` yourself.

## nixlab's own versions

Your `flake.nix` pins nixlab to a release tag (`github:thatbagu/nixlab/v1.24.2`). The weekly update keeps that tag; moving to a newer release is your call: change the tag, then `nix flake update nixlab`. The numbers say what to expect:

| Release | What changes for you |
|---|---|
| **patch** (1.24.1 → 1.24.2) | Fixes. Nothing to do. |
| **minor** (1.23 → 1.24) | New options and features, off or harmless by default. Your `vars.nix` keeps working as it is. |
| **major** (1.x → 2.0) | Something in `vars.nix`, or a step on the cluster, needs changing. |

Every release's notes end with **Upgrading**, which says what to do, if anything. Each release is also tested against the previous one's deployed state (`vm-upgrade`), so an upgrade can't fail on objects Kubernetes won't change in place.

## When an update breaks something

- **An app:** revert the update commit on `main` (`git revert <sha>` and push); Argo CD rolls back. Or roll back one app in the Argo CD UI (History and rollback).
Expand Down