Things I learned while building and operating a homelab on the path to becoming a DevOps engineer.
Each entry is a short summary with a link to a detailed explanation. Organized by topic.
Work in progress. Corrections and feedback welcome.
| Topic | Summary |
|---|---|
| LVM Thin Provisioning | How thin-pool storage works, why df lies inside containers, and how to reclaim space with fstrim |
| Namespaces & nsenter | What Linux namespaces are, how LXC containers use them, and how to enter them from the host |
| ELF Binaries & Corruption | What ELF format is, how to detect a corrupt binary, and how to reinstall it |
| apt & dpkg | apt update vs apt upgrade, dpkg --verify with conffile filtering, dpkg audit and repair, Ansible upgrade: dist modes, cache hygiene |
| systemd Basics | Unit types, systemctl commands, journalctl filtering, persistent journald (Storage= auto trap), mount units, Type=oneshot, drop-in overrides |
| systemd Service Hardening | Restart=on-failure, RestartPreventExitStatus, After= vs Wants=, race-condition fixes, reactive vs proactive ExecStartPre readiness gates, the start-limit trap (reset-failed), fail-open vs fail-closed gates |
| systemd Mount Units & the Network-Mount Boot-Race | fstab→.mount unit escaping, _netdev/nofail (no retry!), why a CIFS share ends up failed after a boot-race, LXC bind-mount propagation needs a container restart, CIFS POSIX-mode is cosmetic |
| Time Synchronization | System clock vs RTC, NTP/chrony vs timesyncd, timedatectl/chronyc, why drift breaks SnapRAID & alert math, LXC inherits host clock vs VM needs its own |
| Bash Scripting Patterns | Strict mode, pre-flight checks, command -v, install -m, HEREDOC, sub-commands, mktemp+trap; $? lies in a pipe → PIPESTATUS[0]; capturing a non-zero exit under set -e (&& rc=0 || rc=$?) to inspect vs || true to swallow |
| SSH Keys | Ed25519 keys, agent forwarding caveats, authorized_keys hygiene, break-glass fallback key, key rotation; additive management never revokes — exclusive: true is per invocation (a loop keeps only the last key), inline join("\n") yields a literal \n, an empty key list truncates the file |
| Cron and Scheduling | crontab vs /etc/cron.d/, systemd timers, log conventions; the decision rule — "cron is fine for simple jobs" hides the premise that the machine is up at the scheduled time, and a failing cron job leaves no state to alert on; calendar vs monotonic timers (Persistent= is ignored on monotonic); Persistent=true does not fire on first start (stamp file, measured); template units (unit@.service, %i, Unit=); systemd escape processing in ExecStart=; retiring a hand-written crontab line with sed -E '/pat/d' (not grep -v, which exits 1 on empty output) |
| Mount Existence vs Mount Identity | A recurring silent-failure class: when a network mount fails, consumers see the empty mountpoint directory underneath it. [ -d ], mountpoint -q and Proxmox mkdir 0 all pass; only findmnt -no FSTYPE asserts identity. Plus: a failed precondition must exit 1, or the systemd unit never reaches failed and no alert can fire |
| Disk Diagnostics | smartctl, dmesg, identifying drive failures, SMART attribute thresholds; who reported the error — hostbyte/driverbyte/sense key, DID_SOFT_ERROR (transport) vs Medium Error (media), cmd_age = timeout; why -H PASSED is worthless as evidence and Pending>0 with Reallocated=0 is the bad case (spares are consumed on write, not read); which controller a disk is really on; comparing boots via journalctl -k -b -N |
| Failing-Disk Data Rescue | Rescue before repair: read-only ro,noload mount, ownership-preserving tar stream (vs rsync), socket ignored vs real I/O errors, ddrescue for damaged metadata, sparse allocated-vs-live |
| Network Tools | nc -zv, ss, findmnt, Python socket fallback, layer-by-layer reachability |
| Input-Device Reconnects | Why a BT controller reuses its eventX path while inputN-derived nodes (LEDs) renumber; key identity on a stable id, detect reconnect by presence not path, re-assert device state after settle; D-Bus bisection + sysfs-authority debugging |
| Atomic File Writes | Crash-safe state files: temp in the same filesystem + fsync + rename/os.replace; why truncate-in-place silently loses data; Ansible/Terraform/etcd parallels |
| DualSense Lightbar: LED Class vs. Raw HID | Kernel LED class vs. raw HID output reports; the firmware "light out" latch (sysfs sticks, hardware dark); why a driver rebind doesn't clear it but a raw colour report does; DualSense report 0x31/CRC32 seed 0xA2 vs USB 0x02; the two-firmware setup-flag gotcha; Steam-Input coexistence; bisection debugging + validated privileged helper |
| Topic | Summary |
|---|---|
| Playbook Structure | Minimum required fields, plays vs tasks, YAML indentation rules |
| Task Control | register, changed_when, ansible.builtin.fail, when with Jinja2 filters, shell vs command, docker --format/Jinja2 collision in ad-hoc commands; ansible.builtin.cron cannot remove an entry it did not write (it finds entries by its own #Ansible: marker) — adoption pattern for hand-deployed jobs; making a role dry-runnable: check_mode: false on read-only probes, when: not ansible_check_mode on systemctl enable; assert as a precondition that deploys first and fails with a diagnosis |
| Privilege Escalation | become, become_user, and how to configure NOPASSWD sudo for Ansible |
| Inventory Groups | Group structure, per-host overrides, host_vars/group_vars, sanitized inventory pattern |
| Serial Execution | Why parallel upgrades are dangerous and how serial prevents resource spikes |
| Ansible Configuration | ansible.cfg settings explained: host_key_checking, pipelining, ControlMaster, fork count |
| Roles | Why roles exist, directory structure, defaults/ vs vars/, files/ vs templates/, binary deployment pattern, handlers |
| Jinja2 Templates | Generating config files from inventory: groups[], hostvars[], for loops, if conditions, mixing static and dynamic sections, template module |
| Ansible Vault | AES-256 secrets encryption in Ansible: why plaintext in Git is permanent, breach window concept, group_vars split pattern, key commands, vault password file |
| SSH Hardening | lineinfile module pattern, regexp workflow for sshd_config directives, --check --diff dry-run habit, handler reload vs restart, idempotency signals; sshd_config.d first-match-wins — a 50-cloud-init.conf drop-in beat the hardened line (changed=0 but unhardened); fix by sorting a 00-hardening.conf before it, verify with sshd -T (effective config, not the line you wrote); verifying the applied change needs a non-multiplexed connection (ControlMaster=no, ControlPath=none) or you reuse a socket and never re-authenticate; check authorized_keys before PermitRootLogin no |
| Docker Compose Updates | docker_compose_v2 for fleet updates: pull/recreate idempotency, per-host compose_projects list + loop/item, group targeting vs safe no-op default scoping; the sync gap — a role that acts on remote state but never ships the repo compose file is a silent no-op (:latest ran for weeks despite a repo pin); {dest, src} list shape, promoting a loop scalar to a dict |
| PostgreSQL Provisioning | Declarative DB-tenant onboarding via community.postgresql: peer auth (become_user: postgres), module-per-step mapping, plus two transferable traps — acl needed for unprivileged become, and never co-locating a secret with the loop item (Ansible dumps it on failure) |
| Walkthrough: Fleet Docker Updates & Postgres Provisioning | Two roles end to end on the shared group→host_vars→role→playbook skeleton: the become/docker.sock root-cause story, pull vs recreate, the verification ladder, and Postgres' four access layers, peer auth, and Vault-backed tenant provisioning |
| Ad-Hoc Commands | ansible <host> -m <module> -a "<args>" syntax, command vs shell, ping (not ICMP), --become trap, and the live-state verification pattern (verify before trusting docs) |
| Fleet Health Checks & hostvars | hostvars magic variable for cross-host data aggregation, multi-play reporting pattern, service_facts, docker_host_info, find+file cleanup loop, copy with Jinja2 content, role-vs-inline decision rule; two silent-no-op bugs — Docker play needs become (ansible user not in docker group), and a report play targeting the control node's inventory name matches zero hosts (control node is deliberately not in inventory → use hosts: localhost + connection: local) |
| GitHub Actions: CI/CD for Ansible | Minimal ansible-lint pipeline on push + PR; why cd ansible && ansible-lint . (CWD-relative ansible.cfg); requirements.yml for collections; common lint rules + fixes; handler name matching gotcha; pipefail explained; pinning the version vs default-profile drift; the profile ladder (min→production) + .ansible-lint config; vault in CI (dummy password; syntax-check parses but never decrypts; internal-error masking); role-name → var-naming cascade + waiver; --fix reformat scope + excluding the vault file; a finding ≠ a defect — read the rule source (_executable_options allow-list; is-failed flagged, reset-failed not); three suppression levels (inline # noqa: id vs bare vs skip_list), pick the narrowest |
| Control-Node Hygiene | Playbooks execute from the working tree, not from a commit — a control node on a feature branch or left mid-merge runs code matching no commit and reports success; --ff-only, the pre-run cleanliness grep, and why CI cannot substitute for it (it only sees committed markers); recovering a stuck merge without losing stashes (rev-list --left-right before touching anything, cp -a incl. .git, merge --abort) |
| Topic | Summary |
|---|---|
| Thin-Pool Recovery | How to diagnose and recover from a full LVM thin-pool on a Proxmox host |
| LXC & VM Management | pct and qm, LXC vs VM, mount points, boot order, nesting=1, /dev/disk/by-id, bind-mount propagation; passthrough safety — a disk both host-mounted and passed to a VM means two kernels on one ext4 and there is no cross-host locking; ps -C kvm vs qm config, correlating by filesystem UUID, safe qm set --delete, and is_mountpoint 1 so a failed mount can't fill the host root |
| Tailscale TUN in Unprivileged LXCs | CT210-pattern: cgroup2.devices.allow + mount.entry for kernel WireGuard, userspace-networking pitfall |
| Hard Shutdown Recovery | LXC boot failures after forced power-off: SMB mount dependency (exit 19), SSH/Tailscale race condition, access hierarchy, recovery order |
| Diagnosing a Frozen VM | Guest hard-freeze while qm status still reads running: guest-health probes (guest-agent + ACPI timeout, blank serial console) vs hypervisor state, qm stop/start recovery, the abrupt-journal-stop freeze signature; two timeline traps — align guest-UTC vs host-local clocks before reasoning, and absence-of-data ≠ absence-of-problem (--list-boots/uptime); detection ≠ response, and why the in-guest NMI watchdog is unreliable under KVM |
| LXC Bindmount: CIFS via Host | CIFS mounts live on the Proxmox host and are bindmounted into LXCs via mp config; systemd automount behavior, stacked mount pitfall, Samba tooling (pdbedit, testparm) |
| Topic | Summary |
|---|---|
| Tailscale | Tailscale IPs, MagicDNS, Tailscale-managed certs, app-layer security boundary, vendor-lock-in considerations |
| Loopback + Tailscale Serve | The 127.0.0.1 + Serve binding pattern, alternatives evaluated, HTTPS/HTTP mismatch |
| nftables alongside Tailscale | Enforcing a boundary the service cannot: own table (never nftables.service — its stock config flush rulesets Tailscale's chains), table inet, counters as audit trail, RemainAfterExit |
| Tailscale ACL Design | Tier-based design, hosts aliases, access matrix, ACL changelog, pre-existing tunnel pitfall |
| Tailscale Debugging | How Tailscale's userspace packet filter works, tailscale ping bypasses ACL, duplicate node key problem, tcpdump as layer-separator, fix via daemon restart |
| Topic | Summary |
|---|---|
| Compose Patterns | Restart policies, network_mode: host, bridge service-name DNS vs localhost trap, logging, depends_on w/ healthcheck, env_file vs environment, PUID/PGID, named-volume mix |
| Daemon Recovery | Docker vs containerd process model, stale task state after ungraceful crash, docker rm -f + compose up recovery |
| Data Root Migration | Moving containerd + Docker data root off the root disk to Aux storage: daemon.json, config.toml, rsync -aH, boot-time dependency, fstrim |
| Bind-Mount Pitfalls | Silent directory creation for missing files, host-networking DNS loss, UID alignment in unprivileged LXCs |
| GPU Passthrough | NVIDIA Container Toolkit, pid: host, deploy.resources.reservations.devices, capability scoping |
| Topic | Summary |
|---|---|
| Prometheus Stack | Scrape jobs, node_exporter, textfile collector pattern, alert rules, node-up ≠ service-up blind spot (blackbox_exporter), up over a window to bound an incident, Alertmanager routing |
| Prometheus Configuration | scrape_configs, rule_files, alertmanagers static_configs, job-naming, retention/lifecycle flags |
| systemd Unit Alerting | The failed unit no alert category covered (20k failures, green dashboard); --collector.systemd + the stock exclude that drops .mount units; why the first alert after enabling it is proof, not a regression; test the negative case |
| PromQL & Alert Rules | for: debouncing, severity labels, fstype filters, aggregations, annotation templating, recording rules |
| Alertmanager Routing | Routes, group_by/wait/interval, repeat_interval, inhibit_rules, silences, Discord webhooks |
| Topic | Summary |
|---|---|
| SnapRAID + MergerFS | Storage stack architecture, sync/scrub discipline, noatime, excludes, multiple content files, category.create=mfs, hash-mismatch recovery, live disk expansion via xattr, empty-disk sync (XOR neutral), status output interpretation |
| CIFS via systemd Automount | Reboot-safe network mounts, x-systemd.automount options, boot-trigger oneshot, app-state vs uploads split |
| SQLite on CIFS Locking | Why database is locked over CIFS (byte-range locks not honored), CIFS-vs-local control test, nobrl vs the local-copy + atomic-swap workaround, and delete-the-source-last (durability ordering) for ingest-then-delete jobs |
| Samba Server Config | smb.conf structure, SMB3-only, mandatory signing, bind interfaces, share types (RW/RO/Ingest) |
| Samba Access Control & SMB Clients | SMB 3.1.1 vs mandatory signing (two gates), hosts allow default-deny (app-layer, not firewall), discovery (nmbd) vs connection (smbd), diagnosing with smbstatus/ss/testparm/%m.log/smbcontrol debug, app vs OS SMB client (Android SAF content:// vs real path), same-host bridge vs Tailscale |
| Topic | Summary |
|---|---|
| PostgreSQL Operations | pg_dumpall, backup scripts with install -m, crontab -u, pg_monitor role, dump validation |
| PostgreSQL CLI | psql meta-commands, pg_stat_activity, pg_terminate_backend, dropdb, restore verification |
| Zero-Trust PostgreSQL Access | Four-layer access: Tailscale ACL + binding + pg_hba hostssl + role privileges |
| Topic | Summary |
|---|---|
| Ollama Deployment | Modelfile syntax, quantization tags, context-window trade-offs, OLLAMA_HOST, ROCm vs CUDA |
| Local LLM Coding Fallback (aider + Ollama) | Self-hosted Claude Code fallback: native Anthropic API vs aider, the KV-cache q8_0 lever, "optimal = largest fully-resident context", size vs size_vram spill check, aider config |
| Multi-Agent Workflows | Spawning subagents in Claude Code, CI/CD parallel-stage analogy, when it helps vs. when it defeats active learning, unlock conditions (Terraform/k8s arc), and the meta-test: can you tell if the output is wrong? |
| Topic | Summary |
|---|---|
| Nextcloud Administration | occ CLI, files:scan, files_external:verify, APCu+Redis cache split, Apache TLS via Tailscale certs |
| Paperless-ngx | Pipeline (Gotenberg+Tika+Redis), CSRF origins, OCR languages, polling vs inotify on CIFS, USERMAP_UID/GID |
| Vaultwarden | Argon2id ADMIN_TOKEN, signups/invitations off, non-root container user, "no SQLite on CIFS" rule |
| Audiobookshelf Library Structure | Author/Series/Book/ folder layout, prefix stripping, series detection regex, dry-run/execute pattern, privacy in public repos |
| Topic | Summary |
|---|---|
| Least-Privilege Patterns | SMB perms, credentials files, .env hygiene, service isolation, sudoers.d, NOPASSWD helper hardening (the binary is the boundary), secret generation, defense in depth |
| Topic | Summary |
|---|---|
| Runbook Methodology | Root-cause process, failure-modes table, layer-by-layer health checks, verification log, doku-first workflow, fail-forward visibility |
| Git Branching Patterns | Cherry-pick workflow, CI dependency trap, feature branch strategy, -d vs -D, filter-repo replacement side effects, filter-branch --msg-filter, cherry-pick conflict resolution; merge-conflict resolution workflow; rename + modify merges (rename detection replays edits onto the new path — silently, verify it); merge vs rebase decided from file overlap (comm -12); ahead/behind in one shot (rev-list --left-right --count A...B, three-dot vs two-dot); --ff-only for a mirror-only node (refusal is the feature); fetch --prune after a remote branch is deleted; sanitizing an already-public history — a forward sanitization commit re-leaks in its own diff, --replace-text misses messages (--replace-message) and filenames (--path-rename), and a rewrite cannot touch GitHub PR metadata (titles/branch names/diff tabs survive force-push — Support or delete only; edit titles via REST); refs/pull/*/head keeps force-pushed commits reachable (fetch them all with +refs/pull/*/head:refs/remotes/origin/pr/* — a force-push frees nothing while a PR ref holds the commit; check for-each-ref --contains and merge-base --is-ancestor <sha> main to see whether deleting PRs is even sufficient); renaming is not removing — verify a sanitizing rewrite by grepping file content across all refs, not commit subjects; keep a gitignored legend + a validate guard against re-introduction; ship-then-soak (merge a verified fix now, refactor later on its own tests-first branch) |
| Conventional Commits | Commit message format with required scope, per-node and thematic scope conventions |
| Repo Validation | Self-validating documentation repos, structural checks, sanitization rules, CI integration; a local check that shells out to an external linter can be silently inert — guard on command -v, print SKIP loudly, pin to CI's version, scope to the diff, test the FAIL path deliberately |
| CI Quality Gates | Why a green pipeline can be an untested one (tests self-skip on missing deps), fixing it, the setup-python vs apt interpreter trap |
| Documentation-vs-Reality Audits | Treat every documented claim as a hypothesis with a live-state verification command; reconciliation direction is a decision (fix doc vs fix reality vs record intent); drift classes (pins, boot order, "managed by X" claims, ACL, naming); "green and correct-looking" as the failure mode; public-repo hygiene (facts not exploit detail) |
| Backup Strategy | 3-2-1 rule, threat coverage matrix, restic with append-only credentials, retention policies, restore verification |
| Claude Code Hooks | Hook events, stdin JSON, additionalContext vs systemMessage, if conditional field, SessionStart context injection, dual-Stop pattern, defense-in-depth with branch protection, hook fatigue |
| Git Commit Hooks | What git hooks are, why .git/hooks/ is not committed, symlink pattern for versioned hooks, commit-msg validation script, local hooks vs CI enforcement, pre-commit/husky overview |
| Dotfiles Management | Template + render pattern, --dry-run flag, pipx ensurepath PATH fix, validate.sh, bootstrap/install split |
Updated at the end of each working session.