Skip to content

Enterprise-grade infrastructure for a user base of one. Powered by 56 cores and anxiety

Notifications You must be signed in to change notification settings

ngodat0103/dev-oops

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

260 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

"Move fast and break things" β€” Mark Zuckerberg
"I moved fast. Things are broken." β€” Me, at 3 AM
"Have you tried turning it off and on again?" β€” My mom, who has heard enough

Welcome to dev-oops β€” my personal laboratory where I cosplay as a DevOps engineer, ARP-spoof my children's tablets, and treat terraform destroy as a form of meditation.

This is what happens when you have more CPU cores than friends.


What is This?

This repository contains enterprise-grade infrastructure for a hobbyist-grade homelab. It's over-engineered, over-documented, and occasionally over-heated.

I treat my homelab like a Fortune 500 company's infrastructure, except:

  • My SLA is "probably up"
  • My incident response is "wake up and panic"
  • My disaster recovery plan is "cry, then restore from backup MinIO"
  • My change management process is git push --force and pray
  • My parental controls involve literal ARP poisoning (see: The Sentry Project)

The Victim (Hardware Specs)

Component Spec Notes
CPU 56 x Intel Xeon E5-2680 v4 @ 2.40GHz Two sockets of raw, slightly-aged power
RAM 62GB Enough to run Kubernetes. Barely.
Boot Mode Legacy BIOS "I don't do UEFI here"
Hypervisor Proxmox VE 9.0.3 The backbone of my chaos
Kernel Linux 6.14.8-2-pve Latest and greatest (until tomorrow)
Electricity Bill Yes I don't talk about this

Storage Situation

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Device      β”‚ Size    β”‚ Purpose                                      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ sda         β”‚ 465.8G  β”‚ Spinning rust from 2014 (the "OG")           β”‚
β”‚ sdb         β”‚ 931.5G  β”‚ More spinning rust (the "backup OG")         β”‚
β”‚ nvme0n1     β”‚ 1.8T    β”‚ The fast boi (VMs live here, briefly)        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The Production Network (ansible/core/inventory.ini)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Host                β”‚ IP             β”‚ What It Does                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ pve-master          β”‚ 192.168.1.120  β”‚ Proxmox Hypervisor (the boss)        β”‚
β”‚ ubuntu-server       β”‚ 192.168.1.121  β”‚ Docker + Traefik (the workhorse)     β”‚
β”‚ teleport            β”‚ 192.168.1.122  β”‚ Zero-trust access (fancy SSH)        β”‚
β”‚ vpn-server          β”‚ 192.168.1.123  β”‚ OpenVPN (for remote chaos)           β”‚
β”‚ hephaestus          β”‚ 192.168.1.124  β”‚ CI/CD runners (Greek god vibes)      β”‚
β”‚ sonarqube           β”‚ 192.168.1.125  β”‚ Code quality (yes, I lint my code)   β”‚
β”‚ core-dns            β”‚ 192.168.1.126  β”‚ Internal DNS (Alpine, 128MB RAM)     β”‚
β”‚ crowdsec            β”‚ 192.168.1.127  β”‚ WAF / Security engine (the bouncer)  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                     β”‚  PRIVATE NET   β”‚  192.168.99.0/24                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ lxc-postgresql-16   β”‚ 192.168.99.2   β”‚ PostgreSQL in LXC (the elephant)     β”‚
β”‚ lxc-kafka           β”‚ 192.168.99.2   β”‚ Kafka (enterprise cosplay)           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Architecture (a.k.a. "The Overkill")

                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β”‚                   THE INTERNET                    β”‚
                    β”‚              (where the danger lives)             β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                            β”‚
                                            β–Ό
                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β”‚                   CLOUDFLARE                      β”‚
                    β”‚    DNS, Firewall, "Please don't DDoS me" layer   β”‚
                    β”‚         Domain: datrollout.dev (nice)             β”‚
                    β”‚              (Managed by Terraform)               β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                            β”‚
                                            β–Ό
                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β”‚               UPTIMEROBOT                         β”‚
                    β”‚     "Is it down? Let me text you at 3 AM"        β”‚
                    β”‚              (Also Terraform'd)                   β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                            β”‚
                                            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                              PROXMOX VE (pve-master)                        β”‚
β”‚                    (The hypervisor that runs everything)                     β”‚
β”‚                              192.168.1.120                                   β”‚
β”‚                                                                              β”‚
β”‚ ═══════════════════════════ 🏭 PRODUCTION ═══════════════════════════════   β”‚
β”‚                                                                              β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚              🐳 UBUNTU-SERVER VM (192.168.1.121)                     β”‚    β”‚
β”‚  β”‚                    "The Docker Workhorse"                           β”‚    β”‚
β”‚  β”‚                  (Managed by ansible/core)                          β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   ══════════════ TRAEFIK v3.6.7 (The Gateway) ══════════════       β”‚    β”‚
β”‚  β”‚   β”‚ :80/:443 β†’ CrowdSec middleware β†’ Services                β”‚      β”‚    β”‚
β”‚  β”‚   β”‚ Let's Encrypt SSL via Cloudflare DNS challenge           β”‚      β”‚    β”‚
β”‚  β”‚   ═══════════════════════════════════════════════════════════       β”‚    β”‚
β”‚  β”‚                              β”‚                                       β”‚    β”‚
β”‚  β”‚                              β–Ό                                       β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”              β”‚    β”‚
β”‚  β”‚   β”‚  GitLab  β”‚ β”‚Vaultwardenβ”‚ β”‚ Jellyfin β”‚ β”‚Nextcloud β”‚              β”‚    β”‚
β”‚  β”‚   β”‚  CI/CD   β”‚ β”‚ Passwords β”‚ β”‚ "Linux   β”‚ β”‚  Files   β”‚              β”‚    β”‚
β”‚  β”‚   β”‚  + Repos β”‚ β”‚           β”‚ β”‚  ISOs"   β”‚ β”‚          β”‚              β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜              β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                           β”‚    β”‚
β”‚  β”‚   β”‚qBittorrentβ”‚ β”‚Agent DVR β”‚ β”‚ useless- β”‚                           β”‚    β”‚
β”‚  β”‚   β”‚ "Linux   β”‚ β”‚ Cameras  β”‚ β”‚  app.yamlβ”‚                           β”‚    β”‚
β”‚  β”‚   β”‚  ISOs"   β”‚ β”‚  πŸŽ₯      β”‚ β”‚    ???   β”‚                           β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                           β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”      β”‚    β”‚
β”‚  β”‚   β”‚              πŸ“Š OBSERVABILITY STACK                      β”‚      β”‚    β”‚
β”‚  β”‚   β”‚  Prometheus β”‚ Grafana β”‚ Loki β”‚ Alloy β”‚ InfluxDB β”‚ cAdvisor    β”‚    β”‚
β”‚  β”‚   β”‚           "Watching containers die in 4K"                β”‚      β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   πŸ’Ύ Backups: restic β†’ rclone β†’ cloud (I learned the hard way)    β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚              πŸ”¬ SONARQUBE VM (192.168.1.125)                         β”‚    β”‚
β”‚  β”‚                    Code Quality Analysis                            β”‚    β”‚
β”‚  β”‚           "Yes, I run static analysis on my homelab code"           β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚                     πŸ“¦ LXC CONTAINERS                               β”‚    β”‚
β”‚  β”‚               (Because VMs are too mainstream)                      β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”      β”‚    β”‚
β”‚  β”‚   β”‚ PostgreSQL 16   β”‚ β”‚ Kafka           β”‚ β”‚ CoreDNS         β”‚      β”‚    β”‚
β”‚  β”‚   β”‚ 192.168.99.2    β”‚ β”‚ 192.168.99.x    β”‚ β”‚ 192.168.1.126   β”‚      β”‚    β”‚
β”‚  β”‚   β”‚ 4GB RAM         β”‚ β”‚ 8GB RAM         β”‚ β”‚ 128MB RAM 😎    β”‚      β”‚    β”‚
β”‚  β”‚   β”‚ (Private Net)   β”‚ β”‚ (Private Net)   β”‚ β”‚ Alpine Linux    β”‚      β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                               β”‚    β”‚
β”‚  β”‚   β”‚ CrowdSec WAF    β”‚                                               β”‚    β”‚
β”‚  β”‚   β”‚ 192.168.1.127   β”‚ ← "You shall not pass"                        β”‚    β”‚
β”‚  β”‚   β”‚ LAPI + AppSec   β”‚                                               β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                                               β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚            πŸ›‘οΈ SECURITY LAYER (The Actually Serious Part)            β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚    β”‚
β”‚  β”‚   β”‚ TRAEFIK v3.6.7 (192.168.30.50 / :80, :443)                   β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Reverse proxy for all services                           β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Let's Encrypt SSL via Cloudflare DNS challenge          β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Prometheus metrics + access logging                      β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ CrowdSec bouncer plugin middleware                       β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Cloudflare trusted IPs (CF-Connecting-IP header)         β”‚  β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚    β”‚
β”‚  β”‚                              β”‚                                       β”‚    β”‚
β”‚  β”‚                              β–Ό                                       β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚    β”‚
β”‚  β”‚   β”‚ CROWDSEC (192.168.1.127) - "The Bouncer"                     β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ LAPI on :8080                                            β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ AppSec engine on :7422                                   β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Detects: XSS, Path Traversal, Brute Force               β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ Mode: LIVE (blocks bad actors in real-time)              β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   β€’ "You shall not pass" energy                              β”‚  β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚    β”‚
β”‚  β”‚   β”‚ COREDNS (192.168.1.126) - Alpine, 128MB RAM                  β”‚  β”‚    β”‚
β”‚  β”‚   β”‚   Internal DNS resolution                                    β”‚  β”‚    β”‚
β”‚  β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚               πŸ” TELEPORT (192.168.1.122)                           β”‚    β”‚
β”‚  β”‚                   Zero-Trust Access                                 β”‚    β”‚
β”‚  β”‚           "SSH but make it enterprise"                              β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚               πŸ”’ VPN-SERVER (192.168.1.123)                         β”‚    β”‚
β”‚  β”‚                      OpenVPN                                        β”‚    β”‚
β”‚  β”‚           "For when you're not at home"                             β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚              πŸ›οΈ HEPHAESTUS (192.168.1.124)                          β”‚    β”‚
β”‚  β”‚              "Named after the Greek god of craftsmanship"           β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚          GitLab Runner β”‚ GitHub Runner β”‚ Maven β”‚ Go β”‚ K8s Tools     β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚                     πŸ•΅οΈ THE SENTRY (Planned)                         β”‚    β”‚
β”‚  β”‚              "Parental Controls via ARP Poisoning"                  β”‚    β”‚
β”‚  β”‚      Because asking nicely doesn't work on tablets at 1 AM          β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                                                                             β”‚
β”‚ ═══════════════════════════ πŸ§ͺ LAB / DEV ════════════════════════════════   β”‚
β”‚                                                                              β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚                    K3s KUBERNETES CLUSTER                           β”‚    β”‚
β”‚  β”‚        🚧 LAB ENVIRONMENT ONLY - NOT PRODUCTION 🚧                  β”‚    β”‚
β”‚  β”‚            (Migration aborted, now it's a playground)               β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚  "I tried to migrate to K8s. K8s won. Now it's where I test things β”‚    β”‚
β”‚  β”‚   before they go to the real Docker setup. Or break things on      β”‚    β”‚
β”‚  β”‚   purpose with Chaos Mesh. Mostly the second one."                 β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   ArgoCD β”‚ Traefik β”‚ Longhorn β”‚ Sealed Secrets β”‚ Chaos Mesh        β”‚    β”‚
β”‚  β”‚   PostgreSQL β”‚ Redis β”‚ MinIO β”‚ Vaultwarden β”‚ qBittorrent           β”‚    β”‚
β”‚  β”‚                                                                      β”‚    β”‚
β”‚  β”‚   Status: ✨ Learning ✨ Testing ✨ Breaking ✨                       β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Repository Structure

dev-oops/
β”œβ”€β”€ ansible/                    # Configuration Management
β”‚   β”œβ”€β”€ core/                   # 🏭 THE PRODUCTION STUFF
β”‚   β”‚   β”œβ”€β”€ inventory.ini      # The network map (192.168.1.x gang)
β”‚   β”‚   β”œβ”€β”€ hephaestus/        # CI/CD runners (Greek god = extra cool points)
β”‚   β”‚   β”œβ”€β”€ lxc/               # PostgreSQL 16, Kafka in containers
β”‚   β”‚   β”‚   β”œβ”€β”€ postgresql/    # The elephant (192.168.99.2)
β”‚   β”‚   β”‚   └── kafka/         # Message queue for enterprise cosplay
β”‚   β”‚   β”œβ”€β”€ teleport/          # Zero-trust access (fancy SSH for fancy people)
β”‚   β”‚   β”œβ”€β”€ ubuntu-server/     # THE DOCKER WORKHORSE
β”‚   β”‚   β”‚   β”œβ”€β”€ apps/          # GitLab, Jellyfin, Nextcloud, qBittorrent...
β”‚   β”‚   β”‚   β”‚   └── useless-app.yaml   # Yes, this exists. No, I won't explain.
β”‚   β”‚   β”‚   β”œβ”€β”€ basic/         # apt, samba, storage, swap, user management
β”‚   β”‚   β”‚   β”œβ”€β”€ observation-and-monitoring/  # Grafana, Prometheus, Loki, Alloy
β”‚   β”‚   β”‚   └── system-cron/   # Backups via restic (I learned my lesson)
β”‚   β”‚   └── vpn-server/        # OpenVPN because WireGuard is too easy
β”‚   β”œβ”€β”€ kubernetes/            # Kubespray configs (deprecated)
β”‚   └── sonarqube/             # Code quality (yes, I lint my YAML. Judge me.)
β”‚
β”œβ”€β”€ kubernetes/                 # πŸ§ͺ LAB ENVIRONMENT ONLY
β”‚   β”œβ”€β”€ argocd/                # GitOps playground
β”‚   β”‚   β”œβ”€β”€ argocd-app/        # Application definitions
β”‚   β”‚   β”‚   β”œβ”€β”€ daemon/        # Kube-Prometheus-Stack, MetalLB
β”‚   β”‚   β”‚   β”œβ”€β”€ stateful/      # PostgreSQL, Redis, MinIO, Longhorn, CHAOS MESH
β”‚   β”‚   β”‚   └── stateless/     # Traefik, Vaultwarden, Sealed Secrets
β”‚   β”‚   └── argocd-crd/        # ArgoCD itself (it's ArgoCD all the way down)
β”‚   └── traefik/               # Ingress controller configs
β”‚   # ⚠️  This is NOT production! Just a place to test K8s concepts
β”‚   #     and break things with Chaos Mesh before giving up and
β”‚   #     going back to Docker like a sensible person.
β”‚
β”œβ”€β”€ tf/                        # Terraform (Infrastructure as Code)
β”‚   β”œβ”€β”€ cloudflare/            # DNS & Storage for datrollout.dev
β”‚   β”œβ”€β”€ proxmox/               # VM provisioning
β”‚   β”œβ”€β”€ openstack/             # Because why not add another cloud?
β”‚   β”œβ”€β”€ uptimerobot/           # "Is it down?" β†’ "Yes, check Discord"
β”‚   └── terraform-module/      # Reusable modules (I're professionals here)
β”‚
β”œβ”€β”€ disaster-recovery/         # For when things go wrong (often)
β”‚   └── vaultwarden/           # Python backup scripts to MinIO
β”‚       └── Backup/            # Because losing passwords is NOT an option
β”‚
└── plans/                     # Future chaos documentation
    └── use-side-arm-arp-interception.md   # *chef's kiss* (see below)

πŸ›‘οΈ Security Stack (The Actually Professional Part)

Traffic flows through multiple security layers before reaching any service:

Internet 🌐
    β”‚
    β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                         CLOUDFLARE                                      β”‚
β”‚   β€’ DDoS protection ("Please don't hurt me")                           β”‚
β”‚   β€’ DNS management (datrollout.dev)                                    β”‚
β”‚   β€’ Firewall rules (Terraform managed)                                 β”‚
β”‚   β€’ Proxy mode enabled (hides real IP)                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
    β”‚ CF-Connecting-IP header
    β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     TRAEFIK v3.6.7                                      β”‚
β”‚   β€’ Reverse proxy on 192.168.1.121:80/443                              β”‚
β”‚   β€’ Let's Encrypt SSL via Cloudflare DNS challenge                     β”‚
β”‚   β€’ Routes: gitlab, vaultwarden, nextcloud, jellyfin, teleport...      β”‚
β”‚   β€’ Every request passes through CrowdSec middleware                   β”‚
β”‚   β€’ Prometheus metrics + structured access logs                        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
    β”‚ crowdsec@file middleware
    β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     CROWDSEC (192.168.1.127)                            β”‚
β”‚   LXC Container - "The Bouncer"                                        β”‚
β”‚                                                                        β”‚
β”‚   LAPI (:8080)           AppSec Engine (:7422)                         β”‚
β”‚   β”œβ”€ Decision API        β”œβ”€ Real-time request analysis                 β”‚
β”‚   β”œβ”€ Ban/Captcha         β”œβ”€ HTTP path traversal detection              β”‚
β”‚   └─ IP reputation       β”œβ”€ XSS probing detection                      β”‚
β”‚                          └─ Generic brute force detection              β”‚
β”‚                                                                        β”‚
β”‚   Mode: LIVE (blocks in real-time, not just logging)                   β”‚
β”‚   Failure behavior: BLOCK (if CrowdSec is down, deny all)              β”‚
β”‚   "I'd rather break the site than let hackers in"                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
    β”‚ βœ… Allowed
    β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                      The actual userful services                       β”‚
β”‚   GitLab β”‚ Vaultwarden β”‚ Nextcloud β”‚ Jellyfin β”‚ SonarQube β”‚ etc.       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Security Scenarios Protected Against

Attack Type Detection Response
Path Traversal crowdsecurity/http-path-traversal-probing 403 Forbidden
XSS Probing crowdsecurity/http-xss-probing 403 Forbidden
Brute Force crowdsecurity/http-generic-bf 403 + Temp Ban
DDoS Cloudflare Mitigation
Bot Traffic CrowdSec community blocklists 403 Forbidden

The "Trust No One" Philosophy

# If CrowdSec AppSec is unreachable:
crowdsecAppsecUnreachableBlock: true  # BLOCK EVERYTHING

# If CrowdSec fails:
crowdsecAppsecFailureBlock: true      # BLOCK EVERYTHING

# Translation: "I'd rather explain downtime than a breach"

πŸ•΅οΈ Parental Controls via Cyberwarfare

ADR Status: Accepted
Codename: Homelab Sentry
mAF (mom Acceptance Factor): Pending review

When Screen Time isn't enough and you have a Proxmox server with existential anxiety, you build a Man-in-the-Middle attack framework for your home network.

The Plan

Normal Network:
iPad πŸ§’ ──────────────────────────► Router πŸ“‘ ──► Internet

After I'm Done:
iPad πŸ§’ ──► Sentry VM πŸ•΅οΈ ──► Router πŸ“‘ ──► Internet
                β”‚
                └── "Is it 1 AM? DROP PACKET."
                └── "Is it homework time? Block YouTube DNS."
                └── "Alert Dad via Telegram Bot."

Features (Planned)

  • ARP Poisoning: Whispers to the iPad: "I am the router now"
  • Time-based blocking: No internet after 1 AM (the hard way)
  • DNS Sinkholing: YouTube resolves to a "Go to bed" page
  • Telegram Bot: /allow 1h when they've been good
  • Graceful Shutdown: Floods correct ARP packets on exit so WiFi doesn't die when Proxmox reboots

Risks

  • IP conflicts if I mess up broadcasts
  • Explaining to my mom why I'm "hacking the children"
  • Slight latency increase (4K streaming might suffer)
  • The kids might learn networking to fight back

The Stack of Chaos

Infrastructure Layer

Tool Purpose Status
Proxmox VE Hypervisor 🟒 Running (pve-master)
Terraform Infrastructure as Code 🟒 Running
Cloudflare DNS & Security 🟒 Running
OpenStack ??? 🟑 It's in the tf folder, I'll figure it out

Configuration Management

Tool Purpose Chaos Level
Ansible Server configuration (🏭 PRODUCTION) πŸ”₯πŸ”₯ Medium (YAML indentation trauma)
ansible/core The actual production playbooks πŸ”₯πŸ”₯ Medium (but it works!)
Kubespray K8s deployment πŸ”₯πŸ”₯πŸ”₯ Deprecated (I gave up)

Container Orchestration

Tool Purpose Environment Chaos Level
Docker Container runtime 🏭 PRODUCTION πŸ”₯πŸ”₯ Medium (I know this one)
Traefik Reverse proxy & SSL 🏭 PRODUCTION πŸ”₯πŸ”₯ Medium (middleware inception)
K3s Lightweight Kubernetes πŸ§ͺ LAB ONLY πŸ”₯πŸ”₯πŸ”₯πŸ”₯ Extreme (it's still Kubernetes)
ArgoCD GitOps deployment πŸ§ͺ LAB ONLY πŸ”₯πŸ”₯ Medium (fun to learn)
Longhorn Distributed storage πŸ§ͺ LAB ONLY πŸ”₯πŸ”₯πŸ”₯ High (distributed = distributed problems)
Chaos Mesh Breaking things on purpose πŸ§ͺ LAB ONLY πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯ MAXIMUM (by design)

Why K8s is lab-only: I tried to migrate from Docker to K8s. I really did. But you know what? Docker Compose + Ansible just worksβ„’. The K8s cluster now serves as a playground for learning, testing configs, and occasionally running Chaos Mesh to watch pods die for educational purposes.

Observability (Watching Things Break)

Tool Purpose Chaos Level
Prometheus Metrics collection πŸ”₯πŸ”₯ Medium
Grafana Pretty dashboards πŸ”₯ Low (the fun part)
Loki Log aggregation πŸ”₯πŸ”₯ Medium
Alloy Telemetry collector πŸ”₯πŸ”₯ Medium (new hotness)
InfluxDB Time-series DB πŸ”₯πŸ”₯ Medium
UptimeRobot External monitoring πŸ”₯ Low (it texts me at 3 AM)

Applications (The Actual Useful Stuff)

App Purpose Why
GitLab Git hosting & CI/CD Self-hosted GitHub at home
Vaultwarden Password manager Because I can't remember anything
Nextcloud File sync Google Drive but with more RAM usage
Jellyfin Media server "Linux ISOs" streaming
qBittorrent Torrent client For "Linux ISOs"
Agent DVR Security cameras Watching the driveway, professionally
PostgreSQL Database The elephant in the room
Kafka Message queue Because why not?
Redis Cache Speed
MinIO Object storage S3 at home (for backups, mostly)
Teleport Zero-trust access SSH but enterprise-grade
SonarQube Code quality Yes, I lint my homelab code
useless-app Unknown The YAML exists. That's all I know.

Security Layer

Tool Purpose Vibe
Traefik v3.6.7 Reverse proxy + SSL The front door
CrowdSec WAF + Threat detection The bouncer
CoreDNS Internal DNS 128MB of pure resolution
Cloudflare DDoS + DNS + CDN The bodyguard
Let's Encrypt SSL certs Free HTTPS via DNS challenge

CI/CD: The Hephaestus System

Named after the Greek god of fire, metalworking, and craftsmanship, our CI/CD runner infrastructure auto-provisions:

  • πŸ”¨ GitLab Runner β€” for the self-hosted git
  • πŸ™ GitHub Runner β€” for the cloud repos
  • β˜• Maven β€” Java builds
  • 🐹 Golang β€” Go builds
  • 🎑 K8s Tools β€” kubectl, helm, the works
  • 🐳 Docker β€” containers all the way down

All managed by Ansible because manually installing runners is for mortals.


Lessons Learned (The Hard Way)

Things I've Broken (So Far)

  • Deleted production database (it was just my passwords, no big deal)
  • Ran terraform destroy on the wrong workspace
  • Forgot to backup before "quick fix"
  • Locked myself out of my own server
  • Filled up the boot disk with logs
  • Created an infinite ArgoCD sync loop
  • Misconfigured firewall, couldn't SSH in
  • Tried to migrate from Docker to K8s
  • Gave up on K8s migration (Docker + Ansible supremacy)
  • Kept K8s cluster anyway as "learning environment" (cope)
  • Installed Chaos Mesh and immediately regretted it
  • Successfully ARP-spoofed my kids (coming soon)
  • Lost data permanently (knock on wood πŸͺ΅)

Lessons Actually Learned

  1. Always backup Vaultwarden β€” hence the Python scripts to MinIO
  2. Docker + Ansible is fine β€” K8s is cool but production uptime is cooler
  3. K8s is great... for learning β€” keep it as a lab, not production
  4. Chaos Mesh is both amazing and terrifying β€” USE WITH CAUTION (in lab only)
  5. Name things after Greek gods β€” makes debugging feel epic
  6. Document your ARP spoofing plans β€” your future self will thank you
  7. LXC for databases, VMs for apps β€” this actually works really well

File Highlight Reel

File What It Does Concern Level
useless-app.yaml Deploys... something? 🀷
use-side-arm-arp-interception.md Tactical child network control πŸ‘€
delete-crd.sh Exactly what it sounds like πŸ’€
chaos-mesh/argo-app.yaml Automated breaking things πŸ”₯
backup.sh (in Vaultwarden) The most important file πŸ™

Getting Started (For the Brave)

# Step 1: Clone this chaos
git clone https://github.com/ngodat0103/dev-oops.git
cd dev-oops

# Step 2: Terraform your cloud resources
cd tf/cloudflare && terraform init && terraform apply

# Step 3: Ansible your PRODUCTION servers (the real stuff)
cd ../../ansible/core
ansible-playbook -i inventory.ini ubuntu-server/basic/apt.yaml       # Base setup
ansible-playbook -i inventory.ini ubuntu-server/apps/gitlab.yaml     # GitLab
ansible-playbook -i inventory.ini ubuntu-server/apps/traefik.yaml    # Reverse proxy
ansible-playbook -i inventory.ini lxc/postgresql/0-manage-postgresql.yaml  # DB

# Step 4: (Optional) Play with K8s lab environment
cd ../../kubernetes/argocd
# This is just for learning, not production. Go wild. Break things.
kubectl apply -f argocd-crd/

# Step 5: Watch it all in Grafana
# Step 6: Get paged at 3 AM by UptimeRobot
# Step 7: Fix it half-asleep
# Step 8: Write a postmortem you'll never read
# Step 9: Repeat

Contributing

This is my personal homelab, so contributions are... unexpected? But if you:

  1. Found a security issue β†’ Please tell me (nicely)
  2. Have a suggestion β†’ Open an issue
  3. Want to judge my YAML β†’ Fair enough
  4. Know why useless-app.yaml exists β†’ Please enlighten me
  5. Have better parental control ideas than ARP poisoning β†’ I'm listening

The Real Architecture

                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β”‚           My Mental State           β”‚
                    β”‚                                     β”‚
                    β”‚    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”‚
                    β”‚    β”‚ Anxiety │────►│ Coffee  β”‚     β”‚
                    β”‚    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜     β”‚
                    β”‚         β–²               β”‚          β”‚
                    β”‚         β”‚               β–Ό          β”‚
                    β”‚    β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”‚
                    β”‚    β”‚ 3 AM    │◄───│ Alerts  β”‚     β”‚
                    β”‚    β”‚ Panic   β”‚    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β”‚
                    β”‚    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                     β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

License

This project is licensed under the "Works On My Machine" license.

You're free to:

  • Copy this and break your own stuff
  • Learn from my mistakes
  • Laugh at my configuration choices
  • Question my parenting techniques
  • Wonder why anyone needs Chaos Mesh at home

Powered by caffeine, spite, and 56 Xeon cores that could heat a small apartment.

About

Enterprise-grade infrastructure for a user base of one. Powered by 56 cores and anxiety

Topics

Resources

Stars

Watchers

Forks

Packages

 
 
 

Contributors 2

  •  
  •