feat(caddy): fail2ban jail to ban repeated RPC basic-auth failures

AGENTS.md

@@ -63,7 +63,8 @@ secrets; generated on host) and #2 (localhost-only by default) hold for both con
   metrics/admin, operator-provisioned eth keystore, required chain knobs (no baked protocol
   facts — sourced from ADRs). **Internal: the anvil devnet** (`playbooks/anvil.yml` →
   baseline + anvil + Caddy basic-auth; anvil stays loopback while Caddy fronts it on public
-  https/443 with auto-TLS — `caddy_public: true`, default; flip to loopback-only for a tunnel)
+  https/443 with auto-TLS — `caddy_public: true`, default; flip to loopback-only for a tunnel;
+  repeated basic-auth failures are fail2ban-banned via the public-only `caddy-rpc` jail)
   — team tooling, not the product. Shared
   DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node stake/registration,
   ADR 019 Phase 2, is an operator step, not automated.)

ansible/README.md

@@ -105,8 +105,9 @@ make add-dev USER_NAME=alice    # mint + reveal a basic-auth dev user
 On the first anvil deploy the shared **mnemonic** and the `dev` basic-auth password are
 printed **once** — save them to the team vault. anvil binds `127.0.0.1` only; **caddy fronts
 it on public https/443** with auto-TLS + per-dev basic auth (`caddy_public: true`, default),
-so the DNS A record must already point at the host. To keep it loopback-only instead (e.g.
-behind a tunnel) set `caddy_public: false` — see the appendix.
+so the DNS A record must already point at the host. Repeated basic-auth failures are
+**fail2ban-banned** (the `caddy-rpc` jail, public listener only — see `roles/caddy`). To keep
+it loopback-only instead (e.g. behind a tunnel) set `caddy_public: false` — see the appendix.
 
 ---
 
@@ -139,6 +140,7 @@ Defaults live in each role (`roles/*/defaults/main.yml`); override in `group_var
 | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node (host_vars); sourced from an ADR/deployment. |
 | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. |
 | `anvil_chain_id` … `rpc_hostname` | see `roles/anvil`,`roles/caddy` | internal devnet knobs. |
+| `caddy_fail2ban` (+ `_maxretry`/`_findtime`/`_bantime`) | `true` (5 / 10m / 1h) | RPC basic-auth brute-force jail; public listener only. |
 
 ---
 

ansible/roles/caddy/README.md

@@ -0,0 +1,81 @@
+# roles/caddy
+
+Puts a **per-dev HTTP basic-auth reverse proxy** in front of the loopback anvil
+RPC. The internal anvil devnet (`playbooks/anvil.yml`) runs `baseline → anvil →
+caddy`; Caddy is the only thing that faces the internet, and only after auth over
+TLS. Internal tooling — **not** part of the public `decdn.node` collection.
+
+## What this role does
+
+1. **Installs Caddy** and ensures `/etc/caddy` (group `caddy`, `0750`).
+2. **Mints basic-auth users** on the host into a TSV registry
+   (`caddy_users_tsv`), bcrypt-hashed via `caddy hash-password`, and renders the
+   `basic_auth` import block (`caddy_basicauth_file`). Credentials are generated
+   on the host — never in the repo. `caddy_initial_user` (`dev`) always exists;
+   `make add-dev USER_NAME=…` mints more.
+3. **Renders the Caddyfile** (validated with `caddy validate`) in one of two
+   shapes:
+   - `caddy_public: true` (default) — public **HTTPS on `rpc_hostname:443`** with
+     auto-TLS (Let's Encrypt). Requires inbound tcp/80+443 (open via
+     `baseline_extra_inbound`).
+   - `caddy_public: false` — loopback plain-HTTP on `caddy_bind_port` (dev/CI, or
+     TLS terminated upstream by a tunnel).
+4. **JSON access log** to `caddy_access_log` (self-rolling, 10 MiB × 5), recording
+   who hit the RPC. (Access logs move from the journal to this file — query it for
+   requests; Caddy's runtime/process logs still go to `journalctl -u caddy`.)
+5. **fail2ban `caddy-rpc` jail** — *public listener only*. Watches the access log
+   and bans, via nftables, any IP that repeatedly fails basic auth (HTTP 401).
+   See below.
+
+## fail2ban: RPC basic-auth brute-force protection
+
+A public RPC behind basic auth invites credential-stuffing. When
+`caddy_public: true` and `caddy_fail2ban: true` (both default), the role installs:
+
+- `/etc/fail2ban/filter.d/caddy-rpc.conf` — matches a `401` for the access logger
+  in the JSON log and captures the connecting `remote_ip`.
+- `/etc/fail2ban/jail.d/caddy-rpc.local` — the `caddy-rpc` jail
+  (`banaction = nftables-multiport`, bans on 80+443).
+
+fail2ban itself comes from `baseline` (always run before `caddy` in `anvil.yml`).
+On the loopback/CI listener the jail is **skipped** — there's no public attack
+surface, and Molecule runs `caddy` without `baseline` (fail2ban absent). Lenient
+defaults: **5 failures within 10m → 1h ban**.
+
+Because Molecule never exercises this path, the role self-checks at deploy: it
+**asserts fail2ban is installed** before writing the jail (actionable error if you
+ran `caddy` standalone without `baseline`), and **verifies the jail loaded**
+(`fail2ban-client status caddy-rpc`) after the restart — so a malformed filter
+fails loud instead of silently never banning.
+
+Inspect / unban on the host:
+
+```bash
+sudo fail2ban-client status caddy-rpc
+sudo fail2ban-client set caddy-rpc unbanip <IP>
+```
+
+> **Note:** baseline's nftables ruleset uses `flush ruleset`; a baseline nftables
+> *reload* (only on template change) clears fail2ban's `f2b-table` until the next
+> fail2ban restart. Pre-existing for the `sshd` jail too.
+
+## Key variables
+
+| Var | Default | Notes |
+|-----|---------|-------|
+| `caddy_public` | `true` | `true` = public HTTPS on `rpc_hostname`; `false` = loopback HTTP on `caddy_bind_port`. |
+| `caddy_acme_email` | `""` | Let's Encrypt account contact; empty = anonymous ACME. |
+| `caddy_bind_port` | `8080` | Loopback listener port (only when `caddy_public: false`). |
+| `rpc_hostname` | `rpc-dev.decdn.org` | Public HTTPS host; printed in `ETH_RPC_URL`. |
+| `anvil_host` / `anvil_port` | `127.0.0.1` / `8545` | Upstream RPC to proxy to. |
+| `caddy_users_tsv` / `caddy_basicauth_file` | `/etc/caddy/…` | On-host credential registry + generated import. |
+| `caddy_initial_user` | `dev` | Always-present basic-auth user. |
+| `caddy_access_log` | `/var/log/caddy/rpc-access.log` | JSON access log; the log the fail2ban jail watches. |
+| `caddy_fail2ban` | `true` | Enable the `caddy-rpc` jail (public listener only). |
+| `caddy_fail2ban_maxretry` | `5` | Failed auths before a ban. |
+| `caddy_fail2ban_findtime` | `10m` | Window the failures must fall within. |
+| `caddy_fail2ban_bantime` | `1h` | Ban duration. |
+
+## Platforms
+
+Debian (bookworm), Ubuntu (jammy, noble).

ansible/roles/caddy/defaults/main.yml

@@ -23,3 +23,16 @@ caddy_users_tsv: /etc/caddy/rpc-dev.users.tsv       # username<TAB>bcrypt regist
 caddy_basicauth_file: /etc/caddy/rpc-dev.basicauth  # generated Caddyfile import
 caddy_initial_user: dev                              # always ensured to exist
 # caddy_extra_user: alice        # set by add-dev-user.yml to mint one more user
+
+# JSON access log (self-rolling) recording who hit the RPC; also the log the
+# fail2ban caddy-rpc jail watches for repeated basic-auth failures.
+caddy_access_log: /var/log/caddy/rpc-access.log
+
+# fail2ban jail banning IPs that repeatedly fail RPC basic auth. Only wired up on
+# the public listener (caddy_public) — loopback/CI has no attack surface and runs
+# without baseline, so fail2ban isn't installed there. Lenient by default:
+# caddy_fail2ban_maxretry failures within findtime -> bantime ban (via nftables).
+caddy_fail2ban: true
+caddy_fail2ban_maxretry: 5
+caddy_fail2ban_findtime: 10m
+caddy_fail2ban_bantime: 1h

ansible/roles/caddy/handlers/main.yml

@@ -8,3 +8,11 @@
   # Under --check on a fresh host the unit isn't installed yet; flush_handlers would
   # otherwise fail trying to restart a non-existent service.
   when: not ansible_check_mode or 'caddy.service' in (ansible_facts.services | default({}))
+
+# fail2ban ships with baseline (always before caddy in anvil.yml); guard the same
+# way so a --check run on a fresh host (unit absent) skips rather than hard-fails.
+- name: Restart fail2ban
+  ansible.builtin.systemd:
+    name: fail2ban
+    state: restarted
+  when: not ansible_check_mode or 'fail2ban.service' in (ansible_facts.services | default({}))

ansible/roles/caddy/tasks/main.yml

@@ -69,6 +69,28 @@
   # caddy_tsv_raw is only defined when the slurp above ran (skipped in check mode).
   when: not ansible_check_mode or caddy_tsv_stat.stat.exists
 
+- name: Ensure the caddy log directory exists
+  ansible.builtin.file:
+    path: "{{ caddy_access_log | dirname }}"
+    state: directory
+    owner: caddy
+    group: caddy
+    mode: "0750"
+
+# `caddy validate` (run as root by the template task below) instantiates the file
+# logger and CREATES the log file if absent — as root, which the caddy daemon then
+# can't open. Pre-create it owned by caddy so validate only ever opens an existing
+# caddy-owned file; touch+preserve keeps this idempotent and corrects ownership drift.
+- name: Ensure the caddy access log is owned by caddy
+  ansible.builtin.file:
+    path: "{{ caddy_access_log }}"
+    state: touch
+    owner: caddy
+    group: caddy
+    mode: "0640"
+    modification_time: preserve
+    access_time: preserve
+
 - name: Install the Caddyfile
   ansible.builtin.template:
     src: Caddyfile.j2
@@ -88,5 +110,88 @@
     state: started
   when: not ansible_check_mode or 'caddy.service' in (ansible_facts.services | default({}))
 
+# --- fail2ban: ban IPs that repeatedly fail RPC basic auth (public listener) ---
+# Only meaningful for the public HTTPS listener; the loopback/CI listener has no
+# attack surface and runs without baseline (so fail2ban isn't installed there).
+# fail2ban itself ships with baseline, which always runs before caddy in anvil.yml.
+
+# Fail loud, at the cause: if the jail is enabled but fail2ban is absent (e.g. a
+# standalone `--tags caddy` run that skipped baseline), the copies below would
+# write config for a non-existent service and the handler would abort later with a
+# confusing "service not found". Skipped under --check on a not-yet-provisioned host.
+- name: Assert fail2ban is installed before wiring the caddy-rpc jail
+  ansible.builtin.assert:
+    that: "'fail2ban.service' in (ansible_facts.services | default({}))"
+    fail_msg: >-
+      caddy_fail2ban is enabled but fail2ban is not installed on this host. It
+      ships with the baseline role — run the full anvil.yml (baseline before
+      caddy), or set caddy_fail2ban=false to skip RPC brute-force protection.
+  when: caddy_public | bool and caddy_fail2ban | bool and not ansible_check_mode
+
+- name: Install the caddy-rpc fail2ban filter
+  ansible.builtin.copy:
+    dest: /etc/fail2ban/filter.d/caddy-rpc.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # MANAGED BY the caddy role — do not edit by hand.
+      # Ban on a basic-auth failure (HTTP 401) in Caddy's one-line JSON access
+      # record, capturing the connecting IP. Order-independent lookaheads: JSON
+      # keys are unordered and Caddy/zap may reorder them across versions, so a
+      # positional `field.*field` regex would silently stop matching. Each
+      # `(?<!\\)` requires an UNescaped quote, so only real root-level keys match
+      # — never a `"status":401` injected inside an escaped header/URI value
+      # (Caddy escapes those to `\"status\":401`). No `logger` assertion needed:
+      # the jail's logpath is Caddy's dedicated access log (only access entries).
+      [Definition]
+      failregex = ^(?=.*(?<!\\)"remote_ip":"<HOST>")(?=.*(?<!\\)"status":401).*$
+      # `%%` is fail2ban config-parser escaping for a literal `%` (a bare `%`
+      # raises InterpolationSyntaxError and the filter won't load). `%%z` parses
+      # the trailing `Z` AS UTC — a literal `Z` would only consume the char and
+      # leave the time read in host-local zone, silently skewing the findtime
+      # window on a non-UTC host. `(?:\.%%f)?` allows the optional fractional
+      # seconds. Requires the Caddyfile's `format json { time_format iso8601 }`.
+      datepattern = "ts":"%%Y-%%m-%%dT%%H:%%M:%%S(?:\.%%f)?%%z
+      ignoreregex =
+  notify: Restart fail2ban
+  when: caddy_public | bool and caddy_fail2ban | bool
+
+- name: Install the caddy-rpc fail2ban jail
+  ansible.builtin.copy:
+    dest: /etc/fail2ban/jail.d/caddy-rpc.local
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # MANAGED BY the caddy role — do not edit by hand.
+      [caddy-rpc]
+      enabled   = true
+      filter    = caddy-rpc
+      backend   = auto
+      logpath   = {{ caddy_access_log }}
+      maxretry  = {{ caddy_fail2ban_maxretry }}
+      findtime  = {{ caddy_fail2ban_findtime }}
+      bantime   = {{ caddy_fail2ban_bantime }}
+      # Ban on the nftables firewall (the host has no iptables); drop 80+443.
+      banaction = nftables-multiport
+      port      = http,https
+  notify: Restart fail2ban
+  when: caddy_public | bool and caddy_fail2ban | bool
+
 - name: Apply caddy config changes now
   ansible.builtin.meta: flush_handlers
+
+# A malformed filter/jail lets fail2ban start but silently never load the jail —
+# leaving the operator believing brute-force protection is active when no IP can
+# ever be banned. Confirm it loaded; fail loud at deploy. Retry: the client socket
+# can briefly lag the restart. (This is the only check on the public path, which
+# molecule never exercises — see roles/caddy/README.md.)
+- name: Verify the caddy-rpc fail2ban jail is loaded
+  ansible.builtin.command: fail2ban-client status caddy-rpc
+  changed_when: false
+  register: caddy_f2b_status
+  until: caddy_f2b_status.rc == 0
+  retries: 5
+  delay: 2
+  when: caddy_public | bool and caddy_fail2ban | bool and not ansible_check_mode

ansible/roles/caddy/templates/Caddyfile.j2

@@ -26,8 +26,19 @@
     # Forward authenticated requests to the local anvil JSON-RPC.
     reverse_proxy {{ anvil_host }}:{{ anvil_port }}
 
-    # Access logging to the journal (auditing who hit the RPC).
-    log
+    # Access logging as JSON to a (self-rolling) file: auditing who hit the RPC,
+    # and the log fail2ban's caddy-rpc jail watches for repeated 401s. The jail's
+    # datepattern parses the `ts` string field, so `time_format iso8601` is required
+    # here (the Caddy default is a Unix-epoch float the datepattern can't read).
+    log {
+        output file {{ caddy_access_log }} {
+            roll_size 10MiB
+            roll_keep 5
+        }
+        format json {
+            time_format iso8601
+        }
+    }
 }
 {% else %}
 {
@@ -49,7 +60,18 @@
     # Forward authenticated requests to the local anvil JSON-RPC.
     reverse_proxy {{ anvil_host }}:{{ anvil_port }}
 
-    # Access logging to the journal (auditing who hit the RPC).
-    log
+    # Access logging as JSON to a (self-rolling) file: auditing who hit the RPC,
+    # and the log fail2ban's caddy-rpc jail watches for repeated 401s. The jail's
+    # datepattern parses the `ts` string field, so `time_format iso8601` is required
+    # here (the Caddy default is a Unix-epoch float the datepattern can't read).
+    log {
+        output file {{ caddy_access_log }} {
+            roll_size 10MiB
+            roll_keep 5
+        }
+        format json {
+            time_format iso8601
+        }
+    }
 }
 {% endif %}