From 67844b5fcbcf81f4bca518261e04c5365d8dee01 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sun, 7 Jun 2026 04:27:17 +0300 Subject: [PATCH 1/2] feat(caddy): fail2ban jail to ban repeated RPC basic-auth failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The anvil devnet RPC sits on public HTTPS behind Caddy basic auth, but failed credentials (HTTP 401) were only logged to the journal — nothing rate-limited or banned credential-stuffing. Switch the Caddy access log to a self-rolling JSON file (iso8601 ts) and add a fail2ban `caddy-rpc` jail that watches it and bans, via nftables, any IP that repeatedly fails basic auth. Lenient default: 5 failures / 10m -> 1h ban (caddy_fail2ban_* knobs). The jail is gated on the public listener (caddy_public) — loopback/CI has no attack surface and runs without baseline (where fail2ban ships). Fail loud on the public path, which molecule never exercises: assert fail2ban is installed before wiring the jail, and verify the jail loaded (fail2ban-client status) after restart. Pre-create the log file owned by caddy so `caddy validate` (run as root) doesn't leave it root-owned and crash the daemon. datepattern uses `%%` (fail2ban config-parser escaping; a bare `%` raises InterpolationSyntaxError and the filter won't load). Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 3 +- ansible/README.md | 6 +- ansible/roles/caddy/README.md | 81 ++++++++++++++++++ ansible/roles/caddy/defaults/main.yml | 13 +++ ansible/roles/caddy/handlers/main.yml | 8 ++ ansible/roles/caddy/tasks/main.yml | 98 ++++++++++++++++++++++ ansible/roles/caddy/templates/Caddyfile.j2 | 30 ++++++- 7 files changed, 232 insertions(+), 7 deletions(-) create mode 100644 ansible/roles/caddy/README.md diff --git a/AGENTS.md b/AGENTS.md index b8333de..3781e4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,7 +63,8 @@ secrets; generated on host) and #2 (localhost-only by default) hold for both con metrics/admin, operator-provisioned eth keystore, required chain knobs (no baked protocol facts — sourced from ADRs). **Internal: the anvil devnet** (`playbooks/anvil.yml` → baseline + anvil + Caddy basic-auth; anvil stays loopback while Caddy fronts it on public - https/443 with auto-TLS — `caddy_public: true`, default; flip to loopback-only for a tunnel) + https/443 with auto-TLS — `caddy_public: true`, default; flip to loopback-only for a tunnel; + repeated basic-auth failures are fail2ban-banned via the public-only `caddy-rpc` jail) — team tooling, not the product. Shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node stake/registration, ADR 019 Phase 2, is an operator step, not automated.) diff --git a/ansible/README.md b/ansible/README.md index 1d10011..46c3865 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -105,8 +105,9 @@ make add-dev USER_NAME=alice # mint + reveal a basic-auth dev user On the first anvil deploy the shared **mnemonic** and the `dev` basic-auth password are printed **once** — save them to the team vault. anvil binds `127.0.0.1` only; **caddy fronts it on public https/443** with auto-TLS + per-dev basic auth (`caddy_public: true`, default), -so the DNS A record must already point at the host. To keep it loopback-only instead (e.g. -behind a tunnel) set `caddy_public: false` — see the appendix. +so the DNS A record must already point at the host. Repeated basic-auth failures are +**fail2ban-banned** (the `caddy-rpc` jail, public listener only — see `roles/caddy`). To keep +it loopback-only instead (e.g. behind a tunnel) set `caddy_public: false` — see the appendix. --- @@ -139,6 +140,7 @@ Defaults live in each role (`roles/*/defaults/main.yml`); override in `group_var | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node (host_vars); sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `anvil_chain_id` … `rpc_hostname` | see `roles/anvil`,`roles/caddy` | internal devnet knobs. | +| `caddy_fail2ban` (+ `_maxretry`/`_findtime`/`_bantime`) | `true` (5 / 10m / 1h) | RPC basic-auth brute-force jail; public listener only. | --- diff --git a/ansible/roles/caddy/README.md b/ansible/roles/caddy/README.md new file mode 100644 index 0000000..d9c4e26 --- /dev/null +++ b/ansible/roles/caddy/README.md @@ -0,0 +1,81 @@ +# roles/caddy + +Puts a **per-dev HTTP basic-auth reverse proxy** in front of the loopback anvil +RPC. The internal anvil devnet (`playbooks/anvil.yml`) runs `baseline → anvil → +caddy`; Caddy is the only thing that faces the internet, and only after auth over +TLS. Internal tooling — **not** part of the public `decdn.node` collection. + +## What this role does + +1. **Installs Caddy** and ensures `/etc/caddy` (group `caddy`, `0750`). +2. **Mints basic-auth users** on the host into a TSV registry + (`caddy_users_tsv`), bcrypt-hashed via `caddy hash-password`, and renders the + `basic_auth` import block (`caddy_basicauth_file`). Credentials are generated + on the host — never in the repo. `caddy_initial_user` (`dev`) always exists; + `make add-dev USER_NAME=…` mints more. +3. **Renders the Caddyfile** (validated with `caddy validate`) in one of two + shapes: + - `caddy_public: true` (default) — public **HTTPS on `rpc_hostname:443`** with + auto-TLS (Let's Encrypt). Requires inbound tcp/80+443 (open via + `baseline_extra_inbound`). + - `caddy_public: false` — loopback plain-HTTP on `caddy_bind_port` (dev/CI, or + TLS terminated upstream by a tunnel). +4. **JSON access log** to `caddy_access_log` (self-rolling, 10 MiB × 5), recording + who hit the RPC. (Access logs move from the journal to this file — query it for + requests; Caddy's runtime/process logs still go to `journalctl -u caddy`.) +5. **fail2ban `caddy-rpc` jail** — *public listener only*. Watches the access log + and bans, via nftables, any IP that repeatedly fails basic auth (HTTP 401). + See below. + +## fail2ban: RPC basic-auth brute-force protection + +A public RPC behind basic auth invites credential-stuffing. When +`caddy_public: true` and `caddy_fail2ban: true` (both default), the role installs: + +- `/etc/fail2ban/filter.d/caddy-rpc.conf` — matches a `401` for the access logger + in the JSON log and captures the connecting `remote_ip`. +- `/etc/fail2ban/jail.d/caddy-rpc.local` — the `caddy-rpc` jail + (`banaction = nftables-multiport`, bans on 80+443). + +fail2ban itself comes from `baseline` (always run before `caddy` in `anvil.yml`). +On the loopback/CI listener the jail is **skipped** — there's no public attack +surface, and Molecule runs `caddy` without `baseline` (fail2ban absent). Lenient +defaults: **5 failures within 10m → 1h ban**. + +Because Molecule never exercises this path, the role self-checks at deploy: it +**asserts fail2ban is installed** before writing the jail (actionable error if you +ran `caddy` standalone without `baseline`), and **verifies the jail loaded** +(`fail2ban-client status caddy-rpc`) after the restart — so a malformed filter +fails loud instead of silently never banning. + +Inspect / unban on the host: + +```bash +sudo fail2ban-client status caddy-rpc +sudo fail2ban-client set caddy-rpc unbanip +``` + +> **Note:** baseline's nftables ruleset uses `flush ruleset`; a baseline nftables +> *reload* (only on template change) clears fail2ban's `f2b-table` until the next +> fail2ban restart. Pre-existing for the `sshd` jail too. + +## Key variables + +| Var | Default | Notes | +|-----|---------|-------| +| `caddy_public` | `true` | `true` = public HTTPS on `rpc_hostname`; `false` = loopback HTTP on `caddy_bind_port`. | +| `caddy_acme_email` | `""` | Let's Encrypt account contact; empty = anonymous ACME. | +| `caddy_bind_port` | `8080` | Loopback listener port (only when `caddy_public: false`). | +| `rpc_hostname` | `rpc-dev.decdn.org` | Public HTTPS host; printed in `ETH_RPC_URL`. | +| `anvil_host` / `anvil_port` | `127.0.0.1` / `8545` | Upstream RPC to proxy to. | +| `caddy_users_tsv` / `caddy_basicauth_file` | `/etc/caddy/…` | On-host credential registry + generated import. | +| `caddy_initial_user` | `dev` | Always-present basic-auth user. | +| `caddy_access_log` | `/var/log/caddy/rpc-access.log` | JSON access log; the log the fail2ban jail watches. | +| `caddy_fail2ban` | `true` | Enable the `caddy-rpc` jail (public listener only). | +| `caddy_fail2ban_maxretry` | `5` | Failed auths before a ban. | +| `caddy_fail2ban_findtime` | `10m` | Window the failures must fall within. | +| `caddy_fail2ban_bantime` | `1h` | Ban duration. | + +## Platforms + +Debian (bookworm), Ubuntu (jammy, noble). diff --git a/ansible/roles/caddy/defaults/main.yml b/ansible/roles/caddy/defaults/main.yml index 0c7f76c..01c2ce0 100644 --- a/ansible/roles/caddy/defaults/main.yml +++ b/ansible/roles/caddy/defaults/main.yml @@ -23,3 +23,16 @@ caddy_users_tsv: /etc/caddy/rpc-dev.users.tsv # usernamebcrypt regist caddy_basicauth_file: /etc/caddy/rpc-dev.basicauth # generated Caddyfile import caddy_initial_user: dev # always ensured to exist # caddy_extra_user: alice # set by add-dev-user.yml to mint one more user + +# JSON access log (self-rolling) recording who hit the RPC; also the log the +# fail2ban caddy-rpc jail watches for repeated basic-auth failures. +caddy_access_log: /var/log/caddy/rpc-access.log + +# fail2ban jail banning IPs that repeatedly fail RPC basic auth. Only wired up on +# the public listener (caddy_public) — loopback/CI has no attack surface and runs +# without baseline, so fail2ban isn't installed there. Lenient by default: +# caddy_fail2ban_maxretry failures within findtime -> bantime ban (via nftables). +caddy_fail2ban: true +caddy_fail2ban_maxretry: 5 +caddy_fail2ban_findtime: 10m +caddy_fail2ban_bantime: 1h diff --git a/ansible/roles/caddy/handlers/main.yml b/ansible/roles/caddy/handlers/main.yml index 873f60b..24cac60 100644 --- a/ansible/roles/caddy/handlers/main.yml +++ b/ansible/roles/caddy/handlers/main.yml @@ -8,3 +8,11 @@ # Under --check on a fresh host the unit isn't installed yet; flush_handlers would # otherwise fail trying to restart a non-existent service. when: not ansible_check_mode or 'caddy.service' in (ansible_facts.services | default({})) + +# fail2ban ships with baseline (always before caddy in anvil.yml); guard the same +# way so a --check run on a fresh host (unit absent) skips rather than hard-fails. +- name: Restart fail2ban + ansible.builtin.systemd: + name: fail2ban + state: restarted + when: not ansible_check_mode or 'fail2ban.service' in (ansible_facts.services | default({})) diff --git a/ansible/roles/caddy/tasks/main.yml b/ansible/roles/caddy/tasks/main.yml index b00429f..b3d1d03 100644 --- a/ansible/roles/caddy/tasks/main.yml +++ b/ansible/roles/caddy/tasks/main.yml @@ -69,6 +69,28 @@ # caddy_tsv_raw is only defined when the slurp above ran (skipped in check mode). when: not ansible_check_mode or caddy_tsv_stat.stat.exists +- name: Ensure the caddy log directory exists + ansible.builtin.file: + path: "{{ caddy_access_log | dirname }}" + state: directory + owner: caddy + group: caddy + mode: "0750" + +# `caddy validate` (run as root by the template task below) instantiates the file +# logger and CREATES the log file if absent — as root, which the caddy daemon then +# can't open. Pre-create it owned by caddy so validate only ever opens an existing +# caddy-owned file; touch+preserve keeps this idempotent and corrects ownership drift. +- name: Ensure the caddy access log is owned by caddy + ansible.builtin.file: + path: "{{ caddy_access_log }}" + state: touch + owner: caddy + group: caddy + mode: "0640" + modification_time: preserve + access_time: preserve + - name: Install the Caddyfile ansible.builtin.template: src: Caddyfile.j2 @@ -88,5 +110,81 @@ state: started when: not ansible_check_mode or 'caddy.service' in (ansible_facts.services | default({})) +# --- fail2ban: ban IPs that repeatedly fail RPC basic auth (public listener) --- +# Only meaningful for the public HTTPS listener; the loopback/CI listener has no +# attack surface and runs without baseline (so fail2ban isn't installed there). +# fail2ban itself ships with baseline, which always runs before caddy in anvil.yml. + +# Fail loud, at the cause: if the jail is enabled but fail2ban is absent (e.g. a +# standalone `--tags caddy` run that skipped baseline), the copies below would +# write config for a non-existent service and the handler would abort later with a +# confusing "service not found". Skipped under --check on a not-yet-provisioned host. +- name: Assert fail2ban is installed before wiring the caddy-rpc jail + ansible.builtin.assert: + that: "'fail2ban.service' in (ansible_facts.services | default({}))" + fail_msg: >- + caddy_fail2ban is enabled but fail2ban is not installed on this host. It + ships with the baseline role — run the full anvil.yml (baseline before + caddy), or set caddy_fail2ban=false to skip RPC brute-force protection. + when: caddy_public | bool and caddy_fail2ban | bool and not ansible_check_mode + +- name: Install the caddy-rpc fail2ban filter + ansible.builtin.copy: + dest: /etc/fail2ban/filter.d/caddy-rpc.conf + owner: root + group: root + mode: "0644" + content: | + # MANAGED BY the caddy role — do not edit by hand. + # Match a basic-auth failure (HTTP 401) in Caddy's JSON access log and + # capture the connecting IP. The whole JSON record is one line, so the + # logger -> remote_ip -> status ordering lets a single failregex span it + # (relies on Caddy's current JSON field emission order). + [Definition] + failregex = ^.*"logger":"http\.log\.access[^"]*".*"remote_ip":"".*"status":401.*$ + # `%%` is fail2ban config-parser escaping for a literal `%` (a bare `%` + # raises InterpolationSyntaxError and the filter won't load). Requires the + # Caddyfile's `format json { time_format iso8601 }` — the string `ts` field. + datepattern = "ts":"%%Y-%%m-%%dT%%H:%%M:%%S + ignoreregex = + notify: Restart fail2ban + when: caddy_public | bool and caddy_fail2ban | bool + +- name: Install the caddy-rpc fail2ban jail + ansible.builtin.copy: + dest: /etc/fail2ban/jail.d/caddy-rpc.local + owner: root + group: root + mode: "0644" + content: | + # MANAGED BY the caddy role — do not edit by hand. + [caddy-rpc] + enabled = true + filter = caddy-rpc + backend = auto + logpath = {{ caddy_access_log }} + maxretry = {{ caddy_fail2ban_maxretry }} + findtime = {{ caddy_fail2ban_findtime }} + bantime = {{ caddy_fail2ban_bantime }} + # Ban on the nftables firewall (the host has no iptables); drop 80+443. + banaction = nftables-multiport + port = http,https + notify: Restart fail2ban + when: caddy_public | bool and caddy_fail2ban | bool + - name: Apply caddy config changes now ansible.builtin.meta: flush_handlers + +# A malformed filter/jail lets fail2ban start but silently never load the jail — +# leaving the operator believing brute-force protection is active when no IP can +# ever be banned. Confirm it loaded; fail loud at deploy. Retry: the client socket +# can briefly lag the restart. (This is the only check on the public path, which +# molecule never exercises — see roles/caddy/README.md.) +- name: Verify the caddy-rpc fail2ban jail is loaded + ansible.builtin.command: fail2ban-client status caddy-rpc + changed_when: false + register: caddy_f2b_status + until: caddy_f2b_status.rc == 0 + retries: 5 + delay: 2 + when: caddy_public | bool and caddy_fail2ban | bool and not ansible_check_mode diff --git a/ansible/roles/caddy/templates/Caddyfile.j2 b/ansible/roles/caddy/templates/Caddyfile.j2 index 6c30a4c..4055988 100644 --- a/ansible/roles/caddy/templates/Caddyfile.j2 +++ b/ansible/roles/caddy/templates/Caddyfile.j2 @@ -26,8 +26,19 @@ # Forward authenticated requests to the local anvil JSON-RPC. reverse_proxy {{ anvil_host }}:{{ anvil_port }} - # Access logging to the journal (auditing who hit the RPC). - log + # Access logging as JSON to a (self-rolling) file: auditing who hit the RPC, + # and the log fail2ban's caddy-rpc jail watches for repeated 401s. The jail's + # datepattern parses the `ts` string field, so `time_format iso8601` is required + # here (the Caddy default is a Unix-epoch float the datepattern can't read). + log { + output file {{ caddy_access_log }} { + roll_size 10MiB + roll_keep 5 + } + format json { + time_format iso8601 + } + } } {% else %} { @@ -49,7 +60,18 @@ # Forward authenticated requests to the local anvil JSON-RPC. reverse_proxy {{ anvil_host }}:{{ anvil_port }} - # Access logging to the journal (auditing who hit the RPC). - log + # Access logging as JSON to a (self-rolling) file: auditing who hit the RPC, + # and the log fail2ban's caddy-rpc jail watches for repeated 401s. The jail's + # datepattern parses the `ts` string field, so `time_format iso8601` is required + # here (the Caddy default is a Unix-epoch float the datepattern can't read). + log { + output file {{ caddy_access_log }} { + roll_size 10MiB + roll_keep 5 + } + format json { + time_format iso8601 + } + } } {% endif %} From c68d89efee9de41cee61f4862e3c38b04ef2e2a8 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sun, 7 Jun 2026 04:40:01 +0300 Subject: [PATCH 2/2] =?UTF-8?q?fix(caddy):=20harden=20fail2ban=20filter=20?= =?UTF-8?q?per=20review=20=E2=80=94=20order-independent=20+=20UTC-aware?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address automated review (gemini-code-assist, copilot) on the caddy-rpc filter: - failregex no longer depends on JSON field order. Caddy/zap may reorder keys across versions, which would silently stop the jail from banning. Switch to order-independent lookaheads, each with an unescaped-quote lookbehind `(? --- ansible/roles/caddy/tasks/main.yml | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/ansible/roles/caddy/tasks/main.yml b/ansible/roles/caddy/tasks/main.yml index b3d1d03..0d0f860 100644 --- a/ansible/roles/caddy/tasks/main.yml +++ b/ansible/roles/caddy/tasks/main.yml @@ -136,16 +136,23 @@ mode: "0644" content: | # MANAGED BY the caddy role — do not edit by hand. - # Match a basic-auth failure (HTTP 401) in Caddy's JSON access log and - # capture the connecting IP. The whole JSON record is one line, so the - # logger -> remote_ip -> status ordering lets a single failregex span it - # (relies on Caddy's current JSON field emission order). + # Ban on a basic-auth failure (HTTP 401) in Caddy's one-line JSON access + # record, capturing the connecting IP. Order-independent lookaheads: JSON + # keys are unordered and Caddy/zap may reorder them across versions, so a + # positional `field.*field` regex would silently stop matching. Each + # `(?".*"status":401.*$ + failregex = ^(?=.*(?")(?=.*(?