feat(caddy): fail2ban jail to ban repeated RPC basic-auth failures

[thiras]

Context

The anvil devnet RPC is exposed on public HTTPS (rpc-dev.decdn.org:443) behind Caddy per-dev basic auth. A failed/missing credential returned HTTP 401 but nothing rate-limited or banned — a public endpoint with basic auth invites credential-stuffing. fail2ban was already installed (baseline role) with only an sshd jail.

What this does

Adds a fail2ban caddy-rpc jail that watches Caddy's access log and bans (via nftables) any IP that repeatedly fails RPC basic auth. Implemented in the caddy role (co-located with the auth + log it protects; keeps the public decdn.node collection's baseline role generic).

• JSON access log — Caddyfile.j2 switches the bare log (journal) to a self-rolling JSON file (/var/log/caddy/rpc-access.log, iso8601 timestamps) that fail2ban parses.
• Filter + jail — gated on caddy_public and caddy_fail2ban; banaction = nftables-multiport matching the host firewall, drops 80+443. Lenient default: 5 failures / 10m → 1h ban (all overridable via caddy_fail2ban_*).
• Handler — check-mode-safe Restart fail2ban.
• Docs — new roles/caddy/README.md + one-line mentions in ansible/README.md and AGENTS.md.

Bug fixes found during verification / review

• caddy validate (run as root) created the log file root-owned, so the caddy daemon couldn't open it → crash / non-idempotent. Fixed by pre-creating the file as caddy:caddy before validate.
• datepattern used a bare % → fail2ban's configparser raises InterpolationSyntaxError and the filter never loads (the jail would be silently dead). Confirmed with fail2ban-regex; fixed to %%.

Fail-loud guards (the public path has no CI coverage — molecule runs caddy_public: false)

• Assert fail2ban is installed before wiring the jail (actionable error for a standalone --tags caddy run that skipped baseline).
• Verify the jail loaded (fail2ban-client status caddy-rpc) after restart, so a malformed filter fails loud at deploy instead of silently never banning.

Verification

• fail2ban-regex against real Caddy iso8601 401 lines: filter loads, date parses, 401 matches + IP captured, 200 correctly missed.
• make lint-ansible (production profile): 0 failures · make lint (markdown/shellcheck/yaml): pass · make security (KICS): no findings in changed files · make build/galaxy-check: collection still builds.
• Molecule (fresh): converge OK, new public-path tasks skip under caddy_public: false, idempotence changed=0, verify "All assertions passed".

Notes

• Access logs move from journalctl -u caddy to the JSON file; runtime logs stay in the journal.
• Pre-existing nftables flush ruleset caveat (a baseline nftables reload clears fail2ban's table until its next restart — affects the sshd jail too) is documented in the role README as a follow-up.
• Follow-up: a dedicated caddy_public: true Molecule scenario would cover the public path (larger effort — needs ACME/TLS in a container).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a fail2ban brute-force protection jail (caddy-rpc) for the public Caddy reverse proxy in front of the Anvil devnet. It configures Caddy to output JSON access logs to a file using ISO8601 timestamps, sets up fail2ban filters and jails to monitor these logs for 401 unauthorized errors, and adds validation tasks to ensure fail2ban is installed and the jail is loaded successfully. The review feedback highlights two critical issues in ansible/roles/caddy/tasks/main.yml: first, the failregex in the fail2ban filter is fragile and vulnerable to log injection due to relying on strict JSON field order and missing root-level key verification; second, the datepattern lacks support for the UTC timezone indicator (Z) and fractional seconds, which can cause timezone mismatches and silently break the jail's findtime window. Both comments provide robust regex and pattern suggestions to resolve these issues.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Copilot]

Pull request overview

This PR adds brute-force protection for the publicly exposed Anvil devnet RPC endpoint by introducing a fail2ban jail that detects repeated HTTP 401 basic-auth failures from Caddy’s access logs and bans offending IPs via nftables.

Changes:

• Switch Caddy RPC access logging from journald to a self-rolling JSON access log file suitable for fail2ban parsing.
• Add a caddy-rpc fail2ban filter + jail (gated on caddy_public and caddy_fail2ban), restart handler, and a post-restart verification that the jail actually loaded.
• Document the caddy role behavior and newly introduced fail2ban variables.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ansible/roles/caddy/templates/Caddyfile.j2	Changes access logging to a JSON file with ISO8601 timestamps for fail2ban parsing.
ansible/roles/caddy/tasks/main.yml	Creates log directory/file with correct ownership; installs fail2ban filter/jail and verifies the jail is loaded.
ansible/roles/caddy/handlers/main.yml	Adds a check-mode-safe handler to restart fail2ban when configs change.
ansible/roles/caddy/defaults/main.yml	Introduces defaults for access-log path and fail2ban jail tuning knobs.
ansible/roles/caddy/README.md	Documents the role and the new fail2ban-based brute-force protection behavior.
ansible/README.md	Mentions the new public-path fail2ban ban behavior and variables.
AGENTS.md	Notes that repeated basic-auth failures are banned via the caddy-rpc jail.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.