Hardening plan 1

README.md

@@ -105,11 +105,27 @@ Branch [main]:
 Use SSH deploy key? [Y/n]:
 ```
 
+Before building, seploy checks free space on the Docker data root (from
+`docker info`, falling back to `/var/lib/docker`); if less than 2 GB is
+free, the deploy aborts before touching any container, with a hint to run
+`docker system prune`. (If the check itself fails, the deploy proceeds.)
+
 Then seploy clones the repo, builds the Docker image, detects the app port
 from the image's `EXPOSE`, starts the container on a random localhost port,
 writes the Caddy config and reloads Caddy. Done — Caddy fetches a TLS
 certificate and your app is live at `https://stats.example.com`.
 
+As a last step, seploy fetches `https://<domain>` (or the app's healthcheck
+path, if set) from the public internet to confirm it is actually reachable
+— not just that the container passed its local health check on
+127.0.0.1. It retries for up to 30 seconds, since Caddy may still be
+fetching a TLS certificate on a first deploy. Success prints
+`edge: https://stats.example.com reachable (HTTP <code>)`; a failure prints
+a warning with a hint (e.g. "the domain does not resolve yet", or "check:
+systemctl status cloudflared" for a tunneled app whose cloudflared is down)
+without failing the deploy, since the release itself is already live and
+healthy.
+
 ### Private repos (deploy keys)
 
 If you answer yes to the deploy key prompt (the default), seploy:
@@ -141,14 +157,38 @@ prints a URL to open in your browser), creates a tunnel named `seploy`,
 and installs it as a service. All tunnel traffic goes to Caddy, which
 routes per app — so one tunnel serves every app on the server.
 
+After setup, the Cloudflare credentials (`cert.pem` and the tunnel's
+credentials file) live in `/etc/cloudflared` — the service and all seploy
+tunnel operations use that copy, so they don't depend on the home directory
+of whoever ran the login.
+
+`cloudflared` installs are pinned to a specific release and checksum-verified:
+seploy downloads the exact version it's tested against and checks its SHA256
+against the digest Cloudflare published for that release before the binary
+is ever installed — an unverified download is never installed. Get the
+newest pinned version with:
+
+```bash
+sudo seploy tunnel upgrade
+```
+
+This downloads and verifies the new version, atomically replaces the
+installed binary, restarts the `cloudflared` service if the tunnel is
+already configured, and prints the resulting `cloudflared --version` output.
+
 After that, app setup gains one question:
 
 ```text
 Expose via Cloudflare Tunnel (DNS record created automatically)? [Y/n]:
 ```
 
 Answer yes and seploy creates the Cloudflare DNS record itself. No A
-records, no waiting for propagation. Check things with:
+records, no waiting for propagation. If the domain already has a
+conflicting DNS record — an A record, or a CNAME to a different tunnel —
+seploy fails loudly and tells you what the domain currently resolves to
+instead of silently leaving the app unreachable; a pre-existing CNAME that
+already points at this server's tunnel (`<tunnel-id>.cfargotunnel.com`) is
+left alone and treated as success. Check things with:
 
 ```bash
 seploy tunnel status
@@ -157,6 +197,55 @@ seploy tunnel status
 Tunneled apps are served by Caddy as plain HTTP on localhost — TLS is
 terminated at Cloudflare's edge, and nothing is exposed publicly.
 
+### Tunnel setup without a browser (token)
+
+`seploy tunnel setup` above needs an interactive login (it prints a URL
+and waits for you to open it), which doesn't work well for scripted
+provisioning or servers with no one watching the terminal. For those cases,
+create the tunnel in the Cloudflare dashboard instead and hand seploy its
+token:
+
+1. In the Cloudflare dashboard, go to **Zero Trust > Networks > Tunnels**
+   and create a tunnel (choose "Cloudflared" as the connector).
+2. Copy the token shown on the install step — it's the long string after
+   `--token` in the install command Cloudflare suggests, not the whole
+   command.
+3. Run, on the VPS:
+
+   ```bash
+   sudo seploy tunnel setup --token <token>
+   ```
+
+   (or set `SEPLOY_TUNNEL_TOKEN` in the environment instead of passing
+   `--token`). This downloads `cloudflared` and installs it as a service
+   running that token — no login, no `tunnel create`, no `config.yml`.
+
+This is a **remotely-managed** tunnel: unlike `seploy tunnel setup`, which
+owns a local `config.yml` and lets seploy create DNS records and ingress
+rules for you, a token-mode tunnel's routing lives entirely in the
+dashboard. For every app you expose via the tunnel, add a **Public
+Hostname** in **Zero Trust > Networks > Tunnels > \<your tunnel\> > Public
+Hostname**:
+
+```text
+Domain:  <your app's domain>
+Service: http://localhost:80
+```
+
+That single step both creates the DNS record and routes the domain to
+Caddy — no separate DNS entry is needed. seploy prints this same reminder
+when you run `seploy <repo-url>` for a tunneled app while in token mode,
+since it cannot create this DNS record automatically (token-mode tunnels
+have no local `cert.pem` for seploy to authenticate DNS API calls with).
+`seploy tunnel status` and `seploy doctor` both report when the tunnel is
+running in token mode.
+
+On OpenRC hosts, note that the token has to be embedded directly in
+`/etc/init.d/cloudflared`'s `command_args` (OpenRC has no equivalent of
+systemd's token-file handling); seploy writes that file with mode `0700`
+so it's readable only by root, but be aware the token is there in plain
+text.
+
 ## Redeploy
 
 Push to your branch, then:
@@ -176,6 +265,20 @@ Container logs are size-capped (10 MB × 3 files per container) and
 Docker's build cache is trimmed after each deploy, so long-running
 servers don't slowly fill their disk.
 
+Add `--no-cache` to force a fresh build without Docker's build cache:
+
+```bash
+sudo seploy deploy lets-talk-statistics --no-cache
+```
+
+Use `--all` to redeploy every configured app, one at a time, in sorted
+order. It's mutually exclusive with a positional app name, continues past
+per-app failures, and prints a summary at the end:
+
+```bash
+sudo seploy deploy --all
+```
+
 ## Deploy automatically on push (webhooks)
 
 ```bash
@@ -210,6 +313,30 @@ This restarts the previous release, health checks it, switches Caddy
 traffic to it, then stops the broken one. Running rollback again switches
 back.
 
+Older releases are kept too (see `keep_releases` under Configuration), so
+you can roll back further than one step by targeting a specific commit:
+
+```bash
+sudo seploy rollback lets-talk-statistics --to a1b2c3d
+```
+
+`--to` accepts a full commit SHA or any unambiguous prefix. It searches the
+current, previous and history releases; if the SHA doesn't match anything,
+or matches more than one release, the command lists the available commits
+and when they were deployed. After a `--to` rollback, the release you left
+becomes "previous" (so a plain `seploy rollback` switches straight back to
+it), and history is left untouched.
+
+To see what's available before rolling back:
+
+```bash
+seploy releases lets-talk-statistics
+```
+
+This lists the current, previous and history releases with their commit,
+deploy time, and whether the release's image is still on disk (an image
+that's missing can no longer be targeted with `rollback --to`).
+
 ## Restart (apply env changes without rebuilding)
 
 ```bash
@@ -218,7 +345,11 @@ sudo seploy restart lets-talk-statistics
 ```
 
 `restart` recreates the current container from the already-built image
-with the fresh env file — seconds instead of a full clone + build.
+with the fresh env file — seconds instead of a full clone + build. It is
+zero-downtime: the replacement container starts on a new port, is health
+checked, and Caddy is switched over to it before the old container is
+stopped. If the replacement fails its health check, the old container is
+left running untouched and `restart` reports an error.
 
 ## Doctor
 
@@ -229,8 +360,19 @@ sudo seploy doctor
 Checks the server (docker daemon, caddy, ports 80/443, disk space, public
 IP) and every app (container running, domain DNS actually pointing at this
 server, tunnel and webhook services healthy), and prints a fix hint for
-anything that fails. New-app setup also runs a DNS preflight and warns if
-the domain doesn't point at the server yet.
+anything that fails. For tunneled apps it verifies the domain's CNAME
+target is this server's tunnel (`<tunnel-id>.cfargotunnel.com`), not just
+that the domain resolves — so it catches a stray A record or a CNAME to
+someone else's tunnel; if the CNAME lookup itself fails it falls back to a
+plain resolve check and notes that the CNAME target wasn't verified.
+New-app setup also runs a DNS preflight and warns if the domain doesn't
+point at the server yet.
+
+For every deployed app it also does the same public-edge reachability check
+as `deploy` — fetching `https://<domain>` (or its healthcheck path) from
+the internet — and reports it as a pass/fail line with a hint (DNS not
+resolving, TLS handshake failing, connection refused, or an HTTP >= 500
+from Caddy/cloudflared/the app itself).
 
 ## Logs
 
@@ -298,6 +440,12 @@ config, reloads Caddy, and removes `/var/lib/seploy/apps/<app>`. The SSH
 deploy key in `/etc/seploy/keys/` is intentionally **kept** — delete it
 manually (and remove it from GitHub) if you no longer need it.
 
+If the app was exposed through a Cloudflare Tunnel (`expose = "tunnel"`),
+its DNS record is **not** removed — the `cloudflared` CLI has no command to
+delete DNS records. `seploy remove` prints a reminder; delete the CNAME
+record for the app's domain yourself in the Cloudflare dashboard
+(DNS → Records) if it's no longer used.
+
 ## App requirements
 
 - The repo must contain a `Dockerfile` at its root (or at the path set in
@@ -308,7 +456,14 @@ manually (and remove it from GitHub) if you no longer need it.
 - The app must listen on `0.0.0.0` *inside* the container (containers are
   published only on the host's loopback, so this is safe).
 - Optional: set `healthcheck = "/healthz"` in `app.toml` to make deploys
-  wait (up to 30s) for a 2xx/3xx response before switching traffic.
+  wait (up to 30s by default, or `health_timeout` if set) for a 2xx/3xx
+  response before switching traffic.
+- Optional: set `health_timeout = "90s"` (a Go duration like `"90s"` or
+  `"3m"`) in `app.toml` to change how long deploys wait for the health
+  check (default 30s) before failing the deploy.
+- Optional: set `build_timeout = "10m"` (a Go duration like `"10m"` or
+  `"1h"`) in `app.toml` to change how long `docker build` is allowed to run
+  (default 30m) before the deploy fails.
 
 ## File layout
 
@@ -337,7 +492,7 @@ config_dir = "/etc/seploy"
 caddy_conf_dir = "/etc/caddy/conf.d"
 caddyfile = "/etc/caddy/Caddyfile"
 default_branch = "main"
-keep_releases = 3
+keep_releases = 3  # images of the last N releases; containers only for the current and previous release
 webhook_port = 8316
 ```
 
@@ -352,6 +507,8 @@ domain = "stats.example.com"
 dockerfile = "Dockerfile"
 port = 3000
 healthcheck = ""
+health_timeout = ""  # optional, Go duration like "90s" or "3m"; default 30s
+build_timeout = ""   # optional, Go duration like "10m" or "1h"; default 30m
 ssh_key = "/etc/seploy/keys/lets-talk-statistics_ed25519"
 expose = ""       # "" = Caddy + DNS, "tunnel" = Cloudflare Tunnel
 memory = ""       # optional, e.g. "512m"