Skip to content

deployment.md

Latest commit

 

History

History
250 lines (196 loc) · 6.27 KB

deployment.md

File metadata and controls

250 lines (196 loc) · 6.27 KB

Deployment Setup (Forge + Nginx + Caddy On-Demand TLS)

This document captures the current server setup and the SSL automation design using Caddy while keeping the same server IP (95.216.207.99).

Overview

  • Server managed by Laravel Forge.
  • IP: 95.216.207.99
  • Nginx keeps handling HTTP on :80.
  • Caddy handles HTTPS on :443 with On-Demand TLS.
  • Caddy reverse proxies HTTPS traffic to Nginx on 127.0.0.1:80.
  • Default app catches all hostnames pointing to this IP (intentional).

Applications on Server

  1. assets.shuttlemath.com
  • Static assets site (games/assets).
  • Nginx-only app in Forge.
  1. shuttlemath.com
  • Node.js app.
  • Deployment script:
cd /home/forge/shuttlemath.com
git pull origin $FORGE_SITE_BRANCH

npm ci
npm run build

pm2 delete shuttlemath || true
pm2 start npm --name "shuttlemath" -- start
  • Nginx proxies to http://localhost:3000.
  1. default
  • Node.js app (catch-all for domains on this IP).
  • Deployment script:
cd /home/forge/default
git fetch
git reset --hard origin/$FORGE_SITE_BRANCH

npm ci
npm run restart
  • Nginx default server proxies to http://127.0.0.1:8080.

Why Caddy Was Added

Forge/Nginx catch-all can route unknown hosts on HTTP, but HTTPS certificates must still be issued per hostname.
Caddy On-Demand TLS issues certs at first HTTPS hit (if approved by ask endpoint), which gives low-effort SSL coverage for newly pointed domains.

Final Traffic Flow

  1. New domain points A to 95.216.207.99.
  2. HTTPS request hits Caddy (:443).
  3. Caddy calls local ask endpoint (127.0.0.1:9123) to decide if cert issuance is allowed.
  4. If allowed, Caddy obtains/serves cert and proxies request to Nginx 127.0.0.1:80.
  5. Nginx default app handles host logic and upstream proxying.

Caddy Installation

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
sudo chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install -y caddy

Ask Service (Domain Allow Check)

Create /opt/caddy-ask/ask.mjs:

import http from 'node:http';
import { lookup } from 'node:dns/promises';
import { isIP } from 'node:net';

const PORT = 9123;
const SERVER_IPS = new Set(['95.216.207.99']); // Add IPv6 too if used
const CACHE_TTL_MS = 5 * 60 * 1000;
const cache = new Map();

function validDomain(d) {
  if (!d || d.length > 253 || isIP(d)) return false;
  return /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+$/i.test(d);
}

async function pointsToUs(domain) {
  try {
    const addrs = await lookup(domain, { all: true, verbatim: true });
    return addrs.some(a => SERVER_IPS.has(a.address));
  } catch {
    return false;
  }
}

http.createServer(async (req, res) => {
  const u = new URL(req.url, 'http://127.0.0.1');
  if (u.pathname !== '/allow') return res.writeHead(404).end();

  const domain = (u.searchParams.get('domain') || '').toLowerCase().trim();
  if (!validDomain(domain)) return res.writeHead(403).end('invalid');

  const now = Date.now();
  const hit = cache.get(domain);
  if (hit && hit.exp > now) return res.writeHead(hit.ok ? 200 : 403).end();

  const ok = await pointsToUs(domain);
  cache.set(domain, { ok, exp: now + CACHE_TTL_MS });
  return res.writeHead(ok ? 200 : 403).end(ok ? 'ok' : 'deny');
}).listen(PORT, '127.0.0.1', () => {
  console.log(`ask service listening on 127.0.0.1:${PORT}`);
});

Create /etc/systemd/system/caddy-ask.service:

[Unit]
Description=Caddy On-Demand TLS ask endpoint
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=www-data
Group=www-data
ExecStart=/usr/bin/node /opt/caddy-ask/ask.mjs
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now caddy-ask
sudo systemctl status caddy-ask --no-pager

Caddyfile (Only HTTPS Listener)

Use /etc/caddy/Caddyfile:

{
	email you@shuttlemath.com
	auto_https disable_redirects
	on_demand_tls {
		ask http://127.0.0.1:9123/allow
	}
}

:443 {
	tls {
		on_demand
		issuer acme {
			disable_http_challenge
		}
	}
	reverse_proxy 127.0.0.1:80
}

Validate and confirm it listens only on :443:

sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo caddy adapt --config /etc/caddy/Caddyfile --adapter caddyfile --pretty | jq '.apps.http.servers|to_entries[]|{name:.key,listen:.value.listen}'

Expected:

{
  "name": "srv0",
  "listen": [
    ":443"
  ]
}

Start Caddy:

sudo systemctl enable --now caddy
sudo systemctl restart caddy
sudo systemctl status caddy --no-pager

Nginx Default App Config (Redirect Placement)

In the existing default app server { ... } block (port 80), keep redirect before location /:

if ($redirect_host != "") {
    return 301 https://$redirect_host$request_uri;
}

This is the correct replacement for:

return 301 $scheme://$redirect_host$request_uri;

Do not add a blanket return 301 https://$host$request_uri; in this server unless you implement loop-safe conditions.

Verification

sudo ss -ltnp | egrep ':80|:443'
sudo systemctl status caddy --no-pager
sudo systemctl status caddy-ask --no-pager
curl -vkI https://h.shuttlemath.com
journalctl -u caddy -n 100 --no-pager

Expected listener state:

  • :80 -> nginx
  • :443 -> caddy

Troubleshooting

  1. Caddy says port 80 in use
  • Caddy config still contains :80 or auto-redirect behavior not disabled.
  • Confirm Caddyfile exactly matches this doc and validate/adapt output shows only :443.
  1. Cert not issued for new domain
  • Domain not resolving to 95.216.207.99 yet.
  • ask endpoint denies domain.
  • DNS propagation delay.
  • Let’s Encrypt rate limits.
  1. Check active Caddy unit options
sudo systemctl cat caddy

Ensure startup uses /etc/caddy/Caddyfile and no unexpected override.

Rollback

sudo systemctl disable --now caddy

Nginx on :80 continues serving HTTP exactly as before.