R4: Dynamic domain source — app-provided domains with automatic TLS (poll + push refresh, per-domain certs, optional stable SAN batching)

[mhenrixon]

Problem / Goal

SaaS apps hosting many customer domains cannot use kamal-proxy for TLS today because the proxy only learns hostnames at deploy time. Reference case (wm3): 1429 domains — 961 covered by a *.enode.site DNS-01 wildcard, ~468 external customer domains — currently served by a Traefik tier fed by generated file-provider YAML, kept current by a daily cron → regenerate → PR → merge → rsync pipeline (310 auto-generated update-traefik-domains-* branches of churn). The app already exposes the list: GET /api/v1/domains with bearer-token auth.

Goal: the proxy learns the domain list from the app at runtime — polls a configurable endpoint and accepts an authenticated "refresh now" nudge — and fully manages Let's Encrypt certs for that list: proactive throttled issuance, background renewal (ARI), per-domain failure quarantine, eviction on removal. Done = a customer domain added in the app's DB serves HTTPS within minutes, with zero deploys, config edits, or PRs. This collapses the entire Traefik tier for the wm3 topology (client → dash LB with TLS → per-host proxies → app).

This feature also fixes three latent defects in the SAN manager it builds on:

1. Unguarded SNI provisioning: SANCertManager.GetCertificate calls provisionCertificate(hello.ServerName) for ANY unknown SNI (san_cert_manager.go:313-314). With a host-less (catch-all) service, Router.GetCertificate (router.go:318-329) consults the SAN manager for arbitrary scanner-supplied names — poisoning batches and burning rate limits. The dynamic set must become a hard allowlist.
2. No background renewal: SAN certs renew only if a TLS handshake happens inside the final 24h of validity (san_cert_manager.go:236,304), and at that point the batch disintegrates (provisionCertificate batches the handshake domain + whatever is pending — not the cert's original set). The separate CertificateRenewalManager (cert_renewal.go) is dead code with zero callers.
3. Whole-batch failure loop: one failing domain fails the entire Obtain, and the error path re-queues ALL domains (san_cert_manager.go:382-388) — a retry loop against LE's 5-auth-failures/identifier/hour and 300-orders/3h limits.

Context (read these first)

• internal/server/san_cert_manager.go — the issuance engine this feature extends: lego client, in-memory HTTP-01 provider, pendingDomains batching, acme.state persistence (atomic .tmp+rename at :654-666), sanCertID hashing (:668-675). Read fully.
• internal/server/router.go:293-330 — GetCertificate chain: CertificateRegistry → serviceForHost → service.certManager. Unknown SNI → ErrorUnknownServerName (handshake abort).
• internal/server/service.go:423-456 — createCertManager: static certs → shared sanCertManager (registers each deploy host) → per-service autocert. service.go:82-96 ServiceOptions (JSON-persisted; new fields need tags and old-state zero-value compat via MarshalJSON/UnmarshalJSON at :273-337). service.go:98-123 Validate() — currently requires non-empty hosts for --tls.
• internal/server/health_check.go — the ticker/ctx-cancel loop + direct-to-target HTTP GET pattern to copy for polling (:38-76, :78-115).
• internal/server/cert_renewal.go — the Start/Stop + WaitGroup lifecycle template (:21-77); currently dead code (do not wire it for SAN — this feature builds its own loop; registry renewal is issue R1: Start CertificateRenewalManager (never started today) #1).
• internal/server/load_balancer.go:124-134, internal/server/target.go:143-152,279-291 — healthy-target enumeration + address/URL building for polling through targets. Note: no public "one healthy target" accessor exists; add one.
• internal/server/server.go:211-227 — buildHandler: where the SAN HTTP-01 interceptor mounts; the nudge endpoint and pre-flight probe handler mount here too. server.go:178-202 — metrics listener pattern (reference only).
• internal/cmd/deploy.go:29-67 + internal/server/commands.go:19-26 — deploy flag → ServiceOptions RPC plumbing.
• internal/cmd/run.go:46-124 — where managers initialize and where the poller/renewal loop must start/stop.
• internal/server/config.go:34-75 — state paths; add DynamicDomainsStatePath().
• internal/metrics/metrics.go:81-101 — SetCertificateExpiry / IncCertificateRenewals / SetCertificateCount exist with zero callers; the reconcile loop wires them (overlaps issue R1: Wire the 3 certificate Prometheus metrics #2).
• Upstream PR implement the On-Demand TLS feature basecamp/kamal-proxy#63 (On-Demand TLS, open) — the autocert HostPolicy seam, local-path ask variant, and the "--tls + on-demand ⇒ Hosts forced to [""]" validation relaxation. Stay merge-compatible: our allowlist is a local, cached equivalent of its ask gate. Related: our issue R4: On-demand TLS with ask endpoint #13 (porting feat(cache): shared RFC 9111 response cache with stale-while-revalidate #63) — this feature supersedes its allowlist half; the external-ask-URL half remains R4: On-demand TLS with ask endpoint #13.

Hard external constraints (Let's Encrypt, verified June 2025 docs + 2026 posts)

Limit	Value
New orders / account	300 per 3h (refill 1/36s)
New certs / registered domain	50 per 7 days
Duplicate certs (exact same SAN set)	5 per 7 days
SANs / certificate	100 (classic profile); 25 on tlsserver/shortlived
Auth failures / identifier / account	5 per hour; zombie-pausing at 1152 consecutive

• Renewals with an IDENTICAL identifier set are exempt from order + per-domain limits (still subject to duplicate-cert 5/week). ARI-based renewals (replaces) are exempt from ALL limits. lego supports ARI (ShouldRenewAt, MakeARICertID); autocert does not (proposal: x/crypto/acme: support the ACME Renewal Information standard in Client golang/go#60958).
• Cert lifetimes shrink: 90d today → 64d (Feb 2027) → 45d (Feb 2028); authz reuse 30d → 10d → 7h. Batch renewal frequency ×2 then ×4, re-validating every member each time.
• OCSP is dead (responders off Aug 2025) — no stapling anywhere.
• Industry practice at 1000+ tenant domains is per-hostname certs (Caddy/CertMagic refuses multi-SAN; Fly.io per-hostname; lua-resty-auto-ssl per-SNI). SAN batches co-publish all tenants in CT logs (privacy leak) and give one dead tenant a 99-domain blast radius.
• Cold-start math at batch size 1: 468 orders ÷ 300/3h ≈ 4.7h one-time. Steady state = exempt renewals. Batching is NOT needed for rate limits.

Decision

Hybrid source + per-domain-first issuance:

1. Poll is the source of truth — per-service --tls-domains-source (path resolved against a healthy target of the service, or absolute URL), default interval 300s with jitter, ETag/If-None-Match, last-known set persisted to dynamic-domains.state so boot doesn't depend on the app being up.
2. Push is a cache-invalidation nudge, not a data channel — POST /.kamal-proxy/domains/refresh on the existing listeners triggers an immediate re-poll. Bearer token REQUIRED (env KAMAL_PROXY_REFRESH_TOKEN, constant-time compare), 404 when unconfigured, 429 under a 10s min-interval. Carries no domain data → no second source of truth, replay-harmless, works cross-host (wm3's Sidekiq hosts ≠ LB host).
3. Batch size is a knob, default 1 — --tls-domains-batch-size=1 issues per-domain certs (industry-aligned; renewals ARI/identical-set exempt; zero rebalancing pathology; no CT tenant-list leak). Values 2–25 enable stable batching: append-only fill of new batches, membership changes ONLY at renewal boundaries (compaction: drop evicted/quarantined, top-up from pending), never reshuffle otherwise. Values >25 rejected (new-profile SAN cap).
4. The dynamic set is a hard allowlist — GetCertificate provisions only domains in (deploy-registered hosts ∪ dynamic set); unknown SNI aborts the handshake. Fixes defect 1.

Alternatives rejected:

• Push-carries-the-list (PATCH/POST of domains): two sources of truth, boot-order dependency on the app, needs idempotency/versioning protocol; a nudge gets the same latency with none of that.
• Poll-only: simple but couples onboarding latency to the interval; tightening the interval globally is wasteful vs. an event-driven nudge.
• Always-batch to 90–100 SANs (what wm3's disabled Traefik generator did): every membership change is a rate-limited new order; one dead tenant blocks 89 domains; duplicate-cert 5/week punishes set oscillation; SAN cap is 25 on newer profiles; CT privacy leak. Retained only as the opt-in knob with the stable planner.

Implementation steps (ordered; all in this repo; branch off san-certificate-batching, merge forward into dash — this extends a dash-only subsystem that does not exist on main)

1. Allowlist gate + config plumbing. Add TLSDomainsSource, TLSDomainsInterval, TLSDomainsBatchSize to ServiceOptions (json tags; zero-value = feature off) with flags in deploy.go. Relax ServiceOptions.Validate(): --tls with empty hosts is allowed when TLSDomainsSource != "" (mirror PR feat(cache): shared RFC 9111 response cache with stale-while-revalidate #63's approach; service becomes the catch-all route). In SANCertManager, add an allowlist (allowedDomains map[string]struct{} unioned with deploy-registered domains); GetCertificate returns ErrCertNotFound for SNI outside it instead of provisioning.
2. DomainSource poller. New internal/server/domain_source.go: per-service loop (health_check.go ticker pattern, cert_renewal.go Start/Stop+WaitGroup lifecycle). Path mode: resolve a healthy target (add a small LoadBalancer.HealthyTargets() accessor), GET http://<target><path> with Host: = first service host (or unset), optional Authorization: Bearer $KAMAL_PROXY_DOMAINS_TOKEN. URL mode: GET the absolute URL. Honor ETag/304. Response schema: {"domains": ["example.com", ...]} — validate each against the hostname grammar, reject payloads >1MB or >10k entries, log+skip *.-prefixed entries (wildcards stay registry-scoped; follow-up). Diff against current set → add/remove via the manager; log added/removed counts.
3. State. DynamicDomainsStatePath() in config.go → dynamic-domains.state: {services: {name: {domains, etag, fetched_at}}, quarantine: {domain: {until, failures}}}, atomic write idiom from san_cert_manager.go:654-666. Load on boot before first poll so certs serve immediately.
4. Refresh nudge. Handler mounted in buildHandler beside the ACME interceptor: POST /.kamal-proxy/domains/refresh → constant-time bearer check against KAMAL_PROXY_REFRESH_TOKEN env (404 if env unset or no service has a source; 401 bad token; 429 if <10s since last; 202 accepted → trigger immediate poll). Log requester IP.
5. Issuance planner + throttle. Replace the ad-hoc pending-batch logic for dynamic domains: a queue drained by a worker honoring a local token bucket (defaults: burst 20, refill 1/40s ≈ 250 orders/3h safety margin under LE's 300) and max 3 concurrent orders. Batch assembly: take up to batch-size domains per order (default 1). Extract a certObtainer interface around client.Certificate.Obtain so planner logic is unit-testable without ACME.
6. Pre-flight self-probe. Before a domain's FIRST issuance attempt: serve a per-boot random nonce at /.kamal-proxy/preflight/<nonce> (mounted in buildHandler), GET http://<domain>/.kamal-proxy/preflight/<nonce> via the default resolver with 5s timeout. Reachable+matching ⇒ eligible; else quarantine (5m first backoff) without burning an ACME order. Skip for renewals.
7. Quarantine. Per-domain failure tracking with backoff 15m → 1h → 4h → 24h (cap). On batch failure, parse lego's authorization errors to identify failing identifiers; quarantine those, re-enqueue the survivors ONCE (not a loop — fixes defect 3). Quarantined domains are excluded from batch assembly and from renewal sets; cleared when removed from the source or on successful issuance.
8. Renewal loop. One background loop (Start/Stop lifecycle, started in run.go after manager init, stopped before server stop): hourly tick; for each managed cert, renew when inside lego's ARI window (ShouldRenewAt) when available, else at NotAfter - lifetime/3 + jitter(0–6h) (works for the 90d and 45d eras — do NOT hardcode 30 days). Renew re-obtains the SAME identifier set minus evicted/quarantined members, passing ARI replaces (lego MakeARICertID); batch-size>1 compaction/top-up happens ONLY here. Swap maps atomically, persist, GC cert directories no longer referenced by any domain (fixes defect 2; sanCertID changes when the set changes — old dirs must be pruned).
9. Eviction. Domain removed from source ⇒ immediately out of the allowlist (no new issuance), cert keeps serving until its renewal drops it; when a cert's domain set becomes empty, delete state entry + cert dir.
10. Observability. Wire SetCertificateCount/SetCertificateExpiry/IncCertificateRenewals from the reconcile/renewal paths (overlaps issue R1: Wire the 3 certificate Prometheus metrics #2 — coordinate, don't duplicate); extend GetStats() with dynamic-domain counts (total/pending/quarantined/certs); slog every refresh diff.
11. CLI (small). kamal-proxy domains <list|stats|refresh> via the existing RPC surface (commands.go) for on-host ops/debugging.
12. Docs. README section (schema, flags, tokens, nudge endpoint, LE-constraints note, wm3-style example); update ROADMAP.md R4 to reference this issue.

Verification gates

• make test — all green; new unit tests: poller (httptest source: 200/304/ETag/schema rejection/oversize), nudge (401/404/429/202 + constant-time token), planner (batch assembly at size 1 and 5; append-only invariant; renewal-boundary compaction only), quarantine backoff progression + survivor re-enqueue-once, allowlist gate (unknown SNI rejected; catch-all no longer provisions arbitrary names), state round-trip (old acme.state without new fields still loads), renewal-set identity (unchanged set preserved for exemption).
• gofmt -l internal/ cmd/ — empty; go vet ./... — clean. (make lint runs in CI.)
• Manual staging pass: kamal-proxy deploy test --tls --tls-staging --tls-domains-source=/domains against a local httptest app on LE staging — first domain issued, nudge picks up an added domain, removed domain evicted at renewal.
• Existing suite unaffected: services WITHOUT --tls-domains-source behave byte-identically (deploy-registered hosts still issue via the existing path).

Out of scope

• Gem-side deploy.yml plumbing (proxy.tls_domains.*) and kamal proxy domains passthrough — paired issue in mhenrixon/kamal.
• Forwarding *. wildcard entries from the source to the CertificateRegistry (follow-up; registry already supports explicit wildcards).
• Loopia DNS provider for *.wm3.se parity (separate S-sized issue; lego supports it, our factory lacks it).
• Porting PR feat(cache): shared RFC 9111 response cache with stale-while-revalidate #63's external ask-URL mode (remains issue R4: On-demand TLS with ask endpoint #13; keep the HostPolicy seam compatible).
• Multi-ACME-host coordination (HTTP-01 tokens are in-memory per proxy; wm3 terminates TLS on the single LB host — document as a topology requirement).
• Certificate revocation on eviction (eviction = stop renewing; revocation adds nothing given CRL-only world).
• No edits to Dockerfile/Makefile/script/release (upstream-owned); no commits to main; no plain v* tags (fork tags are v<base>.<n>).

Execution

Hand to a fresh implementation session (sonnet tier) in ~/Code/mhenrixon/kamal-proxy. Branch off san-certificate-batching (NOT main — this subsystem is dash-only), PR into dash. Release lands as the next v0.9.2.N proxy tag BEFORE any gem release that references it (.claude/rules/upstream-sync.md → Release procedure).

[mhenrixon]

Gem-side plumbing: zoolutions/dash#15 (blocked by this issue)

[mhenrixon]

Implementation up: PR #24 (feature/dynamic-domain-source → dash). All 12 implementation steps done; 83-agent adversarial review ran and its 18 confirmed findings are fixed in-branch. Manual LE-staging pass deferred to pre-release validation.