Spec: Hosted DNS, Cluster Domain and wildcard certificates #1015

Problem Statement

A Cloud user needs working HTTPS hostnames for their Services before they own a domain. Today the daemon reserves a generated domain from a third-party DNS service, stores the reservation in cluster state, and exposes five frozen DNS RPCs. From 0.2.0 every daemon RPC and store row is part of the Stable promise, so that surface would be frozen for a Cloud-only feature. It also leaves a hard capacity limit: every generated hostname shares Let's Encrypt's 50 new certificates per registered domain per week, the apex will not be on the Public Suffix List at launch, and Let's Encrypt is not taking rate-limit override requests.

Solution

Managed hostnames become a Cloud feature, served by a Ployz-run Hosted DNS service. The daemon knows nothing about generated domains.

reserve   Cloud ──▶ Hosted DNS    at founder enrollment (Organization slug as preferred label)
records   Cloud ──▶ Hosted DNS    hourly sync + events: probed ingress IPs → apex A/AAAA; *.name CNAME apex
cert      Cloud ──CSR──▶ Hosted DNS ──DNS-01──▶ Google Trust Services   one wildcard per Cluster Domain
publish   Cloud ──PublishCertificateMaterial(*.name)──▶ daemon           covers every generated hostname
deploy    Cloud ──▶ daemon        managed hostnames arrive expanded as explicit hostnames
release   Cloud ──▶ Hosted DNS    on Organization delete; name retired forever

Each Organization consumes one new wildcard certificate, issued centrally from Ployz's own Google Trust Services account for names under Ployz's apex. Customer Machines never order certificates for generated names. Custom domains keep per-hostname Let's Encrypt HTTP-01 on the daemon, unchanged. Self-hosted Clouds use the same Ployz-hosted Hosted DNS and get all of this.

Decisions: Design the Hosted DNS service (its certificate section is superseded by #1003), Design Cloud's ownership of the cluster domain, Verify ZeroSSL ACME quotas, and the certificate grilling recorded on Spec and tickets for the Hosted DNS and cluster-domain build.

User Stories

1. As a Cloud user, I want my Organization to receive a generated Cluster Domain when my first Server enrols, so that my Services get public hostnames without buying a domain.
2. As a Cloud user, I want the generated name to be based on my Organization slug, so that it is recognisable.
3. As a Cloud user, I want a readable suffix when my slug is taken or reserved, so that I still get a name instead of an error.
4. As a Cloud user, I want my Cluster Domain to survive teardown and re-pairing, so that my URLs do not change when I rebuild.
5. As a Cloud user, I want enrollment to succeed even when Hosted DNS is unreachable, so that DNS never blocks getting a Server running.
6. As a Cloud user, I want Server Settings to show my Cluster Domain with a copy button, so that I can share it.
7. As a Cloud user, I want to see when the records were last published, so that I know whether DNS is current.
8. As a Cloud user, I want to see which ingress Servers are in the record set and which were not reachable on port 80, so that I can fix firewalls.
9. As a Cloud user, I want a Publish now button, so that I can republish after fixing a Server without waiting an hour.
10. As a Cloud user whose domain was never reserved, I want Publish now to reserve it, so that I can recover from an earlier outage.
11. As a Cloud user, I want removed Servers to drop out of DNS, so that visitors are not sent to a Machine that no longer exists.
12. As a Cloud user, I want a failed probe or missing runtime frame to leave the previous records in place, so that a transient failure never blanks my DNS.
13. As a Cloud user, I want the lease renewed hourly with a 7-day TTL, so that a Cloud outage over a weekend loses nothing.
14. As a Cloud user, I want a Service's managed hostname shown as prefix.name in the Service drawer, so that I see the real URL.
15. As a Cloud user without a reserved domain, I want the drawer to say "pending" and link to Server Settings, so that I know what to do.
16. As a Cloud user, I want generated hostnames and PLOYZ_PUBLIC_DOMAIN to render with no Server connected, so that the UI does not depend on a live runtime frame.
17. As a Cloud user, I want deploys that need a managed hostname to try one inline reservation and otherwise fail with a named reason, so that I never deploy a broken hostname silently.
18. As a Cloud user, I want every generated hostname served over valid HTTPS immediately after deploy, so that new Services never wait on certificate issuance.
19. As a Cloud user, I want my Organization's wildcard renewed before it expires, so that HTTPS never lapses.
20. As a Cloud user, I want Server Settings to show the wildcard's expiry, so that I can see certificate health.
21. As a Cloud user, I want deleting my Organization to release its name, so that nothing lingers on my behalf.
22. As a Self-hosted Cloud operator, I want the same generated domains and wildcard certificates with only a Hosted DNS URL default, so that self-hosting needs no CA or DNS setup.
23. As a Self-hosted Cloud operator, I want to point Cloud at a different Hosted DNS endpoint through configuration, so that testing and staging are possible.
24. As a Machine operator, I want the daemon to hold no generated-domain state, RPCs, or tokens, so that the Stable promise does not freeze a Cloud feature.
25. As a Machine operator, I want to publish my own certificate and key for a hostname (for example a Cloudflare Origin certificate), so that I can serve names ACME cannot issue.
26. As a Machine operator, I want published Certificate Material never replaced by ACME renewal, so that my uploaded certificate stays in use.
27. As a Machine operator, I want to clear published material and hand the hostname back to ACME, so that I can undo a publish.
28. As a Machine operator, I want a published wildcard to cover matching hostnames, so that the daemon never orders a certificate that a wildcard already covers.
29. As a Machine operator, I want custom domains to keep automatic Let's Encrypt HTTP-01 issuance, so that nothing changes for my own domains.
30. As a CLI user, I want the dns command and the --no-dns/--dns-endpoint flags gone, so that the CLI only exposes supported operations.
31. As the Ployz operator, I want Hosted DNS to be a small DNS-and-signing service holding no customer private keys (only its own ACME account key), so that its breach exposes no customer key.
32. As the Ployz operator, I want anonymous minting rate-limited per source IP and unused reservations reaped after 24 hours, so that anonymous minting cannot exhaust the namespace, while Hosted Cloud mints for every Organization from one egress IP with a mint key.
33. As the Ployz operator, I want released names retired forever, so that a released name can never be reissued to someone else.
34. As the Ployz operator, I want a record update to rewrite only the apex A/AAAA, with the wildcard a fixed CNAME to the apex, so that address changes touch two records.
35. As the Ployz operator, I want certificate issuance limited to the token holder's own name and its wildcard, so that a token cannot obtain certificates for other names.

Implementation Decisions

Hosted DNS service (new private repository getployz/hosted-dns)

• One Rust axum service on Railway (project ployz-hosted-dns, service api, EU West; EC2 dropped in #984). Route 53 serves one hosted zone for the apex ployz.app. PlanetScale Postgres stores reservations. DNS-only plus certificate signing; no queue.
• API (from #976): anonymous POST /domains {preferred?} → {name, token}; the token is returned once and stored bcrypt-hashed; everything else uses the per-name bearer token. rotate returns a new token and kills the old one at once. PUT records takes the full apex A/AAAA set and is idempotent; *.name is a CNAME to the apex, written once. release retires the name forever. The lease is renewed by any authenticated call and lasts 7 days; reservations with no records are reaped after 24 hours; anonymous minting is rate-limited per source IP. A mint key (Authorization: Bearer <key> on POST /domains, matched against the service's MINT_KEYS list) bypasses the per-IP limit; a wrong key is refused, not downgraded to anonymous. Hosted Cloud uses a key because it mints for every Organization from one egress IP; Self-hosted Clouds mint anonymously.
• Preferred label: [a-z0-9-]{3,40}, no edge hyphens, reserved-word list (www, api, dns, mail, admin, cloud, app, ployz, …). A taken or refused label gets a 4-character random suffix; an invalid one gets a fully random label. The response always carries the granted name.
• Certificate endpoint (new, supersedes #976's "no certificates"): POST /domains/{name}/certificate {csr} → PEM chain. The CSR must name exactly name and *.name; anything else is refused. The service orders from Google Trust Services (one Ployz Google Cloud project, one ACME account bound with EAB) using DNS-01 TXT records it writes in its own zone, waits for Route 53 to sync, finalises, and returns the chain. It stores no private keys and no certificates. The call is synchronous and may take minutes. Google Trust Services is the only CA.
• The service's contract is Cloud-facing only; it is not a daemon surface and is not covered by the Stable promise.

Cloud (dashboard)

• Table organization_cluster_domain, one row per Organization, as decided in Design Cloud's ownership of the cluster domain #985: organization_id (PK, FK cascade), endpoint, name, encrypted_token, reserved_at, lease_renewed_at, records_synced_at, published, timestamps. Adds three optional certificate columns: encrypted private key, certificate chain, certificate_not_after.
• PLOYZ_HOSTED_DNS_URL joins Cloud config with a Ployz-hosted default, in the same shape as the installer URL. Optional PLOYZ_HOSTED_DNS_MINT_KEY is sent on reserve only when set (Hosted Cloud sets it; Self-hosted leaves it unset).
• Reserve in founder enrollment completion; Hosted DNS failure never fails enrollment. Enrollment completion does not see the founder's IP, so it fires the sync event and the sync publishes the founder's records from the runtime frame.
• One Inngest function sync-cluster-domain: event cluster-domain/sync.requested (founder enrollment completion, machine removal, Publish now) plus an hourly cron fanning out one event per Organization that has a Cluster Domain row, paired or not, so a torn-down Organization's lease never lapses; concurrency 1 per Organization. Steps: reserve if the row is missing → probe ingress Machines with a public IP (GET http://<ip>/.ployz-verify must return the Machine id within 5 s) → full-set PUT if the frame exists and the probed set is non-empty → renew lease → ensure certificate. With no Cluster the sync still renews the lease. A name Hosted DNS reports reaped (404) or retired (410) is replaced by a fresh reservation, and its old wildcard is cleared from the Cluster, so deploys never ship a dead name.
• Ensure certificate: with no certificate, or under 30 days to certificate_not_after, generate a fresh key and CSR, call the certificate endpoint, store the key encrypted with APP_ENCRYPTION_SECRET plus the chain, then publish *.name to the Cluster through PublishCertificateMaterial. It also republishes when the Cluster lacks the current material (for example after re-pairing).
• Deploy compile reads the Organization's row instead of the runtime watch frame and emits ${prefix}.${name} as explicit hostnames; the daemon never receives a bare prefix. With no row: one inline reserve, then refuse with a named failure. servicePublicDomain keeps its signature. UI readers take the name from an Organization query. The up.ployz.dev placeholder goes.
• Release: in the organization-scope teardown that deletes the Organization row (that is where Organizations are deleted, not a Better Auth hook), call release once and ignore failure; the cascade removes the row. Project or cluster teardown, pairing reset and machine removal never touch the domain.
• No rename, re-reserve or manual release at 0.2.0.

Core (daemon, SDK, CLI)

• Delete (from #976): the Hosted DNS client module, the hosted_dns cluster row and its store functions, the four DNS RPCs (ReserveDomain, GetDomain, ReleaseDomain, CreateDomainRecords) and their request/response types, hosted_dns_hostname in the runtime watch frame and SDK payloads, IngressHostname::ClusterDomain and its label type, core managedHostnames lowering (managedHostnames stays in the core-owned authored environment document, because Cloud's parse, diff, review and restore all run through it; core lowering refuses a non-empty list, and Cloud expands it to explicit hostnames before lowering), the CLI dns command, --no-dns and --dns-endpoint, machine add/remove DNS refresh, the CLI probe, the Hosted DNS layer-3 suite, and --no-dns in release-qualification scripts.
• Keep: the Caddy /.ployz-verify handler and INGRESS_VERIFY_PATH (Cloud's sync depends on them), ClusterDnsVerdict (the certificate issuance gate), the internal .internal DNS server.
• Add one frozen RPC, PublishCertificateMaterial { hostname, set { certificate_chain_pem, private_key_pem } | clear }, advertised as a Cluster capability. hostname may be an explicit hostname or a single-level wildcard *.x. Set validates the chain, the key match, and that the leaf covers the hostname.
• The certificate row gains optional published: bool (default false; additive). ACME never orders, renews or overwrites a hostname with published material. Clear removes the material and returns the hostname to ACME.
• wanted_certificate_hosts skips an explicit hostname covered by a published wildcard (one label under x). Caddy loads the wildcard material and matches by SNI natively.
• No ZeroSSL fallback, no issuer list and no SetCertificatePolicy: the daemon's issuer behaviour is unchanged.
• Glossary: rewrite Hosted DNS and Ingress Hostname in core/CONTEXT.md (the vocabulary lives in dashboard/CONTEXT.md); add the published and wildcard semantics to Certificate Material.

Testing Decisions

• Test external behaviour at the highest seam; do not test private helpers.
• Hosted DNS: its HTTP API is the seam, run against a fake Route 53 and a fake ACME CA (DNS-01) in process: minting and suffixing, token rotation, full-set idempotent PUT, lease and reaper, retire-forever, CSR scope refusal, certificate issuance.
• Core: Machine RPC contract and capability tests for the RPC removals and PublishCertificateMaterial; the layer-3 certificates suite (prior art: its FakeCa cases) for published material never being renewed, clear returning to ACME, and a published wildcard suppressing ACME orders for covered hostnames. Remove the Hosted DNS layer-3 suite and the fake DNS RPCs in the cloud-init harness.
• Cloud: InngestTestEngine for sync-cluster-domain (prior art: the machine-removal Inngest tests) with a fake Hosted DNS HTTP server and stubbed runtime/probe: reserve, probe filtering, empty set keeps records, lease renewal, certificate ensure and publish. The Effect Postgres harness for the row, enrollment reservation and organization teardown release. Existing managed-service-exports tests move to expansion from the row. Register the new function in the Inngest policies test.
• Release qualification for generated domains stays with What release qualification must prove for 0.2.0.

Out of Scope

• Public Suffix List listing for the apex.
• A ZeroSSL or any daemon-side fallback issuer; SetCertificatePolicy.
• Bring-your-own base domain; rename, re-reserve or manual release of a Cluster Domain.
• A Servers-page public-ingress reachability indicator (#1009).
• Self-hostable Hosted DNS.
• Provisioning the apex, AWS, Route 53, Railway, PlanetScale and Google Cloud project (#984).

Further Notes

• Google Trust Services quotas are 100 new orders per hour per project; one wildcard per Organization plus renewals every ~60 days stays far below that.
• The wildcard key never exists in Hosted DNS; it lives encrypted in Cloud and in the Cluster's certificate row, which already holds private keys for ACME-issued material.
• Store changes are additive per core/DESIGN.md's Stable promise; the certificate-row change should land with or after Freeze tolerant daemon store and enrollment contracts.