From a0737427fde35cc61fe9f3e47f38648a0ac7d69f Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 02:19:26 +0100 Subject: [PATCH 1/9] docs(adr): decide the dedicated sending address for company Email Record how outbound mail gets one stable, reverse-DNS-matched source: one Hetzner Floating IPv4 on one labelled static worker, used only by the Email server through a Cilium egress-gateway policy, with the address on a dummy interface on every worker. It sets the order of the remaining steps, starting with the maintainer creating the approved address. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-sending-address.md | 90 +++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 docs/adr/email-sending-address.md diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md new file mode 100644 index 0000000000..d532c35ad9 --- /dev/null +++ b/docs/adr/email-sending-address.md @@ -0,0 +1,90 @@ +# Email sending address + +**Status:** Proposed + +## Context + +Company Email delivers directly to receiving mail servers (#5481); the +maintainer rejected a hosted relay. A pod's outbound connections leave with the +address of whichever node it runs on. That address changes when the pod moves +or a node is replaced, and it has no reverse DNS for the mail host name, so +receivers would treat the mail as suspect. + +On 2026-11-03 the maintainer approved one dedicated, movable IPv4 address and +its recurring cost. Hetzner blocks outbound SMTP per account, and the protected +no-message check has already reached an external mail server on port 25 from a +cluster node ([proof](../dr/email-provider-smtp-proof.md)). So the account permits +direct delivery; what is missing is a stable sending address. + +## Decision + +- **One Hetzner Floating IPv4** in the cluster's location is the only source + of Email's outbound mail. Its reverse DNS names the configured mail host, and + that name's address record points back to it. SPF authorizes this address, + Email signs with DKIM, and DMARC aligns on the mailbox domain. +- **A Cilium egress-gateway policy** routes only the Email server's + internet-bound traffic through one gateway node, with the Floating IP as the + source. Other workloads keep their current egress. The policy covers IPv4 + destinations only, so a gateway without IPv6 stays selectable. +- **The gateway is one static worker**, selected by a single node label. + Hetzner accepts the Floating IP as a source only from the server it is + assigned to, so it is assigned to that worker. Cilium requires the address + on a network device, so every worker carries it on a dummy interface. This + leaves the main interface, and the node addresses that Cilium and the kubelet + select, unchanged. +- **No high availability.** If the gateway node is lost, Cilium drops the + matching traffic instead of sending it from another address, and Email's + queue retries the messages. Moving the gateway means labelling another static + worker and reassigning the address through a reviewed main-only workflow. No + in-cluster controller or cloud token is added. +- The address and host name stay in private configuration. Public manifests + read them from the platform's encrypted substitution variables. + +## Order + +1. The maintainer creates the Floating IPv4. This is the only account action + and the only new cost. +2. A reviewed main-only workflow assigns it to the gateway worker, sets its + reverse DNS and reads both back. +3. A separate revision enables the egress gateway in Cilium, which restarts + the agents and operator. Existing egress must be unchanged afterwards. +4. Add the dummy-interface address to workers, label the gateway and apply the + Email policy. First confirm that KSail's pinned Talos configuration library + accepts the dummy-interface document. +5. From the Email server's own network identity, repeat the no-message SMTP + check. Confirm that the external server sees the Floating IP as the source. +6. Publish the host's address record and the SPF, DKIM and DMARC records + through the DNS owner, after the maintainer approves them. MX still moves + only after migration and recovery are verified. + +## Alternatives + +- **A hosted relay:** rejected by the maintainer. +- **A dedicated mail node with its own address:** costs a whole server, which + is not authorized. +- **A controller that moves the address automatically:** adds a component and + an in-cluster cloud token. Static workers are rarely replaced, and Email's + queue tolerates a manual move. +- **The nodes' own addresses:** they change when nodes are replaced, so they + cannot be published in SPF or given stable reverse DNS. + +## Consequences + +Cilium applies an egress-gateway policy to a new pod after a short delay. +Until then, that pod's traffic can leave with its node's address and fail SPF. +Activation therefore checks the observed source after the Email server +restarts, before outbound delivery is relied on. A gateway outage delays +outbound mail; it never sends from an unauthorized address. Inbound mail +continues to use the existing load balancer and is unaffected by this +decision. + +Egress gateway is incompatible with Cluster Mesh and CiliumEndpointSlice; +neither is enabled. Replies from the gateway are unencrypted only with XDP +acceleration, which is not enabled either. + +## Sources + +- [Cilium egress gateway](https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/) +- [Hetzner Floating IP configuration](https://docs.hetzner.com/cloud/floating-ips/persistent-configuration/) +- [Hetzner outbound mail policy](https://docs.hetzner.com/cloud/servers/faq/#why-can-i-not-send-any-mails-from-my-server) +- [Talos dummy link configuration](https://docs.siderolabs.com/talos/v1.13/reference/configuration/network/dummylinkconfig) From 4b5d09ddd655e1211766d21572e124b50a2ee838 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 03:24:13 +0100 Subject: [PATCH 2/9] docs(adr): require proof that node addresses stay unchanged The dummy-interface address is meant to leave each node's own addresses alone, but no source establishes that for the kubelet's node address or Cilium's masquerade address. Make the implementing revision prove both, and state the start-up SPF window as something activation measures. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-sending-address.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md index d532c35ad9..9862209a08 100644 --- a/docs/adr/email-sending-address.md +++ b/docs/adr/email-sending-address.md @@ -29,9 +29,11 @@ direct delivery; what is missing is a stable sending address. - **The gateway is one static worker**, selected by a single node label. Hetzner accepts the Floating IP as a source only from the server it is assigned to, so it is assigned to that worker. Cilium requires the address - on a network device, so every worker carries it on a dummy interface. This - leaves the main interface, and the node addresses that Cilium and the kubelet - select, unchanged. + on a network device, so every worker carries it on a dummy interface rather + than on its main interface. The node address the kubelet reports and the + address Cilium masquerades other traffic to must stay the server's own; the + implementing revision proves both, and pins the kubelet's node address if + needed. - **No high availability.** If the gateway node is lost, Cilium drops the matching traffic instead of sending it from another address, and Email's queue retries the messages. Moving the gateway means labelling another static @@ -50,7 +52,8 @@ direct delivery; what is missing is a stable sending address. the agents and operator. Existing egress must be unchanged afterwards. 4. Add the dummy-interface address to workers, label the gateway and apply the Email policy. First confirm that KSail's pinned Talos configuration library - accepts the dummy-interface document. + accepts the dummy-interface document. Afterwards, confirm that every node's + reported address and other pods' egress source are unchanged. 5. From the Email server's own network identity, repeat the no-message SMTP check. Confirm that the external server sees the Floating IP as the source. 6. Publish the host's address record and the SPF, DKIM and DMARC records @@ -71,10 +74,10 @@ direct delivery; what is missing is a stable sending address. ## Consequences Cilium applies an egress-gateway policy to a new pod after a short delay. -Until then, that pod's traffic can leave with its node's address and fail SPF. -Activation therefore checks the observed source after the Email server -restarts, before outbound delivery is relied on. A gateway outage delays -outbound mail; it never sends from an unauthorized address. Inbound mail +Until then, that pod's traffic can leave with its node's address, so a message +sent in that window can fail SPF. Activation measures that window after an +Email server restart, and the outcome decides whether outbound delivery needs +a start-up delay. A gateway outage only delays outbound mail. Inbound mail continues to use the existing load balancer and is unaffected by this decision. From e3f6a9e11599fb5717cc1b14af29a3d15bdabe7f Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 03:24:55 +0100 Subject: [PATCH 3/9] docs(adr): say where the Talos patch reads the private address Talos machine config is rendered at deploy time, so it cannot use Flux substitution variables. Name the deploy-time secret path, as used for the WireGuard key, and state the provider's port block as its default. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-sending-address.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md index 9862209a08..0d42dc0a78 100644 --- a/docs/adr/email-sending-address.md +++ b/docs/adr/email-sending-address.md @@ -11,10 +11,10 @@ or a node is replaced, and it has no reverse DNS for the mail host name, so receivers would treat the mail as suspect. On 2026-11-03 the maintainer approved one dedicated, movable IPv4 address and -its recurring cost. Hetzner blocks outbound SMTP per account, and the protected -no-message check has already reached an external mail server on port 25 from a -cluster node ([proof](../dr/email-provider-smtp-proof.md)). So the account permits -direct delivery; what is missing is a stable sending address. +its recurring cost. Hetzner blocks outbound SMTP by default, per account. The +protected no-message check has already reached an external mail server on port +25 from a cluster node ([proof](../dr/email-provider-smtp-proof.md)), so the +account permits direct delivery. What is missing is a stable sending address. ## Decision @@ -39,8 +39,9 @@ direct delivery; what is missing is a stable sending address. queue retries the messages. Moving the gateway means labelling another static worker and reassigning the address through a reviewed main-only workflow. No in-cluster controller or cloud token is added. -- The address and host name stay in private configuration. Public manifests - read them from the platform's encrypted substitution variables. +- The address and host name stay in private configuration. Kubernetes + manifests read them from the platform's encrypted substitution variables, and + the Talos patch from a deploy-time secret, as the WireGuard key already does. ## Order From 619b9f045f298d52c29dd9afbabb6d6fab81b945 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 04:48:31 +0100 Subject: [PATCH 4/9] docs(adr): gate every Email start and keep inbound off the sending address Address the review of the sending-address decision: - a dedicated sending host name carries the Floating IP's reverse DNS and address record, so the mail host name stays on the load balancer; - moving the gateway removes the old label, and delivery resumes only when exactly one node carries it; - every Email start waits for an in-pod check to see the Floating IP as its source, instead of measuring the policy delay after one restart. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-sending-address.md | 48 +++++++++++++++++++------------ 1 file changed, 29 insertions(+), 19 deletions(-) diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md index 0d42dc0a78..606b0206a2 100644 --- a/docs/adr/email-sending-address.md +++ b/docs/adr/email-sending-address.md @@ -19,9 +19,11 @@ account permits direct delivery. What is missing is a stable sending address. ## Decision - **One Hetzner Floating IPv4** in the cluster's location is the only source - of Email's outbound mail. Its reverse DNS names the configured mail host, and - that name's address record points back to it. SPF authorizes this address, - Email signs with DKIM, and DMARC aligns on the mailbox domain. + of Email's outbound mail. Its reverse DNS names a dedicated sending host, + that host's address record points back to it, and Email greets receiving + servers with that name. The mail host name that clients and inbound mail use + stays on the load balancer. SPF authorizes this address, Email signs with + DKIM, and DMARC aligns on the mailbox domain. - **A Cilium egress-gateway policy** routes only the Email server's internet-bound traffic through one gateway node, with the Floating IP as the source. Other workloads keep their current egress. The policy covers IPv4 @@ -36,9 +38,16 @@ account permits direct delivery. What is missing is a stable sending address. needed. - **No high availability.** If the gateway node is lost, Cilium drops the matching traffic instead of sending it from another address, and Email's - queue retries the messages. Moving the gateway means labelling another static - worker and reassigning the address through a reviewed main-only workflow. No - in-cluster controller or cloud token is added. + queue retries the messages. Moving the gateway means removing the label from + the old worker, labelling another static worker and reassigning the address + through a reviewed main-only workflow. Delivery resumes only after exactly one + node carries the label. No in-cluster controller or cloud token is added. +- **Every start is gated.** Cilium applies the policy to a new pod after a + short delay, and until then its traffic leaves with the node's address. A + start-up check in the Email pod therefore waits until a fixed external + address-reflection endpoint sees the Floating IP as its source, and the + server starts only then. The check shares the pod's address, so the server + inherits the proven route. - The address and host name stay in private configuration. Kubernetes manifests read them from the platform's encrypted substitution variables, and the Talos patch from a deploy-time secret, as the WireGuard key already does. @@ -48,16 +57,18 @@ account permits direct delivery. What is missing is a stable sending address. 1. The maintainer creates the Floating IPv4. This is the only account action and the only new cost. 2. A reviewed main-only workflow assigns it to the gateway worker, sets its - reverse DNS and reads both back. + reverse DNS to the sending host name and reads both back. 3. A separate revision enables the egress gateway in Cilium, which restarts the agents and operator. Existing egress must be unchanged afterwards. -4. Add the dummy-interface address to workers, label the gateway and apply the - Email policy. First confirm that KSail's pinned Talos configuration library - accepts the dummy-interface document. Afterwards, confirm that every node's - reported address and other pods' egress source are unchanged. +4. Add the dummy-interface address to workers, label the gateway, and apply + the Email policy together with its start-up check. First confirm that + KSail's pinned Talos configuration library accepts the dummy-interface + document. Afterwards, confirm that every node's reported address and other + pods' egress source are unchanged, and that a restarted Email server starts + only after its check passes. 5. From the Email server's own network identity, repeat the no-message SMTP check. Confirm that the external server sees the Floating IP as the source. -6. Publish the host's address record and the SPF, DKIM and DMARC records +6. Publish the sending host's address record and the SPF, DKIM and DMARC records through the DNS owner, after the maintainer approves them. MX still moves only after migration and recovery are verified. @@ -74,13 +85,12 @@ account permits direct delivery. What is missing is a stable sending address. ## Consequences -Cilium applies an egress-gateway policy to a new pod after a short delay. -Until then, that pod's traffic can leave with its node's address, so a message -sent in that window can fail SPF. Activation measures that window after an -Email server restart, and the outcome decides whether outbound delivery needs -a start-up delay. A gateway outage only delays outbound mail. Inbound mail -continues to use the existing load balancer and is unaffected by this -decision. +The start-up gate couples one thing to the gateway: while the gateway is +down, a restarting Email server waits, and its inbound mail and mailbox access +wait with it. A running server keeps serving; a lost gateway only delays its +outbound mail. If the pinned Email server can hold just its outbound +queue until the check passes, that narrower gate replaces the start-up gate. +Inbound mail otherwise continues to use the existing load balancer. Egress gateway is incompatible with Cluster Mesh and CiliumEndpointSlice; neither is enabled. Replies from the gateway are unencrypted only with XDP From fe1767263d6d950ba819457a6cbc24d7eca3f2b9 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 05:34:33 +0100 Subject: [PATCH 5/9] docs(adr): keep the existing SPF sender and state the routing reason SPF must keep the existing provider while mail is still sent through it, or the maintainer's current mail fails SPF during the transition. Base the gateway assignment on Hetzner routing the Floating IP only to its assigned server, which is documented, rather than on an inferred source filter. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-sending-address.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md index 606b0206a2..8d2a4087e5 100644 --- a/docs/adr/email-sending-address.md +++ b/docs/adr/email-sending-address.md @@ -22,15 +22,17 @@ account permits direct delivery. What is missing is a stable sending address. of Email's outbound mail. Its reverse DNS names a dedicated sending host, that host's address record points back to it, and Email greets receiving servers with that name. The mail host name that clients and inbound mail use - stays on the load balancer. SPF authorizes this address, Email signs with - DKIM, and DMARC aligns on the mailbox domain. + stays on the load balancer. SPF adds this address and keeps the existing + provider for as long as mail is still sent through it. Email signs with DKIM, + and DMARC aligns on the mailbox domain. - **A Cilium egress-gateway policy** routes only the Email server's internet-bound traffic through one gateway node, with the Floating IP as the source. Other workloads keep their current egress. The policy covers IPv4 destinations only, so a gateway without IPv6 stays selectable. - **The gateway is one static worker**, selected by a single node label. - Hetzner accepts the Floating IP as a source only from the server it is - assigned to, so it is assigned to that worker. Cilium requires the address + Hetzner routes the Floating IP only to the server it is assigned to, so + replies to outbound connections reach only that server. The address is + therefore assigned to the gateway worker. Cilium requires the address on a network device, so every worker carries it on a dummy interface rather than on its main interface. The node address the kubelet reports and the address Cilium masquerades other traffic to must stay the server's own; the From 10497a23389bded843949e1bdaf7720167b33308 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 06:23:32 +0100 Subject: [PATCH 6/9] docs(adr): decide Amazon SES as the outbound relay for company Email On 2026-11-13 the maintainer switched outbound company mail from direct delivery to a hosted relay. Replace the direct-delivery sending-address design with the relay decision: SES in eu-central-1, Easy DKIM and a custom MAIL FROM domain, one send-only SMTP credential kept in OpenBao, and the order of the remaining steps. No dedicated address is bought. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-outbound-relay.md | 78 ++++++++++++++++++++++ docs/adr/email-sending-address.md | 106 ------------------------------ 2 files changed, 78 insertions(+), 106 deletions(-) create mode 100644 docs/adr/email-outbound-relay.md delete mode 100644 docs/adr/email-sending-address.md diff --git a/docs/adr/email-outbound-relay.md b/docs/adr/email-outbound-relay.md new file mode 100644 index 0000000000..a42020d974 --- /dev/null +++ b/docs/adr/email-outbound-relay.md @@ -0,0 +1,78 @@ +# Email outbound relay + +**Status:** Proposed + +## Context + +Company Email first planned direct delivery from the cluster (#5481). That needs +a stable sending address with matching reverse DNS, routed only for the Email +server. On 2026-11-13 the maintainer switched to a hosted relay: *"Let's switch +to a relay, but do a quick research to find the cheapest option that support my +business. We should be able to change to a more advancced and expensive option +later given it is just a relay, but no need to overcomplicatd things from the +start"*. No dedicated address is bought. Inbound mail, migration and recovery +are unchanged. + +## Decision + +- **Amazon SES** on the existing AWS account, pay-per-use, in `eu-central-1`. + At about $0.10 per 1,000 messages it costs pennies at this volume, and its + terms allow ordinary mailbox mail from our own domain. +- **The mail domain is an SES identity** with 2048-bit Easy DKIM, plus a custom + MAIL FROM domain on a subdomain, so both SPF and DKIM align for DMARC. SES + signs every relayed message. It rewrites Message-ID and Date, which can break + Stalwart's own signature, but SES's aligned signature still passes DMARC. +- **Stalwart relays all outbound mail** to the SES SMTP endpoint on port 587, + with STARTTLS and SMTP authentication. +- **One send-only credential.** An IAM user may only `ses:SendRawEmail` on that + identity, with the sender limited to the mail domain. Crossplane creates its + access key, and the provider publishes the derived SES SMTP password. That + password is valid only for the provider's own region, which is why SES runs + in `eu-central-1`. A dedicated push identity stores it in OpenBao, and the + Email server reads it from there. Nobody reads the value. +- **Ownership.** The identity and MAIL FROM domain grant no permission, so the + `aws` tenant owns them. The SMTP user, its policy and its key stay + platform-owned, like the EKS CI role, so the public tenant cannot widen them. +- **DNS.** The three DKIM records, the MAIL FROM MX and SPF records, and DMARC + go through the DNS owner. SPF keeps the existing provider while mail is still + sent through it. +- **Production access** is requested through a reviewed main-only workflow. + Until SES grants it, its sandbox delivers only to verified recipients. +- **Swappable.** Changing relay means changing Stalwart's relay route and its + credential; nothing else depends on SES. + +## Order + +1. Install the SES provider and activate the kinds (#6184). +2. Declare the identity and MAIL FROM domain (devantler-tech/aws#78). +3. Create the send-only SMTP identity and store its credential in OpenBao. +4. Publish the DNS records once the identity's DKIM tokens exist. +5. Request production access. +6. Add Stalwart's relay route and its egress to the SES endpoint. +7. Send real mail to external mailboxes. Check that SPF, DKIM and DMARC pass, + and that replies thread against the sender's Sent copy despite SES's + Message-ID rewrite. If threading fails, report it before switching relay. + +## Alternatives + +- **Direct delivery:** needs a dedicated address, reverse DNS and egress + routing for one server. Superseded by the maintainer on 2026-11-13. +- **SMTP2GO Free:** 1,000 messages a month, hosted in the EU, and its terms + allow this use. It is the fallback if SES fails the test in step 7. +- **Scaleway TEM, Cloudflare Email Service:** transactional-only terms. +- **SES in `eu-north-1`:** the provider would publish an SMTP password for the + wrong region, so it would need a separate derivation step. + +## Consequences + +The account starts in the SES sandbox, so real delivery waits on production +access. A lost or revoked access key stops outbound mail until Crossplane +creates a new one, and the push identity stores it again. Relayed mail and its +metadata are processed by SES in the EU. + +## Sources + +- [SES pricing](https://aws.amazon.com/ses/pricing/) +- [Requesting production access](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html) +- [DMARC with SES](https://docs.aws.amazon.com/ses/latest/dg/send-email-authentication-dmarc.html) +- [Header fields SES changes](https://docs.aws.amazon.com/ses/latest/dg/header-fields.html) diff --git a/docs/adr/email-sending-address.md b/docs/adr/email-sending-address.md deleted file mode 100644 index 8d2a4087e5..0000000000 --- a/docs/adr/email-sending-address.md +++ /dev/null @@ -1,106 +0,0 @@ -# Email sending address - -**Status:** Proposed - -## Context - -Company Email delivers directly to receiving mail servers (#5481); the -maintainer rejected a hosted relay. A pod's outbound connections leave with the -address of whichever node it runs on. That address changes when the pod moves -or a node is replaced, and it has no reverse DNS for the mail host name, so -receivers would treat the mail as suspect. - -On 2026-11-03 the maintainer approved one dedicated, movable IPv4 address and -its recurring cost. Hetzner blocks outbound SMTP by default, per account. The -protected no-message check has already reached an external mail server on port -25 from a cluster node ([proof](../dr/email-provider-smtp-proof.md)), so the -account permits direct delivery. What is missing is a stable sending address. - -## Decision - -- **One Hetzner Floating IPv4** in the cluster's location is the only source - of Email's outbound mail. Its reverse DNS names a dedicated sending host, - that host's address record points back to it, and Email greets receiving - servers with that name. The mail host name that clients and inbound mail use - stays on the load balancer. SPF adds this address and keeps the existing - provider for as long as mail is still sent through it. Email signs with DKIM, - and DMARC aligns on the mailbox domain. -- **A Cilium egress-gateway policy** routes only the Email server's - internet-bound traffic through one gateway node, with the Floating IP as the - source. Other workloads keep their current egress. The policy covers IPv4 - destinations only, so a gateway without IPv6 stays selectable. -- **The gateway is one static worker**, selected by a single node label. - Hetzner routes the Floating IP only to the server it is assigned to, so - replies to outbound connections reach only that server. The address is - therefore assigned to the gateway worker. Cilium requires the address - on a network device, so every worker carries it on a dummy interface rather - than on its main interface. The node address the kubelet reports and the - address Cilium masquerades other traffic to must stay the server's own; the - implementing revision proves both, and pins the kubelet's node address if - needed. -- **No high availability.** If the gateway node is lost, Cilium drops the - matching traffic instead of sending it from another address, and Email's - queue retries the messages. Moving the gateway means removing the label from - the old worker, labelling another static worker and reassigning the address - through a reviewed main-only workflow. Delivery resumes only after exactly one - node carries the label. No in-cluster controller or cloud token is added. -- **Every start is gated.** Cilium applies the policy to a new pod after a - short delay, and until then its traffic leaves with the node's address. A - start-up check in the Email pod therefore waits until a fixed external - address-reflection endpoint sees the Floating IP as its source, and the - server starts only then. The check shares the pod's address, so the server - inherits the proven route. -- The address and host name stay in private configuration. Kubernetes - manifests read them from the platform's encrypted substitution variables, and - the Talos patch from a deploy-time secret, as the WireGuard key already does. - -## Order - -1. The maintainer creates the Floating IPv4. This is the only account action - and the only new cost. -2. A reviewed main-only workflow assigns it to the gateway worker, sets its - reverse DNS to the sending host name and reads both back. -3. A separate revision enables the egress gateway in Cilium, which restarts - the agents and operator. Existing egress must be unchanged afterwards. -4. Add the dummy-interface address to workers, label the gateway, and apply - the Email policy together with its start-up check. First confirm that - KSail's pinned Talos configuration library accepts the dummy-interface - document. Afterwards, confirm that every node's reported address and other - pods' egress source are unchanged, and that a restarted Email server starts - only after its check passes. -5. From the Email server's own network identity, repeat the no-message SMTP - check. Confirm that the external server sees the Floating IP as the source. -6. Publish the sending host's address record and the SPF, DKIM and DMARC records - through the DNS owner, after the maintainer approves them. MX still moves - only after migration and recovery are verified. - -## Alternatives - -- **A hosted relay:** rejected by the maintainer. -- **A dedicated mail node with its own address:** costs a whole server, which - is not authorized. -- **A controller that moves the address automatically:** adds a component and - an in-cluster cloud token. Static workers are rarely replaced, and Email's - queue tolerates a manual move. -- **The nodes' own addresses:** they change when nodes are replaced, so they - cannot be published in SPF or given stable reverse DNS. - -## Consequences - -The start-up gate couples one thing to the gateway: while the gateway is -down, a restarting Email server waits, and its inbound mail and mailbox access -wait with it. A running server keeps serving; a lost gateway only delays its -outbound mail. If the pinned Email server can hold just its outbound -queue until the check passes, that narrower gate replaces the start-up gate. -Inbound mail otherwise continues to use the existing load balancer. - -Egress gateway is incompatible with Cluster Mesh and CiliumEndpointSlice; -neither is enabled. Replies from the gateway are unencrypted only with XDP -acceleration, which is not enabled either. - -## Sources - -- [Cilium egress gateway](https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/) -- [Hetzner Floating IP configuration](https://docs.hetzner.com/cloud/floating-ips/persistent-configuration/) -- [Hetzner outbound mail policy](https://docs.hetzner.com/cloud/servers/faq/#why-can-i-not-send-any-mails-from-my-server) -- [Talos dummy link configuration](https://docs.siderolabs.com/talos/v1.13/reference/configuration/network/dummylinkconfig) From 3099abf8bd7985788e38acd645f41322bbc63726 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 07:10:55 +0100 Subject: [PATCH 7/9] docs(adr): say how Stalwart gets the SMTP username and reaches SES Address the review of the relay decision: the key ID is the SMTP username and travels with the password through one exact OpenBao key, copied by a platform-applied PushSecret rather than a dedicated identity. The relay step must also replace the Email server's all-egress deny with egress to DNS and the SES endpoint, since a deny overrides any allow. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-outbound-relay.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/adr/email-outbound-relay.md b/docs/adr/email-outbound-relay.md index a42020d974..54da8e2f44 100644 --- a/docs/adr/email-outbound-relay.md +++ b/docs/adr/email-outbound-relay.md @@ -26,10 +26,11 @@ are unchanged. with STARTTLS and SMTP authentication. - **One send-only credential.** An IAM user may only `ses:SendRawEmail` on that identity, with the sender limited to the mail domain. Crossplane creates its - access key, and the provider publishes the derived SES SMTP password. That - password is valid only for the provider's own region, which is why SES runs - in `eu-central-1`. A dedicated push identity stores it in OpenBao, and the - Email server reads it from there. Nobody reads the value. + access key and publishes the key ID, which is the SMTP username, and the + derived SES SMTP password. That password is valid only for the provider's + own region, which is why SES runs in `eu-central-1`. A platform-applied + PushSecret copies the username and password to one exact OpenBao key, and the + Email server reads both from there. Nobody reads the values. - **Ownership.** The identity and MAIL FROM domain grant no permission, so the `aws` tenant owns them. The SMTP user, its policy and its key stay platform-owned, like the EKS CI role, so the public tenant cannot widen them. @@ -48,7 +49,10 @@ are unchanged. 3. Create the send-only SMTP identity and store its credential in OpenBao. 4. Publish the DNS records once the identity's DKIM tokens exist. 5. Request production access. -6. Add Stalwart's relay route and its egress to the SES endpoint. +6. Add Stalwart's relay route. Its network policy now denies all egress, and a + deny overrides any allow, so this step replaces the deny with egress to DNS + and to the SES SMTP endpoint on port 587 only. That ends the server's + preparation-only network contract. 7. Send real mail to external mailboxes. Check that SPF, DKIM and DMARC pass, and that replies thread against the sender's Sent copy despite SES's Message-ID rewrite. If threading fails, report it before switching relay. From 3773e9860b9e0f7e27435bb50ac937582b5b4eaf Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 08:21:56 +0100 Subject: [PATCH 8/9] docs(adr): publish the SES records through external-dns Record the DNS method: the platform's external-dns publishes the DKIM CNAMEs and the MAIL FROM domain's MX and SPF records as DNS-only DNSEndpoints, which also requires it to manage MX and TXT records. SPF is checked on the MAIL FROM domain, so the mailbox domain's SPF record is unchanged, and the existing DMARC record keeps its owner. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-outbound-relay.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/adr/email-outbound-relay.md b/docs/adr/email-outbound-relay.md index 54da8e2f44..509257dc8c 100644 --- a/docs/adr/email-outbound-relay.md +++ b/docs/adr/email-outbound-relay.md @@ -34,9 +34,12 @@ are unchanged. - **Ownership.** The identity and MAIL FROM domain grant no permission, so the `aws` tenant owns them. The SMTP user, its policy and its key stay platform-owned, like the EKS CI role, so the public tenant cannot widen them. -- **DNS.** The three DKIM records, the MAIL FROM MX and SPF records, and DMARC - go through the DNS owner. SPF keeps the existing provider while mail is still - sent through it. +- **DNS.** The platform's external-dns publishes the three DKIM CNAMEs and + the MAIL FROM domain's MX and SPF records as DNS-only `DNSEndpoint` + resources. They are new names, so each keeps one owner; external-dns also + manages MX and TXT records for this. SPF is checked on the MAIL FROM domain, + so the mailbox domain's own SPF record, which still lists the existing + provider, is unchanged. The existing DMARC record keeps its current owner. - **Production access** is requested through a reviewed main-only workflow. Until SES grants it, its sandbox delivers only to verified recipients. - **Swappable.** Changing relay means changing Stalwart's relay route and its @@ -47,7 +50,8 @@ are unchanged. 1. Install the SES provider and activate the kinds (#6184). 2. Declare the identity and MAIL FROM domain (devantler-tech/aws#78). 3. Create the send-only SMTP identity and store its credential in OpenBao. -4. Publish the DNS records once the identity's DKIM tokens exist. +4. Publish the DNS records through external-dns once the identity's DKIM + tokens exist. 5. Request production access. 6. Add Stalwart's relay route. Its network policy now denies all egress, and a deny overrides any allow, so this step replaces the deny with egress to DNS From 6f180590e9b7b43525810c20d2cdb1754f5df27e Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 13 Nov 2026 08:56:31 +0100 Subject: [PATCH 9/9] docs(adr): accept the SES relay decision The maintainer chose the hosted relay and Amazon SES in eu-central-1 on 2026-11-13, and the DNS method is settled, so the record is accepted. Part of devantler-tech/platform#5481. Co-Authored-By: Claude Opus 5.5 --- docs/adr/email-outbound-relay.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adr/email-outbound-relay.md b/docs/adr/email-outbound-relay.md index 509257dc8c..cf6cbdfa10 100644 --- a/docs/adr/email-outbound-relay.md +++ b/docs/adr/email-outbound-relay.md @@ -1,6 +1,7 @@ # Email outbound relay -**Status:** Proposed +**Status:** Accepted. The maintainer chose a hosted relay, Amazon SES in +`eu-central-1`, on 2026-11-13. ## Context