k0sctl cannot set controller endpoint ip address for each individual HA controller differently

[chaserhkj]

My nodes are interconnected with a VPN that has private IPs in 192.168.194.* range, the host I am running k0sctl on is in the same VPN, my configuration looks like this:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: primary-cluster
  user: hkj
spec:
  hosts:
  - openSSH:
      address: 192.168.194.1
      user: root
      port: 22
      keyPath: ./secrets/k0sctl.deploy.key
      configPath: /dev/null
    role: controller+worker
    noTaints: true
    useExistingK0s: true
    privateInterface: tap0
    privateAddress: 192.168.194.1
  - openSSH:
      address: 192.168.194.2
      user: root
      port: 22
      keyPath: ./secrets/k0sctl.deploy.key
      configPath: /dev/null
    role: controller+worker
    noTaints: true
    useExistingK0s: true
    privateInterface: tap0
    privateAddress: 192.168.194.2
  - openSSH:
      address: 192.168.194.3
      user: root
      port: 22
      keyPath: ./secrets/k0sctl.deploy.key
      configPath: /dev/null
    role: controller+worker
    noTaints: true
    useExistingK0s: true
    privateInterface: tap0
    privateAddress: 192.168.194.3
  k0s:
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: primary
      spec:
        api:
          k0sApiPort: 9443
          port: 6443
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        konnectivity:
          adminPort: 8133
          agentPort: 8132
        network:
          kubeProxy:
            disabled: false
            mode: iptables
          kuberouter:
            autoMTU: true
            mtu: 0
            peerRouterASNs: ""
            peerRouterIPs: ""
          podCIDR: 10.244.0.0/16
          provider: kuberouter
          serviceCIDR: 10.96.0.0/12
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy
          controlPlaneLoadBalancing:
            enabled: true
            type: Keepalived
            keepalived:
              vrrpInstances:
              - virtualIPs: ["192.168.198.255/21"]
                interface: tap0
                authPass: <redacted>
        podSecurityPolicy:
          defaultPolicy: 00-k0s-privileged
        storage:
          type: etcd
        telemetry:
          enabled: false
  options:
    wait:
      enabled: true
    drain:
      enabled: true
      gracePeriod: 2m0s
      timeout: 5m0s
      force: true
      ignoreDaemonSets: true
      deleteEmptyDirData: true
      podSelector: ""
      skipWaitForDeleteTimeout: 0s
    concurrency:
      limit: 30
      workerDisruptionPercent: 10
      uploads: 5
    evictTaint:
      enabled: false
      taint: k0sctl.k0sproject.io/evict=true
      effect: NoExecute
      controllerWorkers: false

And when I run 'k0sctl apply --config config.yaml`:

$ k0sctl apply --config config.yaml
----------------------------------------------------------------------------------------------------

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl (devel) Copyright 2025, k0sctl authors.
INFO ==> Running phase: Set k0s version
INFO Looking up latest stable k0s version
INFO Using k0s version v1.34.3+k0s.0
INFO ==> Running phase: Connect to hosts
ControlSocket /home/hkj/.ssh/ctrl-b0bad2c1c3758b29ea6964b95527fefc790a6d61 already exists, disabling multiplexing
ControlSocket /home/hkj/.ssh/ctrl-0181e9b08cf056ec6c3d9b42b196fc7f6796901c already exists, disabling multiplexing
ControlSocket /home/hkj/.ssh/ctrl-825a11a131323b6ecf70f70f7753fdd20e2332df already exists, disabling multiplexing
INFO [OpenSSH] root@192.168.194.3:22: connected
INFO [OpenSSH] root@192.168.194.1:22: connected
INFO [OpenSSH] root@192.168.194.2:22: connected
INFO ==> Running phase: Detect host operating systems
INFO [OpenSSH] root@192.168.194.1:22: is running Arch Linux
INFO [OpenSSH] root@192.168.194.2:22: is running Arch Linux
INFO [OpenSSH] root@192.168.194.3:22: is running Arch Linux
INFO ==> Running phase: Acquire exclusive host lock
INFO ==> Running phase: Prepare hosts
INFO ==> Running phase: Gather host facts
INFO [OpenSSH] root@192.168.194.3:22: using miku as hostname
INFO [OpenSSH] root@192.168.194.1:22: using hatsune as hostname
INFO [OpenSSH] root@192.168.194.2:22: using kagumine as hostname
INFO ==> Running phase: Validate hosts
WARN failed to walk k0s.tmp.* files in /usr/local/bin/k0s: stat /usr/local/bin/k0s.tmp.*: file does not exist
WARN failed to walk k0s.tmp.* files in /usr/local/bin/k0s: stat /usr/local/bin/k0s.tmp.*: file does not exist
WARN failed to walk k0s.tmp.* files in /usr/local/bin/k0s: stat /usr/local/bin/k0s.tmp.*: file does not exist
INFO validating clock skew
INFO ==> Running phase: Gather k0s facts
INFO [OpenSSH] root@192.168.194.3:22: found existing configuration
INFO [OpenSSH] root@192.168.194.1:22: found existing configuration
INFO [OpenSSH] root@192.168.194.2:22: found existing configuration
INFO [OpenSSH] root@192.168.194.1:22: is running k0s controller+worker version v1.34.3+k0s.0
INFO [OpenSSH] root@192.168.194.1:22: listing etcd members
INFO [OpenSSH] root@192.168.194.1:22: useExistingK0s=true, reusing existing k0s v1.34.3+k0s.0
INFO [OpenSSH] root@192.168.194.2:22: useExistingK0s=true, reusing existing k0s v1.34.3+k0s.0
INFO [OpenSSH] root@192.168.194.3:22: useExistingK0s=true, reusing existing k0s v1.34.3+k0s.0
INFO ==> Running phase: Validate facts
INFO ==> Running phase: Validate etcd members
INFO [OpenSSH] root@192.168.194.1:22: validating configuration
INFO [OpenSSH] root@192.168.194.2:22: validating configuration
INFO [OpenSSH] root@192.168.194.3:22: validating configuration
INFO ==> Running phase: Configure k0s
INFO [OpenSSH] root@192.168.194.3:22: installing new configuration
INFO [OpenSSH] root@192.168.194.1:22: installing new configuration
INFO [OpenSSH] root@192.168.194.1:22: restarting k0s service
INFO [OpenSSH] root@192.168.194.2:22: installing new configuration
INFO [OpenSSH] root@192.168.194.1:22: waiting for k0s service to start
INFO ==> Running phase: Run Before Apply Hooks
INFO ==> Running phase: Install controllers
INFO [OpenSSH] root@192.168.194.1:22: generate join token for [OpenSSH] root@192.168.194.2:22
INFO [OpenSSH] root@192.168.194.1:22: generate join token for [OpenSSH] root@192.168.194.3:22
INFO [OpenSSH] root@192.168.194.3:22: validating api connection to https://192.168.2.51:9443
INFO [OpenSSH] root@192.168.194.2:22: validating api connection to https://192.168.2.51:9443
INFO * Running clean-up for phase: Acquire exclusive host lock
INFO * Running clean-up for phase: Install controllers
INFO [OpenSSH] root@192.168.194.3:22: cleaning up
INFO [OpenSSH] root@192.168.194.2:22: cleaning up
INFO ==> Apply failed
FATA apply failed - log file saved to /home/hkj/.cache/k0sctl/k0sctl.log: failed on 2 hosts:
 - [OpenSSH] root@192.168.194.3:22: failed to connect from controller to kubernetes api - check networking: context deadline exceeded
command failed: client exec: command failed: command wait: exit status 28
 - [OpenSSH] root@192.168.194.2:22: failed to connect from controller to kubernetes api - check networking: context deadline exceeded
command failed: client exec: command failed: command wait: exit status 28

For some reason k0sctl just keeps trying to validate API connection using the public (or precisely, internet-facing) IP address (on-site LAN address 192.168.2.51) of my 192.168.194.1 node. But these three nodes are not on the same site thus the other two just don't have access to that IP.

[chaserhkj]

Umm seems I'm being dumb and I just need to set

k0s:
  config:
    spec:
      api:
        address: 192.168.198.255

The address is the VIP I used for VRRP.

I feel the doc may still need to be improved a bit on emphasizing the default behavior of selecting the first public-facing IP as the API endpoint. But I'll just close this issue for now.

[chaserhkj]

I think this still creates a relevant problem: when I set the API address as the VIP address above, the embedded CPLB and NLLB of k0s seem to be functioning weirdly which as a result breaks all konnectivity connections.

I believe this is because CPLB and NLLB collects upstream endpoints from the same field, so setting it to just the VIP would 1. just use the leader api server for k8s api. 2. essentially create a loop situation with konnectivity and breaks it completely.

The true solution to this might be to set the address field differently for different HA controllers, but there seems to be none of that kind of functionality in k0sctl hooks might be needed to do this.

[chaserhkj]

For reference for anyone encountering similar problems with this kind of setup, I installed yq on all the controller nodes (kustomize should work as well but I want some oneliners), and did this hook to make it work:

      hooks:
        install:
          before:
            - yq -iY '.spec.api.address = "192.168.194.3"' /etc/k0s/k0s.yaml

(change the ip address to match each individual HA controller's VPN address)

I also completely disabled konnectivity since my VPN is L2 unrestricted, there is no need to use konnectivity, so I am not sure this still works with konnectivity. But this setup works very good for me now.