Introducing IPv6 on existing Cluster

[pschichtel]

Hi there! This is more of a discussion/question, since I'm pretty sure this is not officially supported, neither here nor upstream. I still want to try though. I have an existing 3 node (controller+worker) bare-metal cluster on 1.35.1. This is a home lab, so I'm willing to toy around and potentially break things, even though I'd prefer not to break it.

The cluster is currently on IPv4-only since back when I originally set the cluster up, kube-router didn't support IPv6/Dual-Stack.

Just simply enabling dual stack and settings the cidrs is clearly not enough.

Doing that makes the kube-controller-manage fail with:

failed to mark cidr[10.0.0.0/24] at idx [0] as occupied for node: server: cidr 10.0.0.0/24 is out the range of cluster cidr 1111:1111:1111:1111::1:0/112

From my research that might be because the node objects don't have an IP in the new IPv6 cluster cidr. Unfortunately the IP can't just be added to an existing node.

Can I just drain and delete a node and have k0s register it again?

Any input on this? Thanks in advance!

[juanluisvaladas]

Hi,

This error is very suspicoius and makes us think you changed the primary address family (if this happened this was most certainly an accidental change). Could you please provide:

1- The full k0s configuration of the control plane node (one node should be enough. Feel free to redact stuff but if it's a hostname we need to know if it resolves to IPv4 and/or IPv6 and we need to know if IP addresses are either IPv4 or IPv6(
2- ps auxww | grep k0s in a control plane node.
3- ps auxww | grep -e k0s -e kube-router in a worker node.

Regarding if this is supported or not, kubernetes itself supports it and I don't see why it wouldn't be possible in kube-router, but it's not something that we actively test.

Regarding your actual question:

Can I just drain and delete a node and have k0s register it again?

This should be fine if the problem is that, but we haven't actually tested it. But since the error is coming from kube-controller manager I don't think that's the issue.

[pschichtel]

I added the primary address family explicitly as IPv4 to the config. From what was documented, the default value should have been IPv4 anyway, that's why I thought adding IPv4 explicitly shouldn't change a thing.

I've commented out the primary IPv6 changes and as I said I also added the primaryAddressFamily: IPv4 line. The nodes already have IPv6 addresses on their primary interfaces, but non of the hostnames resolve to IPv6 addresses currently.

Details

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - openSSH:
      address: server1.example.org
      user: root
      port: 22
      keyPath: null
    role: controller+worker
    noTaints: true
    installFlags:
    - --profile=enable-swap
    - --enable-metrics-scraper
  - openSSH:
      address: server2.example.org
      user: root
      port: 22
      keyPath: null
    role: controller+worker
    noTaints: true
    installFlags:
    - --profile=enable-swap
    - --enable-metrics-scraper
  - openSSH:
      address: server3.example.org
      user: root
      port: 22
      keyPath: null
    role: controller+worker
    noTaints: true
    installFlags:
    - --profile=enable-swap
    - --enable-metrics-scraper
  k0s:
    version: v1.35.1+k0s.0
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s
      spec:
        api:
          k0sApiPort: 9443
          port: 6443
          sans:
          - server1.example.org
          - server2.example.org
          - 10.1.1.2
          - 10.1.1.3
          - 127.0.0.1
        extensions:
          storage:
            create_default_storage_class: false
            type: external_storage
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        # controllerManager:
        #   extraArgs:
        #     node-cidr-mask-size-ipv6: "120"
        network:
          provider: kuberouter
          calico: null
          kubeProxy:
            mode: iptables
          kuberouter:
            autoMTU: true
            mtu: 0
            peerRouterASNs: ""
            peerRouterIPs: ""
          primaryAddressFamily: IPv4
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
          # dualStack:
          #   enabled: true
          #   IPv6podCIDR:     1111:1111:1111:1111:0000:0000:0001:0000/112
          #   IPv6serviceCIDR: 1111:1111:1111:1111:0000:0000:0002:0000/108
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy
        storage:
          etcd:
            externalCluster: null
          type: etcd
        workerProfiles:
        - name: enable-swap
          values:
            memorySwap:
              swapBehavior: LimitedSwap
            featureGates:
              NodeSwap: true
            imageMaximumGCAge: 30m

[pschichtel]

Oh also: I reverted the IPv6 options but kept the primaryAddressFamily: IPv4 and that works fine currently.

[juanluisvaladas]

What about keeping the IPv6 configuration with primaryAddressFamily: IPv4 ? How does that work?

[pschichtel]

that was the version that produced the error. I then reverted the IPv6 configuration, but kept primaryAddressFamily: IPv4. It's running like that currently.

[juanluisvaladas]

Sorry I deleted my previous comment while you were answering because I realized I had misunderstood your previous the message, I wasn't expect you to have read it that quickly.

Can you enable the IPv6 configuration with primaryAddressFamily: IPv4 defined explicitly and that still throws the same error?

[pschichtel]

The explicit primaryAddressFamily: IPv4 was part of the configuration that produced the error.

[pschichtel]

I think I'll build a virtualized setup and try this, before trying this on my actual lab.

[pschichtel]

I'm struggling with the setup in the VMs for some reason. I'm getting one node up, but not the second. Still investigating.

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days

[pschichtel]

I'm still working on this, just a little tight on time currently.