Usable with OpenStack with Cilium tunnel routing mode?

[Nils98Ar]

Is there any reason why only gcp, aws and azure are supported?

One difference I can imagine is that in OpenStack environments the default tunnel/encapsulation routing mode of cilium is used. With VXLAN/Geneve tunnels between the nodes.

[Nils98Ar]

And do these issues also apply to kubenetmon?

• [Feature] Support Cilium - add parsing of conntrack information kubecost/features-bugs#101
• [Feature] Support Cilium - add parsing of conntrack information kubecost/features-bugs#101 (comment)

[andreev-io]

Hi @Nils98Ar! We haven't tested kubenetmon with Cilium tunnel routing, but I suspect it won't work out of the box. If you want to make a contribution, it should be fairly simple to add another source of connection information (eBPF or otherwise) on top of conntrack, and the rest of annotation / storage logic will be handled by kubenetmon.

[Nils98Ar]

@andreev-io Thank you for your response! So Cilium in kube-proxy-less mode won't work as well similiar to the kubecost issue?

[andreev-io]

@Nils98Ar We have tested kubenetmon with kube-proxy and kube-proxy-less Cilium modes, but in both cases we used native routing, not tunnel routing. I suspect it won't work with tunnel routing; but it should work with native routing regardless of your kube-proxy setup.

My mental model is that you will have reasonably accurate data in conntrack as long as routing is done by the kernel; it doesn't matter whether NAT and other functions are done in eBPF or iptables. Of course there will be differences, but they're generally inconsequential.

[Nils98Ar]

Okay, so at least this (quote from the kubecost issue) does not seem to be true for kubenetmon?

We are using Cilium in kube-proxy-less mode (no kube-proxy). Routing is done via ebpf host routing. This means that the conntrack information is not available any longer in the Kubernetes node's conntrack system.

As a result of our switch to Cilium, network costs in Kubecost are not attributed correctly any longer.

Are there any other adjustments needed to allow a „generic“ cloud provider? I think currently it‘s only possible to choose between aws, gcp and azure?

[andreev-io]

@Nils98Ar I believe that comment you are referring to conflates two things:

1. Enabling/disabling kube-proxy. In our testing, this has no effect on kubenetmon.
2. Using eBPF host and/or tunnel routing vs native routing. We only tested the native routing mode (normal "kernel" routing), and I suspect that kubenetmon won't work when routing is done via eBPF.

Are there any other adjustments needed to allow a „generic“ cloud provider? I think currently it‘s only possible to choose between aws, gcp and azure?

kubenetmon doesn't make assumptions on which cloud you use (if any). There is some cloud-specific logic to annotate publicly known cloud ranges as such (N.B. someone seems to be working on adding custom IP ranges support too) and some unit tests are inspired by individual clouds, but I think from first principles kubenetmon should work on any random cloud provider.

Have you tried it yet?

[Nils98Ar]

I haven't tried it yet. I wanted to find out if it should work first ;)

My understanding is that kubeProxyReplacement requires eBPF host routing (regardless of whether native or tunnel routing is used). Is this not the case?

[andreev-io]

@Nils98Ar In the environments where we tested kubenetmon, we have legacy host routing configured regardless of whether kube-proxy is enabled or not.

[andreev-io]

@Nils98Ar By the way, supporting eBPF routing while still relying on conntrack is probably relatively simple if you get conntrack info from inside the container network namespace. I haven't experimented with it, though.

[Nils98Ar]

Sounds promising. Then the "the conntrack information is not available any longer in the Kubernetes node's conntrack system" could be the crucial point?

But currently you are collecting the info in the default netns?

[andreev-io]

But currently you are collecting the info in the default netns?

Yeah

[Nils98Ar]

Is there any reason why collecting it inside the pod networks netns shouldn't be default? Assuming that this makes it usage for eBPF host routing...