Fix for NetworkManager Infiniband Errors

[tessalatedSapote]

Hi,
Sorry for the persistence here, but I'd like to use NetworkManager to configure Infiniband on Clear Linux.
I can configure Infiniband networking mostly without error via iproute2, but configuration via NetworkManager errors out as described in a pretty buried issue and a while ago on the forums.

I pretty quickly found the error was the result of just a couple kernel configs, CONFIG_INFINIBAND_IPOIB_CM and CONFIG_INFINIBAND_IPOIB_DEBUG need to be set to y in order for Infiniband interfaces to be able to be set to "Connected Mode". NetworkManager errors out when it's unable to do this (which it always? tries to do) and of course iproute2 cannot set the interface mode.

I made a pull request directly with the kernel package, but it's been a while and I suspect that may not have been the best best place raise attention to this issue and solution.

I built and installed the package myself with the aforementioned configs enabled and had no issue doing so. Configuring Infiniband via NetworkManager with the custom kernel package installed worked perfectly.

I would be shocked if there were any general performance implications tied to these, as I think they're rather localized to Infiniband. Also, all major distributions I looked at had these enabled (e.g. RHEL, Debian, and Arch etc), so it looks like having these configs enabled is pretty standard. I should also note that configuring Infiniband via NetworkManager on those three distributions works without error.

[bwarden]

I can enable CONFIG_INFINIBAND_IPOIB_CM. We generally avoid setting debug options without a very specific reason, though. I couldn't find anything that suggested NetworkManager actually needed that.

[fenrus75]

(filing a ticket here is the right thing to do; we generally do not get notifications for pull requests to our package repos)

…

On Sun, Nov 20, 2022 at 11:19 PM tessalatedSapote ***@***.***> wrote: Hi, Sorry for the persistence here, but I'd like to use NetworkManager to configure Infiniband on Clear Linux. I can configure Infiniband networking mostly without error via iproute2, but configuration via NetworkManager errors out as described in a pretty buried issue <#1095> and a while ago on the forums <https://community.clearlinux.org/t/trouble-setting-up-infiniband-through-network-manager/1266> . I pretty quickly found the error was the result of just a couple kernel configs, CONFIG_INFINIBAND_IPOIB_CM and CONFIG_INFINIBAND_IPOIB_DEBUG need to be set to y in order for Infiniband interfaces to be able to be set to "Connected Mode". NetworkManager errors out when it's unable to do this (which it always? tries to do) and of course iproute2 cannot set the interface mode. I made a pull request <clearlinux-pkgs/linux#25 (comment)> directly with the kernel package, but it's been a while and I suspect that may not have been the best best place raise attention to this issue and solution. I built and installed the package myself with the aforementioned configs enabled and had no issue doing so. Configuring Infiniband via NetworkManager with the custom kernel package installed worked perfectly. I would be shocked if there were any general performance implications tied to these, as I think they're rather localized to Infiniband. Also, all major distributions I looked at have these enabled (e.g. RHEL <https://git.centos.org/rpms/kernel/blob/c8s/f/SOURCES/kernel-x86_64.config#_3394>, Debian <https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/config/config#L1614>, and Arch <https://github.com/archlinux/svntogit-packages/blob/packages/linux/trunk/config#L8280> etc), so it looks like having these configs enabled is pretty standard. I should also note that configuring Infiniband via NetworkManager on those three distributions works without error. — Reply to this email directly, view it on GitHub <#2770>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ54FLSTWJKKYSPDSQ43FTWJMPBHANCNFSM6AAAAAASGKIAXA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tessalatedSapote]

I very much understand not wanting to enable debug options unless necessary, but I've seen some strange and inconsistent behavior without it enabled. I also never found any documentation that explicitly stated NM needed the debug option enabled, but through far too much trial and error, I'm pretty much convinced the safe option would be to just enable CONFIG_INFINIBAND_IPOIB_DEBUG.

NM (usually) works smoothly with infiniband when only CONFIG_INFINIBAND_IPOIB_CM is enabled on 6.0.9-1209.
However, when I was testing on 6.0.7-1207, I had no such luck and infiniband connections would not get set to connected mode without CONFIG_INFINIBAND_IPOIB_DEBUG being enabled.

When on 6.0.9-1209 with both CONFIG_INFINIBAND_IPOIB_DEBUG and CONFIG_INFINIBAND_IPOIB_CM enabled, I never hit any mode setting issues. When the debug parameter was not enabled I did encounter a rare Error: Connection activation failed: The device could not be readied for configuration or some other error.

I have no real explanation for the inconsistency, I thought I saw a slight association with whether the node or infiniband subnet manager booted first, but due to inconsistent behavior I really can't be sure nor do I have any idea why that would have an effect.

With the debug config enabled (of course along with CONFIG_INFINIBAND_IPOIB_CM ), I always got infiniband net sysfs entries for mode and link_mode, without it on some boots they would be present and others not. Again, this behavior was pretty weird and when looking through sources I did not see any reasonable explanation. There could be quirk with my hardware, though I doubt it as I've tested across multiple nodes and systems.

As you can see in the official documentation, "the performance impact of this option is negligible", and from what I've experienced from a stability standpoint I would suggest to just enable it. Also, given my varying experiences between kernel versions, I fear that coasting on the enabled CONFIG_INFINIBAND_IPOIB_CM mostly working is just an invitation for a regression.

[bwarden]

I'll enable CONFIG_INFINIBAND_IPOIB_DEBUG. The inconsistent behavior suggests race conditions at play, but this appears well-contained in the InfiniBand driver. Resulting bugs should go to InfiniBand subsystem maintainers.

[bwarden]

Closing as the fix should be available in latest release now.

[valentijnvenus]

@tessalatedSapote may you share your NetworkManager config now Infiniband connected-mode can be set on Clear Linux please?

[tessalatedSapote]

Thanks @bwarden for the fixes. Sorry for not closing this earlier but everything is now working as expected. I definitely think there's a race condition somewhere, though finding where and on what layer will be pretty involved. I will probably keep poking around a bit and see if I can get some better data to make a report.

@valentijnvenus, there really isn't anything to my NetworkManager configs, I'm just following the Red Hat documentation for setting up an ipoib connection. And to be honest I'm not really even "using" ipoib, I just need an infiniband interface with an ip so it can do its little dance for iSER.

Though for posterity here we go:

nmcli connection add type infiniband con-name mlx5_ib0 ifname ibp182s0 transport-mode Connected mtu 65520 ipv6.method disabled ipv4.method manual ipv4.addresses 192.168.2.2/24

The actual config file looks like this:

[connection]
id=mlx5_ib0
uuid=dead-beef
type=infiniband
interface-name=ibp182s0
timestamp=1670100000

[infiniband]
mtu=65520
transport-mode=connected

[ipv4]
address1=192.168.2.2/24
method=manual

[ipv6]
addr-gen-mode=stable-privacy
method=disabled

[proxy]

Finally, here's the iproute2 command I was using before I got nmcli working:
ip address add 192.168.2.2/24 dev ibp182s0