IPC term buffer NUMA node

[chrisejones]

Hi, we have a multi-socket box and have two processes using Aeron IPC on a different numa node to where the media driver is running. The IPC term buffers are ending up on the media driver node according to /proc//numa_maps.

Do we need a separate media driver on the other socket to avoid this? Is there a way that we could customise this in the channel string maybe? So that the memory can be allocated on a specific NUMA node.

[mjpt777]

If you want to keep the mapped files on the same node then you need a media driver on that node. The media driver will be the first touch on the memory and thus on its node. From the Java driver we have no control. We may have options for the C driver if you would like us to look into that?

[chrisejones]

Yes potentially. We're not currently using the C media driver, but we could look to migrate.

From Java could you do something with hugetlbfs? Have different mounts for different numa nodes and then have a way to customise which is used for a particular channel?

I'll do a bit more investigation on our end to see what impact this is having and get back to you on whether we'd be interested in having this in the C media driver.

[mjpt777]

hugetlbfs may be an option for Linux. We can use some support time to look into this.

[chrisejones]

That would be great, thanks. We are using Linux.

[mjpt777]

On Linux you could try starting the media driver using numactl and putting the threads on on socket and the memory on another. The issue is not just the log buffer, you also need to consider the CnC file which has the counters for tracking progress.

numactl -N 1 -m 0 media-driver

This will result in cache traffic transiting the node interconnects as the media driver accesses memory.

An alternative solution would be to run a media driver focused on IPC support on the required node and set the threading mode to SHARED as the network side will be doing nothing.

[chrisejones]

Ah, I see. I didn't think about the CnC file. I think we'll do as you suggest and run a separate media driver. Thanks for your help.