Add all-to-all benchmark

[pentschev]

To run the benchmark in single-node:

$ python benchmark_multiple_processes_all_to_all.py
[('10.33.225.163', 49583) -> 57555] Transferred bytes: 30.00 MiB, average bandwidth: 1.27 GiB/s, median bandwidth: 1.27 GiB/s
[('10.33.225.163', 57555) -> 49583] Transferred bytes: 30.00 MiB, average bandwidth: 1.00 GiB/s, median bandwidth: 1.00 GiB/s

To run each process separately, allowing multi-node as well:

# Monitor process, used to synchronize workers only
# In this case it will wait for 2 worker processes to connect
$ python benchmark_multiple_processes_all_to_all.py --monitor --listen-interface enp1s0f0 --port 12345 --num-workers 2 --communication-lib ucx
Monitor listening at 10.33.227.163:12345

# Worker 1
$ python benchmark_multiple_processes_all_to_all.py --worker --listen-interface enp1s0f0 --endpoints-per-worker 2 --communication-lib ucx --monitor-address 10.33.227.163:12345
[10.33.227.163:58306 -> 10.33.227.163:56515] Transferred bytes: 60.00 MiB, average bandwidth: 354.44 MiB/s, median bandwidth: 354.44 MiB/s

# Worker 2
$ python benchmark_multiple_processes_all_to_all.py --worker --listen-interface enp1s0f0 --endpoints-per-worker 2 --communication-lib ucx --monitor-address 10.33.227.163:12345
[10.33.227.163:56515 -> 10.33.227.163:58306] Transferred bytes: 60.00 MiB, average bandwidth: 356.47 MiB/s, median bandwidth: 356.47 MiB/s

[quasiben]

cc @jakirkham @gjoseph92 you both may find this PR interesting for running networking experiments which compare Tornado, UCX, and Asyncio

[pentschev]

rerun tests

[jakirkham]

It might be interesting to try with uvloop as well. Since the event loop is largely handled by libuv in C, would expect that performs better than asyncio alone

[pentschev]

Thanks for the suggestion @jakirkham , I added that now.

[pentschev]

rerun tests

[pentschev]

rerun tests

[mrocklin]

I will be curious to see the results that come out of this. If anyone has anything preliminary that they want to share I highly encourage that :)

[jakirkham]

Do we want this in tests/ or would it make sense to include in benchmarks/?

[pentschev]

I'm still thinking about it. I don't want it in tests, but I want a test there (for the UCX part only). However, a lot of the code is going to be shared and we don't have a good place currently where that common code would be visible to both. I'm not even sure the non-UCX code should live in this repo as we'll soon upstream it to OpenUCX, so it doesn't really make sense to have non-UCX code there. I'm still thinking of an appropriate place for this, if you have any suggestions, please let me know.

[jakirkham]

Yeah that makes sense. Will think about it as well.