Performance Tracker

[dragonJACson]

This issue would be used for tracking stride-perf performance between different versions, as a benchmark tool, we try to avoid performance degradation, so we track the performance after every important PR merged in.

[dragonJACson]

2025-06-15

Tools and commits

• stride: 3c11070
• perftest: linux-rdma/perftest@4d645d4

Server

fastfetch

~$ fastfetch
                             ....              lukedyue@node1
              .',:clooo:  .:looooo:.           --------------
           .;looooooooc  .oooooooooo'          OS: Ubuntu 24.04.1 LTS x86_64
        .;looooool:,''.  :ooooooooooc          Host: SYS-110P-WTR (0123456789)
       ;looool;.         'oooooooooo,          Kernel: Linux 6.8.0-51-generic
      ;clool'             .cooooooc.  ,,       Uptime: 16 mins
         ...                ......  .:oo,      Packages: 913 (dpkg)
  .;clol:,.                        .loooo'     Shell: bash 5.2.21
 :ooooooooo,                        'ooool     Terminal: /dev/pts/0
'ooooooooooo.                        loooo.    CPU: Intel(R) Xeon(R) Silver 4314 (32) @ 3.40 GHz
'ooooooooool                         coooo.    GPU: ASPEED Technology, Inc. ASPEED Graphics Family
 ,loooooooc.                        .loooo.    Memory: 3.47 GiB / 125.41 GiB (3%)
   .,;;;'.                          ;ooooc     Swap: 0 B / 8.00 GiB (0%)
       ...                         ,ooool.     Disk (/): 6.32 GiB / 62.45 GiB (10%) - ext3
    .cooooc.              ..',,'.  .cooo.      Local IP (ens1f0np0): 
      ;ooooo:.           ;oooooooc.  :l.       Locale: en_US.UTF-8
       .coooooc,..      coooooooooo.
         .:ooooooolc:. .ooooooooooo'
           .':loooooo;  ,oooooooooc
               ..';::c'  .;loooo:'

stride-probe

rocep81s0f0: ConnectX-6 Dx (PhysicalFunction, FW: 22.43.2566)
    Port 1: HighDataRate (50 Gbps), Width2X, total data rate: 100 Gbps
rocep81s0f1: ConnectX-6 Dx (PhysicalFunction, FW: 22.43.2566)
    Port 1: HighDataRate (50 Gbps), Width2X, total data rate: 100 Gbps
rocep138s0f0: ConnectX-6 Lx (PhysicalFunction, FW: 26.32.2004)
    Port 1: EnhancedDataRate (25 Gbps), Width1X, total data rate: 25 Gbps
rocep138s0f1: ConnectX-6 Lx (PhysicalFunction, FW: 26.32.2004)
    Port 1: QuadrupleDataRate (8 Gbps), Width4X, total data rate: 32 Gbps

NUMA Node

~$ cat /sys/class/infiniband/rocep81s0f0/device/numa_node
0

CPU governor

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Test

Commands & Results

1 QP, all message sizes

# server
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f0 -n 10000 -a

# client
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f1 -n 10000 -a 192.168.1.1

# results
------------------------------  RDMA Write Bandwidth Test  -------------------------------
Device         : rocep81s0f1
Transport      : InfiniBand
QP Count       : 1
Connection Type: RC
MTU            : 4096
GID Type       : RoceV2
Rx Depth       : 512
Tx Depth       : 128
Post List      : 1

----------------------------------  Connection Details  ----------------------------------
QP #00000000: (Local QPN: 0x01ab PSN: 0x33ba5a) -> (Remote QPN: 0x00ab PSN: 0x4cb988)
Local  GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0102
Remote GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0101

----------------------------------  Bandwidth Results  -----------------------------------
 Size (B)     | Iterations   | Avg BW (Gb/s)      | MsgRate (Mpps)     | Time
--------------┼--------------┼--------------------┼--------------------┼------------------
 2            | 10000        | 0.1075             | 6.7199             | 0.00
 4            | 10000        | 0.2195             | 6.8604             | 0.00
 8            | 10000        | 0.4354             | 6.8033             | 0.00
 16           | 10000        | 0.8742             | 6.8299             | 0.00
 32           | 10000        | 1.7566             | 6.8618             | 0.00
 64           | 10000        | 3.5088             | 6.8531             | 0.00
 128          | 10000        | 7.0188             | 6.8543             | 0.00
 256          | 10000        | 13.9935            | 6.8328             | 0.00
 512          | 10000        | 28.0462            | 6.8472             | 0.00
 1024         | 10000        | 55.6304            | 6.7908             | 0.00
 2048         | 10000        | 94.9164            | 5.7932             | 0.00
 4096         | 10000        | 97.4289            | 2.9733             | 0.00
 8192         | 10000        | 97.7128            | 1.4910             | 0.01
 16384        | 10000        | 97.8521            | 0.7466             | 0.01
 32768        | 10000        | 97.9230            | 0.3735             | 0.03
 65536        | 10000        | 97.9591            | 0.1868             | 0.05
 131072       | 10000        | 97.9739            | 0.0934             | 0.11
 262144       | 10000        | 97.9829            | 0.0467             | 0.21
 524288       | 10000        | 97.9876            | 0.0234             | 0.43
 1048576      | 10000        | 97.9900            | 0.0117             | 0.86
 2097152      | 10000        | 97.9910            | 0.0058             | 1.71
 4194304      | 10000        | 97.9914            | 0.0029             | 3.42
 8388608      | 10000        | 97.9917            | 0.0015             | 6.85
 16777216     | 10000        | 97.9918            | 0.0007             | 13.70
 33554432     | 10000        | 97.9922            | 0.0004             | 27.39
------------------------------------------------------------------------------------------

1 QP, all message sizes below 4096B, 16 Post List

# server
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f0 -n 1000000 -l 16 -a --max-msg-size 4096

# client
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f1 -n 1000000 -l 16 -a --max-msg-size 4096 192.168.1.1

# results
------------------------------  RDMA Write Bandwidth Test  -------------------------------
Device         : rocep81s0f1
Transport      : InfiniBand
QP Count       : 1
Connection Type: RC
MTU            : 4096
GID Type       : RoceV2
Rx Depth       : 512
Tx Depth       : 128
Post List      : 16

----------------------------------  Connection Details  ----------------------------------
QP #00000000: (Local QPN: 0x01ac PSN: 0x1e5d9e) -> (Remote QPN: 0x00ac PSN: 0x39626d)
Local  GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0102
Remote GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0101

----------------------------------  Bandwidth Results  -----------------------------------
 Size (B)     | Iterations   | Avg BW (Gb/s)      | MsgRate (Mpps)     | Time
--------------┼--------------┼--------------------┼--------------------┼------------------
 2            | 1000000      | 0.2079             | 12.9957            | 0.08
 4            | 1000000      | 0.4159             | 12.9971            | 0.08
 8            | 1000000      | 0.8285             | 12.9456            | 0.08
 16           | 1000000      | 1.6763             | 13.0964            | 0.08
 32           | 1000000      | 3.3650             | 13.1446            | 0.08
 64           | 1000000      | 6.7485             | 13.1807            | 0.08
 128          | 1000000      | 13.3337            | 13.0212            | 0.08
 256          | 1000000      | 26.0879            | 12.7382            | 0.08
 512          | 1000000      | 50.5058            | 12.3305            | 0.08
 1024         | 1000000      | 87.8533            | 10.7243            | 0.09
 2048         | 1000000      | 95.2908            | 5.8161             | 0.17
 4096         | 1000000      | 97.6171            | 2.9790             | 0.34
------------------------------------------------------------------------------------------

16 QP, all message sizes below 4MiB

# server
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f0 -n 10000 -q 16 -a --max-msg-size $((4*1024*1024))

# client
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f1 -n 10000 -q 16 -a --max-msg-size $((4*1024*1024)) 192.168.1.1

# results
------------------------------  RDMA Write Bandwidth Test  -------------------------------
Device         : rocep81s0f0
Transport      : InfiniBand
QP Count       : 16
Connection Type: RC
MTU            : 4096
GID Type       : RoceV2
Rx Depth       : 512
Tx Depth       : 128
Post List      : 1

----------------------------------  Connection Details  ----------------------------------
QP #00000000: (Local QPN: 0x00ad PSN: 0xf3bacc) -> (Remote QPN: 0x01ad PSN: 0xe0e4e8)
QP #00000001: (Local QPN: 0x00ae PSN: 0x1b2b1d) -> (Remote QPN: 0x01af PSN: 0xcfdb8e)
QP #00000002: (Local QPN: 0x00af PSN: 0x69dd51) -> (Remote QPN: 0x01b0 PSN: 0xb7f8b5)
QP #00000003: (Local QPN: 0x00b0 PSN: 0x1736b4) -> (Remote QPN: 0x01b1 PSN: 0x26c607)
QP #00000004: (Local QPN: 0x00b1 PSN: 0x512f7e) -> (Remote QPN: 0x01b2 PSN: 0x96fe7a)
QP #00000005: (Local QPN: 0x00b2 PSN: 0x8c7dac) -> (Remote QPN: 0x01b3 PSN: 0x0431a0)
QP #00000006: (Local QPN: 0x00b3 PSN: 0x0c0b18) -> (Remote QPN: 0x01b4 PSN: 0x5e7865)
QP #00000007: (Local QPN: 0x00b4 PSN: 0x7ebbac) -> (Remote QPN: 0x01b5 PSN: 0x2aec52)
QP #00000008: (Local QPN: 0x00b5 PSN: 0x2a07ac) -> (Remote QPN: 0x01b6 PSN: 0xedc26c)
QP #00000009: (Local QPN: 0x00b7 PSN: 0xb7a690) -> (Remote QPN: 0x01b7 PSN: 0xe2ac08)
QP #00000010: (Local QPN: 0x00b8 PSN: 0x7911d3) -> (Remote QPN: 0x01b8 PSN: 0x26a7fa)
QP #00000011: (Local QPN: 0x00b9 PSN: 0xf6f5e7) -> (Remote QPN: 0x01b9 PSN: 0x4f31d3)
QP #00000012: (Local QPN: 0x00ba PSN: 0xbf6179) -> (Remote QPN: 0x01ba PSN: 0x05bfde)
QP #00000013: (Local QPN: 0x00bb PSN: 0xd2dcc5) -> (Remote QPN: 0x01bb PSN: 0x92e92d)
QP #00000014: (Local QPN: 0x00bc PSN: 0x42803f) -> (Remote QPN: 0x01bc PSN: 0x5d47c4)
QP #00000015: (Local QPN: 0x00bd PSN: 0x589649) -> (Remote QPN: 0x01bd PSN: 0xb13597)
Local  GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0101
Remote GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0102

----------------------------------  Bandwidth Results  -----------------------------------
 Size (B)     | Iterations   | Avg BW (Gb/s)      | MsgRate (Mpps)     | Time
--------------┼--------------┼--------------------┼--------------------┼------------------
 2            | 160000       | 0.1078             | 6.7404             | 0.02
 4            | 160000       | 0.2160             | 6.7502             | 0.02
 8            | 160000       | 0.4328             | 6.7621             | 0.02
 16           | 160000       | 0.8675             | 6.7772             | 0.02
 32           | 160000       | 1.7328             | 6.7688             | 0.02
 64           | 160000       | 3.4703             | 6.7780             | 0.02
 128          | 160000       | 6.9254             | 6.7631             | 0.02
 256          | 160000       | 13.8143            | 6.7453             | 0.02
 512          | 160000       | 27.6159            | 6.7422             | 0.02
 1024         | 160000       | 54.9392            | 6.7064             | 0.02
 2048         | 160000       | 95.2796            | 5.8154             | 0.03
 4096         | 160000       | 97.6069            | 2.9787             | 0.05
 8192         | 160000       | 97.7997            | 1.4923             | 0.11
 16384        | 160000       | 97.8960            | 0.7469             | 0.21
 32768        | 160000       | 97.9443            | 0.3736             | 0.43
 65536        | 160000       | 97.9684            | 0.1869             | 0.86
 131072       | 160000       | 97.9805            | 0.0934             | 1.71
 262144       | 160000       | 97.9865            | 0.0467             | 3.42
 524288       | 160000       | 97.9895            | 0.0234             | 6.85
 1048576      | 160000       | 97.9911            | 0.0117             | 13.70
 2097152      | 160000       | 97.9918            | 0.0058             | 27.39
 4194304      | 160000       | 97.9922            | 0.0029             | 54.79
------------------------------------------------------------------------------------------

16 QP, all message sizes below 4096B, 16 Post List

# server
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f0 -n 1000000 -l 16 -q 16 -a --max-msg-size 4096

# client
numactl -N 0 -m 0 taskset -c 10 ./target/release/stride-perf write bw -x 3 -d rocep81s0f1 -n 1000000 -l 16 -q 16 -a --max-msg-size 4096 192.168.1.1

# results
------------------------------  RDMA Write Bandwidth Test  -------------------------------
Device         : rocep81s0f1
Transport      : InfiniBand
QP Count       : 16
Connection Type: RC
MTU            : 4096
GID Type       : RoceV2
Rx Depth       : 512
Tx Depth       : 128
Post List      : 16

----------------------------------  Connection Details  ----------------------------------
QP #00000000: (Local QPN: 0x01be PSN: 0x8d1c58) -> (Remote QPN: 0x00be PSN: 0xb699a6)
QP #00000001: (Local QPN: 0x01bf PSN: 0x8b8e2b) -> (Remote QPN: 0x00bf PSN: 0x18ad14)
QP #00000002: (Local QPN: 0x01c0 PSN: 0x75c7e5) -> (Remote QPN: 0x00c0 PSN: 0xd9028d)
QP #00000003: (Local QPN: 0x01c1 PSN: 0xcc9de4) -> (Remote QPN: 0x00c1 PSN: 0xc15c69)
QP #00000004: (Local QPN: 0x01c2 PSN: 0x2e21af) -> (Remote QPN: 0x00c2 PSN: 0x1f7b07)
QP #00000005: (Local QPN: 0x01c3 PSN: 0xa66c58) -> (Remote QPN: 0x00c3 PSN: 0x61d99c)
QP #00000006: (Local QPN: 0x01c4 PSN: 0xb93e91) -> (Remote QPN: 0x00c4 PSN: 0xfcab33)
QP #00000007: (Local QPN: 0x01c5 PSN: 0x55232b) -> (Remote QPN: 0x00c5 PSN: 0x172875)
QP #00000008: (Local QPN: 0x01c6 PSN: 0xee01be) -> (Remote QPN: 0x00c6 PSN: 0xedf935)
QP #00000009: (Local QPN: 0x01c7 PSN: 0x9fd0d7) -> (Remote QPN: 0x00c7 PSN: 0xaaa774)
QP #00000010: (Local QPN: 0x01c8 PSN: 0x020fad) -> (Remote QPN: 0x00c8 PSN: 0x0d5cd6)
QP #00000011: (Local QPN: 0x01c9 PSN: 0x373e5a) -> (Remote QPN: 0x00c9 PSN: 0x907cdc)
QP #00000012: (Local QPN: 0x01ca PSN: 0x500764) -> (Remote QPN: 0x00ca PSN: 0xeb7aa0)
QP #00000013: (Local QPN: 0x01cb PSN: 0x44765c) -> (Remote QPN: 0x00cb PSN: 0xae84d2)
QP #00000014: (Local QPN: 0x01cc PSN: 0x9ae955) -> (Remote QPN: 0x00cc PSN: 0x7ffa4f)
QP #00000015: (Local QPN: 0x01cd PSN: 0x190f67) -> (Remote QPN: 0x00cd PSN: 0x810109)
Local  GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0102
Remote GID: GID: 0000:0000:0000:0000:0000:ffff:c0a8:0101

----------------------------------  Bandwidth Results  -----------------------------------
 Size (B)     | Iterations   | Avg BW (Gb/s)      | MsgRate (Mpps)     | Time
--------------┼--------------┼--------------------┼--------------------┼------------------
 2            | 16000000     | 0.2831             | 17.6958            | 0.90
 4            | 16000000     | 0.5961             | 18.6278            | 0.86
 8            | 16000000     | 1.1973             | 18.7081            | 0.86
 16           | 16000000     | 2.4131             | 18.8524            | 0.85
 32           | 16000000     | 4.7425             | 18.5252            | 0.86
 64           | 16000000     | 9.6053             | 18.7603            | 0.85
 128          | 16000000     | 18.6978            | 18.2596            | 0.88
 256          | 16000000     | 32.8492            | 16.0397            | 1.00
 512          | 16000000     | 44.4591            | 10.8543            | 1.47
 1024         | 16000000     | 88.7554            | 10.8344            | 1.48
 2048         | 16000000     | 95.2535            | 5.8138             | 2.75
 4096         | 16000000     | 97.5680            | 2.9775             | 5.37
------------------------------------------------------------------------------------------

1 QP, all message sizes (ib_write_bw)

# server
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f0 -n 10000 -a -Q 1 --cqe_poll 32 --report_gbits

# client
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f1 -n 10000 -a -Q 1 --cqe_poll 32 --report_gbits 192.168.1.1

# results
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : rocep81s0f1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 CQ Moderation   : 1
 CQE Poll Batch  : 32
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01cf PSN 0x7b37ee RKey 0x1bde00 VAddr 0x0078dbc7e01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 remote address: LID 0000 QPN 0x00cf PSN 0x8cdf8f RKey 0x181e00 VAddr 0x007d96e0e01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          10000            0.11               0.11                 6.731194
 4          10000            0.22               0.22                 6.754713
 8          10000            0.44               0.43                 6.786078
 16         10000            0.87               0.87                 6.774950
 32         10000            1.74               1.74                 6.782216
 64         10000            3.47               3.47                 6.771496
 128        10000            6.96               6.94                 6.781560
 256        10000            13.88              13.87                6.770829
 512        10000            27.77              27.74                6.771459
 1024       10000            55.38              55.35                6.756582
 2048       10000            94.98              94.91                5.792914
 4096       10000            97.45              97.40                2.972452
 8192       10000            97.69              97.67                1.490343
 16384      10000            97.81              97.81                0.746219
 32768      10000            97.89              97.88                0.373375
 65536      10000            97.91              97.91                0.186749
 131072     10000            97.93              97.93                0.093390
 262144     10000            97.94              97.94                0.046699
 524288     10000            97.94              97.94                0.023351
 1048576    10000            97.94              97.94                0.011676
 2097152    10000            97.94              97.94                0.005838
 4194304    10000            97.94              97.94                0.002919
 8388608    10000            97.94              97.94                0.001459
---------------------------------------------------------------------------------------

1 QP, all message sizes below 4096B, 16 Post List (ib_write_bw)

# server
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f0 -n 1000000 -a -Q 1 -l 16 --cqe_poll 32 --report_gbits

# client
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f1 -n 1000000 -a -Q 1 -l 16 --cqe_poll 32 --report_gbits 192.168.1.1

# results
---------------------------------------------------------------------------------------
                    RDMA_Write Post List BW Test
 Dual-port       : OFF          Device         : rocep81s0f1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 Post List       : 16
 CQ Moderation   : 1
 CQE Poll Batch  : 32
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01f1 PSN 0xb464ea RKey 0x1bde00 VAddr 0x00739719001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 remote address: LID 0000 QPN 0x00f1 PSN 0x901f6 RKey 0x181e00 VAddr 0x007033ede01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000000          0.00               0.20                 12.804399
 4          1000000          0.00               0.41                 12.837346
 8          1000000          0.00               0.82                 12.793313
 16         1000000          0.00               1.64                 12.818663
 32         1000000          0.00               3.27                 12.759767
 64         1000000          0.00               6.52                 12.729472
 128        1000000          0.00               12.76                12.458546
 256        1000000          0.00               25.22                12.315481
 512        1000000          0.00               49.37                12.054299
 1024       1000000          0.00               87.89                10.728671
 2048       1000000          0.00               95.21                5.811421
 4096       1000000          0.00               97.57                2.977576
---------------------------------------------------------------------------------------

16 QP, all message sizes (ib_write_bw)

# server
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f0 -n 10000 -a -Q 1 -q 16 --cqe_poll 32 --report_gbits

# client
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f1 -n 10000 -a -Q 1 -q 16 --cqe_poll 32 --report_gbits 192.168.1.1

# results
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : rocep81s0f1
 Number of qps   : 16           Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 CQ Moderation   : 1
 CQE Poll Batch  : 32
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01d1 PSN 0x7c215b RKey 0x1bdf00 VAddr 0x0077ba48401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d2 PSN 0x7d5fa9 RKey 0x1bdf00 VAddr 0x0077ba48c01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d3 PSN 0x91123f RKey 0x1bdf00 VAddr 0x0077ba49401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d4 PSN 0x63fe22 RKey 0x1bdf00 VAddr 0x0077ba49c01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d5 PSN 0xaf768d RKey 0x1bdf00 VAddr 0x0077ba4a401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d6 PSN 0xcc7d6b RKey 0x1bdf00 VAddr 0x0077ba4ac01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d7 PSN 0x4d49ee RKey 0x1bdf00 VAddr 0x0077ba4b401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d8 PSN 0x818a4f RKey 0x1bdf00 VAddr 0x0077ba4bc01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01d9 PSN 0xdb747 RKey 0x1bdf00 VAddr 0x0077ba4c401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01da PSN 0xbb50bd RKey 0x1bdf00 VAddr 0x0077ba4cc01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01db PSN 0x98641d RKey 0x1bdf00 VAddr 0x0077ba4d401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01dc PSN 0xcb6eec RKey 0x1bdf00 VAddr 0x0077ba4dc01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01dd PSN 0xdf3068 RKey 0x1bdf00 VAddr 0x0077ba4e401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01de PSN 0xb0f5f5 RKey 0x1bdf00 VAddr 0x0077ba4ec01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01df PSN 0xae5c0f RKey 0x1bdf00 VAddr 0x0077ba4f401000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e0 PSN 0x107c27 RKey 0x1bdf00 VAddr 0x0077ba4fc01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 remote address: LID 0000 QPN 0x00d1 PSN 0xc74c5d RKey 0x181f00 VAddr 0x00758e5b201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d2 PSN 0xfa7b83 RKey 0x181f00 VAddr 0x00758e5ba01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d3 PSN 0xd3ca11 RKey 0x181f00 VAddr 0x00758e5c201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d4 PSN 0x8aee8c RKey 0x181f00 VAddr 0x00758e5ca01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d5 PSN 0xfb0faf RKey 0x181f00 VAddr 0x00758e5d201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d6 PSN 0xe93ce5 RKey 0x181f00 VAddr 0x00758e5da01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d7 PSN 0xa6ce0 RKey 0x181f00 VAddr 0x00758e5e201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d8 PSN 0x86d759 RKey 0x181f00 VAddr 0x00758e5ea01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00d9 PSN 0xbe8089 RKey 0x181f00 VAddr 0x00758e5f201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00da PSN 0xaa4dd7 RKey 0x181f00 VAddr 0x00758e5fa01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00db PSN 0xb7842f RKey 0x181f00 VAddr 0x00758e60201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00dc PSN 0x744296 RKey 0x181f00 VAddr 0x00758e60a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00dd PSN 0xa40bca RKey 0x181f00 VAddr 0x00758e61201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00de PSN 0xf46aaf RKey 0x181f00 VAddr 0x00758e61a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00df PSN 0xbe2b41 RKey 0x181f00 VAddr 0x00758e62201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e0 PSN 0xaa071 RKey 0x181f00 VAddr 0x00758e62a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          160000           0.11               0.11                 6.614253
 4          160000           0.21               0.21                 6.613067
 8          160000           0.42               0.42                 6.613931
 16         160000           0.85               0.85                 6.611999
 32         160000           1.69               1.69                 6.606919
 64         160000           3.39               3.39                 6.614080
 128        160000           6.77               6.77                 6.610295
 256        160000           13.54              13.52                6.603907
 512        160000           27.01              26.98                6.587738
 1024       160000           53.72              53.66                6.550689
 2048       160000           95.44              95.23                5.812533
 4096       160000           97.57              97.56                2.977244
 8192       160000           97.75              97.75                1.491553
 16384      160000           97.88              97.85                0.746511
 32768      160000           97.91              97.90                0.373444
 65536      160000           97.92              97.92                0.186767
 131072     160000           97.93              97.93                0.093395
 262144     160000           97.94              97.94                0.046700
 524288     160000           97.94              97.94                0.023351
 1048576    160000           97.94              97.94                0.011676
 2097152    160000           97.94              97.94                0.005838
 4194304    160000           97.94              97.94                0.002919
 8388608    160000           97.94              97.94                0.001459
---------------------------------------------------------------------------------------

16 QP, all message sizes below 4096B, 16 Post List (ib_write_bw)

# server
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f0 -n 1000000 -a -Q 1 -q 16 -l 16 --cqe_poll 32 --report_gbits

# client
numactl -N 0 -m 0 taskset -c 10 ./ib_write_bw -x 3 -d rocep81s0f1 -n 1000000 -a -Q 1 -q 16 -l 16 --cqe_poll 32 --report_gbits 192.168.1.1

# results
---------------------------------------------------------------------------------------
                    RDMA_Write Post List BW Test
 Dual-port       : OFF          Device         : rocep81s0f1
 Number of qps   : 16           Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 Post List       : 16
 CQ Moderation   : 1
 CQE Poll Batch  : 32
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01e1 PSN 0x3fd282 RKey 0x1bdf00 VAddr 0x00715bf2801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e2 PSN 0xecd744 RKey 0x1bdf00 VAddr 0x00715bf3001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e3 PSN 0x60413e RKey 0x1bdf00 VAddr 0x00715bf3801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e4 PSN 0xe81ab5 RKey 0x1bdf00 VAddr 0x00715bf4001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e5 PSN 0x1fb324 RKey 0x1bdf00 VAddr 0x00715bf4801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e6 PSN 0x39f3b6 RKey 0x1bdf00 VAddr 0x00715bf5001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e7 PSN 0x5409dd RKey 0x1bdf00 VAddr 0x00715bf5801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e8 PSN 0x25fd12 RKey 0x1bdf00 VAddr 0x00715bf6001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01e9 PSN 0xf1c64e RKey 0x1bdf00 VAddr 0x00715bf6801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01ea PSN 0xca00b8 RKey 0x1bdf00 VAddr 0x00715bf7001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01eb PSN 0x58d3fc RKey 0x1bdf00 VAddr 0x00715bf7801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01ec PSN 0xd16adf RKey 0x1bdf00 VAddr 0x00715bf8001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01ed PSN 0xcbc8df RKey 0x1bdf00 VAddr 0x00715bf8801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01ee PSN 0x1ccaa0 RKey 0x1bdf00 VAddr 0x00715bf9001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01ef PSN 0x4d8ade RKey 0x1bdf00 VAddr 0x00715bf9801000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 local address: LID 0000 QPN 0x01f0 PSN 0xb9644a RKey 0x1bdf00 VAddr 0x00715bfa001000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:02
 remote address: LID 0000 QPN 0x00e1 PSN 0xcbfaab RKey 0x181f00 VAddr 0x00747935201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e2 PSN 0xf139f0 RKey 0x181f00 VAddr 0x00747935a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e3 PSN 0xfe5944 RKey 0x181f00 VAddr 0x00747936201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e4 PSN 0xdf460c RKey 0x181f00 VAddr 0x00747936a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e5 PSN 0xf7c466 RKey 0x181f00 VAddr 0x00747937201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e6 PSN 0x1c259a RKey 0x181f00 VAddr 0x00747937a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e7 PSN 0x2facbb RKey 0x181f00 VAddr 0x00747938201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e8 PSN 0x2f5662 RKey 0x181f00 VAddr 0x00747938a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00e9 PSN 0x743a28 RKey 0x181f00 VAddr 0x00747939201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00ea PSN 0xecbd53 RKey 0x181f00 VAddr 0x00747939a01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00eb PSN 0xf0d731 RKey 0x181f00 VAddr 0x0074793a201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00ec PSN 0xbed9a7 RKey 0x181f00 VAddr 0x0074793aa01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00ed PSN 0x5530d1 RKey 0x181f00 VAddr 0x0074793b201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00ee PSN 0xa30573 RKey 0x181f00 VAddr 0x0074793ba01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00ef PSN 0x489beb RKey 0x181f00 VAddr 0x0074793c201000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x00f0 PSN 0x63c809 RKey 0x181f00 VAddr 0x0074793ca01000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          16000000         0.00               0.39                 24.420571
 4          16000000         0.00               0.79                 24.621000
 8          16000000         0.00               1.57                 24.576349
 16         16000000         0.00               3.15                 24.604192
 32         16000000         0.00               6.32                 24.676312
 64         16000000         0.00               12.65                24.702644
 128        16000000         0.00               25.15                24.558690
 256        16000000         0.00               50.64                24.725404
 512        16000000         0.00               83.39                20.358109
 1024       16000000         0.00               90.94                11.101355
 2048       16000000         0.00               95.26                5.814009
 4096       16000000         0.00               97.57                2.977622
---------------------------------------------------------------------------------------