I/O performance test—InfiniBand Card

ib_read_bw 

Performs a read bandwidth diagnostic. Issued on the Linux InfiniBand host.

Syntax

ib_read_bw [-p TCP_port][-d device][-i IB_port][-m mtu][-o num][-s size][-a][-t depth][-n iters][-b][-V][-e][-F][IP_address]

where:

  • TCP_port is the TCP port.
  • device is the InfiniBand device.
  • IB_port is the InfiniBand port.
  • mtu is the size of the MTU.
  • num is the outstanding reads/atom.
  • size is the size of the messages.
  • depth is the size of the TX queue.
  • iters is the number of message exchanges.
  • IP_address is the IP address of the remote node host.
Description

This InfiniBand software command performs a bandwidth diagnostic between two nodes in the InfiniBand fabric. The command is dependent upon the Internet Protocol, so the InfiniBand fabric must be configured with Internet Protocol over InfiniBand (IPoIB). The command is a client-server, in that a remote node is configured as a server, while a local node performs as a client.

The command is first run locally on the server. The command is then run again locally on the client, pointing to the IP address of the server. The diagnostic checks the bandwidth of the data transfer from the server to the client. The connection uses the Reliable Datagram transport.


Note – This command only functions if your InfiniBand fabric is configured with IPoIB.


Options

The following table describes the options to the ib_read_bw command and their purposes:

Option
Purpose
-p
Uses the TCP port for initial synchronization.
-d
Uses the InfiniBand device.
-i
Uses the InfiniBand port.
-m
Sets the size of the MTU.
-o
Sets the number of outstanding reads/atom.
-s
Sets the size of the message to exchange.
-a
Runs all sizes, from 2 to 223.
-t
Sets the size of the TX queue.
-n
Performs iters message exchanges.
-b
Measures bidirectional bandwidth.
-V
Displays the version information.
-e
Sleeps on CQ events.
-F
Does not fail, even if the cpufreq_ondemand module is loaded.

 

Example

The following example shows how to run a diagnostic between a local node client and a remote node server with the ib_read_bw command. First configure the remote node server:

# ib_read_bw
------------------------------------------------------------------
                         RDMA_Read BW Test
Connection type : RC
     local address:  LID 0x01, QPN 0x990406, PSN 0xd5c3ba RKey 0x3682003a VAddr 0x002ad63cb81000
     remote address: LID 0x05, QPN 0x220406, PSN 0xa67d82, RKey 0x2af2043f VAddr 0x002b364ab80000
Mtu : 1024
#

Note – The output is not displayed until the local node client issues the respective command.


Then run the command on the local node client:

# ib_read_bw 192.168.200.100
------------------------------------------------------------------
                         RDMA_Read BW Test
Connection type : RC
     local address:  LID 0x05, QPN 0x220406, PSN 0xa67d82 RKey 0x2af2043f VAddr 0x002b364ab80000
     remote address: LID 0x01, QPN 0x990406, PSN 0xd5c3ba, RKey 0x3682003a VAddr 0x002ad63cb81000
Mtu : 1024
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec] 
 65536        1000             763.28                763.26
------------------------------------------------------------------
#

 

ib_write_bw 

Performs a write bandwidth diagnostic. Issued on the Linux InfiniBand host.

Syntax

ib_write_bw [-p TCP_port][-d device][-i IB_port][-c RC|UC|UD][-m mtu] [-g posts][-q num][-s size][-a][-t depth][-n iters][-I size][-b][-V] [-N][-F][IP_address]

where:

  • TCP_port is the TCP port.
  • device is the InfiniBand device.
  • IB_port is the InfiniBand port.
  • mtu is the size of the MTU.
  • posts is the number of posts for each queue pair.
  • num is the quantity of queue pairs.
  • size is the size of the messages.
  • depth is the size of the TX queue.
  • iters is the number of message exchanges.
  • IP_address is the IP address of the remote node host.
Description

This InfiniBand software command performs a bandwidth diagnostic between two nodes in the InfiniBand fabric. The command is dependent upon the Internet Protocol, so the InfiniBand fabric must be configured with Internet Protocol over InfiniBand (IPoIB). The command is a client-server, in that a remote node is configured as a server, while a local node performs as a client.

The command is first run locally on the server. The command is then run again locally on the client, pointing to the IP address of the server. The diagnostic checks the bandwidth of the data transfer from the client to the server. The connection uses the Reliable Datagram transport. Optionally, you can set the connection between the nodes to use Reliable Connection, Unreliable Connection, or Unreliable Datagram transport.


Note – This command only functions if your InfiniBand fabric is configured with IPoIB.


Options

The following table describes the options to the ib_write_bw command and their purposes:

Option
Purpose
-p
Uses the TCP port for initial synchronization.
-d
Uses the InfiniBand device.
-i
Uses the InfiniBand port.
-c
Sets the connection type.
-m
Sets the size of the MTU.
-g
Specifies the number of posts for each queue pair in the chain.
-q
Sets the number of queue pairs.
-s
Sets the size of the message to exchange.
-a
Runs all sizes, from 2 to 223.
-t
Sets the size of the TX queue.
-n
Performs iters message exchanges.
-I
Sets the maximum message size to be sent in inline mode.
-b
Measures bidirectional bandwidth.
-V
Displays the version information.
-N
Cancels the peak bandwidth calculation.
-F
Does not fail, even if the cpufreq_ondemand module is loaded.
Example

The following example shows how to run a diagnostic between a local node client and a remote node server with the ib_write_bw command. First configure the remote node server:

# ib_write_bw
------------------------------------------------------------------
                         RDMA_Write BW Test
Number of qp’s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 400 bytes message
     local address:  LID 0x01, QPN 0x9d0406, PSN 0x30ea7b RKey 0x369c003a VAddr 0x002ada954eb000
     remote address: LID 0x05, QPN 0x260406, PSN 0x5e303b, RKey 0x2b0c043f VAddr 0x002b94b3356000
Mtu : 1024
#

Note – The output is not displayed until the local node client issues the respective command.


Then run the command on the local node client:

# ib_write_bw 192.168.200.100
------------------------------------------------------------------
                         RDMA_Write BW Test
Number of qp’s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 400 bytes message
     local address:  LID 0x05, QPN 0x260406, PSN 0x5e303b RKey 0x2b0c043f VAddr 0x002b94b3356000
     remote address: LID 0x01, QPN 0x9d0406, PSN 0x30ea7b, RKey 0x369c003a VAddr 0x002ada954eb000
Mtu : 1024
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec] 
     65536        5000             790.41                790.41
------------------------------------------------------------------
#