Latency not expected for eager zcopy

[haipeng31]

I do the following test case: client send reqeust to server, and server send reponse back to client.
request: with 6 buffers to send.
buffer0: 16byte
buffer1: 100byte
buffer2-5: 64 byte per buffer (total 64 * 4 = 256 byte)

response: with 2 buffer to send
buffer0: 16 byte
buffer1: 16 byte

The I run test the upper message with 3 different mode: eager(bcopy), eager(zcopy), rndv, using ucp_tag_send_nb where data type is IOV and got the following performance result, including latency(depth=1, start from send request till receive reponse) and message cnt per sec(depth = 64) (to make the problem more easy, the response is all sent in eager(bcopy) mode).

mode	latency	message cnt per sec
eager(bcopy)	8us	450000
eager(zcopy)	13us	380000
rndv	15us	90000

there are two points confuse me:

1. why eager(zcpy) latency is 5 us more than eager(bcopy), I guess it may come from one extra ibv_send because am_zcopy has max iov_cnt = 3 limit per ibv_send and I have 6 buffer to send, so it need two ibv_send, but this extra ibv_send may only need 800 + 1 * 371 ns = 1171ns = 1.1us, where does the extra 4us come from, is it from the pgtable_lookup? pgtablet_lookup may cost 180ns per buffer, so it cost about 180ns * 6 = 1080ns = 1 us，sum them only got 2.1 us, much less then 5us.
2. why rndv has only 90000 message cnt per sec， it's much less than eager(bcopy) which is 450000。can you give some advices to figure out how these is happended?

ucx version: https://github.com/yosefe/ucx/releases/tag/v1.9.0-pre38
env:
UCX_MAX_EAGER_LANES=2
UCX_TLS=rc_x
UCX_IB_SEG_SIZE=2k
UCX_RC_RX_QUEUE_LEN=1024
UCX_RC_MAX_RD_ATOMIC=16
UCX_RC_ROCE_PATH_FACTOR=2
UCX_SOCKADDR_CM_ENABLE=y
UCX_RC_MAX_GET_ZCOPY=32k
UCX_RC_TX_NUM_GET_BYTES=10240k

[changchengx]

@haipeng31

1. Please share your test code
2. Could you use typical value to run the test? If you're testing buffer with16byte ~ 64 bytes, we can use inline data.

[haipeng31]

eager(bcopy)
client: UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k
server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k

eager(zcopy)
client: UCX_ZCOPY_THRESH = 256 UCX_RNDV_THRESH=8k
server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k

eager(rndv)
client: UCX_ZCOPY_THRESH = 128 UCX_RNDV_THRESH=256
server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k

[changchengx]

@haipeng31
Update test result in local side. https://github.com/changchengx/ucx/tree/tbdc_latency_pre38_iov_six
Base configuration:

export UCX_MAX_EAGER_LANES=2
export UCX_TLS=rc_x
export UCX_IB_SEG_SIZE=2k
export UCX_RC_RX_QUEUE_LEN=1024
export UCX_RC_MAX_RD_ATOMIC=16
export UCX_RC_ROCE_PATH_FACTOR=2
export UCX_SOCKADDR_CM_ENABLE=y
export UCX_IB_TRAFFIC_CLASS=160
export UCX_RC_MAX_GET_ZCOPY=32k
export UCX_RC_TX_NUM_GET_BYTES=10240k
export UCX_IB_FORK_INIT=n
export UCX_IB_RCACHE_MAX_REGIONS=11000
export UCX_IB_RCACHE_MAX_SIZE=512m
export UCX_IB_MLX5_DEVX_OBJECTS=""

config	latency(us)	CMD
client: UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k	4.19	client: io_demo -d 3072 -k 512 -w 1 -i 0 1.1.25.1 server: io_demo -d 3072 -k 512
client: UCX_ZCOPY_THRESH = 256 UCX_RNDV_THRESH=8k server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k	5.38	client: io_demo -d 3072 -k 512 -w 1 -i 0 1.1.25.1 server: io_demo -d 3072 -k 512
client: UCX_ZCOPY_THRESH = 128 UCX_RNDV_THRESH=256 server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k	7.36	client: io_demo -d 3072 -k 512 -w 1 -i 0 1.1.25.1 server: io_demo -d 3072 -k 512

[changchengx]

@haipeng31

1. For Eager/IOV/bcopy (4.19us)
   Because total data length is 372 < zcopy_thresh, so it’ll use bcopy_single ( ucp_tag_eager_bcopy_single ) to collect( ucp_tag_pack_eager_only_dt ) the iov data segments into one WR sge buffer to send it.

2. For Eager/IOV/zcopy (5.38us)
   Every data segment is too small and it’ll use zcopy_multi (ucp_do_am_zcopy_multi]) to send data. In this way, it needs to construct the iov. And use two send WRs(2 x 3 sge) to send data to server node.
   Compared with 1st case, this case avoids copying buffer. however, it increases the time to construct iov and use one extra WR to send data to server node.

3. RNDV/AM/IOV/bcopy (7.36us)
   It’ll use RNDV/AM protocol( ucp_rndv_progress_am_bcopy ) since it uses IOV operation. Because data length is 372 < AM.max_bcopy, so it’ll use ucp_do_am_bcopy_single to send data. When using RDNV protocol, it needs to send the meta data (rkey etc) first, this could result in extra send WRs than EAGER. In RNDV/AM mode, it still uses bcopy_single at last in this case. It doesn’t have advantage than 1st & 2nd.

[haipeng31]

@changchengx thanks for your answer for the first question, it helps a lot. How about the second question, it is confusing.

[changchengx]

@haipeng31 Pls refer to RNDV/AM/IOV/bcopy (7.36us).
As being said in RNDV/AM/IOV/bcopy, it will use ucp_do_am_bcopy_single. To compare BW between eager/iov/bcopy and rndv/iov/am/bcopy in this case, I created another branch to measure the bandwidth: https://github.com/changchengx/ucx/tree/tbdc_bw_pre38_iov_single
Base configuration:

export UCX_MAX_EAGER_LANES=2
export UCX_TLS=rc_x
export UCX_IB_SEG_SIZE=2k
export UCX_RC_RX_QUEUE_LEN=1024
export UCX_RC_MAX_RD_ATOMIC=16
export UCX_RC_ROCE_PATH_FACTOR=2
export UCX_SOCKADDR_CM_ENABLE=y
export UCX_IB_TRAFFIC_CLASS=160
export UCX_RC_MAX_GET_ZCOPY=32k
export UCX_RC_TX_NUM_GET_BYTES=10240k
export UCX_IB_FORK_INIT=n
export UCX_IB_RCACHE_MAX_REGIONS=11000
export UCX_IB_RCACHE_MAX_SIZE=512m
export UCX_IB_MLX5_DEVX_OBJECTS=""

config	BW(MB/s)	CMD
client: UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k	73.58	client: io_demo -d 372 -k 372 -w 32 -i 0 1.1.25.1 server: io_demo -d 372 -k 372
client: UCX_ZCOPY_THRESH = 128 UCX_RNDV_THRESH=256 server:UCX_ZCOPY_THRESH = 4k UCX_RNDV_THRESH=8k	32.54	client: io_demo -d 372 -k 372 -w 32 -i 0 1.1.25.1 server: io_demo -d 372 -k 372

Note: This branch doesn't follow the same logic as what you've described in this Q&A. In the above branch, client uses IOV data type to send message and data to server, then server use contig data type to receive message and reply message to client under IOV data type. At last, client will use contig to receive the replied message from server. It also could be used to compare BW between eager/iov/bcopy and rndv/iov/am/bcopy. If you're satisfied with this model, please help change the io_demo to be used to measure BW.

[changchengx]

@haipeng31
When sending small data segments with IOV, it's better to use eager/bcopy according to the above analysis.

[changchengx]

@haipeng31
You can use the above branches and methods to measure the latency and BW. If you have other concern, please supply your code to do further analysis.

[changchengx]

@haipeng31
Have your questoins been answered yet?

[changchengx]

@haipeng31 Any feedback?