performance/multi-node: use fi_inject_write.

[e-harvey]

• Use fi_write_inject until the inject threshold is exceeded.
• This is to compare the performance of fi_inject_write to fi_write.

With the changes introduced in this PR we get:

at https://github.com/ofi-cray/libfabric-cray/tree/6cdc8c283feecc2b3b881666598edcf96dd56d5e:

bin$ aprun --network -n2 -t960 -N1 -d1 ./rdm_one_sided -t1
1 threads
# Libfabric Bandwidth Test 
# Size        Bandwidth (MB/s)        Latency (us)        Min Lat (us)        Max Lat (us)
1                         0.24                4.00                4.00                4.00
2                         0.48                3.94                3.94                3.94
4                         0.98                3.90                3.90                3.90
8                         1.94                3.93                3.93                3.93
16                        3.86                3.96                3.96                3.96
32                        7.77                3.93                3.93                3.93
64                       15.49                3.94                3.94                3.94
128                     164.02                0.74                0.74                0.74
256                     326.45                0.75                0.75                0.75
512                     637.01                0.77                0.77                0.77
1024                   1197.02                0.82                0.82                0.82
2048                   1835.43                1.06                1.06                1.06
4096                   2505.19                1.56                1.56                1.56
8192                   5972.43                1.31                1.31                1.31
16384                  7291.29                2.14                2.14                2.14
32768                  8238.93                3.79                3.79                3.79
65536                  8819.31                7.09                7.09                7.09
131072                 9120.97               13.70               13.70               13.70
262144                 9306.11               26.86               26.86               26.86
524288                 9385.54               53.27               53.27               53.27
1048576                9447.40              105.85              105.85              105.85
2097152                9464.41              211.32              211.32              211.32
4194304                9411.90              424.99              424.99              424.99

and at https://github.com/ofi-cray/libfabric-cray/tree/2b63528292d4ead86c120f134edfc6ccb25f8369:

bin$ aprun --network -n2 -t960 -N1 -d1 ./rdm_one_sided -t1
1 threads
# Libfabric Bandwidth Test 
# Size        Bandwidth (MB/s)        Latency (us)        Min Lat (us)        Max Lat (us)
1                         0.82                1.16                1.16                1.16
2                         0.76                2.51                2.51                2.51
4                         4.48                0.85                0.85                0.85
8                        10.52                0.73                0.73                0.73
16                       20.30                0.75                0.75                0.75
32                       36.63                0.83                0.83                0.83
64                       80.39                0.76                0.76                0.76
128                     167.08                0.73                0.73                0.73
256                     332.01                0.74                0.74                0.74
512                     646.42                0.76                0.76                0.76
1024                   1213.19                0.80                0.80                0.80
2048                   1855.98                1.05                1.05                1.05
4096                   2527.70                1.55                1.55                1.55
8192                   5931.69                1.32                1.32                1.32
16384                  7280.67                2.15                2.15                2.15
32768                  8230.45                3.80                3.80                3.80
65536                  8804.75                7.10                7.10                7.10
131072                 9099.18               13.74               13.74               13.74
262144                 9301.24               26.88               26.88               26.88
524288                 9400.84               53.19               53.19               53.19
1048576                9407.68              106.30              106.30              106.30
2097152                9448.38              211.68              211.68              211.68
4194304                9426.98              424.31              424.31              424.31

Signed-off-by: Evan Harvey eharvey@lanl.gov

[jswaro]

motivation for this PR? Why is this being done?

[e-harvey]

motivation for this PR? Why is this being done?

Mainly because @hppritcha asked me to. Also, to compare the performance of fi_write to fi_write_inject.

[jswaro]

Mainly because @hppritcha asked me to. Also, to compare the performance of fi_write to fi_write_inject.

You need to amend your commit and PR descriptions to reflect that.

[sungeunchoi]

This PR should change to add the option to using fi_write_inject. The default behavior should be as it is now.

[sungeunchoi]

Why -j? Seems like -i.

[e-harvey]

Why -j? Seems like -i.

-i is taken, i can swap the flags if you want.

[sungeunchoi]

Oh yeah. I'd use -n.

[e-harvey]

Oh yeah. I'd use -n.

Did you mean -i?