How to Benchmark the WAN connection in a Replication Environment

Answer

To measure NPS -> PTS connection

On Master Puredata Appliance as nz:
nzrepltestpts

On Subordinate PureData as nz:
nzrepltestpts

The speed should be apx 250MB/s.

A slow NPS-> PTS speed will result in many issues and needs to be corrected.

To measure PTS->PTS WAN bandwidth:
Follow steps in
http://www-01.ibm.com/support/docview.wss?uid=swg21576189

Using the WAN address during client selection


To measure PTS->PTS write speed using standard network transfer:

create a large file for use in data movement:

#fallocate /var/nzrepl/file.out -l 50G

This file should ideally be 2x larger than system memory to prevent caching throwing off results

On subordinate PTS:

#nc -k -l -v 1234 > /var/nzrepl/testfiles

On master PTS:

#dd if=/var/nzrepl/file.out bs=64K | nc pts2 1234

DD test will display transfer rate in MB/s upon completion

After running this test, it is important to clear buffers before testing further.

#echo 3 > /proc/sys/vm/drop_caches

To measure PTS-> PTS write speed using PTS protocol

On both master and subordinate PTS as nz, add a new folder on the linux system under replication control

#mkdir /var/nzrepl/networktest/
#ptsconfigure -add networktest/

On Master PTS, move the file created earlier to transfer the file

# mv /var/nzrepl/file.out /var/nzrepl/networktest/

You must use mv, not cp

Take note of the timestamp
#ls -lah --time=ctime --full-time file.out

Example:
#-rw-rw-r-- 1 nz nz 1.0G 2015-03-03 14:37:44.026388406 -0500 file.out

Wait for file to transfer to subordinate PTS

Take note of the timestamp
#ls -lah --time=ctime --full-time file.out

Example
#-rw------- 1 nz nz 1.0G 2015-03-03 14:38:06.410005417 -0500 file.out

In this case, a 1G file transferred in 22.4 seconds, or 44.6 MB/s.

It is important that NTP is used, since a large drift will throw off results.

Network admin should examine the WAN link for errors, but troubleshooting this connection is outside the scope of PureData support.

A helper script is provided to scan for all files transferred under /var/nzrepl on subordinate, to capture the rate at which they transferred.

Run the following 1 time on subordinate, pumping resulting log file into a database or spreadsheet to analyze for trend analysis.

for f in `find /var/nzrepl/ -type f `; do stat $f -c '%s,%Y,%Z' | awk -F "," ' $1>1000000 { print ($1/1000000) /($3-$2)}' >>file.log 2> /dev/null ; done