PowerAI tuning recommendations

Find recommended settings for optimal deep learning performance on the S822LC and AC922 for High-Performance Computing.

Enable Performance Governor

sudo yum install kernel-tools
sudo cpupower -c all frequency-set -g performance

Enable GPU persistence mode

sudo systemctl enable nvidia-persistenced
sudo systemctl start nvidia-persistenced

Set GPU memory and graphics clocks

S822LC with NVIDIA Tesla P100, set clocks to maximum

sudo nvidia-smi -ac 715,1480

AC922 with NVIDIA Tesla V100, set clocks to NVIDIA defaults

sudo nvidia-smi -rac

For TensorFlow, set the SMT mode

S822LC with NVIDIA Tesla P100, set SMT=2.

sudo ppc64_cpu --smt=2

AC922 with NVIDIA Tesla V100, set SMT based on DDL usage:

sudo ppc64_cpu --smt=4    # for TensorFlow WITHOUT DDL
sudo ppc64_cpu --smt=2    # for TensorFlow WITH DDL