NVIDIA GPU Power Capping

The ESSL and NVIDIA library subroutines are highly optimized and for some problem sizes your application may exceed the SW Power cap for one or more of the GPUs. If this happens your performance will be degraded because the frequency of the corresponding GPU clock will be reduced because the GPU is consuming too much power.

You can confirm that this is happening by using nvidia-smi to monitor the GPUs while your application is running.

 nvidia-smi dmon

If you wish to adjust the Power Cap Limit follow these steps:

Determine the current, default and maximum power limit as follows:
```
nvidia-smi -q | grep 'Power Limit'
```
Ensure that persistence mode is being used.
Increase the SW Power Cap limit for all GPUs as follows, where xxx is the desired value in watts:
```
 nvidia-smi -pl xxx
```

Note: You must increase the power limit and set persistence each time the server is booted.

For additional information, see the following URL:

http://international.download.nvidia.com/tesla/pdf/gpu-boost-tesla-k40-app-note.pdf