Power9 Performance Best Practices

Steps

• Ensure that firmware is current
  Fix Central provides latest updates. Install the latest version of FW950 for the ALL Power9 systems.
  Use the Fix Level Recommendation Tool (FLRT) tool to obtain the recommended levels.
  NOTE: Ensure that required HMC level is installed when updating firmware.
   
• NX GZIP
  To take advantage of NX GZIP H/W accelerator on Power9 and onwards processor-based systems.
  Fix Central provides latest updates. Install the latest version of FW950 for the ALL Power9 systems.
  Use the Fix Level Recommendation Tool (FLRT) tool to obtain the recommended levels.
  NOTE: Ensure that required HMC level is installed when updating firmware.
   
• Memory DIMMs
  Follow proper memory plug-in rules found here
   
• Ensure OS level is current
  Fix Central provides the latest updates for AIX, IBM i, VIOS, Linux, HMC and firmware. In addition to that, the FLRT tool provides the recommended levels for each hardware model. Use these tools to maintain your system levels.
   
• SMT8
  To take full advantage of the performance of Power9 CPUs, we recommend clients evaluate the use of
  SMT8. Proper sizing is also recommended to maximize the Power9 improvements. We recommend
  when moving to SMT8 to reboot the partition to get the best performance of this change.
   
• 40GbE adapter
  RHEL7 and onwards: For network bandwidth sensitive workloads, we recommend increase the receive queue size
  from 1024 to 8192.
   
• Sizing a system
  • When migrating to Power9, we recommend considering using SMT8, and size the LPARs based on the SMT8 rPerf values.
     In many instances, this will likely reduce the number of VPs required.
  • Use Workload Estimator (WLE) for sizing LPARs for CPU consumption as it provides better sizing results.
   
• Right-size your Shared LPARs
  • Assign entitled capacity (EC) to sustained peak utilization for LPARs with critical SLA requirement.
  • Assign EC to average utilization and number of virtual CPUs to peak utilization (physical core consumption) for LPARs with non-critical SLA.
  • Ensure the average LPAR utilization is equal or less than 75% of the entitled capacity.
   
• Partition placement
  Current firmware levels ensure optimal placement of the partitions. However, if numerous DLPAR operations are executed on partitions on the CEC, it  is recommended the use Dynamic Partition Optimizer (DPO) to optimize placement (requires current level of firmware).
   
• Compilers
  • IBM XL C/C++ for AIX V16.1 and XL Fortran V16.1 added support for Power9. Also adds support for C++11 and C++14.
  • IBM XL C/C++ for Linux V16.1.1 & XL Fortran V16.1.1 support for P9 ISA.
  • Advanced Toolchain: 11.0-3 and later.
  • gcc: Version 7 of gcc is recommended for P9 ISA support. Also includes support for “-mtune=power9”
  • More information on compiler versions may be found here.
   
• vNIC
  • AIX - Change the following vNIC interface settings as follows:
     # chdev -l ent# -a rx_que_num=8 -a tx_que_num=8 -a rx_que_elem=2048 -a tx_que_elem=1024 -a use_rec_q_val=no
  • Linux distros - Update to latest kernel. Also, set rx/tx queues to maximum.
     # ethtool -L eth# rx 8 tx 8
• Java
  • IBM JDK8 SR5 is the minimum level to exploit Power9.
  • Open JDK 1.8 provides partial support for P9 ISA.
  • Use of 64k size pages normally increases application performance.
   
• IBM i
  Ensure Technology Updates are current.
  These can be found here.
   
• AIX Tunables / VIOS Tunables
  • Tuning a VIOS is not recommended unless directed by VIOS/AIX support.
  • Restricted tunables should not be modified (unless directed by AIX/VIOS development).
  • Tunables should not be migrated across hardware or AIX levels.
  • If tunables have been modified from default, they should be re-evaluated prior to moving to a different processor platform.
   
• AIX CPU utilization
  The AIX operating system is optimized for best raw throughput at higher CPU usage when running with dedicated processors. When running with shared processors, the AIX OS system is optimized to reduce CPU usage (pc column in vmstat). If there is a requirement to reduce CPU usage (pc), use the schedo tunable vpm_throughput_mode to tune the workload and evaluate the benefits of raw throughput vs. CPU usage.
   
• Live Partition Mobility (LPM)
  • VIOS 3.1 runs on native POWER9 mode.
  • VIOS 2.2.5 and above provide performance improvements over prior versions.
  • Use of 10Gb or faster dedicated network for LPM is preferred.
  • Use of dual MSPs can improve performance (minimum requirements: VIOS 2.2.5 and F/W860).
   
• VIOS configuration
  • Shared Ethernet adapters using a 10Gb, 40Gb or 100Gb Mellanox adapter as a backing device should enable the “flip_n_run” attribute, via chdev, on the
     network adapter port :
     # chdev entN -a flip_n_run=yes
  • If configured with shared processors:
    ➢ Assign total entitlement of all VIOS partitions to be 10-15% of cores in shared pool and assign CPU ratio of 2:1 (vCPUs:ec). 
         Refer to the PowerVM Best Practices for additional recommendations.
  ➢ Assign uncapped mode and set variable weight capacity of VIOS partition higher than all client LPARs serviced by VIOS.
        • For performance and flexibility, it is recommended to use IBM i to virtualize internal storage to IBM i.
          If you do want to use VIOS, then follow the wiki at the following link.
        • For vFC, ensure no more than 64 client connections total per physical fcs adapter on the VIOS. Also,  ensure no more than 64 storage ports configured
           per vFC adapter on the client. These are physical limits; practical limits may differ based on workload.
        • For vSCSI disks, ensure the queue_depth for virtual disks is less than or equal the queue_depth of the physical disk in the VIOS.   
    For vSCSI adapters, ensure you configure VTDs based on the following formula:
          Max VTDs = ( 512 - 2 ) / (virtual_q_depth + 3).
  • Only enable the largesend attribute on the SEA (physical adapter backing the SEA) if all LPARs serviced by the VIOS are AIX partitions.
   
• Virtual Ethernet adapters on AIX
  • Increase the virtual Ethernet (vETH) device driver buffers if the partition is dropping packets on the virtual interface even when running with entitled CPU capacity. e.g.
  # chdev –l ent# -a max_buf_xxx=NNNN
  • Set largesend on vETH adapter to improve performance (AIX):
  # chdev –l en# -a mtu_bypass=on (or) ifconfig en# largesend