Installing and using IBM PowerVP and IBM AIX
IBM PowerVP virtualization performance
IBMs PowerVP tool became available in November 2013. It was designed to provide IBM Power Systems™ administrators with performance information in an enhanced visual format. The aim was to accelerate the identification of performance bottlenecks so that performance analysts could make better decisions based on more detailed and comprehensive data from POWER7 (and POWER8) processor-based systems. PowerVP presents both system (frame) and partition-level views of performance data. This has not been possible in the past using any single tool. Administrators might typically need to use many different tools and interfaces to obtain a single, system-wide performance view across an entire Central Electronics Complex (CEC) and to drill down to all individual partitions.
The tool was originally developed for IBM internal use only (known as Sleuth) which helped the IBM development team with rapid development of prototype technology and performance analysis. After a brief demonstration of the tool during an internal, invitation only, event for customers at IBM Austin, almost all of the customer attendees requested that the tool be made available for use outside of IBM.
In this article, I will briefly discuss how to quickly install and configure PowerVP in an AIX environment. I will start by discussing how to install the PowerVP GUI on a Microsoft® Windows® system and then cover how to install the PowerVP agents on an AIX or Virtual I/O Server (VIOS) partition. I'll then show you how to monitor your system and collect system wide metrics for an entire frame by recording and playing back your PowerVP sessions.
I have written about PowerVP a few times on my blog. First, I discussed the challenge I faced while installing a PowerVP service pack, then how to display VCPU affinity, and finally how to collect PowerVP system level data from the AIX command line. Each of these posts referred to version 1.1.2. of PowerVP. In June 2015, IBM released PowerVP version 1.1.3. So, I upgraded my lab environment to this version recently and found that the product and its interface had changed quite a bit. I will discuss the changes and new features at the end of this article.
Downloading and installing PowerVP
To begin, let's download, extract, and install the latest version of PowerVP. Entitled customers (PowerVM Enterprise Edition customers) can download the PowerVP software directly from the IBM Entitled Systems Support website.
Figure 1. Downloading PowerVP from Entitled Systems Support
Though by glancing through you may not be able to find the latest version of PowerVP in Entitled Systems Support, when you look closely and expand all the options, you can find v1.1.3 is listed for download.
Figure 2. Locating PowerVP v1.1.3 in Entitled Systems Support
After downloading the software, you'll end up with a PowerVP package that is named something similar to ESD_-_PowerVP_Standard_Edition_v_1.1.3_62015.zip. When you extract the zip file you'll discover the directory structure, as shown in Figure 3.
Figure 3. PowerVP downloaded and shown in Windows Explorer
To install the PowerVP GUI for Windows, run the PowerVP.exe file from the Windows folder. Select PowerVP Client GUI and click Next.
Figure 4. Installing PowerVP Client GUI
When prompted, select the Install Liberty for PowerVP check box as part of the GUI installation. The following information on IBM PowerVP Introduction and Technical Overview from IBM Redbooks® explains this.
"Starting with Version 1.1.3 PowerVP has a web based GUI. It is packaged in the Web Application Archive (WAR) format and it must be deployed onto an application server. By default, PowerVP GUI uses IBM WebSphere® Application Server Liberty Core. Liberty profile is a new server profile of IBM WebSphere Application Server V8.5. Liberty profile provides all features required to run the PowerVP, it is lightweight, has a small footprint and fast startup time. PowerVP and a configured Liberty profile are packaged into a compressed file. This provides for an easy and efficient distribution and a simplified installation procedure. Because the new PowerVP GUI is web based it is now possible that a single instance of this GUI be accessed by multiple users using web browsers. This eliminates the need to install a console for each PowerVP user and avoids the potential overhead generated by additional performance data requests initiated from multiple consoles. PowerVP users can connect to the web GUI using web browsers. Users must be able to connect to the ports on which the application server is listening. Default port numbers are 9080 for HTTP traffic and 9443 for HTTPS traffic. Port numbers can be changed during the installation process."
Figure 5. Installing Liberty for PowerVP
Figure 6, also from the IBM Redbooks, provides a visual representation indicating where Liberty fits in with the new GUI, the system, and partition-level agents.
Figure 6. PowerVP architecural diagram
After the GUI is installed on your Windows system, the next step is to extract the PowerVP agent for AIX/VIOS. To do this, you need to once again run the PowerVP.exe installer. Select PowerVP Server Agents and click Next.
Figure 7. Installing PowerVP server agents
Select the AIX/VIOS check box and click Next.
Figure 8. PowerVP agent server selection
When prompted for a system-level agent host name or IP address, you can enter anything here as it is ignored (but it cannot blank). All we want to do here is extract the installation software and not connect to an agent. I entered localhost, even though this is not where the PowerVP agent will reside. Click Next.
Figure 9. PowerVP AIX/VIOS agent specification
After completing the install process, you'll find the extracted AIX/VIOS installation file sets in the C:\Program Files (x86)\IBM\PowerVP\PowerVP_Installation\PowerVP_Agent_Installation_Instructions\AIX directory.
Figure 10. Extracted AIX/VIOS filesets
You can now transfer these files to your AIX or VIOS system of choice, essentially wherever you'd like to install and run the PowerVP server agent and any partitions you'd like to monitor as a partition-level agent. Many customers have chosen to install the PowerVP system-level agent on their VIOS. This seems like a logical place to install it as these systems are typically always up and available. Ensure that you copy the powervp.1.1.30.bff file set and the GSKit filesets to the destination system, as both are needed for installation. Of course, you should download and install the latest fixes for PowerVP from the IBM Fix Central website as well.
Figure 11. Downloading PowerVP fixes from IBM Fix Central
Figure 12. Selecting PowerVP fixes for download from IBM Fix Central
The agent installation on AIX is very simple. Make sure that your hardware and system firmware support PowerVP before you install it. The IBM Redbooks, IBM PowerVP Introduction and Technical Overview REDP-5112-00, has a comprehensive list of supported systems and minimum requirements.
If you attempt to install PowerVP on an IBM POWER® processor-based system that does not have the correct firmware level, you can see the following message in the /var/log/powervp.log file.
# cat /var/log/powervp.log 2014-02-14 15:28:01.844 Error - MSG0047: MSG0205: System does not support the required interfaces. Firmware version 7.7 or greater is required.. [syslet.C:6090]
Ensure that the Allow performance information collection check box is selected in the logical partition (LPAR) properties.
Figure 13. Partition Properties – General tab
If this option is not selected, the system level agent might fail to start with the following error.
2015-07-23 04:57:42.634 Error - syslet- Gpci_tools- Partition is not authorized to collect system-wide performance metrics. Enable the 'Allow performance information collection' partition property using the Hardware Management Console.. [syslet.C:11470]
To install the agent on an AIX or VIOS partition, copy the filesets to the system and use installp to install both the GSKit and PowerVP filesets. The GSKit filesets are required for Secure Sockets Layer (SSL) support with PowerVP. Even if you don't plan on using SSL with PowerVP, these filesets must be installed when powervp.rte is installed, regardless.
$ ioslevel 126.96.36.199 $ r oem # installp -agXd . powervp.rte +-----------------------------------------------------------------------------+ Pre-installation Verification... +-----------------------------------------------------------------------------+ Verifying selections...done Verifying requisites...done Results... … Installation Summary -------------------- Name Level Part Event Result ------------------------------------------------------------------------------- GSKit8.gskcrypt64.ppc.rte 188.8.131.52 USR APPLY SUCCESS GSKit8.gskssl64.ppc.rte 184.108.40.206 USR APPLY SUCCESS powervp.rte 220.127.116.11 USR APPLY SUCCESS powervp.rte 18.104.22.168 ROOT APPLY SUCCESS powervp.rte 22.214.171.124 USR APPLY SUCCESS [root@gibopvc2]/tmp/cg/pvp # lslpp -l GSK\* Fileset Level State Description ---------------------------------------------------------------------------- Path: /usr/lib/objrepos GSKit8.gskcrypt64.ppc.rte 126.96.36.199 COMMITTED IBM GSKit Cryptography Runtime GSKit8.gskssl64.ppc.rte 188.8.131.52 COMMITTED IBM GSKit SSL Runtime With Acme Toolkit [root@gibopvc2]/tmp/cg/pvp # [root@gibopvc2]/tmp/cg/pvp # lslpp -l powervp.rte Fileset Level State Description ---------------------------------------------------------------------------- Path: /usr/lib/objrepos powervp.rte 184.108.40.206 APPLIED PowerVP 220.127.116.11 Fixpack for AIX Path: /etc/objrepos powervp.rte 18.104.22.168 COMMITTED PowerVP for AIX
With the agent installed successfully, you can start the PowerVP agent. No further configuration is required at this point. The agent can run as a system-level agent and allow you to connect to it with the PowerVP client GUI.
# /etc/rc.d/rc2.d/SPowerVP # cat /var/log/powervp.log 2015-07-19 04:52:21.911 S824VIO2 - running in partition server mode. 2015-07-19 04:52:23.019 Server running, waiting for clients to request data.
However, if you want to configure this agent as a partition-level agent, you need to run the PowerVP iconfig tool to point the partition-level agent at an existing system-level agent. For example, we can configure the newly installed agent to communicate with an existing system-level agent at IP address 10.1.50.59. Then, we can start the agent using the SPowerVP script. We then confirm that the agent has registered with the system-level agent by reviewing the output in the /var/log/powervp.log file on the client partition.
[root@gibopvc2]/ # oslevel -s 7100-03-04-1441 [root@gibopvc2]/tmp/cg/pvp # /opt/ibm/powervp/iconfig Listen="* 13000" SystemLevelAgent=10.1.50.59 [root@gibopvc2]/tmp/cg/pvp # grep SystemLevelAgent /etc/opt/ibm/powervp/powervp.conf # sample interval from the system level agent. (See SystemLevelAgent # SystemLevelAgent [hostname-or-address [port [ssl]]] # connect to the system level agent. Up to two SystemLevelAgents SystemLevelAgent 10.1.50.59 [root@gibopvc2]/tmp/cg/pvp # /etc/rc.d/rc2.d/SPowerVP [root@gibopvc2]/ # cat /var/log/powervp.log 2015-07-19 04:52:21.911 GIBOPVC2 - running in partition server mode. 2015-07-19 04:52:23.019 Server running, waiting for clients to request data. 2015-07-19 04:52:23.211 Host GIBOPVC2 successfully registered with syslet server 10.1.50.59.
Now that the agent is installed, we can connect to it with the PowerVP GUI. You can start the GUI by doubling-clicking the PowerVP icon on your Windows system. This starts the Liberty server, opens your web browser, and connects you to the PowerVP interface.
Figure 14. PowerVP desktop icon
You should see the following messages as the PowerVP GUI server is started on your Windows system.
Figure 15. Starting PowerVP client on a Windows system
Note: You need to ensure that Java™ is in your path on your Windows system. If it is not, PowerVP will fail to start and you might be presented with an error stating that "javaw cannot be found". You can check if Java is in the path by opening a MS DOS command prompt and entering a Java command, for example:
C:\Users\ChrisG>java -version java version "1.7.0" Java(TM) SE Runtime Environment (build pwi3270sr9-20150417_01(SR9)) IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 x86-32 20150406_242981 (JIT enabled, A OT enabled) J9VM - R26_Java726_SR9_20150406_1443_B242981 JIT - tr.r11_20150401_88894 GC - R26_Java726_SR9_20150406_1443_B242981 J9CL - 20150406_242981) JCL - 20150414_02 based on Oracle 7u79-b14
If java isn't found from the DOS command line, then you can refer to the tips on how to resolve this problem.
To connect to the system-level agent, click New Connection.
Figure 16. PowerVP interface main screen
Then, enter the IP address or the host name of the partition running the system-level agent, followed by the user name and password of the root user (or padmin if running the agent on a VIOS). Then, click Connect.
Figure 17. Connecting to a PowerVP server agent
In the PowerVP main window, you can start exploring each of the main views available, such as System Topology, Node Drill Down, and Partition Drill Down.
Figure 18. PowerVP main panel view
The System Topology view shows the hardware topology of the system that we are connected to in the current session. In this view, we can see the topology of a Power® System S824 server with two processor modules. We can see that each node has two chips or sockets. We can also see numbers in the boxes, which indicate how busy each of the chips are on the system. The lines between the nodes show the traffic on the simultaneous multiprocessing (SMP) fabric between each node. If you click Toggle Buses, the PowerVP GUI shows lines between the processor module boxes and processor nodes, which represent buses. The Toggle Affinity button is intended to show the affinity where every partition has a different colour.
Figure 19. PowerVP system topology view
The Node Drill Down view appears when you click one of the nodes and allows you to see the resources being used by the partitions running on the system. In this view, we can see that this processor module has 12 cores or processors. There are lines showing the buses between the chips. We can also see the memory controllers and the PHB buses which shows traffic to and from our I/O system. We can also see connections to the other processor module; this is the SMP connections to other nodes and shows traffic.
Figure 20. PowerVP Node Drill Down view
The Partition Drill Down view allows us to drill down to resources being used by a specific partition that we clicked. This view opens on a new tab in our web browser. In this view, we can see CPU, memory, disk input/output operations per second (IOPS), and Ethernet being used. We can also get an idea of cache and memory affinity (under the Detailed LSU Breakdown section in Figure 21).
Figure 21. PowerVP Partition Drill Down view
The main panel also provides you with a view of processor utilization for each LPAR on the system. You can easily sort LPARs based on utilization to quickly identify the LPARs that are using the most (or least) CPU across a single system.
Figure 22. Partition utilization view
Overall system processor utilization is available from the main panel also. This view provides a graph of the total processor utilization, over time, for the entire Power server. Directly above this graph, you can find useful information for items such as clock frequency, total cores, platform (AIX, Linux, VIOS, or IBM i), system model/serial number, and sample rate.
Figure 23. Overall system utilization view
One very useful feature of PowerVP is the ability to record and playback your PowerVP sessions. By clicking Start Recording, PowerVP starts to record your session to your local system (in my case, my Windows notebook).
Figure 24. Playback and recording interface
I can then load this recording at a later date for playback inside the PowerVP GUI.
Figure 25. Loading recorded PowerVP for playback
PowerVP and VIOS advisor
A new feature in version 1.1.3 allows you to run the VIOS advisor (part) directly from the PowerVP GUI. When you connect to a system-level agent on a VIOS instance, you will be presented with the VIOS Performance Advisor panel in the GUI. You can configure PowerVP to run the VIOS advisor at a particular time or you can run it on demand. You can also retrieve previously created VIOS advisor reports from the GUI. This is a very nice feature.
Figure 26. VIOS advisor interface
When I clicked Run Advisor on my VIOS, I noticed that a new topas_nmon and part process were started. The process ran for 10 minutes (default) and then a new .tar file was created in /opt/ibm/powervp/advisor.
# pwd /opt/ibm/powervp/advisor # ls -ltr total 672 -rw-r--r-- 1 root system 369 Jul 23 14:11 ioscli.log -rw-r--r-- 1 root staff 337920 Jul 23 14:22 s824vio2_150723_14_11_34.tar # # ps -ef | grep topas_nmon root 14614550 1 0 15:55:20 - 0:00 /usr/bin/topas_nmon -X -s 15 -c 40 -t -w 4 -l 150 -I 0.1 -ytype=advisor -o ./s824vio2_150723_15_55_20/ -youtput_dir=/opt/ibm/powervp/advisor/./s824vio2_150723_15_55_20/ -ystart_time=15:55:20,Jul23,2015 # ps -ef | grep part root 16908532 7077922 0 15:55:19 - 0:00 /usr/bin/perl /usr/ios/utils/part -i 10 #
A new tab was automatically opened in my web browser and showed me the VIOS advisor report for my VIOS. Impressive stuff!
Figure 27. VIOS advisor report
Refer to the PowerVP Redbooks for more information about how to configure and use this option.
Changes in PowerVP version 22.214.171.124
Several features and functions have changed since the last release of PowerVP. Here is a short list of the important changes I've encountered so far:
- The new PowerVP web interface does not support the older level of PowerVP agents. You need to update both the system-level and partition-level agents to v1.1.3 for the new version to function.
- The old PowerVP GUI was previously installed on Windows as a PowerVP.exe application. This has been replaced by a launch-powervp.bat file (a shortcut will be created on your desktop). This starts the Liberty server for the GUI. You must select to install Liberty for this file to be installed. Figure 28 lists the contents of my PowerVP GUI Installation directory for my Windows laptop.
Figure 28. PowerVP launch-powervp.bat file in Windows
I also came across this useful tip in the PowerVP IBM Redbooks. PowerVP can record large amounts of data when recording is enabled. So, you should make sure that you have sufficient space available to store recorded data on your local system. It is recommended to increase the sample rate from the default of 1 second to reduce the amount of data collected during recording. The sample rate can be changed by editing the /etc/opt/ibm/powervp/powervp.conf file and changing SampleInterval to a larger value. You need to only change the sample interval on the system-level agent (the partition-level agents pick up the sample interval from the system-level agent). After modifying the powervp.conf file, you must restart the PowerVP system-level agent (syslet on AIX).
The aim of this article is to help you quickly install and configure PowerVP in your AIX environment. I encourage the reader to review the available PowerVP material from IBM (and in particular, the PowerVP Redbooks) to learn more about the features and functions of the tool. This tool, finally, provides IBM Power Systems administrators with a single method of obtaining some important performance data in their POWER7 and POWER8 processor-based systems environment.