Troubleshooting and performance analysis of different applications in Versioned WPARs

Versioned Workload Partitions (WPARs) is a new revolution in the area of WPARs in the IBM AIX® operating system. Applications that are supported in existing operating systems can also be configured to run in Versioned WPARs. However, there can be issues while configuring and running the same applications in Versioned WPARs. This article addresses trouble shooting of problems that might occur when common applications such as IBM WebSphere® Application Server, IBM DB2® and others are started in Versioned WPARs. This article also discusses the performance comparison of applications and benefits of moving an existing setup(on older hardware) to new IBM POWER7® hardware in the form of Versioned WPARs.

Sougata Sarkar (sougsark@in.ibm.com), System Software Engineer, IBM

sougataSougata Sarkar is System Software Engineer working with AIX UPT release team, Bangalore. His areas of expertise include oracle,WAS,Hibernation and WPARs.



Sangeeth Keeriyadath (k.sangeeth@in.ibm.com), Staff Software Engineer, IBM

sangeethSangeeth Keeriyadath is a Staff Software Engineer, working with AIX development team in Bangalore, India. Apart from his expertise in AIX Workload Partitions, his technical interests include scripting and developing web based applications.



Vijayalakshmi Krishnamurthy (vijayalk@in.ibm.com), Staff Software Engineer, IBM

vijayalakshmiVijayalakshmi Krishnamurthy is Staff Software Engineer working with AIX UPT release team, Bangalore. Her areas of expertise include NFS, Virtualization features like VIOS, NPIV, LPM and WPARs.



Srikanth R Joshi (srikanth.joshi@in.ibm.com), Systems Software Engineer, IBM

SrikanthSrikanth R Joshi is Systems Software Engineer working with AIX UPT release team, Bangalore. His areas of expertise are AIX WPARs, AIX Virtualization concepts like VIOS, vSCSI, NPIV, LPM, VIOS Shared Storage Pool and SAN.



19 April 2012

Also available in Chinese

Overview of Versioned WPARs

When WPARs were first introduced, there was a restriction to have the WPAR created with the same operating system as on the global LPAR (which holds the WPAR). This restriction is there till AIX 6.1. From AIX 7.1, IBM has come up with a new technology that enables us to have WPARs created with an older level of AIX, known as Versioned WPAR.

Versioned WPAR is supported from AIX 7.1 onwards on POWER7 hardware and later.

The pre-requisite is to have the following filesets : wio.common, vwpar.52 and bos.wpars to be present in /usr/sys/inst.images on an LPAR where versioned WPARs are being created. As of now, only AIX 5.2 SP8 onwards is supported inside Versioned WPAR.

Advantages of using Versioned WPARs:

  1. Customer can deploy their applications that run in their global LPARs to Versioned WPARs. By this, they can make use of other major advantages of WPARs, such as security, multiple user environment, hardware resource requirements and more.
  2. Customer can save a lot of time as they don't need to reconfigure their applications after deploying to Versioned WPARs.
  3. Customer can get better performance by making use of the latest Power Systems hardware (POWER7). They can also use enhanced features like Active Memory Expansion (AME), Remote Restart (RR), Hibernation and more, which are not there in older hardware.
  4. Versioned WPARs are capable of having all the features like RootVG WPAR, migration and other normal WPAR features.

    Versioned WPARs are created using mksysb of an AIX 5.2 LPAR, which has the required applications installed and configurations done.


Workload setup and configuration details on the AIX 5.2 LPAR

In this setup on the AIX 5.2 LPAR, the Trade workload is installed on top of WebSphere Application Server and it uses DB2 as the backend database. For accessing the “trade workload” over HTTP, IBM HTTP Server (IHS) is installed.

Trade is a performance benchmark workload developed by WebSphere that simulates the electronic stock broker, where thousands of transactions can be run. User transactions originate from the Web browser, which in turn talks to the WebSphere Application Server that is connected to the backend DB2 Server, generating lots of transactions.

Overview of the steps to install and configure the Trade workload on WebSphere Application Server:

  1. WebSphere Application Server, IBM HTTP Server (IHS) and DB2 database software need to be installed in 5.2 LPAR.
  2. We need to start the DB2 server and create the database. The WebSphere Application Server instance needs to be started.
  3. After WebSphere Application Server is successfully started, we need to configure the Trade workload on top of it.
  4. We need to generate the plug-in for IHS from WebSphere Application Server. This plug-in (XML file) needs to be copied to IHS and the IHS server has to be started.
  5. The Trade link is http://<hostname>/trade/ and scenario link is http://< hostname>/trade/scenario

Creation of Versioned WPARs from the backup image of an AIX 5.2 LPAR.

  1. A backup image of an existing AIX 5.2 partition was created using smitty mksysb.
  2. Versioned WPAR can be created from the backup image. mkwpar -C -B backup-image is used to create the Versioned WPAR.

Problems and solutions for applications on versioned WPARs

The following example illustrates this:

  1. Legacy AIX 5.2 LPAR name is thick04cor01.upt.austin.ibm.com (running on Power 5)
  2. AIX 71B LPAR name is thick04.upt.austin.ibm.com (running on Power 7)
  3. Versioned WPAR name is thick03cor03.upt.austin.ibm.com (running on Power 7 and using AIX 71B kernel)
  4. A mksysb of 5.2 LPAR (i.e. thick04cor01) was taken and VWPAR thick03cor03 was created with that backup image in 7.1 LPAR (that is thick04)

Let's look at a few of the problems in deploying various applications ad their solutions.

WebSphere Application Server fails to start inside the Versioned WPAR

The problem is that after deploying the Trade workload to the Versioned WPAR, the WebSphere Application Server instance fails to start. The following code snippet shows the error.

Snippet 1: WebSphere Application Server start error in versioned WPAR
# hostname
thick03cor03
# pwd
/usr/IBM/WebSphere/AppServer/bin
# ./startServer.sh server1
ADMU0116I: Tool information is being logged in file
           /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU3100I: Reading configuration for server: server1
ADMU3028I: Conflict detected on port 8880. Likely causes: a) An instance of the
           server server1 is already running b) some other process is using port 8880
ADMU3027E: An instance of the server may already be running: server1
ADMU0111E: Program exiting with error:
           com.ibm.websphere.management.exception.AdminException: ADMU3027E: An
           instance of the server may already be running: server1
ADMU1211I: To obtain a full trace of the failure, use the -trace option.
ADMU0211I: Error details may be seen in the file:
           /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log
#

The problem, as shown in Snippet 1, with the starting of the WebSphere Application Server instance is that a different host name was used while creating the Versioned WPAR, (thick03cor03) than was the 5.2 LPAR (thick04cor01). The solution is to fix the WebSphere Application Server configuration to reflect the new host name. This is shown in Snippet 2:

Snippet 2: Solution to WebSphere Application Server start error in Versioned WPAR
# hostname
thick03cor03
# pwd
/usr/IBM/WebSphere/AppServer/bin
# ./wsadmin.sh -conntype none
WASX7357I: By request, this scripting client is not connected to any server process.
Certain configuration and application operations will be available in local mode.
WASX7029I: For help, enter: "$Help help"
wsadmin>$AdminConfig list Node
thick04cor01Node01(cells/thick04cor01Node01Cell/nodes/thick04cor01Node01|node.xml#Node_1)
wsadmin>$AdminTask changeHostName {-nodeName thick04cor01Node01 -hostName thick03cor03}
 
wsadmin>$AdminConfig save

wsadmin>exit

# hostname
thick03cor03
# pwd
/usr/IBM/WebSphere/AppServer/bin
# ./startServer.sh server1
ADMU0116I: Tool information is being logged in file
           /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU3100I: Reading configuration for server: server1
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server server1 open for e-business; process id is 12058694
#

Problem 2: DB2 server fails to start inside the Versioned WPAR due to licensing problem

After deploying the Trade workload to the Versioned WPAR, the DB2 server fails to start due to licensing problem. The following snippet shows the error:

Snippet 3: DB2 start error inside Versioned WPAR
# hostname
thick03cor03
# su - db2inst1
$ db2start
lscfg: device proc* not found.
09/01/2011 05:16:28      0   0   SQL8000N DB2START processing failed;
a valid product license was not found. If you have licensed this product,
ensure the license key is properly registered.
You can register the license via the License Center or db2licm command line utility.
The license key can be obtained from your licensed product CD.
SQL1032N No start database manager command was issued.  SQLSTATE=57019
$

The DB2 server failed to start due to an expired license. A new DB2 license needs to be purchased for POWER7 and applied. The following snippet shows how to solve this problem:

Snippet 4: Solution to DB2 start error inside Versioned WPAR
# hostname
thich03cor03
# pwd
/home/db2inst1/sqllib
# /opt/IBM/db2/V9.1/adm/db2licm -a db2ese_u.lic

LIC1402I License added successfully.


LIC1426I This product is now licensed for use as specified in the License Agreement
and License Information documents pertaining to the licensed copy of this product.
USE OF THE PRODUCT CONSTITUTES ACCEPTANCE OF THE TERMS OF THE IBM LICENSE AGREEMENT
AND LICENSE INFORMATION DOCUMENTS, LOCATED IN THE FOLLOWING DIRECTORY:
"/opt/IBM/db2/V9.1/license/en_US.iso88591"
#

Problem 3: DB2 server fails to connect to the database inside Versioned WPAR due to wrong host name

After deploying the Trade workload to the Versioned WPAR, DB2 fails to connect to the database. Snippet 5 shows the error. This is due to different host names for AIX 5.2 LPAR and Versioned WPAR name.

Snippet 5: DB2 server fails to connect database due to wrong hostname
# hostname
thick03cor03
# pwd
/home/db2inst1/sqllib
# cat db2nodes.cfg
0 thick04cor01.upt.austin.ibm.com 0
# vi db2nodes.cfg

To solve the problem shown in Snippet 5, we need to edit db2nodes.cfg and change the host name to the Versioned WPAR name. Snippet 6 shows what needs to be modified in db2nodes.cfg.

Snippet 6: Solution to DB2 server connect error inside the Versioned WPAR
# hostname
thick03cor03
# pwd
/home/db2inst1/sqllib
# cat db2nodes.cfg
0 thick03cor03.upt.austin.ibm.com 0
# su - db2inst1
$ db2start
lscfg: device proc* not found.
09/01/2011 06:05:54     0   0  SQL1026N  The database manager is already active.
SQL1026N  The database manager is already active.
$ db2 connect to tradedb1

     Database Connection Information

  Database server        = DB2/AIX64 9.1.5
  SQL authorization ID   = DB2INST1
  Local database alias   = TRADEDB1

$

Problem 4: IBM HTTP Server fails to load Trade inside VWPAR

After deploying the Trade workload to the Versioned WPAR, the IBM HTTP Server is unable to connect to the Trade application due to a wrong host name.

The plugin file is present in /usr/IBM/HTTPServer/Plugins/config/webserver1/. The solution is we have to change "Transport Hostname="thick04cor01.upt.austin.ibm.com" ..." to "Transport Hostname="thick03cor03.upt.austin.ibm.com" ..." in the plugin-cfg.xml file.

Please note that for IHS to pick up the latest configuration changes it needs to be restarted. The restart needs to be done from "/usr/IBM/HTTPServer/bin/apachctl -k stop and /usr/IBM/HTTPServer/bin/apachctl -k start"

Also note that some of these problems can be avoided by using a Versioned WPAR host name that is the same as that of the 5.2 LPAR. The reason for having different host names is to allow both the LPAR and the VWPAR to coexist and operate simultaneously without any IP conflict.


Performance measurement between POWER5 LPAR and Versioned WPAR on POWER7

To measure the performance of the Versioned WPAR (which is deployed on POWER7) and the POWER5 LPAR, we are taking the same example of the Trade workload, which we used to explain problems and solutions in Versioned WPARs. We will be running the akstress workload, which will be used to generate automated transactions on the Trade application that was set up for both the 5.2 LPAR ( thick04cor01) and 5.2 Versioned WPAR (thick03cor03).

akstress has been configured as follows:

  • Clients (number of browsers): 4
  • ThreadsPerClient (number of threads within each client) : 4
  • TimedRun (number of seconds for the test to be run) : 300

We are using the nmon tool to generate the performance data. Nmon runs on both servers (POWER7 as well as POWER5) for a period of time when akstress was running on both the POWER5 LPAR as well as POWER7 LPAR and usage data was collected for various hardware resources. This data was analyzed using nmon_analyzer.

5.2 LPAR performance data

  • akstress results
    • Pages attempted: 63861
    • Pages to be attempted: 150000
    • Pages per second: 212.16
    • Requests completed: 63862
    • Requests per second: 212.17
  • nmon results
    • Average processor usage in %:
      • User : 77.4
      • System : 17.3
      • Wait : 2.8
      • Idle : 2.5
      • Overall : 94.7
    • Average Disk Write in KB/sec: 323.171

The following snapshots (Figure 1 and Figure 2) shows the nmon result graph for average processor usage and average disk usage when on a POWER5 LPAR (5.2 LPAR).

Figure 1 shows the AIX 5.2 LPAR performance graph on POWER5. It shows processor usage in % by user, system and wait plotted against time with 2 second intervals, for a total period of 5 minutes.

Figure 1. POWER5 5.2 LPAR performance graph for average processor usage
5.2 LPAR Performance

Figure 2 shows the AIX 5.2 LPAR performance graph on POWER5. It shows disk read/writes in KB/sec and I/O per second plotted against time with 2 second intervals, for a total period of 5 minutes.

Figure 2. POWER5 5.2 LPAR performance graph for average disk usage
5.2 LPAR Performance

5.2 Versioned WPAR performance data

  • akstress results
    • Pages attempted: 74473
    • Pages to be attempted: 150000
    • Pages per second: 247.42
    • Requests completed: 74476
    • Requests per second: 247.43
  • nmon results
    • Average processor CPU usage in %:
      • User : 83.4
      • System : 10.5
      • Wait : 1.1
      • Idle : 5.0
      • Overall : 93.9
    • Average disk write in KB/sec: 435.3

The following snapshots (Figure 3 and Figure 4) show the nmon result graph for average processor usage and average disk usage when on a 5.2 Versioned WPAR (POWER7 LPAR).

Figure 3 shows the AIX 5.2 WPAR (on AIX 7.1 LPAR) performance graph on POWER7. It shows processor usage by user, system and wait plotted against time with 2 second intervals, for a total period of 5 minutes.

Figure 3. POWER7 5.2 Versioned WPAR performance graph for average processor usage
5.2 LPAR Performance

Figure 4 shows the AIX 5.2 WPAR(on AIX 7.1 LPAR) performance graph on POWER7. It shows disk read/writes in KB/sec and I/O per second plotted against time with 2 second intervals, for a total period of 5 minutes.

Figure 4. POWER7 5.2 Versioned WPAR performance graph for average disk usage
5.2 LPAR Performance

From the akstress data, we are seeing that 74476 transactions were completed in the 5.2 Versioned WPAR (on POWER7 LPAR) compared to 63861 transactions on a 5.2 LPAR. Also, from nmon analyzer, we see the major difference in average processor and disk usages


Related enhancements of versioned WPAR

These are the other enhancements to Versioned WPAR, for which the support has been extended recently :

  • ISD support

    WPAR Manager plug-in version 2.3 for IBM System Director 6.3 onwards supports Versioned WPAR administration (Create, Start, Stop, Remove, Change, Migrate, and so on).

  • Versioned WPAR support for other versions of AIX:

    AIX 5.3 Versioned WPAR support is being provided from AIX 71 TL 1 onwards.


Conclusion

In this article, we have started with introducing versioned WPARs and its advantages. Then, we discussed how different applications can be deployed to Versioned WPAR. Then, we saw different problems we face when working with applications like WebSphere Application Server, IHS and DB2. We also saw the solutions to each of the problems we face. Then we get onto performance part by running the workload like akstress on both POWER5 LPAR and Versioned WPAR, which is deployed on a POWER7 LPAR. At the end, we saw the related enhancements by IBM on Versioned WPARs

Resources

Learn

Get products and technologies

  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=808596
ArticleTitle=Troubleshooting and performance analysis of different applications in Versioned WPARs
publish-date=04192012