Performance monitoring with Enterprise Workload Manager

Today, multi-tier Web applications and clusters are very common in IT environments. System administrators need to know how their Web applications are performing to satisfy established business goals or service level agreements (SLA). There are several tools to let you monitor systems and network capacity; in fact, most middleware has some means of providing internal statistics. Other tools and service providers can tell you how your applications are performing from different viewpoints around the world.

But, very few tools can give you an end-to-end view of every transaction coming across your middleware, from the moment the request hits the front-end Web server all the way to the back-end database, across multiple hops and multiple platforms. That's the unique capability of Enterprise Workload Manager for Multiplatforms (EWLM), a component of the IBM Virtualization Engine Server Suite. (The IBM Virtualization Engine documentation details more about this comprehensive platform to help virtualize your infrastructure; see Resources.)

For any given transaction, EWLM can tell you the path that was taken, the time it has spent at each middleware hop, and other system statistics. With this data, EWLM can draw a topology of the middleware hosting your application, which helps you identify a potential bottleneck server among a cluster, if it occurs.

Of course, middleware can mean different things in different contexts. From Wikipedia: "Middleware is now used to describe database management systems, Web servers, application servers, content management systems, and similar tools that support the application development and delivery process." In this article, middleware refers to the bundle of IBM HTTP Server, Application Server, and DB2 UDB together.

Much IBM middleware has been enhanced in the past few years to exploit the Application Response Measurement (ARM) open standard as a way to provide internal transaction statistics to any external tool supporting this standard. Application Server (Express, Base, ND, XD) and DB2 UDB have been ARM instrumented, as well as the WebSphere plug-in for IBM HTTP Server (IHS) and Microsoft™ Internet Information Server (IIS). You can now monitor your Web application's performance from end to end if you're using this middleware. However, ARM instrumentation is disabled by default; I'll explain how you can enable ARM instrumentation.

You'll need the EWLM Managed Server (MS), an agent that runs inside each server to collect the ARM statistics and send them to the EWLM Domain Manager (DM). The DM aggregates all the data and presents it through the EWLM Control Center, which you can access from any Web browser. Figure 1 illustrates the high-level architecture.

The overall process is:

1. Install EWLM DM on the management server.
2. Install EWLM MS on each machine running the middleware.
3. Enable ARM platform support in the operating system of each machine.
4. Create an EWLM DM configuration ID.
5. Create an EWLM MS configuration ID.
6. Create an EWLM domain policy.
7. Enable ARM instrumentation in DB2 UDB.
8. Enable ARM instrumentation in the Application Server.
9. Enable ARM instrumentation in IHS.
10. Use the EWLM Control Center to see the transactions statistics.

Prerequisites

EWLM supports IHS, Application Server, and DB2 on many platforms; consult the EWLM Information Center (see Resources) for the latest information. ARM instrumentation of the middleware is identical on most platforms (except i5/OS® and z/OS®), but the ARM platform support layer and related security procedures are specific to each operating system. For convenience, in this article I'll mainly discuss AIX, with some mention of Linux-specific items when necessary.

The software discussed in this article:

• AIX 5.3 ML1
• SLES 9.0 SP3
• IBM Enterprise Workload Manager V2R1
• IBM HTTP Server 6.0.2
• WebSphere Application Server 6.0.2, Network Deployment
• DB2 UDB 8.2 Enterprise Edition

You should have practical knowledge of the middleware and the operating systems. You should have already installed the EWLM DM on the management server and EWLM MS on each node where the middleware is running. Both could run in the same system, provided you have enough processing power and memory.

The test environment consists of four logical partitions on a pSeries® 570 box. Each partition is configured with the desired capacity of one processor (POWER5™ Micro-Partitioning enabled) and 2GB of memory. You can do this exercise on any pSeries stand-alone box or iSeries® partition as long as you can run AIX 5.3 on them. I have AIX and Linux running in 64-bit mode, and the same for DB2 UDB. EWLM is installed as part of the IBM Virtualization Engine CD stack, which I won't discuss in this article.

Enable ARM platform support

First I'll explain how to enable ARM platform support for AIX and Linux.

For AIX

As shown in Figure 1, neither the middleware nor any ARM-instrumented application really interacts with EWLM MS directly. They send data to the ARM platform support and MS, in turn, fetches the data from the ARM platform support and sends it over to DM. So, you must start with enabling ARM support in AIX.

Make sure that the ARM support package, named bos.net.ewlm.rte, is installed. Use the command

# lslpp -l bos.net.ewlm.rte

to query if it's installed. If it is not already installed, you'll have to get it from the AIX installation CD. You can use SMIT to install the package.

After confirming that the package is installed, you can enable ARM support as follows:

• Issue smitty ewlm.
• Change/show the status of the EWLM services.
• Enable the EWLM services.
• When prompted to enable EWLM services now, at next boot, or both, select F4 to choose both, then select Enter to confirm.

Run ewlmcfg -q to confirm that EWLM support is enabled. You should get the message:

EWLM is enabled.
No process is using the EWLM services.

ARM support in AIX should now be enabled. When starting EWLM MS, if it's complaining that the platform support is not enabled, run ewlmcfg -q again to verify. Look for the following line in /etc/inittab:

rcnetwlm:23456789:wait:/etc/rc.netwlm start> /dev/console 2>&1 # Start netwlm

Make sure this line is not commented out and that it is being invoked properly. I've seen problems where AIX was initializing another service before it and that service never exited, so AIX never got to the point of running this service. If in doubt, you could try to initialize it manually, which doesn't cause any harm if it was already initialized. Run the command /etc/rc.netwlm start, which might yield the message:

/usr/lib/drivers/netwlm already loaded
Error loading kernel extension: cfgnetwlm failed

For Linux

Before running EWLM MS on Linux, check "Required critical updates for IBM Virtualization Engine version 2.1" for additional patches to apply to the operating system (Resources). If you're compiling your own kernel, make sure you don't disable the Class-Based Kernel Resource Management (CKRM) feature in the kernel. The ARM platform support relies on CKRM to get statistics from the operating system. You also need to install fixpack 020 on top of the Linux MS. Linux MS was deactivated in the GA version.

The ARM platform support in Linux is shipped in the EWLM MS package and is installed as part of MS installation. Normally, you don't have to do anything special for that. The ARM daemon should be started at system startup. You can verify that it started using the ps aux | grep ewlmarmd command. If it's not running, verify the existence of the file /etc/init.d/ewlm and make sure the daemon's not being disabled during the startup sequence. Use your favorite platform tool for verification, such as YaST2 or chkconfig.

On the Linux platform, EWLM MS uses a filesystem mounted at /mnt/relayfs, so be cautious not to mount anything at /mnt; create a subdirectory under /mnt and use that instead.

Another important directory that you should not delete is /tmp/ewlmarmd. Every time there's an application making ARM registration, some related files are created in it. If it is deleted by accident, you can simply restart the ARM daemon, which will recreate it, but then you'll have to restart all the ARM applications as well because all the registrations would be lost. Use /etc/init.d/ewlm restart to restart.

Create an EWLM DM configuration ID

Before you can use EWLM DM, you must create a configuration ID similar to an Application Server profile or a DB2 instance. You can use the command line or the GUI wizard to do so. Creating the configuration ID creates a working directory for this configuration ID, assigns several TCP/IP ports to the DM, and creates an Application Server profile to run the Control Center application.

Using the configuration wizard

Make sure you have a working X window environment. This has to be done by root, using

# /opt/IBM/VE2/EWLM/bin/configWizardDM.sh

When the panel opens, select Create domain manager and Control Center and Next.

At step 1 of 8, the wizard asks for the configuration ID name; don't use any special characters; I'll use chekimDM. The wizard creates the directory /opt/IBM/VE2/EWLM/ewlmdm_instances/servers/chekimDM.

In the next step, the domain name is only for display purposes. You can use long words and spaces. The IP address could be your actual address (I prefer to put 0.0.0.0 as a way to tell DM to listen to all IP addresses on the system). You can keep the default port assignments or set your own as shown in Figure 2.

For step 3, it's not necessary to enable SSL encryption between DM and MS for an intranet environment.

Step 4 asks if you want to replace the default SSL certificate provided by WebSphere, which is running the Control Center application with Global Security enabled. Again, in a controlled environment, it's not really necessary. Leave the EWLM default choice and select Next.

In step 5, do not enable the firewall broker unless you know what you're doing.

Step 6 is for the user ID that will be used to stop and manage WebSphere for the Control Center. It has to be root because Global Security is using local OS authentication.

For step 7, you can assign operating system users or groups to different roles for the Control Center operations, as shown in Figure 3. You have to assign at least one user or group to the Administrator role to use EWLM; however, you can add or remove any assignment later on if you want.

At step 8, leave the box unchecked. Creating the DM configuration ID is a lengthy process, so I prefer to get it done properly and then start DM manually afterward.

After about 10 minutes it's complete, as shown in Figure 4. Notice the Control Center URL at the bottom of the window. You'll have to substitute your actual IP address or hostname in the URL.

Using the command line

You can perform this task with the command line, too. This is especially useful if you are not root and you need to provide the command to be run by another person who has root authority. These commands do exactly what the configuration wizard did:

# cd /opt/IBM/VE2/EWLM/bin
# ./createDM.sh chekimDM -ma 0.0.0.0 -mp 5775 -wasPorts 5775 -jp 5800 -dn chekimDomain
-auth None -adminUser root -adminPW password
					
# ./changeCC.sh -addUser chekimDM -adminUser root -adminPW password -roleUser root
-role Administrator

To start the DM and Control Center, use:

# cd /opt/IBM/VE2/EWLM/bin
# ./startDM.sh chekimDM
# ./startWAS.sh chekimDM

Now, log in to the Control Center to see if everything is working properly. Open Internet Explorer to https://dm_hostname:5777/webui and log in with the user ID and password that you provided as the Administrator role. After successful login, you should see the panel shown in Figure 5. Make sure you allow pop-ups for this URL.

Troubleshooting

Now what if you have errors and the wizard doesn't complete? You need to know where and why is it failing, and you can find this information in the execution logs. There are three log files created during the process in two directories:

• /var/adm/ewlm/ewlmdm/Diagnostics/createDM_timestamp.log
• /var/adm/ewlm/ewlmdm/Diagnostics/wizardDM_timestamp.log
• /var/adm/ewlm/ewlmdm/ConfigWizardDiagnostics/ConfigurationWizard.log

If there are errors during the WebSphere profile creation process, you should be able to determine the issue from the createDM log file. If you have port conflict issues, try to stop all the running Application Server instances and make sure the ports are available for use.

If the failure occurs after 90 percent of the process, most likely you're having a problem with the LDAP server. By default, after the WebSphere profile is created and the Control Center application deployed, it tries to register your newly created DM configuration ID into the LDAP server that was specified during Virtualization Engine (VE) installation. If a problem occurs during this process, your newly created DM is dropped so that you can recreate it again. It only keeps the log files to tell you what went wrong.

The LDAP registration is only used by the VE Console. If you do not use the VE Console in your environment and you thought that LDAP was not needed anymore, you do need the LDAP server for creating and deleting DM. If for some reason you can no longer get your LDAP server back, there's a workaround to skip the LDAP registration during DM creation and removal. Edit the file /opt/IBM/VE2/EWLM/bin/ewlmdm_environment.sh and comment out the following two lines:

# VE_BASE=/opt/IBM/VE2
# VE_LIB=/opt/IBM/VE2/lib

Rerun the wizard or command line. It won't try to do any LDAP registration. (Don't do this if you are planning to use the VE Console in the future or you won't be able to hook up to this DM from the VE Console bridge.)

If you want to delete a DM configuration ID, you must use the deleteDM.sh command or the wizard. Do not simply delete the working directory; there is lots of cleanup work that needs to be done.

Create an EWLM MS configuration ID

You must create a configuration ID on each EWLM MS node using either the wizard or command line. This process is much shorter than creating a DM configuration ID.

I'm running the MS process as root. You could run MS as a non-root user (see the Information Center in Resources to find out how). As before, I'll discuss the configuration wizard, command line, and troubleshooting.

Using the configuration wizard

Run the configuration wizard as root using

# /opt/IBM/VE2/EWLMMS/bin/configWizardMS.sh

At the startup panel, select Create managed server and Next.

For step 1 of 4, the configuration ID should not contain any special characters. Mine is chekimMS. A working directory is created at /ewlmData/ewlmms/servers/chekimMS.

At step 2, you can provide the DM's hostname or IP address. Don't forget to change the port if you assigned a different one earlier during DM creation.

Step 3 is for SSL encryption and must match the setting in the DM. Select None and then Next.

At step 4, if you have no direct communication between DM and MS, you may use a standard HTTP proxy to bridge them, making sure the CONNECT method is enabled. Otherwise, just select Next to start the MS creation.

And that's it. Make sure you wait until the message PROCESSING COMPLETE is shown.

Using the command line

To do all the wizard steps with one command, run the following command on all the MS nodes:

# /opt/IBM/VE2/EWLMMS/bin/createMS.sh chekimMS -ma hci246 -mp 5773 -auth None

To start the MS, enter:

# /opt/IBM/VE2/EWLMMS/bin/startMS.sh chekimMS

If you go to the Control Center to Manage > Managed servers, you should see the new MS joining the domain. If you don't see it right away, click the refresh icon at the top right of the window a few times.

Troubleshooting

The MS creation process is quite straightforward, but where you might run into problems is when you start the MS. It might not start successfully if the platform support is not set up properly, in which case you should revisit Enable ARM platform support.

Another potential problem you might have eventually is with filesystem space. By default, the MS configuration ID is created at the root filesystem under the /ewlmData directory. AIX allocates very little space to this filesystem by default. If your MS is running into exceptions and it starts dumping XML diagnostics files, it might quickly fill up the root filesystem. As a remedy, you could increase the root filesystem size or move /ewlmData's content to a larger filesystem and create a symlink at /ewlmData pointing to the new place.

Create an EWLM domain policy

EWLM is shipped with sample policies to get you started, but I'll create a new one together in this section. You'll want to specify in the policy how to classify incoming requests into transaction classes that are subsets of a service class and specify a performance objective for the service class. I'm going to use a basic Web application called Trade that simulates a simple trading Web application in which users can buy or sell stocks and query their portfolio. For those who don't have access to this application, you can use any other sample Web application for this exercise or just read along.

To begin the domain policy creation, log in to the Control Center again, select Domain policies, and then select New....

The domain policy wizard appears. Give the policy a name and a short description.

At the next step, you can keep the default EWLM workload and select Next.

In the Service classes step, create a new service class by clicking on New....

You should see the details of the service class's performance objective; all the fields are self-explanatory, as shown in Figure 6. After filling in all the values, select OK and then Next.

Now you'll add an application to the policy to classify requests coming into this application by selecting Add... in the Applications step. Remember, only the entry application matters; that's where each incoming request is being classified and the classification is carried on to all subsequent middleware hops. Each application here has a certain number of predefined filters that are pertinent to it.

For the Trade application I'm using, all the incoming requests hit the IHS first, so I'm adding the IBM Webserving Plugin application to my policy.

To create a transaction class, make sure the drop-down list is on the IBM Webserving Plugin entry and select New.... Give it a name and assign it to the position 1 of Trade service class, as shown in Figure 7. Remember that order is very important in the classification rules. You always want to have the most specific rule first, followed by the more generic ones.

You need to add a rule to this transaction class by selecting New.... My classification rule is based on the filter EWLM:URI and the value is /trade(\*) without any quotes, as shown in Figure 8. Any incoming request starting with /trade is classified in this transaction class. You can create very complex rules by grouping several individual rules with logical operators, but for this exercise let's keep it simple.

When you're done adding the rule, make sure the new transaction class is positioned before the default one that will catch everything else coming into this application. Select Finish to save the policy.

After the policy is saved, you must deploy it to actually make the policy effective. Select the newly created policy and then select Deploy... in the drop-down list; select Go.

Next, you'll be prompted for which service policy to activate. Because you didn't create any new service policies, there's only one that you can pick to activate. Select it and then Deploy. You can actually deploy a domain policy without activating any service policy, but I see no point in doing that.

Make sure the service policy is successfully activated on all the managed servers, as shown in Figure 9.

At any time you can tell which policy is active by looking at the top of the panel. You can also see the service policy that is active on any managed server. Normally, all the managed servers should have the same active service policy, but sometimes a server might be down or disconnected while a domain policy is being deployed, thus the discrepancy. As soon as the server is connected back to the domain manager, it attempts to activate the current policy.

You're now ready to receive transaction statistics from the middleware. But, you'll need to enable the ARM instrumentation in your middleware first.

Enable ARM instrumentation in DB2 UDB

Beginning with version 8.2 of DB2 UDB, ARM instrumentation is already turned on by default, but you have to tell it where to look for the ARM library, libarm4.a (or libarm4.so for Linux). For the instance db2inst1, use:

# su - db2inst1
$ db2set DB2LIBPATH=/usr/lib

Before you can start DB2, you need to give the user db2inst1 some special privileges on AIX to register itself to ARM platform support. You have to be root to do this:

# chuser "capabilities=CAP_ARM_APPLICATION,CAP_PROPAGATE" db2inst1

On Linux you simply have to add the user db2inst1 to the group ewlm_armusers (created during MS installation). In both cases, you might have to log out and log back in to make the change effective. Then you can start the DB2 instance.

For Linux operating systems, only DB2 for Linux on Intel x86 platforms has ARM instrumentation support. Linux on all other hardware platforms does not have ARM support yet, so it's not useful to do this. Future versions of DB2 UDB will have ARM support on all Linux hardware platforms.

You may now start the DB2 instance; Listing 3 shows if the DB2 instance has successfully registered to ARM.

# lsarm -a
APPL: IBM DB2 Universal Database

On Linux this command is found at /opt/IBM/VE2/EWLMMS/usr/bin/lsarm.

Troubleshooting

The most common DB2 ARM instrumentation problem is caused by unauthorized registration, either because it lacks the required privilege or the privilege was not yet effective. To double check that on AIX, run:

# lsuser db2inst1
db2inst1 id=108 pgrp=db2grp1 groups=db2grp1,staff,dasadm1 home=/home/db2inst1
shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL
admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22
registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=
false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8
minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= capabilities=
CAP_ARM_APPLICATION,CAP_PROPAGATE fsize=-1 cpu=-1 data=491519 stack=32767
core=-1 rss=-1 nofiles=2000 time_last_login=1139331346 tty_last_login= host_last_login=
hci246 unsuccessful_login_count=0 roles=

On Linux, run id db2inst1 and make sure it's in the right group. Another possible problem on Linux is that, if for any reason the directory /tmp/ewlmarm4 is not accessible to db2inst1 because it has the wrong directory permission or it might be deleted by accident during clean up activities, then lsarm won't show any application.

A more serious problem would be that DB2 somehow couldn't find or load the ARM library. In that case, you would have to capture some traces with the db2trc tool. If you are not familiar with this tool, see the DB2 command reference. Listing 5 shows a quick trick that tells how far DB2 has gone in loading the ARM library.

# su - db2inst1
$ db2trc on -f 1.trc                     (turn on tracing)
$ db2start                               (start DB2)
$ db2trc off                             (turn off tracing)
$ db2trc format 1.trc 1.txt              (format binary trace file to text)

$ cd sqllib/db2dump
$ grep -i arm 1.txt

        rc = 0x800F00FC = -2146500356 = SQLO_INV_PARM
2948    entry DB2 UDB base sys utilities sqleLoadArmFunctions fnc (1.3.5.1132.0)
2949    data DB2 UDB base sys utilities sqleLoadArmFunctions fnc (3.3.5.1132.0.1)
        libarm4.a
        6C69 6261 726D 342E 61                     libarm4.a
        6172 6D5F 7265 6769 7374 6572 5F61 7070    arm_register_app
        6172 6D5F 7374 6172 745F 6170 706C 6963    arm_start_applic
        6172 6D5F 7265 6769 7374 6572 5F74 7261    arm_register_tra
        6172 6D5F 7374 6172 745F 7472 616E 7361    arm_start_transa
        6172 6D5F 6269 6E64 5F74 6872 6561 64      arm_bind_thread
        6172 6D5F 626C 6F63 6B5F 7472 616E 7361    arm_block_transa
        6172 6D5F 756E 626C 6F63 6B5F 7472 616E    arm_unblock_tran
        6172 6D5F 756E 6269 6E64 5F74 6872 6561    arm_unbind_threa
        6172 6D5F 7374 6F70 5F74 7261 6E73 6163    arm_stop_transac
        6172 6D5F 6465 7374 726F 795F 6170 706C    arm_destroy_appl
        6172 6D5F 6765 745F 6572 726F 725F 6D65    arm_get_error_me
        6172 6D5F 7374 6F70 5F61 7070 6C69 6361    arm_stop_applica
        6172 6D5F 6765 745F 636F 7272 656C 6174    arm_get_correlat
2999    data DB2 UDB base sys utilities sqleLoadArmFunctions fnc (3.3.5.1132.0.10)
3000    exit DB2 UDB base sys utilities sqleLoadArmFunctions fnc (2.3.5.1132.0)
11569   entry DB2 UDB base sys utilities sqleArmRegisterAgent fnc (1.3.5.1135.0)
11578   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.10)
11579   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.20)
11580   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.30)
11581   exit DB2 UDB base sys utilities sqleArmRegisterAgent fnc (2.3.5.1135.0)
11880   entry DB2 UDB base sys utilities sqleArmRegisterAgent fnc (1.3.5.1135.0)
11883   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.10)
11884   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.20)
11885   data DB2 UDB base sys utilities sqleArmRegisterAgent fnc (3.3.5.1135.0.30)
11886   exit DB2 UDB base sys utilities sqleArmRegisterAgent fnc (2.3.5.1135.0)

The result of grep in Listing 5 is when DB2 has successfully loaded all the ARM functions from libarm4.a. If you're seeing this in your trace file but your DB2 instance still doesn't show up in lsarm, then you know that it is not related to ARM library loading. Most likely it is a user-privilege problem or ARM-platform-support problem.

If DB2 couldn't find libarm4.a, you would only see the first five rows from Listing 5 and wouldn't see the other ARM functions in the trace file. If you have to call technical support for this kind of issue, providing them with such a trace file is very helpful.

Enable ARM instrumentation in Application Server

Enabling ARM in Application Server is a two-step process. First, you enable Request Metrics globally. Then, inside each AppServer's setting, you tell it where to look for the ARM implementation class and JNI library. You can thus selectively enable ARM instrumentation in each AppServer of your WebSphere cell. With WebSphere Application Server Express, Base, and Network Deployment editions, ARM instrumentation is applicable to AppServers only. With WebSphere Application Server XD edition, you can apply that to Proxy Servers and On Demand Routers as well.

To enable Request Metrics at the cell scope, log in to the WebSphere Administrative Console, go to the Request Metrics panel, and use the following settings (shown in Figure 10).

• Check Enable Request metrics
• Components to be instrumented = ALL
• Trace level = Hops
• Check Application Response Measurement(ARM) agent
• Agent Type = ARM4
• ARM transaction factory implementation class name = com.ibm.wlm.arm40SDK.transaction.Arm40TransactionFactory

Now you must go into each AppServer (or other types of server in the case of WebSphere Application Server XD) to add custom properties to the Java™ Virtual Machine (JVM). As shown in Figure 11, from Application servers > yourAppServer > Process Definition > Java Virtual Machine > Custom Properties, create the following properties:

• ws.ext.dirs = /opt/IBM/VE2/EWLMMS/classes/ARM
• com.ibm.websphere.pmi.reqmetrics.PassCorrelatorToDB = true

The PassCorrelatorToDB property should be set to true only if you have an ARM-instrumented DB2 instance at the next hop; otherwise, it's preferable to set it to false or just not set it to avoid some minor overhead. If your WebSphere application is connecting to other types of database servers or if your back-end DB2 instance is an older version, you can skip this property. Note that only the new DB2 Universal JDBC Driver (db2jcc) supports flowing ARM correlator to downstream hop. If you're using another JDBC provider, ARM correlator won't flow to the next hop regardless of the database server.

• java.library.path = /usr/lpp/ewlm/java/lib_64.

You only need to set the java.library.path property if your Application Server is a 64-bit edition; otherwise, Application Server is able to find the EWLM JNI library at /usr/lib by itself. On Linux, the 64-bit JNI library is at /usr/lib64 directory.

Save the changes and restart the AppServers to make ARM instrumentation effective. Verify if ARM registration is successful by using:

# lsarm -a

APPL: WebSphere:APPLICATION_SERVER

Although this procedure is fairly straightforward for a handful of AppServers, it could become tedious if you have a few dozen on different platforms to ARM instrument because each platform might need a different directory path. For that purpose, I've written a wsadmin script called was_arm.jacl, available at Download, to automate the process. The script will interactively loop through a list of servers (AppServer, ODR, and so on) in a WebSphere ND or XD cell to add JVM custom properties for ARM instrumentation. It has the default directory paths for all the EWLM supported platforms except i5/OS and z/OS. You are free to use or modify it to fit your own environment. I strongly advise you to become familiar with this procedure before using the script; otherwise, you might not know exactly what has been done to your WebSphere environment.

If you're running Application Server as a non-root user, then the same security rule applies as with DB2. On AIX, you would have to add some privileges to the user:

# chuser "capabilities=CAP_ARM_APPLICATION,CAP_PROPAGATE" userID

On Linux you would have to add that user into the group ewlm_armusers and log in again to make it effective.

If for any reason you want to temporarily disable ARM instrumentation in your entire WebSphere cell, you can simply uncheck Enable Request metrics on the Request Metrics panel and leave those JVM custom properties alone. If you want to disable ARM permanently, it is better to remove those properties.

Troubleshooting

What if you followed all the steps and don't see WebSphere in lsarm? The first thing to check is whether the user running WebSphere has the required privilege to register to ARM. The second thing to look for is whether WebSphere has been able to load the ARM implementation class provided by EWLM in the file arm4.jar. You can make this determination from the SystemOut.log during Application Server startup. If there is no error, then the ARM implementation class was found and loaded. If there are any errors, you should double check the JVM custom property ws.ext.dirs.

When all is well, you should see the following in the SystemOut.log:

0000000a PmiRmArmWrapp I   PMRM0001I: ARM implementation is:
com.ibm.ws.pmi.reqmetrics.PmiRmArmXfcArm4Impl
0000000a PmiRmArmWrapp I   PMRM0004I: PmiRmInit
servername=6.0.0.0/hcp121Cell01/hcp121Node02/TradeServer1/APPLICATION_SERVER

When arm4.jar is not found, you'll see the following in the SystemOut.log:

0000000a PmiRmArmWrapp I   PMRM0001I: ARM implementation is:
com.ibm.ws.pmi.reqmetrics.PmiRmArmXfcArm4Impl
0000000a PmiRmArmWrapp W   PMRM0002E: Failed to instantiate:
com.ibm.ws.pmi.reqmetrics.PmiRmArmXfcArm4Impl

You'll also see the exception java.lang.NoClassDefFoundError in the SystemErr.log.

Sometimes the problem is less obvious to detect. For example, if the JNI library libewljarm4.a is not found, you might not see any error message in the logs or you might see the exception java.lang.UnsatisfiedLinkError in the SystemErr.log, depending on the circumstance of the failure. For those cases where no obvious error is seen, you should turn on PMI tracing with pmi.rm=all, as shown in Figure 12.

If WebSphere is not able to find or load the JNI library, you should see the following in the trace.log:

0000000a PmiRmArmXfcAr 3   init: creating newArmApplication object with parms:
WebSphere:APPLICATION_SERVER, cell=hcp121Cell01, node=hcp121Node02.TradeServer1,
server=TradeServer1
0000000a PmiRmArmXfcAr 3   init: newArmApplicationDefinition error_code=-15001,
error_msg=-15001
0000000a PmiRmArmXfcAr 3   init: newArmApplication error_code=-15001, error_msg=-15001

The error code -15001 is returned for any problems while loading or calling the JNI library. You would see this error code if the JNI library was not found, or if registration to ARM platform support failed. You should then double check the property java.library.path. To read about all the ARM return code descriptions, search for "ARM return codes" in the IBM EWLM Information Center (see Resources for more information).

Enable ARM instrumentation in IHS

Technically, you're going to enable ARM in the Application Server plug-in running inside IHS, not in IHS itself. After you enable request metrics in WebSphere, you just have to regenerate the plug-in configuration file to enable ARM instrumentation to the plug-in. Look for the following lines in your new plugin-cfg.xml:

<RequestMetrics armEnabled="true" loggingEnabled="false" rmEnabled="true" 
traceLevel="HOPS">

Remembering the user privilege, by default IHS is running as the user nobody. But this time on AIX, you don't have to give nobody any additional capabilities. It is inherited when you start IHS with root, which is normally required for IHS to listen on port 80 or 443. If you have a different setting in your environment, you should give the same capabilities as usual. On Linux, you always have to add nobody in the group ewlm_armusers.

After you restart IHS with the new plugin-cfg.xml, you should see it registered to ARM like this:

# lsarm -a
APPL: IBM Webserving Plugin

Troubleshooting

If there is an error while the Application Server plug-in is trying to initialize ARM or making ARM registration, you would see it in the plug-in log file, normally at /opt/IBM/WebSphere/Plugins/logs/http_plugin.log.

Beware of a major difference between IHS and the previous middleware:

• With DB2 and Application Server, if ARM support is not initialized successfully, everything is still fully functional but no ARM data gets transmitted.
• With the Application Server plug-in, if ARM support fails to initialize, the plug-in will be totally stuck. IHS will still serve static content, but won't be able to forward any request to WebSphere.

If you're debugging a strange case of Application Server plug-in failure, keep in mind that ARM support might make it stuck sometimes. If in doubt, change armEnabled=false in the plug-in configuration file and try again.

Viewing transaction statistics using the Control Center

Now that you've turned on instrumentation everywhere, let's run some transactions and see what statistics you can get out of EWLM. For this exercise I've set up a cluster of three Application Server nodes front-ended by an IHS node with a DB2 database server at the back end. One of my servers is acting as Application Server and IHS at the same time.

After hitting the Trade application several times from my browser, the Control Center is telling me how the cluster is performing in regards to the performance objective previously set in the domain policy. As shown in Figure 13, my goal was to have 95 percent of the transactions belonging to the Trade service class respond within half a second (500ms). I actually got 99 percent in that range. EWLM has given me a performance index of 0.5 for the transactions occurring during that time interval. A performance index of less than 1.0 means I'm exceeding the performance objective. If this goal was the SLA for this application, I know I'm exceeding the SLA and my customer should be happy!

Select Trade service class, select Server topology in the Action drop-down list, then select Go. A pop-up window appears showing the system topology that EWLM has discovered on the fly while transactions are being run. As shown in Figure 14, you also see that there were 75 requests being sent to hci245 Application Server node while 28 and 29 requests were sent to the two other nodes. At this point, EWLM is only capturing statistics about those transactions; it does not influence the outcome in any way.

Now let's select the Trade service class again by choosing Application topology, and then selecting Go. A pop-up window shows the same topology, but in the application perspective (Figure 15).

In both topology graphs, you can actually click on each box to drill down to a more detailed level, down to the individual process with its respective statistics. You can position the mouse cursor on the box that you want and the statistics table will show up.

In conclusion

I've just scratched the surface in describing what the EWLM Control Center is and the types of statistics it can present to you. In this article, I showed you how to enable ARM instrumentation in various middleware. You looked at the summarized and granular statistics that you can get from the EWLM Control Center, as well as the topology graphs showing the transaction paths. I demonstrated how you can get those statistics without touching the application code, using off-the-shelf IBM middleware.

As IBM continues to ARM-instrument more and more middleware, you will see the benefit of your EWLM investment growing. It's only a matter of time before you can have an end-to-end view of your Web transactions all the way back to your legacy applications running on the mainframe, or your corporate ERP. Meanwhile, if you want to get the same benefits from your home-made applications, you can ARM instrument them yourself! Just search for "EWLM APIs" in the Information Center or visit The Open Group (see Resources).

These lessons can also be applied to a WebSphere Portal Server or WebSphere Commerce Server environment as well, provided that it's running on Application Server 6.0 or higher.

Back to top

Download

Information about download methods

Resources

Learn

• IBM EWLM Information Center: The information center provides documentation, an overview, and frequently asked questions about EWLM.
  
  
• Required critical updates for IBM Virtualization Engine V2.1: Read about the lastest recommended patches to apply.
  
  
• Specified operating environments: This section of the EWLM Information Center explains software and hardware requirements for each component of EWLM.
  
  
• ARM: The Open Group has a section on ARM that shows you how to extend your enterprise management tools directly to applications, creating a comprehensive end-to-end management capability that includes measuring application availability, application performance, application usage, and end-to-end transaction response time.
  
  
• IBM Virtualization Engine: See the other components of the Virtualization Engine server suite.
  
  

Discuss

• developerWorks blogs: Get involved in the developerWorks community.
  
  
• Dave Bartlett's blog: Hear what this IBM VP of Autonomic Computing has to say about the state of autonomic computing technology.
  
  
• Autonomic Computing: An insider's perspective: Drop in on the discussion forum to get an insider's viewpoint.
  
  

About the author

CheKim Chhuor currently works at IBM Poughkeepsie in the System Verification Test team. He has many years of Web infrastructure consulting experience, and holds many IBM WebSphere®, DB2®, and e-business certifications. His current focus is on grid and autonomic computing. You can contact him at chhuor@us.ibm.com.

Comments

Back to top

Trademarks

Table of contents

• Prerequisites
• Enable ARM platform support
• Create an EWLM DM configuration ID
• Create an EWLM MS configuration ID
• Create an EWLM domain policy
• Enable ARM instrumentation in DB2 UDB
• Enable ARM instrumentation in Application Server
• Enable ARM instrumentation in IHS
• Using the Control Center
• In conclusion
• Download
• Resources
• About the author
• Comments