Optimizing the Application Life Cycle Process with Application Management

A common complaint amongst our customers is the chasm between the application development team and the IT team on the necessity of Application Monitoring. We consistently see applications rolled into production with inadequate monitoring; application monitoring is an afterthought, once a critical problem occurs. This article will attempt to narrow the gap between development and the IT department, and thus improve the application development life cycle by describing how to use WebSphere® Studio Application Monitor and Tivoli Monitoring for Transaction Performance along with Rational to take an application from development, through test, into production, and back into development for problem resolution. Using these tools will allow development to spend more time on developing new features and new products, and less time fixing code that has already been released into production.

Sandra Tipton (stipton@us.ibm.com), Senior Software Engineer, IBM

Sandra Tipton is responsible for product design for IBM Tivoli's Application Management product suite. Sandra was previously the technical lead for the 5.2 version of Tivoli Monitoring for Transaction Performance.



28 May 2005

Introduction

Dave arrived at the office this morning excited to spend time working on a new snazzy feature for his application. After grabbing a cup of coffee, he quickly glances at his urgent emails, and sees a note from his favorite Operations Support engineer.

For a moment, he considers pretending not to have seen the note - so that he can work on new code, rather than debug - but he knows that his annual assessment is coming up, and supporting production is one of his commitments. The email says that there is a performance problem that is affecting their gold service customers - which is preventing them from completing an online transaction.

First, he attempts to reproduce the problem in his environment - of course, the transaction completes with lightening speed! For the entire morning, he looks at the data dump and the log files sent to him by support. As usual, the information provided does not clearly identify where the source of the problem is (or why it landed on his desk in the first place!).

After lunch, he adds a lot of extra debug code - but now he will have to wait until the next window where the services organization will allow him to upload debug code to the production system.

Finally, after missing out on dinner and the new Star Wars movie with his friends - he is able to reproduce the problem on the production environment, gather the debug information he needs, and fix the problem!

Does this scenario ring true for you as a developer?

Ideally, as developers, we would prefer to spend most of our time on new and exciting tasks. In this paper, we will walk through how to use the tools from IBM to implement the best practices described by ITIL Application Management to allow you, as a developer, to devote more time to developing new features and functions, by spending less time solving problems on code that has been delivered to production.

The Information Technology Infrastructure Library (ITIL) [1] is a framework that describes best practices for the IT service industry. These best practices do not prescribe the development process you must use, or the tools to use - but instead describe the tasks, procedures, ownership, and handshakes between process step owners. The application management process of ITIL defines the application management lifecycle as depicted in figure 1 below.

Figure 1. The Application Life Cycle
The Application Life Cycle

As developers, the process steps in the Application Development Phase triangle should be familiar. Depending on the development lifecycle process you have adopted, there are likely iterations within this phase

  • The requirements process step is where the development team works closely with the business stakeholders to identify what the requirements and their relative priorities are.
  • The design process step is where the specifications are written to provide direction on how to implement the requirements.
  • The build process step includes the coding and various levels of testing the code to prepare the application for production.

The IT operations and support department typically handle the service management phase.

  • The deployment process step includes all the planning required to roll the application out to production.
  • The operate process step includes all the tasks to ensure that the application remains available for the end users.
  • The optimize process step is used to evaluate how the application is meeting the needs of the end users, and will identify new requirements that should be considered by development.

Customer Pain Points

Based on numerous interviews conducted with various IT organizations we have observed that they experience common pain points. This section will briefly describe some of these pain points. The following sections will describe the steps you can take to alleviate these pain points.

In the application development phase, prioritization is typically given to new features and functions, rather than investment in maintainability. Therefore, very little attention is given to building in function to ensure that the application can be easily monitored and maintained once it is deployed into production.

Once the application development phase completes, there is a gap between development and service management where the eligibility of the application for deployment to the production environment is determined. Development must be able to demonstrate that the application will perform as expected once it is deployed.

In the operate process of the service management phase, monitoring tends to be reactive rather than proactive. Although minimal monitoring may be enabled initially, which will identify overall responsiveness and availability of the application - more precise monitoring of individual components is not typically established until a critical problem occurs.

When a problem does occur in the operate process of the service management phase, the service management team needs the ability to isolate the cause of the problem quickly. They need to be able accurately identify where the problem is, and whether the problem is environmental, or is due to an application error.

Finally, the service management team needs to provide to the development team adequate contextual information about application errors that are identified in production.

Using IBM Tivoli's Application Management products, in conjunction with IBM Rational Robot, these pain points felt by the development and IT organization can be reduced measurably. The following sections will explain various steps that developers can take to address these pain points. The steps involved are not time consuming for the developer, and will in fact allow the developer to spend more time focusing on developing new code, rather than fixing production problems. This article explains the steps for using IBM Tivoli Monitoring for Transaction Performance, WebSphere® Studio Application Monitor, Application Response Measurement (ARM) instrumentation, and Rational Robot to ease these pain points.


What to do when you are developing your application

Development focus on feature and function vs. maintainability

For all development projects, there are 3 variables that can be managed: the delivery date, the number of resources, and the content of the deliverable. Typically, the delivery date and the number of resources tend to have little flexibility. Therefore, when a project begins to slip, the focus is on what content can be removed. New features and functions tend to take priority over maintainability.

During the Build process step of the Application development phase, taking the time to enable the application to be readily monitored once in production will be beneficial to you as a developer in addition to the IT management team responsible for the application once it is in production.

There are two mechanisms to select from to enable your application for monitoring by IBM Tivoli Application Manager products. These steps are not time consuming, and the investment up front can save you time in the long run. Enabling your application for monitoring will save you time by reducing the number of production problems you receive that are system or environmental problems, rather than application problems. The application problems that you receive will also more precisely pinpoint the location of the problem.

The first thing you can do as a developer to enable your application for monitoring is to include Application Response Measurement(ARM) instrumentation in your code. Including the ARM calls will allow production monitoring with minimal performance impact. Information about the ARM API is available at http://www.opengroup.org/arm. This instrumentation can be added to your java application code or c code. It is not necessary to add instrumentation to J2ee code that will run on an Application Server - this code will be auto-instrumented by IBM Tivoli Monitoring for Transaction Performance at run time. The following java code example illustrates how to insert ARM calls into your code.

Listing 1. Inserting ARM calls into Java code
/********************************************** 
* 1. Define ARM application and user
*    ArmJni.armInit (application_name, user_name)
**********************************************/
int appl_id = ArmJni.armInit("ArmSample1", "*"); 
       
/********************************************** 
* 2. Define Transactions
*    ArmJni.armGetId (application_id,
transaction_name,transaction_detail,
metric_info_set)
**********************************************/
               
// Transaction A - 
int tranA_id = ArmJni.armGetId(
	appl_id, "TransactionA", "My TransactionA", null);                 
           
// Transaction B
int tranB_id = ArmJni.armGetId(
	appl_id, "TransactionB", "My TransactionB", null);                 
      
// Generate num_transactions of transactions
       
for (int i = 0; i < num_transactions; i++){
/********************************************** 
* 3. Start and Stop Transaction A and B
*    ArmJni.armStart(transaction_id, 
   	metric_data_set)
*    ArmJni.armStop (transaction_handle, 
   	status, metric_data_set)
**********************************************/
int startAHandle = ArmJni.armStart(tranA_id, null);
// Transaction A code here
ArmJni.armStop (startAHandle, 
   	ArmJni.ARM_GOOD, null)

int startBHandle = ArmJni.armStart(tranB_id, null);
// Transaction A code here
ArmJni.armStop (startBHandle, 
   	ArmJni.ARM_GOOD, null)
}

The other choice you have to enable your application for monitoring is to record scripts using Rational Robot (VU or GUI) to exercise the various functions of your application and insert ARM calls into these scripts. In Tivoli Monitoring for Transaction Performance version 5.3, a simplified wrapper for the ARM API has been provided to make it easier to include ARM calls in Rational playback scripts.

The following example shows how to insert Application Response Measurement (ARM) calls into a Rational Robot GUI script. The first step is to record the transaction steps, and user the Timer Start and Timer Stop feature of the record tool to indicate the start and end of each transaction step.

Listing 2. Rational Robot GUI Script with Start and Stop timings
Sub Main
Dim Result As Integer
'Initially Recorded:11/29/2004 5:45:07 PM
'Script Name:IBMTest

StartTimer "IBMSite"StartTimer "HomePage"
StartBrowser "http://www.ibm.com","WindowTag=WEBBrowser"
StopTimer "HomePage"StartTimer "ProductsPage"
Window SetContext,"WindowTag=WEBBrowser",""
Browser NewPage,"HTMLTitle=IBM United States",""
HTMLLink Click,"HTMLText=Products",""
StopTimer "ProductsPage"StartTimer "ServicesPage"
Browser NewPage,"HTMLTitle=IBM Products -United States",""
HTMLLink Click,"HTMLText=Services and solutions",""
StopTimer "ServicesPage"

Window CloseWin,"",""
StopTimer "IBMSite"
End Sub

The next step is to replace the StartTimer and StopTimer calls in the Robot GUI Script with the Simplified Wrapper for ARM calls.

Listing 3. Rational Robot GUI Script with ARM calls
'TMTP:Added Header'$include "SWARM.sbh"
Sub Main

Dim Result As Integer
'Initially Recorded:11/29/2004 5:45:07 PM
'Script Name:IBMTest

'TMTP:Added ARM base name 
	and overall start
arm_start_with_name_rt("IBMTest")'TMTP:Added ARM start for HomePage
arm_start_with_name_rt("HomePage") StartBrowser "http://www.ibm.com", "WindowTag=WEBBrowser" 'TMTP:Added ARM stoparm_stop_rt'TMTP:Added ARM start for ProductsPage
arm_start_with_name_rt("ProductsPage") Window SetContext, "WindowTag=WEBBrowser","" Browser NewPage, "HTMLTitle=IBM United States","" HTMLLink Click,"HTMLText=Products","" 'TMTP:Added ARM stop
arm_stop_rt'TMTP:Added ARM start for ServicePagearm_start_with_name_rt("ServicePage") Browser NewPage, "HTMLTitle=IBM Products -United States","" HTMLLink Click, "HTMLText=Services and solutions","" 'TMTP:Added ARM stoparm_stop_rt Window CloseWin,"","" 'TMTP:Added ARM stop for overall transactionarm_stop_rt End Sub

Using either of these mechanisms during the build process step of the application development phase will make the application easier to maintain during the service management phase, and bridge the gap back into development from service management should any problems arise in production.


How to test your application before it is put into production

Determining production eligibility

Once the build process step completes, there is a gap to bridge between application development and service management so that the application may be successfully deployed into production.

In order to accurately predict how an application will perform once deployed to a production environment - it is crucial that the application is tested in an environment that closely simulates the production environment, and that resource usage information is collected and compared against the requirements established during the requirements process step, and/or compared against the baseline of a previous release of the application. Typically, reports should be produced and included as part of a request for change to be reviewed by a Change Advisory Board.

The reports for determining production eligibility should be completed after functional testing has validated that the application features meet the functional requirements. You can use the scripts recorded during the build process step to generate reports that will demonstrate to the IT management team that your application is ready for deployment to production.

In some cases, your organization may be fortunate enough to have the same number of servers as the application will be deployed to once in production, and these servers will have similar resource specifications (same processors, memory size, etc). If so, the scripts should be scheduled such that they play back in intervals that mimic a typical production load.

In the case where you have fewer servers in the test environment than the production environment, or less powerful servers than those that are used for production - the number of requests made by the scripts should be throttled to account for the discrepancy - so that the data collected will reflect what should occur in production.

In the WebSphere® Studio Application Monitor product, you should set the current monitoring level and sampling rate for the server(s) the application is deployed to. This is accomplished by selecting the Administration ->Monitoring On Demand menu option. Use the monitoring on demand to set the monitoring level to "(L1) Production Level" and click on the "System Default" for the sampling rate. This will collect metric data without skewing the data results.

The duration of the playback of the scripts is dependant upon the number of data points and the granularity of data you will need to present in your production eligibility reports. A typical report should be aggregated by hour of the day for a period 4-24 hours, or by minute of the hour for a period of 1 hour.

Once the playback of the scripts has completed, gather the reports needed to show that your application has not exceeded the resource requirements identified during the requirements process step. The WebSphere® Studio Application Monitor UI Performance Analysis Report tool provides a simple wizard for you to chart the trend for a variety of metrics.

An example of typical metrics that would be used for analysis of production eligibility are:

  • Response Time
  • CPU Time
  • Throughput
  • System CPU Usage
  • JVM CPU Usage
  • JVM Heap Memory Usage

Figure 2 below shows an example WebSphere® Studio Application Monitor report that charts the trend for CPU Usage aggregated by hour over a day. The data on the report should be compared to the requirements defined in the requirements process step. For example, the requirement could be specified as "The application should not exceed utilizing more than 20% of the System CPU". The System CPU trend report shows that System CPU did not exceed 7% over a 24 hour period.

Figure 2. System CPU Trend Report
System CPU Trend Report

Using WebSphere® Studio Application Monitor reports to demonstrate production eligibility will allow developers to quickly and accurately demonstrate to the IT management organization the impact that the new or modified application will have on the production environment.

Reactive Monitoring

Based on interviews conducted with numerous customers, it is common for applications to be deployed into production without monitoring enabled. Unfortunately, the result is that the need to enable the application for monitoring is identified when a critical problem occurs.

If steps are taken during the development phase, as outlined in the section Development focus on feature/function vs. maintainability, monitoring the application once deployed into production only requires a few steps in Tivoli Monitoring for Transaction Performance. The IT management team will be able to quickly set up a monitoring policy that will identify when an application is not performing as expected, or when the application is unavailable, and alert them. Figure 3 below is an example of the dashboard in Tivoli Monitoring for Transaction Performance, which displays application status to the IT management team.

Figure 3. Dashboard View
Dashboard View

Similarly, the IT management team will set up data collectors on the application server where the application will run to ensure that the impact on the production environment behaves as expected. Figure 4 below is an example of the overall health of the application servers that are being monitored by WebSphere® Studio Application Monitor

Figure 4. Enterprise Overview
Enterprise Overview

The eventuality is that monitoring of the application will be required to be enabled, and the cost of enabling monitoring is less if taken into account during the build process step of the application development phase.


How to troubleshoot problems with your application while in production

Problem Isolation

In the service management phase, when the monitoring generates an alert, the operations team needs to be able to quickly and accurately isolate the source of the problem.

First, in IBM Tivoli Monitoring for Transaction Performance, drill down on the application from the dashboard to the topology for the transaction instance that is in violation. The topology will display an icon on the component that is the worst performing as shown in figure 5 below

Figure 5. Topology View
Topology View

Then in WebSphere® Studio Application Monitor, display the detailed information for the server identified on the Tivoli Monitoring for Transaction Performance topology as the source of the problem. Figure 6 below is an example of the status of the application server as displayed in WebSphere® Studio Application Monitor

Figure 6. Server Overview
Server Overview

Using IBM Tivoli Monitoring for Transaction Performance for problem isolation will reduce the number of non-application errors that are sent to development for resolution.

Problem Resolution

One of the difficulties developers face is that the tools used in development to debug an application are typically not the same tools used during production to monitor the application or isolate the problem.

Bridging the gap between the service management phase and the development phase by providing more useful information to development can be accomplished by following a few simple steps. When an application is ARM instrumented in the Build process step of the development phase of the life cycle, the data can be used for problem resolution, and tie directly back to the source code editor in Rational.

Tivoli Monitoring for Transaction Performance has an Eclipse plug-in available for Rational Application Developer - the IBM Problem Resolution Toolkit. The toolkit will allow you to import production data to be analyzed.

In order for you to import the production data into the Problem Resolution Toolkit, you will need to be provided the following information from the IT Management staff (this information will be prompted for when you import the data into the Eclipse plug-in).

  • The name of the policy for the application which has an error
  • The name of the management server where the data resides
  • A user name and password for authentication to the management server
  • A time and date range that the application error was seen in production
  • The name of the system that hosts the application server where the problem occurred
  • One specific transaction instance when the problem occurred - this data should be collected from the transaction topology that was used to isolate the problem

Once the data is imported, you can use the toolkit to analyze the application error. The data can be analyzed with 3 different views:

  • A UML class diagram
  • A method detail view
  • A method statistics view

For example, the method details are displayed in figure 7 below

Figure 7. Method Details
Method Details

From these views, the developer can click right click on any problematic method, and directly open the source code in Rational Application Developer

Therefore, the information that development can now receive about a production problem will be able to point them directly to the source code involved, and reduce the amount of time spent sorting through production logs for debug information.


Conclusion

Once the problem is resolved, and the code is fixed, we have come full circle in the Application Life Cycle. Easing the pain points in each step requires that the recommendations made in previous steps are followed.

When development accounts for monitoring during the build process step, the benefits will be felt by both the service management team during the operate process step, and the development team when an application error is found in production, and turned over to the development team to be resolved.

Using the tools during the process steps as described in this article will reduce the pain points felt during the Application Life Cycle, and will allow development to spend more time working on new products and features, and less time debugging production application errors.

Resources

  1. Bon, Kemmerling, Pondman, IT Service Management, an Introduction, ISBN 90-806713-63
  2. Moeller, Ceylan, Bhuiyan, Graziani, Henley, Veress, IBM Redbook, End-to-End e-business Transaction Managment Made Easy, ISBN 0738499323
  3. DeCarlo, Khimani, Mackler, IBM Redbook, Planning for the Installation and Rollout of WebSphere Studio Application Monitor 3.1 , ISBN 073849223X

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Tivoli (service management) on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Tivoli, WebSphere
ArticleID=83853
ArticleTitle=Optimizing the Application Life Cycle Process with Application Management
publish-date=05282005