Introduction

Reliability, availability, and serviceability (RAS) include those aspects of hardware and software design and development, solution design and delivery, manufacturing quality, technical support service and other services that contribute to assuring that the offering will be available when the customer wants to use it.

RAS assures that it will reliably perform the job and that if failures do occur, they will be non-disruptive and will be repaired rapidly. It guarantees that after repair, the user may resume operations with a minimum of inconvenience. These characteristics must be stable and exhibit sustained performance over the entire offering life irrespective of IBM® provided changes or enhancements.

You should have a good understanding of WebSphere® Application Server, DB2®, WebSphere Process Server, and its components before reading this article.

Reliability overview

The reliability of a system can be defined in multiple ways. For the scope of our testing, the following are considerations:

• The system is considered reliable if it is up and running for seven continuous days, with a constant load flowing through the system.
• All the events are accounted for at the destination.

Software used

• WebSphere Process Server V6.0.2
• WebSphere Application Server V6.0.2.17
• JDK 1.4
• MQ Series 6.1
• IBM Tivoli Directory Server 6.0

Endurance test configuration details

This article uses endurance testing to test a WebSphere Process Server system for memory leaks or other problems that can occur with prolonged execution. To increase the reliability of a product or system, all the components, such as Business Process Execution Language (BPEL), the Human Task Manager, and so forth, should work properly throughout their life. But in practice, it is not possible to test the product or system for its entire life due to cost and time constraints.

We performed accelerated life tests to bridge this gap. Accelerated life tests are used to subject the subcomponents or systems to constant minimal stress and changing conditions, like adding or deleting components for a shorter duration. This provides the expected life of the product for different levels of system parameters. Once the test is complete, you will be able to predict the life of the component at a certain confidence level.

Test setup and load

In the test setup, the Private Exchange (PE) scenario runs for seven continuous days. The load and size of the Business Object is constant.

• Business Object size = 20 Kilobytes
• Pumping rate = 30 Business Objects per minute

While the PE scenario was running, the insurance scenario was deployed on the same setup and a variable number of quote and claim requests were sent throughout a five-day period. This is done to simulate a change to the environment and to see how the product responds to change.

Runtime tools used

To test runtime tools interaction with the server, the following runtime tools were launched and used throughout seven-day period:

Admin console

Admin console to install and uninstall applications, configure system or application parameters, access various tools, start or stop applications, and so forth. The Admin console was used to install or uninstall all the application EAR files.

Failed event manager

Failed event manager to view failed events, discard or resubmit them with or without modifications. All the failed events were monitored using Failed Event Monitor. Some failed events were resubmitted and then monitored.

CBE browser

CBE browser to monitor the events. In the PE Scenario some of the events are added to Common Event Infrastructure (CEI) database. CBE browser was used at least an hour each day for event monitoring.

Business Process Choreographer Explorer

Business Process Choreographer Explorer to monitor Business Process Choreographer failed events, and Human Tasks. Since Business Process Choreographer Explorer lists all the Human Task Manager tasks, it was used at least three times a day for 45 minutes to claim the tasks waiting for the human approval.

Business Rules manager

Business Rules manager to deploy and manage Business rules. This was used to modify and redeploy existing business rules at least one hour a day.

Human Task Manager

Human Task Manager to manage human tasks such as claim, complete, and escalate tasks. Since Business Process Choreographer Explorer lists all the Human Task Manager tasks, it was used at least three times a day for 45 minutes to claim the tasks waiting for the human approval.

Relationship manager

Relationship manager to access relationships, add, and change relationship values. This was used to modify and redeploy existing relationships at least once a day.

Scenarios used

Two scenarios are used during our endurance run:

• Private Exchange Scenario
• Insurance Scenario

The private exchange scenario focuses on item synchronization for suppliers, and the insurance scenario focuses on how customers purchase insurance and file claims online.

Private exchange scenario

The Private Exchange Scenario focuses on the Item Synchronization for a Suppliers solution, which automatically integrates a supplier's item business processes with those of its trading partners. For the endurance test, a JSP was used at the source and a flat file adapter was used for the destination.

Insurance scenario

The Insurance scenario focuses on creating a Web enabled system that lets the customer purchase the insurance and file claims online. Before purchasing the insurance, the customer can request a quote. Then depending upon the customer history and the current company policy, the request is processed and the results are sent to the customer.

System design: ND7 topology

You can see the topology used in Figure 3. Tivoli Directory Server was used for authentication. The databases specific to WebSphere Process Server were on one machine and the databases specific to our applications were on another machine. Since we used ND7 Topology, we had a horizontal ME cluster.

Nine-step checklist for the setup

1. Recommend 4 GB of RAM or more in every machine where WebSphere Process Server nodes are installed.
2. Recommend 80+ GB of total hard disk space (partitioned or single).
3. Turn off Windows automatic updates (including anti-virus updates).
4. Tune each server JVM logs so that it creates 100 logs of 20 MB. You could create a cron job for taking backup of or move these logs files to save on disk space, and also to avoid any unpredicted rollover of logs.
5. Follow instructions for tune up parameter instructions for WebSphere Process Server and database. See Appendix B: ND7 tuning parameters.
6. After completing ND7 setup, take backups of all databases and save them. This helps to restore DB to the clean state if necessary.
7. Change DB2 logging for each database from circular to archive.
8. Archive logging creates ".log" files on hard disk depending on the size specified, one can archive the older files to some other hard disk partition or removable media to save disk space.
9. Shut down any applications that might interfere with the performance and stability of the machine.

Tuning parameters

Endurance testing requires tuning to ensure optimum performance.

The various components and parameters that need to be tuned are as follows:

• Application Server
• Java™ Virtual Machine
• Database
• WebSphere MQ
• Application Parameters
• Trace and Logs
• Connection Factories
• Tuning for Activation Specifications
• DB2 Connection Pool settings
• Tuning for Event Sequencing
• Performance Monitoring Infrastructure
• JCA Adapters

Refer to Appendix B for the tuning parameters used in our endurance testing.

Problems and solutions

Our team implemented the following solutions for the stated problems, see Table 1.

Table 1. Problems and solutions implemented

Back to top

Where do I look for missing events?

1. For every event consumed by WebSphere Process Server, there must be an ID associated with it. Check the database and write down the ID for which you did not see an appropriate output.
3. Open the admin console and check for failed events. Check the eventID of each failed event, and check if that matches with the ID that you found in step 1. If yes, then the event is not missing since it is traceable and can be resubmitted.
4. Open Business Process Choreographer Explorer and check the BPEL failed events. If there are any failed events, check the ID of each failed event and check to see whether that matches with the ID from step 1. If yes, then this event is traceable, too.
5. Check your e-mails (if you have e-mail notification configured), and see if you have received any email regarding the failed event inside BPEL. Make sure to match the ID from the e-mail with the ID from step 1. If yes, then you can count this event as the event is traceable.
6. Check your queues to see if there are events still lurking around and not really missing. Please note that you could create a custom application to check your queues.

Monitoring DB2 system health

During an endurance run, it is also very important to continuously check the performance of the database, which is DB2 in this case.

DB2 V8 introduces two new features to help you monitor the health of your DB2 systems: the Health Monitor and the Health Center. These tools add a management by exception capability to DB2 Universal Database by alerting you to potential system health issues. This enables you to address health issues before they become real problems that affect your system's performance.

The Health Monitor is a server-side tool that constantly monitors the health of the database instance, even without user interaction. If the Health Monitor finds that a defined threshold has been exceeded (for example, the available log space is not sufficient), or if it detects an abnormal state for an object (for example, an instance is down), the Health Monitor will raise an alert.

When an alert is raised, two things can occur:

• Alert notifications can be sent by e-mail or to a pager address, allowing you to contact whoever is responsible for a system.
• Preconfigured actions can be taken. For example, a script or a task (implemented from the new Task Center) can be run.

Constantly checking the system performance helped us in completing our endurance run successfully.

Heap Space Analyzer: Analyze JVM performance

During our endurance run, it was important to constantly keep a tab on the memory usage of WebSphere Process Server 32-bit or 64-bit JVM logs to ensure there were no memory leaks, and so forth. Several tools can help you check for memory usage by analyzing the native_stderr.log. However, for the sake of simplicity, we built our own command line tool to analyze the 32-bit and 64-bit JVM logs.

How the HeapSpaceAnalyzer works

HeapSpaceAnalyzer is a Java-based performance analyzer tool.

The user basically feeds the JVM logs to the HeapSpaceAnalyzer, which in turn analyzes the data and prints out important information:

• Average used Heap space
• Maximum used Heap space
• Average value of allocation failures

The tool also provides additional data:

• Total number of allocation failures
• Average used heap space for every 'n' cycles
• Maximum contiguous space available after compaction

Refer to Appendix A for how to use the above mentioned tool.

Conclusion

Even though endurance is just one part of Reliability testing, it is still an important aspect. This article started off explaining endurance and then went on to explain the configuration details and scenarios used for our endurance testing. We have mentioned the problems that we faced during our endurance run and explained how we solved them. The tuning parameters section is very important and useful as it suggests what parameters need to be tuned for your setup. We sincerely hope that the information presented in this article is useful to people in the field as well as IBM customers alike.

Appendix A

How does the Heap Space Analyzer work?

HeapSpaceAnalyzer is a Java class and takes two parameters:

• ARG 1: Name of the log file (native_stderr.log);
• ARG 2: Sampling Rate

How to determine the Sampling Rate:

Assume we have about 2000 GC cycles. The sampling rate argument would calculate the Average used heap space during each sampling rate.

This would help us determine if the heap space is exponentially increasing (ultimately resulting in OutOfMemory) or is steadily maintained within a certain range.

Below is a sample output that the HeapSpaceAnalyzer generated from the log we generated on a 64-bit JVM where we used large objects. The result shows that the Average Heap Space used is stable after a certain number of cycles.

C:\SHARE\AnalyzeTool>java HeapSpaceAnalyzer.class native_stderr.log 100

Listing 1. Analyzing Log File: native_stderr500.log

			*************************************************
			------------------------------------
			FINAL ANALYSIS OF NATIVE_STDERR LOG
			------------------------------------
			Total number of GC cycles = 1373.0
			Average Heap in use = 2.0 GB
			THE SAMPLE RATE SPECIFIED = 100.0
			------------------------------------
			Sample Number 0: Heap in use after GC = 1.7 GB
			Sample Number 1: Heap in use after GC = 1.9 GB
			Sample Number 2: Heap in use after GC = 1.9 GB
			Sample Number 3: Heap in use after GC = 1.9 GB
			Sample Number 4: Heap in use after GC = 2.0 GB
			Sample Number 6: Heap in use after GC = 2.0 GB
			Sample Number 7: Heap in use after GC = 2.0 GB
			Sample Number 8: Heap in use after GC = 2.0 GB
			Sample Number 9: Heap in use after GC = 2.0 GB
			Sample Number 10: Heap in use after GC = 2.0 GB
			Sample Number 11 Heap in use after GC = 2.0 GB
			

Appendix B

Application server tuning

In WebSphere Process Server admin console, select Application servers => server1 => Thread Pools => Default and change the following properties:

Under Application servers, choose BPEL1 => Message Listener Service => Thread Pool change.

Under Application servers, choose server1 => Container service => Transaction Service and change the following property for all servers.

Under Troubleshooting, choose Logs and Trace => server =>JVM Logs and change the following values.

JVM tuning

From the WebSphere Process Server admin console, choose Servers => Application Servers, then do the following:

1. Click the server name.
2. Under the server infrastructure, expand Java and Process Management by clicking on the + icon.
3. Select the link Process definition.
4. From Process definition, choose Additional properties and select Java Virtual Machine.
5. Under general properties, select (check) verbose garbage collection.
6. Increase max heap size to 1024 and above depending on configuration.
7. Paging (that is, page faults) can cause severe performance problems for your application and result in long garbage collection pauses. To avoid paging, ensure that is not set to more than 75 percent of the physical memory of the system.
8. Refers to the existence of free chunks of memory in the heap that are too small for object allocation.

The following recommendations will help you avoid fragmentation:

• The easiest way to avoid fragmentation is to increase the heap size within its natural limits.
• Normally, a partial compaction of each old collection occurs with the garbage collection. If you think this default compaction ratio is either insufficient or"overkill," you can turn it down, thus reducing pause times, by using some combination of the compaction start-up options listed here.
• Use a generational garbage collector. During a young collection (nursery garbage collection) the objects that are found live in the nursery are moved to the old generation. This has the positive side-effect of compacting the objects while they are moved.

For Generic JVM arguments, set:

• "-Dibm.dg.trc.print=st_verify -Xgcpolicy:optavgpause -Xcompactgc –Xnopartialcompactgc"
• Initial heap size to 512
• Maximum heap size to 1536

For a very useful guide to performing tune up parameters for JVM, see Resources.

Database tuning

The following script can be run in the database server hosting WebSphere Process Server database: WPRCSDB, BPEDB

Listing 2. Run this script in the database server hosting WebSphere Process Server

-- Tuning for Websphere Process Server db
CONNECT TO <Websphere Process ServerDBName > USER <username > USING <password >;
UPDATE DATABASE CONFIGURATION   USING LOGFILSIZ       2500;
UPDATE DATABASE CONFIGURATION   USING LOGPRIMARY      100;
UPDATE DATABASE CONFIGURATION   USING LOGSECOND       100;
CONNECT RESET;

-- Tuning for CEI db
CONNECT TO <CEIDBName > USER <username > USING <password >;
UPDATE DATABASE CONFIGURATION   USING LOGFILSIZ       2500;
UPDATE DATABASE CONFIGURATION   USING LOGPRIMARY      100;
UPDATE DATABASE CONFIGURATION   USING LOGSECOND       100;
CONNECT RESET;

-- Tuning ME db
CONNECT TO <MEDBName > USER <username > USING <password >;
UPDATE DATABASE CONFIGURATION   USING LOGFILSIZ       2500;
UPDATE DATABASE CONFIGURATION   USING LOGPRIMARY      100;
UPDATE DATABASE CONFIGURATION   USING LOGSECOND       100;
CONNECT RESET;

-- Tuning for BPE db
CONNECT TO <BPEDBName > USER <username > USING <password >;
UPDATE DATABASE CONFIGURATION   USING APP_CTL_HEAP_SZ 144;
UPDATE DATABASE CONFIGURATION   USING APPGROUP_MEM_SZ        13001;
UPDATE DATABASE CONFIGURATION   USING CATALOGCACHE_SZ 521;
UPDATE DATABASE CONFIGURATION   USING CHNGPGS_THRESH  55;
UPDATE DATABASE CONFIGURATION   USING DBHEAP          600;
UPDATE DATABASE CONFIGURATION   USING LOCKLIST        5000;
UPDATE DATABASE CONFIGURATION   USING LOCKTIMEOUT   30;
UPDATE DATABASE CONFIGURATION   USING LOGBUFSZ        245;
UPDATE DATABASE CONFIGURATION   USING LOGFILSIZ       2500;
UPDATE DATABASE CONFIGURATION   USING LOGPRIMARY      100;
UPDATE DATABASE CONFIGURATION   USING LOGSECOND       100;
UPDATE DATABASE CONFIGURATION   USING MAXAPPLS        200;
UPDATE DATABASE CONFIGURATION   USING MAXLOCKS        57;
UPDATE DATABASE CONFIGURATION   USING MINCOMMIT       1;
UPDATE DATABASE CONFIGURATION   USING NUM_IOCLEANERS  6;
UPDATE DATABASE CONFIGURATION   USING NUM_IOSERVERS   10;
UPDATE DATABASE CONFIGURATION   USING PCKCACHESZ      915;
UPDATE DATABASE CONFIGURATION   USING SOFTMAX         440;
UPDATE DATABASE CONFIGURATION   USING SORTHEAP        228;
UPDATE DATABASE CONFIGURATION   USING STMTHEAP        2048;
UPDATE DATABASE CONFIGURATION   USING DFT_DEGREE      1;
UPDATE DATABASE CONFIGURATION   USING DFT_PREFETCH_SZ 32;
UPDATE DATABASE CONFIGURATION   USING UTIL_HEAP_SZ    11663;
UPDATE DATABASE MANAGER CONFIGURATION USING SHEAPTHRES         27416;
UPDATE DATABASE MANAGER CONFIGURATION USING INTRA_PARALLEL     OFF;
UPDATE DATABASE MANAGER CONFIGURATION USING MAX_QUERYDEGREE    2;
UPDATE DATABASE MANAGER CONFIGURATION USING MAXAGENTS          400;
UPDATE DATABASE MANAGER CONFIGURATION USING NUM_POOLAGENTS     120;
UPDATE DATABASE MANAGER CONFIGURATION USING NUM_INITAGENTS     0;
UPDATE DATABASE MANAGER CONFIGURATION USING FCM_NUM_BUFFERS    4096;
UPDATE DATABASE MANAGER CONFIGURATION USING PRIV_MEM_THRESH        32767;
ALTER BUFFERPOOL IBMDEFAULTBP SIZE 214990;
SET CURRENT QUERY OPTIMIZATION = 5;
CONNECT RESET;

WebSphere MQ tuning

1. Start WebSphere MQ Explorer.
2. On left-hand side, inside navigator, select the manager that you created for endurance testing and right-click it.
3. In the pop-up menu, select properties. This opens a new window "<manager-name> Properties".
4. In this window, on left-hand side, select Channels.
5. On the right-hand side, set "max channels" and "max active channels" to 500.
6. Click on Apply and select Okay to close the properties window.

Application parameter tuning

Trace and logs

Configuring the size and number of rollover log files created for SystemOut and SystemErr log files as well as trace files. This file size and numbers will vary depending on the use case being tested.

1. From the WebSphere Process Server admin console, choose Troubleshooting => Logs and Trace.
2. Click on the Server name for which you want to set the settings.
3. Select JVM logs.
4. Under SystemOut settings, in log file rotation, set the required size for log files and also set the number of historical logs.
5. Under SystemErr settings in log file rotation, set the required size for the log files and also set the number of historical logs.

It is advisable to have no trace during the endurance run other than default trace.

For the singleton server, file size range depends upon whether it is hosting any applications. Singleton hosted Web services for Insurance services in the RAS Endurance Scenario. In this case, it created a single SystemOut.log file of size 5 MB.

Connection Factories Tuning

To send 30 BO per min, do the following connection factory settings:

From the WebSphere Process Server admin console, select Resources => JMS Providers => default messaging => BPEL cluster scope => JMS Connection Factory.

Select BPECF and click on Connection Pool properties under "Additional Properties".

Repeat the same for BPECFC and HTMCF in JMS Connection Factory.

Under Cell Scope, choose JMS Queue connection factory => CommonEventInfrastructure_QueueCF => and connection pool properties.

Calculate the throughput for CEI by sending 30 events and observe the logs created under the CEI browser. Take the difference between the time the first event was sent (say x) by client and last event received (say y) by CEI browser (y –x). Multiply this difference (rounded to closet bigger integer) by 30 to identify the minimum number of active connections for the CEI Queue connection factory. Assuming the difference is 3 (hence the active connection can reach 3x30=90), set the minimum and maximum connections as:

Under "JMS Topic connection factory" select CommonEventInfrastructure_AllEventsTopicCF and click on connections pool properties. Set the connections values to:

Under Resources, choose JMS Providers => WebSphere MQ.

For all cluster scope as well as cell scope, change "connection pool settings" for all WebSphere MQ connection factories listed. Change the minimum and maximum connection values to:

Tuning for activation specs

In WebSphere Process Server admin console, select Resources => JMS Providers => Default messaging => Default messaging provider. Change the scope to BPEL cluster and select option: JMS activation specification => BPEApiActivationSpec and change the following properties:

Repeat the same procedure for BPEInternalActivationSpec, HTMInternalActivationSpec

For rest of the Activation Specs, set the properties to:

DB2 Connection Pool Settings

In WebSphere Process Server admin console, select JDBC Providers => DB2 Universal JDBC Provider =>BPEL Cluster => Data Sources => BPEDataSourceDb2 => Connection pool properties and change the following properties:

For application related data source EventsTally, Flatfile, INSWEBDB, PrivateExchange, and so forth, set the connection pool properties to:

Under CEI Cluster, choose Data Sources => event => Connection pool properties and change the following properties:

Repeat the settings for "event catalog" data source.

Under ME Cluster, choose Data Sources => DS for ME => Connection pool properties and change the following properties:

Repeat this for all other data source in this cluster scope.

Under Cell scope, choose Data Sources => WebSphere Process Server Datasource => Connection pool properties and change the following properties:

Repeat this for all other data source in cell scope.

Tuning for event sequencing

1. In WebSphere Process Server admin console, select Work managers => es-workmanager and set Maximum number of threads = 40. The default value is 20. This needs to be changed to match BPEApiActivationSpec(JMS activation specification => BPEApiActivationSpec and BPEInternalActivationSpec, the max concurrent value is 40) .
2. For ND7: every WebSphere Process Server installation, check the file /opt/wps/properties/EventSequencing.properties and change maxActiveMessages from 100 to 0.

Performance Monitoring Infrastructure

In WebSphere Process Server admin console, select Monitoring and Tuning => Performance Monitoring Infrastructure (PMI) => server1. Enable Performance Monitoring Infrastructure (PMI).

JCA-JDBC Adapter application tuning

Let's say we are sending 30 BO per min, we need to tune JCA-JDBC adapter polling so that 30 events are picked for processing every minute.

In WebSphere Process Server admin console, select Applications => Enterprise Applications => JCAJDBCAdapterApp => Connector Modules => CWYBC_JDBC.rar => JCAJDBCAdapterApp.IBM JDBC Adapter => J2C activation specification => commonj.connector.runtime.InboundListener => Custom properties.

Change the value for:

This will poll 10 events every 20 seconds hence 30 events per minute.

JCA-FLATFILE Adapter application tuning

For sending 30 BO per min, we tune the flat file adapter application as follows:

Choose Application => Enterprise Applications => JCAFlatFileApp => Connector Modules => CWYFF_FlatFile.rar => JCAFlatFileApp.IBM WebSphere Adapter for Flat Files => J2C connection factories => javax.resource.cci.ConnectionFactory => Connection pools. Change the following properties:

Back to top

Download

Information about download methods

Resources

Learn

• WebSphere Application Server for z/OS Information Center
  Provides a useful guide in doing tune up parameters for JVM and JVM memory tuning.
  
  
• WebSphere business integration zone
  Resources to help you advance your business integration skills.
  
  
• The Safari technology bookstore
  The bookstore provides books on various technical topics.
  
  

Get products and technologies

• Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  

Discuss

• developerWorks blogs
  Get get involved in the developerWorks community.
  
  

About the authors

Comments

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Endurance test configuration details
• Scenarios used
• System design: ND7 topology
• Problems and solutions
• Where do I look for missing events?
• Monitoring DB2 system health
• Heap Space Analyzer: Analyze JVM performance
• Conclusion
• Appendix A
• Appendix B
• Listing 2. Run this script in the database server hosting WebSphere Process Server
• Download
• Resources
• About the authors
• Comments

Next steps from IBM

• Try: WebSphere Application Server
• Webcast: Using Rational Application Developer with WebSphere Application Server to increase productivity
• Briefing: Building SOA solutions and managing the service lifecycle
• Buy: WebSphere Application Server

My developerWorks community

Dig deeper into WebSphere on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers