Troubleshooting Performance Issues in WebSphere Lombardi Edition and Teamworks

Resolving The Problem

This article details information about troubleshooting performance issues with TeamWorks.

First it's nice to get an overall picture of the performance issue:

1. Where specifically are you seeing performance problems?
• Authoring Environment
• User Experience - launching a bpd, progressing to the next coach in a flow, ...
• Custom solution code
• Systemic - impacting all areas,
• etc...
2. Can you quantify the performance issue?
• How long does a specific action take?
• Do you have logging that shows where the time is being spent?
• Do you have performance tools to help analyze or collect data?
• etc...
3. Is this a new problem or have you always had it in this environment?
4. Do you see this same problem in other environments on the network?
5. Is the performance problem consistent or does it change?
• Performance is worse at certain times of day?
• Performance is bad under load?
• Performance improves after a restart of the application server?
• etc...

Answers to the above questions might guide the approach that is taken, but the following information is typically relevant in any situation.

What are the hardware specifications?

1. Application server hardware specifications:
• Specific hardware being used
• # of CPUs
• Speed of CPU
• Total memory installed
• Network connection information/speed
2. Database server hardware specifications:
• Specific hardware being used
• # of CPUs
• Speed of CPU
• Total memory installed
• Network connection information/speed
• Physical proximity to the App Server.
3. Other hardware being used:
• Specific hardware being used
• # of CPUs
• Speed of CPU
• Total memory installed
• Network connection information/speed
• Physical proximity to the App Server.

1. Application Server:
• Specific version of OS
• Specific version of app server software
• What JVM are you running?
• Specific JVM version (java -version from the appropriate path)
• Have you applied any special config settings to the JVM - turned off all or part of JITC, etc...
• Configuration of the app server (is the app server clustered? if so how many nodes? vertical or horizontal?)
• Max and min heap size
• Max and min JDBC connection pool size for all database connections being used in the application including the default TeamWorks connection pools.
• Max execution threads.
• Memory utilization of machine from the OS level.
• Memory utilization of the app server.
• CPU utilization of the machine(s).
• Other software on this machine:
• What else is running on this system?
• What is the load of those applications?
• Are there performance issues with that software? etc...
2. Database Server:
• Specific version of OS
• Specific version of database server software
• Configuration of the database server (is this a database cluster of some kind? etc.)
• Max connections allowed to database server
• Memory utilization of machine from the OS level.
• CPU utilization of the machine(s).
• Is this a shared database? If so what load is on the db from the other apps?
• Other software on this machine:
• What else is running on this system?
• What is the load of those applications?
• Are there performance issues with that software? etc...

There are several TeamWorks specific configuration files that should be reviewed. Lombardi Support or Services might want to review these as part of the troubleshooting process. In general we don't recommend making configuration changes to help troubleshoot potential performance issues unless you understand what you are changing and why you are making the changes.

• .../process-server/resources/log4j.xml
• Are all logging "categories" set to error?
• .../process-server/resources/config/TeamWorksConfiguration.running.xml
• teamworks log files (default location of .../process-server/logs/*.* and .../process-server/startup/logs/*.*), app server log files if applicable.

Some basic tests are typically helpful in narrowing down the scope of performance issues and isolating the root cause. Here are some tests that might be applicable depending on the performance issue you are having:

1. Test the scenario with a basic BPD with just an "Activity with the Default Service".
2. Test with the stock Default Human Service or the stock Default System Service.
3. Test static networks speeds from the app server by pulling up an image from the TeamWorks install. A good selection of images can be found here:
   .../process-server/applications/teamworks.ear/teamworks/coachImages/
   To display one of these from the browser you would go to a URL like http://process_server_host:process_server_port/teamworks/coachImages/uttower.gif

There are 2 types of logging that can be very helpful in isolating performance problems:

1. Solution code logging - Custom logging in the solution code helps to determine where the performance problems lie. This not only helps identify areas in the solution code that might benefit from performance tuning but also helps highlight the TeamWorks product areas to focus on during diagnosis.
2. TeamWorks Instrumentation logging - Instrumentation logging writes logging data and timing in a light weight binary format that has minimal to no impact on the performance of the system. These logs will be requested by support or engineering when needed. You can learn more about Instrumentation logging in the TeamWorks Administration Guide.