TeamWorks Performance Tuning Tips

Resolving The Problem

The following are just guidelines. The settings that are right for you will be highly dependent on your hardware and the types of processes you run. You should also consult the documentation that came with your app server. Teamworks is running on the app server, so vendor specific app server tuning is very important.

Memory Tuning:

(Note that the appropriate max memory settings may vary depending on your OS, available memory, TW version, and JVM version)

In general, for BEA WebLogic and JBoss installs: Edit the wrapper.conf for the performance server and set it to:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=768
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=768

Edit the wrapper.conf for the process server and set it to:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1024

Setting the min and max heap size equal is generally recommended, but check with your JVM vendor and performance tuning tips for your specific JVM for specific recommendations on your configurations.

Edit your log4j.xml for the process and performance server.
Make sure ALL categories are set to "error".
The appenders can be set to other log levels as long as the categories are set to error.

Specific to your Application Server, make sure ALL the application server logs are to "error". Please consult your application server vendor to determine how to set the application server log level. Note that for TeamWorks installs that use JBoss you will want to modify the log4j-jboss.xml files located in the <TW_Install_Dir>/process-server/conf/ and <TW_Install_Dir>/performance-server/conf/ directories. See KL #160 for additional info on changing the JBoss log levels.

See Understanding and Tuning the Event Manager.

As documented in the TeamWorks Install and Config Guide please make any changes to the TeamWorks xml configuration files via the appropriate 100Custom.xml file.

From the <TW_Install_Dir>/process-server/config/system/00Static.xml:
• <skip-uca-sst> (default = true) - Determines whether Undercover Agents have access to the deprecated Shared Symbol Table. If you&rsquo;re not using the Shared Symbol Table, setting this to true will speed up Undercover Agent execution and reduce some contention on the lsw_uca table. Lombardi recommends that this be set to true on production systems.
• <maintain-uca-last-run-time> (default = false) - Determines whether TeamWorks updates the "last_run_time" in the lsw_uca table. This is not used internally, but it is displayed in the Authoring Environment. Setting this to false will reduce contention on the lsw_uca table. Lombardi recommends that this be set to false on production systems.
• <lock-event-manager-tables> (default = false) - Determines whether TaskLoader and cancelTasks need to grab a table lock on lsw_em_task before performing their work. This should be set to false, to increase the parallelism of these activities across a cluster of process servers, but this parameter allows us to revert back to the "old" locking behavior. You should never change this unless directed to do so by Lombardi Support.
• <cache-task-categories> (default = false) - This parameter applies specifically to the code used to populate the "old" Task Manager (from TW5.0.x and prior) folder list. It specifies whether Task Manager should use a single cached list of categories for all folders (each folder will contain the same list of categories). If this is true, this cache will be refreshed every <category-cache-refresh-interval> seconds; otherwise, TeamWorks will get the list of categories for each folder as before. If you enable this, you will reduce database traffic and increase overall throughput. The trade-off is that (1) certain categories may not appear until the cache refreshes and (2) all categories will appear under each folder even if no tasks in the folder are assigned to that category.
• <category-cache-refresh-interval> (default = 300) (300 seconds = 5 minutes) - This parameter is the number of seconds the category cache will be refreshed.
• <skip-task-counts> (default = false) -This parameter applies to the task list display of the "old" Task Manager from TW5.0.x and prior. At the top of the task list, TeamWorks shows the total number of tasks in the current folder and category, and the total number of overdue tasks in the current folder and category. If this setting is true, TeamWorks task list display performance will improve with the following changes in behavior: (1)"0" will be displayed in place of the current counts and (2) the "Last" link will be disabled. If this setting is false, the behavior will be unchanged from TeamWorks 5.0.x.
• <all-classes-refresh-interval> (default = 300) (300 seconds = 5 minutes) -This parameter determines the number of seconds between server-side refreshes of the Variable Type cache. This cache is used when deserializing TWObjects from XML, which happens when invoking an Undercover Agent, invoking a system-lane activity, or delivering a message to a BPD instance. In TW 5.5.5 and prior, the recommended setting for development environments is 0 because Variable Types change often in development. The recommended setting for production environments is between 300 (5 minutes) and 3600 (1 hour). Note that if you import a Variable Type into an environment that has this parameter set at a greater than 0 value, you will have to wait that amount of time before the cache is correct. For example, if you have the parameter set to 300 and you import a Library File with a changed Variable Type 'foo', then you must wait 5 minutes to be certain that 'foo' is correct in the cache.In TW 5.5.6 and beyond the behavior of this cache has changed. It is no longer necessary to set the variable to a very low number because the cache is refreshed on the save of a variable type. There is also now a "Runtime TWClass Cache" reset option on the "Manage Caches" page in the TeamWorks Admin Console.
• <create-sent-tasks> (default = false) - When a task is sent from one user to another (or from a service running as one user to another), TeamWorks creates two tasks: one that shows up in the recipient's Inbox folder, and one that shows up in the sender's Sent folder. This Sent task is sometimes useful so that the sender of the task can see its details and is aware that they sent a task; in an environment where this information is unnecessary, it's useful to be able to turn off this behavior to reduce the growth of the LSW_TASK table and related tables. If these sent tasks aren't important, setting this to false will reduce the growth of the process server database's LSW_TASK table (and related tables) and improve performance.

From the <TW_Install_Dir>/process-server/config/system/80EventManager.xml:
• <kick-on-schedule> (default = true) - When set to true a newly-scheduled task will force the Event Manager into an immediate poll of lsw_em_task, to reduce the time between when a new task is scheduled and when it will be executed. This helps with latency - a newly-scheduled "right now" task will be executed almost immediately - but hurts overall throughput, because the TaskLoader ends up being more active than it would be otherwise. If kick-on-schedule is false, newly-scheduled tasks won't be picked up until the next time the Event Manager polls lsw_em_task (up to the loader-long-period), which will increase latency, but will also increase overall throughput, by reducing the chatter and contention on the lsw_em_task table. For a system with heavily loaded Event Manager, this setting should be set to false.

From the <TW_Install_Dir>/process-server/config/system/99Local.xml:
• <cached-objects-ttl> (default = 0) (seconds) - This controls the version checks that occur against the TeamWorks Persistent Object cache. By default this is set to 0 meaning that any lookups from the cache are checked against the current version in the DB. We have found that this is a light weight call; however, in an environment where the objects should not be changing, such as production, you can increase this ttl setting and gain some performance improvement.Note that a non-zero value for this variable means when changes are made to your processes in production (including imports) you either need to wait the number of seconds that cached-objects-ttl is set to or you need to restart the process server before the system would be in a stable state. Note that in versions of TeamWorks prior to 5.5.6, this setting also effects EPVs. Any changes to EPVs will not be picked up by the system until the cached-objects-ttl has passed.
• <reloadable-jar-location-load-only-once> If you are using the dynamic classloader make sure that the reloadable-jar-location-load-only-once value is set to true. If false performance will be drastically impacted.

If you are using MQ, have your MQ admin set MaxChannels and MaxActiveChannels to at least 1000

Increase the PersistenceServices EJB max-beans-in-free pool setting to 100. To do this in WebLogic:
• unjar process-server/applications/teamworks.ear/twcore-ejb.jar
• Edit meta-inf/weblogic-ejb-jar.xml and change <max-beans-in-free-pool> from 20 to 100
• Put the changed file back into twcore-ejb.jar

When running TeamWorks under a high load a large number of database connections can be used. One rule of thumb for the value of available connections in your JDBC max database connection pool size is to set it to at least 2x the <max-thread-pool-size> that is set in the 80EventManager.xml (or 100Custom.xml). Note that depending on your solution code and the user load more connections than this might be needed. Also please note that this is per app server node in a cluster, so the total number of connections available at the database will need to equal the max connections for the pool * # of nodes in the cluster.

In WebLogic we have found it useful to set the min and max connection pool size to the same value. This is a BEA recommendation and more information can be found about it here:http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/WLSTuning.html#1140691

In the .../startup/wrapper.conf set the wrapper.ping.timeout = 0. This will prevent the system from restarting when it cannot ping the JVM for some reason. If the wrapper can't ping the JVM this usually means that there is some kind of problem with the system or the system is very busy doing something. In either case if the system restarts itself we will have a hard time troubleshooting the real problem. You should set this on all managed servers for both the process and performance server. Note that as of TW 5.5.4 this is the default setting for this timeout.

• Change the Thread Count for the Execute Queue from 20 to 60.
  You do this from the WL console screen: processerver>servers>TeamWorksServer(or what ever your managed server is named)
• Click Show Advanced option (under the Configuration/ General tab)
• Click Configure Execute Queues at the bottom of the screen
• Click on the weblogic.kernel.Default queue
• Increase the thread count to the appropriate value.
• Change the JMS Thread Pool Size from 15 to 50.
  You do this from the WL console screen: processerver>servers>TeamWorksServer(or what ever your managed server is named)
• Click the Services tab
• Click the JMS tab
• Increase the JMS Thread Pool Size to the appropriate value.
Note: These changes will not take affect until after you restart the WebLogic server.

Make sure that JITC is enabled for the IBM JVM (typically used with the WebSphere App server or other App Servers running on AIX).

Contact your app server vendor for any specific requirements specific to your OS.

For WebSphere Application Server tuning information (such as JVM and kernel parameters) please review:
• IBM RedPaper: "WebSphere Business Integration V6 Performance Tuning"
• IBM Redbooks publication: "WebSphere Application Server V6 Scalability and Performance Handbook"