![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
In dealing with automobile accidents, a customer submits his claim to his insurance company. A claims adjuster usually is assigned to a claim to investigate. The customer usually receives a phone call, a letter, or an e-mail from the adjuster introducing herself and informing the customer of the general process her company follows. The adjuster then reviews the customer's policy to see what coverage and deductibles he has and evaluates the type and extent of damage, injuries, or both that resulted from the accident (photos will also be taken). In most cases, the adjuster may simply ask the customer to get an estimate for the cost of repairing the car and pay the customer a final amount. The end result is that a check gets written to the appropriate parties. The case is then closed.
In all of these steps, updates are made to a FileNet content management system—records are created and updated, and photos are uploaded and tagged for each claim as the process moves forward.
The workload used in this article simulates the above scenario. The workload consists of a client Java™ test driver implementing the FileNet Content Engine APIs. It supports concurrent user sessions. It can also be executed in separate client machines, all of which drive the load onto a single FileNet back end. The back end consists of multiple instances of FileNet Content Engine application servers. All of the Content Engine's servers connect to DB2 9.7 databases that serve as the repository of metadata for all the claim documents.
The workload's mix is as follows:
- 10% - Create claims documents
- 45% - Modify / annotate claim documents
- 10% - Browse all claims
- 15% - Search claims
- 20% - Retrieve claim documents
Figure 1 illustrates the test topology:
Figure 1. System under test
Table 1 lists the details of the hardware and software levels used in this study:
Table 1. FileNet system under test for this article
| Tier | Hardware | Software |
|---|---|---|
| Load drivers | IBM xSeries x326 2xCPU 4GB RAM InfiniBand | SuSe 10 SP2 Linux; FileNet Java Test Driver |
| FileNet P8 WebSphere Server | Intel Xeon® X5570 CPU X2 18GB RAM SSDs InfiniBand | Microsoft® Windows® Server 2008 R2; FileNet P8 V4.5.1; WebSphere V7.0 cluster |
| DB2 database server | Intel Xeon® X5570 CPU X2; 6GB RAM InfiniBand SAN | Windows Server® 2008 R2; DB2 V9.7 |
FileNet Content Engine offers two types of APIs:
- Java 4.0 API that uses EJB transport (RMI/IIOP for WebSphere)
- Web Services Interface (WSI) that uses HTTP transport
Both transports are supported in the Content Engine's jace.jar client code. As a client application, the test driver uses the Java APIs, and therefore uses the EJB transport.
WebSphere supports EJB transport by using Object Request Broker (ORB). The ORB is used to manage communications between client applications (in this case, the test driver written in Java) and server applications (in this case, FileNet Content Engine application server), as well as among WebSphere components themselves.
The ORB manages inbound and outbound requests for remote Java technology objects, like EJB components. ORB provides a framework for clients to locate EJBs on the server and invoke operations on those as if they were local. Such requests are made using messages. Messages can be large or small, depending on the workload. A more complex workload, such as the insurance scenario in this article, requires ORB message fine-tuning. Default ORB fragment size can serve well for a lot of workloads, but a fine-tuned ORB message size can boost performance in addition to database tuning and other WebSphere tuning, such as JDBC connection pool size and Web thread pool.
Some workloads running in a FileNet Content Engine environment can present an interesting resource usage pattern: there is no disk I/O bottleneck, no memory limitation, no client-driver performance issues, yet the CPUs are under-utilized. For example, in the insurance scenario, prior to ORB fragment size tuning, maximum-achieved CPU utilization is about 55%, as shown in Figure 2:
Figure 2. CPU usage before tuning ORB fragment size
This article focuses on tracing and tuning the ORB messages in the FileNet Content Engine application server so as to increase CPU utilization.
Turn on ORB tracing in WebSphere
To enable WebSphere ORB tracing, follow the steps below.
- In the WebSphere Administrative Console, navigate to Application servers > server1 > Container Services > ORB \ service.
- On this properties page, select the ORB Tracing check box to enable ORB tracing. You should also see general ORB properties such as Request timeout, Request retries count, and so on, on the same page.
- To set the appropriate PMI tracing level, navigate to Troubleshooting > Logs and Trace > server1 > Diagnostic trace service > Change log detail levels.
- Click on the ORBRas, and select All Messages and
Traces. You should see the settings box showing an entry
similar to
*=info: ORBRas=all. Here, you can turn on other traces if desired.Use log levels to control which events are processed by Java logging. Click on Components to specify a log detail level for individual components, or click on Groups to specify a log detail level for a pre-defined group of components. Click on a component or group name to select a log detail level. Log detail levels are cumulative; a level near the top of the list includes all the subsequent levels.
- Remember to disable tracing by changing the setting back to No
Logging after troubleshooting.
There is no need to restart the application server if you are using WebSphere V6.1 or later. You will find a trace file normally under /usr/IBM/WebSphere61ND/AppServer1/profiles/AppSrv01/logs/server1/trace_xxx.log.
When tracing is turned on, a lot of entries will be logged. You can lower your workload by reducing the number of users and duration of the runs. The key is to be able to collect enough samples (around 10 transactions in this case) so that a good coverage is achieved in terms of ORB requests. Listing 1 shows a sample of ORB tracing output:
Listing 1. Sample ORB traces collected from running test workload
11/2/09 11:40:40:671 PST] 00000040 ORBRas 3 com.ibm.rmi.ras.Trace dump:81 RT=2: P=364261:O=0:WSTCPTransportConnection [addr=10.0.1.68,port=41312,local=56772] IN COMING: Fragment Message Date: November 2, 2009 11:40:40 AM PST Thread Info: RT=2:P=364261:O=0:WSTCPTransportConnection[addr=10.0.1.68, port=41312,local=56772] Local Port: 56772 (0xDDC4) Local IP: 10.0.1.238 Remote Port: 41312 (0xA160) Remote IP: 10.0.1.68 GIOP Version: 1.2 Byte order: big endian Fragment to follow: Yes |
In the above listing, Request ID: 1219 has
several fragments in a message. It is easy to determine how many fragments
are in the request by running a grep (UNIX) or
find (Windows) command. For example:
Listing 2.
find commandc:\>find "Request ID: 1219" trace_09.11.02_11.40.43.log ---------- TRACE_09.11.02_11.40.43.LOG Request ID: 1219 Request ID: 1219 Request ID: 1219 Request ID: 1219 Request ID: 1219 Request ID: 1219 |
As the result, the request with ID 1219 had six fragments—five fragments are 1012 bytes, and one fragment is 160 bytes. Let's take a look at another request (see Listing 3):
Listing 3. Sample ORB traces collected from running test workload
Fragment Message Date: November 2, 2009 11:40:41 AM PST Thread Info: ORB.thread.pool : 0 Local Port: 56772 (0xDDC4) Local IP: 10.0.1.238 Remote Port: 41312 (0xA160) Remote IP: 10.0.1.68 GIOP Version: 1.2 Byte order: big endian Fragment to follow: Yes Message size: 1012 (0x3F4) -- Request ID: 1154 0000: 47494F50 01020207 000003F4 00000482 GIOP............ 0010: FFFFFFFD 00000001 00BDBDBD 7FFFFF0A ............... 0020: FFFFFFFF FFFE21C4 00000014 00000010 ......!......... 0030: 00000000 51003010 00030000 00012315 ....Q.0.......#. 0040: FFFFFFFD 00000065 00BDBDBD FFFFFFFF .......e........ 0050: FFFE3878 00010002 00BDBDBD FFFFFFFF ..8x............ 0060: FFFE3930 00BDBDBD 00130008 00BDBDBD ..90............ 0070: FFFFFFFF FFFE3964 00110006 00BDBDBD ......9d........ 0080: FFFFFFFF FFFE39A4 00010006 00BDBDBD ......9......... 0090: FFFFFFFF FFFE39D0 00000001 00010005 ......9......... 00A0: 00BDBDBD FFFFFFFF FFFE3A08 00BDBDBD ..........:..... 00B0: 7FFFFF0A FFFFFFFF FFFE2130 00000014 .........!0.... ...... |
A similar analysis shows that Request ID 1154 has 117 occurrences. At 1012 bytes for each fragment, this request had 118,170 bytes (117 x 1012) in message size. ORB messages are sent in serialized fashion, making this request lock on the ORB connection longer than shorter requests. Under a heavy load, this can contribute to queuing at ORB communication between the Java client and the FileNet WebSphere applications, and this was the cause for under-utilized CPUs (as results will show later).
As the analysis above shows, ORB fragment size determines how many fragments in a request have to be sent on the ORB connection. The more fragments there are in a message, the longer an ORB connection is needed. A balance between fragment size and the length of a average request can boost performance. Certainly, if all requests in a workload are uniform in size, it would be easy to set the fragment size. Usually a workload is more complex, like the insurance scenario where create, update, search, browse, and retrieve transactions are mixed together, and the workload is executed under concurrent users. The size of documents can vary, too.
So, how do you determine the optimum ORB fragment size?
You can take the approach of determining an average message size. By running the workload for a reasonable amount of time—one hour—we collected a large amount of ORB traces. This "peak hour" approach not only covered documents with various sizes, but it also made sure that all transactions were well-represented in this duration. When we analyzed the ORB traces, we found that the largest message size is 120KB, the smallest 6KB, and the average size is 40KB. The distribution shows that majority of the ORB messages are between 30K and 48K. This tells us that our initial ORB Fragment Size of 1012 was simply too small, and that caused a large amount of fragments to be created, which bottlenecked on the ORB connection.
There are two places you must set this value: server and client. On the
server, you can specify new values for the ORB custom properties in the
administrative console. Any value you specify takes precedence over any
JDK or WebSphere Application Server default values for these properties.
The ORB custom properties settings that you specify in the administrative
console are stored in the WebSphere Application Server server.xml system
file and are passed to an ORB in a properties object whenever an ORB is
initialized. To use the administrative console to set ORB custom
properties, in the administrative console, click on Application
Servers > Application Servers >
serverName > ORB Service >
Custom Properties. Set a custom property with name
com.ibm.CORBA.FragmentSize to your desired
value. In our case, we set it to 40K, which was the average size for the
majority of the ORB messages. The following are the specifications for
com.ibm.CORBA.FragmentSize:
- Units: Bytes
- Default: 1024
- Range: From 64 to largest value of Java integer type that is divisible by 8
This parameter should also be set on the client side with a
-D system property if using a stand-alone Java
application. We appended our Java test driver with
-Dcom.ibm.CORBA.FragmentSize=40000.
The reduced number of ORB fragments in turn reduced the contention in ORB connection. This is reflected in the difference in CPU usage. Figures 3 and 4 below show the before and after screens:
Figure 3. FileNet Content Engine CPU usage: Before
Figure 4. FileNet Content Engine CPU usage: After
Table 2. Difference in throughput
| Throughput | Before | After |
|---|---|---|
| Transactions per hour | 1.7 million | 2.8 million |
In the performance environment, the FileNet application server and the database server reside on multi-core, multi-socket servers. Both servers run Microsoft Windows Server 2008 R2. Windows Server 2008 R2 comes with some new features that will impact performance. At minimum, you should evaluate the following areas:
- Server role - select the appropriate pre-allocated resources per role (for example, Directory Server, as opposed to Application Server)
- Server core - Use server core if possible; OS without UI to increase performance and improve security
- WSRM (Windows System Resource Manager) - Use WSRM to manage resources allocated to applications
- Built-in network improvement over previous versions of Windows
- Disk write-caching - substantial improvement in storage performance
- Power management - inactive processor suspension to save power consumption
DB2 9.7 built-in FileNet Registry settings
There are several useful performance parameters that are built into the DB2
engine for the FileNet applications. All of them can be automatically set
with the DB2_WORKLOAD registry variable set to
FILENET_CM. The actual command and output are
as follows:
Listing 4.
DB2_WORKLOAD=FILENET_CM output%> db2set DB2_WORKLOAD=FILENET_CM %> db2set DB2_WORKLOAD=FILENET_CM DB2_SKIPINSERTED=YES [DB2_WORKLOAD] DB2_USE_ALTERNATE_PAGE_CLEANING=YES [DB2_WORKLOAD] DB2_EVALUNCOMMITTED=YES [DB2_WORKLOAD] |
DB2 9 has the autoconfigure feature that can simplify database tuning for a
particular workload. The feature calculates and displays initial values
for the buffer pool size, database configuration, and database manager
configuration parameters, with the option of applying these recommended
values. For example, you can specify
workload_type to be simple, mixed, or complex.
DB2 will automatically adjust buffer pool size and memory allocations. It
takes into account that simple workloads tend to be I/O-intensive and
mostly transactions, whereas complex workloads tend to be CPU-intensive
and mostly queries. The following code snippet is an example of running
autoconfigure after a database is created:
db2 autoconfigure using mem_percent 95 workload_type mixed tpm 45000 admin_priority \ performance apply DB only |
Other WebSphere tuning parameters to consider for FileNet workloads:
- Increased thread pool size of EJB objects in WebSphere.
- Increased JDBC connection pool size in WebSphere.
- Increased JVM heapsize.
- Disabled RC4 and MD5 calls in WebSphere.
- Created additional Content Engine JVMs to reduce heap/vector lock contention.
- On Windows, apply JVM garbage collector -Xgcpolicy:gencon settings to reduce GC periods. For other platforms, please refer to the P8 Performance Tuning Guide in the Resources section.
This article defined the FileNet workload used to tune ORB fragment size in order to boost CPU utilization. It showed how workloads determine optimum performance settings in FileNet applications and that there is never "one size fits all" when it comes to performance tuning. This article also provided the detailed application server runtime logging and tracing with vital data points needed to provide useful analysis. Finally, it provided the exact places to tune ORB messaging and additional tunings in Windows Server 2008 R2, DB2, and WebSphere.
The author extends his thanks to Christopher J Blythe, Tim Morgan, and Dave Royer for providing guidance and clarification on the topics discussed in this article.
Learn
- "Troubleshooting: FileNet Content Engine Performance" (IBM
technote, December 2009): Performance problems in Content Engine could
come from a wide range of causes. Explore the possible issues to look into
when performance problems happen.
- "Understanding how EJB calls operate in WebSphere Application Server
V6.1" (developerWorks, July 2008): Gain a basic understanding of
how EJB communication works in the context of WebSphere Application
Server.
- "DB2 Database
AUTOCONFIGUREcommand" (IBM DB2 Database for Linux, UNIX, and Windows Information Center, March 2010): Understand the DB2 DatabaseAUTOCONFIGUREcommand. -
PerformanceWiki: Find quick
platform tuning tips.
- developerWorks
Information Management zone: Learn more about Information
Management. Find technical documentation, how-to articles, education,
downloads, product information, and more.
- Product documentation for FileNet P8 Platform: Find the IBM FileNet P8 Performance Tuning Guide and other useful product documentation.
- Stay current with developerWorks technical events and webcasts.
Get products and technologies
- Build your next
development project with IBM trial
software, available for download directly from
developerWorks.
Discuss
- Participate in the discussion forum.
-
WebSphere Application Server forum: Discuss WebSphere Application
Server issues.
- Participate in developerWorks
blogs and get involved in the My developerWorks
community; with your personal profile and custom home page, you
can tailor developerWorks to your interests and interact with other
developerWorks users.
Jesse F Chen is a senior software engineer in the Information Management group with a focus on software performance and scalability. He has 12 years of IT experience in software development and services. His recent projects include FileNet and DB2 integration and optimization, Master Data Management, and WebSphere Product Center. Jesse is also a frequent contributor and speaker at multiple ECM conferences on topics of performance, scalability, and capacity planning. He earned a Bachelor's degree in Electrical Engineering and Computer Science from the University of California, at Berkeley, and a M.S. degree in Management in Information and Systems from New York University.
Table of contents
Next steps from IBM
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
