WebSphere Peformance - Alexandre Polozoff's Point of View
This is the page to follow if there seem to be any Maximo performance or stability problems.
polozoff 110000N2A2 Tags:  tcp connection ip over fail refused windows poor performance 2,910 Views
As I travel the world working performance problems I never see Microsoft Windows environments used outside the developer's desktop. Surprisingly these past couple of weeks I've been working in an environment where Microsoft Windows is used for the IBM HTTP Server tier with the WebSphere Application Server plug-in. Under normal operating conditions everything seems to work nominally.
However, much to my surprise, if we took down any of the application servers in the cluster of this very large cell I saw an anomaly. When the plug-in was attempting to route traffic to the downed application servers there seemed to be a really long lag on the connection refused processing. In fact, I was seeing least a second to get through the TCP/IP roundtrip. This made no sense to me. One of my colleagues, Keys Botzum, took a Java application and ran it on both Windows and Solaris. The application simply tried to connect to localhost (to eliminate any DNS lookups or network latency from the test) on a port no one was listening to and looped around 20 times. On Windows the test took slightly over 20 seconds. On Solaris, less than a second (which was the behaviour I was expecting on Windows).
If you are, or planning to, use Microsoft Windows on the IHS tier be aware of this strange failure scenario on Windows. I'll try to investigate and see if there are any Windows settings to help tune this. Though the plan is to move off Windows to Redhat Linux which right now sounds like the right move to me.
IBM provides a tool to help with analysis of client side performance. One of the benefits for doing this is to help identify static objects that are not being cached at the browser. In addition, for JSF based applications, this tool helps identify large client side caches.
polozoff 110000N2A2 1,718 Views
I am often asked specifically what metrics should be monitored in WebSphere Application Server. This list can be considered a starting point. The WAS admin will frequently use "Custom" PMI setting and disable any metrics not in this list.
Suggested: Pool Size, Active Thread Count
Optional: ActiveTime (avg. time in use), ConcurrentHungThreadCount.
For WESB, per-mediation: Mediation.ThreadCount
Database - JDBC Data Source Connection Pools.
Messaging - JMS Queue Connection Factory Connection Pools.
Suggested: PoolSize, WaitingThreadCount, WaitTime,
Suggested: BufferedReadBytesCount, BufferedWriteBytesCount
Suggested: AvailableMessageCount, LocalMessageWaitTime
Transactions (per application server)
Memory Management. (Java GC).
Suggested: Heap Size, % Free after GC, % Time spent in GC.
On Solaris this tuning page has the details that I often follow for Option 1.
The past few weeks meeting with various WebSphere Application Server-based customers reminded me of the importance of the basic and fundamental performance tuning tasks. The InfoCenter provides information on tunings at the OS level, TCP/IP, JVM, etc. I have visited no less than 3 different environments running WebSphere Application Server without these base tunings. Just by applying the base tunings to the OS and JVM we saw as much as 99% less garbage collection, improved response time, throughput and less CPU utilization with the same production loads. The best part of following these instructions is the administrator does not need to be a performance guru to realize these gains. These improvements also help save money requiring less capacity going forward.
In the "Tuning the JVM" section I have never been disappointed with the "Option 1" settings. Options 2 and 3 require the ability to place the application under load/stress test. If you do not have a load/stress test environment (i.e. you have to test in production) then stick with "Option 1".
Notice that "Tuning Performance" has several sections for both application developers and WebSphere administrators. This is because we all know that to realize the best performance gains one has to optimize the application code. Runtime tuning can realize 5-15% but application code improvements can see 300% and higher performance improvements.