WebSphere Peformance - Alexandre Polozoff's Point of View

IBM's new application monitoring solution

During the Q&A session of a customer performance presentation one of the questions asked about performance in migrating from a bare metal environment to a virtualized cloud environment. This is a really good question!  As I've said many times when it comes to performance you can not manage what you can not measure. The cloud doesn't escape the rules of Computer Science and Performance 101.

This is where IBM's SmartCloud Monitoring - Application Insight can help. The features page describes how this new application monitoring solution can provide insight into application performance and user experience. Providing capabilities such as dynamic monitoring of cloud applications for a variety of public and private cloud providers including Amazon EC2, VMware and IBM's PureApplication System (IPAS). Diagnostic drill downs and embed monitoring technology to aid troubleshooting and root cause analysis. The resource page provides articles on topics such as service management and proactive application management. A wiki on best practices, a forum for technical questions, a blog for the latest news and updates and a community to engage with IBM experts or peers at other enterprise organizations is all provided from the community page. Finally, documentation and an extensive knowledge base can be found on the SmartCloud support page.

Why is this important? Primarily because clouds are comprised of many VM instances. If not properly configured or tracked the underlying resources the VMs need (i.e. CPU, RAM, network, disk, etc) can easily be over committed resulting in perplexing performance problems. Inside the VM everything may look nominal but without visibility into the cloud itself troubleshooting is near impossible prolonging the negative performance and end user experience. No one wants to prolong a bad end user experience.

In addition, production monitoring data can be fed back into the cloud capacity planning organization. This allows for production data to be used in their calculation models to maintain Service Level Agreements (SLAs) and availability requirements. Cloud infrastructure, while seemingly boundless, can suffer from resource availability as applications grow and mature if no one is monitoring the environment.

From an administration and infrastructure perspective cloud technologies are presenting new and exciting technologies to further simplify those tasks. Last year when I was working on the IPAS performance team I was really enthused by patterns and the powerful capabilities behind them especially where consistency and repeatability is a must. Even more impressive is the symbiotic integration of various IBM technologies like in the mobile space with Worklight developing their IPAS support and mobile application platform patterns.

[Edit to correct typo and added tags and title and a couple of links to more specific content]

I had a blog post on socketRead issues causing hung thread in this blog post and using timeouts when communicating with a database back in 2009.  These problems don't happen very often anymore as networks and databases tend to run on fairly robust environments.  You can imagine my surprise when late last week I received an email from a colleague working with an application suffering from the same symptoms in that blog.

The reason for setting the timeouts is to be able to fail fast as opposed to the application appearing to be non-responsive. 

However, what to do if the timeout doesn't seem to be kicking in?  In this case the first thing to do is to open a PMR with IBM Support. 

Data really needs to be collected in at least two places.  On the application server and on the database. 

For the database http://www.ibm.com/developerworks/data/library/techarticle/dm-0812wang/ provides information on how to use db2top to collect data when the problem is occurring. 

Then for the application server (this happens to be for BPM) http://www-01.ibm.com/support/docview.wss?uid=swg21611603 there are various sets of mustgathers to collect data in order for IBM Support to run analysis on. 

Networks can also have hiccups and by running tcpdump on both the application server and database side one can use various protocol inspection tools like Wireshark to look at the underlying network communications to see what, if any, problems may be occurring there. 

Perennially I stumble into the question of when is the performance tuning effort finished?  How do you know if enough has been done or there is still further to go? 

Personally, I think this should be dictated specifically around Service Level Agreements (SLAs).  If the throughput / response time is within the SLAs then we are done tuning.  Any more tuning should only be done if the application (in a new release) starts slipping SLAs or the SLAs themselves have changed.

But not every one has defined or agreed upon SLAs which muddles the water.  Say, for example, we're building a new application.  We have a number of components that we're performance testing but we don't know if the results we are seeing are the best results that can be achieved.  Well, that makes sense.  If we have no experience working with a particular end point (i.e. database, messaging engine, etc) then obviously our lack of knowledge with that package or our application code will hamper the analysis.

This is where application monitoring comes in.  Properly installed and configured the application monitoring tool should be able to provide enough information as to whether we have the best throughput and response time.  Look at the metrics, compare CPU and memory utilization.  Make sure all the resources under load are as close to maxed out as possible without introducing contention.  Look at the throughput / response time graphs as load is ramped up and then how it does after reaching steady state.  Once all these have come together you are probably ready to go the next step and over saturate the environment.  Once you've proven that over saturation has occurred (because of the noted degradation) then you can probably tune no better.  And it will probably take a lot of tuning work to get to the point where you are able to saturate all the resources.  This will be through a variety of methods like increased thread pools, vertical/horizontal scaling out, increased memory settings, garbage collection policy analysis, changing application code, etc. 

However, is it worth it?  Does an organization really want to pay for the most optimal tuning when it may not be necessary?  These efforts can be long and drawn out (i.e. months instead of weeks).  It also can tie up a lot of highly technically skilled people on an effort that provides for some unknown possible improvement.  Lets look at the only scenario I can think of (can you think of another?). 

High volume applications tend to also be resource hogs requiring gobs (i.e. hundreds or thousands) of JVMs and the supporting infrastructure.  This can get expensive real quick especially when factoring in all the administration, configuration, maintenance costs are factored in.  In this one scenario I can see justification, even beyond defined SLAs, to have as best performance as possible in order to reduce the capacity needed to support the application. 

 Sessions

Suggested: Session Active Count

Optional: Session size

PrepStmtCacheDiscardCount (for JDBC)

ActiveCount, CommittedCount, RolledBackCount

ProcessCpuUsage