 | Level: Introductory James Kochuba (kochuba@us.ibm.com), Support Staff Engineer, IBM
Daniel Julin (dpj@us.ibm.com), Senior Technical Staff Member, IBM
12 Dec 2007 
There are a lot of troubleshooting resources available to help you with your IBM® software products. Here are 10 you should be sure to have in your toolbox.
From IBM WebSphere Developer Technical Journal.
In each column, The Support Authority discusses resources, tools, and other elements of IBM Technical Support that are available for WebSphere products, plus techniques and new ideas that can further enhance your IBM support experience.
This just in...
As always, we begin with some new items of interest for the WebSphere® community at large:
- Version 3 of the
IBM Software Support Toolbar
has just been released. The toolbar is a free plug-in for Internet Explorer and Firefox browsers that provides quick links to the most commonly-needed Web resources for supporting all IBM software brand products and a search engine. The new version is updated with enhanced content and search options, a new menu for quick access to common Web-based tools, and, in particular, a new
SWG Quick Reference Page
. If you have an older version of the toolbar installed, you can update it automatically through your browser’s built-in mechanism for updating plug-ins. If not, you can download it from the toolbar page.
- Registration for the
Impact 2008
conference is now open, to be held in Las Vegas April 6-11, 2008. This conference brings together thousands of WebSphere practitioners, and will, as always, include many presentations and labs specially targeted for people involved in supporting WebSphere-based applications.
- A new Redbook on
Problem Determination for WebSphere Application Server V6.1
is approaching completion, and is currently available as a series of RedPieces.
- If you liked
12 ways you can prepare for effective production troubleshooting
, which appeared in the July version of this column, or wanted to get another perspective on this topic, listen to the December 13 WebSphere Technical Exchange Webcast on how to
Prepare for Problem Determination for WebSphere Application Server
. The speaker is also a member of the team who produced the earlier article. While there, be sure to check-out other webcasts in the same series, since many of them are on problem determination topics.
- A new version of the
IBM Dump Analyzer for Java
tool has just been released in the IBM Support Assistant. This tool analyzes a system dump from a WebSphere (or other) JVM to examine its state and diagnose a variety of problems at the Java and WebSphere level. The new 1.1.0 version contains about 50 fixes and new features, particularly in the area of increased analysis functions and enhanced robustness of the underlying DTFJ JDK library used to read the dump. It also introduces the ability for you to implement and add your own customized analysis modules to this tool.
Continue to monitor the various support-related Web sites, as well as this column, for news about other tools as we encounter them.
And now, on to our main topic...
Helping you help yourself
Throughout the year, this column has presented several of the major initiatives and tools being developed within the IBM WebSphere Support community. As noted before, IBM Support tries to organize all resources through two major channels:
- The IBM Support Assistant, which provides a centralized front end for searching information from many sources and a centralized repository for problem determination tools.
- The IBM Support Web site, which serves as the primary access point on the Web for searching and browsing support information.
In general, these two front ends should be your starting point for finding the resources that you need, and we are always and systematically updating them. But we have a dynamic environment where resources are constantly being created and updated, some of which might be hosted on a variety of other sites, such as alphaWorks, developerWorks, the IBM Information Centers, IBM Redbooks, and so on. The number of resources available can sometimes be overwhelming, and as a result some great resources can occasionally be overlooked.
For the final Support Authority column of the year, we're taking a slightly different approach. As you know, there is a vast inventory of tools and utilities in the IBM support repositories to assist you with various support issues. But rather than tell you about "everything" that is available to help you, we have handpicked 10 particularly useful tools and documents that you might not be aware of. Some we have covered here before, but if any of these are new to you, check them out now and put them in your virtual toolbox so you can be ready for the new year:
-
Developing a client to determine a hung thread problem
-
IBM Trace and Request Analyzer for WebSphere Application Server
-
Trace Analyzer for WebSphere Application Server
-
IBM Thread and Monitor Dump Analyzer for Java Technology
-
New and improved JDBC test tool
-
Dynamic Cache Statistics Collector and Visualizer for IBM WebSphere Application Server
-
Sample scripts for WebSphere Application Server
-
System administration for WebSphere Application Server
-
WebSphere Application Server Troubleshooting Guide
-
Detailed system requirements documents
The sections that follow provide a short description of each of these items and links to the information on the Web.
1. Developing a client to determine a hung thread problem
Problems don't always happen when administrators are watching and waiting for them. Capturing the required documentation for a problem can be extremely cumbersome and time consuming if the system is not automated to monitor and react. This article presents a simple example of how to create a JMX program and use it to listen for a notification, specifically the "hung thread notification" generated by WebSphere Application Server when a hang condition is suspected. Once a notification is received, the tool will react by generating helpful debug documentation, like a thread dump. The article also contains example programs with instructions for configuring and executing to monitor for different types of diagnostics events exported through JMX notifications.
Example usage scenario
|
Here is a hung thread message that is logged in the WebSphere Application Server server's SystemOut.log file:
WSVR0605W: Thread threadname has been active for hangtime and may be hung. There are totalthreads threads in total in the server that may be hung.
During the periodic review of all the servers' system and native logs, the hung thread message is something you should search for. If found, then implementing this tool in your environment will help capture the Java™ thread dump to troubleshoot what the "threadname" was doing. If you determine that the thread is running normally, you might want to increase the hung thread detection timeout; if there is problem, then the thread dump is the first element of diagnostics.
|
2. IBM Trace and Request Analyzer for WebSphere Application Server
This tool helps find the source of unusual delays, slow behavior in a system, or situations when the system appears to hang and then recover. It examines traces files from WebSphere Application Server and the HTTP plug-in, identifies individual requests through these files, and points to areas where a particular operation took a long time to complete, which helps you review suspicious areas in the traces to determine if something is wrong.
Example usage scenario
|
Suppose that you observe that an application server is taking longer than usual to start. You enable WebSphere trace, collect that trace during the entire duration of the startup of the application, then load the trace into the Trace and Request Analyzer tool. The tool will quickly highlight if there are any "gaps" in the trace (that is, periods when some operation started and did not generate any further trace statements until it finally completed much later). Finding that operation will tell you where to focus further investigation, or it might tell you (based on the nature of the operation) what the problem is right away.
|
3. Trace Analyzer for WebSphere Application Server
This tool is sometimes confused with the Trace and Request Analyzer tool above, but it is in fact quite different. Where the Trace and Request Analyzer focuses mainly on scanning once through the trace to easily locate areas of particular interest or specific delays in execution, the Trace Analyzer tool is a highly interactive utility that enables you to browse through and examine complex trace files from WebSphere Application Server in detail. This tool helps bring together multiple different files and formats them to a single common view, and provides extensive navigation and search features to quickly find your way through the trace file. This tool's capabilities include:
- Visual trace presentation
- Advanced navigational engine
- Nested search and filter support
- Dynamic call sequence analysis
- Call duration measurement
- Trace highlighting and mark-up
- Entry and exit record pairing.
4. IBM Thread and Monitor Dump Analyzer for Java Technology
A common technique for diagnosing problems in a WebSphere process is to obtain and examine a snapshot of the Java process called a "javacore" or "thread dump." The javacore shows the thread and process information about a specific Java or WebSphere Application Server process at one moment in time. If WebSphere Application Server hangs, the thread dumps are extremely helpful to determine what is happening. But the information inside a thread dump can be complex and hard to interpret, so several tools were developed over time to assist with this analysis. The Thread Analyzer, which is a different tool that is available in the IBM Support Assistant, was discussed in a previous column. The IBM Thread and Monitor Dump Analyzer for Java Technology is an alternative tool, not yet available in IBM Support Assistant. It provides a different perspective from Thread Analyzer to help diagnose JVM monitor locks and thread activities so you can identify the root cause of hangs, deadlocks, and resource contention or monitor bottlenecks.
Example usage scenario
|
After implementing a program to capture the right files for a suspected hung thread (see above), you can use this tool to review the Java thread dump. One of the potential problems it might show is a deadlock: many threads competing for access to a common synchronization block that constitutes a bottleneck, or many threads that are currently waiting for a response from a remote database, and so on.
|
5. New and improved JDBC test tool
The JDBC test tool is a Java program that can verify database connectivity between a system and a database. This tool can test DB2®, Oracle®, Microsoft® SQL Server (using several drivers), Cloudscape™, Sybase, and Informix® databases. This tool will verify that the Java runtime environment and classpath are setup correctly. Not only does this tool verify the connection and configuration from the JVM to the database, but it provides information from the database (such as number of connections and arbitrary SQL statements). This is helpful for debugging WebSphere Application Server JDBC connection issues with a specific database and confirming that the client receives valid information.
6. Dynamic Cache Statistics Collector and Visualizer for IBM WebSphere Application Server
This tool helps collect statistics that provide insight into the WebSphere Dynamic Cache (DynaCache) subsystem. DynaCache is used to speed up access to common Web pages or fragments served by the application server, but it needs to be carefully configured and tuned to obtain the maximum performance benefit, while at the same time minimizing memory and disk overhead. This tool interacts with the JMX MBean running in WebSphere Application Server to retrieve statistics about the state, health, performance, composition, and efficiency of DynaCache.
7. Sample scripts for WebSphere Application Server
Scripting with the wsadmin utility is a common technique for automating many administrative and diagnostic tasks in WebSphere Application Server. The IBM Information Centers provides a number of basic scripting examples, and recent versions of WebSphere Application Server also provide facilities to assist in script generation. A collection of sophisticated administrative task scripts can serve as a helpful library of jacl or jython sample scripts. This article provides downloadable, "as is" samples scripts subdivided into four major categories for WebSphere Application Server v5.0 to v6.1. Scipts are available for:
- Automated deployment
- Security configuration
- Configuration changes
- Administration.
The article will give you a starting point for new automation administrators and guidance to veteran administrators to help automate more complex tasks.
8. System administration for WebSphere Application Server
Beyond the use of wsadmin and scripting, you might sometimes want to perform more complex functions or implement specialized tools that cannot be easily performed with simple scripts. For this purpose, WebSphere Application Server V5 and above opened the administrative JMX APIs to the public allowing you to create a custom Java administrative program. This article will help you understand how to write a JMX client for administration configuration purposes from the JMX Javadoc. The high level steps to create the JMX client configuration administrator program is included along with education on the WebSphere Application Server design. This helps provide the complete picture of how to create an administrative program and what your program will do inside WebSphere Application Server. The article includes a sample program so you can begin experimenting with a JMX client configuration administrator program.
Example usage scenario
|
Suppose you want a tool so developers can deploy applications on development machines, but your organization has restrictive processes or security requirements that make it impractical to use the WebSphere Application Server admin tools. The JMX administrative client helps you create a tool so you can execute all the steps required for the deployment and opens up the usage of the to developers without opening up the WebSphere Application Server security.
|
9. WebSphere Application Server Troubleshooting Guide
The troubleshooting guide on the WebSphere Application Server support site breaks down the product into major topic areas. A matrix guides you though the different problem components for each topic, and helps you find other relevant material as well. Each problem component has contains sections on:
-
Learning More to help educate you on the topic.
-
Troubleshooting for providing specific debugging steps you can take before calling IBM Support.
-
Collecting Data for information on how to collect the diagnostic information required for IBM Support to assist with problem determination.
-
Analyzing Data for instructions on how to interpret some of these diagnostics.
-
Contacting IBM.
Nearly all problem components offer helpful information for each of the above problem determination steps.
As an alternative to using this troubleshooting guide directly, you can also be guided through the step-by-step process by using the IBM Guided Activity Assistant, offered as a tool in the IBM Support Assistant.
10. Detailed system requirements documents
Ever wonder about the minimum level of software and hardware that is required for WebSphere Application Server (or some other WebSphere product) -- or to be supported by IBM? IBM publishes minimum level lists of specific product versions for the operating system, high availability solutions, security, Java, Web servers, databases, transaction managers, and much more, that run and interact with each version of a WebSphere product. This documentation helps identify the lowest level of patches required on a product before your WebSphere product will work properly.
Conclusion
These 10 helpful tools and documents are only a few of the items available to help you support your WebSphere systems. To find more items of this type (besides the IBM Support Assistant and the main IBM Support Web site), check out the Featured Documents Web page for each WebSphere product. These pages contain links with abstracts to the most requested IBM documents found on each product, and are updated periodically to keep up with current trends and hot issues. Each page presents information in these sections:
- In the spotlight
- Planned maintenance
- Education
- General self-help resources.
See the Featured Documents page for WebSphere Application Server for an example.
We will continue to keep you informed of major tools, new items, and any other emerging items of interest through Support Authority articles like this one. Each column is contributed by IBM technical support engineers from different organizations, and provides insight to new and existing information to further improve the support experience. Some of these articles provide high level best practices and guides to improve the troubleshooting timing for a WebSphere solution, and others provide technical details on specific hot topics for education purposes.
And don't forget to look in the products themselves! Several tools -- like the Classloader Viewer, Install Verifier, and Configuration Validator -- are now integrated in or shipped with WebSphere Application Server.
Resources Learn
Get products and technologies
Discuss
About the authors | 
| |
James Kochuba is the technical team leader for IBM support personnel whose primary focus is systems management and workload management. James' role requires focus on customer issues to help guide both support engineers and customers toward problem source identification and, ultimately, to problem resolution. |
| 
| |
Daniel Julin has 20 years of experience developing and troubleshooting
complex online systems.
As technical area lead for the WebSphere Serviceability Team, he currently focuses
on helping the team define and implement a collection of tools and techniques to
assist in problem determination for WebSphere Application Server and to maximize
the efficiency of IBM support. He also occasionally assists directly in various
critical customer support situations. |
Rate this page
| |
