In each column, The Support Authority discusses resources, tools, and other elements of IBM Technical Support that are available for WebSphere products, plus techniques and new ideas that can further enhance your IBM support experience.
This just in...
As always, we begin with some new items of interest for the WebSphere community in general, including several new interesting problem determination tools that have been released in the past month:
The new Extensible Verbose Toolkit is a tool for visualizing and analyzing many complex aspects of verbose GC logs from a JVM, and for diagnosing various issues related to garbage collection behavior, such as memory leak, poor performance, and so on. This toolkit is available through the IBM Support Assistant.
The new IBM Trace and Request Analyzer for WebSphere Application Server analyzes trace files from the application server and HTTP plug-in to help track down the source of delays in the processing of requests and other activities in the server. This utility is available through alphaWorks.
The new Dynamic Cache Statistics Collector and Visualizer for WebSphere Application Server collects and visualizes statistics that provide insight into the state, health, performance, composition, and efficiency of the dynamic cache subsystem in WebSphere Application Server. This utility is available through alphaWorks.
The uses of the Log and Trace Analyzer continue to expand. A recent developerWorks article series called Build a framework for problem determination triage provides an excellent look at using a lightweight version of this tool: the Log and Trace Analyzer for Java™ Desktop.
The IBM Support Assistant team has launched the ISA Jump Start Program. Customers enrolled in this program will be guided through the various steps of analyzing, customizing, and deploying IBM Support Assistant and its features in a manner that is specific to their own environment. To find out more about enrollment, contact the IBM Support Assistant team.
Continue to monitor the various support-related Web sites, as well as this column, for news about other tools as we encounter them.
And now, on to our main topic...
What is the IBM Guided Activity Assistant?
As an IBM customer, there is a lot of information, tools, and recommended processes available to you to help you perform troubleshooting tasks. However, it may not always be clear which tool or process to use, or where the information is that you need. The IBM Guided Activity Assistant (IGAA) is a new tool that brings together all three of these support elements -- information, tools, and processes -- to help you solve problems in an easier and more consistent manner. IGAA takes you step-by-step through the IBM-recommended problem determination process and exposes you to information, utilities, and best practices at the points where they are appropriate and relevant.
IGAA aims to help you answer:
- What should I do next?
- What diagnostic data should I analyze?
- What tool should I use to analyze it?
- How do I install and interact with that tool?
and other support-related process questions.
IGAA helps you at every step of a problem determination scenario.
If you are a new user and want to understand what step to take next, IGAA can explain to you in detail what next step is appropriate, and provide supporting information to give you a better understanding of the problem and the recommended solutions. IGAA does this with the philosophy of Guide first, educate later. This means IGAA tries to expose you to enough information to help you work through the current problem, but not too much that it requires extensive reading just to accomplish the first step. After you have resolved your problem, you can then retrace your steps to educate yourself at a deeper level by following designated links out to appropriate Redbooks, Information Center pages, IBM Education Assistant modules, and other resources.
If you are an advanced user and want a quick reminder of the problem determination steps to take, IGAA helps you by only showing the information necessary to move you quickly through the steps. At each point along the path, additional information is only a click away if you need specific details about any step in the problem determination workflow.
While the primary goal of IGAA is to guide you through the process of problem determination, IGAA also helps you stay organized as you work through the problem determination process. At times, you might need to collect diagnostic data for deeper analysis. Keeping these files organized and associated with your problem determination session helps to keep the process efficient. You can upload the files or link to them if they are too large to make copies; those files can then be passed seamlessly to the appropriate tools that are recommended in the flows.
If IGAA does not have a flow that helps you solve your problem, it can help you engage the appropriate IBM support representatives. IGAA will quickly bundle up the information you have already gathered and send that information up to IBM Support, who can then pick up right where you left off, saving a significant amount of information gathering time.
Now that you know what IGAA is at a high level and how it can help you, the next section will help give you an even better understanding of the value of IGAA with a real world example of how you can leverage IGAA to help solve problems.
Anatomy of the IGAA screen
IGAA is available as a tool delivered through the IBM Support Assistant and must be installed into the IBM Support Assistant prior to use (see Frequently asked questions about the IGAA). Assuming that has already been done, you can find IGAA simply by starting IBM Support Assistant, navigate to the Tools panel, and select one of the products for which IGAA is currently available (for example, WebSphere Application Server). You will then see a list of tools available for that product, and IGAA will be one of these tools. Once you start IGAA, you will see a window similar to Figure 1.
Figure 1. IBM Guided Activity Assistant layout
IGAA's default layout consists of three primary sections:
Quick Views (blue section)
Quick Views are small expandable areas that provide session management and other capabilities for IGAA users. This section consists of:
- Saved Sessions: Resume, delete, import, and export sessions.
- Session Information: See the session metadata defined when creating the session.
- Session Files: Add and organize diagnostic data files collected during IGAA flows.
- Session Notes: Enter short notes describing observed session details, such as tool analysis results and session information.
- Session History: See the series of steps that have been taken for this session.
- Glossary: View the definition of common terms used in the IGAA flows.
You can customize which side of the panel you want the Quick Views section to appear (left or right) simply by dragging the Quick Views title bar from one side to the other.
Wizard (red section)
The Wizard section is the primary area for making path decisions as you progress through an IGAA flow. In addition, if there are specific steps or information that the content author determined to be particularly important, that information is presented in this section. The [+] symbols next to most path choices indicate that additional information is available to help you understand more about your path choices without reading the extensive details in the lower section.
Details (green section)
This section provides extra details about the current step that you are viewing. Additional text, links to existing resources (such as Tech Notes, Information Centers, and Redbooks), and processing notes are all contained in this section. If you find you need extra information to help you work through an IGAA flow, look here for specific details.
Together, these three sections make up the interface with which you will work through IGAA flows. Let's take a look at a common scenario to illustrate how IGAA can help you identify and solve problems.
This scenario highlights all the major steps involved in using IGAA to resolve a typical problem in a large WebSphere Application Server deployment, from the end-to-end perspectives of a system administrator and a support engineer. This scenario is for illustrative purposes only to highlight key IGAA features, and does not necessarily describe all details of an actual problem situation. At the same time, don't be apprehensive by all the steps shown in this scenario; while the intention is to show every significant step to demonstrate the full picture of what is happening throughout this process, the point is that IGAA will guide you through each of these steps in a manner that makes them flow naturally and painlessly. In practice, a user who does not leverage IGAA would most likely follow the same or similar steps, but simply without the benefit of a tool for guidance.
So let's begin.
Kathy is a system administrator for a large online store. She receives a call from the Customer Care center to inform her that over the last 30 minutes many shoppers on the Web site have complained about seeing a blank page when they attempt to check out. Kathy is not sure where to begin, so she starts the IBM Support Assistant and launches IGAA to guide her through the process of narrowing down possible causes. The first screen that displays looks like Figure 2.
Figure 2. Welcome
Start the troubleshooting session
Kathy starts an IGAA troubleshooting session and fills in the session information, shown in Figure 3. Since the problem has been described at a high level as a blank page, she names this session "Blank Page". She clicks Next.
Figure 3. Start session
Select the type of activity
At this point, Kathy is presented with a screen like Figure 4, instructing her to select one of a few main types of activities for which IGAA can provide assistance. Since she is not quite sure what is causing the blank page problem, she selects Guide me, instructing IGAA to guide her through the process of finding the components that may cause a blank page to show up as a symptom.
Figure 4. New session
Select a problem category
The next screen (Figure 5) shows a few choices for broad categories of problems. Kathy looks at the expandable details of the possible action choices and notices that "blank page" is a common symptom for the first choice. She chooses performs poorly or behaves unexpectedly, which is the first action covering unexpected behavior.
Figure 5. Select symptom
Select a sub-category of problem
Figure 6 lists a few different potential sub-types of problems under the general problem heading of "Application performs poorly or unexpectedly," which Kathy had selected. Again, she is not certain which action to choose next, so she looks at the details in the bottom section and determines that does not receive a response seems to be most appropriate for her problem. She selects the first option as the best option.
Figure 6. Select appropriate action
Pick a topic area to drill down
In Figure 7, IGAA advises Kathy that the problem could originate in a few different areas, which she may have to investigate in turn. Because she is the application server administrator and she suspects that this is an application server problem, she decides to drill down into the application server steps first to determine if the primary problem exists there.
Figure 7. Select area to investigate
Check possible causes
Kathy works through the list of items in Figure 8 to see if the problem is a known application server problem. She sees the information discussing Java memory problems, then discovers a java.lang.OutOfMemoryError exception in the SystemOut.log file that occurred during the time that customers were reporting blank page problems. While this does not explain what is causing the out-of-memory (OOM) errors quite yet, it is a clue as to which path to choose next. She chooses the Out Of Memory action and presses Next.
Figure 8. Select problem type
Identify a Java out-of-memory problem
On Figure 9, Kathy chooses an action that specifically helps to troubleshoot out-of-memory problems. She selects that option with confidence.
Figure 9. Out of memory
Find diagnostics information, or save the session
In Figure 10, IGAA instructs Kathy that she needs to collect some diagnostics information for in-depth analysis. For the purpose of this example, let's say that Kathy is at the end of her shift. She decides to export the guided activity session and hand it over to Barb, the administrator working during the next shift. Barb will be able to pick up where Kathy left off by importing the session Kathy had created. Kathy exports the saved "Blank Page" guided activity session and e-mails it to Barb.
When Barb receives the file, she imports it into her copy of IGAA and resumes the session, beginning on the same screen where Kathy left off. She reviews the work Kathy has completed by studying the Session History quick view, and then starts her troubleshooting effort. She knows that Verbose Garbage Collection is on by default in the company's application servers, so she responds to IGAA indicating that a file containing Verbose GC information already exists.
Figure 10. VCG logging
Add diagnostics information to the session
In Figure 11, Barb is asked to collect the appropriate diagnostic files. She follows the instructions in the guided activity and saves the verbose GC log in the session. After collecting the log, she moves on to begin diagnostic analysis.
Figure 11. Data collection
Analyze the verbose GC log file
IGAA instructs Barb that she needs to invoke the Pattern Modeling and Analysis tool (PMAT) that will analyze the verbose GC log (Figure 12). She invokes that tool directly from IGAA by pressing the Launch Pattern Modeling and Analysis Tool button. Once the tool starts and prompts her for an input file, she responds to the dialog telling the tool which log to analyze.
Figure 12. Launch Pattern Modeling and Analysis Tool
The PMAT tool automatically performs its analysis and displays the results in a new window (Figure 13).
Figure 13. PMAT Analysis Results Table
Note analysis results in the session
Barb notices that the analysis results indicate Java heap exhaustion (sixth entry listed in the results in Figure 13). She opens the Session Notes quick view in the same IGAA window from which she ran the analysis tool, and enters a notation of that fact in the guided activity session before she pressing Next to move on to Figure 14.
Figure 14. Add a note to the IGAA session
Evaluate verbose GC analysis results
Now, Barb needs to indicate to IGAA what the results of the analysis were such that IGAA can properly guide her to the next step (Figure 15). Because there is heap exhaustion, she chooses that action to continue.
Figure 15. Evaluate heap exhaustion
Collect and analyze heap dump
At this point, IGAA instructs Barb to collect a heapdump file and save it to the session, using steps very similar to those outlined in steps 9 and 10 to collect a verbose GC log. The next screen (Figure 16) instructs her to invoke the Memory Dump Diagnostic for Java tool (MDD4J) to analyze the heapdump. Again, this is similar to step 12 in which the PMAT tool was launched.
Figure 16. Launch Memory Dump Diagnostic for Java
Evaluate heapdump analysis results
The analysis results (not shown here) indicate a memory leak in the application. In Figure 17, Barb chooses the appropriate action for the next step in the flow.
Figure 17. Memory leak
Work offline to fix the memory leak problem in the application
Because there is a memory leak in the application, Barb knows that she will have to debug the code and deploy a patch to the application running in production. That will take time, but the problem is preventing customers from making purchases. She knows she must provide a workaround (such as cycling the servers periodically or increasing the heap sizes) until the memory leak can be fixed. She saves her activity session and will resume it when she is ready to test the corrected application. Barb makes note of what caused the problem and that a temporary solution has been implemented on the Web site's servers. Having restored the operation of the production web site, Barb continues investigating the memory leak in the application's code. Eventually, she identifies the leaking objects and fixes the application to eliminate the problem.
Validate the solution
After implementing what she expects will be the fix for the memory leak in the application, Barb places the fixed application into a test environment and then resumes her activity session to verify the solution (Figure 18).
Figure 18. Validate the solution
Test and verify if symptoms still exist
IGAA helps Barb test the solution and prompts her to verify that none of the previous symptoms (or any new symptoms) sill exist (Figure 19). This time, because there are no further symptoms, Barb selects the action None of listed symptoms exist.
Figure 19. No symptoms
Close the troubleshooting session
Because all of the symptoms have been eliminated, she chooses the action to end the activity session (Figure 20).
Figure 20. Symptoms eliminated
IGAA reports that the problem is solved. Barb saves the session as an archive of the troubleshooting process in the event that a similar problem occurs in the future.
Figure 21. Problem fixed
Frequently asked questions about the IGAA
How do I obtain the IBM Guided Activity Assistant tool?
IGAA is a tool the runs as a plug-in to the IBM Support Assistant. To obtain the IGAA tool, you need to first download the IBM Support Assistant, and then download IGAA). See IBM Guided Activity Assistant (IGAA) plug-in for IBM Support Assistant (ISA) V3 for more details.
How do I obtain content for IGAA?
IGAA separates the content from the tool itself, which makes it very easy to dynamically add or remove content without having to reinstall IGAA. See IBM Guided Activity Assistant (IGAA) plug-in for IBM Support Assistant (ISA) V3 for instructions on obtaining IGAA content.
Is IGAA a WebSphere-only tool? Does IGAA provide WebSphere-only content?
No, IGAA is not a WebSphere-only tool. Most of the problem determination content available today is for WebSphere Application Server-related problems, but additional content is forthcoming for other IBM products.
What content is available on IGAA today?
At this writing, problem determination content is available for WebSphere Application Server in these areas:
- Phase 1 (problem determination and filter)
- Java out-of-memory
- HTTP Server and plug-in
- Java hang
- Java crash
- System management
- J2EE Connector (J2C)
How do I learn about new content on IGAA?
To learn about new content available on IGAA, visit the ISA Forum for additions and updates to IGAA content. You can also learn about new content via the IBM Support Assistant Updater Component. See IBM Guided Activity Assistant (IGAA) plug-in for IBM Support Assistant (ISA) V3 for more information on the updater feature.
How do I get updates for IGAA?
Updates for IGAA can be obtained via the Updater component in IBM Support Assistant. Click on Available Plug-ins Updates in the IBM Support Assistant to obtain updates for IGAA.
I want to leave feedback about IGAA. What do I do?
You can post comments or questions to IGAA on the feedback section of the IGAA Support Page.
You can get the IBM Guided Activity Assistant today by downloading it freom the IBM Guided Activity Assistant (IGAA) plug-in for IBM Support Assistant (ISA) V3 page. If you would like to see more topics available on IGAA, you can provide us with your suggestions either through the feedback capability in IGAA, or via the feedback section of IGAA Support Page. We hope to hear from you!
- The Support Authority: If you need help with WebSphere products, there are many ways to get it
- IBM Software Support Web site
- IBM Software Support Handbook
- Product documentation
- IBM developerWorks
- IBM Redbooks