In each column, The Support Authority discusses resources, tools, and other elements of IBM® Technical Support that are available for WebSphere® products, plus techniques and new ideas that can further enhance your IBM support experience.
As always, we begin with some new items of interest for the WebSphere community at large:
- New updates have been made to the IBM Guided Activity Assistant,
which is available as the Guided Troubleshooter
in the IBM Support
Assistant, or you can get it from alphaWorks:
- The Java™ troubleshooting guided activities now include assistance with solving performance problems.
- Bookmark specific steps in a guided activity by using the new link feature. (This capability is available from the alphaWorks version only.)
- Get these latest IBM Support
Assistant add-ons for these WebSphere products:
- IBM WebSphere Application Server V7
- IBM WebSphere Application Server Feature Pack for Web 2.0
- IBM WebSphere Business Monitor 6.1
- Interested in migrating from IBM WebSphere Business Integration Server Foundation to IBM WebSphere Process Server V6.1? See this new Redbook to learn more.
- Noticed a change in your My Support subscriptions? My Support was recently replaced with the new My Notifications service.
Continue to monitor the various support-related Web sites, as well as this column, for news about other tools as we encounter them.
And now, on to our main topic...
Phase 1 problem determination (Phase 1 PD) is the concept of identifying, understanding, and potentially resolving a software problem at first look. It is analogous to someone who is sick visiting a general physician for the first phase of their diagnosis; the doctor takes the patient's temperature, checks blood pressure, and examines external maladies, but if more in-depth analysis is needed, the doctor enlists the services of a lab for further tests or refers the patient to a specialist for advanced cases. Likewise, Phase 1 PD assists you with either resolving problems, or refers you to more advanced troubleshooting tools, or even to IBM technical support.
Because its purpose is initial diagnosis, Phase 1 PD primarily examines log files. Traces and other diagnostics (such as heap or thread dumps) can provide valuable data, but they can be expensive to collect during run time, or they might only be only available when re-creating a software failure. (How to collect and analyze these diagnostics are outside the scope of Phase 1 PD.)
Log Analyzer is a freely available tool developed by the IBM Tivoli® Autonomic Computing organization. It processes the log files of over 200 different IBM products and enables you to sort, filter, correlate, and analyze log entries from these products. For example, this tool makes it easy for you to see what has occurred in the log files of different IBM WebSphere Application Server instances, IBM HTTP Server instances, DB2® applications, and others, all at the same time.
When combined with the IBM Support Assistant, Log Analyzer is even more powerful. Log Analyzer is case-aware, meaning that it can view log files that have been collected inside of the Support Assistant case manager. Further, it is able to leverage IBM Support Assistant agent technology to collect log files from remote systems, making it very easy to collect and analyze logs from production systems. You can learn more about collecting logs in the Support Authority column Collecting diagnostic information using the IBM Support Assistant.
The remainder of this article describes a scenario that can be solved using Phase 1 PD.
Using WebSphere Application Server Version 6.X, you can configure an application server's log file to "roll over" at certain intervals. In other words, the server will archive the log files to prevent the file from growing so large that it is unmanageable. In this scenario, the log file rollover fails to archive as expected on your Microsoft® Windows®-based WebSphere Application Server installation.
The first step in Phase 1 PD is to gather all relevant diagnostic information. There are two ways of collecting this data using Log Analyzer and the IBM Support Assistant:
- Use the data collector.
- Manually choose files using the systems explorer.
Both approaches assume that either the tools are installed on the same system as the failing product or that an IBM Support Assistant Agent is installed on the failing system.
The data collector gathers a large number of files based on your input and stores them in a .zip file that can be consumed by Log Analyzer. It is accessible in the IBM Support Assistant under the Analyze Problem activity. In order for the data collector to work with a particular product, you must install that product's product add-on from the IBM Support Assistant's updater. You can learn more about agent technology and data collection from this earlier Support Authority column.
The systems explorer enables you to browse the file systems of local and remote hosts that are connected to the same IBM Support Assistant agent manager. Similar to other file browsers, the systems explorer enables you to view the contents of files on these remote hosts. Unlike conventional file browsers, the systems explorer lets you transfer files from remote (or local) file systems into a new or existing case in the IBM Support Assistant's case manager. Log Analyzer can access both the systems explorer and the case manager from within its import wizard.
To simplify this scenario, assume that the WebSphere Application Server installation with the failing log file rotation is on the same system as the IBM Support Assistant workbench:
- Navigate to WebSphere Application Server under C:\IBM\WebSphere\ApplicationServer\profiles\AppSrv01\logs\server1 and add the SystemOut.log file to a new case.
- IBM Support Assistant prompts you for a case name and an optional description. You can also create incidents or folders to help organize your case. For now, just create a case called "Log file rollover issue."
- When you click OK to close this dialog, the SystemOut.log file will be copied into the IBM Support Assistant's case manager where it can be accessed by tools such as the Log Analyzer. It is possible to multi-select from this view and add more than one log file at a time. (Figure 1).
Figure 1. IBM Support Assistant
Now that you have collected the "extra large" log file into IBM Support Assistant's case manager, you are ready to launch Log Analyzer. You can do this through the IBM Support Assistant by switching to the Analyze Problem activity and selecting the Tools tab. If Log Analyzer does not appear in the tools catalog, you will need to install it using the Updater:
- From the Update menu, select Find new... => Tools Add-ons.
- Select Log Analyzer from the dialog under the JVM-based Tools heading.
- After accepting the license agreement, you can complete the installation.
Once Log Analyzer has started, you can import the log file that you added to your case in the IBM Support Assistant's case manager.
Select File => Import Log File from Log Analyzer. A dialog similar to the one in Figure 2 will appear.
Figure 2. Import log file
From here, you can select the SystemOut.log file under the Log file rollover issue case that you imported into the IBM Support Assistant earlier. It is possible to import more than one log file at a time, but for this scenario, you only need SystemOut.log. Log Analyzer should correctly identify this log file as a WebSphere Application Server log file. You do not need to modify the filter settings, nor should you register this file with Log Analyzer's Log Discovery Service since it is already on the local file system.
After clicking Finish, you will see the log file in a table view, similar to Figure 3.
Figure 3. Table view of log file
Now you are ready to analyze the logs.
The first step is to quickly identify all error messages in this log file. Log Analyzer comes with a built-in filter called "Show error log records only" that will filter out all log records that are not considered an error message. Log Analyzer also gives you the ability to create custom filters, such as filtering out log records that are older (or newer) than a certain date, that contain messages with certain text, or whose severity is above or below a specific level, and so on. (Figure 4)
Figure 4. Log record filter
In this particular log file, there is only one error message, and when you click on it you see that an unexpected exception occurred while trying to archive one log file to another. The exception text mentions (Figure 5):
Unable to rename file
Figure 5. Archive exception
To perform analysis on this error message, you need to import the latest symptom catalog from IBM. IBM maintains over 20 symptom catalogs for multiple versions of multiple products. You can also create a custom symptom catalog using the Symptom Editor tool (also available for download via the IBM Support Assistant). The symptom catalog maps matching log records with additional information including recommendations and actions to resolve the problem.
For this problem, you will import the symptom catalog for WebSphere Application Server Version 6.1:
- Start by selecting File => Import Symptom Catalog...
- From the Import System Catalog wizard, select the From Remote host
option and choose the Symptom catalog for IBM WebSphere Application Server,
Version 6.1, then click Finish (Figure 6). This will download and install the symptom catalog. Once installed, you are ready to perform some analysis.
Figure 6. Import symptom catalog
- Right click on the error message in the log view and select Analyze Selection. The first time you click this will take some time while Log Analyzer compiles the symptom catalog(s).
- Once finished, you will see a single hit in the Symptom Analysis Results view
(in the bottom right pane of Figure 7). The Recommendation and actions tab shows you that the log file rollover threshold was reached, but some problem prevented successful rollover. The next step is to determine if the problem is a WebSphere Application Server error or some other problem.
Although the symptom catalog gives you some information -- you learned that the application server was unable to rename the SystemOut.log file to a SystemOut_<timestamp>.log file – the information is not really sufficient to solve the problem. But don’t throw in the towel just yet. Try one more thing: search.
Figure 7. Symptom analysis results
- Return to the log view, right-click on the error message, and select Search Message ID in IBM Support Assistant. This will kick off a search request using the IBM Support Assistant's search mechanism, which simultaneously searches IBM Support documentation, developerWorks articles, newsgroups, forums, and the Google search engine. The search results will appear in the lower right pane. (Figure 8)
Figure 8. Search results
If you click on the IBM Software Support Documents results, you will see the results shown in the previous image. The first document seems unrelated as it is referring to IBM Rational® Application Developer. While it is possible that you might learn something from this document, the next two seem a little more promising.
By clicking on the second document link, Log Analyzer will open a browser window to that document. If you prefer to open the document in your own browser, simply right click on the link and select Copy link to clipboard. This will enable you to paste the URL into your favorite browser.
This document seems to tell you what you need to know. Paraphrasing the document, it mentions that (1) some non-WebSphere Application Server process most likely has a lock on the log file, which is preventing the application server from renaming it, and (2) that a fix exists that can overcome this log rollover problem, but (3) that while the fix works, the best performing option would be to stop the process that is locking the log file (usually a text editor, or a program to tail the log, and so on).
Thanks to the Phase 1 PD process and these freely available tools from IBM, you are able to resolve this particular problem quickly and without involving IBM Support.
Phase 1 PD can be a very effective process for resolving software problems. The scenario in this article showed you how you can use the Log Analyzer to solve a problem simply by examining the standard output log for WebSphere Application Server. In many cases, Log Analyzer can make it quick and easy for you to identify a problem’s cause and implement a solution.
IBM provides several free tools that you can use to facilitate problem determination. Coupled with Phase 1 Problem Determination, solutions to software-related problems could be just around the corner.
The Support Authority: If you need help with WebSphere products, there are many ways to get it
Webcast: Autonomic Computing WebSphere Technical Exchange
MustGather: Read first for all WebSphere Application Server products
IBM Software Support Web site
IBM Education Assistant
Get products and technologies
Andy McCright is the lead serviceability developer for WebSphere Application Server, based in the Rochester, Minnesota lab. In his five years with IBM, he has contributed to the development of the IBM Support Assistant, the Log Analyzer tool, and assisted customers with various WebSphere-related problems.
Russell Wright has several years of experience developing and supporting data communications and middleware software including WebSphere Application Server. He currently manages the deployment of troubleshooting tools for the IBM Support Assistant and is a developer for the IBM Guided Activity Assistant.