Level: Introductory Andy McCright (andymc@us.ibm.com), Serviceability Developer, IBM WebSphere Application Server,
IBM
Russell Wright (rbwright@us.ibm.com), WebSphere Serviceability Development, IBM
12 Nov 2008 
When you encounter a problem, the first thing you usually want to determine
is whether there is a quick resolution available, or if you're
going to need advanced assistance. The freely available Log Analyzer tool can help you save critical
troubleshooting time when you need to make this initial evaluation.
From IBM WebSphere Developer Technical Journal.
In each column, The Support Authority discusses resources, tools, and other
elements of IBM® Technical Support that are available for WebSphere® products,
plus techniques and new ideas that can further enhance your IBM support
experience.
This just in...
As always, we begin with some new items of interest for the WebSphere
community at large:
- New updates have been made to the IBM Guided Activity Assistant,
which is available as the Guided Troubleshooter
in the IBM Support
Assistant, or you can get it from alphaWorks:
- The Java™ troubleshooting guided activities now include assistance with solving performance problems.
- Bookmark specific steps in a guided activity by using the new link
feature. (This capability is available from the alphaWorks version only.)
- Get these latest IBM Support
Assistant add-ons for these WebSphere products:
- IBM WebSphere Application Server V7
- IBM WebSphere Application Server Feature Pack for Web 2.0
- IBM WebSphere Business Monitor 6.1
- Interested in migrating from IBM WebSphere Business Integration Server
Foundation to IBM WebSphere Process Server V6.1? See this new Redbook to learn more.
- Noticed a change in your My Support subscriptions? My Support was recently
replaced with the new My Notifications service.
Continue to monitor the
various support-related Web sites, as well as this
column, for news about other tools as we encounter them.
And now, on to our main topic...
What is Phase 1 problem determination?
Phase 1 problem determination (Phase 1 PD) is the concept of identifying,
understanding, and potentially resolving a software problem at first look. It is
analogous to someone who is sick visiting a general physician for the first phase
of their diagnosis; the doctor takes the patient's temperature, checks blood
pressure, and examines external maladies, but if more in-depth analysis is needed,
the doctor enlists the services of a lab for further tests or refers the patient
to a specialist for advanced cases. Likewise, Phase 1 PD assists you with either
resolving problems, or refers you to more advanced troubleshooting tools, or even to IBM technical support.
Because its purpose is initial diagnosis, Phase 1 PD primarily examines log
files. Traces and other diagnostics (such as heap or thread dumps) can provide
valuable data, but they can be expensive to collect during run time, or they might
only be only available when re-creating a software failure. (How to collect and
analyze these diagnostics are outside the scope of Phase 1 PD.)
IBM currently provides the Log Analyzer tool to aid in Phase 1
PD. This tool is available as an add-on to the IBM Support Assistant .
What is the Log Analyzer?
Log Analyzer is a freely available tool developed by the IBM Tivoli®
Autonomic Computing organization. It processes the log files of over 200 different
IBM products and enables you to sort, filter, correlate, and analyze log entries
from these products. For example, this tool makes it easy for you to see what has
occurred in the log files of different IBM WebSphere Application Server instances,
IBM HTTP Server instances, DB2® applications, and others, all at the same time.
When combined with the IBM Support Assistant, Log Analyzer is even more powerful.
Log Analyzer is case-aware, meaning that it can view log files that have been
collected inside of the Support Assistant case manager. Further, it is able to
leverage IBM Support Assistant agent technology to collect log files from remote
systems, making it very easy to collect and analyze logs from production systems. You can learn more about collecting logs in the Support Authority column Collecting diagnostic information using the IBM Support Assistant.
The remainder of this article describes a scenario that can be solved using Phase 1 PD.
Sample scenario
Using WebSphere Application Server Version 6.X, you can configure an application
server's log file to "roll over" at certain intervals. In other words, the server
will archive the log files to prevent the file from growing so large that it is
unmanageable. In this scenario, the log file rollover fails to archive as expected
on your Microsoft® Windows®-based WebSphere Application Server installation.
A. Collect the appropriate log files
The first step in Phase 1 PD is to gather all relevant diagnostic information.
There are two ways of collecting this data using Log Analyzer and the IBM Support
Assistant:
- Use the data collector.
- Manually choose files using the systems explorer.
Both approaches assume that either the tools are installed on the same system as the failing product or that an IBM Support Assistant Agent is installed on the failing system.
The data collector gathers a large number of files based on your input and stores
them in a .zip file that can be consumed by Log Analyzer. It is accessible in the
IBM Support Assistant under the Analyze Problem activity. In order for the data
collector to work with a particular product, you must install that product's
product add-on from the IBM Support Assistant's updater. You can learn more about
agent technology and data collection from this
earlier Support Authority column.
The systems explorer enables you to browse the file systems of local and remote
hosts that are connected to the same IBM Support Assistant agent manager. Similar
to other file browsers, the systems explorer enables you to view the contents of files on these remote hosts. Unlike conventional file browsers, the systems explorer lets you transfer files from remote (or local) file systems into a new or existing case in the IBM Support Assistant's case manager. Log Analyzer can access both the systems explorer and the case manager from within its import wizard.
To simplify this scenario, assume that the WebSphere Application Server installation with the failing log file rotation is on the same system as the IBM Support Assistant workbench:
- Navigate to WebSphere Application Server under C:\IBM\WebSphere\ApplicationServer\profiles\AppSrv01\logs\server1 and add the SystemOut.log file to a new case.
- IBM Support Assistant prompts you for a case name and an optional description. You can also create incidents or folders to help organize your case. For now, just create a case called "Log file rollover issue."
- When you click OK to close this dialog, the SystemOut.log file will be copied into the IBM Support Assistant's case manager where it can be accessed by tools such as the Log Analyzer. It is possible to multi-select from this view and add more than one log file at a time. (Figure 1).
Figure 1. IBM Support Assistant
B. Import the logs into Log Analyzer
Now that you have collected the "extra large" log file into IBM Support
Assistant's case manager, you are ready to launch Log Analyzer. You can do this
through the IBM Support Assistant by switching to the Analyze Problem
activity and selecting the Tools tab. If Log Analyzer does not appear in the tools catalog, you will need to install it using the Updater:
- From the Update menu, select Find new... => Tools Add-ons.
- Select Log Analyzer from the dialog under the JVM-based Tools heading.
- After accepting the license agreement, you can complete the installation.
Once Log Analyzer has started, you can import the log file that you added to your
case in the IBM Support Assistant's case manager.
Select File => Import Log
File from Log Analyzer. A dialog similar to the one in Figure 2 will appear.
Figure 2. Import log file
 | | The Log Discovery Service is most appropriate when importing
log files from remote systems using the IBM Support Assistant agent. The Register
file option (which only works for files on remote file systems) adds the selected
log files to a repository of log files for the IBM Support Assistant agent on that
system. When you need that log file again, you can find it more easily in the import wizard. |
|
From here, you can select the SystemOut.log file under the Log file
rollover issue case that you imported into the IBM Support Assistant earlier. It is possible to import more than one log file at a time, but for this scenario, you only need SystemOut.log. Log Analyzer should correctly identify this log file as a WebSphere Application Server log file. You do not need to modify the filter settings, nor should you register this file with Log Analyzer's Log Discovery Service since it is already on the local file system.
After clicking Finish, you will see the log file in a table view, similar to Figure 3.
Figure 3. Table view of log file
Now you are ready to analyze the logs.
C. Analyze the logs and find known solutions
The first step is to quickly identify all error messages in this log file. Log Analyzer comes with a built-in filter called "Show error log records only" that will filter out all log records that are not considered an error message. Log Analyzer also gives you the ability to create custom filters, such as filtering out log records that are older (or newer) than a certain date, that contain messages with certain text, or whose severity is above or below a specific level, and so on. (Figure 4)
Figure 4. Log record filter
In this particular log file, there is only one error message, and when you click on it you see that an unexpected exception occurred while trying to archive one log file to another. The exception text mentions (Figure 5):
Unable to rename file
C:\was6109\profiles\AppSrv01\logs\server1\SystemOut.log to
C:\was6109\profiles\AppSrv01\logs\server1\SystemOut_08.08.11_12.18.11.log
Figure 5. Archive exception
To perform analysis on this error message, you need to import the latest symptom catalog from IBM. IBM maintains over 20 symptom catalogs for multiple versions of multiple products. You can also create a custom symptom catalog using the Symptom Editor tool (also available for download via the IBM Support Assistant). The symptom catalog maps matching log records with additional information including recommendations and actions to resolve the problem.
For this problem, you will import the symptom catalog for WebSphere Application Server Version 6.1:
- Start by selecting File => Import Symptom Catalog...
- From the Import System Catalog wizard, select the From Remote host
option and choose the Symptom catalog for IBM WebSphere Application Server,
Version 6.1, then click Finish (Figure 6). This will download and install the symptom catalog. Once installed, you are ready to perform some analysis.
Figure 6. Import symptom catalog
- Right click on the error message in the log view and select Analyze Selection. The first time you click this will take some time while Log Analyzer compiles the symptom catalog(s).
- Once finished, you will see a single hit in the Symptom Analysis Results view
(in the bottom right pane of Figure 7). The Recommendation and actions tab shows you that the log file rollover threshold was reached, but some problem prevented successful rollover. The next step is to determine if the problem is a WebSphere Application Server error or some other problem.
Although the symptom catalog gives you some information -- you learned that the
application server was unable to rename the SystemOut.log file to a
SystemOut_<timestamp>.log file – the information is not really sufficient to solve the problem. But don’t throw in the towel just yet. Try one more thing: search.
Figure 7. Symptom analysis results
- Return to the log view, right-click on the error message, and select Search Message ID in IBM Support Assistant. This will kick off a search request using the IBM Support Assistant's search mechanism, which simultaneously searches IBM Support documentation, developerWorks articles, newsgroups, forums, and the Google search engine. The search results will appear in the lower right pane. (Figure 8)
Figure 8. Search results
If you click on the IBM Software Support Documents results, you will see the
results shown in the previous image. The first document seems unrelated as it is
referring to IBM Rational® Application Developer. While it is possible that you might learn something from this document, the next two seem a little more promising.
By clicking on the second document link, Log Analyzer will open a browser window
to that document. If you prefer to open the document in your own browser, simply
right click on the link and select Copy link to clipboard. This will enable you to paste the URL into your favorite browser.
This document seems to tell you what you need to know. Paraphrasing the document, it mentions that (1) some non-WebSphere Application Server process most likely has a lock on the log file, which is preventing the application server from renaming it, and (2) that a fix exists that can overcome this log rollover problem, but (3) that while the fix works, the best performing option would be to stop the process that is locking the log file (usually a text editor, or a program to tail the log, and so on).
Thanks to the Phase 1 PD process and these freely available tools from IBM, you are able to resolve this particular problem quickly and without involving IBM Support.
Summary
Phase 1 PD can be a very effective process for resolving software problems. The scenario in this article showed you how you can use the Log Analyzer to solve a problem simply by examining the standard output log for WebSphere Application Server. In many cases, Log Analyzer can make it quick and easy for you to identify a problem’s cause and implement a solution.
IBM provides several free tools that you can use to facilitate problem determination. Coupled with Phase 1 Problem Determination, solutions to software-related problems could be just around the corner.
Resources Learn
Get products and technologies
Discuss
About the authors | | | Andy McCright is the lead serviceability developer for WebSphere Application Server, based in the Rochester, Minnesota lab. In his five years with IBM, he has contributed to the development of the IBM Support Assistant, the Log Analyzer tool, and assisted customers with various WebSphere-related problems. |
| | | Russell Wright has several years of experience developing and supporting data communications and middleware software including WebSphere Application Server. He currently manages the deployment of troubleshooting tools for the IBM Support Assistant and is a developer for the IBM Guided Activity Assistant. |
Rate this page
| 
