The Support Authority: Using Log Analyzer for Phase 1 problem determination

When you encounter a problem, the first thing you usually want to determine is whether there is a quick resolution available, or if you're going to need advanced assistance. The freely available Log Analyzer tool can help you save critical troubleshooting time when you need to make this initial evaluation. This content is part of the IBM WebSphere Developer Technical Journal.

Share:

Andy McCright (andymc@us.ibm.com), Serviceability Developer, IBM WebSphere Application Server, WSO2 Inc

Andy McCright is the lead serviceability developer for WebSphere Application Server, based in the Rochester, Minnesota lab. In his five years with IBM, he has contributed to the development of the IBM Support Assistant, the Log Analyzer tool, and assisted customers with various WebSphere-related problems.



Russell Wright (rbwright@us.ibm.com), WebSphere Serviceability Development, EMC

Russell Wright has several years of experience developing and supporting data communications and middleware software including WebSphere Application Server. He currently manages the deployment of troubleshooting tools for the IBM Support Assistant and is a developer for the IBM Guided Activity Assistant.



12 November 2008

Also available in Chinese

In each column, The Support Authority discusses resources, tools, and other elements of IBM® Technical Support that are available for WebSphere® products, plus techniques and new ideas that can further enhance your IBM support experience.

This just in...

As always, we begin with some new items of interest for the WebSphere community at large:

  • New updates have been made to the IBM Guided Activity Assistant, which is available as the Guided Troubleshooter in the IBM Support Assistant, or you can get it from alphaWorks:
    • The Java™ troubleshooting guided activities now include assistance with solving performance problems.
    • Bookmark specific steps in a guided activity by using the new link feature. (This capability is available from the alphaWorks version only.)
  • Get these latest IBM Support Assistant add-ons for these WebSphere products:
    • IBM WebSphere Application Server V7
    • IBM WebSphere Application Server Feature Pack for Web 2.0
    • IBM WebSphere Business Monitor 6.1
  • Interested in migrating from IBM WebSphere Business Integration Server Foundation to IBM WebSphere Process Server V6.1? See this new Redbook to learn more.
  • Noticed a change in your My Support subscriptions? My Support was recently replaced with the new My Notifications service.

Continue to monitor the various support-related Web sites, as well as this column, for news about other tools as we encounter them.

And now, on to our main topic...


What is Phase 1 problem determination?

Phase 1 problem determination (Phase 1 PD) is the concept of identifying, understanding, and potentially resolving a software problem at first look. It is analogous to someone who is sick visiting a general physician for the first phase of their diagnosis; the doctor takes the patient's temperature, checks blood pressure, and examines external maladies, but if more in-depth analysis is needed, the doctor enlists the services of a lab for further tests or refers the patient to a specialist for advanced cases. Likewise, Phase 1 PD assists you with either resolving problems, or refers you to more advanced troubleshooting tools, or even to IBM technical support.

Because its purpose is initial diagnosis, Phase 1 PD primarily examines log files. Traces and other diagnostics (such as heap or thread dumps) can provide valuable data, but they can be expensive to collect during run time, or they might only be only available when re-creating a software failure. (How to collect and analyze these diagnostics are outside the scope of Phase 1 PD.)

IBM currently provides the Log Analyzer tool to aid in Phase 1 PD. This tool is available as an add-on to the IBM Support Assistant .


What is the Log Analyzer?

Log Analyzer is a freely available tool developed by the IBM Tivoli® Autonomic Computing organization. It processes the log files of over 200 different IBM products and enables you to sort, filter, correlate, and analyze log entries from these products. For example, this tool makes it easy for you to see what has occurred in the log files of different IBM WebSphere Application Server instances, IBM HTTP Server instances, DB2® applications, and others, all at the same time.

When combined with the IBM Support Assistant, Log Analyzer is even more powerful. Log Analyzer is case-aware, meaning that it can view log files that have been collected inside of the Support Assistant case manager. Further, it is able to leverage IBM Support Assistant agent technology to collect log files from remote systems, making it very easy to collect and analyze logs from production systems. You can learn more about collecting logs in the Support Authority column Collecting diagnostic information using the IBM Support Assistant.

The remainder of this article describes a scenario that can be solved using Phase 1 PD.


Sample scenario

Using WebSphere Application Server Version 6.X, you can configure an application server's log file to "roll over" at certain intervals. In other words, the server will archive the log files to prevent the file from growing so large that it is unmanageable. In this scenario, the log file rollover fails to archive as expected on your Microsoft® Windows®-based WebSphere Application Server installation.

A. Collect the appropriate log files

The first step in Phase 1 PD is to gather all relevant diagnostic information. There are two ways of collecting this data using Log Analyzer and the IBM Support Assistant:

  • Use the data collector.
  • Manually choose files using the systems explorer.

Both approaches assume that either the tools are installed on the same system as the failing product or that an IBM Support Assistant Agent is installed on the failing system.

The data collector gathers a large number of files based on your input and stores them in a .zip file that can be consumed by Log Analyzer. It is accessible in the IBM Support Assistant under the Analyze Problem activity. In order for the data collector to work with a particular product, you must install that product's product add-on from the IBM Support Assistant's updater. You can learn more about agent technology and data collection from this earlier Support Authority column.

The systems explorer enables you to browse the file systems of local and remote hosts that are connected to the same IBM Support Assistant agent manager. Similar to other file browsers, the systems explorer enables you to view the contents of files on these remote hosts. Unlike conventional file browsers, the systems explorer lets you transfer files from remote (or local) file systems into a new or existing case in the IBM Support Assistant's case manager. Log Analyzer can access both the systems explorer and the case manager from within its import wizard.

To simplify this scenario, assume that the WebSphere Application Server installation with the failing log file rotation is on the same system as the IBM Support Assistant workbench:

  1. Navigate to WebSphere Application Server under C:\IBM\WebSphere\ApplicationServer\profiles\AppSrv01\logs\server1 and add the SystemOut.log file to a new case.
  2. IBM Support Assistant prompts you for a case name and an optional description. You can also create incidents or folders to help organize your case. For now, just create a case called "Log file rollover issue."
  3. When you click OK to close this dialog, the SystemOut.log file will be copied into the IBM Support Assistant's case manager where it can be accessed by tools such as the Log Analyzer. It is possible to multi-select from this view and add more than one log file at a time. (Figure 1).
Figure 1. IBM Support Assistant
Figure 1. IBM Support Assistant

B. Import the logs into Log Analyzer

Now that you have collected the "extra large" log file into IBM Support Assistant's case manager, you are ready to launch Log Analyzer. You can do this through the IBM Support Assistant by switching to the Analyze Problem activity and selecting the Tools tab. If Log Analyzer does not appear in the tools catalog, you will need to install it using the Updater:

  1. From the Update menu, select Find new... => Tools Add-ons.
  2. Select Log Analyzer from the dialog under the JVM-based Tools heading.
  3. After accepting the license agreement, you can complete the installation.

Once Log Analyzer has started, you can import the log file that you added to your case in the IBM Support Assistant's case manager.

Select File => Import Log File from Log Analyzer. A dialog similar to the one in Figure 2 will appear.

Figure 2. Import log file
Figure 2. Import log file

The Log Discovery Service is most appropriate when importing log files from remote systems using the IBM Support Assistant agent. The Register file option (which only works for files on remote file systems) adds the selected log files to a repository of log files for the IBM Support Assistant agent on that system. When you need that log file again, you can find it more easily in the import wizard.

From here, you can select the SystemOut.log file under the Log file rollover issue case that you imported into the IBM Support Assistant earlier. It is possible to import more than one log file at a time, but for this scenario, you only need SystemOut.log. Log Analyzer should correctly identify this log file as a WebSphere Application Server log file. You do not need to modify the filter settings, nor should you register this file with Log Analyzer's Log Discovery Service since it is already on the local file system.

After clicking Finish, you will see the log file in a table view, similar to Figure 3.

Figure 3. Table view of log file
Figure 3. Table view of log file

Now you are ready to analyze the logs.

C. Analyze the logs and find known solutions

The first step is to quickly identify all error messages in this log file. Log Analyzer comes with a built-in filter called "Show error log records only" that will filter out all log records that are not considered an error message. Log Analyzer also gives you the ability to create custom filters, such as filtering out log records that are older (or newer) than a certain date, that contain messages with certain text, or whose severity is above or below a specific level, and so on. (Figure 4)

Figure 4. Log record filter
Figure 4. Log record filter

In this particular log file, there is only one error message, and when you click on it you see that an unexpected exception occurred while trying to archive one log file to another. The exception text mentions (Figure 5):

Unable to rename file C:\was6109\profiles\AppSrv01\logs\server1\SystemOut.log to
C:\was6109\profiles\AppSrv01\logs\server1\SystemOut_08.08.11_12.18.11.log

Figure 5. Archive exception
Figure 5. Archive exception

To perform analysis on this error message, you need to import the latest symptom catalog from IBM. IBM maintains over 20 symptom catalogs for multiple versions of multiple products. You can also create a custom symptom catalog using the Symptom Editor tool (also available for download via the IBM Support Assistant). The symptom catalog maps matching log records with additional information including recommendations and actions to resolve the problem.

For this problem, you will import the symptom catalog for WebSphere Application Server Version 6.1:

  1. Start by selecting File => Import Symptom Catalog...
  2. From the Import System Catalog wizard, select the From Remote host option and choose the Symptom catalog for IBM WebSphere Application Server, Version 6.1, then click Finish (Figure 6). This will download and install the symptom catalog. Once installed, you are ready to perform some analysis.
    Figure 6. Import symptom catalog
    Figure 6. Import symptom catalog
  3. Right click on the error message in the log view and select Analyze Selection. The first time you click this will take some time while Log Analyzer compiles the symptom catalog(s).
  4. Once finished, you will see a single hit in the Symptom Analysis Results view (in the bottom right pane of Figure 7). The Recommendation and actions tab shows you that the log file rollover threshold was reached, but some problem prevented successful rollover. The next step is to determine if the problem is a WebSphere Application Server error or some other problem.

    Although the symptom catalog gives you some information -- you learned that the application server was unable to rename the SystemOut.log file to a SystemOut_<timestamp>.log file – the information is not really sufficient to solve the problem. But don’t throw in the towel just yet. Try one more thing: search.

    Figure 7. Symptom analysis results
    Figure 7. Symptom analysis results
  5. Return to the log view, right-click on the error message, and select Search Message ID in IBM Support Assistant. This will kick off a search request using the IBM Support Assistant's search mechanism, which simultaneously searches IBM Support documentation, developerWorks articles, newsgroups, forums, and the Google search engine. The search results will appear in the lower right pane. (Figure 8)
    Figure 8. Search results
    Figure 8. Search results

If you click on the IBM Software Support Documents results, you will see the results shown in the previous image. The first document seems unrelated as it is referring to IBM Rational® Application Developer. While it is possible that you might learn something from this document, the next two seem a little more promising.

By clicking on the second document link, Log Analyzer will open a browser window to that document. If you prefer to open the document in your own browser, simply right click on the link and select Copy link to clipboard. This will enable you to paste the URL into your favorite browser.

This document seems to tell you what you need to know. Paraphrasing the document, it mentions that (1) some non-WebSphere Application Server process most likely has a lock on the log file, which is preventing the application server from renaming it, and (2) that a fix exists that can overcome this log rollover problem, but (3) that while the fix works, the best performing option would be to stop the process that is locking the log file (usually a text editor, or a program to tail the log, and so on).

Thanks to the Phase 1 PD process and these freely available tools from IBM, you are able to resolve this particular problem quickly and without involving IBM Support.


Summary

Phase 1 PD can be a very effective process for resolving software problems. The scenario in this article showed you how you can use the Log Analyzer to solve a problem simply by examining the standard output log for WebSphere Application Server. In many cases, Log Analyzer can make it quick and easy for you to identify a problem’s cause and implement a solution.

IBM provides several free tools that you can use to facilitate problem determination. Coupled with Phase 1 Problem Determination, solutions to software-related problems could be just around the corner.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Business process management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Business process management, WebSphere, Commerce
ArticleID=350756
ArticleTitle=The Support Authority: Using Log Analyzer for Phase 1 problem determination
publish-date=11122008