System administration in WebSphere Application Server V8.5, Part 3: High Performance Extensible Logging (HPEL)

IBM® WebSphere® Application Server V8.0 introduced a new way to store and work with log and trace content called High Performance Extensible Logging. HPEL was further improved in the newly released WebSphere Application Server V8.5 making it even more useful. This article tells you what HPEL is all about, and why it matters. This content is part of the IBM WebSphere Developer Technical Journal.

Don Bourne (dbourne@ca.ibm.com), WebSphere Serviceability Architect, IBM

Don Bourne is the WebSphere Serviceability Architect. Don joined IBM in 1996 and helped to build what is now known as WebSphere Commerce. Don has worked in various roles in server development, performance measurement and improvement, and problem determination. Don joined the WebSphere Application Server team in 2003, and has focused on making the application server, and stack products more easily serviceable.



Igor Belyi (belyi@us.ibm.com), Staff Software Engineer, IBM

Igor Belyi is a developer in the WebSphere Serviceability group. Igor joined IBM in 1999 through acquisition of Transarc Corporation. Igor worked as a developer on Encina/TxSeries/CICS middleware product, Dump Anaylzer framework, IBM Support Assistant, and Knowledge Centered Support. Igor joined WebSphere Serviceabiliity team in 2006.



Scott Highbarger ( shighbar@us.ibm.com), Advisory Software Engineer, IBM

Scott Highbarger is an Advisory Software Engineer in the IBM Software Group.



01 August 2012

Also available in Chinese Japanese

Introduction

Try WebSphere Application Server V8.5 now

Try IBM WebSphere Application Server V8.5 trial version free for 60 days, or get IBM WebSphere Application Server for Developers V8.5 now at no cost, and with no expiration date!

IBM WebSphere Application Server V8.5 enhances the High Performance Extensible Logging (HPEL) feature that was first introduced in V8.0. As the name suggests, HPEL was created to significantly outperform the existing application server logging and tracing mechanism, plus provide a powerful way to extend log and trace entries with extra fields. Beyond that, HPEL provides a number of ease of use improvements for working with log and trace content. Enabling HPEL is quick and easy, and does not require any code changes to your applications.

This article provides a primer for administrators considering trying HPEL.

Throughout this article, "HPEL" will refer to the new log and trace system in WebSphere Application Server V8.5, and "Basic" will refer to the pre-existing log and trace system.


Introducing HPEL

HPEL is a new log and trace content storage and access system for WebSphere Application Server. HPEL stores log and trace content produced by the existing log and trace APIs, in particular the java.util.logging API.

HPEL addressses a number of common needs:

  • Log and trace performance

    If you have worked with IBM support teams, you might have been asked to turn on tracing for certain components. This might have been problematic to do on your production systems, because enabling tracing could significantly affect the performance of your applications. HPEL log and trace is significantly faster than Basic log and trace, making tracing much less impactful to your application performance. Using the DayTrader benchmark and EJBContainer=fine trace specification, HPEL boosted overall benchmark performance by 33% compared to using the same trace specification with Basic log and trace (measured using WebSphere Application Server V8.0).

  • Log and trace entry extensibility

    Being able to extend log and trace records can be very useful. For example, you might want to add a userID field to every log and trace record so that you can search your logs or traces later to find records with a particular userID value. There is no mechanism for doing this with the Basic log and trace system.

  • Automating log/trace analysis

    As a WebSphere Application Server administrator, you might have had a need to automate the scan of SystemOut.log and SystemErr.log for significant events. This is non-trivial with the Basic log and trace system because log and trace entries can span multiple lines in the log/trace files. The SystemOut.log and SystemErr.log files also rolled (meaning that when they reached a certain size, they were renamed and truncated back to 0 length), adding further complexity to automation.

  • Application separation

    To facilitate problem determination, you might want to separate the logs and traces for each application into a separate file. There is no mechanism with the Basic log and trace system to make that easy to do. If you had created your own log files per application (perhaps using Log4J), you might have found it cumbersome to correlate content in your log files with WebSphere Application Server log files, or to correlate log and trace content between applications when looking for patterns.

  • Request tracking

    You might have wanted to gather the log and trace records that made up a single request to your server, where a request might have been served by multiple application server threads, or different application server JVMs (such as for web services requests). The Basic log and trace system does not provide that capability. HPEL delivers this when used in combination with Cross Component Trace (XCT).

  • Remote access to logs/trace

    The Basic log and trace system provides the ability to view chunks of the SystemOut.log, SystemErr.log, or trace.log in the administrative console (WebSphere Integrated Solutions Console). HPEL provides a greatly improved user interface capable of filtering and browsing content more easily. HPEL also provides the ability to export log and trace content from the administrative console without need for operating system access. HPEL log and trace content can also be remotely accessed via a JMX MBean interface, or via the HPEL API.

  • Access to z/OS® logs/trace

    If you run your applications on the z/OS platform, you might have difficulty getting your application developers to look at the log and trace content when problems occur, as they are more comfortable in a Windows® or Linux® environment. HPEL lets z/OS and other operating system teams work with logs and trace in a consistent way.

If you have used previous versions of WebSphere Application Server, you're probably familiar with the main run time log and trace files. On distributed systems, these are the SystemOut.log, SystemErr.log, trace.log, and activity.log. HPEL replaces all of these files with a log repository, a trace repository, and an optional text log.

The Basic log and trace system stores log content in multiple places, including the SystemOut.log, the service log (activity.log), and the trace.log (whenever trace is enabled). System.err content is stored separately – in the SystemErr.log – making it challenging to correlate with content in the SystemOut.log. Trace content is stored in the trace.log. This makes the trace.log the primary log for problem determination when trace is enabled, and the SystemOut.log the primary log for problem determination when trace is not enabled.

In contrast, HPEL stores log, System.out, and System.err content in a log data repository, and trace content in a trace data repository (Figure 1).

To view the content from the log data repository and trace data repository, you can use the logViewer command line tool. The logViewer command provides access to all content in both repositories. It also enables live monitoring – similar to running tail -f SystemOut.log – and enables filtering so you can restrict what content it provides as output.

HPEL also provides a text log (which is optional, but enabled by default). The text log mirrors the content of the log data repository, but does so in plain text, with the same format you're used to from SystemOut.log. The text log can be enabled to additionally mirror the content of the trace data repository, though for performance reasons, you should avoid doing that on a production system. The text log is handy if you need a plain text log for any reason.

Figure 1. Comparison of where content is stored in Basic and HPEL modes
Figure 1. Comparison of where content is stored in Basic and HPEL modes

Configuration

The switch from Basic mode to HPEL mode is done on a server-by-server basis. The main Troubleshooting > Logs and Trace > serverName panel provides a button with which to switch each server to HPEL. Switching back to Basic is just as easy, but is done via the Change log and trace mode link on the same panel. In either case, you will need to restart the server whose log and trace mode you changed in order for changes to take effect.

The administrative console also provides HPEL configuration options for the log data repository, trace data repository, and text log, and lets you independently specify data retention and buffering options for each.

Be aware that HPEL can be configured for use on your deployment manager and node agent processes as well. Enabling HPEL on your node agent process is a good idea, as it enables you to use the Log Viewer in the deployment manager GUI to access the logs of any server on that node, even when those servers are stopped.


New in WebSphere Application Server V8.5

HPEL was first introduced in WebSphere Application Server V8.0. The initial design included the ability for the log and trace data repositories to have extensible records, but that capability was not surfaced to application developers or administrators.

In WebSphere Application Server V8.5, the application server uses this extensibility to store the application name in log and trace records created during requests to Java™ EE applications. It also uses this capability to store the Cross Component Trace requestID, created when Cross Component Trace is enabled, in the log and trace records. Finally, an API is provided in V8.5 to enable you to add your own extensions to log and trace records.

To make use of these extension fields, the logViewer command line tool was extended to include an -includeExtensions option, enabling it to filter log and trace records by the content of extension fields.

Out of the box, WebSphere Application Server V8.5 tracks appName and requestID extensions, but you can feel free to add your own extensions to your log and trace entries using the com.ibm.websphere.logging.hpel.LogRecordContext API.

Let's take a practical look at how you might typically use HPEL.


Using logViewer command

When problems occur on your production application servers, it can be a time consuming task to discover and isolate the most important content in your log and trace files. Imagine that somewhere in your server's log files an error occurred on one of your systems.

Traditionally, you would have three options for finding problems in your log files:

  • Look at each log manually, perhaps grep-ing for " E " or " W " or "Error" or "Exception," and so on.
  • Write scripts to do the above searches.
  • Use tools to mine your logs for warnings, errors, and exceptions.

HPEL stores its log and trace data efficiently in binary repositories, which you can then access, filter, and format using the logViewer command line tool.

In its simplest form, you can output all content from your logs and trace using this simple command from your profile bin directory (output shown in Figure 2):

./logViewer.sh

Figure 2. Output from logViewer.sh command
Figure 2. Output from logViewer.sh command

Let's try to do something more interesting though. To find any warning or error messages in your log file you can use this command (output shown in Figure 3):

./logViewer.sh -minLevel warning

Figure 3. Output from logViewer.sh command
Figure 3. Output from logViewer.sh command

Notice in Figure 3 that the server was restarted, and by default the logViewer output includes the content from all ”instances” of the server. Importantly, logViewer never alters the repositories, it just provides a filtered, formatted view.

Now, let's further restrict your query to only see content since the last time you restarted your server. That's easy too, with the -latestInstance option (output shown in Figure 4):

./logViewer.sh -minLevel warning -latestInstance

Figure 4. Output from logViewer.sh command
Figure 4. Output from logViewer.sh command

Maybe, you've even got a dozen applications running on the same WebSphere Application Server JVM -- and you'd like to check for warning or errors in your LemonadeStand application. Using WebSphere Application Server V8.5, you can filter your log and trace content by Java EE application name, leveraging the appName extension as follows (output shown in Figure 5):

./logViewer.sh -minLevel warning -latestInstance -includeExtensions appName="LemonadeStand"

Figure 5. Output from logViewer.sh command
Figure 5. Output from logViewer.sh -minLevel warning -latestInstance -includeExtensions appName=”LemonadeStand” command

Now that you've found an interesting error in your logs, you might want to see what happened during the rest of the request on the same thread. This time, you can also eliminate any trace records by setting the minLevel option to info (output shown in Figure 6):

./logViewer.sh -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel info

Figure 6. Output from logViewer.sh command
Figure 6. Output from logViewer.sh command

Using HPEL with Cross Component Trace

If you are using HPEL in combination with the new Cross Component Trace (XCT) capability introduced in WebSphere Application Server V8.5, and the error you are interested in occurred during an HTTP or SIBus request, you can filter your logs by (the XCT supplied) requestID. To see the requestIDs, rerun the previous query with the -format advanced option (output shown in Figure 7):

./logViewer.sh -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel info -format advanced

Figure 7. Output from logViewer.sh command
Figure 7. Output from logViewer.sh command

With the output in advanced format, you can see the requestID for that message. As another tip, viewing your logs in advanced format is also a great way to get the full logger name from the log entry (which appears as the 'source' field in log/trace records); this is helpful in case you want to enable more granular tracing for a particular logger. Similarly, you can see the name of the thread that logged/traced each entry in the advanced format.

Armed with the requestID of the request that had the error you are interested in, you can now query the repository to see other log and trace entries that are present in the same request. HPEL stores the requestID as an extension to log and trace records, so you use the -includeExtensions option, as follows (output shown in Figure 8):

./logViewer.sh -includeExtensions requestID=AAAKLwUVkuH-AAAAAAAAAAC

Figure 8. Output from logViewer.sh command
Figure 8. Output from logViewer.sh command

Share your log and trace data

Having isolated a problem in your log files, you might want to send the logs for your LemonadeStand application to a teammate. You can store your output in a log in plain text using the -outLog option:

./logViewer.sh -includeExtensions appName="LemonadeStand" -outLog /tmp/myLogs.log

Alternatively, you could store your filtered output as a new HPEL repository, using the extractToNewRepository option, so your colleagues can use the logViewer command on their system to continue to manipulate the data:

./logViewer.sh -includeExtensions appName="LemonadeStand" -extractToNewRepository /mySharedDrive/myLogs

Upon receiving word of your great work, your colleague could then access what you stored in the log. To do so, they would specify the location of the repository using the repositoryDir option:

./logViewer.sh -repositoryDir /mySharedDrive/myLogs


View log or trace data in real time

When developing or troubleshooting applications, it's often helpful to be able to monitor the log output in real time. Perhaps you are aware of a problem and are trying to get it to reoccur, or you want to watch for a specific event as part of testing your application. At one point or another, you've probably found a need to fire up a shell window and execute a tail -f SystemOut.log to monitor your log output.

HPEL can make this type of task easy by enabling you to monitor a log repository for new events without having to monitor multiple log files or worry about files rolling over, interfering with your ability to tail them. Try it out by simply invoking the HPEL logViewer tool with the -monitor option:

./logViewer.sh -monitor

A key benefit with using HPEL's monitor option is that it works across platforms, providing you a consistent option for monitoring new log and trace events, regardless of your hardware or operating system. This includes the z/OS® platform, so it's possible to easily view or work with your application logs on z/OS without needing to be a z/OS system administrator or have access to log spools.

The monitor feature can be combined with filtering options to watch for specific events. Imagine you need to monitor a verbose logging application for critical error messages that pertain to a specific username that is part of a message. You could do this with something like the following to see all new severe or fatal log events that contain the string “user33” as part of their message, without the need for grep or other tooling:

./logViewer.sh -monitor -minLevel SEVERE -message “*user33*”

As you might expect, it is possible to combine multiple filter options to further fine tune what log events you want displayed. For example, you could further limit the records displayed to those generated by the LemonadeStand application as follows:

./logViewer.sh -monitor -minLevel SEVERE -message “*user33*” -includeExtensions appName=”LemonadeStand”

By combining options for the logViewer tool, administrators can easily view all or only the specific log data that is required directly from the HPEL repository. LogViewer provides a consistent and powerful tool for viewing WebSphere Application Server log and trace data regardless of platform. It does not require that any special tooling be installed or administrators to copy log files to their workstations.

Now that you've seen how easy it can be to use HPEL to view live log data and filter that output from a command line, lets look at how easy it is to view log data from within the WebSphere Application Server administrative console.


View log or trace data from the admin console

Another common scenario that many system administrators may have faced is the need to ensure limited operating system access to key production application servers, while still allowing application support teams or development groups access to the production application's logs. WebSphere Application Server now provides an improved web-based log viewing and export utility via the administrative console when HPEL is enabled.

While Basic mode log and trace has for some time provided simple log viewing capabilities, HPEL provides an improved console log viewing experience. The HPEL Log Viewer (not to be confused with the logViewer command line tool) can be used to view HPEL log and trace repository data by anyone authorized to access the administrative console. An application support specialist with Monitor role access to the administrative console can view log data without needing OS level access to the production servers or having to wait for their system administrator to collect the log files on their behalf.

Figure 9. Administrative Console view of log contents
Figure 9. Administrative Console view of log contents

The console's Log Viewer provides all the same powerful filtering options found in the command line logViewer tool. To view log and trace data via the administrative console, navigate to Troubleshooting > Logs and trace and select your HPEL enabled application server from the list, then choose the View HPEL logs and trace link on the Logging and tracing panel.

The full log repository (or even a filtered subset of the repository) can be exported to a .zip file for local analysis, if required, directly from the administrative console's Log Viewer. Simply select the export button at the top of the Log Viewer. You can choose to export the entire repository or just your current (filtered selection) to a new binary HPEL repository. Anyone you share the repository with can use the logViewer command to manipulate the file later. If preferred, you can export the entire repository or the current selection to a text file in basic or advanced format as well. The logViewer command-line tool can read HPEL repositories from inside an exported zip file -- no unzipping needed.

Figure 10. Administrative Console export log dialog
Figure 10. Administrative Console export log dialog

Conclusion

HPEL utilizes a new way to store log and trace data for improved performance and extensibility. At the same time, HPEL maintains compatibility with your existing log manipulation scripts by providing the same data formats you are already familiar with from WebSphere Application Server logs. New command line and administrative console tools expedite the tasks of log and trace monitoring and filtering. In addition to the tools provided, tool providers can utilize the new HPEL Java API, which provides access to HPEL repositories locally as well as remotely, and output results in basic or advanced text formats, or even store them in a new HPEL repository (see Resources).

Switching to and from HPEL in WebSphere Application Server can be done quickly in the administrative console and requires a server restart to take effect. This ability to easily switch between logging modes means you can try out HPEL and switch back if you find any need for the Basic mode. IBM support teams are able to work with HPEL repositories – whether exported from your administrative console, output from the logViewer command, or simply zipped up from your server logs directory.

Resources

Learn

Get products and technologies

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=828518
ArticleTitle=System administration in WebSphere Application Server V8.5, Part 3: High Performance Extensible Logging (HPEL)
publish-date=08012012