IBM Support

Troubleshoot: Logging Problems in IBM WebSphere Application Server

Troubleshooting


Problem

Trace analysis information for the logging component in the WebSphere® Application Server.

Symptom

Does not display.

Resolving The Problem

This document contains troubleshooting information for logging problems in the WebSphere® Application Server. This can help address common issues with this component before calling IBM support and save you time.

Tab navigation

Overview

You will find common questions in this section that may help resolve your logging problems.


1. What are IBM recommendations regarding SLF4J and Logback?

In general, we recommend that customers use JUL when possible/practical. JUL provides a good, ubiquitous logging infrastructure, which is suitable for most applications, and integrates with WebSphere Application Server log and trace. Using JUL directly also avoids extra log/trace API "hops" that are introduced by using other log/trace frameworks that map to JUL. The WAS logs are generated using JUL Handlers connected to the root JUL Logger.

In case you don't want to use JUL, or you have code that is already instrumented with another logging package, SLF4J or Logback can be helpful. For example, you may have code already written with Log4J. Log4J log/trace is not routed to WAS log/trace files, and Log4J loggers don't show up in the WAS tree of loggers you can control from the admin console. Further, trace spec settings you make in WAS do not control Log4J logger settings. This is where SLF4J or Logback can come in handy, as you can use them to intercept Log4J calls and forward them to the logging implementation of your choice. In order to integrate well with WAS logs/trace, we would recommend you bind your SLF4J/Logback API to map your log/trace content to JUL -- this will give you the ability to see your logs/trace in the WAS log/trace files, and enable you to have easy admin console control over all of your loggers.

2. How can I get WAS logs to go to syslog or the Windows EventLog?

There are a number of places that users have asked to be able to direct logs to in the past. Most notably, a number of customers have asked for the ability to send logs to syslog.

WebSphere Application Server sends all of its logs to java.util.logging (JUL), and customers can customize what happens to the log/trace events by way of using JUL Handlers. Once content is sent to JUL, it can be handled in a variety of flexible ways:

1) By default, we store the content in our log files, such as SystemOut.log, trace.log, or the newer log and trace repositories of the High Performance Extensible Logging framework we added in WebSphere Application Server V8.0. We also send the log content over JMX notifications, which allows for remote access.

2) If customers want to send content to places like SMTP, syslog, event log, or a few other places, they can use SLF4J to route WAS logs to Log4J, and use the corresponding Log4J Appenders. In this configuration, WAS logging routes to java.util.logging, SLF4J's Handler routes the request to Log4J, and Log4J can be configured to write log records to a syslog Appender.

3) If users want to do something even more custom, they can write their own JUL Handlers and plug them into WebSphere Application Server.

3. I am unable to turn on WebSphere Application Server trace. The trace state sets to *=off?

The systemOut.log shows:
 
[5/26/15 14:10:44:746 EDT] 000000d9 ManagerAdmin  I   TRAS0018I: The trace state has changed. The new trace state is *=off.

The problem is related to the below two JVM arguments:
 
-Djava.util.logging.configureByLoggingPropertiesFile=TRUE
-Djava.util.logging.config.file=<path>/logging.properties

Verify if the above 2 JVM arguments are configured. If yes, temporary disable it so WAS trace can be enabled.

4. The WebSphere Application Server logging has stopped, what may have caused this?

The user might have disabled the trace by setting trace output to none.

Setting runtime trace output to none impacts the JVM logs. To do that, the runtime trace spec will be set to *=off, which means to turn off all log levels. If customers just want to disable the trace for debug information only, we recommend setting the runtime trace spec, accordingly, instead of setting runtime trace output to none. For example, set to *=info, which is the default.

Verify if the commons-logging.jar and commons-logging.properties exist in the JVM classpath. If yes, suggest to remove it and put them inside the application (with parent_last setting) or put them in the isolated shared library.

5. How can I separate my logs into separate files for each application?

In WAS V8.5, we added the application name to log and trace records stored using HPEL (High Performance Extensible Logging).  Since HPEL lets you format and filter your log content after it has been persisted to the HPEL log/trace repository, this lets you decide if you want to see your application content on its own (helpful for uncluttered debug of an application), or together with the application server log/trace content (helpful to see impact of server on applications, and vice versa).

For example, if you had 3 applications called app1, app2, and app3, you could create app specific logs as follows:
WAS_INSTALL_ROOT\bin> logViewer -includeExtensions appName=app1 > app1.log
WAS_INSTALL_ROOT\bin> logViewer -includeExtensions appName=app2 > app2.log
WAS_INSTALL_ROOT\bin> logViewer -includeExtensions appName=app3 > app3.log

or see everything in one file as follows:
 
WAS_INSTALL_ROOT\bin> logViewer > all.log

6. How can I add my own java.util.logging.Handler to WAS?

To create a Handler you need to extend the java.util.logging.Handler class as follows (this example also shows how to filter for only the most severe messages):
 
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.LogRecord;
public class MyHandler extends Handler {
    public void close() {
    }
    public void flush() {
    }
    public void publish(LogRecord record) {
        if (record.getLevel().intValue() >= Level.SEVERE.intValue()) {
            // do something...
        }
    }
}
In your application you can add that Handler to the root logger as follows:
import java.util.logging.Handler;
import java.util.logging.Logger;
...
Handler myHandler = new MyHandler();
Logger logger = Logger.getLogger("");
logger.addHandler(myHandler);
...

7. Why can't I set my own log levels using the Logger setLevel method?

The application server can run in two modes.  

By default, the logger levels are managed, and we expect any changes to logger levels to be done only through wsadmin or through the admin console.  When the trace spec update is applied, we explicitly set each logger to the level specified in the trace spec -- we do not leave any named loggers at level set to null.  So, while you can change a logger's level using the setLevel method, that change only lasts until you apply your next trace spec update from wsadmin or the admin console.  Basically, changes you make via logger.setLevel are temporary at best, and do not apply automatically to children since the children won't have a null level.

Alternately, you can have the server to be configured by the JRE's logging.properties file by setting the java.util.logging.configureByLoggingPropertiesFile property.  In this mode, the application server will not touch the logger levels -- you cannot update the levels through the admin console, nor through wsadmin.  See Java virtual machine custom properties for more information on this setting. In this mode, you can specify anything you would normally specify in your logging.properties file.  Be careful not to use a console handler though, as that will lead to an infinite loop, since we take data sent to System.out, turn it into LogRecords, and send it to the root logger.



8. How can I get java.util.logging to initialize from a jre/bin/logging.properties file in the application server?

You can do this via system property: java.util.logging.configureByLoggingPropertiesFile=true

Note that if you do set up your server in this mode, we will not manage your logger levels via the admin console.

Note also that we will ignore any console handler entries found in the logging.properties file, since we route System.out to the root logger, which would mean that a console handler would cause an infinite recursion.

If you do use this system property, and you want to change logger levels, you have two choices:

 
  • 1) Change the level in the logging.properties file, and reload the file using the LogManager.readConfiguration method

    2) Set the Logger levels programatically using the Logger.setLevel method

9. How can I get the full logger name from log and trace records?

The full logger name can be handy to know for enabling trace. How you get to it depends on what log and trace mode you use:

Basic log and trace mode:

The basic log format (not to be confused with basic log and trace mode), which is the default format for SystemOut.log / SystemErr.log / trace.log has a truncated version of the logger name:
 
[4/14/11 9:41:45:218 EDT] 00000021 QuickLogTest W hello fine 54857

There is no way to get the full logger name from basic formatted logs. You need to switch to advanced format for your logs/trace if you want to see the full logger name. Once in advanced format, log contents appear as follows:
 
[4/14/11 9:41:45:218 EDT] 00000021 W UOW= source=com.ibm.somelogger.QuickLogTest org= prod= component= thread=[WebContainer : 4]
hello fine 54857

HPEL log and trace mode:

You can use the HPEL LogViewer command line tool to output the log/trace content in either basic or advanced format.

> logviewer -format basic
 
....
[4/14/11 9:41:45:218 EDT] 00000021 QuickLogTest W hello fine 54857
....

> logviewer -format advanced
 
....
[4/14/11 9:41:45:218 EDT] 00000021 W UOW= source=com.ibm.somelogger.QuickLogTest org= prod= component= thread=[WebContainer : 4]
hello fine 54857
...

10. How can I write HPEL content to a database?

If you want to write to a Database, any of the following options will work:

1) Write your own log handler (that writes to a database) and plug it into JSR47. (see the Java doc for java.util.logging.Handler)

2) Write an HPEL reading Java application to read the content from an HPEL log repository and write it to a database

3) Write an HPEL reading Java application to read the content from an HPEL log repository and write it out in .csv format, then just use the common database commands to load the .csv content into the database.

11. How do you tune HPEL for optimal performance in tWAS?

In the server.xml, jvmEntries element:
- Ensure debugMode is set to false
- Ensure RAS JMX notifications are disabled by including -Dcom.ibm.ejs.ras.disablerasnotifications=true in genericJvmArguments
- Ensure -Xquickstart is not in the list of genericJvmArguments
 
<jvmEntries xmi:id="JavaVirtualMachine_1183122130078" verboseModeClass="false" verboseModeGarbageCollection="false" verboseModeJNI="false" runHProf="false" hprofArguments="" debugMode="false" debugArgs="-Dcom.ibm.ws.classloader.j9enabled=true -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" genericJvmArguments="-Dcom.ibm.ejs.ras.disablerasnotifications=true">

In hpelModel.xml:
- Ensure textLog is disabled.
 
<hpelTextLog xmi:id="HPELTextLog_1" enabled="false" dataDirectory="${SERVER_LOG_ROOT}"/>

12. How can I tell who is writing to System.err / System.out?

Add the following to your genericJvmArguments:
 
-Xtrace:print=mt,trigger=method{com/ibm/ejs/ras/System*.*,jstacktrace}

13. How can I identify what code is writing to System.out.println?

Add the following to your genericJvmArguments and restart the server to see a stack trace for each call to System.out.println:
 
-Xtrace:print=mt,trigger=method{java/io/PrintStream.println,jstacktrace}

A snippet of the stack trace for a sample application, which calls System.out.println(), with the above JVM argument set, which identifies that the Hello.printHello (Hello.java:15) writes to System.out.
 
03:41:01.245*0x3277900 mt.0 > java/io/PrintStream.println(Ljava/lang/String;)V Bytecode method, This = 7fffff49f98
03:41:01.247 0x3277900 j9trc_aux.0 - jstacktrace:
03:41:01.248 0x3277900 j9trc_aux.1 - [1] java.io.PrintStream.println (PrintStream.java:829)
03:41:01.248 0x3277900 j9trc_aux.1 - [2] com.ibm.jvm.io.ConsolePrintStream.println (ConsolePrintStream.java:311)
03:41:01.250 0x3277900 j9trc_aux.1 - [3] Hello.printHello (Hello.java:15)
03:41:01.251 0x3277900 j9trc_aux.1 - [4] Hello.main (Hello.java:9)Hello World
03:41:01.253 0x3277900 mt.6 < java/io/PrintStream.println(Ljava/lang/String;)V Bytecode method

14. How can I see who called a particular method?

Add the following to your genericJvmArguments:
 
-Xtrace:print=mt,trigger=method{java/util/logging/Logger.setLevel,jstacktrace}

15. What are the .owner files for in the log directory?

The server creates a .owner file for each of the log files. The file content indicates which cell/node/server is the rightful user of the similarly named log file. This is used as a sort of locking mechanism to make sure that users don't improperly configure their systems. If two or more servers are configured to use the same log directory and file name, we can detect it using the .owner file and we will print a warning in the log file.

For more information, see Java virtual machine (JVM) log settings

16. How can I correlate information from thread stacks in the javacore with information in my SystemOut.log file?

There are a few ways to correlate content in WAS logs/trace with content in javacore thread dumps:

Best option, if you are using WebSphere Application Server 8.0 / 8.5:

Applies to:
WAS 8.0.0.4 and above running with Java 626 SR4 and above
WAS 8.5.0.2 and above running with Java 7 SR3 and above or Java 626 SR4 and above

In recent releases, we have made it possible to have the thread id used in the WebSphere Application Server log/trace match with the thread id used in javacores.  In WAS 8.0.0.4 and above, you can use a system property to make the thread id in the logs/trace match the Java thread id.  In WAS 8.5.0.2 and above, the thread id used in the logs/trace matches with the Java thread id by default. For more information, see PM60913: Introduce ability to use Java Thread ID in Logs instead of LogRecord Thread ID and JVM log interpretation  

Another option if you are using High Performance Extensible Logging (HPEL) / Binary Logging:

Applies to:
Full profile WAS 8.0.0.0 and above
Liberty profile WAS 8.5.5.0 and above

Full Profile:
Use the HPEL logViewer command to output the logs in advanced format so you can see the full thread name:
> logViewer -format advanced            (basic is the default format)

Liberty Profile:
Use the binaryLog command line tool to output the logs in advanced format so you can see the full thread name:
> binaryLog view --format=advanced

You will get output that looks like the following:
 
[4/14/11 9:41:45:218 EDT] 00000021  W UOW= source=com.ibm.somelogger.QuickLogTest org= prod= component= thread=[WebContainer : 4]

The javacore contains the same thread name:
 
3XMTHREADINFO      "WebContainer : 4" J9VMThread:0x1578D700, j9thread_t:0x150D0D04, java/lang/Thread:0x02517CE0, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x15B0, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/io/async/AsyncLibrary.aio_getioev2(Native Method)

Another option if you are using Advanced format for your SystemOut.log / trace.log:

Applies to:
All currently supported WebSphere Application Server releases.

Change your log/trace format to Advanced.  The Advanced format contains the full thread name, as does the javacore.

Advanced format Log:
 
[4/14/11 9:41:45:218 EDT] 00000021  W UOW= source=com.ibm.somelogger.QuickLogTest org= prod= component= thread=[WebContainer : 4]

The javacore contains the same thread name:
 
3XMTHREADINFO      "WebContainer : 4" J9VMThread:0x1578D700, j9thread_t:0x150D0D04, java/lang/Thread:0x02517CE0, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x15B0, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/io/async/AsyncLibrary.aio_getioev2(Native Method)

17. How do I deal with TRAS0216W message from Liberty when using logstashCollector-1.0 feature?  
 

If you see the following message in your messages.log:

TRAS0216W: The logstash collector was unable to connect to the logstash server on the specified host logstash_host and port number 5043

Set -Djavax.net.debug=all in the Logstash jvm.options and then look for this: 

ssl: Ignoring alias key: signature does not conform to negotiated signature algorithms

where key is the key for the certificate that I made using openssl and the client sends the ClientHello message with its supported cipher suites, and then immediately gets a handshake_failed error (because the server, Logstash, could not reply because it doesn't have an SSL certificate)

Add -Djavax.net.debug=all to jvm.options for your Liberty server and restart. Look for this message in your messages.log when the SSL handshake occurs between Liberty and Logstash:

[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O [Raw read]: length = 5
[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O 0000: 15 03 03 00 02 .....
[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O [Raw read]: length = 2
[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O 0000: 02 28 ..
[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O Default Executor-thread-15, READ: TLSv1.2 Alert, length = 2
[6/4/19 17:28:46:514 EDT] 00000056 SystemOut     O Default Executor-thread-15
[6/4/19 17:28:46:515 EDT] 00000056 SystemOut     O , RECV TLSv1.2 ALERT:
[6/4/19 17:28:46:515 EDT] 00000056 SystemOut     O fatal,
[6/4/19 17:28:46:515 EDT] 00000056 SystemOut     O handshake_failure
[6/4/19 17:28:46:515 EDT] 00000056 SystemOut     O Default Executor-thread-15, called closeSocket()
[6/4/19 17:28:46:515 EDT] 00000056 SystemOut     O Default Executor-thread-15, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

This will indicate that Logstash is sending an error in reply to ClientHello. This is because Logstash doesn't have a certificate it can use.

Use openssl to test connection to Logstash:

openssl s_client -showcerts -connect logstash_host:logstash_port_number
CONNECTED(00000003)
140444464277312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1535:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 303 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

If you have the above problem, the workaround is to use an OpenJDK JVM to run Logstash instead of the IBM JDK. This will enable Logstash to recognize and use the certificate created by openssl.



What to do next 
If the preceding troubleshooting did not help solve your problem, see the Mustgather for logging problems to continue investigation.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD28V","label":"WebSphere Application Server Liberty Core"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF017","label":"Mac OS"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
19 June 2022

UID

swg21995397