Contents


Collect troubleshooting data for WebSphere Liberty for Java on Bluemix

A developer's quick guide of the most commonly asked questions

Comments

To troubleshoot IBM® WebSphere® Application Server Liberty for Java applications in IBM Bluemix®, you must collect data on the state of the running application. This troubleshooting data includes thread dumps, heap dumps, trace logs, HTTP access logs, the messages.log file, the console.log file, first-failure data capture (FFDC) logs, and verbose garbage collection (GC) logs.

Many articles are written about how to collect troubleshooting data from on-premises applications that run in the Liberty profile, such as the following examples:

Other articles are written about common problems that you might see when you use Liberty for Java in Bluemix and how you can quickly resolve those problems, such as these examples:

But, how you collect the required troubleshooting data from Liberty for Java on Bluemix is different. This article fills that gap. It explains how to collect the troubleshooting data from Liberty for Java on Bluemix by addressing frequently asked questions (FAQs). After you collect this data, you can diagnose the problem by using various no-cost diagnostic tools, such as the following examples:

Documentation conventions

This article uses the following conventions:

  • Cloud Foundry command-line interface (CLI) commands. You must run the Cloud Foundry CLI commands, as indicated by cf. These commands are preceded by the dollar sign ($).
  • Listings. All configuration files (for example, XML and YAML files) are snippets of the actual files and not the complete file. Any deleted lines from the files are shown as an ellipsis (...).

System requirements

The information in this article applies to the following systems:

  • Bluemix Cloud Foundry: DEA and Diego architectures
  • Liberty for Java buildpack V3.0 and later
  • Java runtime environments (JRE): IBM JRE and OpenJDK V7 and later

Bluemix Cloud Foundry: DEA and Diego architectures

Bluemix is among the world's largest Cloud Foundry implementations. Droplet Execution Architecture (DEA) was the original Cloud Foundry architecture. The Diego architecture, which is the current Cloud Foundry architecture, offers enhanced capabilities for developing cloud native applications, and for hosting and constructing cloud platforms.

By using Bluemix, you can run applications on both architectures, but Diego is the default architecture, with all new applications deploying to the Diego architecture. Existing applications that run on the DEA architecture will continue on the DEA architecture for some time, but they will need to be migrated to the Diego architecture. For more information about how to migrate applications, see Upgrading from DEA to Diego architecture. To learn more about the Diego architecture, see:

The information in this article applies to applications that run on either the DEA architecture or the Diego architecture. Most of this information is the same for both architectures, with differences between the architectures noted. To determine whether your application is running on the DEA architecture or the Diego architecture, enter the following command:

$ cf has-diego-enabled <appname>

A response of true indicates that the application uses the Diego architecture. A response of false indicates that the application uses the DEA architecture. If you receive the error message that indicates that "has-diego-enabled" is not a valid command, install the Diego-Enabler plug-in.

Liberty for Java buildpack V3.0 and later

The Liberty for Java buildpack provides a complete runtime environment to run Java Enterprise Edition (Java EE) 7 applications on the Liberty runtime on the Bluemix Cloud Foundry platform as a service (PaaS). The current version of Liberty buildpack is V3.8, which provides the Liberty runtime V17.0.0.1 stable release by default or the Liberty runtime V2017.2.0.0 monthly release if you explicitly enable it.

The Liberty for Java buildpack is frequently updated on Bluemix. Therefore, if you want to target a specific version, specify that version in the application's manifest file or use the cf push command with the -b option. For the list of available Liberty for Java buildpacks, see Latest updates to the Liberty buildpack.

The information in this article applies to Liberty for Java buildpack V3.0 and later.

Java Runtime Environment: IBM JRE and OpenJDK V7 and later

The Liberty for Java buildpack also provides the Java runtime environment. The default JRE is the IBM JRE. Optionally, you can enable OpenJDK by setting the application's ENV environment variable to 'openjdk'. To set this variable, use the cf set-env command or the application's manifest file as shown in the following example:

# manifest.yml 

applications:  
- name: <appname>  
env: 
   ENV: 'openjdk' 

...

You can customize either of the JREs by setting the environment variables as explained in Customizing the JRE.

The information in this article applies to both the IBM JRE and the OpenJDK JRE. Any differences in collecting troubleshooting data from the JREs on Bluemix are noted, and both solutions are detailed. Although the information in this article was not tested on the OracleJDK JRE, the same (or similar) solutions that are provided for the OpenJDK JRE apply to the OracleJDK JRE because the OpenJDK JRE is based on the OracleJDK JRE.

1

How do I generate a thread dump?

With Liberty for Java, you can generate thread dumps by using the Bluemix console. To generate a thread dump:

  1. Log in to Bluemix.
  2. Select the application, and click Runtime.
  3. In the Runtime panel, select one of the running Instances from which to generate the dump, and click Dump.
    Figure 1. Select instance from which to generate the dump
    Select instance from which to generate the dump
    Select instance from which to generate the dump
  4. In the next window (Figure 2), generate the thread dump. In most cases, generating a thread dump does not cause a performance impact. Therefore, you might want to generate multiple thread dumps for troubleshooting purposes:
    1. Select Thread Dump, and then click Generate Dump to generate a single thread dump for the selected instance.
    2. Optional: If you want to generate multiple thread dumps, click Generate Dump again to generate another thread dump. Each thread dump is listed in the File menu.
    3. When you are finished, click Close to close the window.
    Figure 2. Generate a thread dump
    Generate a thread dump
    Generate a thread dump
2

How do I download a thread dump?

To download thread dumps, use the Cloud Foundry CLI.

DEA architecture

For applications that use the DEA architecture, enter the cf files command to get the trace.log file:

$ cf files <appname> dumps/javadump.20170404.141344.40.0002.txt

Diego architecture

For applications that use the Diego architecture, enter the cf_ssh command to get the trace.log file:

$ cf ssh <appname> -c "cat dumps/javadump.20170404.141344.40.0002.txt" > javadump.20170404.141344.40.0002.txt
3

How do I interactively generate a heap dump?

To interactively generate a heap dump, you use the Bluemix console similar to the way you use it to generate thread dumps as explained in 1. How do I generate a thread dump?:

  1. Select one of the running Instances from which to generate the dump, and click Dump (see Figure 1).
  2. In the next window (Figure 3), select Heap Dump, and click Generate Dump. In the File box, the generated heap dump is in the *.hprof format because we set the application's Java virtual machine (JVM) to OpenJDK. If the application is using the IBM JRE, it produces the heap dump in the IBM *.phd format. For more information about these dump types, see Debugging from dumps.
    Figure 3. Generate a heap dump
     Generate a heap dump
    Generate a heap dump

Heap dumps can also be quite large. Their relative sizes vary based on their heap dump type. With the IBM PHD type, the heap dump can be as large as 20% of your application's memory limit. With the OpenJDK HPROF type, the heap dump can be as large as 100% of memory. If the heap dump is too large to fit in your application's disk quota, you might need to decrease your memory limit, which by default decreases your application's maximum heap size. Or, you might need to increase your disk limit if your application cannot run with a smaller heap. You can adjust these limits by using the cf scale command or the application's manifest file as shown in the following example:

# manifest.yml 

applications:    
- name: <appname>    
  memory: <memory_limit>    
  disk_quota: <disk_limit>  

...

Generating heap dumps not only consumes a lot of your disk allocation, but it also consumes a lot of your application's memory and CPU allocation. For that reason, use caution when you interactively generate heap dumps.

4

How do I generate a heap dump on an out-of-memory error?

By default, when an out-of-memory error occurs, the application is stopped, and another instance is started, but no heap dump is generated. This situation most likely occurs because the application's local file system is short-lived. That is, when the application stops, resources that are assigned to that instance are reclaimed. Therefore, any changes that are made to the local file system since the application was started, such as the heap dump, are removed. The instance is then restarted with a new local file system (disk image), and you no longer have the heap dump.

Although your application can generate the heap dump and write it to the local file system, the heap dump will disappear, in a matter of seconds, unless you delay the application from exiting. For this reason, you must generate the heap dump and delay the application from exiting to give you enough time to download the generated heap dump before it gets deleted.

To generate the heap dump and delay the application from exiting, use the cf push command with the -c option as shown in the following example:

$ cf push <appname> -c ".liberty/initial_startup.rb;sleep 5m"

In this example, you run the command in the same folder as the application's enterprise archive (EAR) or web archive (WAR) file and manifest file. Otherwise, you can specify the path names to the EAR or WAR file by using the -p option and to the manifest file by using the -filename option.

Now that you know how to delay the application restart long enough so that you can download the heap dump, you can configure the JVM to generate a heap dump on an out-of-memory error. Configuring the JVM to trigger the heap dump does not impact the performance of the running application. Generating the heap dump occurs only on an out-of-memory error. In addition, you need the heap dump to analyze any out-of-memory errors. For those reasons, configure the application (JVM) to generate the heap dump by using one of the following methods:

  • Set JAVA_OPTS in the application's manifest file.
  • Set the JVM options in the server configuration.

The following sections describe both options for configuring the JVM to generate a heap dump.

Option 1: Set JAVA_OPTS in the application manifest file (manifest.yml)

This approach is the best practice because you can push a stand-alone application without concern for the server configuration.

For applications that use the IBM JDK, set these Java options:

# manifest.yml 

... 

env: 
   JAVA_OPTS: '-Xdump:heap:events=systhrow,filter=ja va/lang/OutOfMemoryError'

For applications that use OpenJDK, set these Java options:

# manifest.yml 

... 

env: 
   JAVA_OPTS: '-XX:+HeapDumpOnOutOfMemoryError'

Option 2: Set the JVM options in the server configuration (server.xml)

If you use this approach, in addition to pushing the application, you must also push the server configuration. To push the server configuration as a packaged server or as a server directory along with the application, see Options for pushing Liberty applications.

For applications that use the IBM JDK, set these Java options:

# jvm.options 

-Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError 

...

For applications that use the OpenJDK, set these Java options:

# jvm.options 

-XX:+HeapDumpOnOutOfMemoryError 

...
5

How do I download a heap dump?

Download heap dumps by using the Cloud Foundry CLI.

DEA architecture

For applications that use the DEA architecture, you can use either of the following options:

  • Use the cf app command to get the application's globally unique identifier (GUID):
    $ cf app <appname> --guid
  • Use the cf curl command with the –output option to download heap dumps. This command works well with raw binary data.
$ cf curl /v2/apps/<app_guid>/instances/<instance_id>/files/dumps/java.20170403.171501.40.0001.hprof --output java.20170403.171501.40.0001.hprof

Conversely, do not use the cf files command to download heap dumps because it has trouble processing binary data. It might add characters to the heap dump, which creates problems for your diagnostic tools to parse them.

Diego architecture

For applications that use the Diego architecture, use the cf ssh command to download the generated dump files that might contain binary data:

$ cf ssh <appname> -c "cat dumps/ java.20170403.171501.40.0001.hprof" >
java.20170403.171501.40.0001.hprof

Don't use the cf files command because it is not available on the Diego architecture and cannot handle binary files.

6

How do I generate a system dump?

If you are not familiar with the value of system dumps, see the article Debugging from dumps. That said, at the time of writing this article, you cannot generate system dumps on the Bluemix Cloud Foundry platform, although the reason why is not known. It might be because of the performance implications of dumping all memory in the process, the storage implications of taking up a lot of disk space, or the security implications of storing and transferring the raw data in the process. Regardless, if you are interested in this feature, see Generating and obtaining system dumps from Liberty for Java on Bluemix.

Although the Bluemix Cloud Foundry platform does not support generating system dumps, you can generate HPROF heap dumps on OpenJDK, which is akin to system dumps as shown in Figure 3. HPROF heap dumps include all the Java memory contents, including the thread stacks and stack frame locals. However, it doesn't include native memory contents. For that reason, system dumps are better for evaluating excessive native memory usage. For example, with a system dump, you can look at all native memory, including direct byte buffers. See Excessive memory usage by direct byte buffers.

7

How do I enable trace logs?

With Liberty for Java, you can enable trace logs by using the Bluemix console. This process begins similar to the steps in 1. How do I generate a thread dump?

  1. On the Runtime panel, select one of the running instances, and click Trace (see Figure 1).
  2. Enable the trace logs (Figure 4):
    1. Enter the trace specification. In this example, we enabled the trace specification for the web container component. For the list of available trace specifications, see Setting up trace in the Liberty profile. Click Submit Trace. You now see the "Trace specification is set successfully" message.
    2. Scroll to the bottom of the panel, and click Close.
      Figure 4: Enabling trace logs
      Enabling trace logs
      Enabling trace logs
8

How do I download the trace logs?

To download the trace.log file, use the Cloud Foundry CLI as explained for each architecture.

DEA architecture

For applications that use the DEA architecture, enter the following command to get the trace.log file:

$ cf files <appname> logs/trace.log

Diego architecture

For applications that use the Diego architecture, enter the following command to get the trace.log file:

$ cf ssh <appname> -c "cat logs/trace.log" > trace.log
9

How do I enable HTTP access logs?

You can use HTTP access logs to track the number and response times of HTTP or HTTPS requests. These logs can contain essential troubleshooting data, but they cause the server to write out every request to the access log, causing it to consume a lot of your application's disk quota on Bluemix. For this reason, HTTP access logs are disabled by default. You can enable HTTP access logging to write the requests to the http_access.log file by adding the <accessLogging> element to the server.xml file as shown in the following example:

# server.xml 

... 

<httpEndpoint httpPort="${port}" id="defaultHttpEndpoin filepath='./http_access.log'/>  
</httpEndpoint> 

...

Enabling access logging limits the number of files to the current http_access.log file and two archives. It also limits their size to 20 MB a piece, which means that 60 MB of access logs can be stored on disk. These access logs are also automatically rotated.

If you enable access logs:

  • Limit the size and number of access logs. If you do not, you might exceed your application's disk quota. If you need to increase your disk quota, see the listing in 3. How do I interactively generate a heap dump? to set memory and disk limits in the manifest.yml file.
  • Push the updated server configuration (server.xml file) along with your application to Liberty for Java on Bluemix. You push a packaged server or a server directory as explained in Options for pushing Liberty applications.
10

How do I download HTTP access logs?

You download the http_access.log file by using the Cloud Foundry CLI as follows for the two architecture types.

DEA architecture

For applications that use the DEA architecture, enter the following command to get the HTTP access log file:

$ cf files <appname> logs/http_access.log

Diego architecture

For applications that use the Diego architecture, enter the following command to get the HTTP access log file:

$ cf ssh <appname> -c "cat logs/http_access.log" > http_access.log
11

How do I download the messages.log file?

Download the messages.log file by using the Cloud Foundry CLI.

DEA architecture

For applications that use the DEA architecture, run the following command to get the messages.log file:

$ cf files <appname> logs/messages.log

Diego architecture

For applications that use the Diego architecture, enter the following command to get the messages.log file:

$ cf ssh <appname> -c "cat logs/messages.log" > messages.log
12

How do I download the console.log file?

Liberty for Java does not have a console.log file. If you are familiar with the Liberty on-premises server, you know that the console.log file contains the redirected standard output and standard error from the JVM process. With Liberty for Java on Bluemix, the messages.log file contains the redirected standard output and standard error messages from the JVM. To see the standard output and standard error messages in Liberty for Java:

  1. Download the messages.log file as explained in 11. How do I download the messages.log file?.
  2. Look for entries in the messages.log file that are prefixed with SystemOut and SystemErr.
13

How do I download the FFDC log files?

You can download the first-failure data capture logs by using the Cloud Foundry CLI as follows for the two architecture types.

DEA architecture

For applications that use the DEA architecture, use the cf download command to download the FFDC logs:

$ cf download <appname> logs/ffdc

If you receive an error indicating that download is not a valid command, use the following commands to install the command:

$ cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org/ 
$ cf install-plugin cf-download -r CF-Community

The cf download command is similar to the cf files command except that you use the cf download command to download all the files in a directory. You can also use both commands to download textual data, but they cannot handle raw binary files. You can also use the cf download command to download a single file with the --file option.

Diego architecture

For applications that use the Diego architecture:

  1. Run the following command to see the FFDC log files:
    $ cf ssh <appname> -c "ls -l logs/ffdc"
  2. Download the individual FFDC log files:
$ cf ssh <appname> -c "cat logs/ffdc/ffdc_file_name>" > <local_ffdc_file_name>
14

How do I enable verbose GC log files?

You have two options to enable verbose GC log files as explained in the following sections.

Option 1: Enable verbose GC logs by using the manifest.yml file

For applications that use the IBM JDK, set these Java options:

# manifest.yml 

... 

env: 
   JAVA_OPTS: '-verbose:gc -Xverbosegclog:./verbosegc.%seq.log,5,50000'

For applications that use OpenJDK, set these Java options:

# manifest.yml 

... 

env: 
     JAVA_OPTS: '-verbose:gc -Xloggc:./verbosegc.log -XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDateStamps 
-XX:+PrintGCDetails'

Option 2: Enable verbose GC logs by using the jvm.options file

As a best practice, set Java options in the application's manifest file so that you can push a stand-alone application without concern for the server configuration. To be complete, you can also set Java options in the jvm.options file. If you decide to use this approach, see Options for pushing Liberty apps for details on how to push the server configuration as a packaged server or server directory.

For applications that use the IBM JDK, set these Java options:

# jvm.options 

# Enable verbose GC 
-verbose:gc 

# Specify alternate verbose GC log 
-Xverbosegclog:./verbosegc.%seq.log,5,50000 

...

For applications that use the OpenJDK, set these Java options:

# jvm.options 

# Enable verbose GC 
-verbose:gc 

# Specify alternate verbose GC log 
-Xloggc:./verbosegc.log 

# Specify additional verbose GC  options 
-XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=5 
-XX:GCLogFileSize=20M 
-XX:+PrintGCDateStamps 
-XX:+PrintGCDetails 

...
15

How do I download verbose GC log files?

To download the verbose GC log files, use the cf commands as follows for the two architectures.

DEA architecture

For applications that use the DEA architecture, run this command:

$ cf files <appname> app/wlp/usr/servers/defaultServer/verbosegc.log

Diego architecture

For applications that use the Diego architecture, run this command:

$ cf ssh <appname> -c "cat app/wlp/usr/servers/defaultServer/verbosegc.log" > verbosegc.log

Conclusion

This article answered the top FAQs that users have about collecting troubleshooting data for Liberty for Java on Bluemix. By reading this article, you learned how to download the essential data for troubleshooting applications that are running on the Liberty for Java buildpack in the IBM Bluemix Cloud Foundry platform. When you tap into this essential data, you can use your normal diagnostics tools to troubleshoot the problem. You can also open a service request and send this data to IBM Support to expedite the time in finding a solution.

Acknowledgements

The author thanks Kevin Grigorenko and MiYoung Hong for their valuable input and assistance.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Middleware
ArticleID=1045306
ArticleTitle=Collect troubleshooting data for WebSphere Liberty for Java on Bluemix
publish-date=05012017