How-tos

How to Automatically Capture Diagnostic Data from Liberty Buildpack Applications

Share this post:

In a previous post entitled “How to deploy IBM Support Assistant 5 to store and analyze diagnostic data” I described how to deploy an IBM Support Assistant application and I said that it would be ideal to be able to transfer data directly from applications in Bluemix to the IBM Support Assistant application so there’s no need to download anything.

This is possible because the IBM Support Assistant provides a REST interface to upload files to an existing case. Additionally, the IBM runtime for Java that is included in the Liberty buildpack provides the ability to execute command lines or scripts on certain events. Putting the two together means it becomes possible to automatically upload files to the IBM Support Assistant application.

 

REST interfaces provided by the IBM Support Assistant

 

The IBM Support Assistant actually provides REST interfaces for both case creation and file upload, but its less complicated if you create the cases directly in IBM Support Assistant ahead of time. That way the case your uploading the data to can be pre-configured in your application.

As described above, the IBM runtime for Java is able to execute command lines on certain events, so the most straight forward way to call the REST interface supplied by the IBM Support Assistant is to use curl. The following curl command will upload a given file to a specified case through the REST interface:

    curl -X POST -v -k -o /home/vcap/logs/curl.log -F file=@'{file_name}' --url http://{host name}.mybluemix.net/rest/1/files/{case number}/file_upload?path=/{case number}/

In the above command, the following items need to be replaced:

  1. {file bame}: this is the path and name of the file that you wish to upload
  2. {host name}: this is the location of your IBM Support Assistant application
  3. {case number}: this is the case number for the IBM Support Assistant case you want to upload the file to

If you configured user accounts in IBM Support Assistant for security, you also need to add your user name and password to the curl command:

    curl --user {user name}:{password} -X POST -v -k -o /home/vcap/logs/curl.log -F file=@'{file_name}' --url http://{host name}.mybluemix.net/rest/1/files/{case number}/file_upload?path=/{case number}/

In the above command, you also need to replace {user_name} with your user name and {password} with your password.

Now that you know how to upload files, you need to be able to trigger the command line when a failure event occurs. You can do this using the -Xdump options on the IBM runtime for Java.

 

Using the dump engine in the IBM runtime for Java

 

One of the features of the IBM runtimes for Java is a dump engine that is configured using the -Xdump Java command line option.

This provides the ability to request a number of dumps types to be generated on any number of dump events.

Dump types

The available dump types include:

  • stack: this writes the status and Java stack of the thread hitting the event to stderr.
  • console: this writes a basic thread dump to stderr
  • system: this requests the creation of an operating system based core (memory) dump
  • java: this writes an application summary to a text file
  • heap: this writes a graph representation of the Java heap to a file
  • tool: this runs a command-line program

The last of these is what will allow us to run the curl command to upload diagnostic data files to the IBM Support Assistant deployment.

Dump events

Whilst there’s a number of events that the dump types can be triggered on, ranging from JVM start and stop through to exceptions being thrown and caught and classes being loaded or unloaded, I’ll explain how to trigger a tool dump on an OutOfMemoryError, and use that to upload data to the IBM Support Assistant.

Triggering a dump on an OutOfMemoryError requires the configuration of a dump on a systhrow event, an Exception thrown by the JVM, with a filter of java/lang/OutOfMemoryError. This can be achieved as follows:

    -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError

This requests a tool dump to be done when a systhrow event occurs with an identifier of java/lang/OutOfMemoryError.
The final piece is to the command line to run when the tool dump is triggered. This is done using an additional exec= clause to the configuration. The following will run the curl command to invoke the IBM Support Assistant REST call:
     -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec="curl --user {user name}:{password} -X POST -v -k -o /home/vcap/logs/curl.log -F file=@'{file_name}' --url http://{host name}.mybluemix.net/rest/1/files/{case number}/file_upload?path=/{case number}/"

More information on how to configure -Xdump options for the IBM Java Runtime is available in the “Using dump agents” section of the IBM Java 7.1 KnowledgeCenter

 

Automatically capturing and uploading data on an OutOfMemoryError

 

Now that we can automatically run curl to invoke the REST interface in the IBM Support Assistant on an OutOfMemoryError, we just need to make sure that we have some data to send, and know how to find it.

Multiple dump types can be triggered on the same event, and the order in which the dumps are triggered is configurable with the tool configured by default to run last. We can enable a heap dump OutOfMemoryErrors in addition to the running on the tool dump by adding the following Java command line option:

    -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError

This generates a file called heapdump{date}.{time}.{pid}.phd which is pre-configured in Bluemix to write to the /dumps directory.
The dump engine provides a %last substitution is replaced with the full name and location of the last dump file written, and that can be used to upload the heapdump file. However it more useful to be able to find an upload multiple data files. This can be done using the operating system find command. The configuration below searches for any file in the application container with the process ID in the file name, and uploads it to the IBM Support Assistant:
    -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec="find /home/vcap -name '*%pid*' -exec curl --user {user name}:{password} -X POST -v -k -o /home/vcap/logs/curl.log -F file=@'{}' --url http://{host name}.mybluemix.net/rest/1/files/{case number}/file_upload?path=/{case number}/ \;"

This could be modified to send the contents of the logs or dumps directories instead.

 

Summary

 

Combining the use of -Xdump:tool in Liberty buildpack applications with an IBM Support Assistant deployment in Bluemix means that you can automatically capture diagnostic data and analyse it using tools like Memory Analyzer without having to download anything locally.
ISA5 in Bluemix

Configurations can be created for any number of problem types. If you have problems scenarios that your struggling to configure -Xdump for, leave a comment and I’ll see if we can find a way to do it.

More stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading