Set up trace and get a full dump for WebSphere Liberty

Resolving The Problem

For tracing components in traditional WebSphere Application Server refer to MustGather: Read first for WebSphere Application Server.

To request dumps for Liberty running with IBM SDK:

Run the Liberty dump command on the target Liberty server to request the desired dump files. The dump type can be any combination of thread, heap or system. It is specified by:

1. --include=thread
2. --include=heap
3. --include=system
4. --include=thread,system,heap (or any combination)

NOTE: For Oracle SDKs the only supported type is "heap".

The location of the dump file will be returned when the command completes:

• For Windows platforms, run:
  <LIBERTY_HOME>\bin\server.bat dump <serverName> --include=<desired dump type>
  
  For UNIX platforms, run:
  <LIBERTY_HOME>/bin/server dump <serverName> --include=<desired dump type>
• Collect the resulting dump zip file.
  Note: The current Liberty dump command also gathers deployed applications.

Enabling Trace on Liberty:

There are 2 formats for trace output - binary or HPEL which uses fewer resources and disk space but has to be formatted to be readable and BASIC which is readable. Binary HPEL format requires an addition to the bootstrap.properties file before starting servers and includes binaryTrace in the trace specifications. Once the bootstrap.properties file is edited, trace is picked up dynamically for both formats. . The trace strings are specific to a module or component within WebSphere Liberty that you want to troubleshoot. Examples are for the BASIC format.

1. To enable tracing on the target Liberty server:
   1. To enable binary logging, first add this line to the bootstrap.properties file in the LIBERTY_HOME/usr/servers/<server> location :
      
      websphere.log.provider=binaryLogging-1.0
       
   2. Open the server.xml file in the root directory of the target server:
      <LIBERTY_HOME>/usr/servers/<server>
      
      Where LIBERTY_HOME is the location of the extracted binary or is <AppServer>/wlp if Liberty is installed with WebSphere Application Server Version 8.5
      Add the <logging> tag to the server.xml file with the following elements:
      
      For example, to trace for the web container component, you would add the following snippet to the server.xml file, with the appropriate traceSpecification not this example uses binaryTrace format (explanation to each of these elements is provided in the product documentation)
           
      <logging  traceSpecification="com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all"
      traceFileName="trace.log"
      maxFileSize="20"
      maxFiles="10"
      traceFormat="BASIC" />
       
2. If possible, stop the server and clear the logs under the following directory:
   <LIBERTY_HOME>/usr/servers/<serverName>/logs
    
3. Recreate the problem and collect the information.
    
4. Follow instructions to send diagnostic information to IBM Support.

Trace specifications available

The following trace strings can be used as the trace specification with the instructions above to diagnose an issue in a specific component:

• Deploy (EAR/WAR/EBA)
  app.manager*=all

• Connections/Connection Pooling/Transaction
  RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all

  For a ConnectionWaitTimeoutException, J2CA0045E, or Connection related issue, use this trace string
  RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all:ConnLeakLogic=all

• Classloading
  ClassLoadingService=all:SharedLibrary=all
  
  Also, so that native JVM classloader tracing is captured, add the following JVM argument to <LIBERTY_HOME>/usr/servers/<server name>/jvm.options Add the argument, save, and stop-restart the server:
  -verbose:class
  
  Note: osgi classloading issues require the use of the OSGI console see section on OSGI console

• Configuration
  com.ibm.ws.config.*=all

• JAX-RS (RESTful Web Services)
  com.ibm.ws.jaxrs*=all:com.ibm.websphere.jaxrs*=all:org.apache.wink.=all:org.apache.cxf.=all

• JAX-WS (Web services)
  org.apache.cxf.*=all:com.ibm.ws.jaxws.*=all

• JNDI and Naming
  Naming=all:NamingService=all

• JSF
  com.ibm.ws.jsf*=all:org.apache.myfaces*=all:com.ibm.ws.webcontainer*=all:com.ibm.ws.jsp=all

• JavaServer Pages (JSP)
  com.ibm.ws.jsp=all

• Java Management Extensions (JMX) or JMX client API (Server)
  jmx.rest.server.connector=all:com.ibm.ws.jmx.connector.server.rest.*=all

• Java Persistence API (JPA)
  JPA=all

• Java Transaction Service (JTS)
  Transaction=all:com.ibm.ws.transaction.services.WebAppTransactionCollaboratorImpl=all

• Plug-in Generation
  com.ibm.ws.webcontainer.osgi.mbeans.*=all

• PMI/Performance Tools
  
  General:
  com.ibm.websphere.monitor.*=all:com.ibm.ws.monitor.*=all:com.ibm.websphere.pmi.*=all:com.ibm.websphere.pmi.*=all:com.ibm.ws.monitors.*=all
  
  In case the ConnectionPoolStats MBean is required, please use the following instead of the above PMI trace string:
  *=info:com.ibm.websphere.monitor.jmx.*=all:com.ibm.websphere.monitor.meters.Meter*=all:com.ibm.websphere.monitor.meters.Counter*=all:com.ibm.websphere.monitor.meters.Gauge*=all:com.ibm.websphere.monitor.meters.StatisticsMeter=all:com.ibm.websphere.monitor.meters.StatisticsReading=all:com.ibm.ws.monitor.*=all:com.ibm.websphere.pmi.*=all:com.ibm.ws.monitors.*=all

• Aries / OSGi
  org.apache.aries.jndi.*=all

• Security
  MustGather: Security problems for WebSphere traditional and Liberty
  MustGather: SSL problems on WebSphere Liberty
  ​​​​​MustGather: Web Single Sign-on problems with WebSphere Application Server
  MustGather: Web Services Security (WS-Security) problems with WebSphere Application Server
  MustGather: SPNEGO problems on WebSphere Liberty

• Servlet Engine/Web Container
  com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all

• WebSocket
  websockets=all:HTTPChannel=all:TCPChannel=all:com.ibm.ws.webcontainer.*=all:com.ibm.wsspi.webcontainer.*=all:com.ibm.ws.webcontainer31.*=all:HTTPDispatcher=all

• Sessions and Session Management
  com.ibm.ws.session.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all
  
  In addition:
  RRA=all (for database persistence issues)
  
  Additional trace for scheduler problems:persistentExecutor=all:concurrent=all:EJBContainer=all:RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all  

• SIBJMS connection problems
  SIBTrm=all

• SIBJMS Communications and TCP/IP
  SIBCommunications=all:SIBJFapChannel=all:TCPChannel=fine:com.ibm.io.async.*=all

• SIBJMS Client Applications
  SIBJms*=all:SIBCommunications=all:SIBJFapChannel=all:        
  SIBMessageTrace=all:SIBTrm=all:SIBJmsRa=all:SIBRa=all

• SIBJMS locked messages
  SIBProcessor=all:SIBMessageTrace=all

• SIBJMS MDB
  SIBMessageTrace=all:SIBJmsRa=all:SIBRa=all

• SIBJMS message format and schema
  SIBMfp=all:SIBCommunications=all

• SIBJMS message processor
  SIBProcessor=all:SIBMessageTrace=all

• SIBJMS message store
  SIBMessageStore=all

• SIBJMS performance
  SIBMessageTrace=all

• SIBJMS publish/subscribe
  SIBMatchSpace=all:SIBProcessor=all

• SIBJMS security
  SIBSecurity=all

Please see the full trace specifications in the related information URL.