IBM Support

Set up trace and get a full dump for WebSphere Liberty

Troubleshooting


Problem

Use this document to set up trace strings and request dumps for WebSphere Application Server Liberty since there is no administrative console available.

Resolving The Problem


For tracing components in traditional WebSphere Application Server refer to MustGather: Read first for WebSphere Application Server.

To request dumps for Liberty running with IBM SDK:

Run the Liberty dump command on the target Liberty server to request the desired dump files. The dump type can be any combination of thread, heap, or system. It is specified by:

  1. --include=thread
  2. --include=heap
  3. --include=system
  4. --include=thread,system,heap (or any combination)

NOTE: For Oracle SDKs the only supported type is "heap".


The location of the dump file will be returned when the command completes:
  • For Windows platforms, run:
    <LIBERTY_HOME>\bin\server.bat dump <serverName> --include=<desired dump type>

    For UNIX platforms, run:
    <LIBERTY_HOME>/bin/server dump <serverName> --include=<desired dump type>
     
  • Collect the resulting dump zip file.
    Note: The current Liberty dump command also gathers deployed applications.
     
Configuring Full Core Dumps
The OS must be set up to produce full system core dumps.  Please reference the appropriate information for your operating system:
"Guidelines for setting ulimits"
http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21469413
"Crash on AIX produces no core or a truncated core"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21052642
** Steps 1 - 3 ONLY
"Crash on Linux produces no core or truncated core"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21115658
** Steps 1 - 3 ONLY
"MustGather: Crash on Windows"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21053924
** Section "Configuring Full Core Dumps"


Enabling Trace on Liberty:

There are 2 formats for trace output - binary or HPEL, which uses fewer resources and disk space but has to be formatted to be readable and BASIC, which is readable. Binary HPEL format requires an addition to the bootstrap.properties file before starting servers and includes binary trace in the trace specifications. Once the bootstrap.properties file is edited, trace is picked up dynamically for both formats. . The trace strings are specific to a module or component within WebSphere Liberty that you want to troubleshoot. Examples are for the BASIC format.

  1. To enable tracing on the target Liberty server:
    1. To enable binary logging, first add this line to the bootstrap.properties file in the LIBERTY_HOME/usr/servers/<server> location:

      websphere.log.provider=binaryLogging-1.0
       
    2. Open the server.xml file in the root directory of the target server:
      <LIBERTY_HOME>/usr/servers/<server>

      Where LIBERTY_HOME is the location of the extracted binary or is <AppServer>/wlp if Liberty is installed with WebSphere Application Server Version 8.5
      Add the <logging> tag to the server.xml file with the following elements:

      For example, to trace for the web container component, you would add the following snippet to the server.xml file, with the appropriate traceSpecification not this example uses binaryTrace format (explanation to each of these elements is provided in the product documentation)
           
      <logging        traceSpecification="com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all"
      traceFileName="trace.log"
      maxFileSize="20"
      maxFiles="10"
      traceFormat="BASIC"       />
       
  2. If possible, stop the server and clear the logs under the following directory:
    <LIBERTY_HOME>/usr/servers/<serverName>/logs
     
  3. Recreate the problem and collect the information.
     
  4. Follow instructions to send diagnostic information to IBM Support.


Trace specifications available

The following trace strings can be used as the trace specification with the instructions above to diagnose an issue in a specific component:

  • Aries / OSGi
    org.apache.aries.jndi.*=all
  • Classloading
    ClassLoadingService=all:SharedLibrary=all

    Also, so that native JVM classloader tracing is captured, add the following JVM argument to <LIBERTY_HOME>/usr/servers/<server name>/jvm.options Add the argument, save, and stop-restart the server:
    -verbose:class

    Note: osgi classloading issues require the use of the OSGI console see section on OSGI console
  • Configuration
    com.ibm.ws.config.*=all
  • Connections/Connection Pooling/Transaction RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all:ConnLeakLogic=all
  • Deploy (EAR/WAR/EBA)
    app.manager*=all
  • EJB
    EJBContainer=all:Injection=all
  • Java Batch
    com.ibm.jbatch.*=all:com.ibm.ws.jbatch.*=all
  • Java Management Extensions (JMX) or JMX client API (Server)
    jmx.rest.server.connector=all:com.ibm.ws.jmx.connector.server.rest.*=all
  • Java Persistence API (JPA)
    JPA=all
  • Java Transaction Service (JTS)
    Transaction=all:com.ibm.ws.transaction.services.WebAppTransactionCollaboratorImpl=all
  • JavaServer Pages (JSP)
    com.ibm.ws.jsp=all
  • JAX-RS (RESTful Web Services)
    com.ibm.ws.jaxrs*=all:com.ibm.websphere.jaxrs*=all:org.apache.wink.*=all:org.apache.cxf.*=all
  • JAX-WS (web services)
    org.apache.cxf.*=all:com.ibm.ws.jaxws.*=all
  • JNDI and Naming
    Naming=all:NamingService=all
  • JSF
    com.ibm.ws.jsf*=all:org.apache.myfaces*=all:com.ibm.ws.webcontainer*=all:com.ibm.ws.jsp=all
  • Managed Executors and Thread Context Propagation
    concurrent=all
    If an issue with concurrency constraints (for example, max async or max queued),
    concurrent=all:concurrencyPolicy=all
    If an issue with context propagation to managed executor tasks,
    concurrent=all:context=all
    If an issue with the underlying thread pool,
    concurrent=all:concurrencyPolicy=all:Threading=all
  • MQJMS:
    If the IBM® MQ resource adapter is installed using either the wmqJmsClient-1.1 or wmqJmsClient-2.0 feature, 
    enable the following trace:
    *=info:JMSApi=all:Transaction=all:WAS.j2c=all:EJBContainer=all:Injection=all
     
    If the IBM® MQ resource adapter is installed using generic JCA support, trace can be enabled by adding the following entry to the WebSphere Liberty file jvm.options:
    -Dcom.ibm.msg.client.commonservices.trace.status=ON
    -Dcom.ibm.msg.client.commonservices.trace.outputName=C:\Trace\MQRA-WLP_%PID%.trc
    This results in trace being written to a trace file called MQRA-WLP_<process identifier>.trc in the directory C:\Trace.
  • ORB:
    For IIOP and Yoko ORB issues:
    IIOP=all:org.apache.yoko.*=all:yoko.verbose.*=fine

    For JNDI and Naming issues:
    Naming=all

    CosNaming issues can involve both IIOP and JNDI, so enable both sets:
    Naming=all:IIOP=all:org.apache.yoko.*=all:yoko.verbose.*=fine
     
  • Plug-in Generation
    com.ibm.ws.webcontainer.osgi.mbeans.*=all
  • PMI/Performance Tools

    General:
    com.ibm.websphere.monitor.*=all:com.ibm.ws.monitor.*=all:com.ibm.websphere.pmi.*=all:com.ibm.websphere.pmi.*=all:com.ibm.ws.monitors.*=all

    In case the ConnectionPoolStats MBean is required, use the following instead of the above PMI trace string:
    *=info:com.ibm.websphere.monitor.jmx.*=all:com.ibm.websphere.monitor.meters.Meter*=all:com.ibm.websphere.monitor.meters.Counter*=all:com.ibm.websphere.monitor.meters.Gauge*=all:com.ibm.websphere.monitor.meters.StatisticsMeter=all:com.ibm.websphere.monitor.meters.StatisticsReading=all:com.ibm.ws.monitor.*=all:com.ibm.websphere.pmi.*=all:com.ibm.ws.monitors.*=all
  • Servlet Engine/Web Container
    com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all
  • Sessions and Session Management
    com.ibm.ws.session.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all

    In addition,:
    RRA=all (for database persistence issues)

    Additional trace for scheduler problems:persistentExecutor=all:concurrent=all:EJBContainer=all:RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all  
  • SIBJMS connection problems
    SIBTrm=all
  • SIBJMS Communications and TCP/IP
    SIBCommunications=all:SIBJFapChannel=all:TCPChannel=fine:com.ibm.io.async.*=all
  • SIBJMS Client Applications
    SIBJms*=all:SIBCommunications=all:SIBJFapChannel=all:        
    SIBMessageTrace=all:SIBTrm=all:SIBJmsRa=all:SIBRa=all
  • SIBJMS locked messages
    SIBProcessor=all:SIBMessageTrace=all
  • SIBJMS MDB
    SIBMessageTrace=all:SIBJmsRa=all:SIBRa=all
  • SIBJMS message format and schema
    SIBMfp=all:SIBCommunications=all
  • SIBJMS message processor
    SIBProcessor=all:SIBMessageTrace=all
  • SIBJMS message store
    SIBMessageStore=all
  • SIBJMS performance
    SIBMessageTrace=all
  • SIBJMS publish/subscribe
    SIBMatchSpace=all:SIBProcessor=all
  • SIBJMS security
    SIBSecurity=all
  • WebSocket
    websockets=all:HTTPChannel=all:TCPChannel=all:com.ibm.ws.webcontainer.*=all:com.ibm.wsspi.webcontainer.*=all:com.ibm.ws.webcontainer31.*=all:HTTPDispatcher=all

Please see the full trace specifications in the related information URL.

[{"Product":{"code":"SSD28V","label":"WebSphere Application Server Liberty Core"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF022","label":"OS X"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 January 2022

UID

swg21596714

Page Feedback