IBM Support

MustGather: Object Request Broker (ORB) problems with WebSphere Application Server

Troubleshooting


Problem

Collecting data for problems with the IBM WebSphere Application Server Object Request Broker component. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.

Resolving The Problem

Tab navigation

Collecting data for problems with the IBM WebSphere Application Server Object Request Broker component. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.

Collecting data manually

Outline

Basic Diagnostic Questions 
Third Party ORB interoperability Questions 
Data to Collect 
Collector Tool
ORB trace 
WebSphere Application Server V8.5 and V9.0
ORBRas Trace Only (+ WAS Component Trace) 
ORBRas Trace + CommTrace 
Standalone (Thin) Java Client 
ORBRas Trace + CommTrace 
ORBRas Trace + CommTrace + WAS Component Trace 
J2EE Client
ORBRas Trace + CommTrace (+ WAS Component Trace)
Performance Considerations 
How to check IBM Java ORB build version in WebSphere Application Server

 

Basic Diagnostic Questions

  1. What is the WebSphere software version of both the client and server?  Is there a non-WebSphere client/server involved?
     
  2. What exception(s) are being seen in the logs?
       
  3. Is the problem reproducible at will or intermittent?
     
  4. How often and under what circumstances does this problem occur?
     
  5. Has this problem always occurred or has it just started to occur?  What, if any, recent changes have been made to the environment (WebSphere, JDK, application, OS, increased client load, etc)?

Third Party ORB interoperability Questions

  1. Provide Third-Party Server/ORB software name (e.g. WebLogic), version, language, and platform information.
       
  2. Is WebSphere Application Server being used as the server or client to the third-party ORB?
     
  3. Was the approach recommended in the CORBA Interop samples followed?
     
  4. Where was the Interface Definition Language (IDL) compiled (what tool was used)?
     

Data To Collect

Collector Tool

The collector tool output is required to troubleshoot an ORB issue.  The resulting collector.jar provides valuable configuration data that provides a better understanding of your topology and settings.

Detailed instructions on how to run the collector program can be found in the Knowledge Center:

Gathering information with the collector tool
 

NOTE:

  • Run the tool from the Dmgr.
  • Run the collector tool from a directory outside the WebSphere Application Server product installation root directory otherwise an error will occur. 
  • Upload the collector.jar file that is created.
ORB Trace
  • In most cases, a matching client and server-side trace is needed.
  • There are 2 basic kinds of ORB trace:
    • ORBRas
      • Basic ORB method tracing
    • ORB CommTrace
      • Payload data tracing for each incoming and outgoing ORB message
      • Not always required for every type of problem, but recommended in most cases
  • Can be configured 2 ways
    • Startup, requires a server restart
    • Runtime, no server restart required

WebSphere Application Server V8.5 and V9.0

 

ORBRas Trace Only (+ WAS Component Trace)

ORBRas trace can be configured 2 ways:

  • Startup (Static)
    • Settings are made in the Configuration tab for log setting menu panels
    • Requires a server restart
    • Remains ON through server restart(s)
  • Runtime (Dynamic)
    • Settings are made in the Runtime tab for log setting menu panels
    • No server restart required
    • Remains ON only until the server is restarted

NOTE:  The instructions below assume the use of Basic Trace Mode rather than High Performance Extensible Logging (HPEL) trace mode.  From the Admin Console path Troubleshooting > Logs and Trace > server_name click the Switch back to Basic Mode button to run in Basic Mode.

Instructions

  1. Change the trace spec string.
    1. Administrative Console path: 
      Servers > Server Types > Application Servers > server_name > Change log detail levels
    2. Select the Configuration or Runtime tab.
    3. Remove any previous entries in the text field and enter:
      *=info:ORBRas=all
    4. Add any other WebSphere component trace string(s), if desired:
      *=info:ORBRas=all:WLM*=all:Naming=all
    5. Select Apply or OK to save changes.
       
  2. Change the log size and number of log files.
    1. Administrative Console path:  
      Servers > Server Types > Application Servers > server_name > Diagnostic Trace Service
    2. Select the Configuration or Runtime tab.
    3. Change the following fields:
      • Maximum Number of Historical Files to 10
      • Maximum File Size to 100MB
    4. Ensure Trace Output Format is set to Basic
    5. Select Apply or OK to save changes.
       
  3. Restart the sever if using Startup trace and recreate the problem.  No restart is required if using Runtime trace.
     
  4. Collect the following data:
     profile_root/logs/server_name/trace_<timestamp>.log
    profile_root/logs/server_name/trace.log
    profile_root/logs/server_name/SystemOut*.log
    profile_root/logs/server_name/SystemErr*.log
    profile_root/logs/ffdc folder
     

ORBRas Trace + CommTrace

  • Considered a Startup trace and requires a server restart.
  • This is the recommended method for turning on ORB trace in most cases.

Instructions

  1. Select the appropriate Administrative Console path of the JVM to be traced.
    1. Application Server: 
      Servers > Server Types > Application Servers > server_name > Container Services > ORB Service
    2. Node Agent:
      System Administration > Node Agent > nodeagent > ORB Service
    3. Deployment Manager:
      System Administration > Deployment Manager > ORB Service
       
  2. Select the ORB Tracing check box to enable both ORBRas and ORB CommTrace.
     
  3. If needed, per steps 1 and 2 in the previous “ORBRas Only (+ WAS Component Trace) section”:
    1. Change the log size and number of log files.
    2. Set any additional WebSphere component trace strings.
       
  4. Restart the affected JVM.
     
  5. Collect the following data:
     profile_root/logs/server_name/trace_<timestamp>.log
    profile_root/logs/server_name/trace.log
    profile_root/logs/server_name/SystemOut*.log
    profile_root/logs/server_name/SystemErr*.log
    profile_root/logs/ffdc folder

Standalone (Thin) Java Client

  • Trace is configured through a variety of “-D” java arguments.
  • Example:
     $JAVA_HOME/bin/java -classpath com.ibm.ws.ejb.thinclient_9.0.jar;<list_of_your_application_jars_and_classes> -Djava.naming.provider.url=iiop://<your_application_server_machine_name> -Dcom.ibm.CORBA.Debug=true –Dcom.ibm.CORBA.CommTrace=true <fully_qualified_class_name_to_run>

ORBRas Trace + CommTrace

  • -Dcom.ibm.CORBA.Debug=true
    • Equivalent to turning on ORBRas=all
  • -Dcom.ibm.CORBA.CommTrace=true
  • -Dcom.ibm.CORBA.Debug.Output=logName
    • Can be a file name, full-path or relative-path file name
    • If property is not set, defaults to orbtrc.DDMMYYYY.HHmm.SS.txt
    • Default location is the directory from which the java command was run
  • -Dcom.ibm.CORBA.Debug.Component=name
    • Optional.  Use only when full ORB trace is prohibitive
    • Only available in IBM JDK 8.0
    • Use in conjunction with -Dcom.ibm.CORBA.Debug=true to limit trace output to specific ORB subcomponent(s).
    • Name can be a single component or comma-separated list
    • Name options:  DISPATCH, MARSHAL, TRANSPORT, CLASSLOADER, ALL

ORBRas Trace + CommTrace + WAS Component Trace

Set the java -D arguments in the previous “ORBRas + CommTrace” section, in addition to the following:

  • -Dcom.ibm.ejs.ras.lite.traceSpecification=traceSpecString
    • TraceSpecString example:  WLM*=all:com.ibm.ws.security.*=all
  • -Dcom.ibm.ejs.ras.lite.traceFileName=logName
    • Must be the same value as logName set in com.ibm.CORBA.Debug.Output
  • -Dcom.ibm.ejs.ras.lite.maxFileSize=fileSize
    • Size in Mb
    • Only needed when the client log exceeds 100Mb
  • -Dcom.ibm.ejs.ras.lite.maxFiles=numFiles
    • Only needed when setting maxFileSize
  • -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
  • -Djava.util.logging.configureByServer=true

J2EE Client

  • Trace is configured through a variety of “-CC” launchClient.bat(.sh) arguments.
  • Example:
    install_root/bin/launchClient.sh <ear_file> -CCDcom.ibm.CORBA.CommTrace=true -CCtrace=ORBRas=all -CCtracefile=orbtrace.txt -CCtraceMode=basic

ORBRas Trace + CommTrace (+ WAS Component trace)

  • -CCtrace=ORBRas=all
    • Add other WAS component trace spec strings as desired separated by colons.  e.g. ORBRas=all:Naming=all
  • -CCDcom.ibm.CORBA.CommTrace=true
  • -CCtracefile=logName
    • This must be set otherwise output goes to the console
    • Can be a file name, full-path or relative-path file name
    • Default location is the directory from which the java command was run.
  • -CCtraceMode=basic
    • Ensure this is set to prevent Advanced trace mode which is not recommended.
  • -CCDcom.ibm.CORBA.Debug.Component=name
    • Optional.  Use only when full ORB trace is prohibitive
    • Only available in IBM JDK 8.0
    • Use in conjunction with –CCtrace=ORBRas=all to limit trace output to specific ORB subcomponent(s).
    • Name can be a single component or comma-separated list
    • Name options:  DISPATCH, MARSHAL, TRANSPORT, CLASSLOADER, ALL

Performance Considerations

  • ORBRas only
    • Any trace will incur a performance cost.  However, the majority of customers are able to run ORBRas trace with a minimal performance cost.  No performance benchmark is given due to the high variability in server environments.
  • CommTrace
    • With CommTrace enabled, ALL the data in both client request messages and server reply messages will be printed in hex format.  Again, most customers are able to run with CommTrace with a minimal performance cost.  Should it become an issue, the ORBRas-only option still remains.
How to check IBM® Java™ ORB build version in WebSphere® Application Server.

Exchanging data with IBM Support

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:

  • Service Request (SR)
  • E-mail
  • FTP to the Enhanced Customer Data Repository (ECuRep)

Instructions for exchanging information with IBM Support

Read first and related MustGathers

MustGather: Read first for WebSphere Application Server

Classloader problem Security Problem JNDI Problem JTS Problem Session Management Problem Workload Management Problem

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Object Request Broker (ORB)","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.0;7.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"8.0;7.1;7.0;6.1;6.0.2;6.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Object Request Broker (ORB)","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0;6.1","Edition":"Base;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
04 February 2020

UID

swg21157005

Page Feedback