Troubleshooting
Problem
Resolving The Problem
Tab navigation
- Learning more
- Troubleshooting
- Collecting data- selected tab,
Collecting data for problems with the IBM WebSphere Application Server Object Request Broker component. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.
Collecting data manually
Outline
Basic Diagnostic Questions
- What is the WebSphere software version of both the client and server? Is there a non-WebSphere client/server involved?
- What exception(s) are being seen in the logs?
- Is the problem reproducible at will or intermittent?
- How often and under what circumstances does this problem occur?
- Has this problem always occurred or has it just started to occur? What, if any, recent changes have been made to the environment (WebSphere, JDK, application, OS, increased client load, etc)?
Third Party ORB interoperability Questions
- Provide Third-Party Server/ORB software name (e.g. WebLogic), version, language, and platform information.
- Is WebSphere Application Server being used as the server or client to the third-party ORB?
- Was the approach recommended in the CORBA Interop samples followed?
- Where was the Interface Definition Language (IDL) compiled (what tool was used)?
Data To Collect
Collector Tool
The collector tool output is required to troubleshoot an ORB issue. The resulting collector.jar provides valuable configuration data that provides a better understanding of your topology and settings.
Detailed instructions on how to run the collector program can be found in the Knowledge Center:
Gathering information with the collector tool
NOTE:
- Run the tool from the Dmgr.
- Run the collector tool from a directory outside the WebSphere Application Server product installation root directory otherwise an error will occur.
- Upload the collector.jar file that is created.
- In most cases, a matching client and server-side trace is needed.
- There are 2 basic kinds of ORB trace:
- ORBRas
- Basic ORB method tracing
- ORB CommTrace
- Payload data tracing for each incoming and outgoing ORB message
- Not always required for every type of problem, but recommended in most cases
- ORBRas
- Can be configured 2 ways
- Startup, requires a server restart
- Runtime, no server restart required
WebSphere Application Server V8.5 and V9.0
ORBRas Trace Only (+ WAS Component Trace)
ORBRas trace can be configured 2 ways:
- Startup (Static)
- Settings are made in the Configuration tab for log setting menu panels
- Requires a server restart
- Remains ON through server restart(s)
- Runtime (Dynamic)
- Settings are made in the Runtime tab for log setting menu panels
- No server restart required
- Remains ON only until the server is restarted
NOTE: The instructions below assume the use of Basic Trace Mode rather than High Performance Extensible Logging (HPEL) trace mode. From the Admin Console path Troubleshooting > Logs and Trace > server_name click the Switch back to Basic Mode button to run in Basic Mode.
Instructions
- Change the trace spec string.
- Administrative Console path:
Servers > Server Types > Application Servers > server_name > Change log detail levels - Select the Configuration or Runtime tab.
- Remove any previous entries in the text field and enter:
*=info:ORBRas=all - Add any other WebSphere component trace string(s), if desired:
*=info:ORBRas=all:WLM*=all:Naming=all - Select Apply or OK to save changes.
- Administrative Console path:
- Change the log size and number of log files.
- Administrative Console path:
Servers > Server Types > Application Servers > server_name > Diagnostic Trace Service - Select the Configuration or Runtime tab.
- Change the following fields:
- Maximum Number of Historical Files to 10
- Maximum File Size to 100MB
- Ensure Trace Output Format is set to Basic
- Select Apply or OK to save changes.
- Administrative Console path:
- Restart the sever if using Startup trace and recreate the problem. No restart is required if using Runtime trace.
- Collect the following data:
profile_root/logs/server_name/trace_<timestamp>.log
profile_root/logs/server_name/trace.log
profile_root/logs/server_name/SystemOut*.log
profile_root/logs/server_name/SystemErr*.log
profile_root/logs/ffdc folder
ORBRas Trace + CommTrace
- Considered a Startup trace and requires a server restart.
- This is the recommended method for turning on ORB trace in most cases.
Instructions
- Select the appropriate Administrative Console path of the JVM to be traced.
- Application Server:
Servers > Server Types > Application Servers > server_name > Container Services > ORB Service - Node Agent:
System Administration > Node Agent > nodeagent > ORB Service - Deployment Manager:
System Administration > Deployment Manager > ORB Service
- Application Server:
- Select the ORB Tracing check box to enable both ORBRas and ORB CommTrace.
- If needed, per steps 1 and 2 in the previous “ORBRas Only (+ WAS Component Trace) section”:
- Change the log size and number of log files.
- Set any additional WebSphere component trace strings.
- Restart the affected JVM.
- Collect the following data:
profile_root/logs/server_name/trace_<timestamp>.log
profile_root/logs/server_name/trace.log
profile_root/logs/server_name/SystemOut*.log
profile_root/logs/server_name/SystemErr*.log
profile_root/logs/ffdc folder
Standalone (Thin) Java Client
- Trace is configured through a variety of “-D” java arguments.
- Example:
$JAVA_HOME/bin/java -classpath com.ibm.ws.ejb.thinclient_9.0.jar;<list_of_your_application_jars_and_classes> -Djava.naming.provider.url=iiop://<your_application_server_machine_name> -Dcom.ibm.CORBA.Debug=true –Dcom.ibm.CORBA.CommTrace=true <fully_qualified_class_name_to_run>
ORBRas Trace + CommTrace
- -Dcom.ibm.CORBA.Debug=true
- Equivalent to turning on ORBRas=all
- -Dcom.ibm.CORBA.CommTrace=true
- -Dcom.ibm.CORBA.Debug.Output=logName
- Can be a file name, full-path or relative-path file name
- If property is not set, defaults to orbtrc.DDMMYYYY.HHmm.SS.txt
- Default location is the directory from which the java command was run
- -Dcom.ibm.CORBA.Debug.Component=name
- Optional. Use only when full ORB trace is prohibitive
- Only available in IBM JDK 8.0
- Use in conjunction with -Dcom.ibm.CORBA.Debug=true to limit trace output to specific ORB subcomponent(s).
- Name can be a single component or comma-separated list
- Name options: DISPATCH, MARSHAL, TRANSPORT, CLASSLOADER, ALL
Set the java -D arguments in the previous “ORBRas + CommTrace” section, in addition to the following:
- -Dcom.ibm.ejs.ras.lite.traceSpecification=traceSpecString
- TraceSpecString example: WLM*=all:com.ibm.ws.security.*=all
- -Dcom.ibm.ejs.ras.lite.traceFileName=logName
- Must be the same value as logName set in com.ibm.CORBA.Debug.Output
- -Dcom.ibm.ejs.ras.lite.maxFileSize=fileSize
- Size in Mb
- Only needed when the client log exceeds 100Mb
- -Dcom.ibm.ejs.ras.lite.maxFiles=numFiles
- Only needed when setting maxFileSize
- -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
- -Djava.util.logging.configureByServer=true
J2EE Client
- Trace is configured through a variety of “-CC” launchClient.bat(.sh) arguments.
- Example:
install_root/bin/launchClient.sh <ear_file> -CCDcom.ibm.CORBA.CommTrace=true -CCtrace=ORBRas=all -CCtracefile=orbtrace.txt -CCtraceMode=basic
ORBRas Trace + CommTrace (+ WAS Component trace)
- -CCtrace=ORBRas=all
- Add other WAS component trace spec strings as desired separated by colons. e.g. ORBRas=all:Naming=all
- -CCDcom.ibm.CORBA.CommTrace=true
- -CCtracefile=logName
- This must be set otherwise output goes to the console
- Can be a file name, full-path or relative-path file name
- Default location is the directory from which the java command was run.
- -CCtraceMode=basic
- Ensure this is set to prevent Advanced trace mode which is not recommended.
- -CCDcom.ibm.CORBA.Debug.Component=name
- Optional. Use only when full ORB trace is prohibitive
- Only available in IBM JDK 8.0
- Use in conjunction with –CCtrace=ORBRas=all to limit trace output to specific ORB subcomponent(s).
- Name can be a single component or comma-separated list
- Name options: DISPATCH, MARSHAL, TRANSPORT, CLASSLOADER, ALL
Performance Considerations
- ORBRas only
- Any trace will incur a performance cost. However, the majority of customers are able to run ORBRas trace with a minimal performance cost. No performance benchmark is given due to the high variability in server environments.
- CommTrace
- With CommTrace enabled, ALL the data in both client request messages and server reply messages will be printed in hex format. Again, most customers are able to run with CommTrace with a minimal performance cost. Should it become an issue, the ORBRas-only option still remains.
- <washome>/java/bin/java -Xbootclasspath/p:<washome>/java/jre/lib/ext/ibmorb.jar com.ibm.rmi.util.Version
- Follow instructions to send diagnostic information to IBM support
For a listing of all technotes, downloads, and educational materials specific to the ORB component, search the WebSphere Application Server support site.
Related information
Submitting information to IBM support
Steps to getting support for WebSphere Application Server
MustGather: Read first for WebSphere Application Server
Troubleshooting guide for WebSphere Application Server
Exchanging data with IBM Support
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Read first and related MustGathers
Related Information
Was this topic helpful?
Document Information
Modified date:
04 February 2020
UID
swg21157005