Troubleshooting
Problem
Collecting data for Service Integration Bus Java™ Message Service (JMS) problems with IBM WebSphere Application Server. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time. []Note:[] This MustGather document applies only to messaging in WebSphere Application Server V7.0 and higher that involves SIB JMS Provider (aka default messaging provider). For problems involving the MQ JMS provider , refer to MustGather: MQ JMS.
Environment
Resolving The Problem
Tab navigation
- Learning more
- Troubleshooting
- Collecting data- selected tab,
Collecting data for Service Integration Bus Java™ Message Service (JMS) problems with IBM WebSphere Application Server. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
Note:This MustGather document applies only to messaging in WebSphere Application Server V7.0 and higher that involves SIB JMS Provider aka default messaging provider. For problems involving MQ JMS Provider, refer to MustGather:MQ JMS
MustGather: Read first for WebSphere Application Server
For a list of all technotes, downloads, and educational materials, search the WebSphere Application Server support site.
Troubleshooting guide for WebSphere Application Server
Steps to get support for WebSphere Application Server
Click on the problem that best describes your SIB JMS problem. This will provide you with a listing of the documentation the support team requires to diagnose your problem.
Note1:How to enable trace in WebSphere Application server
Note2:You must gather output of the collector tool run against the deployment manager profile
Note3:It is recommended to collect static traces for all the problems
| Adminconsole/wsadmin scripting If you are having difficulty configuring the Service Integration Bus using Administrative Console or the wsadmin scripting client please gather the following trace: SIBAdmin=all:Webui=all:com.ibm.websphere.management.*=all:com.ibm.ws.management.commands.sib.*=all Note:The above trace must be enabled on the deployment manager process. |
| BusLink If your application fails to send messages over the buslink or if the buslink status is unhealthy or if you have any other issues with the buslink please gather the following trace: SIBProcessor=all:SIBCommunications=all:SIBMessageTrace=all:SIBTrm=all:SIBJms*=all:SIBMfp=all Note:The above trace must be enabled on both sides(sender and receiver) of the BusLink. |
| Communication/Connection/MDB/JMS client problems(CWSIT0088E errors) If your client application(MDB/JMS) fails to connect to the messaging engine or if the requests from your client application are not routed to the desired messaging engine then please gather the following trace: On the server(jvm hosting the messaging engine): SIBCommunications=all:SIBTrm=all:WLM*=all:SIBJms*=all:SIBJFapChannel=all:SIBMessageTrace=all:SIBJmsRa=all:SIBRa=all:SIBProcessor=all On the client( or the jvm hosting the client application): SIBCommunications=all:SIBTrm=all:WLM*=all:SIBJms*=all:SIBJFapChannel=all:SIBMessageTrace=all:SIBJmsRa=all:SIBRa=all Append the following trace on the client and server if you suspect a problem in the network layer: TCPChannel=fine:com.ibm.io.async.*=all In order to trace clients running outside the application server, for example using the launchClient tool, be sure to follow the Client MustGather tracing instructions. For example, you may need to pass the trace file name and trace strings on the command line: launchClient -CCtracefile=C:\Temp\trace.txt -CCtrace=SIBJMS*=all:SIBCommunications=all Note1:Concurrent traces must be collected from the messaging engine and the client Note2:Gather output of netstat and ping commands |
| Mediation Handler Framework This component of the Service Integration Bus is responsible for running "mediation handlers" which can perform special processing or alter the flow of messages in a bus. These are not the same as "mediations" which are a feature in the WebSphere Enterprise Service Bus and WebSphere Process Server products. Gather the following trace if a mediation handler is not being invoked correctly. SIBMediations*=all:SIBMfp=all:SIBMfpApi=all:SIBMessageTrace=all |
| Messaging Engine Startup Gather the following trace if your messaging engine fails to start:
Also describe where the messaging engine stores its database, including the version, release and maintenance level of any databases and JDBC drivers.
Also describe where your filestore files are located(on the local disk, shared file system,details including version of the shared file system) |
| Messaging Engine failover/starting up on multiple jvms Gather the following trace if your messaging engine does not failover or it attempts to start on multiple jvms: SIBAdmin=all:SIBProcessor=all:SIBMessageStore=all:HAManager=finest:DCS=finest:RMM=finest:TCPChannel=finest:WLM*=all Note1:The above trace must be enabled on all the jvms in the messaging cluster Note2:FFDCs and SystemOut.logs must be collected from the deployment manager and all the jvms in the messaging cluster |
| MQLink If your MQLink does not start or if you are unable to send/receive messages over the MQ Link or if you have any other issues with the MQ Link please collect the following trace: SIBProcessor=all:SIBCommunications=all:SIBMfp=all:SIBMqFapChannel=all:SIBTrm=all:SIBPsb=all Also gather a WebSphere MQ trace of the queue manager to which the link is configured. Note:Concurrent traces must be collected from SIB messaging engine and MQ Queue Manager |
| Performance If messages are not passing through the bus as quickly as you expect i.e. if there is a delay in messages reaching the destination or messages being delivered to the consumer please gather the following trace: On the jvm hosting the messaging engine: SIBMessageTrace=all:SIBCommunications=all:SIBProcessor=all:SIBJmsRa=all:SIBRa=all:SIBJms*=all on the jvm hosting the message consumer application: SIBMessageTrace=all:SIBCommunications=all:SIBJmsRa=all:SIBRa=all:SIBJms*=all Note:Concurrent traces must be collected from the messaging engine jvm and the message consumer application jvm. Also provide some basic information : 1.What is the size of the application messages? 2.If you are using transactions , what kind, eg xa transactions? 3.What is the name of the problem Queue/Topic? 4.What is the name of the connection factory/activation spec used by your application? 5.How many messages are on the problem Queue/Topic? 6.What is the message reliability level ex:Best Effort Nonpersistent,Express nonpersistent? |
| Publish/Subscribe Messaging If your subscriber applications are unable to consume messages or if you receive any errors while publishing or consuming messages please gather the following trace: On the jvm hosting the messaging engine: SIBMatchSpace=all:SIBProcessor=all:SIBCommunications=all:SIBTrm=all:SIBMessageTrace=all On the jvm hosting the producer/consumer application: SIBMatchSpace=all:SIBCommunications=all:SIBTrm=all:SIBMessageTrace=all Note:Concurrent traces must be collected from the messaging engine jvm and producer/consumer application jvm. |
| Stuck/locked messages Messages may be locked when the Service Integration Bus is routing messages between engines or when they are being delivered to MDBs or JMS applications. If your messages remain in a locked status much longer than you expect, follow these instructions. The message state is not dynamically updated in the Administrative Console, so be sure to refresh your view of a destination to see whether the message gets unlocked or delivered.
Gather the following trace: On the jvm hosting the messaging engine(s): SIBJms*=all:SIBCommunications=all:SIBJFapChannel=all:SIBMessageTrace=all:SIBTrm=all:SIBJmsRa=all:SIBRa=all:SIBProcessor=all On the jvm hosting the message consumer application: SIBJms*=all:SIBCommunications=all:SIBJFapChannel=all:SIBMessageTrace=all:SIBTrm=all:SIBJmsRa=all:SIBRa=all Javacores: collect 4-5 javacores 1 min apart The javacores must be collected from the jvm hosting the messaging engine and also from the jvm hosting the message consumer application Messagestore dumps: Collect 2 messagestore dumps, one before reproducing the problem and one afterwards Output of printSIBUSSummary script Note1:It is recommended to refer to Understanding Stuck Messages before collecting the mustgather Note2:Concurrent traces must be collected from the jvm hosting the messaging engine and the jvm hosting the message consumer application Note3:If multiple messaging engines are configured in your environment, gather the traces from all the jvms hosting the messaging engines Also provide some basic information : 1.What is the size of the application messages? 2.If you are using transactions , what kind, eg xa transactions? 3.What is the name of the problem Queue/Topic? 4.What is the name of the connection factory/activation spec used by your application? 5.How many messages are on the problem Queue/Topic? 6.What is the message reliability level ex:Best Effort Nonpersistent,Express nonpersistent? |
| Security If your JMS application or MDB fails to connect to the messaging engine due to SINotAuthorizedException or any other SIB security errors gather the following trace: SIBProcessor=all:SIBSecurity=all:WAS.j2c=all Note:The above trace must be enabled on the JVM hosting the messaging engine |
| SIB Web Services If you are having difficulty configuring the SIB Web Services or web service messages are not being handled properly gather the following trace: com.ibm.ws.sib.webservices.*=all:com.ibm.wsspi.wssecurity.*=all:com.ibm.ws.webservices.wssecurity.*=all:com.ibm.xml.soapsec.*=all:com.ibm.ws.webservices.*=all:com.ibm.wsgw.*=all |
| Transactions/XARecovery If you are having problems with messages not committing or rolling back as expected, whether part of a local or a global XA transaction or if you see CWSIT0008E/CWSIT0019E/CWSIT0103E errors accompanied by WTRN0005W gather the following trace: SIBProcessor=all:SIBCommunications=all:SIBJms*=all:SIBMessageTrace=all:SIBJmsRa=all:SIBRa=all:Transaction=all:EJBContainer=all:WAS.j2c=all:RRA=all |
| WS-Addressing If you suspect a problem with WS-Addressing support in the Service Integration Bus gather the following trace: com.ibm.ws.wsaddressing.*=all:com.ibm.ws.transaction.*=all:com.ibm.ws.webservices.*=all |
| WS-Notification If you suspect a problem with WS-Notification support in the Service Integration Bus gather the following trace: *=info:com.ibm.ws.sib.wsn.*=all:com.ibm.ws.sib.wsnotification.*=all:SIBMessageStore=all:SIBProcessor=all:SIBWsn=all |
In addition to collecting the traces that are specific to your problem, please gather the following:
- Run the collector tool against the Deployment Manager Profile to produce a JAR file containing your WebSphere Application Server configuration files and other logs that are useful to the WebSphere Application Server support team.
- Gather all the JVM logs from the server, namely SystemOut.log, SystemErr.log, and the native stdout and stderr log files.
- Gather all the First Failure Data Capture (FFDC) files. The FFDC files are in the ffdc subdirectory at the same location as the JVM logs.
- Follow instructions to send diagnostic information to IBM support to send the files mentioned in the preceding steps.
For a listing of all technotes, downloads, and educational materials specific to JMS, search the WebSphere Application Server support site
Related information
Submitting information to IBM support
Steps to getting support for WebSphere Application Server
MustGather: Read first for WebSphere Application Server
Troubleshooting guide for WebSphere Application Server
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Service Integration Technology","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF025","label":"Platform Independent"},{"code":"PF012","label":"IBM i"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Network Deployment","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSQH9M","label":"WebSphere Process Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"SIB (service integration bus)","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"6.2;6.1;6.0.2;6.0.1;6.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7J6S","label":"WebSphere Enterprise Service Bus"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"SIBus","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"","label":"i5\/OS"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSFTN5","label":"IBM Business Process Manager Advanced"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSQP76","label":"IBM Operational Decision Manager"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSDKKW","label":"WebSphere Partner Gateway Advanced Edition"},"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SSWLGF","label":"WebSphere Service Registry and Repository"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Product Synonym
WMQ MQ