Show overview layout Help

Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser.

Dynamic troubleshooting guide for WebSphere Message Broker

Use this dynamic guide to help you determine the cause and resolution of problems that you might have experienced with WebSphere Message Broker.

Are you unable to run any broker runtime commands?

You are unable to run any broker runtime commands at all. This kind of problem is usually caused by an issue with the installation or the environment that the commands are being run from.

You see an error message indicating that the command is an "invalid executable" or you see an error message saying "cannot execute binary file"

You are using the wrong Message Broker binary files for your operating system.

Resolution

Install the correct version of Message Broker for your operating system. Remember that binary files available for Linux are not cross compatible between different hardware architectures. For example the installation for Linux on PowerPPC is not compatible with a zLinux host.

If you are using an operating system such as Linux where there are both 32-bit and 64-bit versions of the product available, ensure that you have installed the correct version to match your operating system.

On UNIX platforms you can use the "file" command to check if a binary file is a 32-bit or 64-bit file.

On UNIX you observe the message "error while loading shared libraries: libImbCmdLib.so"

The environment is not configured correctly leading to essential library files not being found when you invoke the command.

You need to run mqsiprofile

On UNIX and Windows platforms the WebSphere Message Broker environment is configured by a script called "mqsiprofile.sh" (UNIX) or "mqsiprofile.cmd" (Windows). This script configures environment variables that are required for WebSphere Message Broker to run successfully, including setting up the Library Path.

Resolution

Run the mqsiprofile script before running a Broker command. On Windows operating systems ensure that you run commands from a "Broker Command Window".

You have run mqsiprofile, but see issues loading shared libraries

If you have run mqsiprofile and still observe a problem loading shared libraries, particularly libImbCmdLib.so, then the problem is likely to be an error in your environment.

Resolution

Correct any issues with the PATH, LIBPATH, or LD_LIBRARY_PATH environment variables.

You get en error saying "bad interpreter"

You do not have the Korn shell installed. On UNIX platforms the Korn shell is required by some commands.

Problems starting a broker runtime

Starting a broker runtime involves a number of stages, depending on how you started the broker, that must all complete successfully.

Starting a broker runtime involves a number of stages: establishing an environment where any command can run, running the start command, starting the broker processes, and initialising those processes to the point where they can process data. The following questions try to identify which stage has problems.

Broker startup problems with the WebSphere Message Broker toolkit

The Message Broker toolkit (Brokers view) provides the ability to start and stop local brokers, but with some considerations.

  • The Message Broker toolkit (Brokers view) can only be used to start and stop local brokers that have the same version as the toolkit.
  • The brokers are started by a toolkit action, but this causes the mqsistart command to be issued.
  • If the start command returns successfully, the broker continues to start in the background, and reports further status to the system log (and not through the toolkit).
  • The toolkit cannot be used to start remote brokers.
I cannot see the broker that I want to start

Only brokers that match the version of the WebSphere Message Broker Toolkit can be managed in the Toolkit.

Resolution

Check that the version of your Toolkit matches the version used to create the broker. To check the toolkit version use the Help->About dialog.To check the version of the runtime issue the mqsiservice -v command

If you're broker level is earlier than the toolkit you can migrate your broker to a later version.

I get a toolkit error when I issue a start request

Review the text in the pop-up window.

Resolution

If you are not able to resolve the problem, contact IBM Support, and supply the following additional diagnostic information:

The start request has been accepted

In this case, we assume that your toolkit start action was accepted, but the broker has not processed any messages.

When you get to this point, all methods of starting the broker go down the same path. For more information about investigating this issue further, check the information about "mqsistart" under Broker startup by command on the broker machine .

Broker startup by WebSphere Message Broker Explorer

The Message Broker Explorer provides the ability to start and stop local brokers. The brokers are started by an action in the Message Broker Explorer, but in the background an mqsistart command is issued.

You get an error message when you issue a start request in the WebSphere Message Broker Explorer

Review the text in the pop-up window.

Resolution

If you are not able to resolve the problem, contact IBM support, and supply the following additional diagnostic information:

The start request has been accepted in the WebSphere Message Broker Explorer

In this case, we assume that your Explorer start action was accepted, but the broker has not processed any messages.

When you get to this point, all methods of starting the broker go down the same path. For more information about investigating this issue further, check the information about "mqsistart" under Broker startup by command on the broker machine .

Broker startup by command on the broker machine

The mqsistart command could fail immediately, if your environment or permissions are not correct, but some of the broker startup happens in the background once the command has returned.  Your broker is only up and ready for processing once all its components are running, and in the toolkit once its execution groups show as running.  On z/OS the start request is issued as a console command, not a UNIX command.

On z/OS, the MVS START console command returns an error.

The START console command causes the started-task JCL from your PROCLIB to be run under the started task definition. If this JCL cannot be found or you have a permission error, the actual console command fails. If the console command works, the broker's startup JCL runs as a job, and you can check the output of that job using SDSF. If you have a customisation error, it would be shown in the job output.

Resolution

Check the MVS SYSLOG to see if any errors were associated with the console command. If the SYSLOG reports success, find the job output for the started task to see more information about what problems occurred during startup.

Errors running mqsistart

To be able to run the mqsistart command, you need the correct environment and permissions, but much of this setup is shared with all mqsi commands.

If you can run other commands but not mqsistart, there may be a problem with your broker definition which mean that it just cannot be started at all.

Are you able to run any other mqsi commands from your command prompt (such as mqsilist)?
You cannot run any mqsi command

The mqsistart command shares environmental and user requirements with all broker administration commands.

The command prompt or shell from which you run the command is the exact environment used.

If you cannot run any command, then you have a basic environment problem; check the information in Are you unable to run any broker commands? .

You can run other mqsi commands, but you cannot run mqsistart on your broker.

On both Windows and UNIX, the mqsistart command uses the data stored in the broker data files to get key information before the broker processes can be started. If the mqsistart command gives an error, something has gone wrong at this stage before the broker processes are even launched.

You get a BIP8846S error starting a broker.

A broker is associated with a particular version of the WebSphere Message Broker product, so it is not possible to use one version of the product to start a broker created at a different version; for example, you cannot use the V7 product to start a V6.1 broker.

If you try to start a broker using a profile configured for the wrong WebSphere Message Broker version, then you see the following error:

BIP8846S: Configuration data for component 'brokerName' is not usable by this product version (level is 'level' should be 'level').
Resolution

You have run mqsiprofile that belongs to a different version of the product. Run the mqsiprofile script that matches the version that the broker was created with, and then try the command again. Alternatively, if you are trying to move this broker to a newer version of the product, then migrate the broker using the mqsimigratecomponents command.

You get BIP8223 when trying to execute 'mqsistart'.

You see the following error when running mqsistart: BIP8223: User userName is not a member of group groupName. The user must be a member of the mqbrkrs group in order to have authority to start the Broker.

Resolution

Add the userid being used to start the broker to the mqbrkrs group and retry the command.

The mqsistart command returns a BIP8018E error

The mqsistart command does not work if the broker is already started. Instead, if you try to issue the command on a running broker you see a BIP8018E error.

BIP8018E: Broker 'brokerName' is running
Resolution

The broker is already running. Use your operating system tools (for example, ps -es on UNIX, Task Manager on Windows, or SPSF on z/OS) to verify that the bipservice, biphttplistener, bipbroker, and DataFlowEngine processes are running.

If these processes are not running, then see the information provided under Your start broker command returned OK, but the broker does not appear to be running .

The mqsistart command returns a BIP8026E error

You see the following BIP message after running mqsistart:

Resolution
  1. Ensure the broker service ID is a member of mqm and mqbrkrs groups.
  2. Log on with the broker service ID on the machine.
  3. Run the following command: mqsichangebroker <broker> -a <broker service Id's password>
  4. Start the broker
You get another error when running mqsistart

Review the text in the error, and see whether it explains what is preventing the broker from being launched at all.

Resolution

If you are not able to resolve the problem from the error description, then check the logs for errors related to startup.

  • On Windows systems, check the event log
  • On UNIX systems, check the syslog
  • On z/OS systems, check the JobLog

If this still does not resolve your problem, contact IBM Support with a description of the error.

On Windows, you have a problem starting the broker's service directly.

On Windows, every broker has its own associated Windows service. This can be started manually or configured to start with the machine.

Resolution

There is a stored user and password with the Windows service, and if you change the service user's password then you must update the service definition with the new value before the broker can be started. This error shows up only in the Windows system log.

Also check that the product is installed in the location defined in the Windows service; for example, if you have uninstalled the product and then reinstalled it in a new location. If you run the mqsistart command from a command prompt, the service definition is updated with the correct path from your current profile.

Your start broker command completed successfully, but the broker does not appear to be running.

After the broker has been told to start, a set of processes is started in the background to run the broker. If these processes encounter a startup problem they normally write errors to the logs.

When the broker has been started, the overall parent process restarts any child processes that stop abnormally, so you might encounter processes that appear to come and go, or that change their process ID repeatedly.

  • On Windows and UNIX, these processes are called bipservice, bipbroker, DataFlowEngine and biphttplistener, and the broker's name is in the arguments to those commands.
  • On z/OS, you get one address space for a broker (which runs with a STEPNAME of BROKER) plus for each execution group, one other address space whose STEPNAME is based on the execution group name. All z/OS jobs have a user ID of the broker's started task ID.

You always get a bipservice and bipbroker process, or a started task with STEPNAME BROKER on z/OS, but you only get additional processes after you deploy execution groups.

You can check which processes are present by using the Task Manager on Windows, the "ps" command on UNIX, or by examining the held output on z/OS.

You cannot see any broker processes after mqsistart

There is an initial "Configuration Verification Procedure (CVP)" step that is the first part of a broker startup. This process checks that the environment and configuration of the system are sufficiently correct for the broker to start.

A BIP8873I message is written to say this process is starting. If any serious problems are detected then a set of error messages are written then the broker stops, and not retried.

You get a BIP8875W to say that configuration verification has failed.

Resolution

Check the logs for error messages. These errors are written to syslog, Event Viewer or the joblog for the broker's started task. To verify the environment before starting, you can also run the CVP process ( mqsicvp <brokerName> command) independently of broker startup.

Does the CVP process indicate that there are errors with the environment or Broker configuration?
You see a BIP8875 message or the CVP process produces one or more errors.

The CVP process checks four main types of configuration, database configuration, the execution environment, WebSphere MQ configuration, and Java configuration. The BIP message number can be used to determine which type of configuration you have a problem with.

Examining the error message, which resource is the CVP process reporting a problem with?
You have a configuration problem with Java (BIP8892E)

The level of Java must be at a supported level. (The CVP process checks that the level of Java is supported.)

Resolution

Ensure that the Java being used is at supported level.

  • On UNIX and Windows systems, ensure that the JRE contained in the $MQSI_FILEPATH/jre16 directory is the one that was shipped with the product. To determine which level of Java is in the product's JRE directory, run the mqsiprofile script and then using the which java command.
  • On z/OS, ensure that the JAVA_HOME variable in the broker's env file is set to a supported version of the JRE.

If you are certain that the level of Java is supported, and that the CVP process has misidentified the version of Java you are running, then you can disable the CVP process by setting the following environment variable:

MQSI_DISABLE_CVP=1

Note that although disabling the CVP process should allow you to work around a misidentified JRE, this also disables all other CVP checking. It is therefore recommended that if you need to use this environment variable, that you raise a PMR with IBM Support to verify that you are running in a supported configuration, and to resolve any issues identifying your Java version.

If you need to raise a PMR with IBM Support, please include the output of the mqsicvp command, the broker's environment, and the output of running the command java -version in the broker's environment.

CVP reports an error related to database connectivity.

The CVP process, when run as part of broker startup, checks some basic ODBC configuration parameters to ensure that WebSphere Message Broker has the capability to connect to remote databases.

Are you running CVP as part of WebSphere Message Broker startup or are you running the mqsicvp tool as a standalone command?
You am running CVP as part of WebSphere Message Broker startup and have an error related to database configuration

During broker startup, the CVP process verifies the environment is configured correctly for ODBC, but connection is not made to individual databases.

You see a BIP8288 Error from CVP

You observe the following error:

Resolution

Check the file indicated in the BIP message.

  • If this file is the wrong file, then update the value of ODBCINI in the broker's profile to point to the correct file.
  • If the file is correct but the Broker can not read the file then use chmod to ensure that the file is readable and writable by the Broker's ID.
You see a BIP8289E Error from CVP

You see the following error from CVP:

Resolution

Examine the directory name given in the error message

  • If the directory name is incorrect then update the broker's profile to set ODBCSYSINI so that it points to the directory containing the odbcsys.ini file.
  • If the ODBCSYSINI variable points to the correct directory and that directory contains an odbcsys.ini file then use chmod to ensure that the Broker user has read and write permissions on this file.
You see a BIP8291E Error from CVP

You see the following error from the CVP process:

Resolution

Ensure that the IE02_PATH variable is set correctly

You see a BIP8296 Error from CVP

You see the following error from the CVP process:

Resolution

Update the Broker's profile envornment so that the ODBCSYSINI variable points to the directory containing your odbcsys.ini file.

You see a BIP8297 Error from CVP

You see the following error from the CVP process:

Resolution

Examine the error message to determine which file is affected, and which of the two ODBC environment variables this is used by. Ensure that the variable is pointing to the correct location.

If the variable is pointing in the correct location then ensure that the files are not zero length. If you do not have a sample odb.ini or odbcsysini file then use the examples in the $MQSI_FILEPATH\ODBC\UNIXodbc directory as a starting point.

You see a BIP8293 Error from CVP

You see the following error from the CVP process:

Resolution

Check that the ODBCUOINI envenomed variable point to the correct file and that the file exists and the broker has read/write permissions on the file.

You see another error from CVP

If you see a different error from CVP it may be safe to ignore if you are not using ODBC to connect to user databases.

Are you using ODBC to connect to User Databases? Note that ODBC is the default technology used to connect to databases from ESQL.
Yes You are using ODBC functionality

An unexpected error has occurred during CVP checking. Check to see if any other paths through the flowchart describe your symptoms.

Resolution

Review the error text to determine if this gives you the information required to resolve the problem. If you are still unable to resolve the problem, contact IBM Support, and supply the following additional diagnostic information:

No You are not using ODBC to connect to databases

If you are not using ODBC functionality then it is safe to ignore warning messages relating to ODBC configuration during broker startup.

Resolution

If you would prefer that these checks are not performed in your environment the you can set the following environment variable in the Broker's profile:

MQSI_ODBC_SUPRESS_STARTUP_CHECKING=1
You are running CVP using the standalone command mqsicvp and have an error or warning related to database connectivity

When running mqsicvp in standalone mode the CVP process performs the same environment checking as when CVP is run on startup, but additionally tries to connect to the database.

Does the error message that you see relate to a specific datasource or the ODBC environment as a whole?
A specific data source

The mqsicvp command tries to connect to either the database specified on the command line or, if no database is specified, tries to connect to any database that has both an entry in the odbc.ini file and a corresponding identity set using mqsisetdbparms.

Resolution

Ensure that the correct username and password has been configured for this data source using mqsisetdbparms.

Review the output of the mqsicvp command; if you have an SQLSTATE and Native error code consult the documentation for your database provider for further help resolving the problem.

The general ODBC environment

Checking the general ODBC environment is common between the CVP process used when starting a broker and running the mqsicvp command.

You have a configuration problem with WebSphere MQ (BIP8883W)

The Broker uses several internal message queues on it's host WebSphere MQ queue manager for internal communication messages between different components. If any of these queues do not exist, or if the Broker processes do not have permission to access them, then the Broker does not function correctly. Therefore the CVP process flags an error.

Resolution

Check for messages that identify the MQ object that is causing the problem. Check in the Broker syslog on UNIX, the Event Log on Windows, or the JobLog on z/OS, for one of the following BIP messages:

You see a BIP8885E message in the Broker logs

The BIP8885E message indicates that there has been a problem connecting to the Broker's parent WebSphere MQ queue manager. The message contains two useful inserts that can help you resolve the problem:

Resolution

Ensure that the queue_manager_name insert is correct and that the queue manager can be started successfully. To determine the reason why the Broker cannot connect to the queue manager, use the "mqrc" command supplied with WebSphere MQ. If required, examine the MQ error logs that are are located in $MQ_DATA_PATH/errors and $MQ_DATA_PATH/Qmgrs/queue_manager_name/errors directories and have names like AMQERRxx.Log where xx is a number. Use the WebSphere MQ logs to resolve any MQ errors.

You see a BIP8887E message in the Broker logs

The BIP8887E message indicates that the CVP process could not access an expected queue on the broker's queue manager. The broker uses internal queues to communicate between different components in the overall WebSphere Message Broker product. If these queues do not exist, or if the Message Broker user does not have sufficient permission to access the queues, then Message Broker does not function as expected and therefore CVP reports an error.

The BIP8887E message also reports the "MQRC" reason code that indicates the reason that the underlying WebSphere MQ installation has given as the cause of the failure when access to the queue was attempted. The BIP887E message looks like this:

BIP8887E: Verification failed for queue 'queue_name' on queue manager 'queue_manager_name' while issuing 'operation'. MQRC:<return_code>MQCC:<completion_code>

The CVP process checks for the existence, and access permissions on the following MQ Queues:

  • SYSTEM.BROKER.ADMIN.QUEUE
  • SYSTEM.BROKER.EXECUTIONGROUP.QUEUE
  • SYSTEM.BROKER.EXECUTIONGROUP.REPLY
  • SYSTEM.BROKER.AGGR.CONTROL
  • SYSTEM.BROKER.AGGR.REPLY
  • SYSTEM.BROKER.AGGR.REQUEST
  • SYSTEM.BROKER.AGGR.TIMEOUT
  • SYSTEM.BROKER.AGGR.UNKNOWN
  • SYSTEM.BROKER.EDA.COLLECTIONS
  • SYSTEM.BROKER.EDA.EVENTS
  • SYSTEM.BROKER.TIMEOUT.QUEUE
  • SYSTEM.BROKER.WS.ACK
  • SYSTEM.BROKER.WS.INPUT
  • SYSTEM.BROKER.WS.REPLY
Resolution

Examine the MQRC code and the queue name given in the BIP8887E message. To identify the symbolic name for the failure reason, use the "mqrc" command supplied with WebSphere MQ.

The following are common failures:

  • MQRC 2085 (MQRC_UNKNOWN_OBJECT_NAME)

    This indicates that the queue was not found on the underlying queue manager. Ensure that the queue exists; if necessary create the queue on the queue manager.

  • MQRC 2101 (MQRC_OBJECT_DAMAGED)

    The state of the WebSphere MQ queue manager has become damaged. For recovery options, see the following WebSphere MQ Infocenter topic Recovering damaged objects .

You see a BIP8888E message in the Broker logs

The BIP8888E message indicates that the CVP process was unable to successfully disconnect from the queue manager during the CVP checking. Being unable to successfully disconnect from the queue manager could cause the Broker to leak native resources during operation, and therefore the CVP process flags this as an error.

Resolution

Ensure that the queue_manager_name insert is correct and that the queue manager can be started successfully. To determine the reason that the Broker can not connect to the queue manager, use the "mqrc" command supplied with WebSphere MQ. If required examine the WebSphere MQ error logs, which are located in the directories $MQ_DATA_PATH/errors and $MQ_DATA_PATH/Qmgrs/queue_manager_name/errors , and have named like AMQERRxx.Log where xx is a number. Use the WebSphere MQ logs to resolve any MQ errors.

The mqsistart command appears to run correctly, but I cannot see any Broker processes at all

If the mqsistart command has completed successfully, but you do not see any broker processes at all, then it is likely that an error has caused the bipbroker process to terminate unexpectedly.

For more information about the problem, examine the Event Log on Windows, the syslog on UNIX, or the joblog on z/OS. Check the $MQSI_WORKPATH/common/errors directory for core files, abend files or javacore files created at the time of failure.

Resolution

For help to resolve your problem, contact IBM Support, and supply the following additional diagnostic information:

  • The output from running the mqsidc command on the runtime machine
  • The output from the mqsistart command
  • Any abend files, core files or javacore files in $MQSI_WORKPATH/common/errors
You can see a bipservice process, but other processes do not start, or come and go

A broker comprises three main processes, each of which performs a different function:

There is a problem with the administrative agent starting (bipbroker)

When the broker starts, the first process started is the bipservice process, which monitors the broker admin agent and restarts it if the process fails. The admin agent is represented by the bipbroker process, and the agent is responsible for passing configuration messages between administration tooling and the execution groups.

During broker startup the admin agent must be started first, because it is responsible for creating the execution group processes and then sending to each execution group a request to start.

If the bipservice or bipbroker processes recycle or do not appear, it is most likely due to the admin agent encountering a problem.

For any error messages that indicate that the bipbroker process has abended, check the Windows Event Viewer, UNIX Syslog, or z/OS Joblog. Examine the contents of the $MQSI_WORKPATH/common/errors directory and look for abend files coming from the bipproker process.

Do you see any abend files relating to the Admin Agent or the bipbroker process?
The broker agent (bipbroker) process abended on broker startup and the abend file shows that it occurred in the function semctl().

The Broker uses a type of shared memory called semaphores to synchronize the startup of the broker and execution group processes. This is necessary to prevent multiple attempts to start the same broker being run simultaneously, which could lead to an inconsistent runtime state.

Resolution

Follow these steps to resolve the problem:

  1. Shut down all the broker processes on the system.
  2. Go to the /var/mqsi/locks directory. Run cat <file1> and note down the semaphore Id
  3. Run the command: ipcrm <semaphore Id>
  4. Run the command: rm -rf <file1>
  5. Repeat steps 2-4 for all the files in the /var/mqsi/locks directory
  6. Restart the broker
There are no abend files generated by the bipbroker process

Contact IBM support for help resolving your issue.

Resolution

Contact IBM support for help resolving your issue. Please provide the following documentation:

Problem with the broker-wide listener (biphttplistener) during startup

It is possible that the biphttplistener process is crashing, or abending during startup. Examine the $MQSI_WORKPATH/common/errors directory  for abend files related to the biphttplistener process.

The biphttplistener process produced an abend file showing biphttplistener ran out of memory.

There may be large HTTP messages (for example, greater than 8 MB) on SYSTEM.BROKER.WS.* queues.

Resolution

Export the following environment variable in the broker service Id's profile - export MQSI_LISTENER_BUFFER_SIZE=<new value in Kb>

This environment variable enables the user to set the threshold at which larger buffers are allocated for messages larger than the size set by the variable. Allocating a temporary buffer to handle a larger message adds a slight performance overhead, but gives the advantage that the HTTP Listener does not hold large buffers for each of its threads. Therefore, you should set this variable to an appropriate value to give the balance of memory usage and performance that you require.

There are no abends produced by biphttplistener

Contact IBM support for help resolving your issue.

Resolution

Contact IBM support for help resolving your issue. Please provide the following documentation:

There are other abend files prouced by biphttplistener

Contact IBM support for help resolving your issue.

Resolution

Contact IBM support for help resolving your issue. Please provide the following documentation:

The execution group process (DataFlowEngine) is unstable

If the DataFlowEngine process appears and disappears, then the problem is related to the execution group startup.

Resolution

Check that you have adequate free space in the $MQSI_WORKPATH directory and in your systems temps directory

ANSWERS:

Space in the $MQSI_WORKPATH directory is low

If you run out of space on this filesystem it can cause the DataFlowEngine to crash unexpectedly, and can cause a loop where the execution group continuously tries to restart.

The broker writes information to the $MQSI_WORKPATH directory for internal operation. This file system holds the broker registry, the configuration repository for all defined brokers and the binary trace logs for all brokers on the system.

Resolution
Ensure that you have enough disk space in $MQSI_WORKPATH directory. For example:
  • Check /var/mqsi/common/log for any large trace files or any core dumps. If there are any large trace files, then move them out of this directory.
  • If there are a large number of binary log files in the $MQSI_WORKPATH/common/log directory, then consider disabling trace and clearing out old trace files.
  • You can use the mqsreporttrace command to determine what trace is enabled for a given broker.
Space in the system temp directory is low

The Java Virtual Machine component of the execution group uses the systems temp directory to unpack nodes packaged as PAR files at runtime. If this space is not available then the classes for some nodes might not be available at runtime.

This can cause an error starting up the broker, or in some cases the broker can start successfully but you cannot see a BIP4512S error indicating that a java.lang.NoClassDefFoundError has been encountered.

The class referenced in this exception is contained in the lib directory of a PAR file that is part of the WebSphere Message Broker installation.

Resolution

Complete either of the following actions:

  • Make further space available in the system temp directory
  • Use one of the following methods to specify the location of this temporary JAR directory:
    • Use the environment variable TMPDIR.
    • Set the system property java.io.tmpdir.

To determine the minimum required TEMP space, and allow at least 50 MB of space per execution group in this directory, check the following page in the information centers - Memory and disk space requirements . You might need more space if you deploy large user-defined nodes or other JAR files to the broker.

There is plenty of space available

Examine the Event Log on Windows, the syslog on UNIX, or the joblog on z/OS and look for one of the following errors around the time of the broker startup.

Do you see any of the following errors in the logs?
You see a BIP2624E message in the syslog

The BIP2624E message is issued if the broker encounters a WebSphere MQ error. This message reports the specific WebSphere MQ error that occurred.

Resolution

If you are running WebSphere Message Broker as a trusted WebSphere MQ application, ensure that your user ID is a member of the mqm group. On UNIX and Linux, specify the user ID mqm as the service user ID when you create the broker.

You see a BIP2308E message in broker syslogs

The BIP2308E message is issued by an execution group (DataFlowEngine process) when a failure occurs during initialization while loading libraries (LIL files) and dependencies required for a successful startup. The library file name that is failing to load is included in this message.

Resolution
  1. Check that the file exists in the path specified in the BIP2308E message.
  2. Check that the permissions on the file are similar to other library files, and that Message Broker has access to the file for read and execute.
  3. Check that the Broker startup environment (for example, environment variable LIBPATH) is correct and does not have duplicate environment variables.
You see a BIP2374W message in broker syslogs

The file JDBC_XA_SWITCH_SPMF2 is used to store information required for shared memory operations between the queue manager and the broker, to support JDBC XA.

This requires access by both the mqm and mqbrkrs groups the file, and therefore requires world writeable permission. If the execution group process (DataFlowEngine) does not have the required permissions, then the execution group cannot start and throws the following BIP2374W warning when restarted after configuring broker for JDBC XA.

BIP2374W: The execution group cannot write to the file 'JDBC_XA_SWITCH_SPMF2' under /var/mqm.

This problem can occur if the Broker service ID does not have appropriate permissions to write to this directory.

Resolution

Ensure that the Broker service ID can write to the file JDBC_XA_SWITCH_SPMF2 in the /var/mqm directory.

You see a BIP3400E message during startup even if no adapter nodes deployed

The following error can occur even if there are no adapter components deployed to the execution group:

Resolution

Export the environment variable MQSI_HIDE_XSDZIP_FROM_ADAPTERS=1 in the broker service ID's profile and then try to restart the broker.

You see BIP7409 message in syslog

The BIP7409 message is issued by the execution group (DataFlowEngine process) when a failure is encountered during startup while creating a JVM. This message indicates that the execution group process was unable to create a JVM due to an invalid option.

Perform the following diagnostic activity:

  1. Check that any parameters passed in using the IBM_JAVA_OPTIONS or _JAVA_OPTIONS variables are valid. The command java -version should return successfully when issued from the broker's profile environment.
  2. Check that you do not have an invalid JVM option set in the jvmSystemProperty property on the ComIbmJVMManager object.
Resolution

In the Broker startup environment, set the following environment variable.

MQSI_DISABLE_JVM_SYSTEM_PROP = 1

This causes the execution group to start without using the jvmSystemProperty value. The jvmSystemProperty value can be corrected after startup, and then the broker restarted to resolve the problem.

At Broker level 8.0.0.3 and 7.0.0.6, you can also run the following command to change the jvmSystemProperty value while the Broker is in a stopped state.

mqsichangeproperties brokerName -e egName -o ComIbmJVMManager -n jvmSystemProperty -v "" -f

If the issue is caused by an invalid option set on either of the IBM_JAVA_OPTIONS or _JAVA_OPTIONS environment variables then correct this option in the broker profile's environment and restart the Broker.

In the syslog You see a BIP2060W message from the broker indicating that an execution group has ended abnormally

The BIP2060W message is issued by the admin agent (bipbroker process) when it detects that one of the execution group's processes has disappeared unexpectedly.

The admin agent automatically restarts the failed execution group to provide automatic recovery for tansient failures. However it is good practice to diagnose the root cause of the problem.

Perform the following diagnostic activity:

  1. Examine the UNIX syslog, windows event viewer or zOS joblog for the following messages: BIP2111E , BIP2226E , BIP2227E , BIP2228E
  2. Examine the $MQSI_WORKPATH/common/errors and the directory which mqsistart was run from, look for files containing the words "core", "corefile", "javacore", .phd which have creation timestamps which match up with the time the problem occurred.
Abend files have been generated

Broker generates abend files when it encounters a critical problem leading to process termination. The abend files are generated in $MQSI_WORKPATH/common/errors directory.

These abend files are text files, so can be viewed with any text editor. They have a header block of information that includes product version, date, time, and details, followed by a stack of the process.

To resolve most abend files, you need assistance from IBM support but you should review the content for stack backtraces related to any C user-defined nodes that you may have deployed. You can also use the information in the abend file to search for known issues which have already been fixed by previous APARs.

If you are unable to find a known issue, and the failure does not relate to a C user-defined node, then you should contact IBM support and provide the following documentation:

If you are able to reproduce the issue with service trace enabled then also provide a service trace of the affected ExecutionGroup.

You see corefiles generated by the broker

A corefile is produced when a program crashes unexpectedly. A corefile contains the state of the process at the time of failure.

Resolution

Contain IBM support for further diagnosis. Include the following additional information to aid diagnosis:

On some platforms you can perform additional diagnostic activities to aid the problem determination process.

  • On AIX systems:
    1. Ensure that you environment is configured to produce full core files, and generate a core file with these options set
    2. Use dbx to examine the corefile and issue the "where" command on each thread. This is helpful, because the native libraries used on the failing systems must be replicated on another machine to extract information about what execution was occurring at the time of the failure.
  • On Linux systems:
    1. Use gdb to examine the corefile and issue the thread apply all backtrace command.
You see javacore files or heapdumps produced by the execution groupp

Javacore files and heapdumps are generated by the JVM when certain error conditions are encountered. In some cases these will be caused by Java code associated with your Broker application, such as for example Java Compute Nodes or Java user-defined Nodes.

Resolution

Open the Javacore file in a text editor and examine the stack backtrace of the failing thread. Try to determine if the back trace indicates that execution was in Java code associated with your application.

Open the heapdump file using a tool such as Memory Analyser Tool or Heap Analyser. Look for leak suspects and try to determine if these objects are being referenced by classes associated with Java Code for your application or if these objects are associated with internal Message Broker use of Java.

In general Message Broker classes will be in packages with names starting in com.ibm.broker. Other IBM components which are shipped as part of message broker will typically be in packages whose names start with "com.ibm".

Which of the following best describes your problem?
  • Heapdumps or Javacore files indicate that there may be a problem with my Java application code in a Java Compute Node or Java user-defined Node
  • Heapdumps or Javacore files indicate that there is a problem with Message Broker's internal use of Java, or You are unable to tell conclusively if the problem is in application level code or Message Broker code
I can see a stable set of broker processes, but my broker isn't processing messages

If the set of broker processes is stable, then the execution group has started successfully but there may still be a problem with individual message flows.

Examine the state of the message flows, by using the administrative tools or the mqsilist command. Also check the Event viewer on windows, the syslog on UNIX, or the joblog on z/OS for errors related to message flows starting.

You see messages related to global caching problems when trying to start the broker

If the global cache components fail to start, then this may cause message processing in flows that depend on the global cache to fail. The global cache is initialized during broker startup, so any errors related to the cache starting appear when you are starting the broker.

You are using global caching while running a multi-instance broker configuration, and processing fails over to the standby broker. You may see BIP7195E messages reported by this broker depending upon the setup of your global cache.

If an active broker fails, and a standby broker starts up to take its place on a different machine, the catalogs and containers in that broker fail to start (as they are binding to the wrong hostname).

However, all execution groups can make client connections to the global cache, assuming there is a catalog still running in another broker. The original active broker needs to be restored for the cache components (catalogs and containers) within that broker to restart and rejoin the cache.

Resolution

When running in a high-availability environment, ensure that there are catalog servers running in other brokers.

Cache access from flows fail, and using the listhosts command returns a CWXSI0062E error

When two execution groups are stopped and only one is started, cache access fails and the 'listhosts' command results in 'CWXSI0062E' indicating that the catalog service is unavailable. It is only when the second instance of the execution group is started when the cache can be used and the 'listhosts' command shows both online hosts.

You can have an execution group running on two different brokers on different servers. Both execution groups are configured as catalog and container services. When both execution groups are running, caching works as expected. Either instance of the execution group can be stopped and caching continues to work as expected. The problem observed is when both execution groups are stopped and only one is started. Cache access fails and the 'listhosts' command results in 'CWXSI0062E' indicating that the catalog service is unavailable. It is only when the second instance of the execution group is started when the cache can be used and the 'listhosts' command shows both online hosts.

Resolution

If an active broker fails, and a standby broker starts up to take its place on a different machine, the catalogs and containers in that broker fail to start (as they are binding to the wrong hostname). However, all execution groups can make client connections to the global cache, assuming there is a catalog still running in another broker. The original active broker needs to be restored for the cache components (catalogs and containers) within that broker to restart and rejoin the cache.

None of these options describe the problem

If the global cache is still failing to start correctly, then contact IBM Support for help with troubleshooting.

Resolution

Please contact IBM Support and provide the following additional documentation to help with problem determination:

Do you have message flows that say they are up, but that are not processing any messages?

If a message flow is reporting it's status as running, but is not processing messages, then it is likely that either all instances of that flow are busy processing messages, or there is a configuration problem.

  • All instances of the flow are busy processing messages and therefore new messages cannot be picked up by the flow
  • There may be a problem with the configuration of the input node in your flow.
Does the flow respond to stop or start requests? (That is, when stopping or starting the flow does the state of the flow change when examined with mqsilist?)
The flow responds to start or stop requests

If the flow is responding to start or stop requests then it means that it is unlikely the problem is caused by a delay processing messages. Instead the problem is likely to either be a configuration issue with the input node or no messages are arriving on the input transport.

Resolution

Follow the troubleshooting advice specific to the input node that you are using in your flow:

Troubleshooting for the MQInput node

The MQInput node connects to the broker's parent queue manager and receives messages on the queue specified in the queueName parameter on the node.

Resolution

Examine the flow to determine the correct queue and perform the following diagnostic activity:

  1. Using WebSphere MQ Explorer, check the open input count on the input queue. Right click on the queue and then select Status. Confirm that the DataFlowEngine process has an open input handle on the queue.
  2. Check the queue depth of the input queue to confirm that there are messages waiting to be processed
  3. Check the $MQ_DATA_PATH/errors and $MQ_DATA_PATH/qmgrs/<queue_manager_name>/errors directory for FDCs and errors in the AMQERRXX.LOG files
  4. Use user trace to determine if the MQInput node is actually making MQGET calls and the MQ return codes from these calls, search in the UserTrace for BIP2636 messages.
Troubleshooting for the HTTPInput or SOAPInput nodes

The HTTPInput and SOAPInput nodes receive messages over sockets controlled by the broker listeners. By default, the HTTP nodes use the broker-wide listener (biphttplistener) and the SOAP nodes use the embedded execution group listener.

Resolution

Perform the following diagnostic activity to determine why these flows are not receiving messages.

  • Use the following command to determine which listener the SOAP and HTTP Nodes are configured to use.
    mqsireportproperties <broker> -e <execution group> -o ExecutionGroup -r
  • Examine the httpNodesUseEmbeddedListener and soapNodesUseEmbeddedListener properties to determine which listener is used by each set of nodes.
  • For nodes using the embedded listener check which port is configured using the following command:
    • (SSL nodes) mqsireportproperties <broker> -e <execution group> -o HTTPSConnector -n port
    • (Non-SSL Nodes) mqsireportproperties <broker> -e <execution group> -o HTTPConnector -n port
  • For nodes using the broker-wide listener check which ports is configured using the following command:
    • (SSL Nodes) mqsireportproperties <broker> -b httplistener -o HTTPSConnector -n port
    • (Non-SSL Nodes) mqsireportproperties <broker> -b httplistener -o HTTPConnector -n port
  • When you have determined which port is in use, run the following command to confirm if the port is open:
    netstat -aon | grep <port>
  • Use TCP/IP trace tools, such as wireshark, to verify that the underlying TCP/IP packets are reaching the broker port.
Troubleshooting for the JMSInput node

Perform the following diagnostic activity to resolve your issue with the JMS nodes.

Resolution
  • If the nodes are using a point-to-point model, check the JMS destination on the server to confirm there are messages present to be received
  • Using an external JMS client confirm that you can connect to the JMS provider using the setting provided to the broker
  • Ensure that the JMS client for your provided is correctly references in the JMSProvider configurable service. It is recommended that jars are placed outside the message broker install path.
Troubleshooting for the FileInput node

The FileInput node can operate either on local files or on remote files through FTP or SFTP. Follow the troubleshooting instructions most relevant to your situation.

File nodes configured for FTP

FTP is a text-based protocol. Perform the following diagnostic steps to help resolve your problem.

Resolution

If you have a problem logging in, ensure that you have set the correct login details using mqsisetdbpoarms.

  1. Take a TCP/IP trace by using a tool like wireshark. Filter the trace for packets containing ftp and ftp-data. You should be able to see the command flow between the FTP server and the broker.
  2. Ensure that the expected files are being returned from the NLST command. Check the return codes of FTP operations to confirm that the FTP server is not returning an unexpected error.
File nodes configured for SFTP

SFTP is an encrypted protocol based on SSH. Examine the Event Log on Windows, the syslog on UNIX, or the z/OS joblog for errors relating to SFTP connection.

Resolution

Perform the following diagnostic activities:

  1. Ensure the you have the correct login details set using mqsisetdbparms.
  2. If this fails to resolve the issue then contact IBM Support supplying the following documentation:
    • The output from running the mqsidc command on the runtime machine
    • PI of the message flow
    • Output from the mqsireportproperties <broker> -c FtpServer -o AllReportableEntityNames -r
    • An execution-level service trace of the failure
File nodes configured for local file

The FileNodes operate from an input directory and have a pattern based match for their files. Perform the following diagnostic activity.

Resolution
  1. Check if there are files in the input directory, check if these are readable by the File Input node.
  2. Check the files in the input directory match the pattern on the File Input Node
  3. Check the mqsitransitin subdirectory under the input directory. If there are files stuck here, stop the flow,  move them back to the File Input directory removing the pre-prended EG-UUID from the filename. Delete any locks in the mqsitransitin/locks directory before re-enabling the flow.
  4. If you have applications writing directly to the input directory ensure that the hold an exclusive lock on the file until they have finished their processing. If this is not possible then write files into the directory using a file name which does not match the pattern on the FileInput Node and then rename the file once writing is complete.
The flow does not respond to start or stop requests

Whenever a configuration request such as a start or stop request is made the execution group needs to temporarily stop the affected flow processing messages by taking a lock, the message flow releases this lock when it finishes processing a message and takes it again at the start of message processing..

Are your message flows not starting upon broker startup?

Message flows do not process messages while they are in the stopped state. Since WebSphere Message Broker Version 8 you can deploy flows with a stopped sate. If your flows display as stopped in the admin tooling or in mqsilist output then the first thing to check is if the flows are expected to be deployed in the "stopped state".

Examine the BAR file used to deploy the affected flows or application. On the configure tab examine the "Start Mode" property at the message flow level. What is the Start Mode value?
  • Maintained  - If the start mode is set to the default "Maintained" value, then any flows that were stopped when the broker were stopped are stopped when the broker starts up.
  • Manual  - If the start mode is set to the "Manual" value then the flow or application must be explicitly started after starting the broker.
  • Automatic  - If the start mode is set to "Automatic" then the flow should be automatically started when the broker is started
The start mode is set to "Maintained"

When the Start Mode property of a flow or application is set to "Maintained", when the Broker is started it will restore the last known state of this flow. If the flow was previously stopped then you will need to start the message flow manually using mqsistartmsgflow or the admin tooling.

Resolution

Start the message flow using mqsistartmsgflow or using the admin tooling.

The Start Mode is set to "Manual"

If the Start Mode is set to "Manual" the flow or application is not started when the broker starts up. The flow must be started manually using the mqsistartmsgflow command or the admin tooling.

Resolution

Start the flow or application manually using the mqsistartmsgflow command or the admin tooling.

The Start Mode is set to "Automatic"

If the Start Mode is set to "Automatic" or if the Start Mode is set to "Maintained" and the last state of the flow was started when the broker was shutdown, then the flow should be started automatically when the broker starts up.

If you start the message flow manually, does this resolve the problem?
Starting the Message flow manually resolves the problem

If the message flow starts successfully when you issue an mqsistartmsgflow command, or issue a start command via the admin tooling then the most likely issue is with the Start Mode defined on the flow.

If you have checked the start mode, then it is possible that your flows depend on resources that were not available at the time of message broker startup.

Typically built-in nodes do not cause the startup of a flow to fail if resources are not present, however throwing exceptions during the constructor or onInitialize() methods of Java Compute node can cause these symptoms.

Examine the Syslog on UNIX Systems, the Event Viewer on Windows systems, or the JobLog on z/OS for any errors relating to the starting of message flows.

Examine any dependencies that your application code may have during the constructor or onInitialize() methods of Java Compute nodes.

Resolution

Resolve any dependencies in Java Compute nodes, where possible use "lazy initialization" so that resources are obtained on first message to avoid throwing exceptions while the message flow is being constructed or initialized.

Starting the flow manually does not resolve the problem

If starting the flow manually does not resolve the problem then examine the Event Log on Windows, the syslog on UNIX or the JobLog on zOS for errors related to flow startup. Review the messages and see if the error text helps you resolve the issue.

Resolution

If you are still unable to resolve the problem then open a PMR to request support. Please collect the following materials to help IBM Support with a root cause analysis:

  • mqsidc output
  • Startup trace
I can see a stable set of broker processes, but administrative requests are not accepted

After all of the broker processes have been started, each execution group loads and starts all deployed message flows. When all of the flows have been started, the execution group issues a BIP2154I message with the following text:

Check the Windows Event Viewer, UNIX Syslog or zOS Joblog to see if there is a
BIP2154I message present with a timestamp indicating that it was output after you started the mqsistart command.
You do do not see a BIP2154I message

If you do not see a BIP2154I message stating that the "execution group has finished with Configuration Message" then the execution group may still be processing the initial loading of the message flows from group's repository.

This processing is common with the process that occurs when you deploy a barfile to an execution group. For more information, use the troubleshooting guide for problems deploying artifacts .

You see a BIP2154I message

If you see a BIP2154I message from each execution group stating that it has finished with a configuration message, then the the execution groups have started successfully and should be processing messages.

Execution groups are not processing messages

Many configuration operations performed on the broker require an execution group to perform work and respond to the admin agent. In some cases these requests cannot be performed while there is a message in flight in a message flow.

For example it is not possible to redeploy an application while that application is processing a message. Under normal conditions, when a configuration request is made then the execution group will process as soon as all affected flows have finished processing their current message.

If the execution groups are not processing messages, because execution is hung in the message flow then resolving this problem allows administrative activities to proceed normally.

See the section about troubleshooting when messages are not being processed by a flow .

Execution groups are processing messages normally, but administration requests are still not being processed successfully

If the execution groups are processing messages normally, then the administration request should not be blocked by message processing.

Examine the Event Log on windows, the syslog on UNIX, or the joblog on zOS.

Do you see the following BIP2080E error message?
BIP2080E execution group 'egName' failed to return an internal configuration message response within a 'timeout' second time period.
You see a BIP2080E message in the log

You see a BIP2080E message like the one below:

This indicates that a message was sent to the execution group for processing, but the message could not be processed successfully within the time period allowed.

The time it takes for startup is based on several factors, including performance of the machine, how busy it is at the time of start up, and the complexity of the execution group that is starting.

More configuration start up time is needed if the execution group has lots of flows, many dependencies that need resolving, or very complex flows.

For large deployments the default timeout period of 150 seconds is not always sufficient for processing to complete.

Resolution

Increase the time out value. This can be done using the mqsichangebroker command.

Use the -k flag along with the time in seconds for time out. More information can be found in the following infocenter topic: Setting configuration timeout values  in the WebSphere Message Broker Version 8 information center.

Increasing the timeout resolves the problem

If increasing the timeout resolves the problem you may consider changing your environment to improve startup times.

Resolution

If increasing the timeout resolves the problem you may wish to consider splitting up your configuration or, if you are running in a virtualized environment, providing additional CPI resource to the Broker system to improve startup times.

Increasing the timeout does not resolve the problem or increasing the timeout resolves the problem but the configuration message takes an abnormally long time to complete.

If increasing the timeout resolves the problem, but performance of startup is still unacceptable. Or if increasing the timeout does not work then there may be some tuning you can perform to improve startup up times.

You are running on z/OS and observe slow startup times

Athough all execution groups are started concurrently, they are required to load the same set of broker libraries. If these libraries have not been defined as shared-library programs, then while one execution group process is loading the libraries, the others wait until the underlying MVS locks on the libraries have been released.

The time taken for each execution group to start varies considerably, depending on where they were in the 'queue' to load the broker libraries and the overall startup time of the broker and all its execution groups is increased proportional to the number of execution groups.

If WebSphere Message Broker is running in a shared filesystem sysplex environment, then delays accessing files through the coupling facility can also result in slow startup times.

Resolution
  • Define broker files as shared-library programs, as described in the information center topic Defining WebSphere Message Broker files as shared-library programs .
  • Set an adequate OMVS SHRLIBRGNSIZE value. If this variable is set too low, starting the broker results in the HIGHWATER usage value reaching the SYSTEM LIMIT that has been set, and then not all of the libraries can be loaded into the shared region. This means that each of the execution groups will have to load these libraries themselves. If other applications are not also using the shared library region, then 268435456 (256Mb) is the recommended minimum value for SHRLIBRGNSIZE
  • Ensure that the following directories are mounted locally to the LPAR in which WebSphere Message Broker is started:
    • ++HOME++ is the location of the environment file (ENVFILE) used to create the runtime environment in which WebSphere Message Broker runs.
    • ++INSTALL++ refers to the WebSphere Message Broker installation directories.
    • ++COMPONENTDIRECTORY++ is the location where all deployed configuration is written to and read from by the WebSphere Message Broker runtime libraries.
    • ++JAVA++ is the location of the Java installtion.
    • ++MQPATH++ is the location of the WebSphere MQ installation.

    For more information about mounting these directories, see the information center topic Mounting file systems .

You are running on AIX

On AIX platforms there is a high degree of customization available in the way that underlying subsystems work. Some of these options are known to cause issues.

Run the env command from the broker's profile environment, and then look for the values of MALLOCOPTIONS, MQSI_SYSLOG_CONS_OFF, and LDR_CNTRL.

MALLOCOPTIONS=disclaim is set

The MALLOCOPTIONS=disclaim setting enables the malloc subsytem to return memory to the operating system when the memory has been freed. Normally, freed memory is reserved for use in future allocations by the same process.

Although the MALLOC=disclaim setting can reduce the size of the DataFlowEngine process, it is very CPU intensive.

Resolution

Disable the MALLOCOPTIONS=disclaim setting or set MALLOCOPTIONS to use a less CPU-intensive malloc strategy.

You do not see the variable MQSI_SYSLOG_LOG_CONS_OFF

The MQSI_SYSLOG_LOG_CONS option enables syslog calls to write directly to the system console if there is an error when using the system logger.

On some systems if the syslog is under heavy load then using the LOG_CONS option means the DataFlowEngine process can be forked. This causes an unexpected duplicate process.

If you see duplicate DataFlowEngine processes, then you should prevent broker from using the MQSI_SYSLOG_LOG_CONS option.

Resolution

Turn off the MQSI_SYSLOG_LOG_CONS option, by setting MQSI_SYSLOG_LOG_CONS_OFF=1 in the broker service ID's profile, and then restart the broker.

LDR_CNTRL=USERREGS is not set

This environment variable controls the way user registers are treated on function calls. Some levels of the JVM require these to be stored in order to operate properly.

Resolution

Set the variable LDR_CNTRL=USERREGS in the broker profile.

Non-platform specific suggestions

If you have enabled tracing then that can impact performance considerably at startup. Issue the following command to determine if trace is turned on:

Resolution

If trace is turned on, then disable it by using the mqsichangetrace command. For more information, see the information center topic Stopping service trace .

You do not see a BIP2080E message in the broker log

If you do not see BIP2080E messages in the broker log, then it is possible that the problem is caused by a build up of messages on the queues used for internal communication between the admin agent (bipbroker) and the execution groups (DataFlowEngine).

Resolution

Stop the broker, and then examine the SYSTEM.BROKER.ADMIN.*, SYSTEM.BROKER.DEPLOY.* and SYSTEM.BROKER.EXECUTIONGROUP.* queues. If there is a build up of messages on these queues, then clear the queues and then restart the broker.

You have a problem deploying artifacts to the broker runtime...

You have a problem when you try to deploy a Message Broker BAR file to a broker runtime. Try to eliminate the toolkit deploy method as the source of the problem, then investigate other deploy methods and components used.

There are multiple ways to deploy a resource; for example, using the Message Broker Toolkit, using Message Broker Explorer, using the mqsideploy command, or even using the CMP API directly.

All of these methods share some common processing when the deploy request has reached the broker runtime. Deploying using the toolkit is considered the default option, so the first step is to try deploying this resource in the toolkit to eliminate the toolkit deploy method as the source of the problem.

Try deploying with the Message Broker toolkit to see if this resolves the problem. Alternatively, follow through the following troubleshooting topics, to see if resolving a problem with some processing common to all deploy methods resolve your problem.

Problem deploying from the toolkit or a general deploy problem

When deploying a BAR file to a broker runtime, there are two main types of processing involved. Both can be investigated by deploying from the toolkit.

  • The application performing the deploy action makes several CMP API calls to the Broker, which in turn cause messages to be sent to the Broker components. The Broker components then return messages, in an asynchronous manner, to the deploying application to indicate the success or failure of the deploy action. This whole set of operations can be thought of as the processing of the deploy message itself and can be treated as one class of problem.
  • In addition to the handling of the deploy message processing, the runtime needs to create objects representing the artifacts in the BAR file being deployed. It is usually best to consider this part of processing as a separate class of problem.

There are several ways to determine if the deploy message processing completed successfully.

When deploying from the toolkit, does the "Progress Information" dialog box disappear and there are no red errors reported in the deployment log pane?

  • No  (The deploy message was not processed successfully)
  • Yes  (The deploy message was processed successfully but something else was wrong.)

Examining the UNIX syslog, Windows event viewer, or z/OS joblog, can you see a BIP2154 message that corresponds to this deploy action?

  • No  (The deploy message was not processed successfully)
  • Yes  (The deploy message was processed successfully but something else was wrong.)
The deploy message was not processed successfully

When deploying a BAR file, the deployment request is made from the toolkit by using the CMP API to the admin agent (BIP broker process). This involves several components that need to be investigated.

Follow each link for more help troubleshooting each component that is involved in the deploy action.
Deploy problems with an execution group

During the deploy of resources, the execution group emits a series of informational BIP messages to feedback the status of the deploy. These can be found in the syslog on UNIX platforms, in the event log on Windows platforms, or in the joblog on z/OS.

The expected sequence of messages for an example deploy is as follows:

  1. BIP2152: ( BRK8.GenericRoutingEG ) configuration message received from broker.

    An execution group received a command from the Broker.

    No user action is required.

    This message indicates that the bipbroker process has passed a configuration message to the execution group for it to handle and marks the start of the execution group entering deploy processing.

  2. BIP2153: ( BRK8.GenericRoutingEG ) About to ''Change'' an execution group.

    This message indicates that the execution group has identified that it needs to change its configuration based on the message received.

  3. BIP2155: ( BRK8.GenericRoutingEG ) About to ''create '' the deployed resource ''ZDISTR01_731_IDOC_outbound'' of type ''.APPZIP''.

    You see messages like BIP2155 for resources like applications and message flows contained in the BAR file. You see one BIP2155 for each resource as it is handled by the execution group.

  4. BIP 2154: ( BRK8.GenericRoutingEG ) Execution group finished with Configuration message.

    This indicates that the execution group has finished with the deploy message and has sent a confirmation to the bipbroker process. At this point the deploy message has nominally been processed successfully.

The source of the problem can be isolated by determining at which of these steps in the deploy process that failure occurred.

Resolution

Follow through the BIP messages given in this topic and determine to which of these deploy steps your BAR file got in deploy processing. Examine the error immediately after the last known good step, and review the content to determine the root cause of the problem.

If necessary use the mqsiexplain command to find the user response for a particular error message.

The deploy message was processed successfully, but something else is wrong

If BIP2154 has been observed in the UNIX syslog, windows event viewer or zOS joblog then the deploy itself has completed successfully with none of the Broker components detecting a problem during processing the deploy request.

Deployed to a version 8 or later broker, but something did not work as expected

By default at version 8 or later, artifacts are deployed in the same format as the toolkit uses without an intermediate compilation step. This can cause a change in behaviour in some cases, compared to earlier versions of WebSphere Message Broker.

For example, message flows are deployed as .msgflow files instead of the compiled message flow format (.cmf files) used in earlier versions of Message Broker. Similarly ESQL files are deployed as individual objects instead of being compiled into the message flow when the BAR file is built.

This can cause a change in behaviour in some cases, but it is possible to return to the old method of deployment used in earlier versions of WebSphere Message Broker. This should help exclude the deploy format as the root cause of the problem, and provide a workaround for the issue.

Resolution

To revert to the previous deployment method (.cmf file), open your BAR file in the Message Broker Toolkit and then tick the "Compile and inline resources" check box before rebuilding the BAR file. Redeploy the modified BAR file.

You have problems deploying a BAR file to a version 7 broker, or with compiled resources to a version 8 or above broker

At version 7 or earlier, the resources used in a message flow are compiled into an intermediate format before deployment to the broker (*.cmf files). In this format the resources are pulled into the message flow definitions prior to deployment. For example, ESQL files appear inline in the XML representation of the flow.

You have multiple flows that use the same resources, but seem to be using different copies of the resources

When using applications and libraries, applications have their own copy of the following types of resource:

Resolution

If you want changes made in a library to take effect across all applications, then you must rebuild and redeploy every application that uses the library.

The deploy action appears to have been successful, but one or more artifacts do not appear in either the brokers view in the toolkit or in the Message Broker Explorer

If the deploy action has completed successfully without any error messages, but the deployed resources do not appear in the administrative tools, then the most likely explanation is that the BAR file does not contain the expected resources.

Are all of the expected artifacts present in the BAR file?
Some of the artifacts I expected are not present in the BAR file

The BAR file must include all artifacts needed by your message flow or application.

Resolution

Rebuild the BAR file, but making sure that you include all relevant resources on the prepare tab of the toolkit. Remember that any resources that have an error in your workspace will not be included in the built BAR file.

Help using this guide

This guide comprises a set of information nodes connected by links. You can click a link to move to the linked node, or can click a node to go directly to that node.

Scroll bars are provided for you to browse across the set of nodes.

An Overview presentation is provided for compatible web browsers. The Overview gives you a smaller-scale iew of all nodes in the guide. In the Overview, you can click any node to go directly to that node.