Automated SNMP Alerts in WebSphere Message Broker

Since message flows often contain business-critical functionality, administrators need to receive immediate notification of errors during flow execution. This article shows you how to enhance message flow monitoring to automatically send Simple Network Management Protocol (SNMP) alerts to an administrative console in order to streamline your organization's error notification and error response processes.

Aloak Kapoor (aloak@ca.ibm.com), Application Architect, IBM Global Business Services, IBM

Aloak Kapoor is an Application Architect with IBM Global Business Services. He has several years of experience in the design and development of large-scale enterprise systems, and he has a Bachelor's degree and a Master's degree in Computer Science. You can contact Aloak at aloak@ca.ibm.com.



Bobbee Broderick (rkbroder@us.ibm.com), Solutions Specialist, IBM Software Services for WebSphere, IBM

Photo of Bobbee BroderickBobbee Broderick is an Application Integration and Middleware (AIM) Solutions Specialist with IBM Software Services for WebSphere (ISSW). His responsibilities include architecture, design, and development for customer engagements for WebSphere Message Broker and WebSphere MQ (including WebSphere MQ FTE and WebSphere MQ AMS). He is currently the ISSW Technical Lead for WebSphere MQ. You can contact Bobbee at rkbroder@us.ibm.com.



17 October 2012

Also available in Chinese

Introduction

In WebSphere Message Broker, message flows can use built-in functionality to send notifications when problems occur. For example, the SMTP node can send an e-mail notification, or the File Output node can log issues to a file. In many cases, a better option for problem notification is to send SNMP alerts directly to an existing management console. Medium and large enterprises often have a system and network management console that monitors the health and performance of key IT components. Sending notifications directly to the management console has several advantages:

  • Real-time, push notification of problems to a centralized monitoring location
  • Faster response, since notifications are pushed to a (likely optimized) existing console
  • Reduced costs because new software and processes do not need to be implemented and learned
  • Reduced testing

This article shows you how to configure a message flow to push SNMP alerts to a management console in real-time. The next section describes the business scenario that will be used in this article, including the basic message flow. The article then shows you how to add automated SNMP alerts to the message flow and how to test the alert functionality in a development environment. The conclusion summarizes the concepts presented and offers some ideas on how to extend the functionality in this article.

Terminology note

In this article, the terms SNMP alert and SNMP trap are used interchangeably to refer to the built-in notification message in the SNMP protocol.

Prerequisites

A Microsoft Windows development environment with the following components installed:

  • WebSphere Message Broker V7 (including Message Broker Explorer)
  • WebSphere MQ (any version compatible with WebSphere Message Broker)
  • WebSphere Message Broker Support Pac IAM3 (log4j). Some familiarity with log4j -- the popular Java logging framework -- is helpful but not required.
  • Some familiarity with WebSphere MQ and WebSphere Message Broker development
  • No prior experience with the SNMP protocol or with a specific management console is required

Scenario: Message flow with potential points of failure

To illustrate the concepts of automated SNMP alerts in WebSphere Message Broker, this article uses a fictitious retail company called the ABC Company that uses a message flow to process orders. The message flow retrieves a product order from an input queue, calls an external inventory system, and then places a completed order message on another queue:

Figure 1. Order Processing for ABC Company
Order processing for ABC Company

To set up your local development environment for this scenario, complete the following steps:

  1. Locate an existing local broker for which you've set up the log4j SupportPac.
  2. Create two local queues named ABC.ORDERS and ABC.PROCESSED_ORDERS on the broker's queue manager.
  3. Download and import the message flow project CompanyABC_OrderProcessing_MFP at the bottom of the article.

The dependency: A call to an external system

Take a closer look at the message flow that you just imported. The first and third nodes perform simple get and put operations on MQ queues. The Compute node contains the key functionality -- a call to an external inventory system, which is a potential point of failure in the message flow, because the external system may go down, or the communication link to it may fail. Listing 1 shows the Extended Structure Query Language (ESQL) code for the Compute node.

Listing 1. Code to call external inventory system
CREATE COMPUTE MODULE processOrder_CallInventorySystem
    CREATE FUNCTION Main() RETURNS BOOLEAN
    BEGIN

        DECLARE ResultOfInventoryUpdate BOOLEAN;
        SET ResultOfInventoryUpdate = UpdateInventorySystem(InputRoot);
        IF (ResultOfInventoryUpdate) THEN
            SET OutputRoot = InputRoot;
            RETURN TRUE;
        ELSE
            DECLARE ErrorMsg CHARACTER 'Cannot communicate with inventory system';
            THROW USER EXCEPTION CATALOG 'BIPv700' MESSAGE 2951 VALUES(ErrorMsg);
        END IF;
     END;

     --Simulates an update call to an external inventory system.  
     --Returns false if link to external system is down, and true otherwise. 
     CREATE FUNCTION UpdateInventorySystem(IN element REFERENCE) RETURNS BOOLEAN 
     BEGIN
	
         RETURN TRUE;

     END;

END MODULE;

For simplicity, the code uses the UpdateInventorySystem function to call the external inventory system. In practice, this function would contain the actual code to communicate with a database or a separate application. The code in this example uses return values of TRUE to simulate a successful call to the external system and FALSE to simulate a failure.

Adding SNMP alerts

To manage the potential failure of the external inventory system, you will now add automated SNMP alerts to the message flow, using the log4j logging framework. In the log4j framework, you log messages to a destination by creating a logger and associating an appender (destination) with it. The logger logs messages that meet or exceed its defined log-level to the appender. To enable output of SNMP traps, the key step is to configure an SNMP appender in log4j.

The SNMP appender converts the log messages it receives into SNMP traps, and forwards them to an SNMP listener. Implementations of SNMP appenders for log4j already exist, so you do not have to create an appender from scratch. The next section shows you how to create a log4j configuration file that sets up a logger and appender.

Creating the log4j configuration file

  1. Create a new, blank XML file and name it log4jConfig.xml.
  2. Place this file in the broker's classpath (for example, <Broker Install Dir>\classes). In addition, place the log4j.dtd file in the same location, as described in the instructions for the log4j SupportPac. The log4j.dtd file is bundled with the log4j SupportPac download.
  3. Add the following XML content to the log4jConfig.xml file:
    Listing 2. log4j configuration file
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
    
    <log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
    
        <appender name="TRAP_LOG" class="org.apache.log4j.ext.SNMPTrapAppender">
            <param name="ImplementationClassName" 
                      value="org.apache.log4j.ext.JoeSNMPTrapSender"></param>
            <param name="ManagementHost" value="127.0.0.1"></param>
            <param name="ManagementHostTrapListenPort" value="162"></param>
            <param name="EnterpriseOID" value="1.3.6.1.4.1.24.0"></param>
            <param name="LocalIPAddress" value="127.0.0.1"></param>
            <param name="LocalTrapSendPort" value="161"></param>
            <param name="GenericTrapType" value="6"></param>
            <param name="SpecificTrapType" value="12345678"></param>
            <param name="CommunityString" value="public"></param>
            <param name="ForwardStackTraceWithTrap" value="true"></param>
            <param name="Threshold" value="WARN"></param>
            <param name="ApplicationTrapOID" 
                      value="1.3.6.1.4.1.24.12.10.22.64"></param>
            <layout class="org.apache.log4j.PatternLayout">
              <param name="ConversionPattern" 
                        value="%d{ISO8601} %-5p [%-10t] %m%n"></param>
            </layout>
        </appender>
    
        <logger name="abcLogger">
            <level value="WARN"/>
            <appender-ref ref="TRAP_LOG" />
        </logger>
    
        <root>
        </root>
    
    </log4j:configuration>
  4. Save the log4jConfig.xml file.
  5. This file specifies a log4j appender named TRAP_LOG:
    <appender name="TRAP_LOG" class="org.apache.log4j.ext.SNMPTrapAppender">

    This appender references an external class named SNMPTrapAppender, which is an open-source implementation of an SNMP appender, and is available under the Apache 2.0 license. The rest of this section assumes that you are using SNMPTrapAppender, but you may specify an alternative log4j SNMP appender implementation class if you prefer.
  6. Download the SNMPTrapAppender JAR file.
  7. Save the SNMPTrapAppender JAR file in <Broker Workpath>/shared-classes.
  8. Look once again at the log4j appender configuration, specifically the parameter named ImplementationClassName:
            <param name="ImplementationClassName" 
                      value="org.apache.log4j.ext.JoeSNMPTrapSender"></param>
    SNMPTrapAppender relies on a separate code library to actually send the SNMP traps. The ImplementationClassName parameter specifies the library that is used for this purpose: joeSNMPTrapSender, which is an open-source library available under the GNU LGPL license V2.1 or later.
  9. Download V0.3.4 of the joeSNMP library.
  10. Unzip the download and extract joeSNMP.jar, which can either be found in the output/lib directory, or generated by running the build scripts provided with the download.
  11. Place joeSNMP.jar in <Broker Workpath>/shared-classes.
  12. Observe two additional parameter elements in the log4j appender configuration: ManagementHost and ManagementHostTrapListenPort. These two parameters specify the IP address and port number of the SNMP listener that will receive the trap messages. You can leave these at the default values, but you may need to change them for different application environments, depending on the address and port of the SNMP listener in each environment.
  13. Look over the remaining param elements specified for the TRAP_LOG appender. Most of the remaining values are related to the format and content of the SNMP trap message. They are already set to reasonable defaults, but you may change them to customize your SNMP trap according to your needs.
  14. Underneath the appender, a simple logger has been defined that outputs all log messages at level WARN or higher to the TRAP_LOG appender:
        <logger name="abcLogger">
            <level value="WARN"/>
            <appender-ref ref="TRAP_LOG" />
        </logger>

Creating the log4j node

To fully enable SNMP alerts in your message flow, drag a log4j node onto the flow, and connect its input terminal to the failure terminal of the Compute node as shown below:

Figure 2. Message flow with SNMP alerts
Message flow with SNMP alerts

Table 1 specifies the properties that you must set on the log4j node:

Table 1. log4j node properties
Property NameProperty Value
Log4j Config Filelocation of the config file you created in the previous section, for example:
C:\Program Files\IBM\MQSI\7.0\classes\log4jConfig.xml
Log LevelWARN
Log TextError communicating with external inventory system.
Logger NameabcLogger

Notice that you have set the log4j node to log WARN level messages whenever an unhandled error or failure occurs in the Compute node. Moreover, since you set up the log4j node to log to abcLogger from your configuration file, all appenders you specified for that logger will automatically receive the log messages. Therefore, the SNMP appender will be invoked whenever a failure or error occurs in the Compute node.

Testing the SNMP functionality in a development environment

Testing the alert functionality presents a challenge because access to your system and network management console may not be available at development time, from a development environment. To work around this limitation, download a standalone SNMP trap listener application, and configure it to receive traps on your local development machine. This article uses the SNMP trap daemon that is part of the Net-SNMP suite of applications. Follow the steps below to set up this listener on your local machine.

Setting up a local SNMP listener

  1. Go to the Net-SNMP project home page, review the license agreement, and download the latest binaries. The rest of this section assumes that you are using Net-SNMP V5.6.1.1.
  2. Run the Net-SNMP executable setup file.
  3. Select only the Net-SNMP Trap Service component for installation.
  4. Select an install directory on your development machine and complete the installation.
  5. Create an empty file that will contain the custom configuration for the SNMP trap daemon and add the following text to it: disableAuthorization yes . Name the file snmp.conf and save it inside the bin subdirectory of the Net-SNMP install directory that you specified in the previous step. As a default, Net-SNMP only accepts SNMP traps from authorized users. Since this tool is being used on a development machine for simple testing, the configuration file you create in this step allows any user to send traps to the local listener. Once development testing is completed, you must stop the SNMP listener to prevent unauthorized users from sending traps to it.
  6. Open a command prompt and navigate to install_dir/bin, where install_dir is the install directory you chose for Net-SNMP.
  7. Run the SNMP trap daemon with the following command-line options: ignore the default configuration file, use the custom configuration file you created in this section, and log output to standard out:
    snmptrapd.exe -C -c snmp.conf -Lo.
  8. The SNMP trap daemon should run. Ignore warnings about missing MIB files, since you are only using the daemon for simple testing purposes. The daemon listens for traps on default port 162. You may need administrator privileges on the machine for the SNMP trap daemon to open and listen on the port.
  9. If you experience any errors when trying to launch the SNMP trap daemon, see the help documentation for the snmptrapd server in the file install_dir/docs/Net-SNMP.chm.

Testing the SNMP alerts

Now that a local SNMP listener is configured and running on your development machine, you can test the functionality by causing the Compute node in the message flow to throw an exception. Recall from the above section The dependency: A call to an external system that the Compute node was configured to throw an exception whenever its UpdateInventorySystem function returned false. To test the flow:

  1. Change the code in the UpdateInventorySystem function to return FALSE.
  2. Save the message flow project.
  3. Stop and restart the target broker that will run the flow to ensure that the broker picks up the new files placed in its classpath in the previous steps.
  4. Generate a new BAR file containing the message flow and deploy it to an execution group on the target broker.
  5. Using MQ Explorer, place a test message on the ABC.ORDERS queue.
  6. You should see a message appear in the SNMP trap daemon command console, indicating that an SNMP trap message was sent by the broker and caught by the local SNMP listener. In this case, all steps were completed successfully, and SNMP alerts are enabled in the message flow.
  7. If you do not see a message appear in the SNMP trap daemon, there are several possible causes given the large number of components involved. Carry out the following tasks to locate the problem:
    1. Run the flow in debug mode, and ensure that flow execution reaches the input terminal of the log4j node. Check that no error is thrown by the log4j node.
    2. If the input message appears to be properly consumed by the log4j node, check for any errors or warnings in the log generated by the log4j node. The node's log file can be found in
      <Broker Workpath>\components\BrokerName\BrokerID\console.txt.
    3. Ensure that WARN has been specified as: the value of the logger's level child element, the value of the appender's Threshold parameter, and the value of the log4j node's LogLevel property.
    4. Review the installation instructions for the IAM3 log4j support pack, and ensure that the necessary JAR files have been placed in the correct directories. Also ensure that the user account under which the Message Broker service runs has read access to these files.
    5. Stop and restart the Message Broker service. Place a new test message on the ABC.ORDERS queue, and check if a message appears in the SNMP trap daemon.
    6. If no message appears in the SNMP trap daemon, check the Windows Event Log for any errors or warnings triggered by Message Broker.

This test exercised all of the elements of the SNMP configuration that you created. The test message on the input queue raised an exception in the Compute node, which triggered a log4j message to be sent out by the logger you set up. The logger was configured with an SNMP appender, which transformed the log message to an SNMP trap, and sent the trap to the SNMP listener in your development environment.

Conclusion

This article showed you how to configure WebSphere Message Broker to send SNMP alerts in real time. You began by creating a simple message flow that had a potential point of failure during its communication with an external system. You then constructed a log4j configuration file in which a log4j logger sent messages to an SNMP appender, and you downloaded a log4j SNMP appender class and an SNMP-sending subsystem. Next, you used a log4j node from a Message Broker SupportPac to tie everything together by invoking the log4j logger from your configuration file whenever the message flow failed to contact the external system. Finally, to test the functionality in a development environment, you installed and configured an SNMP listener on your local machine.

There are several interesting ways you can extend the ideas presented in this article. For example, you can make the log message generated in the SNMP alert dynamic by parsing the exception tree when an error occurs, extracting the root exception message, and then storing this message inside the Message Broker Environment tree. The log4j node can then point its log text property to the variable value inside the Environment tree.

Another useful technique is to associate multiple types of log4j appenders to a single logger, and configure each appender with a different severity threshold. The SNMP appender can trigger only at the highest severity level, pushing SNMP alerts to an administrator only when serious errors occur, and not when minor or routine issues surface.


Download

DescriptionNameSize
Code sampleCompanyABC_OrderProcessing_MFP.zipNN KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=840544
ArticleTitle=Automated SNMP Alerts in WebSphere Message Broker
publish-date=10172012