Real-time event processing and dashboard visualization for Sterling Commerce Managed File Transfer

This article shows how you can use a sample dashboard, provided for download, to integrate IBM® Sterling® Commerce Managed File Transfer products with IBM Business Monitor to achieve greater visibility of your business transactions. You'll learn how to install the dashboard in your environment and how you can customize it for your own needs. This content is part of the IBM Business Process Management Journal.

Wilfred Jamison, Ph.D., Comprehensive Visibility Architect, IBM

Dr. Wilfred C. JamisonDr. Wilfred Jamison is currently a lead architect on the Sterling Commerce Integration Development team, where he is focused on comprehensive visibility across Sterling Commerce products and the rest of the IBM Software Group portfolio. In the past, Wil has been a technical lead in business process management, a customer advocate, and an expert in WebSphere Application Server performance and security. He was also active in cloud-based development efforts, such as Blueworks Live.


developerWorks Contributing author
        level

Palanisamy Gounder (pgounder@us.ibm.com), Software Engineer, IBM

Palani Gounder photoPalani Gounder is the leader of the IBM Sterling Control Center (SCC) development team, and collaborates with the various IBM Sterling B2B development teams for products that are monitored by SCC. Prior to that, he was a member of IBM Sterling Integrator development team and worked on the trading partner profile management module. Palani has over 18 years of software development experience with a wide range of software technologies. He has worked as a software engineer with Sterling Commerce for more than 10 years.



Thurman Pylant (tpylant@us.ibm.com), Software Architect, IBM

Thurman Pylant photoThurman Pylant has been with Sterling Commerce since 1996. He was responsible for the development of the Connect:Direct Browser User Interface, Connect:Direct FTP+ and is currently the architect for IBM Sterling Control Center development. He is also a key member of the Connect:Direct for z/OS development team. Previously, Thurman was a lead developer on the other Connect:Direct mainframe products. Before joining Sterling Commerce, Thurman spent 25 years with IBM working on mainframe operating system development and support of those operating systems.



29 September 2011

Introduction

In a developWorks column a couple of months ago, we introduced an early version of sample dashboard for the Sterling Managed File Transfer (MFT) portfolio using IBM Business Monitor V7.5. This article is a follow-up to that column, in which we'll provide the final dashboard for download, and show you how you can use it in your own environment.

If your business enterprise is either planning to have or currently has installations of IBM Sterling Control Center V5.3, IBM Sterling Connect:Direct, IBM Sterling File Gateway, IBM Business Monitor V7.5 and optionally IBM WebSphere MQ V7.0, or if you are simply interested in learning about an integration scenario between Sterling products and Business Monitor, this article is for you.

In this article, we'll describe the sample dashboard and the rationale behind its design in greater detail, discuss how the integration between Sterling Control Center and Business Monitor is implemented, and show you how to set it up to run it in your own environment, as well as how to customize it to your own needs.


What the dashboard provides

IBM Sterling Control Center provides monitoring solutions for both IBM Sterling Connect:Direct and IBM Sterling File Gateway. The importance of monitoring was illustrated in the abovementioned developWorks column. We are taking this monitoring solution to the next level by leveraging the real-time event processing capabilities of IBM Business Monitor, as well as the extensible user interface (UI) model used by Business Space, which is based on iWidgets technology (refer to iWidget Overview).

The sample dashboard will help you achieve better visibility with your products. The dashboard provides:

Improved overall user experience

The new dashboard provides better usability through data visualization. Better visual and graphical representation helps you interpret data more easily and quickly, through the use of colors, shapes and dimensions as visual cues. It provides better data organization and layout to makes navigation more natural and intuitive. You can customize your dashboard settings to display specific information based on your needs or roles.

Additional visibility information

The integration of Sterling Control Center and IBM Business Monitor enables the viewing of data through the dashboard that is not currently available in Sterling Control Center.

Maximized use of resources

Customers who currently have installations of Sterling Control Center and Business Monitor, or those who are planning to have them installed, can further maximize the value of their investments, and recognize a quicker time-to-value for setting up a dashboard-oriented visibility solution.

A step towards a cohesive and consistent visibility solution for Sterling

Aside from leveraging the real-time event processing, data visualization and business analytics capabilities of Business Monitor, this dashboard is a use case that we hope can shape and mold a common visibility framework for the whole Sterling Commerce portfolio. A common visibility framework introduces coherence and consistency to all visibility solutions in Sterling. It can also facilitate the implementation of integrated visibility solutions when two or more Sterling products are used together in an enterprise.


An integrated visibility dashboard for managed file transfer

Visibility solutions in managed file transfer (MFT) products can be improved through event monitoring and the use of dashboard technologies provided by IBM Business Monitor V7.5. This is possible because Sterling Control Center V5.3, and later versions, are enabled to emit events to Business Monitor. This means that Business Monitor dashboards can be readily created in Business Monitor's UI framework, called the Business Space. A dashboard is a space in Business Space, which consists of one or more pages.

The sample MFT dashboard for Sterling Connect:Direct and Sterling File Gateway consists of five pages, as shown in Figure 1.

  • Connect:Direct Current Activity
  • Connect:Direct Activity History
  • Sterling File Gateway Current Activity
  • Sterling File Gateway Activity History
  • Tools
Figure 1. The sample MFT dashboard containing five pages
The sample MFT dashboard containing five pages

(See a larger version of Figure 1.)


Which contents to focus on?

One of the major tasks when designing a dashboard, and possibly the most important one, is deciding what contents to show and how they should be laid out. A secondary task is personalizing and customizing the dashboard to complement the desired overall user experience. This includes deciding on the color scheme, titles, labels, banners and themes, and so on. In our dashboard, we chose to stick with the old and proven principles of simplicity, clarity, consistency and relevance. But the more important guideline we kept in mind is was to understand our target users. In this case, our target users are business operators and solution specialists.

As for the contents, we focused on the fundamentals, which for Connect:Direct means monitoring for files that are received and sent, as well as for failed file transfer transactions. For Sterling File Gateway, we monitor deliveries, routes, and route failures.

We organized the contents by clearly separating between Connect:Direct and Sterling File Gateway, where each product has two pages. The first page for each product serves as a landing page - the Connect:Direct Current Activity page and the File Gateway Current Activity page, respectively – which is intended to display the recent file transfer activities in the enterprise. Studies have shown that most users will be interested first in finding out the status of the current activities in the business and the overall performance. Moreover, the more real-time the information is, the more useful and helpful it is to them. Business Monitor processes data as it arrives and presents it to a dashboard in near real-time depending on a specified refresh interval.


Which widgets to use?

The Connect:Direct Current Activity page contains four types of widgets. These are the same widgets that we introduced in our developWorks column; namely, Key Performance Indicators (KPIs), Alerts, Report Viewer and Instances, as shown in Figure 1. The KPIs widget entitled File Transfer Status (Last Hour) shows the number of transmitted files, the number of files received and the number of transfer failures within the last one-hour period. This KPI widget is likely the very first thing a user would want to look at and, because the user is interested in the most current activities, narrowing the scope to a one-hour window should satisfy that purpose. Meanwhile, the Alerts widget entitled Dashboard Alerts shows both the read and unread alerts of failed transfers. This is likely the second thing that a user would be looking at, because any failure is critical to the business and therefore must be attended to.

The Report Viewer widget provides daily file transfer status (sent, received and failed transfer) for each day of the current month. The thinking behind this widget is that a user, after seeing the recent status of the file transfer activities in the last hour, may want to reference and compare the current status with those in the previous days. This can provide more insight as to whether or not what he or she is seeing now is normal or aberrant. In the latter case, a user may decide to investigate the situation further.

The Instances widget on this page displays both completed and in-progress Connect:Direct processes for the day. One row per process is displayed at the top level. You can view all steps associated with each process by clicking the icon under the Connect:Direct Steps column. The icon under the first column indicates whether a process has completed successfully or not. The warning icon indicates that the process did not complete successfully. Since this widget contains all the details of all transactions for the day, you can use it to do various types of analysis. He or she can search for a specific transaction, sort transactions based on a given column, and so on.

The second page, Connect:Direct Activity History shown in Figure 2, is intended to show historical data of all past transactions. Unlike the first page, this page will be visited less often, probably on a periodic basis or when a problem arises where some historical information would be helpful for investigation and analysis. This page contains graphical charts showing files sent by node, files received by node, and an overall status both by originating and remote nodes. Note that even though the reports are only showing month-to-date data, the time window can be changed easily right there on the dashboard (via the settings). The results are updated immediately. Even the report title can be changed very quickly. A useful feature of the report is that you have the ability to drill down based on the dimensions that were defined. Under the hood, Business Monitor uses IBM Cognos Business Intelligence V10.1 for all reporting and business intelligence features.

Figure 2. The Connect:Direct Activity History page
The Connect:Direct Activity History page

(See a larger version of Figure 2.)

The third page, shown in Figure 3, contains the current activities for Sterling File Gateway. You'll notice that the widgets used and their layout are exactly the same as those in Connect:Direct. This is because we applied the same design philosophy and usage pattern we discussed earlier.

In summary, the Sterling File Gateway Current Activity page contains KPI and other widgets related to most recent File Gateway activities. The performance KPI widget contains KPIs for total deliveries, total routes and total failed routes for the last one-hour period. The Report Viewer widget has month-to-date information about arrived files, routes and deliveries. The instance widget called Today’s Arrived Files shows today's arrived files that were processed and the arrived files that are in the process of being routed and delivered.

Figure 3. The Sterling File Gateway Current Activity page
The Sterling File Gateway Current Activity page

Figure 4 shows the File Gateway Activity History page, which contains widgets that show historical (month to date) information about Arrived Files by Producer, Delivered Files by Consumer, and complete file status by both producer and consumer. Again, this historical information can be used for regular reporting and for problem analysis.

Figure 4. The Sterling File Gateway Current Activity page
The Sterling File Gateway Current Activity page

(See a larger version of Figure 4.)

Finally, Figure 5 shows last page of the sample dashboard, called the Tools page. It contains three widgets that you can use to manage some of the contents and configuration in the dashboard. For example, you can use the KPI Manager to create new KPIs or to change the settings of an existing KPI such as the colors, the range values, and so on. Meanwhile, you can use the Report Designer widget to create new report templates or modify an existing one. You can also add more alert types, change alert subscriptions and more, using the Alert Manager widget.

Figure 5. The Tools page
The Tools page

(See a larger version of Figure 5.)


Enabling event monitoring

Now that we've described the sample MFT dashboard, let's learn how to deploy it and try it out in your own environment. To that end, it helps to provide a high-level background on how things work when using Business Monitor.

IBM Business Monitor is an event-based processing system. It receives and processes events as they arrive from external systems that are being monitored. The processing of these events, that is, consuming and manipulating the data contained in the events, may vary from system to system depending on specific visibility requirements. The mechanism for injecting specific processing logic is done by writing a monitor model for the system being monitored. For more information about monitor models, refer to IBM Business Monitor V7.5 Information Center.

With a monitor model in place, you can create KPIs, alerts, instances and others based on what the monitor model makes available. The source for the two sample monitor models (developed for processing Connect:Direct and Sterling File Gateway events respectively) is included for download with this article.

If you are familiar with monitor models, you can browse through the source to see the different metrics defined and the modeled KPIs using the Business Monitor Development Toolkit. You will also find the dimensional models used for the report widgets. For example, to show Arrived Files by Producer, a dimensional model was created with an Arrived Files measure and a dimension level of Producer.

Business Monitor receives an event and then passes it to the appropriate monitor model, which in turn processes it and stores the processed information in a database. The information that you see on the dashboard is taken out of this database.

A monitor model needs to know the definition of all events that it will be processing so that it knows exactly how to access data from an event depending on which event type it is. These definitions are described in an event definition file in XSD (XML Schema Definition) format. The two monitor models that we built share the same event definition file.

By the same token, the monitored system must emit events in the format that its corresponding monitor model expects. Hence, the monitored system must also know these event definitions using the same event definition file. This is how a common event data protocol is established.

Sterling Control Center is the system that emits events to Business Monitor. Although the monitored systems are actually Connect:Direct and Sterling File Gateway, the collection of the visibility data is done by Sterling Control Center, which then creates the events for emission to Business Monitor. Hence, Sterling Control Center serves as an intermediary. Figure 6 shows the event flow from the monitored systems to Business Monitor.

Figure 6. Event flow between Sterling Control Center and Business Monitor
Event flow between Sterling Control Center and Business Monitor

Notice that from Sterling Control Center, the events are transmitted to Business Monitor through JMS with IBM WebSphere® MQ as the message service provider. Sterling Control Center does not currently support the embedded messaging provider in Business Monitor. If you don't have IBM WebSphere MQ, you can use ActiveMQ, which is supported by Sterling Control Center. However, we have only tried our sample with WebSphere MQ, so our instructions will not contain anything specific to ActiveMQ.

For the sample dashboard to work, you will need to enable the emission of events on the Sterling Control Center side and the monitoring and processing of events on the Business Monitor side. The next section provides an overview of the steps for setting up and configuring the sample in your own environment. The detailed steps are provided in the downloadable Readme.PDF file.


Set up and configure the dashboard

The main products to be set up and configured for the dashboard are Sterling Control Center, Business Monitor and WebSphere MQ (as mentioned, ActiveMQ is a possible alternative). All the required resources for sample dashboard can be downloaded from the Downloads section of this article.

At a high level, you need to complete the following steps. The ReadMe.pdf file included in the download contains detailed instructions for these steps.

  • Install IBM Sterling Control Center V5.3 and its most recent maintenance release.
  • Install and configure IBM WebSphere MQ V7.0 or later.
  • Configure IBM Sterling Control Center to emit events to IBM WebSphere MQ.
  • Install IBM Business Monitor V7.5 or later.
  • Configure JMS resources on IBM Business Monitor to receive events.
  • Download mft_resources.zip and extract the following:
    • The monitor model application EAR files, IBM Sterling Connect DirectApplication.ear and IBM Sterling File GatewayApplication.ear
    • The sample dashboard file, IBM+Sterling+MFT.zip
    • The Cognos Reports files contained in CognosReports.zip
  • Deploy the EAR files into the Business Monitor server.
  • Import the sample dashboard into Business Space.
  • Deploy the Cognos reports associated with the sample dashboard.

View your data in the dashboard

Now that you have the sample dashboard set up and configured in your environment, it is time to view some data in your dashboard. Your Sterling Control Center should be already enabled to emit events to Business Monitor, so any activities in your Connect:Direct and/or Sterling File Gateway installations should be captured by Sterling Control Center. Therefore, your dashboard should automatically display your data as shown in our example. If there are no current activities, perform some test activities in your Connect:Direct or Sterling File Gateway or both. Initiate some processes that involve a couple of different protocols. Make sure that you also try creating scenarios with failures to test your KPIs and alerts. As you drive more and more activities, you should notice the dashboard accumulate more data especially in the Instances widget. Once you have enough data, you can also play with the Report Viewer widgets.

In the next section, we'll give you a brief overview of customizing your dashboard.

Customizing the sample dashboard

Now that you have seen the sample dashboard working in your own environment, you may want to customize it. In this section, we'll describe some of the basic customizations you can do. For more detailed information, refer to IBM Business Monitor V7.5 Information Center and the developerWorks series Business activity monitoring with IBM Business Monitor V7.5.

Here is just a short list of some of the basic things you can do:

  • Add and remove pages from your dashboarde.
  • Add and remove widget from a page.
  • Configure the layout of a page depending on how you want to arrange your widgets.
  • Change the labels for your pages and rearrange the order of the pages.
  • Change the title of your widgets.
  • Configure the settings of each widget. Every widget has its own set of settings that you can configure.
  • Change the theme of your dashboard, including changing the banner as well as the color scheme.
  • Update access permission to your dashboard.
  • Create new alerts and subscribe or unsubscribe to existing alerts.
  • Create new KPIs.
  • Design new report templates.
  • Customize your reports.

We'll discuss only a few of the items on this list just to give you a general idea of how customization is done.

To add another page in the sample dashboard, do the following:

  1. Click the "+" icon located next to the rightmost page, as shown in Figure 7.
    Figure 7. Click the + icon to add a new page
    Click the + icon to add a new page
  2. A pop-up dialog appears in which you can enter the name of the page and a brief description. Choose Create an empty page and click OK. You should see an empty page added to the end.
  3. To start configuring your page, you need to switch to the edit mode by clicking Edit Page, as shown in Figure 8. Note that you can do the same step for any existing pages.
    Figure 8. Click on Edit Page located in the upper right corner of the current page
    Click on Edit Page located in the upper right corner of the current page

    You know that you are in edit mode if you see the set of available widgets, as shown in Figure 9. You can hover on any widget icon to see a brief description of that widget on the right side.

    Figure 9. The set of available widgets
    The set of available widgets

    (See a larger version of Figure 9.)

    As mentioned, a dashboard is composed of a set of widgets which are implemented as iWidgets (see iWidget Overview). A widget is a self-contained UI element that implements a specific function. For example, you have seen the Alert widget, whose function is to show all alerts that you have subscribed to as they occur in the monitored system. You can read an alert or forward it to another user.

    We recommend that you consult the IBM Business Monitor V7.5 Information Center and the developerWorks series Business activity monitoring with IBM Business Monitor V7.5 learn more about these widgets.

  4. You can drag and drop your chosen widget icon into the page and place it in the desired position. A new page will have a default layout, but you can change it by clicking the Change Layout icon located near the Save button, as shown in Figure 10.

    Note the different layout options. You can change the layout even for existing pages. Once you have dragged and dropped the widgets you are interested in, click Save. Remember that you need to configure the widget by going to the widget settings, before you can use it and see some data.

    Once you've configured your widgets, click Finish Editing.

    Figure 10. Change the layout of your page
    Change the layout of your page

Most widgets offer a few settings that you can configure while in the edit mode of the current page. For example, you can change the settings of the KPI widget in the Sterling File Gateway Current Activity page. To do this, do the following:

  1. Switch to edit mode as described above, then go to the KPI widget entitled File Transfer Status (Last Hour) and click the Settings icon located in the upper right corner of the widget, as shown in Figure 11.
  2. Notice that you can change the title of the widget by selecting Rename. If don't want to show a border around the widget, click Choose Skin and then No Border. The menu also allows you to change the position of the widget on the page. If you don't want to make this widget visible, choose Hide. This may sound odd, but there are situations in which hiding the widget is the right option.
    Figure 11. Step to change settings for a widget
    Step to change settings for a widget
  3. Choose Edit Settings to change the available widget-specific settings. Figure 12 shows the settings that are available for the KPI widget. The first tab lists the available KPIs to choose from. The selected KPIs will appear in the widget. As you can see, the three KPIs you have seen before are currently selected. Try to deselect one of them and you'll see that only two KPIs will appear after you click Apply and exit the edit mode.
    Figure 12. The KPIs tab for the KPI Widget Settings
    The KPIs tab for the KPI Widget Settings
  4. The Layout tab shown in Figure 13 has to do with the appearance of the KPIs in the widget. The display type provides you option for how you want the KPIs to display. You can choose to show them in table format, as full gauges (the current selection), as half-gauges, as bars, and so on. You can choose a horizontal or vertical arrangement or a grid. You can also choose the size of the KPIs themselves. Finally, you can specify the refresh rate, which is the amount of time (in seconds) to wait before the current values of the KPIs are fetched and updated in the widget.
    The Layout tab for the KPI widget settings
    The Layout tab for the KPI widget settings

You now have a good idea of how to customize your dashboard. Remember that the most important things to find out are the specific functions of the different widgets and the available settings for each.


Conclusion

In this article, you've learned how you can improve the monitoring of file transfer activities with IBM Sterling Control Center V5.3 and IBM Business Monitor V7.5 using a more advanced set of tools or widgets in a dashboard. We have provided a sample dashboard that you can easily try out in your environment by following the steps in this article. Once you understand the sample dashboard, you should be on your way to creating your own dashboard that meets your needs. With IBM Business Monitor's extensibility, you can start with the sample dashboard and modify it by adding/removing widgets, customizing them, and so on.

If you're already familiar with Business Monitor and have developed monitor models before, you can also view the monitor models we provided with this article and modify them for your own purposes.

In the future, we hope to extend our work to include the remaining products in the MFT portfolio. We may also improve the design of the dashboard depending on the feedback that we receive.


Acknowledgments

The authors would like to acknowledge those who have been instrumental in the successful delivery of this work and this article. Special thanks to Steven Woodcock, Susan Roberts and Michael P. Etgen. We also like to thank Wayne Salter from IBM Cognos development, Eric Wayne and John Alcorn from IBM Business Monitor development, and Paul Coniglio and Adam Orentlicher from Sterling Commerce Integration team.


Downloads

DescriptionNameSize
Sample dashboard filesmft_resources.zip2.7MB
Sample dashboard filesReadMe.pdf487KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=760737
ArticleTitle=Real-time event processing and dashboard visualization for Sterling Commerce Managed File Transfer
publish-date=09292011