GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 2: Building SOA applications with reusable integration services

Part 2 provides an overview of the WSN specification and its implementation in WebSphere® Application Server V6.1, and describes how it has been used in a GPASS implementation to solve the SAP Ledger scenario introduced in Part 1.

Akram Bou-Ghannam, Ph.D., Executive IT Architect (Senior Certified), IBM

Akram Bou-Ghannam photoDr. Bou-Ghannam is an Executive IT Architect (Senior Certified) leading efforts to transform IBM into a globally integrated on-demand enterprise through SOA and dynamically reconfigurable services and architectures. His expertise in complex event processing and intelligent agents is helping IBM to realize the smart and agile enterprise – an enterprise that is capable of real-time discovery and response to actionable business situations. His SOA enterprise information architecture extends SOA into the data layer through the provision of information services, and helps achieve the enterprise vision of becoming information-centric in an SOA world. Dr. Bou-Ghannam is an IBM Master Inventor, and an accomplished author with multiple internal and external publications to his name.

You can reach Dr. Bou-Ghannam at akram@us.ibm.com.



Matt Roberts (matt.roberts@uk.ibm.com), WS-Notification Architect and Development Lead, IBM

Matt RobertsMatt Roberts is the architect and development team lead for support of WS-Notification in WebSphere Application Server. He is an active member of the OASIS WS-Notification technical committee responsible for producing the WS-Notification specifications, and works both inside and outside IBM to promote adoption of WS-Notification. Matt has worked on the Service Integration Bus development team for the past five years, with responsibility for design and implementation of a range of components including the Publish/Subscribe Bridge (for pub/sub interoperability between SIB and WebSphere Businesss Integration Message Broker), Java™ Message Service (JMS), and topology and routing management. Before joining the SIB team, Matt worked in the WebSphere MQ JMS development team. You can contact Matt at matt.roberts@uk.ibm.com.



Allandel Manipon (allandel@us.ibm.com), Enterprise Service Hub Application Architect, IBM

Allandel Manipon photoAllandel Manipon works in IBM Global Application Management Systems Delivery. He is an IT Application Architect and.Development Team Lead for Enterprise Services Hub (ESH). Allandel is currently working on the SOA transformation in EBI, focusing on creating a reusable publish and subscribe application, business application integration, and SOA Web services. Allandel joined IBM in 2000. His areas of interest and expertise include SOA, IT architecture and methodology, WebSphere Application Server, WebSphere Process Server, WebSphere MQ, WebSphere Message Broker and DataStage Transformation Extender. You can reach Allandel at allandel@us.ibm.com.



05 December 2007

Introduction

In Part 1 of this series, we introduced Generalized Publish and Subscribe Services (hereafter called GPASS) – a framework that extends an enterprise service bus (hereafter called ESB) engine with features that greatly simplify data synchronization, and helps to dramatically cut the cost of information distribution in an enterprise, enabling enterprises to implement new business requirements quicker and cheaper. We described the functions that GPASS provides and gave a brief introduction to the way in which those functions are accessed within the open standards infrastructure provided by the OASIS WS-Notification standard.

In Part 2 of the series, we look in more detail at how GPASS uses WS-Notification, before describing how WS-Notification is positioned in WebSphere Application Server (hereafter called Application Server) as a mechanism for communicating with other aspects of your enterprise service bus. Finally we look at a detailed example showing GPASS in action as part of the internal IBM enterprise systems.

WS-Notification specification overview

WS-Notification is made up of three specification documents; WS-BaseNotification, WS-BrokeredNotification and WS-Topics, which can be downloaded free from the OASIS WS-Notification Technical Committee Web site.

Many other articles exist describing the terminology and concepts defined by the WS-Notification standard and we refer the curious reader to those resources, such as the IBM Systems Journal article on WS-Notification (for more information see Resources). However, it's useful to describe here at a high level the parts of the specifications that are of specific interest in the GPASS solution.

WS-BaseNotification

The WS-BaseNotification specification defines the basic application entities that participate in WS-Notification interactions. Figure 1 shows how the five primary entities work together to pass data through the system.

Figure 1. WS-Notification entities
Figure 1. WS-Notification entities

Initially, the Subscriber is responsible for setting up a subscription between the NotificationProducer Web service and a NotificationConsumer Web service. This subscription is managed by the SubscriptionManager Web service working on behalf of the producer.

Subsequently, when a Situation is observed by the Publisher, the Publisher creates a notification message and passes it to the NotificationProducer. It is the responsibility of the producer to establish whether the notification message matches the subscription that has been registered, and, if so, to send the notification message to the consumer.

When writing WS-Notification applications, it's important to consider whether or not the entity the application is implementing is required to expose a Web service endpoint. Entities that are invoked by external stimulus must by necessity expose an endpoint. For example, each of the three entities in rectangular boxes in Figure 1 expose one or more operations that can be invoked by other applications (for example Subscribe on a NotificationProducer for creating subscriptions, or Notify on a NotificationConsumer for receiving notifications). Other entities, such as the Subscriber, are responsible only for driving those services that do expose an endpoint, such as to create a subscription.

Creating and managing a subscription

The most important aspect of the WS-Notification model is that of creating a subscription. It is this act that causes notification messages to flow between the producer and the consumer, and there are many options available that control the exact match criteria and delivery style to ensure that the consumer receives the most appropriate notifications.

In this section, we examine the Subscribe operation that is exposed by a NotificationProducer (and NotificationBroker as described below) Web service endpoint and highlight those parts that are of particular interest in GPASS.

For a Web service to act as a NotificationProducer it must support the Subscribe message exchange – that is to say that the WSDL for the Web service must include in its portType definition an operation that contains the subscribe request and response messages defined by the WS-Base Notification specification. By implementing the Subscribe exchange, the producer service is required to send a notification to each notification consumer that has a subscription registered whenever the producer has a message to send and the filter conditions expressed in the subscription are satisfied.

There are various options for controlling the set of notification messages and the location to which they are sent, as shown in the pseudo-schema for the Subscribe request in Listing 1:

Listing 1. Subscribe request pseudo-schema (reproduced from the WS-BaseNotification specification)
<wsnt:Subscribe> 
  <wsnt:ConsumerReference> 
    wsa:EndpointReferenceType 
  </wsnt:ConsumerReference> 
  <wsnt:Filter> 
    [ <wsnt:TopicExpression Dialect="xsd:anyURI"> 
         {any} ? 
      </wsnt:TopicExpression> | 
      <wsnt:ProducerProperties Dialect="xsd:anyURI"> 
         {any} ? 
      </wsnt:ProducerProperties> | 
      <wsnt:MessageContent Dialect="xsd:anyURI"> 
         {any} ? 
      </wsnt:MessageContent> | 
         {any} * 
    ] * 
  </wsnt:Filter> ? 
  <wsnt:InitialTerminationTime> 
    [xsd:dateTime | xsd:duration] 
  </wsnt:InitialTerminationTime> ? 
  <wsnt:SubscriptionPolicy> 
    [ <wsnt:UseRaw/> | 
         {any} 
    ] * 
  </wsnt:SubscriptionPolicy> ? 
        {any}* 
</wsnt:Subscribe>

Three of the elements of this request message are of particular interest in the GPASS environment:

  • The ConsumerReference is a WS-Addressing endpoint reference (EPR) that identifies the location of the NotificationConsumer service to which matching notification messages will be sent by the producer. In GPASS this address often points to an Information Consumer Proxy Service that acts on behalf of the eventual information consumer to implement the GPASS services, such as scheduling, transformation and distribution.
  • The Filter element, and in particular the TopicExpression filter, is used to determine the specific subset of all available notification messages that should match this new subscription. This is important because the majority of notifications passing through the GPASS system are of interest only to a small number of consumers, so we improve the efficiency of the system by avoiding sending unwanted notifications. The other types of filters (ProducerProperties and MessageContent) provide additional specification-defined mechanisms for filtering the messages that match the subscription. Each application that uses GPASS defines its own topic structure that enables subscribers to define the necessary filter expressions to receive exactly the subset of notifications they're interested in.
  • The SubscriptionPolicy element provides an extensibility mechanism to allow the Subscriber to indicate additional constraints that are not formally defined by the WS-Notification specification. In GPASS these include details of the delivery schedule or transformation requirements as described in Part 1. The GPASS service receiving the subscription request processes the GPASS elements from the subscription policy, resulting in the creation of an information consumer proxy service, before passing the remainder of the request to the NotificationBroker provided by the application server.

In answer to a Subscribe request, the NotificationProducer returns a response message providing details of the new Subscription that has been created, including a reference (WS-Addressing EndpointReference) to the subscription object that is managed by the associated SubscriptionManager. This subscription reference is used subsequently to manage the lifetime of the subscription; for example, to delete or renew the subscription.

The interaction diagram in Figure 2 illustrates the Subscribe/Notify message flows.

Figure 2. Initiating the flow of notifications
Figure 2. Initiating the flow of notifications

WS-Topics

The WS-Topics specification defines a mechanism by which notification messages can be classified in order to indicate the type of situation for which a notification was created. This is achieved by associating each notification with a topic, and means that subscribers can define the specific category of event that they are interested in hearing about.

Topics provide a coarse-grained filtering mechanism that allows large sets of uninteresting notifications to be excluded quickly. For example, in a sports results scenario a subscriber can indicate that he or she is only interested in receiving notifications about football, which excludes any events about baseball or hockey. More fine-grained control of filtering can be achieved using other filtering mechanisms, such as the message content filter defined in WS-BaseNotification. For example, by selecting only those football games in which the home team beat the away team. In many situations, the topic does not actually appear in the body of the notification message itself since the classification of the notification is made at a higher level than the generation of the notification content.

Topics also provide a way for a producer to advertise the type of notifications that it's likely to produce, and are also often used as the basis for an access control scheme.

The GPASS system uses only the "Full" topic dialect defined by WS-Topics. This topic dialect defines an easily understandable subset of the XPath that provides a straightforward mechanism for defining hierarchical topic structures (suitable for classifying the complex events passing through the GPASS environment), as well as useful wildcarding functions to enable a group of related topics to be received by a consumer using a single subscription.

The highlights of the Full topic expression dialect are:

  • Ability to define root level topics using a simple QName; for example, "sports"
  • Ability to create hierarchical topic structures using the "/" separator to denote descendants; for example, "sports/football"
  • Ability to define wildcard expressions that enable a consumer to receive a whole class of event rather than a topic; for example "sports//." receives all event published under the root "sports" topic.
  • Ability to combine two or more topics as part of a single subscription using the conjunction operator "|". For example "sports/football | sports/baseball" enables a consumer to receive events on those two sports using a single subscription without receiving events relating to "hockey".

These facilities, along with the topic structure defined for each application that uses GPASS, enable the system to support a wide range of subscription patterns including the following:

  • All events related to a specific geography
  • All events related to a specific customer
  • All events related to a specific order
  • All new order events for any customer in a specified geography

WS-BrokeredNotification

The WS-BrokeredNotification specification extends the definitions made in the WS-BaseNotification specification to define the concept of a NotificationBroker, which is an intermediary service to which producers and consumers can connect in order to pass notifications.

Critically, the NotificationBroker is capable of accepting subscription requests from consumers, as well as receiving notification messages from producers. The broker is then responsible for matching the notifications with the subscriptions and sending them to the consumer. In this way, the broker takes on some of the more painstaking functions of the producer, freeing developers of producer applications to concentrate on the business task of observing situations and generating the appropriate notifications without having to worry about the challenging but mechanical task of managing subscriptions and matching them to notifications.

Advantages of the brokered notification pattern are as follows:

  • Relieves the publisher of having to implement message exchanges associated with the notification producer; for example, managing subscriptions (SubscriptionManager) and distributing notifications (NotificationProducer)
  • Avoids the need for synchronous communications between the producer and the consumer
  • Can reduce the number of inter-service connections and references
  • Acts as a finder service; for example, if a new publisher is added that publishes notification x, a consumer does not have to issue a new subscription if it is already subscribed to the broker with x
  • Provides anonymous notification, which means that publishers and consumers need not be aware of each other’s identity

In many scenarios, the NotificationBroker service is implemented by a middleware provider, ensuring that the brokering facilities are written to enterprise quality expectations and often providing additional value-add services over and above the basic definition of the service, for example logging, transformation, or quality of service enhancements above those required by the specification.

In the GPASS solution, the NotificationBroker service is provided by Application Server as described in the next section. GPASS provides a façade NotificationBroker service (labeled "GPASS service" in Figure 5), which intercepts elements of the subscription requests that are specific to GPASS and delegates their handling to an Information Consumer Proxy Service. GPASS sends the remainder of the subscription request, containing only the basic WS-Notification constraints, to the Application Server NotificationBroker, where it is managed by the server.

WS-Notification in Application Server V6.1

Application Server V6.1 provides a feature that enables a server instance to be configured to expose a NotificationBroker Web service endpoint. Applications are able to connect to this Web service endpoint using the mechanisms defined by the WS-Notification standard in order to send and receive notification events. In particular, since the message exchanges are defined by the specification, applications connecting to the broker can be running in any environment or written in any programming language – not only in WebSphere applications but also in other Web service frameworks, such as Apache or .NET, and including applications written in languages like C++, C# or PHP. This means that the server NotificationBroker is able to sit at the center of a heterogeneous IT environment.

The concept of a NotificationBroker is defined in the WS-Notification specification, and anyone can write a Web service implementation that exposes a NotificationBroker port. However, by using Application Server, the GPASS solution avoids the need to to concentrate on implementing the many and various requirements of the specification by taking advantage of a out-of-the-box implementation, which has already been tested to ensure its correct functioning and stability.

There are other benefits beyond that of specification implementation that come from using the server NotificationBroker, including seamless integration with Application Server clustering and high availability infrastructure and mechanisms for applying enterprise-level security requirements. You can derive further benefit from the way that the Application Server NotificationBroker is implemented, using the WebSphere service integration bus (SIBus) as the underlying technology by which notifications are passed between producer and consumer. This means that not only is the broker built upon established technology, but it can be used as an on-ramp to the SIBus from which notifications can flow to other non-WSN applications such as JMS, or to other parts of IBM's ESB fabric including WebSphere ESB, WebSphere Process Server or WebSphere Message Broker.

Figure 3 gives you a high-level look at how the Application Server NotificationBroker sits on top of the service integration bus.

Figure 3. WS-Notification in Application Server V6.1
Figure 3. WS-Notification in Application Server V6.1

Notify operations (originating from a Publisher or NotificationProducer application) result in publication of a message to the SIBus topic space associated with the WSN topic namespace requested by the client application. Subscribe requests result in creation of a durable subscription on a topic/selector. Messages delivered to the SIBus durable subscription are asynchronously converted into <notify> messages and invoked on the WSN NotificationConsumer endpoint.

For more information on the WS-Notification facilities provided by Application Server, as well as code samples, see Resources.

Using GPASS to implement the SAP Ledger scenario: A real-world example

This section describes the SAP Ledger scenario, a real-world example from the IBM enterprise. We'll show how to implement the SAP Ledger scenario using GPASS.

Overview of the SAP Ledger scenario

The SAP Ledger project is part of a large IBM business initiative that aims to establish a SAP Enterprise Shared Services instance by integrating the following enterprise applications into a single SAP instance:

  • Ledger (discussed here)
  • Fixed Assets
  • General Procurement
  • Enterprise SAP, or ESAP

In the case of the Ledger scenario, line of business systems such as ERP, CRM or SCM applications wish to subscribe to receive notification of updates, additions or deletions for offerings (products) or customer profiles in which they have a particular interest. The data to satisfy these information requests is provided by a combination of two master data sources; the Customer Information (CI) and Offering Information (OI) systems as shown in Figure 4:

Figure 4. Simplified system context diagram of the SAP Ledger scenario
Figure 4. Simplified system context diagram of the SAP Ledger scenario

The EBI (Enterprise Business Information) Service Hub can provide information on any and all customers and offerings, but the LOB application is usually only interested in receiving notifications on a specific subset of customers or offerings at any given time – for example, those customers with whom they are currently working, or offerings that they use. In order to ensure that the LOB application is not flooded by information that is not relevant to them, the LOB application is able to specify a set of filter criteria that limit the information that is returned.

Additionally, in most real-world scenarios, the consumers of information expect the data they receive to be formatted differently than the format of the systems that provide the input data. The Service Hub must thus be capable of recording what format of data the consumer is interested in receiving, and then transforming the input data format into the output data format as the event flows through the system. In many cases this transformation can take a number of steps. Given a potentially large number of input and output formats it is often uneconomical to write converters from every input format to every output format, and instead may be necessary to convert from input format A to some neutral format X before applying the converter than turns X into output format B.

Figure 5 shows the GPASS architectural model from Part 1.

Figure 5. GPASS architectural model
Figure 5. GPASS architectural model

In the SAP Ledger scenario, the role of the Information Producer is fulfilled by the Customer Information and Offering Information systems, which are connected to the NotificationBroker service by a publisher (see step 4 in Figure 5). Events that are output by these systems are thus written to the NotificationBroker service, where they can be consumed by other GPASS entities.

The Information Consumer is the LOB application that wishes to receive event updates. The application does this by accessing the SAP Instance, which plays the role of the Adapter.

The role of the Subscriber is handled by a Web front-end to which users of the LOB application can connect in order to create or update the set of data in which they are interested. The Web front-end translates the interests expressed by the LOB application into subscription conditions on the notification topic hierarchy, and passes this data to the GPASS service as part of a Subscribe request (step 1 in Figure 5).

The basic elements of the subscription request are passed straight to the NotificationBroker (step 2), while the extended GPASS elements such as transformation and scheduling cause an Information Consumer Proxy Service to be created (step 3) that will receive notifications from the broker (step 5) before applying the value-add processing and delivering the resulting data to the Information Consumer Adapter (step 10).

Implementing the SAP Ledger Scenario solution with WS-Notification and Application Server V6.1

The GPASS system uses WS-Notification to handle the everyday aspects of the publish/subscribe pattern – for example, matching of incoming notification messages to active subscriptions. The open standard nature of WS-Notification means that GPASS is able to function on top of any standards-compliant implementation of WS-Notification.

In the case of the SAP Ledger scenario, we'll use the WS-Notification implementation provided by Application Server V6.1, described in the previous section. This solution has the advantage of being fully integrated with the Application Server infrastructure and capable of supporting the clustering and workload management that you would expect in a large-scale system.

Figure 6 shows in more detail the various elements within the system and how they interact together to achieve the overall goal of delivering notifications from the information producer to the information consumer while adhering to the additional business constraints supported by GPASS.

Figure 6. Conceptual diagram of the system interactions
Figure 6. Conceptual diagram of the system interactions

Let's trace the flow of information through the system from start to finish to show how the various components of the system interact.

First, the information that defines the particular interest profile of a given LOB application (consumer) is configured using a graphical user interface as shown in Figure 7. The user interface collects the information necessary to formulate the WS-Notification subscribe request – that is, the details of where the consumer application is located, and the topics of interest that should be delivered to that consumer.

Figure 7. The subscription management interface
Figure 7. The subscription management interface

The topic hierarchy defined by the SAP Ledger scenario and entered through the graphical interface provides a classification of notification events that enables the consumer application to filter the notifications to exclude those it is not interested in.

For example, in Figure 7 the administrator has configured a subscription topic that selects notifications on all materials (note the "//" character after "Material", which indicates that any number of levels can follow it) where the sales type is "IBMSaleable", the product type is "MD", there is no sales organization defined, and the notification relates to "ALL" partners:

esh:Material//SalesType/IBMSaleable/ProductType/MD/SalesOrg/NONE/SalesOrgEnd/Partner/ALL

The type of subscription made depends upon the business requirements of the LOB application, but the flexibility of the defined topic hierarchy means that it is capable of supporting a wide range of expressions; for example, all notifications of a given product type, all notifications relating to a specified partner, all notifications for a specific material type in a given sales type etc.

When the administrator has finished defining the subscription criteria in the user interface, he or she clicks Subscribe as shown in Figure 7. The information is then formed into a WS-Notification subscribe request that is invoked against the GPASS service shown in Figure 6.

The GPASS service extracts the advanced subscription information specified in the WS-Notification subscription policy element and uses it to configure an Information Consumer Proxy Service to handle the distribution and scheduling constraints. The remaining elements of the subscription request are then passed directly to the underlying Application Server NotificationBroker with the endpoint address of the new consumer proxy service, where they result in a subscription being created inside the application server. Notifications that match the subscription conditions (such as the topic on which they are published) are then delivered by the server to the nominated consumer proxy service, where they are processed according to the distribution and scheduling constraints, and subsequently delivered to the eventual consumer application in an appropriate format.

At a physical level, the services offered by the EBI Service Hub (marked ESH above) are split across two different types of server because of the different functionality provided on each. Figure 8 shows the way this split is implemented in the SAP Ledger scenario; an equivalent separation exists for other scenarios supported by the EBI Service Hub.

The WS-Notification aspects of GPASS are hosted on an Application Server V6.1 server, which exposes a NotificationBroker service to which the CI and OI publishers connect to publish notifications. The GPASS service also sits on the V6.1 server, where subscribers can connect to register the interests of the LOB application users. If appropriate, the GPASS service causes the creation of an Information Consumer Proxy Service, which runs on the second server.

The second server is an instance of WebSphere Process Server V6 (Process Server). Ideally, it wouldn't be necessary to use two different servers. However, since the various adapter components (such as the SAP Ledger adapter) are written using the SCA programming model and must, therefore, be run inside Process Server, which does't support WS-Notification, we need to use both products to fulfill the scenario requirements.

The recently announced WebSphere Process Server V6.1 is based on Application Server V6.1 and includes the WS-Notification support that GPASS requires. In the future, this topology will be condensed to use a single server both to expose the NotificationBroker endpoint and host the required adapter components.

Figure 8. Physical server view of selected components of the SAP Ledger scenario
Figure 8. Physical server view of selected components of the SAP Ledger scenario

It is also important to understand that the SAP Ledger scenario is not the only scenario supported by the EBI Service Hub in which GPASS operates. This scenario is just provided as a useful example of a real-world system that takes advantage of GPASS. Many other scenarios are supported by the Service Hub in order to support the necessary line of business application, as shown in Figure 9. Note the central role played by GPASS in supporting each of the applications using the EBI Service Hub.

Figure 9. SAP Ledger scenario system context diagram
Figure 9. SAP Ledger scenario system context diagram

Summary

In this two-part series of articles, you've learned about Generalized Publish and Subscribe Services (GPASS), which builds on the open standards WS-Notification implementation provided in Application Server V6.1. GPASS extends ESB connectivity to simplify data synchronization, thus helping you dramatically cut the cost of information distribution in your enterprise. We described the overall architecture of GPASS showing how it builds upon the basic functions provided by Application Server, as well as the value-add function implemented by GPASS to simplify common distribution, scheduling and transformation problems. We highlighted the areas of WS-Notification used by GPASS to provide this support before examining how GPASS can interact with other elements of your enterprise service bus through the SIBus. Finally, we looked in detail at a real-world example in which IBM uses GPASS to integrate LOB applications with the various necessary master data sources of the enterprise.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, SOA and web services
ArticleID=272755
ArticleTitle=GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 2: Building SOA applications with reusable integration services
publish-date=12052007