Making all the applications in an enterprise work in an integrated manner to provide unified and consistent data and functionality is a difficult task. It involves integrating various kinds of applications, such as custom-built applications (C++, Java™, or Java 2 Platform, Enterprise Edition [J2EE]), packaged applications (such as SAP CRM applications), and legacy applications (mainframe IBM® CICS® or IBM Information Management System [IMS™]). Furthermore, these applications might be dispersed geographically and run on varied platforms. There might be a need for integrating applications that are outside the enterprise.
As enterprises grow, and as the complexity involved in the enterprise integration has increased progressively with time, a number of integration patterns have also evolved over time. Thus, today a substantial number of integration patterns exist. These integration patterns vary from the simple file-based data transfer between applications to the complete SOA-based integration patterns.
Parts 1 and 2 of this series trace the evolution of these patterns with a view towards helping you understand all the basic concepts and features involved in SOA-based integration patterns. Some of the concepts and features covered include:
- Consumer and provider of services
- Loose coupling
- Code reuse and layering
- Language independence
- Platform independence
- Definition, publication, and discovery of services interface (that is, registry concepts)
- Evolution of the enterprise service bus (ESB) from point-to-point integration involving the concepts of connectivity, marshaling, and mediation
- Coarse granularity
This description of the evolution of integration patterns also emphasizes the important point that many of the preceding technologies and technical concepts have significantly contributed to the development of SOA-based integration patterns.
Another reason to discuss the older patterns is that even in today's world, a combination of older and newer patterns coexist. For example, many ESB implementations support a file-transfer mechanism for integration. In a similar manner, many application servers, such as IBM WebSphere®, have both the capabilities of an Object Request Broker (ORB) and asynchronous messaging.
It's interesting to note that recently an IBM proposal to standardize a Service Integration Maturity Model (SIMM) was accepted by the board of The Open Group. The description of earlier integration patterns in Parts 1 and 2 of this series can greatly help in determining and clarifying maturity levels of the application domain in SIMM.
Of all the concepts, the major driving principle of the march towards SOA-type integration patterns is the idea of loose coupling. This drive for loose coupling has occurred because the number and kinds of applications being integrated has progressively grown very large. This requires integration patterns to minimize the effect on other applications due to changes made to one application. Another business reason for requiring loose coupling is that businesses require agility to meet today's changing business requirements. So the integration pattern must allow for this agility and be flexible. As you'll see in this article, while the path towards loose coupling and agility hasn't been a straight one, in general as we moved from older patterns toward SOA, the couplings between applications and software components have become weaker.
The second most important principle that's been employed in the development of a services-based architecture is the emphasis on maximizing code reuse. This code reuse results in more reliable and efficient code, because the same code is tested again and again. To implement code reuse, usually layering (also known as componentization) is required. Layering, or componentization, refers to pulling out different pieces of the code as separate software components so that at run time multiple applications can use the same code to, for example, make a network connection to remote applications. Layering also promotes loose coupling, because the internal workings of each layer can be changed without affecting other layers or applications.
Let's start the discussion about integration patterns with the simplest integration involving only data sharing between the applications. This will help ease you into the concept of connectivity between different applications.
There are, in principle, three ways to share data between two applications:
The first one, perhaps the most common, is through files. This is because storing data in files is the universal way to store data in a system. In this pattern of integration one application writes data to a file, while another application reads data from the same file. However, there are two major problems with this approach of sharing data between applications:
- Data isn't shared in real time, as there's usually a lag (mainly dependent on business cycles) between writing and reading from the files.
- There's tight coupling between the two applications sharing the data. Thus, changes (which modify the format or the content of the file) in the application producing the file must be accompanied by the changes in the application that consume the file.
The second pattern of sharing data is similar to the first and uses a common database. In this case, one application writes data to a database, while the other applications read from the database. Again, the data isn't shared in real time, as there's usually a lag between writing to the database by one application and reading of the data by other applications. This lag between writing and reading of the data is due to the fact that the reading application isn't aware of when the data is written by the writing application. Also, there's tight coupling between applications, because changes made to the database have a ripple effect, which makes the design of the shared database difficult.
To avoid the problem of stale data, you need real-time connection between applications sharing data. This is termed connectivity. The most rudimentary way to establish a connection between two applications is through sockets. Sockets let one application listen at a given port on a given machine for the incoming data, while another application can write to the same socket using the IP address and port address of the first application. The listening application can read the data as soon as the second application writes the data. So the data is shared in real time, and the problem of data staleness is eliminated.
Because the overhead associated with applications that communicate through sockets is very low, direct socket programming leads to efficient communication. It's also interesting that most of the modern methods of communication, such as message-oriented middleware (MOM) and messages under the hood, rely on socket programming. However, there are a number of shortcomings of the socket programming approach:
- The major problem with socket programming is that only data can be shared directly, not the functionality.
- The API for socket programming is rather low level and, therefore, difficult to use.
- Because the API is low level, socket programming isn't suitable for dealing with complex data types.
- The connectivity code is buried in the applications, so it can't be reused.
- Socket programming is also not platform independent, because applications on both ends must explicitly account for the byte ordering differences (little endian versus big endian) on different platforms, such as mainframe and UNIX®.
- There's also tight coupling between the two applications, because socket connection is point to point.
Thus, we need a solution that lets applications share data and functionality, and avoid low-level network programming. RPC was the first approach that allowed applications to share functionality, which is covered next.
The introduction of RPC was an important step, as it introduced some important concepts and features and specified the basic steps that are necessary for sharing functionality. Recall that services are basically about sharing functionality between applications and between components. Thus, it's instructive to discuss this integration pattern in some details.
RPC is also known as a client/server model and is a rung above socket programming. It eliminates the need for network programming. Briefly, RPC provides a function-oriented interface. The developer defines a function, much like those in functional languages like C, and generates code that makes the function look like a normal function to the caller. RPC is powerful enough to be the basis of client/server applications.
The first valuable concept that was introduced by RPC was that of a service provider, called a server, and a service consumer, called a client. In RPC, a server application provides a function that can be called in a normal manner by a consumer (client) application. The basic sequence is that a server (application) is started, which waits for the client to make a request. When the server receives the request from the client, the server executes the local function and returns the return value for the function to the client. The complete process is shown in Figure 1.
Figure 1. Remote Procedure Call complete process
One of the import components that's introduced in Figure 1 is the client stub. To the client, the client stub appears to be the actual procedure that it calls. The purpose of the stub is to package the arguments to the remote procedure, possibly put them into a standard format, and then build one or more network messages. This packaging of arguments is called marshaling. An important aspect of this marshaling is that the byte ordering differences between different platforms are handled automatically using a standard called external data representation (XDR), thus making RPC platform independent.
The second important component that RPC introduced is the RPC runtime library. The client stub uses the functions provided in the RPC runtime library to make a systems call into the local kernel to send the packaged message over the network to the server machine using a protocol, such as Transmission Control Protocol (TCP). In other words, RPC run time encapsulates all the systems calls necessary for the connectivity—that is, to send the packaged arguments over the network. Thus, the programmer doesn't need to know any systems programming.
On the server machine side, as the network message is received by the network routines in the kernel, it's sent to the server stub by the use of RPC run time. The server stub unmarshals the input parameters and invokes the requested local procedure in the server routines. After the local procedure is completed, the server stub marshals the return value into one or more network messages and sends the packaged return value to the server kernel through the use of RPC run time. The server kernel sends the message to the client machines using the network protocol, such as TCP. The client stub reads the network messages from the kernel through the use of RPC runtime routines. After possibly converting the return values, the client stub finally returns to the client function. This step appears to be a normal procedure return to the client.
RPC also introduced a rudimentary way of defining an interface between the client and the server through the use of a specification file. The RPC specification file may be considered the first step in the development of the services interface in today's world. An example of such a configuration file (square.x) is provided in Listing 1. This file is used to generate the skeleton code for both the server and the client using a tool like rpcgen. Note that the specification is specific to a language such as C and, therefore, requires both the server and the client to be written in the same language.
Listing 1. An example of RPC specification file, which defines a remote procedure for calculating the square of a number
struct square_in { /* input (argument) */
long arg1;
};
struct square_out { /* output (result) */
long result;
};
program SQUARE_PROG {
version SQUARE_VERS {
square_out SQUAREPROC (square_in) = 12; /*procedure number*/
} = 23; /* version number */
} = 0x31231234; /* program number */
|
To summarize what RPC is, it's important to note that RPC allowed for the first instance of real distributed computing by letting applications share functionality. In the process, a number of new concepts were introduced, including:
- Service provider (server) and the service consumer (client).
- Platform independence.
- Concept of interface definition.
- Marshaling of input and output parameters.
- Encapsulation of the systems calls in a library that are necessary to communicate over the network.
However, RPC has a number of shortcomings, including:
- The major disadvantage is that there's little room for code reuse. This is because the code for marshaling and unmarshaling, and the code for network communication is buried in the client and server applications.
- RPC isn't language independent, and the client and the server must employ the same programming language.
- There's also very tight coupling between the applications. The reason is that because the calls are synchronous, the client application must wait for the server to complete the procedure before it can proceed further. Also, it's possible to overcome this problem by using multithreading; however, it introduces another level of programming complexity that has inherent risks in terms of garbage collection.
- The integration of the client and server is point to point and, therefore, isn't suitable when a number of applications need to be integrated.
- RPC is also not suitable if a large number of remote calls are involved. This is because of the synchronous nature of the call, which doesn't allow the client to proceed before the server completes its work .
This article described two of the earliest integration patterns. The first one, socket programming, is suitable for sharing data in real time, while the second one, RPC, is suitable for sharing functionality. You learned the concepts of connectivity, service provider and service consumer, and platform independence.
To improve on RPC, two paths have been taken. The first is the method of distributed objects (also known as Object Request Broker), and the second method is that of asynchronous messaging. The distributed objects approach focuses on code reuse and language independence, while asynchronous messaging addresses the problem of tight coupling between applications. In Part 2 of this series, you'll first learn about the distributed objects approach, as it's more closely aligned with RPC. It's perhaps important to point out that today most of the application servers are based on ORB technology.
Learn
- Learn more about SOA architecture in the book
Enterprise SOA: Service-oriented Architecture Best Practices.
- Check out
UNIX Network Programming, Volume 2: Interprocess Communications.
- Read more about Socket programming in
UNIX
Network Programming, Volume 1: The Sockets Networking API.
- Check out Patterns of Enterprise Application Architecture, an excellent resource for integration patterns.
- Learn more about SIMM.
- The SOA and Web services zone on IBM developerWorks hosts hundreds of informative articles and introductory, intermediate, and advanced tutorials on how to develop Web services
applications.
- Play in the IBM SOA Sandbox! Increase your SOA skills through practical, hands-on experience with the IBM SOA entry points.
- The IBM SOA Web site offers an overview of SOA and how IBM can help you get there.
- Stay current with developerWorks technical events and webcasts.
- Browse for books on these and other technical topics at the
Safari bookstore.
- Check out a quick Web services on demand demo.
- Get an RSS feed for this series. (Find out more about RSS.)
Get products and technologies
- Innovate your next development project with
IBM trial software, available for download or on DVD.
Discuss
- Participate in the discussion forum.
- Get involved in the developerWorks community
by participating in developerWorks blogs, including the following SOA
and Web services-related blogs:
- Service Oriented Architecture -- Off the Record with Sandy Carter
- Best Practices in Service-Oriented Architecture with Ali Arsanjani
- WebSphere SOA and J2EE in Practice with Bobby Woolf
- Building SOA applications with patterns with Dr. Eoin Lane
- Client Insights, Concerns and Perspectives on SOA with Kerrie Holley
- Service-Oriented Architecture and Business-Level Tooling with Simon Johnston
- SOA, ESB and Beyond with Sanjay Bose
Dr. Waseem Roshen is an IT architect in the Enterprise Architecture and Technology Center of Excellence of IBM Global Business Services in Columbus, Ohio. He works on enterprise architecture and integration. He's also a Sun Certified J2EE Architect, has published 60 articles, and has worked on 24 patents.
Table of contents
Next steps from IBM
Rational Tester for SOA Quality is an automated web services testing and SOA testing tool that supports Web services SOAP, HTTP(S), JMS, WS-Security, UDDI, and RESTful standards.
- Try: The no-charge Rational Service Tester for SOA Quality helps quality assurance professionals validate web service based SOA application..
- Article: See the whats new with Rational Service Tester for SOA Quality, and its Generic Service Client, which provides a single client to interact with any SOA service type.
- Tutorial: Get hands-on experience with this Transport for SOAP feature tutorial using Rational Tester for SOA Quality.
- Buy: Rational Service Tester for SOA Quality
