If your business is like most, your business strategy and core-competency analysis have identified where you need to focus on business re-engineering and IT modernisation. A typical outcome (at least for some of an enterprise's core aspects) is the transformation to a service-oriented business and its dependency on an SOA. Business operations are designed around a set of services that require current IT systems to be re-engineered and integrated as service providers. The transformation may also introduce new service providers. The goal is to be able to build new composite applications, such as internal workplace portal applications or Internet commerce applications, in an efficient way to address specific business needs.
As part of this challenging transformation, enterprises generally engage in a number of significant business and IT programmes. The primary transformation deliverables include:
- Alignment of the enterprise architecture around a business service model.
- Introduction of a service-based IT infrastructure that can realise the relevant parts of the business service model.
- Transformation and extension of, or applying a wrapper to, legacy IT systems so that they can be integrated into this service-based IT infrastructure.
- A robust design and development environment and delivery processes that enable efficient and flexible implementation of supporting IT applications.
These primary deliverables of the transformation create a secondary, consequential set of challenges to implement. These ensure that an enterprise delivers on its transformation intent and can monitor and adapt to change to achieve flexibility. These transformation challenges typically aim to adhere to the following principles:
- The enterprise must continue to provide an overall robust and compliant set of business operations.
- A governance process and design authority is needed to ensure that the overall business service model and IT system implementations adhere to the overarching transformation principles.
- The enterprise must achieve an integrated view of business and IT service management so that measurement and responsive change can be delivered.
Some enterprises are fortunate to be able to make the business transformation to SOA across the entire organisation. This approach presents a more challenging implementation for the primary transformation deliverables, but consequentially provides an easier environment for growing and developing the secondary transformation deliverables. This type of scenario usually occurs when C-level executives who want to drive business and IT change across the enterprise provide proactive sponsorship to aid their strategic agendas.
However, most enterprise transformations are incremental -- significant and strategic, but nonetheless incremental. This typically means that legacy IT systems are re-engineered so they can participate in support of new business operations -- for example, combine with other IT systems to facilitate straight-through processing of customer orders. These IT systems also continue to perform all, or least a large majority, of their current capabilities. Business operations outside the scope of the SOA transformation still continue.
In this transformation scenario, it's imperative that you consider careful implementation design for the SOA IT infrastructure. Data processing within this set of IT systems impacts and is impacted by data processing outside of that scope. The overall data-processing environment must continue to be robust and compliant for overall business operations.
The example in Figure 1 shows that three legacy systems have been engineered to be service providers.
Figure 1. Legacy IT in new SOA landscape
In Figure 1, the business-transformation programme has built an SOA infrastructure by which business services are published and can be consumed by new applications. This infrastructure typically includes middleware to enable multiple-protocol connections between software, asynchronous and synchronous message-flow management, message translation, rule execution, and process choreography. Composite applications have been built that use these services, and even legacy systems themselves have been enhanced by taking advantage of invoking new services.
Process and information architecture in an SOA
To implement an SOA infrastructure, the key asset that's produced, maintained, and enhanced is the business service model. Well-understood techniques, such as IBM® Service Oriented Modeling and Architecture (see Resources for more information), are used to develop that model. In a typical enterprise, this includes a mixture of top-down analyses, bottom-up analyses, and goal-service modeling. The transformation's business objective results in a process and information architecture that's used to drive implementations of new business operations and supporting IT applications:
- Process architecture is a functional hierarchy of actions that can be combined to form a set of business processes underlying enterprise operations. It's responsible for flow control of humans to systems and system to systems. For example, it might include the actions that need to happen for a customer to place an order for a product and be put into a satisfied state waiting for the delivery of that product.
- Information architecture is a set of related data and content that's accessed, manipulated, and stored to provide state and meaning to the enterprise operations. It performs the information management of data and content, whether in the context of a simple access event or process flow. An example is the data and content that's consumed and generated during a product-development life cycle (concept to manufacturing, and so on).
The process and information architecture viewpoints are often developed together and demonstrated through such things as a process-data (CRUD) matrix. Collectively, they define a number of services that compose the service architecture. A service is a set of operations called where message data is passed in. The service implementation performs some processing that manipulates some data or content, usually with persistent storage or access, and returns a result. Services can be used synchronously and asynchronously, and invocation is generally brokered by an integration infrastructure. In a completely decoupled system, some services are responsible for causing changes to process state (for example, information presented as requested, new address for customer X has been validated). Other services are responsible for causing changes to the information state, ensuring data and content have been manipulated into a desired state that future service events can rely on (for example, a new shipping address is in effect for customer X).
Legacy data-processing challenges
Legacy IT systems will undoubtedly be part of your enterprise's service-provider implementation and be integrated into the SOA infrastructure. Legacy includes not only traditional mainframe processing, but also enterprise resource planning (ERP) applications, internal and external Web applications, workflow applications, scanning and printing systems, and so on. A number of things can happen to enable these legacy systems to be used and combined into a service-provider implementation:
- The processing function is exposed and made accessible so that it can be combined into a service-provider implementation. This frequently means exposing a transaction via a new protocol and invocation mechanism, such as SOAP wrappers on CICS® transactions.
- Functions may need re-engineering to maintain processing integrity when exposing the transaction. The transaction might be something existing or created (facade) to call one or more other existing transactions. For these first two bullets, a service-granularity strategy should influence the design of the invocation call to these exposed services.
- Certain processing functions may need to be "switched" off because data processing is now carried out by the new functions being invoked through the service-provider implementation.
- Existing access rights as well as security and audit requirements must be enforced for the new access mechanisms.
- Quality of Service (QoS) criteria for existing systems may need to be extended or supplemented. For example, off-line batch-processing windows may need to be covered by stand-in processing solutions.
- SOA may introduce a new and unforeseen load on existing systems, which negatively impacts existing processes.
When the legacy IT systems are exposed in an effort to participate in the SOA, certain functions that support business operations outside the transformation's scope continue data processing. This typically requires understanding and dealing with data currency and synchronisation issues.
Users continue to invoke the legacy IT systems using the original application means, typically through a user interface (such as a mainframe dumb terminal or a packaged application's UI). The SOA depends on a set of data schemas that are used to describe the messages that flow around the SOA ecosystem. These data schemas are realised as physical data messages that are integrated within a collaborating set of service operations; this can include runtime data translation and semantic mapping, or augmentation of message data. This data is then used and processed by the underlying legacy IT system.
The set of processes that can be executed with the SOA relies on a collective set of data and content that needs to be in particular states. The parallel processing via SOA functions and legacy IT systems can cause instability in that state model.
In the example in Figure 1, the transformation has enabled a new Internet application that has straight-through processing capabilities to set up and administer insurance policies for customers. The new applications can trigger contact with the customer and offer a renewal deal when an insurance policy is coming close to renewal. However, some existing business operations outside of the SOA scope could cause problems. Customers have the means to notify the company by any number of routes that they have moved residence. Even though the policy was first purchased through the new Internet application, a customer may not use that SOA application to inform the company of an address change, perhaps sending a letter instead. A back-office process causes the customer's new address to be applied only to the customer relationship management (CRM) system. Overnight the existing CRM system's batch processing updates the Insurance Contract Administration system with customer data updates. Because the new SOA-based systems are designed around near-real-time transactions, data synchronisation timing issues now result. During that time, the new application could try to contact the customer about renewals at an old address, when in fact a responsive and context-aware SOA application should be contacting that person at the new address about cancelling the old policy and setting up a new one.
Hang on a minute, service provider
The SOA architect's role should theoretically be concerned with the design and implementation of a service-enabling infrastructure. SOA modeling techniques guide you to decide the services your process and information architecture will consume, contract with a service provider, and build applications that invoke service operations as part of process execution and data processing. The service consumer doesn't care how the service provider implements the service as long as it states it will process and manage data according to an agreed set of contracts (service operations and their associated QoS).
In certain scenarios this model may work, especially if you're sourcing service-provider implementations external to the enterprise. This article focuses on specific but common scenarios being exercised by many enterprises adopting SOA today.
The reality is that an overall solution architect in an SOA transformation has many responsibilities, especially in an enterprise that's making incremental transformations. Those responsibilities are carried out within the bounds of the enterprise governance process (see the article "A case for SOA governance" [developerWorks, August 2005] for more information).
First, the solution architect is concerned with designing and implementing the SOA infrastructure; this includes identifying the legacy IT systems that will act as software systems that form part of a service-provider implementation. Second, the solution architect provides the development processes and environment for assembling service-consuming applications. This includes design patterns and standards as well as tooling environments. But third, the solution architect must be cognisant of the overarching business and IT architecture responsibilities. The business architecture will state what is reasonable for the business strategy with its sourced capabilities and contracts. The IT architecture will deliver IT systems necessary to support the overall business operations. Within any reasonably sized to large enterprise, this is usually a large portfolio of tactical and strategic applications.
To build a robust information-management architecture that enables overall enterprise compliance, you can consider a number of design strategies and principles that address the issues.
Plan for scope changes in business process analysis
During a transformation programme, a common scenario is that system analysis and design discovers undesirable data-processing scenarios. Business operations that were previously out of scope might need to be brought into scope so that they can be modified to accommodate a new set of systems. The new SOA-based systems may even need to provide internal applications for these changed business operations to use, such as an extra processing step to update some customer details.
Publish and subscribe to legacy entity changes
When agreed entity and attribute data is changed in a legacy IT system that's participating as a service provider, extra processing needs to be developed to inform the SOA. Ideally this should be real time, but more often than not it's in the form of a scheduled batch activity. Applying instrumentation code to legacy IT systems that detects when data records have changed is likely to be costly, if not nearly impossible, in older and unstructured application code.
One thing your enterprise must do is assess the data owned by a legacy system and the system's capability to notify interested parties of changes. The high frequency of changes to data that's important to the SOA, combined with a legacy system's inability to generate record-level event triggers (so a daily batch extract might be best achievable), might lead to a decision to tackle re-engineering or replacing the legacy system. Make sure the business services' QoS is aligned with the planned engineering effort. Indicate how the QoS will improve over time, with the planned increment engineering improvements, such as moving from daily batch processing of key service data to real-time message-driven processing.
Information service query facade
Operations of the information-management services should present simple actions that can be applied to data or content. The query facade hides how the information is accessed and manipulated, and across how many physical IT systems. Basically, the query facade can implement two patterns to achieve data integration. Because of the variety of scenarios and legacy systems within an enterprise, it's most likely both will be used to implement the services:
- Take the data to the query. Collate slowly changing information into a holding data store (a warehouse, for example), and act upon the data there. Extract, Transform, and Load (ETL) technologies can be used here.
- Take the query to the data. For rapidly changing transactional data, the action must go to the physical source to manipulate it. You can use connector and adapter technology to make those accesses (for example read and update).
Within the SOA, the actual query facade could be implemented using a variety of patterns, such as federation and population (see Data Integration application patterns in Resources for more information), and technologies like Service Data Objects and Data Access Services (see Resources). The overall design patterns (get more information in "Toward a pattern language for Service-Oriented Architecture and Integration, Part 1" [developerWorks, July 2005]) for the various service implementations will influence these decisions. But in this problem space, the implementation design would need to be extended to make the service more intelligent. On the assumption that solutions are in place (to agreed levels of QoS) for legacy systems to publish entity data changes, the information service needs to subscribe to the changes (which and how often) and apply the changes to respective underlying physical systems, such as those that perform the data synchronisation.
This is the place in the architecture where you make master entity decisions. Which legacy IT system should be regarded as the master and in which transaction scenarios? This is where master data management technology can help your implementation, but of course the options and decisions, balanced with the integration capabilities of the legacy system, should be a significant part of the architecture definition.
The transformation to SOA introduces a new IT infrastructure that includes integration middleware and new composite applications, among other entities. Business rules are no longer contained within single applications. State is effectively managed through the execution of business rules. As part of developing the information-management design, the types and location of business-rules logic must be understood.
The types of business rules within an SOA include:
- Referential-integrity rules
- Application-behaviour rules
- Cross-application integration rules
- Transaction-processing business-logic rules
- Data-entity processing rules, such as stored procedures
The strategy should be to reuse rather than reproduce business rules. With the introduction of an SOA, you must carefully consider the execution of existing and new business rules (to support robust execution of services). The rules logic itself (wherever it's located in the context) might need to be exposed as a service that can be called to maintain integrity for certain processing scenarios.
Potential design-strategy impacts
The potential impacts to solutions that you must consider when reaching these design decisions include:
- Double-key processing. Efficiencies gained in one business area by introduction of the SOA need to be balanced against extra data-entry tasks (and therefore increased risk of data-entry errors). But it may be the only option for coping with data-synchronisation timing scenarios. These scenarios are not usually taken lightly. Your enterprise needs to establish a head count of persons to perform event-driven double keying to avoid the time delay of a nightly batch update. You'll counter this cost against developing an automated interface in a system to assist in data synchronisation. This circular analysis should assess whether the time required to implement automated interfaces (especially testing) compromises the overall delivery time scales.
- Unnecessary feedback loops. An update to a customer record in the CRM system is followed by a manual or automated call to the SOA to inform other applications of this out-of-bounds customer data change. To update the customer, the SOA calls its information service, which in itself calls out to the original CRM system and updates the customer data. The information-services architecture needs to implement different styles of update customer depending on the usage model.
- Scope creep. When an SOA scope statement is defined, lack of knowledge about legacy IT systems and what their process and information-management capabilities are undoubtedly leads to change. The derivation of the service model drives out that legacy analysis, only to reveal that a more substantial SOA infrastructure implementation is required. Consider budget and planning for phases of discovery before defining the overall scope of business services and engineering programme.
This article has outlined some areas for you to focus on to deliver successful business-transformation programmes incorporating SOA. The challenge of a business transformation to an SOA is significant. Most enterprises approach transformation incrementally by introducing an SOA infrastructure from which new composite applications can be built. They use legacy IT systems to implement the service providers.
For an enterprise to remain robust and compliant across all its operations, the transformation programme must develop an enterprise information-management solution, which includes — but is beyond the scope of — the SOA. This solution should address the issues of consistency and synchronisation of data and content across the legacy IT systems in the SOA so that information services can be safely exposed for consumption.
The solution architect responsible for delivering the SOA transformation must also balance the capabilities of the legacy IT systems against the needs of the SOA in the context of the overall governed enterprise architecture and budget. Technologies for data integration and master data management factor in an implementation strategy for the information-management solution.
We would like to thank Rick Robinson, Patrick Dantressangle, Bob Lojek, and Claudio Cozzi for their reviews and guidance.
Learn
- Read Ali Arsanjani's article,
"Service-oriented modeling and architecture"
(developerWorks, November 2004) for more information about the analysis and design
activities required to build an SOA.
- Tilak Mitra's article
"A case for SOA governance"
(developerWorks, August 2005) strengthens your awareness of the importance of SOA
governance for enterprises where IT is a key organization.
- Learn more about
data Integration application patterns
from the IBM Patterns for e-business site.
- "Introduction to Service Data Objects"
(developerWorks, September 2004) by Bertrand Portier and Frank Budinsky provides
an introduction to next-generation data programming with Service Data Objects.
- Learn more about the
Service Data Objects
specification.
- Ali Arsanjani's article
"Toward a pattern language for Service-Oriented Architecture and Integration, Part 1: Build a service eco-system"
(developerWorks, July 2005) discusses patterns focused on the migration, modeling,
design, and implementation of SOA and Service-Oriented Integration.
- Stay current with
developerWorks technical events and webcasts.
-
The IBM SOA Web site
offers an overview of SOA and how IBM can help you get there.
- Visit the
SOA and Web services zone on developerWorks
to learn more about SOA.
- Browse for books on these and other technical
topics at the
Safari bookstore.
Get products and technologies
- Innovate your next
development project with
IBM trial software,
available for download or on DVD.
Discuss
- Participate in the discussion forum.
- Participate in
developerWorks blogs, and
get involved in the developerWorks community.
Jeremy Caine is a senior IT architect and technology strategy consultant with IBM Global Business Services. He works on large business-transformation and system-integration programmes, delivering complex IT systems, Web technology solutions, and enterprise application integration strategies, primarily with financial services and government clients. Jeremy is also an active member of many worldwide IBM technical communities, providing thought leadership and supporting the creation of offerings for areas, such as SOA. He is an experienced methods leader and member of the IBM architectural teaching team.
Joe Hardman is a senior IT architect with IBM Global Business Services. He specialises in SOA and Web application architecture and security. His recent projects include the cahoot Internet bank and Polaris imarket, a Web service-enabled e-marketplace for the UK commercial insurance industry. He is currently helping a leading global oil company with its SOA programme.
