Federated business and IT
Over recent years we've seen a rapid adoption of SOA as the underpinning of enterprise architectures. Many enterprises are now moving beyond SOA projects that focused on specific, departmental business problems to more complex SOA installations, extending the reach of SOA, making it ubiquitous, and supporting end-to-end business transactions across business unit boundaries.
More and more of these enterprises are adopting a federated approach as a means to scale their SOA implementations across business units that reflect the governance, organizational, and geographical structures of their businesses. There are many interrelated dimensions to a federated approach to business when an enterprise consists of multiple business units: examples include connectivity federation, data federation, service metadata federation, business process federation, and governance federation.
Federation reflects the fact that in a large-scale SOA, communities of participants with different requirements often make different choices when implementing their piece of the enterprise SOA. Federation can connect those islands of SOA together into something that approximates a truly enterprise-spanning SOA. Federation can increase the value of shared services, decrease the cost of duplication, and enhance business agility and flexibility.
Today, many federation efforts are done as bottom-up or reactive federation, mainly driven by IT. Such efforts often result when multiple business units have adopted SOA, but have used different approaches or infrastructures, or when a business needs to integrate new business units from mergers or acquisitions. The objective is to mesh these islands of SOA, enabling service interactions between the islands in order to enable cross domain tasks. Meeting the objective means addressing the multiple dimensions of federation.
But federation is not only used to weave together existing SOA efforts. In many cases, federation is a good way to represent the nature of business and governance in an enterprise. Thus we see a growing number of situations where top-down or proactive federation is applied to reflect the business structure of an enterprise as a set of collaborating and coordinated business units and communities and the varying requirements on infrastructure in support of the participating business units. Fundamentally, the objective is the same, enabling service interactions between the business units to enable cross-domain tasks, and likewise meeting the objective means addressing the multiple dimensions of federation. In a proactive situation, however, the ability to determine up-front the approach or infrastructures used in the individual business units often simplifies the task of creating the federated enterprise. We think that most enterprises will require a mixture of reactive and proactive federation.
As a result, Enterprise Architects are beginning to take on a Federation Architect role, defining the federation profile of the enterprise to cover all of the federation dimensions and matching business requirements leading to federated structures with matching federated IT underpinnings. This article focuses on the connectivity aspects of federation.
In SOA, the focus is naturally on services, as they enable the business of the enterprise. In particular, much of the value of SOA derives from service reuse. Historically, service reuse has started within a particular business unit, and hence it is natural to refer to a business unit as a service domain. The service domains in an enterprise reflect its underlying organizational, business, geographical, or governance structure.
Connectivity is about enabling interactions between services, according to the requirements of the enterprise, to facilitate service reuse. Historically, connectivity has been considered mostly within a particular service domain. Within a domain, a connectivity infrastructure enables four basic principles of connectivity: service visibility, service management, service security, and service governance.
Service visibility in a service domain refers to the ability of a service consumer to "see" and thus interact with a service provider. A visibility infrastructure constructed from registry and service bus components supports this visibility. In particular, the service bus may need to enable basic interoperability via service virtualization, as described in Part 1 of this article series.
Service management in a service domain enables understanding of and sometimes dynamic adaptation to changing service conditions, such as response time, in that domain. For example, as more consumers use a service, response time may increase, which may result in additional resources allocated to the service, or in rejection of some consumer requests. In many environments, service management is encapsulated in management products, and often assisted by a combination of service registry and service bus components via aspect-oriented connectivity, as described in described in Part 1.
Service security in a service domain guards the integrity of the domain by securing access to services. For example, access can be restricted such that not all consumers that can "see" a service can actually invoke it. Security policies associated with services at authoring time are stored in a policy repository, and are enacted in policy enforcement points in the connectivity infrastructure that make use of policy evaluation handlers for specific types of policies. In many environments, service security is mostly encapsulated in security products, and can be assisted by a combination of service registry and service bus components via aspect-oriented connectivity.
Service governance is to a large extent about service lifecycle management, thus about setting policies and identifying enforcement processes for the other parts of the connectivity infrastructure in the domain. For example, governance defines policies for which consumers can "see" a service, whether the service is monitored, and which consumers can access a service, and it enables enforcement points in the other parts connectivity infrastructure.
The goal of federated connectivity is to facilitate service reuse across the entire enterprise, by enabling service interactions between services in different service domains, as well as services within a domain. Federated connectivity thus lets business processes span business unit boundaries, increasing the value of shared services, decreasing costs associated with duplication, and enhancing business agility and flexibility. Federated connectivity requires that one address service visibility, service management, service security, and service governance to create the federated enterprise.
Of course, the basic architecture principles for the connectivity underpinnings of an SOA still apply. Federated connectivity is achieved by recursive application of basic connectivity principles. More specifically, one can implement federated connectivity by adapting the various patterns that serve intra-domain connectivity needs to serve inter-domain connectivity needs; the sections below offer some examples. In part, this is achieved by simplifying and abstracting the more refined intra-domain patterns, and in part by increased attention to aspects of the patterns that are not as important in intra-domain scenarios as they are across federated topologies. This recursive application of connectivity principles eliminates reliance on any specialized connectivity technologies, enhancing the ability to federate in both reactive and proactive situations. As a result, the Enterprise Architect needs to take the following steps to define the federation profile of an enterprise:
- Identify the service domains (business units).
- Define the scope of service reuse across domain boundaries, including which domains share services and which can consume shared services, thus enhancing the value of existing services, and increasing business process agility by tapping into resources across the enterprise, independent of their specific host and owner.
- Map the above reuse policies to requirements on the SOA infrastructure for the individual domains and for the connectivity fabric connecting them.
- Design governance, visibility, security, and management infrastructures for the resulting federated enterprise.
Let's explore the question of how to use federation principles to establish enterprise-spanning connectivity in more detail with an example. Figure 1 shows a hypothetical enterprise consisting of two domains, named Retail and e-business. The Retail domain represents a "legacy" business activity dealing with customer accounts. The Account service provides functions used by consumers in the Retail domain to implement the necessary business activities relevant to that domain. Within the Retail domain, visibility is achieved via a service registry and repository (registry for short), which typically also assists in governance. We assume that Retail has robust security and management infrastructures as well. A domain service bus (bus for short) decouples the interactions between the consumers and the Account service, and helps enforce various connectivity related policies defined by the security and management infrastructures.
Figure 1. Federation example
The e-business domain represents a new business opportunity for the enterprise. That domain implements a different set of business activities, one of which is a business process named AccountProcess. Like the Retail domain, the e-business domain includes infrastructures for service visibility, service management, service security, and service governance. The products and technologies used to build the connectivity infrastructure in e-business could be different than those in Retail; for example, in the security credentials (defining consumer identity) required for authentication and authorization.
AccountProcess needs account functions already offered by the Account service in Retail -- thus the need for federation of the e-business and Retail domains, and for federated connectivity between e-business and Retail. Any cross-domain connectivity must comply with the connectivity policies of both domains and the applicable enterprise-wide policies. Thus "federated connectivity" is really about federating service visibility, service management, and service security, combined with cross-domain governance. The domain service buses play an important role, offering mechanisms to help enable inter-domain service visibility, service management, service security, and service governance.
Federated connectivity examined
As suggested above, the ESB pattern for enabling sophisticated service connectivity within a domain scales well to federated connectivity across domains. What changes in comparison to connectivity within a single domain is focus; the ESB concepts described in Part 1 -- service virtualization and aspect-oriented connectivity -- describe pretty much what happens in federated connectivity. Certain connectivity aspects that are straightforward within a domain, where you can assume a degree of common understanding of services offered and a moderately homogenous security model and service administration community, take center stage when federating service domains. In fact, new usage patterns of the established concepts need to be introduced. Let's examine the most important aspects of federation in more detail.
In some cases, the desired intra-domain visibility is achieved by advertising the service producer via a registry, such as IBM® WebSphere® Service Registry and Repository. In other cases, it is more appropriate to "make visible" a virtualized service abstraction of the actual provider to accommodate requirements the consumers might have on the provider that are not directly supported by the original provider, or to offer added service levels. A service bus, such as IBM WebSphere Message Broker, IBM WebSphere Enterprise Service Bus, or the IBM WebSphere DataPower Integration Appliance XI50 is used to create the abstraction, called a proxy, which becomes visible instead of the service itself.
What does federation mean for visibility? Consider the example above. What must we do to enable AccountProcess in e-business to interact with Account in Retail, considering the differences in policies, QoS expectations, and other characteristics of the different domains? A number of federation patterns can be applied, such as direct federation and indirect federation.
The direct pattern shares visibility metadata describing Account with the e-business registry, where you "project" the service metadata originally owned by the registry in Retail into the registry in e-business. Given visibility to Account in e-business, the developer of AccountProcess might use Account metadata at development time, in effect hard-coding direct connectivity. Alternatively, Account metadata might be used at runtime, perhaps via a mediation already in the e-business bus, allowing dynamic connectivity.
Often, direct connectivity is not possible or desirable. Federation Architects may "inject" the proxy pattern and advertise a proxy hosted by the bus in one or even both of the domains to create interoperability by dealing with syntactic mismatches or to ensure compliance with domain-specific policies. The proxy or proxies delegate to the original service in the Retail domain. The indirect federation pattern is a variation on the well-established delegation pattern, and its main purpose is to establish a local control point for customizing access to the remote service according to the needs of consumers and the general policies in the e-business domain, the needs of the providers and the general policies applicable in the Retail domain, or both.
No matter what means is used, the Federation Architect must have the ability to examine the visibility infrastructure in Retail, to identify the services in Retail to be shared, and to modify the visibility infrastructure in e-business in order to share services with e-business. In addition, the Federation Architect may need to inject proxies into the domain buses to enable basic interoperability and policy compliance in the domains.
Service management within a domain has several aspects: monitoring and management policies that determine the desired behavior, policy enforcement agents sprinkled into the service interaction infrastructure that enact those policies, policy decision services that interpret policies and feed into the enforcement points, and a service management dashboard that enables federation administrators to monitor and manage the underlying service interactions across domains. In many environments, service management is encapsulated in products like Tivoli® Composite Application Manager for SOA, independent of the runtime containers for the services, and is sometimes facilitated by a combination of service registry and ESB infrastructure.
What does federation mean for service management? It requires configuration of policy enforcement points across domains in a consistent and coordinated fashion, as well as monitoring of consumer-provider interaction across domains. Consider again the example. Once AccountProcess and other consumers in e-business begin using Account in Retail, Account will get extra load, and consume additional resources. What must we do to enable understanding that e-business is driving the additional load? This means federation of metrics produced by the domains' management infrastructures, letting administrators examine the domains' metrics in a single console for the entire federation, and chasing the interaction from requester in the e-business domain through any federation proxies to the Account service. The Federation Architect must be able to join and control the disparate management infrastructures. For example, what if you want to throttle traffic from e-business so that the enterprise continues to run smoothly? The Federation Architect may need the ability to inject proxies implementing policy enforcement points into either e-business or Retail and drive their behavior from management metrics.
Intra-domain service security is mostly encapsulated in a product or in a product set, like Tivoli Identity Manager and Tivoli Access Manager. These products can be assisted by a combination of service registry and service bus components to achieve the desired security characteristics.
What does federation mean for service security? At a basic access control level, it is not unlikely that the requirements vary in the domains involved in a federation. The federation proxy pattern for establishing service visibility across domains is a good vehicle to enforce domain-specific authorization and access control policies.
In our example, we can assume that Retail already required consumers in Retail to have the correct identity to access Account. Generally, consumers in e-business would not have an identity relevant to Retail, so you would need to ensure that the Retail identity requirements are met. In the best case, e-business would have a matching security infrastructure and the Federation Architect would need the ability to configure that infrastructure. In some cases, the infrastructures won't match, and the Federation Architect would need to inject a proxy into e-business or Retail using identity mapping and configuration technology, as offered by Tivoli Federated Identity Manager.
Service governance in a service domain involves defining policies and processes that, for example, control the life cycle of services in the domain, establish service visibility in the various stages of that life cycle, define service monitoring and management requirements, and define identity and access criteria. Service governance usually involves cooperation between the service visibility, management, and security infrastructures.
A federated approach to service governance consists of establishing commonality across implementations, typically specifying services standards, security requirements, patterns, ways of expressing qualities of service and other policies, lifecycle management processes, and governance mechanisms. Federation involves taking control of the configuration of service reuse across domain boundaries, facilitating and controlling promotion of providers from services shared within a domain to enterprise-wide shared services, and establishing policies that direct the enforcement points to achieve the visibility, management, and security goals for the enterprise relevant for those services.
In our example, we can assume that Retail already has some governance infrastructure that would control the life cycle of the Account service from the point where it is conceived, through its promotion to production, to its retirement or replacement by a new version. That governance infrastructure would also control decisions like the one to share the service with other domains -- which requires quite a bit of thought about provisioning capacity for additional load as well as the impact of requirements from extra-domain consumers.
In the simplest of cases, that infrastructure could be extended to encompass e-business as well. Depending on the needs of the enterprise, you might need to let e-business and Retail retain some autonomy, and thus have independent governance infrastructures. The Federation Architect needs to view the federation as a whole, but impact the individual governance infrastructures to meet the goals of the enterprise.
Summing it up
We've seen that federating connectivity across multiple service domains for the most part is recursive application of the same principles and techniques that apply within a service domain. The big differences are scope and focus.
The Connectivity Architect for a domain uses the domain visibility infrastructure to enable interaction with services within a domain, engages the domain management and security infrastructures to provide the non-functional requirements for the domain, and injects mediations into a domain service bus to facilitate dynamic intra-domain service interaction and assist the domain management and security infrastructures. The Federation Architect must in effect have an enterprise-wide view and control of service visibility and engage the various domain management and security infrastructures to provide the non-functional requirements for the domains. Some aspects of connectivity require more focus due to mismatches between domains, and the Federation Architect may need to inject proxies into one or more domain service buses, or even inject a new service bus to facilitate dynamic inter-domain service interaction and perhaps deal with mismatches in the non-functional requirements.
Figure 2. Federation configuration
Figure 2 shows the requirements and a solution in the context of our example. Generally, the Federation Architect needs to assess and understand the existing domains, and to create and configure the federation. Specifically, the Federation Architect needs to access and control the service visibility, management, security, and governance infrastructures in the e-business and Retail domains, and project visibility metadata and inject proxies as needed into the domain service buses to produce the desired cross-domain connectivity. In our example, showing a somewhat extreme but sometimes realistic use case, the AccountProcess in e-business interacts with the Account service in Retail via two proxies: the Account proxy in the e-business service bus might do identity mapping, and the Account proxy in the Retail service bus might ensure that consumers from the e-business domain receive lower priority than consumers from the Retail domain.
Federated connectivity challenges
Federation poses some interesting challenges for the connectivity underpinnings in support of scaling the reach of the SOA infrastructure across the enterprise. Federated connectivity comes in reactive and proactive variants. Reactive federation is about coherently connecting service consumers and service providers in existing autonomous domains, with distinct service buses; such domains exist in an enterprise due to technical or business considerations of different business units, mergers, acquisitions, and so on. Proactive federation is about designing the required connectivity across autonomous domains created to reflect the business, technical, organizational, geographical, or governance structures in the enterprise. In reality, most enterprises will require a mixture of reactive and proactive federation.
Federation must support situations where the domain service bus actually consists of multiple service bus nodes, each devoted to specific connectivity tasks within the domain. Similarly, there may in fact be more than one registry within the domain.
Federation must handle the heterogeneous nature of most enterprises. Only in rare cases, even in proactive situations, will the same infrastructure products or technologies for visibility, management, security, and governance be used across all the disparate domains.
Federation must support different levels of domain autonomy regarding service visibility and the attendant management and security requirements. Some enterprises can force strong central control, such that all service visibility decisions, inter-domain and even intra-domain, are made centrally. In other cases, there may be weak central control, where intra-domain decisions are made within the domain, and decisions about inter-domain visibility are made at the enterprise level. Some domains may allow no central control, and all service visibility decisions must be made within the domain, with federation simply facilitating that domain's connectivity goals in terms of exposing services to other domains or using services from other domains. And of course, combinations of domains with different levels of autonomy are possible within an enterprise.
Federation must allow for diversity in connectivity topologies, and this diversity may be the biggest challenge for a Federation Architect. The coherent connectivity demands of different types of enterprises will differ. For example, as shown in Figure 3 below, the topology resulting from the connectivity requirements for a retail company with a headquarters and hundreds to thousands of stores will differ greatly from the topology resulting from the connectivity requirements for a world-wide news service with a few peers. Topology concerns include balancing performance optimization within a particular topology versus the resulting manageability. More challenging is that the needs of an enterprise may evolve over time and require a more complex topology. For example, the "directly connected" news service may need additional domains, or introduce more flexibility or centralized policy enforcement using a more centralized "backbone" or "brokered" topology.
Figure 3. Federation topology examples
All these challenges are surmountable, even today, in one-off situations, given sufficient effort and the use of the appropriate principles and technologies. The real challenge is to formalize the role of the Federation Architect and provide the hypothetical "federation configurator" that will enable the Federation Architect to produce robust federations in a consumable and efficient fashion.
This article has shown that a very important dimension of the federated enterprise is federated connectivity across the multiple service domains within a federation, extending service reuse between as well as within the domains. Federated connectivity can be effectively modeled as recursive application of connectivity principles and techniques that apply in a single service domain regarding service visibility, service management, service security, and service governance. We expect that future developments will formalize this model and offer consumable tooling support for both intra-domain and inter-domain connectivity.
- SOA governance white papers
These three IBM white papers explain why SOA is fast becoming the IT software approach of choice for many major enterprises, and SOA governance is critical to successful SOA adoption.
- Introduction to SOA governance
This article explores how IBM defines SOA governance and explains why it is critical to the success of your SOA project.
- WebSphere DataPower Integration Appliance XI50 product page
- WebSphere ESB developer resources page
- WebSphere ESB product page
- WebSphere Message Broker developer resources page
- WebSphere Message Broker product page
- WebSphere Service Registry and Repository product page
- Tivoli Access Manager product page
- Tivoli Composite Application Manager product page
- Tivoli Federated Identity Manager product page
- Tivoli Identity Manager product page
- developerWorks WebSphere Business Integration zone
For developers, access to WebSphere Business Integration how-to articles, downloads, tutorials, education, product info, and more.
- WebSphere Business Integration products page
For both business and technical users, a handy overview of all WebSphere Business Integration products
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- Technical books from IBM Press
Convenient online ordering through Barnes & Noble.
- developerWorks technical events and Webcasts
Free technical sessions by IBM experts that can accelerate your learning curve and help you succeed in your most difficult software projects. Sessions range from one-hour Webcasts to half-day and full-day live sessions in cities worldwide.