Agile integration:
A modern approach to enterprise hybrid integration

Overview

Today, more than ever, organizations require integration of applications, data and systems to transform engagement models and optimize channel performance while still maintaining business continuity. Traditional approaches to integration are costly and can’t keep up with escalating demand.

Your business needs a modern, agile approach to integration. It should empower extended teams to create integrations, leverages a complete set of integration styles and capabilities, and increases overall productivity.

Organizations pursuing digital transformation must embrace new ways to use and deploy integration technologies so they can quickly realize the benefits of multicloud, decentralization and microservices. You must evolve your approach to integration in order to allow you to boldly build new customer experiences rather than forcing models for architecture and development that detract from your organization’s productivity.

Many organizations have started embracing agile application development techniques, such as microservices architecture, and are now starting to see the benefits of that shift.

This smart paper is derived from an IBM Redbook that explores the merits of what we refer to as agile integration—a container-based, decentralized, microservices-aligned approach for integrating solutions that delivers the agility, scalability and resilience required by digital transformation.

Agile integration enables you to break free from heavily centralized integration architectures that cannot support the demand. Instead, integration becomes a critical enabler of innovation. This new approach spans from people and process to architecture and technology by leveraging modern practices and capabilities including APIs, microservices architecture, cloud-native design, DevOps, event-driven architecture and container-based infrastructure.

Traditional integration vs. Agile integration

Integration has changed

Enterprises are looking to digital transformation as the key to building new personalized and connected experiences across a network of applications that leverages data of all types.

Bringing together processes and information sources at the right time and in the right context is difficult at best—particularly when you consider the aggressive adoption of software-as-a-service (SaaS) business applications. Data from new sources needs to be injected into business processes to create competitive differentiation.

The value of application integration for digital transformation

When you consider your agenda for building new customer experiences, you must focus on how data is accessed and made available for the services and APIs that power these initiatives. Agile integration can help you:

The integration landscape is changing to keep up with enterprise and marketplace computing demands, but how did we get from SOA (service-oriented architecture) and ESBs (enterprise service bus) to a modern, containerized, agile approach to integration?

The journey so far—SOA and the ESB pattern

Before you can look forward to the future of agile integration, you need to understand what came before. SOA patterns emerged at the start of the millennium, and the initial wide acceptance of SOA standards heralded a bright future where every system could discover and talk to any other system using synchronous exposure patterns.

SOA was typically implemented using the ESB pattern which provides standardized synchronous connectivity to back-end systems typically over web services. While many enterprises successfully implemented the ESB pattern, it’s popularity has led to a number of challenges.

The result was that creation of services by a specialist SOA team sometimes became a bottleneck for projects rather than an enabler.

Despite these challenges, the centralized ESB pattern does bring some benefits, especially for organizations that have a highly skilled integration team with a low attrition rate, and who receive a predictable and manageable number of new integration requirements. A single, centralized ESB certainly simplifies consistency and governance of implementation. However, many organizations have fluid and dynamic requirements to manage and are under pressure to implement integration using similar cloud-native technologies and agile methods as are being used in other parts of the organization. A case in point is the move to a microservices architecture typically found in the application development space.

SOA versus a microservices architecture

While SOA and a microservices architecture seem similar and are often confused, they are in fact completely separate concepts.

The use of SOA and the associated ESB pattern is an enterprise-wide initiative to make the data and functions in systems of record readily available to new applications. Solution designers create reusable, synchronous interfaces such as web services and RESTful APIs to expose the systems of record, so new, innovative applications can be created more quickly by incorporating data from multiple systems in real time.

SOA and microservices refer to different scopes.

A microservices architecture, on the other hand, is a way of writing an individual application as a set of smaller (microservice) components in a way that makes the application more agile, scalable and resilient.

In summary, SOA is about real-time integration between applications, whereas a microservices architecture is about how the applications are built internally.

The case for agile integration

Why have microservices concepts become so popular in the application space? Because they represent an alternative approach to structuring applications.

Rather than an application being a large silo of code running on the same server, microservices can help you build applications that are composed of a collection of smaller, completely independently running components.

A microservices architecture enables three critical benefits:

With those benefits in mind, what would it look like if we re-imagined integration, which is typically deployed in centralized silos, with a new perspective based on microservices architecture? That’s what we call an “agile integration.”

A comparison of traditional and agile integration architectures

There are three related, but separate aspects to agile integration:

People and process: Decentralized integration ownership

SOA tended to result in the creation of centralized integration teams and infrastructure to implement the service layer. The central team knew their integration technology well, but often didn’t understand the applications they were integrating, so translating requirements could be slow and error prone. This dependency created ongoing friction given the pace at which projects could run.

Many organizations would have preferred the application teams own their own integration, but the technology and infrastructure at the time didn’t enable that. The move to agile integration with more fine-grained, cloud-native deployment, makes it possible to distribute the responsibility for creating and maintaining integrations. This works together with API management, enabling application teams to manage the exposure of their own APIs.

A comparison of centralized and decentralized ownership of integrations

Furthermore, microservice-based applications often prefer to communicate internally using events. The increased decoupling further improves the team’s autonomy to make changes independently. Agile integration also considers how best to enable teams to publish and consume event streams within and across application boundaries.

Architecture: Fine-grained integration deployment

Traditional integration is characterized by the centralized deployment of integrations in the ESB pattern. In this model, all integrations are deployed to a single heavily nurtured pair of integration servers and can result in a bottleneck for projects. Any deployment to the shared servers runs the risk of destabilizing existing critical interfaces. Furthermore, teams running individual projects can’t independently choose to upgrade the integration middleware to gain access to new features.

Using the same concepts as a microservices architecture, we could break up the enterprise-wide ESB into smaller, more manageable and dedicated pieces. These “fine-grained integration deployment” patterns provide specialized, right-sized containers that improve agility, scalability and resilience, and look very different than the centralized ESB patterns of the past.

A simplified comparison of a centralized ESB to a fine-grained integration deployment

Fine-grained integration deployment draws on the benefits of a microservices architecture. Let’s revisit what we listed as microservices benefits in light of fine-grained integration deployment:

Technology: Cloud-native integration infrastructure

Agile integration requires that integration capabilities be deployed using cloud-native principles. This enables a cloud-native platform such as one based on Kubernetes to perform much of the deployment, scaling and availability tasks in a standardized way. Agile integration normalizes the integration skillset required to set up and look after a given capability and ensures that capability is portable across a multicloud landscape.

Adopting a cloud-native approach impacts the ways your DevOps teams will interact with the environment and the solution overall to increase efficiency as more solutions are moved to lightweight, portable architectures.

In addition, the cloud-native approach and technology simplifies automation of build and deployment pipelines and enables cleaner setup and tear down of testing environments. These benefits contribute to much shorter build-release cycles and a significant increase in productivity.

Moving to a cloud-native approach is a significant change. Some high-level characteristics include:

If the above principles are adhered to, container orchestration platforms like Kubernetes provide auto scaling, load-balancing, deployment, internal routing, reinstatement and more in a standardized way to significantly simplify platform administration.

Modern integration portfolios must be able to embrace the core principles and standards of a cloud-native approach. Integration runtimes, API management, event streams and messaging capabilities must all leverage container orchestration platforms to provide the portability and operational consistency required by today’s solutions.

Hybrid integration platforms for agile integration

An integration platform—or what some analysts refer to as a “hybrid integration platform”—brings together the core integration capabilities cohesively so organizations can build integrations for their business solutions in a more efficient and consistent way.

A hybrid integration platform is a collection of integration capabilities that run both on premises and in the cloud. It should allow people with different skills levels (integration specialists and non-specialists) to perform a wide variety of integration patterns and deploy them discretely to improve agility.

Non-specialists should be provided with a simpler experience that guides them through solving straightforward problems, while integration specialists have expert levels of control to deal with the more complex patterns. This promotes decentralization as users can work independently, yet collaboratively with shared assets, while preserving governance.

Your integration platform should be specifically designed to take advantage of cloud native infrastructure as it becomes more common place.

Read this Ovum report about the factors driving hybrid integration platform adoption.

IBM Cloud Pak for Integration

IBM Cloud Pak for Integration brings together the key set of integration capabilities into a cohesive platform that is simple, fast and trusted.

It allows you to easily build powerful integrations and APIs in minutes, provides leading performance and scalability, and offers unmatched end-to-end capabilities with enterprise-grade security.

IBM Cloud Pak ® for Integration offers a unified set of integration capabilities. These capabilities can be deployed on almost any platform, providing maximum flexibility. Furthermore, deployment into Red Hat® OpenShift® managed container environment allows your teams to run their own containers while delivering auto-scale and self-healing to help avoid putting your integrations at risk. It makes use of operators to drive agility and leverages the dashboards, monitoring, logging, tracing and security systems of OpenShift to ensure consistency and provide visibility across all integration solutions and endpoints—including on-premises and hybrid, multi-cloud.

A visual representation of the key capabilities of IBM Cloud Pak for Integration

IBM Cloud Pak for Integration is the most complete hybrid integration platform in the industry and includes all of the key integration capabilities your team needs.

Application and data integration

Connect applications and data sources on premises or in the cloud to coordinate the exchange of business information so that data is available when and where it’s needed.

API lifecycle management

Expose and manage business services as reusable APIs for select developer communities both internal and external to your organization. You can adopt an API strategy to accelerate how effectively they can share their unique data and services assets to fuel new applications and new business opportunities.

Enterprise messaging

Ensure real-time information is available from anywhere at any time by providing reliable message delivery without message loss, duplication or complex recovery in the event of system or network issue.

Event streaming

Analyze data associated with an event and respond to it in real time. Your business can tap into unused data, take advantage of real-time data insights and create responsive customer experiences.

High-speed data transfer

Move huge amounts of data between on premises and cloud or cloud-to-cloud rapidly and predictably with enhanced levels of security to facilitate fast adoption of cloud platforms when data sets are very large.

Secure gateway

Extend connectivity and integration beyond the enterprise with edge capabilities that protect APIs, the data they move, and the systems behind them.

IBM Cloud Pak for Integration is the platform for agile integration. With it, we’ve seen companies speed their integration development by 3 times, reduce their costs of integration by over 1/3 and increase their overall operational agility.

Learn more about IBM Cloud Pak for Integration