Agile integration:
A modern approach to enterprise hybrid integration


Today, more than ever, organizations require integration of applications, data and systems to transform engagement models and optimize channel performance while still maintaining business continuity. Traditional approaches to integration are costly and can’t keep up with escalating demand.

Your business needs a modern, agile approach to integration. It should empower extended teams to create integrations, leverage a complete set of integration styles and capabilities, and increase overall productivity.

Organizations pursuing digital transformation must embrace new ways to use and deploy integration technologies so they can quickly realize the benefits of multicloud, decentralization and microservices. You must evolve your approach to integration in order to allow you to boldly build new customer experiences rather than forcing models for architecture and development that detract from your organization’s productivity.

Many organizations have started embracing agile application development techniques, such as microservices architecture, and are now starting to see the benefits of that shift.

This smart paper is derived from an IBM® Redbooks® (PDF, 40.2 MB) publication that explores the merits of what we refer to as agile integration—a container-based, decentralized, microservices-aligned approach for integrating solutions that delivers the agility, scalability and resilience required by digital transformation.

Agile integration enables you to break free from heavily centralized integration architectures that cannot support the demand. Instead, integration becomes a critical enabler of innovation. This new approach spans from people and process to architecture and technology by leveraging modern practices and capabilities including APIs, microservices architecture, cloud-native design, DevOps, event-driven architecture and container-based infrastructure.

Traditional integration vs. Agile integration

Integration has changed

Enterprises are looking to digital transformation as the key to building new personalized and connected experiences across a network of applications that leverages data of all types.

Bringing together processes and information sources at the right time and in the right context is difficult at best—particularly when you consider the aggressive adoption of software-as-a-service (SaaS) business applications. Data from new sources needs to be injected into business processes to create competitive differentiation.

The value of agile integration for digital transformation

With so much data, distributed so widely, the speed at which you can bring that data together and make it available via APIs and events is fundamental to the success of modern innovations. Agile integration accelerates your ability to:

The integration landscape is changing to keep up with enterprise and marketplace computing demands, but how did we get from SOA (service-oriented architecture) and ESBs (enterprise service bus) to a modern, containerized, agile approach to integration?

The journey so far—SOA and the ESB pattern

Before you can look forward to the future of agile integration, you need to understand what came before. SOA patterns emerged at the start of the millennium, and the initial wide acceptance of SOA standards heralded a bright future where every system could discover and talk to any other system using synchronous exposure patterns.

SOA was typically implemented using the ESB pattern which provides standardized synchronous connectivity to back-end systems typically over web services. While many enterprises successfully implemented the ESB pattern, it’s popularity has led to a number of challenges.

The result was that creation of services by a specialist SOA team sometimes became a bottleneck for projects rather than an enabler.

Despite these challenges, the centralized ESB pattern does bring some benefits, especially for organizations that have a highly skilled integration team with a low attrition rate, and who receive a predictable and manageable number of new integration requirements. A single, centralized ESB certainly simplifies consistency and governance of implementation. However, many organizations have fluid and dynamic requirements to manage and are under pressure to implement integration using similar cloud-native technologies and agile methods as are being used in other parts of the organization. A case in point is the move to a microservices architecture typically found in the application development space.

SOA versus a microservices architecture

While SOA and a microservices architecture seem similar and are often confused, they are in fact completely separate concepts.

The use of SOA and the associated ESB pattern is an enterprise-wide initiative to make the data and functions in systems of record readily available to new applications. Solution designers create reusable, synchronous interfaces such as web services and RESTful APIs to expose the systems of record, so new, innovative applications can be created more quickly by incorporating data from multiple systems in real time.

SOA relates to enterprise service exposure

SOA and microservices refer to different scopes.

A microservices architecture, on the other hand, is a way of writing an individual application as a set of smaller (microservice) components in a way that makes the application more agile, scalable and resilient.

In summary, SOA is about real-time integration between applications, whereas a microservices architecture is about how the applications are built internally.

The case for agile integration

Why have microservices concepts become so popular in the application space? Because they provide an alternative approach to structuring applications that enables greater agility.

Rather than an application being a large silo of code running on the same server, microservices can help you build applications that are composed of a collection of smaller, completely independently running components.

A microservices architecture enables three critical benefits:

With those benefits in mind, what would it look like if we re-imagined integration, which is typically deployed in centralized silos, with a new perspective based on microservices architecture? That’s what we call an “agile integration.”

Traditional integration architecture
Centrally provisioned and administered

2 rows of 3 boxes connected to a central rectangle

Agile integration architecture
Fine-grained deployment, decentralized ownership, and cloud native infrastructure

2 clouds with 3 API boxes connected to each other plus another box with 3 API boxes

There are three related, but separate aspects to agile integration:

People and process: Decentralized integration ownership

SOA tended to result in the creation of centralized integration teams and infrastructure to implement the service layer. The central team knew their integration technology well, but often didn’t understand the applications they were integrating, so translating requirements could be slow and error prone. This dependency created ongoing friction given the pace at which projects could run.

Many organizations would have preferred the application teams own their own integration, but the technology and infrastructure at the time didn’t enable that. The move to agile integration with more fine-grained, cloud-native deployment, makes it possible to simplify and automate much of the build, test and deployment pipeline. This in turn enables us to distribute the responsibility for creating and maintaining integrations beyond the central team.

Centralized integration team

Row of 9 integration boxes with arrows to integration team circle with arrows to row of 3 application developer boxes

Decentralized integration ownership

3 groups of randomly placed integration boxes with arrows to row of 3 application developers and integration specialist box groups

This goes hand in hand with the increased importance of API management. Teams need to be able to manage the exposure of their own APIs, rather than referring to a centralized team of API gateway specialists to do that. API management enables application teams to perform that configuration themselves, and also populate developer portals with definitions of the APIs and events such that consumers of these can discover them easily, and self-onboard to use them.

Furthermore, microservice-based applications often prefer to communicate internally using events. The increased decoupling further improves the team’s autonomy to make changes independently. Agile integration also considers how best to enable teams to publish and consume event streams within and across application boundaries.

Architecture: Fine-grained integration deployment

Traditional integration is characterized by the centralized deployment of integrations in the ESB pattern. In this model, all integrations are deployed to a unique heavily nurtured pair of integration servers and can result in a bottleneck for projects. Any deployment to the shared servers runs the risk of destabilizing existing critical interfaces. Furthermore, teams running individual projects can’t independently choose to upgrade the integration middleware to gain access to new features.

Using the same concepts as a microservices architecture, we could break up the enterprise-wide ESB into smaller, more manageable and dedicated pieces. These “fine-grained integration deployment” patterns provide specialized, right-sized containers that improve agility, scalability and resilience, and look very different than the centralized ESB patterns of the past. By taking advantage of the cloud native infrastructure we will discuss next, it is much easier to set up automated pipelines that do not require specialists skills to run. This radically simplifies the mechanical deployment process leaving more time to focus on the business value of the integration.

Centralized ESB

Row of 4 consumer boxes, row of 9 integration boxes and row of 3 provider cylinders

Fine-grained integration deployment

Row of 4 consumer boxes, 9 randomly placed integration boxes and row of 3 provider cylinders

Fine-grained integration deployment draws on the benefits of a microservices architecture. Let’s revisit what we listed as microservices benefits in light of fine-grained integration deployment:

Technology: Cloud-native integration infrastructure

Agile integration requires that integration capabilities be deployed using cloud-native principles. This enables a cloud-native platform such as one based on Kubernetes to perform much of the deployment, scaling and availability tasks in a standardized way. Agile integration normalizes the integration skillset required to set up and look after a given capability and ensures that capability is portable across a multicloud landscape.

Adopting a cloud-native approach impacts the ways your DevOps teams will interact with the environment and the solution overall to increase efficiency as more solutions are moved to lightweight, portable architectures.

In addition, the cloud-native approach and technology simplifies automation of build and deployment pipelines and enables cleaner setup and tear down of testing environments. These benefits contribute to much shorter build-release cycles and a significant increase in productivity.

Moving to a cloud-native approach is a significant change. Some high-level characteristics include:

If the above principles are adhered to, container orchestration platforms like Kubernetes provide auto scaling, load-balancing, deployment, internal routing, reinstatement and more in a standardized way to significantly simplify platform administration.

Modern integration portfolios must be able to embrace the core principles and standards of a cloud-native approach. Integration runtimes, API management, event streams and messaging capabilities must all leverage container orchestration platforms to provide the portability, operational consistency and automation required by today’s solutions.

Hybrid integration platforms for agile integration

An integration platform—or what some analysts refer to as a “hybrid integration platform”—brings together the core integration capabilities cohesively so organizations can build integrations for their business solutions in a more efficient and consistent way.

A hybrid integration platform is a collection of integration capabilities that run both on premises and in the cloud. It should allow people with different skills levels (integration specialists and non-specialists) to perform a wide variety of integration patterns and deploy them discretely to improve agility.

Non-specialists should be provided with a simpler experience that guides them through solving straightforward problems, while integration specialists have expert levels of control to deal with the more complex patterns. This promotes decentralization as users can work independently, yet collaboratively with shared assets, while preserving governance.

Your integration platform should be specifically designed to take advantage of cloud native infrastructure as it becomes more common place.

Read this Ovum report about the factors driving hybrid integration platform adoption.

IBM Cloud Pak for Integration

IBM Cloud Pak® for Integration brings together the key set of integration capabilities into a coherent platform that is simple, fast and trusted.

Embrace an agile, AI-accelerated approach to integration that enables extended teams to create integrations, leverages a complete set of integration styles and capabilities, and embeds AI and automation across the integration lifecycle. Speed integration development, reduce costs, and increase overall operational efficiency while maintaining enhanced security, governance and availability.

Easily build powerful integrations and APIs in minutes, provides leading performance and scalability, and offers unmatched end-to-end capabilities with enterprise-grade security.

IBM Cloud Pak® for Integration, part of the IBM Automation platform, is the most complete hybrid integration platform in the industry and includes all of the key integration styles and capabilities your team needs. Deploy workloads on any on-premises or cloud environment for maximum flexibility.

Application and data integration

From complex legacy systems to simple SaaS applications, connect applications and data sources on premises or in the cloud to coordinate the exchange of business information so that data is available when and where it’s needed.

Pre-built smart connectors and a common, shareable asset repository increases speed of delivery and eliminates concerns about mismatched sources, formats, or standards.

Low-code/no-code integration tooling leverages natural language processing (NLP) and AI to offer custom mapping suggestions when building integration flows.

API management

Expose and manage business services as reusable APIs for select developer communities both internal and external to your organization. You can adopt an API strategy to accelerate how effectively they can share their unique data and services assets to fuel new applications and new business opportunities.

Leverage AI models, pre-trained on API calls and sales or marketing data from sources like your CRM to surface and highlight business value analytics. Scale dynamically based on demand by utilizing AIOps practices to detect, identify and resolve production performance issues and feeding the information back into the integration assembly process to train future best practices-based integration asset creation.

Enterprise messaging

Ensure real-time information is available from anywhere at any time by providing reliable message delivery without message loss, duplication or complex recovery in the event of system or network issue. Fully automated failover, dynamically distributed messaging workloads, high-throughput, and suggestions for queues to reuse help to provide a highly available and low-latency solution.

Event streaming

Analyze data associated with an event and respond to it in real time. Your business can tap into unused data, take advantage of real-time data insights and create responsive customer experiences.

High-speed data transfer

Move huge amounts of data between on premises and cloud or cloud-to-cloud rapidly and predictably with enhanced levels of security to facilitate fast adoption of cloud platforms when data sets are very large.

Platform-level security, automation and monitoring

Extend connectivity and integration beyond the enterprise with edge capabilities that protect APIs, the data they move, and the systems behind them. Detect production issues and act based on related automations.

IBM Cloud Pak for Integration is the platform for agile integration. With it, we’ve seen companies speed their integration development by 300%, reduce their costs of integration by over 33% and increase their overall operational agility.

Learn more about IBM Cloud Pak for Integration