A modern approach to enterprise hybrid integration
The case for agile integration
Why have microservices concepts become so popular in the application space? Because they provide an alternative approach to structuring applications that enables greater agility.
Rather than an application being a large silo of code running on the same server, microservices can help you build applications that are composed of a collection of smaller, completely independently running components.
A microservices architecture enables three critical benefits:
With those benefits in mind, what would it look like if we re-imagined integration, which is typically deployed in centralized silos, with a new perspective based on microservices architecture? That’s what we call an “agile integration.”
People and process: Decentralized integration ownership
SOA tended to result in the creation of centralized integration teams and infrastructure to implement the service layer. The central team knew their integration technology well, but often didn’t understand the applications they were integrating, so translating requirements could be slow and error prone. This dependency created ongoing friction given the pace at which projects could run.
Many organizations would have preferred the application teams own their own integration, but the technology and infrastructure at the time didn’t enable that. The move to agile integration with more fine-grained, cloud-native deployment, makes it possible to simplify and automate much of the build, test and deployment pipeline. This in turn enables us to distribute the responsibility for creating and maintaining integrations beyond the central team.
This goes hand in hand with the increased importance of API management. Teams need to be able to manage the exposure of their own APIs, rather than referring to a centralized team of API gateway specialists to do that. API management enables application teams to perform that configuration themselves, and also populate developer portals with definitions of the APIs and events such that consumers of these can discover them easily, and self-onboard to use them.
Furthermore, microservice-based applications often prefer to communicate internally using events. The increased decoupling further improves the team’s autonomy to make changes independently. Agile integration also considers how best to enable teams to publish and consume event streams within and across application boundaries.
Architecture: Fine-grained integration deployment
Traditional integration is characterized by the centralized deployment of integrations in the ESB pattern. In this model, all integrations are deployed to a unique heavily nurtured pair of integration servers and can result in a bottleneck for projects. Any deployment to the shared servers runs the risk of destabilizing existing critical interfaces. Furthermore, teams running individual projects can’t independently choose to upgrade the integration middleware to gain access to new features.
Using the same concepts as a microservices architecture, we could break up the enterprise-wide ESB into smaller, more manageable and dedicated pieces. These “fine-grained integration deployment” patterns provide specialized, right-sized containers that improve agility, scalability and resilience, and look very different than the centralized ESB patterns of the past. By taking advantage of the cloud native infrastructure we will discuss next, it is much easier to set up automated pipelines that do not require specialists skills to run. This radically simplifies the mechanical deployment process leaving more time to focus on the business value of the integration.
Fine-grained integration deployment draws on the benefits of a microservices architecture. Let’s revisit what we listed as microservices benefits in light of fine-grained integration deployment:
Technology: Cloud-native integration infrastructure
Agile integration requires that integration capabilities be deployed using cloud-native principles. This enables a cloud-native platform such as one based on Kubernetes to perform much of the deployment, scaling and availability tasks in a standardized way. Agile integration normalizes the integration skillset required to set up and look after a given capability and ensures that capability is portable across a multicloud landscape.
Adopting a cloud-native approach impacts the ways your DevOps teams will interact with the environment and the solution overall to increase efficiency as more solutions are moved to lightweight, portable architectures.
In addition, the cloud-native approach and technology simplifies automation of build and deployment pipelines and enables cleaner setup and tear down of testing environments. These benefits contribute to much shorter build-release cycles and a significant increase in productivity.
Moving to a cloud-native approach is a significant change. Some high-level characteristics include:
If the above principles are adhered to, container orchestration platforms like Kubernetes provide auto scaling, load-balancing, deployment, internal routing, reinstatement and more in a standardized way to significantly simplify platform administration.
Modern integration portfolios must be able to embrace the core principles and standards of a cloud-native approach. Integration runtimes, API management, event streams and messaging capabilities must all leverage container orchestration platforms to provide the portability, operational consistency and automation required by today’s solutions.