Building a DevOps pipeline for an API Connect and Microservices architecture

Share this post:

There are many lessons to learn from a DevOps implementation in a greenfield development project, where 20 new cloud native microservices were built and deployed to Bluemix. Our client’s business partners created end-user applications, driving mindshare and loyalty to our client’s brand.

In this post, we’ll talk about how we used API Connect to socialize, document, and govern access to these microservices consumed by business partners. We’ll examine how we used them to build high value web and mobile applications that leveraged the data and function provided by these microservices. We’ll also show how we evolved our development practice using Agile and DevOps techniques to streamline the deployment of these microservices and vastly reduce the time to production of updates while reducing risk.

Understanding the challenges to manage the solution

We dealt with a few issues in order to manage the solution:

  1. Management of build artifacts
  2. Testing microservice versions for promotion
  3. Configuring microservice environments
  4. Deploying artifacts to live environments while maximizing uptime
  5. Packaging APIs into meaningful, controllable sets
  6. Managing versioning and microservice compatibility (interoperability)
  7. Aligning the presented API layer with the microservice deployments
  8. Monitoring state

The surface area of the solution included approximately 20 microservices. These were deployed across the four environments, databases in each environment, and API Connect with its configuration of APIs and products. While 20 microservices may seem relatively small, the number of microservices and APIs across the environments exceeded 100 moving parts to be managed consistently in a repeatable and predictable manner. With the project’s rapid changes, this couldn’t be done efficiently without automation—even with a dedicated, skilled operations team.

Topology of microservices and API Connect on Bluemix

Microservices solution architecture

Our architecture consisted of three main layers:

  1. API Connect providing governance and a consistent access point for all microservices
  2. A set of microservices providing domain-specific features for clients and interfacing with third-party services
  3. Data storage in dashDB and IBM Cloudant

Get started with IBM API Connect.

This architecture was completely replicated in four environments:

  1. Development: Tracking the latest “development” code versions
  2. QA: The latest code release for integration test runs
  3. Pre-production: For partner integration and testing
  4. Production

We modeled these environments as separate Bluemix spaces within a single Bluemix organization. We provided project members with per-space permissions appropriate to their roles, allowing isolation of concerns and confidentiality of production information.

Bluemix architecture org chart

We provisioned a single API Connect instance in a space isolated from the environment-specific spaces. This allowed us to handle all API configurations, management, deployment, and monitoring from this central instance of API Connect. The API products defined in API Connect were parameterized so that they proxied to microservices in each of the environments. As microservices moved through the development life cycle, the APIs that provided access to the microservices aligned with and pointed to the microservices.

How we structured our DevOps toolchain

Our toolchain consisted of the following technologies:

Technologies in the project toolchain

We used Flowdock for team communication and for build and deploy notifications. We used Rally for tracking all work items for the team. Every microservice had its own GitHub repository. Commits to a repository kicked off a Jenkins build, which created both microservice artifacts (WAR files, Bluemix manifests) and API Connect configuration artifacts (YAML files). UrbanCode Deploy (UCD) deployed microservices and APIs to Bluemix, which provided the runtime environment. We used Runscope for health monitoring of the APIs, microservices, and some downstream components. API Connect provided valuable insight into the stability and performance of the microservices, which highlighted opportunities for improvements around response time and latency.

In future articles, we will focus on the details of the Code, Deliver, and Run steps in our toolchain and show how they addressed the challenges listed.

This is a complex system. To help you better understand it, we’ve split up what we’ve learned. Stay tuned for the next in a series of articles covering the following topics:

  • Versioning and Swagger for documentation and code skeleton generation
  • Deep dive into API DevOps
  • Deep dive into microservice DevOps

Learn more about microservices on Bluemix.

More Trending Stories

Centralized Configuration for Spring Apps on Bluemix Kubernetes

In this tutorial, we'll adapt the Centralized Configuration guide from the Spring website to be deployed to Bluemix Kubernetes. We'll host our configuration files in a Github repository. Before getting started, you should follow the guide or download the complete version.

Continue reading

Monitoring & logging for IBM Bluemix Container Service with Sematext

In this blog post we discuss how Sematext integrates with IBM Bluemix Container Service to provide monitoring and logging visibility of your containerized applications, as they run in production. In the sections below, we demonstrate how to set up a Kubernetes cluster in Bluemix and how to set up Sematext in this cluster. IBM Cloud has monitoring and logging capabilities in the platform, but we know our customers operate in a multi-cloud or hybrid cloud environment and we are very excited to partner with Sematext, enabling operational consistency across those environments. We worked with Alen Komljen, an Automation Engineer from Sematext, to create the following content and perform the technology validation.

Continue reading

99.95% availability. Balancing release velocity and reliability

Availability and reliability are rarely at the front of developers minds when delivering new applications on Bluemix. The ease and speed of creating and deploying new features is very seductive.

Continue reading