Master continuous integration and delivery with the IBM Devops Toolchain
While our Bluemix Developer Advocate team continues working on the Logistics Wizard demo, we wanted to take a moment to share some key aspects of our development process. In this post, I’ll take a step back from our implementation and talk about how we use GitHub, Travis CI, and our custom Devops Toolchain to streamline the whole development process for continuous integration and delivery.
What’s the Logistics Wizard demo we’re working on? It’s an end-to-end supply chain management solution that showcases hybrid cloud integration, microservices, real-world predictive analytics, and much more. You can learn more about it in this earlier post by Frederic Lavigne.
Architectural Chaos Theory
The demo consists of multiple applications and services working in conjunction to deliver the end-to-end experience. Just a quick look at the overall architecture diagram can make you dizzy.
Bluemix to the rescue
Bluemix simplifies the task of managing such a system, but the decisions of what tools and techniques to use for development and deployment are left up to the team. To make development simpler, we store and manage each application’s source code in its own git repository. That means that the applications are loosely coupled every step of the way, and we need a way to bring them together in the form of a release. This post details how we use GitHub, Travis CI, and our custom Devops Toolchain to perform continuous integration and delivery and streamline this process.
It all starts with
dev branch stores the source code of our most up-to-date application running on Bluemix. All new work originates from this code base, whether it’s a large feature or a tiny hotfix. When a developer picks up an issue (such as Issue #123), he or she creates a new feature branch — let’s call it
feature_123 — off of the code base in
dev. From here, the developer is free to do whatever is necessary to get that feature implemented. Pull several all-nighters. Write everything in Vim. Doesn’t matter, as long as that branch is ultimately pushed upstream with working code. This is where things get interesting.
feature_123 branch now needs to be merged with
dev, so the developer opens a new pull request. Once the PR is created, Travis CI runs a build on the pull request and executes the unit tests to ensure that nothing has broken.
The results of this build are displayed directly on the PR page in GitHub, letting the team know the changes can be safely merged. Another developer then manually reviews the PR and gives the obligatory “LGTM!” if he or she approves. Otherwise, the committing developer must update the code and continue this cycle until it is accepted. Once the PR is approved, the developer can go ahead and merge with
dev. If multiple commits are included, we recommend squashing and merging to help maintain a clean and traceable git history.
Immediately after the merge commit happens, the toolchain comes into play. Our toolchain listens for commits to
dev, initiating a build and deploy when this occurs. The deployed dev app gives our designers and product owner a chance to review the changes and ensure the user interface and experience are as designed. If this final review passes, the issue is closed and awaits the day it finally sees the light of production.
The last merge
master branch has remained in pristine condition, always reflecting the code in production. However, there comes a time in every development cycle when the
master branch must grow up and ingest changes made to its little
When the team is ready to cut a release, we double-check that all the features we targeted for the release have been merged to
dev and that it is in a stable state (in other words, the most recent Travis build has passed and the dev app looks good). We now go about the same process as above, only this time there are two differences. If the Travis build fails, the
master branch will not allow a merge until the issue has been fixed. This prevents easily detected issues from ever making it to production. We also do not do a squash and merge here as we want our git history for
master to mirror the
dev branch. Once the merge completes, the toolchain picks up the new merge commit and begins the final leg of the process.
Toolchain will now run the same unit tests executed by Travis, but this time on the same infrastructure that the app runs on in production. This allows us to catch potential issues that were not seen in the Travis builds. We also run coverage tests here and send the reports to Coveralls.io to make sure we are consistently creating unit tests that keep up with our expanding code base.
If all looks good, we initiate a blue-green deployment to production. This technique requires our new app instance to start successfully before bringing down our old instance, providing us with a zero-downtime deployment.
Aside from CI & CD, we can use the Toolchain for a number of things. Our team has made some custom changes to allow users to configure application environment variables in the setup stage. For instance, our application sends emails using Google’s SMTP server and several email accounts, each of which is used in a separate environment. Using our own custom toolchain, we collect this information on setup and propagate it to the respective environment variables. Here is how we do it:
Create a custom yaml file called
email.jsonto act as the input form.
toolchain.ymlfile to collect these fields and use them in the pipeline envs.
Ingest the environment variables in the deployment pipeline and use them for commands like updating your production app’s environment variables.
The email settings are added to the pipeline configuration form of the toolchain setup.
In this way, we can collect custom information during the toolchain setup and deploy new versions of the application system without the user touching any code!
This end-to-end process seems complex at first and takes some time to set up, but it pays for itself in several key ways. This approach lets us:
Work simultaneously on different features through dev-based branching
Detect issues early because continuous integration tests changes at each commit
Automate dev builds and deployments for visual QA
Complete full unit and code coverage tests before code reaches production
Automate deployment to production for continuous delivery
If you have any questions about our process or comments on how your team does continuous integration and/or delivery, post in the comments below!
Our toolchain was initially constructed from the microservice toolchain template, which includes a few additional features that we stripped out.
For a detailed look at our end-to-end development process, check out the detailed breakdown in our wiki.
If you have yet to sign up for Bluemix, what are you waiting for? Sign up today and start developing the next killer app.