September 23, 2014 | Written by: Adalberto Medeiros
Share this post:
Though OpenStack is a large and complex project at present, its development community is structured very well in comparison to other large open source projects or proprietary development practice communities, to the degree that OpenStack is even considered a role model. It is particularly interesting to note how this community has developed an infrastructure for code review, testing and continuous integration.
In this series of posts, I will discuss continuous integration (CI), looking at how the OpenStack community adopted this practice and the advantages of integrating your own CI environment into OpenStack. I will give a brief description of each of the main components and what you need to know in order to kick off your own services. This is not intended to be a tutorial, but I will provide links to several useful tutorials in my last post.
The OpenStack CI environment includes a series of projects that contain unit tests, functional tests, integration tests, a patch review system and automatic builds. Most of those projects are hosted within the OpenStack Infrastructure project and managed by the OpenStack Infrastructure Team, but tools like Tempest, DevStack and Grenade are essential to provide required automation and the environment that makes CI effective. Those systems are integrated with the build and patch review system (Gerrit), in order to provide the appropriate verification for every change proposed to OpenStack projects. This can ensure that usability, policies, reliability, security, dependencies and other environmental characteristics are properly maintained. Behind all this, you need a cloud infrastructure to run it all.
The first things to understand are how the patch review process works in OpenStack and what main tools are used to automate it.
Continuous integration flow
In the simplest terms, there are five steps in the process:
1. Someone submits a patch for review, which is then managed by Gerrit. Any configured CI environment may be listening for new patches on this service by a configured Zuul scheduler.
2. The Zuul gate receives those changes, and then checks for dependencies and merges changes.
3. The changes will then be handled as jobs by the build automation system Jenkins. Behind that. other tools will create the images and virtual machines to run the jobs (Nodepool) and automate the creation of the jobs that need to be tested on the environment (Jenkins Job builder).
4. Each project hosted by OpenStack has a set of job types. Depending on the specific purpose , each job may use existing open source tools and OpenStack projects such as Tempest and Devstack. A job can be used for the following purposes:
• Unit tests
• Code language checks (Python, pep and pylint)
• Integration tests (using Tempest and DevStack)
• Upgrade tests (using Grenade, DevStack and Tempest)
5. Finally, job results (failures and successes) are reported back to Gerrit and may influence patch reviews such as voting, verifying or even merging the patch code.
This whole procedure is automated. I mentioned a few services that are used in the process (such as Gerrit, Zuul, Nodepool, Jenkins), but there are other parts that you’ll need to make the whole system work. Those tools will be presented in the third post of this series.
Once you have a general understanding of how the CI system works, we can discuss why your company or group of developers may want to create your own CI environment, what you will need in order to accomplish that and what size infrastructure you should consider for your purposes.
Feel free to contact me on Twitter or LinkedIn, or leave a comment below to let me know of your own experiences with continuous integration environments