October 2, 2014 | Written by: Adalberto Medeiros
Share this post:
This is the final post in a series on continuous integration (CI) environments in OpenStack. In previous posts (Part I, Part II), I’ve discussed how a CI environment works and why you should consider creating your own instead of relying on the environment provided by OpenStack. Here I’ll talk about how to accomplish that.
What services do I need?
This repository provides a huge list of services and tools that can be used to set up a CI environment. For the purpose of this post, I will point out the main services that you should have knowledge of and that you will need to replicate in your environment.
• You will need your own Git repository, like Github.
• You will need a Puppet server to rule all nodes and services and host the puppet recipes.
• Jenkins is the build engine that will run the jobs.
• Gearman is the job server, which will decide which job will run on the available nodes (single-use virtual machines).
• Jenkins Job Builder automates the job creation for the builder. It is an OpenStack Infrastructure project.
• Zuul serves as the gating system that will handle the patches and decide which set of jobs to run. It is an OpenStack Infrastructure project. See this link for the Zuul reference manual.
• Nodepool is the node launcher for jobs. It creates and manage single-user virtual machines. It is also an OpenStack Infrastructure project.
• Config is not a service, but a repository that contains the configuration files and puppet recipes you will use to automate the configuration for your own services.
You will need at least four servers to host these services:
- One for Zuul and the Gearman daemon
- One Jenkins (with Gearman plugin) and the job builder
- One for the puppet master
- One for Nodepool
You also need a Git repository to store your puppet configuration and other servers for services like logging, tracking and monitoring.
How can I put it all together?
The glue here is basically the puppet recipes. Most of those recipes are located in the config repository for openstack/infra. You will need to create your own Git repository with a module for your service recipes. Your <your enterprise>_openstack-project will basically copy or use the recipes from the original config/modules/openstack_project. For instance, you might have your own zuul.pp recipe in your module, with the credentials and keys for the system and other customizations. You might create your own jenkins_job_builder recipes to manage your jobs according to your needs.
Here is an example of projects managed by the OpenStack Infrastructure team, with puppet scripts.
It is also important to learn how to use the Hiera and Hieradata to add your own account passwords, keys, names and other specific data. Moreover, you will need to understand how the YAML scripts work for your macros and job builder. For instance, you can create the three jobs we mentioned before copying the scripts that are under modules/openstack_project/jenkins_job_builder to your own repository
One challenge here is maintaining your services and configuration while also keeping up with OpenStack Infrastructure community updates. In a future post, I will discuss the effort to synchronize your own repository with the one in the community.
Further, you might also need to provide a couple of other scripts to make the jobs work. For the jobs noted in my previous post, you need the devstack-gate scripts to set up the variables and run DevStack and Tempest jobs. Again, you would can create your repository [your company]-devstack-gate and copy the devstack-gate scripts (devstack-vm-gate.sh, devstack-vm-gate-wrap.sh) and the necessary YAML configuration, modifying them according to your needs.
Thus, we have covered the general idea of having your own continuous integration environment, as well as the knowledge and tools you can use to make it happen.
Here are some links that can provide step-by-step assistance to build your environment.
I recommend starting with a puppet and creating the puppet-master, along with your own config project. Once you understand this, go ahead and configure Jenkins, Nodepool and Zuul servers, use your puppet recipes and try to use as much as you can from the community. Being active in the community and checking the #openstack-infra channel on IRC Freenode are great methods to find resources and help.
Let me know if this series has been helpful. Feel free to contact me on Twitter or LinkedIn if you’d like to continue the conversation.