Contents


Agile DevOps

Continuous software delivery in the cloud

Leverage an open Continuous Delivery platform

Comments

Content series:

This content is part # of # in the series: Agile DevOps

Stay tuned for additional content in this series.

This content is part of the series:Agile DevOps

Stay tuned for additional content in this series.

When implemented effectively, Continuous Delivery (CD) embraces the DevOps philosophy of collaboration between traditionally separate teams: development and operations. CD is the automated implementation of the build, deploy, test, and release process. With CD, software is always in a releasable state. With every change to any part of the software system — infrastructure, application code, configuration, or data — a release candidate goes through a delivery pipeline where the software is created, environments are built, databases are constructed, tests are run, and software is deployed. Many tools make up a platform capable of moving the software system along this pipeline.

This article introduces an open source platform for CD called OpenDelivery, developed by Stelligent. OpenDelivery combines open source tools, a cloud infrastructure, and scripts. It supports the concept that the whole software system is code. The scripts are all versioned in a GitHub version-control repository. OpenDelivery uses the numerous tools listed in Table 1 to create the platform, and it supports Rails, Grails, and Java™ development on Linux™. In this article, I'll describe the basic steps for running the platform, using the delivery pipeline, and running jobs to provision environments and deploy applications.

Table 1. Tools used by the OpenDelivery platform
ToolDescription
Amazon Web Services (AWS) CloudFormationOpenDelivery uses CloudFormation — a template language for describing the provisioning of AWS resources — for the full infrastructure automation of Jenkins environments and the configuration of AWS resources for target environments.
Amazon Elastic Compute Cloud (EC2)OpenDelivery uses EC2 for all its compute instances.
Amazon CloudWatchOpenDelivery uses CloudWatch to monitor some basic performance metrics and sends Simple Notification Service (SNS) notifications when thresholds are exceeded.
Amazon Route 53Route 53 is a highly available and scalable DNS that OpenDelivery uses to set domains to endpoints based on user preferences.
Amazon SimpleDBSimpleDB is a NoSQL database that is highly available and scalable. OpenDelivery uses it to store pipeline configuration such as IP addresses and CloudFormation stack names.
Amazon Simple Notification Service (SNS)OpenDelivery uses SNS to send messages when it creates environments.
Amazon Simple Storage Service (S3)OpenDelivery uses S3 to store ephemeral files used by the platform.
CapistranoCapistrano is a Ruby-based deployment-specific DSL that OpenDelivery uses to deploy applications to environments.
GitHubGitHub hosts the OpenDelivery source code. Users of OpenDelivery must also use a GitHub account for their application code.
JenkinsJenkins is a Continuous Integration (CI) server. OpenDelivery uses Jenkins as its platform for running all builds and deployments of application code, configuration, data, and infrastructure changes.
PuppetPuppet is an infrastructure automation tool with its own domain-specific language (DSL). All of the infrastructure automation for the target environments is written in Puppet and called by CloudFormation.
RubyMost of the scripts that automate the pipeline are written in Ruby.

Prerequisites

The three prerequisites for successfully installing the OpenDelivery platform are:

  • CloudFormation: Sign up for AWS CloudFormation to gain access to all of the required services that OpenDelivery uses.
  • GitHub: You must have a GitHub account. It's free to sign up with GitHub.
  • Route 53: You need to set up a hosted zone in Route 53 for a domain that you own. This domain will be used in your OpenDelivery scripts to address the resources that are created as a part of the platform.

See Related topics for a link to the OpenDelivery Quick Start Guide. Be aware that in using OpenDelivery, you will be charged AWS fees for resource usage.

Provisioning the continuous-integration platform

The first step in setting up the OpenDelivery CD platform for your projects is to provision the Jenkins CI server in AWS. To begin, run the CloudFormation setup template. You will be prompted to enter the following parameters:

  • ApplicationName: The CNAME prefix for the Jenkins CI environment you're creating. For example http://ApplicationName.domainname.com.
  • GithubUsername: Your GitHub username.
  • ProjectName: The name of your GitHub project. For example, if my project's GitHub URL is https://github.com/stelligent/continuous_delivery_open_platform, then my ProjectName is continuous_delivery_open_platform.
  • Email: The email address where you'd like messages sent.
  • HostedZone: The domain name you own and have created a hosted zone for in Route 53.
  • GithubOrganization: The name of your GitHub organization. For example, if my project's GitHub URL is https://github.com/stelligent/continuous_delivery_open_platform, then my GithubOrganization is stelligent.
  • GithubPassword: Your GitHub password for the GithubOrganization and ProjectName.
  • KeyName: The name of your EC2 key pair. Do not include any file suffixes.
  • InstanceType: The EC2 instance type. Use the default if you are unsure.

After you complete the wizard, the platform will take about 20 minutes to build. When it's done building, go to the CloudFormation Console and select the check box next to the stack with jenkinsstack in its name. Then, click the Outputs tab. Copy the value from the Domain key row and enter this value in your web browser to launch the newly created Jenkins CI server running in AWS. You'll see the initial Jenkins configuration, shown in Figure 1:

Figure 1. Initial Jenkins configuration after it's provisioned using CloudFormation
Screenshot of the Jenkins CI server showing the AddUserToEmailGroup, Build, BuildAMI, CapistranoModuleBuild, CommonStepDefinitionGemBuild, CreateTargetEnvironment and DeployApplication Jenkins jobs. It also shows two tabs: DeliveryPipeline and Email Group Administration.
Screenshot of the Jenkins CI server showing the AddUserToEmailGroup, Build, BuildAMI, CapistranoModuleBuild, CommonStepDefinitionGemBuild, CreateTargetEnvironment and DeployApplication Jenkins jobs. It also shows two tabs: DeliveryPipeline and Email Group Administration.

A CD pipeline

With OpenDelivery, a standard delivery pipeline is provided out of the box. It runs the StartDeliveryPipeline, Build, CreateTargetEnvironment, and DeployApplication Jenkins jobs in sequence. If any of these jobs fails, the pipeline fails and software is not deployed.

An example of a delivery pipeline — as represented in Jenkins — is shown in Figure 2:

Figure 2. Delivery pipeline as represented in the Jenkins CI server
Screenshot of a delivery     pipeline as represented in the Jenkins CI server. There are three separate stages the software goes through. First it runs any setup using the StartDeliveryPipeline job after a build is run through a continuous integration process, then it creates a target environment with the CreateTargetEnvironment job and finally it runs the third stage to deploy the application into the target environment as represented in the DeployApplication job. These three stages run as a unit of work in which if any of the jobs fail, it all fails.
Screenshot of a delivery pipeline as represented in the Jenkins CI server. There are three separate stages the software goes through. First it runs any setup using the StartDeliveryPipeline job after a build is run through a continuous integration process, then it creates a target environment with the CreateTargetEnvironment job and finally it runs the third stage to deploy the application into the target environment as represented in the DeployApplication job. These three stages run as a unit of work in which if any of the jobs fail, it all fails.

In Figure 2, the software goes through three stages. First, after a build is run through a CI process, Jenkins runs any required setup using the StartDeliveryPipeline job. Then it creates a target environment with the CreateTargetEnvironment job. Finally, it deploys the application into the target environment as represented in the DeployApplication job. These three stages run as a unit of work in which if any of the jobs fails, the pipeline fails.

By establishing a delivery pipeline such as this, you assess a release candidate for risk — as more and more automated processes occur — with each step in the pipeline the software goes through. This means that once it successfully completes the pipeline, the release candidate could potentially be released to production if the business chooses to do so. You can also configure the pipeline to deploy to production.

Provisioning target environments

In the context of the OpenDelivery platform, environments are defined as the operating system on EC2 instances along with other provisioned resources — such as security groups, domain configuration, and storage. Once an environment is provisioned, software can be deployed to it. The Jenkins job that is primarily responsible for running the scripts that create these environments is the CreateTargetEnvironment job.

The delivery pipeline in OpenDelivery creates environments from scratch by calling a CloudFormation template. The template executes Puppet scripts (calling puppet apply). (For more information on Puppet, see "Agile DevOps: Infrastructure automation.")

Listing 1 shows the portion of the CloudFormation template that calls Puppet:

Listing 1. CloudFormation snippet from production.template that calls Puppet
"# Install Ruby 1.9.3\n",
"rpm -Uvh /tmp/ruby-1.9.3p0-2.amzn1.x86_64.rpm\n",

"# Install Puppet 3.0.1 from Rubygem\n",
"gem install puppet --no-rdoc --no-ri\n",
"groupadd puppet\n",

"# Run Puppet\n",
"puppet apply --modulepath=/home/ec2-user/modules /home/ec2-user/manifests/site.pp\n",
...

OpenDelivery provides these scripts in its GitHub repository. You can alter the standard scripts for the servers and configuration in use in your infrastructure.

Running deployments

Automated deployments run in Capistrano. You can use Capistrano to deploy to multiple types of development and OS platforms. These deployments are run from Jenkins jobs onto environments created from the CloudFormation and Puppet scripts described in the preceding section. The Jenkins job that is primarily responsible for running the scripts that run the deployment is the DeployApplication job.

The snippet in Listing 2 shows a Capistrano script that runs preconfigure and deploy tasks:

Listing 2. Capistrano snippet from deploy.rb that resets an environment and runs a deployment
task :preconfigure do
  run "cd #{deploy_to} && sudo rm -rf #{deploy_to}/#{artifact_name}"
  config_content = from_template("config/templates/s3_download.erb")
  put config_content, "/home/ec2-user/s3_download.rb"
  run "sudo chmod 655 /home/ec2-user/s3_download.rb"
end

task :deploy do
  run "sudo ruby /home/ec2-user/s3_download.rb --outputdirectory #{deploy_to}/ \
    --bucket #{s3_bucket} --key #{artifact}"
  case
  when language == "rails"
    run "cd #{deploy_to} && sudo tar -zxf #{artifact}"
    run "sudo chown -R ec2-user:ec2-user #{deploy_to}/#{artifact_name}"
  end
end
...
case
when language == "rails"
after "preconfigure", "deploy", "database:configuration", "stripe:initializer", \
      "bundler:install", "database:migration", "virtualhost:configuration", \
      "whenever:set", "postconfigure", "restart"

You'll find additional Capistrano recipes in a subdirectory within the directory named deployment in the OpenDelivery GitHub repository. In Listing 2, examples of these recipes are stripe (stripe.rb) and virtualhost (virtualhost.rb). You can add, modify, or delete recipes based on your deployment needs.

Continuous testing

The OpenDelivery platform includes essential Cucumber acceptance tests that run against the infrastructure (the CI and target environments) and deployments. These tests run every time that changes are applied to scripts that provision environments or run deployments from the CreateTargetEnvironment and DeployApplication jobs, respectively. The example in Listing 3 is from a Cucumber script in the OpenDelivery platform that verifies that certain server components are installed and operational:

Listing 3. Snippet from a Cucumber script in OpenDelivery (production.feature) that runs automated tests against an environment
Feature: Scripted Provisioning of Target Environment
As a Developer
I would like my target environment provisioned correctly
so I can deploy applications to it

Background:
Given I am sshed into the environment

Scenario: Is the proper version of Postgresql installed?
When I run "/usr/bin/postgres --version"
Then I should see "8.4"

Scenario: Is the proper version of Apache installed?
When I run "/usr/sbin/httpd -v"
Then I should see "2.2"

...

You can modify or extend the supplied tests to test more environment or deployment configuration.

Build, deploy, test, and release with every change

In this article, you learned how OpenDelivery leverages open source tools and configuration to create a platform for continuous software delivery in the cloud, and how you can take advantage of the platform. Continuous delivery works best for development and operations teams that work together. The next installment talks about how to break down departmental silos and create cross-functional DevOps teams.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=DevOps, Java development, Open source, Cloud computing
ArticleID=853833
ArticleTitle=Agile DevOps: Continuous software delivery in the cloud
publish-date=01082013