Automation for the people: Speed deployment with automation

Leverage automation to move software through different environments quickly

Automated builds aren't just for development teams — they can be extended to facilitate moving software from development all the way into production. In this installment of Automation for the people, automation expert Paul Duvall describes how to use Ant with Java™ Secure Channel for remotely deploying software into multiple target environments.

Share:

Paul Duvall (paul.duvall@stelligent.com), CTO, Stelligent Incorporated

Paul DuvallPaul Duvall is the CTO of Stelligent Incorporated, a consulting firm and thought leader in helping development teams optimize Agile software production. He is the co-author of the Addison-Wesley Signature Series book, Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley, 2007). He also contributed to the UML 2 Toolkit (Wiley, 2003) and the No Fluff Just Stuff Anthology (Pragmatic Programmers, 2007).



08 January 2008

Also available in Chinese Russian Japanese

About this series

As developers, we work to automate processes for users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

Have you ever noticed that for many teams, moving software from development environments into production is often an afterthought? I've run across teams with deployment timelines that range anywhere from weeks to even months — time wasted, in my opinion. Why not leverage the same automation that's found in automated builds and drastically reduce the time spent on deployment configuration issues and thus, reduce those infrastructure inefficiencies?

Just think about it for a second: The inefficiencies that may exist in deploying software mean a delay in the time it takes to put your killer application into the hands of users. What's worse, some people think deployment is something like pulling a band-aid off (it'll only hurt for a second) when in fact on most projects, the deployment pain lingers and it occurs again and again with every delivery.

Besides a delay in going live, an ineffective deployment infrastructure makes teams less adaptive to change, which often tempts teams into throwing too much functionality into a release (because releases seldomly occur). It becomes a vicious cycle: The business wants software in the hands of users quickly, but that takes too long, so everyone plans for big bang releases to maximize the window of opportunity.

Deployment can be painless

The basic process of deployment includes compilation, integrating data changes (such as database tables), remotely deploying a distribution (JARs and WARs, for example) to other machines, and managing those machines' resources. Nevertheless, there's actually much more that can be automated as well, including creating installation media, testing, generating user documentation, and so on. In this article, I'll keep to the basics and demonstrate how all of these processes can become a part of your automated build process. Specifically, you'll learn the following processes:

  • Deploying binaries to remote machines
  • Externalizing configuration properties
  • Remotely updating a MySQL RDBMS
  • Remotely configuring Jakarta's Tomcat Servlet container

By automating these processes, you will ultimately deliver software faster to your customers with fewer headaches.


The requisite tools

The central tool for automating deployment is a build script; in my case, Ant will be the driver. My Ant script will use properties files (which are unique to targeted environments like staging and production, for example), interface with a MySQL database via Ant's sql task, and use Java Secure Channel (JSch) to copy files to remote machines (via Secure-Copy protocol (SCP)) and to stop and start Tomcat services (via SSH).

Figure 1 illustrates a high-level architectural view of this process. It is critical that all software assets are stored in a version control repository because an ideal build process begins by checking out source code (and configuration files) and compiling and packaging components locally, then remotely executing SQL statements, followed by deploying the distribution and restarting Tomcat.

Figure 1. High-Level build architecture for remote deployment
High-Level build architecture for remote deployment

You can automate all of these processes and, by doing so, can trigger deployment with a single command or mouse click, or even schedule them to execute without human intervention. Is that exciting or what?


Externalize those properties

Configuration values such as file locations, host names, database names, and port numbers that may vary from one target environment to another should never be hardcoded (in source code, for instance). These attributes are more appropriately managed in .properties files. By externalizing these properties, you can use the same build script to compile in one environment and deploy in other environments without modifying or recompiling source code.

The rule of properties

If a value will change from one environment to the next, put it in an external .properties file. If there is more than one reference to a value within a build script, it should go in a property within the build script itself (that is, build.xml). If you know there's only ever going to be one reference to a value, it doesn't need to go in a property (although, this is quite an assumption).

Listing 1 demonstrates a simple example of defining a property in an Ant build script that allows you to pass a .properties file as a system parameter (for example, test.properties), which contains all of the values for a particular target environment. property.file.location may resolve to a path such as C:\Documents and Settings\patrick.henry\test.properties. For example, from the command line, you can type: ant -Dproperty.file.location=C:\Documents and Settings\patrick.henry\test.properties.

Listing 1. Externalize property attributes
<property file="${property.file.location}" />

An example target environment .properties file is shown in Listing 2. The attribute values in this file should (or could) vary from one target environment to the next, but the attribute names should always be the same.

Listing 2. Example attributes and corresponding values in property file
db.database=brewery
db.username.system=root
db.password.system=sa
db.username=root
db.password=sa
db.hostname=my-hostname.domain.com
db.driver=com.mysql.jdbc.Driver
db.port=3306
db.url.system=jdbc:mysql://${db.hostname}:${db.port}/
db.url=jdbc:mysql://${db.hostname}:${db.port}/${db.database}

By externalizing property attributes and values, you can create a more flexible build and deployment architecture that can support multiple target environments.


Simplicity at the core

Deploying software into another environment shouldn't be a nasty process; it should be as easy as typing "deploy," if you ask me. Luckily, build systems like Ant make this a reality. By logically defining a workflow that executes a series of steps in sequence, you can then create a single command that invokes them.

The Ant targets enumerated in the depends attribute in Listing 3 define, at a high level, automated deployment at its finest. First, the script will remove any previously generated artifacts from a local environment (using the clean target), compile source code, remotely create a database, apply test data, start the database, and finally remotely deploy a WAR file into a Tomcat container residing in the target environment.

Listing 3. Key targets executed in remote deployment
<target name="build" 
  depends="clean, compile, refresh-database, remote-tomcat-deploy" />

Refreshing a database and remotely deploying assets isn't an easy task; however, with some clever scripting, you'll be walking tall in no time!


Automating DBAs

When setting up a target test environment, there are often a lot of manual processes occurring, such as configuring a database, inserting test data, removing old entries, and other repetitive (and often error-prone) processes. The good news is that dealing with databases during deployment doesn't have to be painful.

Data Definition Language (or DDL) statements, such as dropping an existing database, creating a database, and creating database users, followed by Data Manipulation (or DML) statements, such as insert statements, can easily be scripted and run as part of an Ant build script. What's more, these statements can be executed remotely.

For example, by passing a db.url.system property (from a target environment .properties file) as shown in Listing 4, a build script can execute SQL statements against a remote database:

Listing 4. A script to create a database and insert data
<target name="refresh-database" depends="create-database,insert-data" />
<target name="create-database">
  <sql driver="${db.driver}"
    url="${db.url.system}" 
    userid="${db.username.system}"
    password="${db.password.system}"
    src="${database.dir}/create-database.sql"> 
    <classpath>
      <pathelement location="${mysql-connector.jar}"/>
    </classpath>
  </sql>
</target>
...
<target name="insert-data">
  <sql driver="${db.driver}"
    url="${db.url}"
    userid="${db.username}"
    password="${db.password}"
    src="${database.dir}/insert-data.sql">
    <classpath>
      <pathelement location="${mysql-connector.jar}"/>
    </classpath>
  </sql>
</target>

The contents found in the insert-data.sql file shown in Listing 5 are called from the insert-data target in Listing 4. Any SQL statements, be they DDL or DML, can be executed in a similar way using Ant's sql task.

Listing 5. SQL statements which insert data
insert into beer(id, beer_name, date_received) values 
  (1, 'Samuel Adams Lager','2006-12-09');
insert into beer(id, beer_name, date_received) values 
  (2, 'Guinness Stout','2006-12-29');
insert into beer(id, beer_name, date_received) values 
  (3, 'Olde Saratoga Lager','2007-02-14');
insert into beer(id, beer_name, date_received) values 
  (4, 'Sierra Nevada Pale Ale','2007-05-14');

Now that I've updated a remote database, the next logical step is to deploy some assets to a remote environment running Tomcat.


Distribution and deployment

Implementing remote deployment isn't all that different than local deployment, it just requires a different channel by which things can be securely copied from one place to another (from a build machine to a target environment). In most enterprises, security is a priority, so simple FTPing and telnet don't always work. In this case, SCP and SSH can easily do the job. Leveraging these channels is easy through Ant; in fact, I often use the sshexec and scp tasks from JSch to remotely copy files and run commands on remote machines.

Going to production

Although certain software applications (Software as a service, or SaaS, for example) are changing the frequency of deployments, going to production is no trivial activity. You'll want to add application and database rollbacks to ensure you can get the software system back into its prior state. Such additions can be the difference between making and losing millions of dollars. Just like testing the software system itself, the automated deployment process must be rigorously tested as well.

Securely copying files with SCP

SCP provides the capability to securely copy assets from one machine to another. There are different tools that support SCP. Logically, in Ant, JSch will use SCP to copy assets (like JAR files) without human intervention, or as I'm rather fond of saying: automatically.

In Listing 6, the scp task (provided by JSch) copies (in this case) a WAR file from a build machine to a remote machine. The JSch library (jsch-0.1.36.jar), by the way, must be in Ant's classpath to make use of the scp task.

Listing 6. Securely copying a WAR file from one machine to another
<target name="copy-tomcat-dist">
  <scp file="${basedir}/target/brewery.war" 
  trust="true" 
  keyfile="${ssh.key.file}"
  username="${ssh.username}"
  passphrase=""
  todir="${ssh.server.username}:${ssh.server.password}@${ssh.server.hostname}
  :${tomcat.home}/webapps" />
</target>

When calling the scp task, you need to provide the location of the local file you wish to copy, along with the location of a local private SSH key file (ssh.key.file in Listing 6, used for secure authentication). Finally, you will also need to provide a location on the remote machine (ssh.server.hostname in Listing 6) where you want scp to place the local file (or files).

Remotely invoking processes with SSH

Just like with SCP, running commands on a remote machine often requires a secure mechanism, such as SSH. In Listing 7, I use the JSch sshexec Ant task to stop and then restart a Tomcat container residing on a remote machine. Presumably, this is the same machine to which the build process just securely copied a series of assets, like WAR files.

Listing 7. Stopping and starting a remote Tomcat instance
<target name="remote-tomcat-stop>
  <sshexec host="${ssh.hostname}"
  port="${ssh.port}"  
  keyfile="${ssh.key.file}"
  username="${ssh.username}"
  passphrase=""
  trust="true"
  command="${tomcat.home}/bin/shutdown" />
  <sleep seconds="${sleep.time}" />
</target>
...
<target name="remote-tomcat-start">
  <sshexec host="${ssh.hostname}"
  port="${ssh.port}"
  username="${ssh.username}" 
  passphrase=""
  trust="true"
  keyfile="${ssh.key.file}" 
  command="${tomcat.home}/bin/startup" />
  <sleep seconds="${sleep.time}" />
</target>

In Listing 7, I provide the machine name where Tomcat resides, the port number for Tomcat (usually this will be 8080), a private key file (ssh.key.file) so that the build script can securely access the machine, and a specific command to execute. In this case, you can see that I'm invoking the shutdown command followed by the startup command.

Logically, with this last series of steps, I'm done: I've configured a remote database, I've moved a Web application to a remote machine, and bounced an instance of Tomcat. Thus, a new version of an application is now running for people to test or even to use normally.


Automated deployment for the people

Hopefully I've shown you how automating a deployment process can easily become a reality. Moving software from development and into the hands of your customers doesn't or shouldn't need to be a manual process nor does it need to be distinctly separated from the development team's build process. In fact, with this approach, releasing software can be as simple as pressing a button, which, of course, can considerably impact a development team's ability to deliver features frequently.

Resources

Learn

Get products and technologies

  • JSch: Download Java Secure Channel for secure communication.
  • Ant: Download Ant and start building software in a predictable and repeatable manner.
  • Tomcat: Download the Tomcat Web container and start building Web applications quickly.
  • DbUnit: Download DbUnit and use XML to manage data manipulation.
  • Example code: Example scripts from the article.

Discuss

  • Improve Your Code Quality discussion forum: Regular developerWorks contributor Andrew Glover brings his considerable expertise as a consultant focused on improving code quality to this moderated discussion forum.
  • Accelerate development space: Regular developerWorks contributor Andrew Glover hosts a one-stop portal for all things related to developer testing, Continuous Integration, code metrics, and refactoring.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=280732
ArticleTitle=Automation for the people: Speed deployment with automation
publish-date=01082008