Part 1 of this series began by looking at a Software as a Service offering called SmallPayroll.ca. The application was picked up and moved to Amazon Elastic Compute Cloud (Amazon EC2). Then, in Part 2, the application was made more robust by adding redundancy, backups, and more reliable disks.
Now, it's time to look beyond stability and toward scalability. Because servers are rented by the hour, would it make more sense to run extra servers only when needed? What about asynchronously processed jobs?
Content distribution networks (CDNs), which cache static content on the edge of the network, used to be costly. Cloud computing has made CDNs accessible to even the smallest site. There's a big performance boost to be had by using a CDN, so let's set one of those up, too.
A key element of growing your infrastructure dynamically is that a new instance is going to have to attach itself to the production environment without outside intervention. It would be possible to continuously re-bundle an Amazon Machine Instance (AMI) after each change, such as a code deployment or database change. However, a bit of scripting means that a server can deploy itself without too much work.
SmallPayroll.ca has a deployment process like most other applications:
- Pull the code from the repository, such as Subversion or git.
- Start the application servers.
- Verify that the application started correctly.
- Add the new server to the load balancer pool.
The instance will also have to configure a database server in /etc/hosts.
Pulling the code from the repository
The SmallPayroll.ca code base is stored in a Subversion repository. Subversion is a source code control system that tracks changes to the code base and allows developers to branch and merge code to work on features in a separate environment.
At the simplest level, a Rails application can run right out of a
checked-out copy of the source code. When changes are made, the production
server performs an update and is restarted. A Ruby Gem called
Capistrano manages these deployments in a way
that allows for centralized control of multiple servers and easy rollback
if problems are encountered.
Rather than checking out the code by hand, a new server bootstraps the
Capistrano process. This is more work up front,
but it means that Capistrano can later manage
the server quite easily. Listing 1 shows the contents of the initial
Capfile, which will be placed in the payroll user's home directory.
Listing 1. A Capfile to bootstrap the Capistrano deployment
load 'deploy'
# Where is the code? This will need to be customized!
set :repository, "http://svn.smallpayroll.ca/svn/payroll/trunk/"
set :scm_username, "DEPLOY_USERNAME"
set :scm_password, "DEPLOY_PASSWORD"
# Deploy to the local server
server "localhost", :app, :web, :db, :primary => true
# By default Capistrano tries to do some things as root through sudo - disable this
set :use_sudo, false
# Define where everything goes
set :home, '/home/payroll'
set :deploy_to, "#{home}"
set :rails_env, "production"
|
The Capfile is executed by the cap command.
Capistrano expects that it will be using the
Secure Shell (SSH) to connect to a server, even if it's the local host.
Listing 2, which is to be run as the application user, prepares the server
so that the application user can use SSH to connect to the local server as
itself and the Capistrano tasks can be run.
Listing 2. Preparing the environment
# Create the SSH directory and generate a secure key pair mkdir .ssh chmod 700 .ssh ssh-keygen -b 1024 -t dsa -q -f .ssh/id_dsa -N "" # Allow the user to SSH in to the server without a password cat .ssh/id_dsa.pub >> .ssh/authorized_keys # SSH is very fussy about permissions! Nothing should be readable # by other users chmod 600 .ssh/* # SSH in once, ignoring host keys, so that the server will save the host key ssh 127.0.0.1 -o StrictHostKeyChecking=false /bin/false |
Listing 2 is divided into three sections:
- The first two commands create an SSH directory that's only readable by
the application user. The third command creates a 1024-bit Digital
Signature Algorithm (DSA) key
(
-b 1024 -t dsa), turns off output (-q), and specifies the name of the key and that there is no password. - Copy the user's public key to the authorized_keys file, which allows the user holding the corresponding private key to log in without a password. Make sure that no one else can read files in the directory.
- Without a saved host key, SSH and
Capistranoprompt to verify the key, which breaks the automated deployment. The last command logs in to the local server, ignoring host keys, and executes/bin/false. This is enough to save the host key.
The final step in the process creates the environment and deploys the
current version of the application. Listing 3 shows how to use
Capistrano to take care of these tasks.
Listing 3. Deploying the application with Capistrano
cap deploy:setup chmod 700 /home/payroll cap deploy |
Listing 3 runs the deploy:setup task, which
creates the directory structure underneath the application home directory
consisting of directories named shared and releases.
Each deployment goes in its own directory in the releases folder. The
shared directory is used for logs and other elements that can be shared
among separate deployments.
Before deploying the application, the second command changes the
permissions on the home directory to be 700.
This is an artifact of deploying to the home directory instead of a
subdirectory, as the setup task opens the permissions of the deployment
directory to allow group writes. The SSH daemon does not allow key-based
authentication if the home directory if people other than the user can
write to it.
The deploy task is finally run, which checks the
code out of Subversion, puts it in a release directory, and makes a
symbolic link (symlink) called current. With this symlink in
place and each deployment in a separate directory, it is easy to roll back
to previous versions of the application.
Launching application servers automatically
Even though it's possible to build a system that automatically adds more Web servers when the load gets high, it's not always a great idea. Web traffic fluctuates considerably, so you would want to make sure the traffic is going to stay around before launching another server. You will need about 10 minutes to determine whether the load you are seeing merits a new server, then another 5 to 10 minutes to launch an instance and get a new Web server into your pool. This process may not be effective for a Web server, but it can work well for job servers.
You can still grow and shrink your application servers on a schedule. For example, most users of the SmallPayroll.ca application do so from about 8:00 a.m. to 9:00 p.m. on week days. To improve response time for the registered users and reduce server costs, three servers will be run on weekdays from 8:00 a.m. to 9:00 p.m. and two servers during other times. The deployment scripts are already written: All you need to do is start the server and add it to the load balancer.
The cron facility seems the natural place to
turn the extra instance on and off on a preset schedule. The ideal
crontab would look like Listing 4.
Listing 4. A crontab that starts and stops servers automatically
0 8 * * 1-5 $BASEDIR/servercontrol launch app 0 21 * * 1-5 $BASEDIR/servercontrol terminate app |
You enter the code in Listing 4 in a user's
crontab on a machine outside the virtual
environment. At 8:00 a.m. on days 1 through 5 (Monday through Friday), a
script named servercontrol is run with the parameters
launch app. At 9:00 p.m., the same command is
run with parameters of terminate app. The
script will understand these parameters to be an operation and a server
role. For now, the roles will always be app.
The next step, then, is to write the servercontrol script, which is run
from cron to start servers on demand. Listing 5
shows the script.
Listing 5. The servercontrol script
#!/bin/bash
AMI=ami-ad7e95c4
OP=$1
ROLE=$2
if [ "$OP" == "" -o "$ROLE" == "" ]; then
echo You\'re doing it wrong.
echo $0 operation number role
exit
fi
case "$OP" in
"launch")
# Launch the instance and parse the data to get the instance ID
DATA=`ec2-run-instances -k main -d "DB=10.126.17.1;ROLE=$ROLE" $AMI`
INSTANCE=`echo $DATA | grep INSTANCE | cut -f 6 -d ' '`
echo $INSTANCE is your instance
# Keep on checking until the state is "running"
STATE=""
while [ "$STATE" != "running" ]; do
STATE=`ec2-describe-instances $INSTANCE |awk '/^INSTANCE/ {print $6}'`
echo the state is $STATE
sleep 10
done
# Keep track of which instances were started by this method
echo $INSTANCE >> $HOME/.ec2-$ROLE
# Now that the instance is running, grab the IP address
IP=`ec2-describe-instances $INSTANCE |awk '/^INSTANCE/ {print $13}'`
# If this is to be an app server...
if [ "$ROLE" == "app" ]; then
# Check the web server to make sure it returns our content
UP=0
while [ $UP -eq 0 ]; do
OUTPUT=`curl -s -m 5 http://$IP`
if [ $? -eq 0 ]; then # curl was successful
echo $OUTPUT | grep -qi 'welcome'
if [ $? -eq 0 ]; then
UP=1
else
sleep 5
fi
fi
done
# Register with the load balancer
elb-register-instances-with-lb smallpayroll-http --instances $INSTANCE
fi
;;
"terminate")
# Grab the instance ID. It's the last line of the file
FILE=.ec2-$ROLE
# Assuming the file exists, of course
if [ ! -f $FILE ]; then
echo No dynamic instances have been started for the $ROLE role
exit
fi
# The last instance started is the last line of the file
INSTANCE=`tail -1 $HOME/.ec2-$ROLE`
# Assuming there's something in that file, that is
if [ "$INSTANCE" == "" ]; then
echo No dynamic instances have been started for the $ROLE role
exit
fi
# Terminate the instance
ec2-terminate-instances $INSTANCE
# Take the instance out of the load balancer
elb-deregister-instances-from-lb smallpayroll-http --instances $INSTANCE
# Delete the last line of the file
sed -i '$d' $FILE
;;
*)
echo "You may only launch or terminate"
exit
;;
esac
|
This script may look long, but a good part of it is error checking. The
script starts by parsing parameters and breaking the
launch and terminate
functions into a case statement. For the launch, the script starts the
instance and waits until the instance has entered the
running state. The script then gets the IP
address of the instance and waits until a Web request to the server
successfully returns and contains the word welcome.
Stopping the instance is much easier. The instance IDs are written to a dotfile in the user's home directory, with newer instances being appended to the file. To stop the instance, the script reads the last line of the file to get the instance ID of the most recently started instance, issues a termination command, removes the instance from the load balancer, and then deletes the last line of the file containing the instances.
Note that each role has its own file. Right now, there is only the web role, but this will be expanded later.
One curious item from Listing 5 is the -d
parameter that was passed to ec2-run-instances.
This parameter contains information that the instance can read by visiting
a special URI that's only accessible to the instance. This information is
in the form of a string. In Listing 5, the server's role and the database
server are passed to the instance.
The init script is run during system boot and configures the instance with a current version of the application, along with starting the appropriate application configuration. Listing 6 uses the information passed from the control script to make those configuration decisions. The code in this listing can be part of a SYSV startup script, or it can go in the rc.local file to be run once on boot.
Listing 6. The application startup script
USER=payroll
HOME=/home/payroll
SRC=/etc/smallpayroll
# If this is a fresh AMI, set up the application directories
if [ ! -d $HOME/releases ]; then
echo "Setting up environment"
cp $SRC/Capfile $HOME
# Listing 2
su - $USER -c "cd $HOME && sh $SRC/setup_environment"
fi
echo "Deploying the application"
su - $USER -c "cd $HOME && /opt/ree/bin/cap deploy"
# Grab the user supplied data. 169.254.169.254 returns data unique to the instance.
USER_DATA=`/usr/bin/curl -s http://169.254.169.254/2007-01-19/user-data`
DBHOST=`echo $USER_DATA | sed 's/.*DB=\([0-9\.]*\).*/\1/'`
ROLE=`echo $USER_DATA | sed 's/.*ROLE=\([a-zA-Z]*\).*/\1/'`
logger "Starting application with DBHOST=$DBHOST and ROLE=$ROLE"
# If available, put the dbhost in /etc/hosts
if [ "$DBHOST" != "" ]; then
sed -i '/dbhost/d' /etc/hosts
echo "$DBHOST dbhost" >> /etc/hosts
fi
# Depending on the role...
case "$ROLE" in
'app')
# Web server... start up mongrel
su - $USER -c "mongrel_rails cluster::start \
-C $HOME/current/config/mongrel_cluster.yml"
;;
*)
logger "$ROLE doesn't make sense to me"
;;
esac
|
Listing 6 starts by initializing some variables. Next, the application
user's home directory is checked to see if the
cap deploy:setup task has been run before; if
not, the task is run. Next, a deployment is run so that the latest code is
available.
Now that the code is available, the script checks the metadata that was
passed to the instance and, with some sed
magic, extracts the components to variables. If the DBHOST variable is
set, this value is put into /etc/hosts so that the application knows where
to find the database. The role is checked; if the server is destined to be
an application server, then the mongrel servers are started.
Together, Listing 5 and Listing 6 are a fair bit of code, but they set the
groundwork for automated the starting and stopping of any kind of server.
With the crontab from Listing 4 in place, the
extra server will come online during peak periods and turn off when the
site is less busy. Next, you extend this framework to launch different
kinds of servers.
One common technique for making dynamic Web sites more efficient is to move long-running requests to a background process. These long-running jobs are usually not as time sensitive as a real request. An application might send a request to run a report to a job processing system and, using Asynchronous JavaScript + Extensible Markup Language (Ajax), poll for completion in the background. The user sees some sort of spinner that indicates the application is working, but the user is not tying up a mongrel process that could serve more interactive requests.
This approach does not remove the need to have ample resources available to process the jobs. If the job queue becomes too long, users will get tired of waiting for their reports. This seems to be an ideal use case for dynamic server launching. Something monitors the backlog of jobs; if the backlog crosses a certain threshold, a new server will be launched to help out. After some time, the server will be torn down.
Background processing is provided by the excellent
delayed_job gem—specifically, the
collectiveidea fork (see Resources). This gem lets you fire off jobs in one line of code
and implements a priority queue so that your important jobs don't wait
behind routine jobs. The job processing daemon runs out of the Rails
application directory and uses the database to request work. This means
that you can extend the current scripts to handle
delayed_job daemons.
Updating the init script to support job processing servers
Recall from Listing 6 that the script checks its
instance metadata to see what was passed from the servercontrol script.
The ROLE parameter dictates the server's job,
with app meaning an application server. The instructions for each
server type are wrapped in a case statement. Listing 7 extends this case
statement to handle delayed_job roles.
Listing 7. Handling the startup of a delayed_job server
case "$ROLE" in
'app')
# For an application server, start the mongrels
su - payroll -c "/opt/ree/bin/mongrel_rails cluster::start \
-C /home/payroll/current/config/mongrel_cluster.yml"
;;
'job')
# For a job server, figure out what kind of machine this is, and run
# an appropriate number of job processing daemons
TYPE=`curl -s http://169.254.169.254/2007-08-29/meta-data/instance-type`
case "$TYPE" in
'm1.small') NUM=2 ;; # 2 per ECU * 1 ECU
'm1.large') NUM=8 ;; # 2 per ECU * 4 ECUs
*) NUM=2 ;;
esac
su - payroll -c "RAILS_ENV=production $HOME/current/script/delayed_job -n $NUM start"
;;
*)
logger "$ROLE doesn't make sense to me"
;;
esac
|
The script checks the role of the server. If the server is an application
server, then mongrels are started. If the server is a job server, the
script checks to find out what kind of instance it is running on. This
information is available from another URL on the 169.254.169.254 virtual
host. As a rough estimate, the script launches two
delayed_job workers per Elastic Compute Unit
(ECU). Your workload may differ.
At this point, the servercontrol script can
launch a new job server by passing launch job
on the command line.
You have several ways to monitor the queue backlog. You could add a job and time how long it takes to be processed, starting a new server if the time is outside of a threshold. The downside to this method is that if the queue really gets backed up, it will take a long time to determine that you're backed up. The simplest solution is to query the database for the number of outstanding requests and to serve this in a controller. Listing 8 shows such a controller.
Listing 8. A controller that shows the length of the queue
class QueueController < ApplicationController
def length
render :text => Delayed::Job.count(
:conditions => "priority > 0 AND failed_at IS NULL").to_s
end
end
|
Listing 8 simply shows the length of the queue—specifically, jobs with a priority greater than 0 and those that have not been processed (that is, haven't failed)—and renders this number directly rather than passing it to a template. Browsing to /queue/length gives you the current queue backlog.
Launching new job servers in response to demand
Now that you can easily determine the length of the queue, you need a script to act on this data. Listing 9 shows such a script.
Listing 9. Launching more job servers, if needed
#!/bin/bash
# What's the length of the queue
QUEUE=`curl -s http://app.smallpayroll.ca/queue/length`
# How many servers are running now? (zero out if file doesn't exist)
SERVERS=`wc -l $HOME/.ec2-job`
if [ "$SERVERS" == "" ]; then SERVERS=0; fi
# launch up to two servers while the queue is over 20
if [ $SERVERS -le 2 -a $QUEUE -gt 20 ]; then
servercontrol launch job
fi
# Terminate one instance if the queue is under 5
if [ $SERVERS -gt 0 -a $QUEUE -lt 5 ]; then
export TZ=/usr/share/zoneinfo/UTC
LAST=`tail -1 $HOME/.ec2-job`
# But only if the server has run for at least 45 minutes
UPTIME=`ec2-describe-instances $LAST | \
awk '/INSTANCE/ {t=$10; gsub(/[\-:T\+]/, " ", t); print systime() - mktime(t) }'`
if [ $UPTIME -gt 2700 ]; then
servercontrol terminate job
fi
fi
|
The code in Listing 9 should be run from cron
every 5 minutes. The code first gets the length of the queue and the
number of job servers currently running. The number of job servers is
gleaned from the length of the .ec2-job file that contains the instance
IDs of the dynamically run servers. If the length of the queue is more
than 20 and there are fewer than two extra job servers running, then a
server is launched. If there is more than one server running and the queue
is less than 5, the script does some more checking to see if it should
terminate an instance.
The script first sets the time zone to Coordinated Universal Time (UTC) by
setting the TZ environment variable. It then gets the instance ID of the
last-run job server and queries to get the startup time. This time is fed
into some awk substitution to arrive at the
time, in seconds, that the server has been alive. If the server has been
alive for more than 45 minutes, then the instance can be turned off. If
not, the server stays alive.
The 45-minute hurdle is there to prevent against prematurely turning off a server. If the queue subsides, and then backs up again, the server will still be there.
Using a content distribution network
When someone goes to your Web site, he or she is also loading images, Cascading Style Sheets (CSS), and JavaScript code. Services called CDNs cache the static assets and distribute them across many servers around the Internet. As a result, you can provide faster access to these assets for your users and allow them to download multiple files in parallel. Amazon has provided a CDN service called Amazon CloudFront—a pay-as-you-go offering like its other services.
Assets that are to be served from a CDN are requested from a different server, so the URL will change. As an example, http://app.smallpayroll.ca/stylesheets/style.css will load the style sheet from the application server, but http://cdn0.smallpayroll.ca/stylesheets/style.css loads from the CDN. If the CDN does not have the asset, it pulls the asset from the origin before caching and passing the asset along to the user.
CloudFront is slightly different from other CDNs in that the origin is an Amazon Simple Storage Service (Amazon S3) bucket. To use CloudFront, you must first populate an Amazon S3 bucket with your static assets, and then rewrite your URLs to use CloudFront hosts.
From the CloudFront main page (see Resources for a link), click the link to sign up. You will have to wait for an activation e-mail before continuing.
Once you have received confirmation of your activation, go to your Amazon Web Services console page and click the Amazon CloudFront tab. You will see the page shown in Figure 1.
Figure 1. The CloudFront Distributions dashboard
Note that no distributions have been created. A distribution is merely a bucket for files that will be tied back to an Amazon S3 bucket. Click Create Distribution to get to the page shown in Figure 2.
Figure 2. Creating a distribution
Perform the following tasks:
- For Delivery Method, select Download.
- In the Origin drop-down list, select the name of an Amazon S3 bucket or make one up now that you will create with your Amazon S3 tool of choice.
- For Logging, select Off unless you plan to use these logs.
- In the CNAMEs field, type four names underneath your
domain, such as
cdn0.smallpayroll.cathroughcdn3.smallpayroll.ca. - In Comments, enter whatever text you wish.
- For Distribution Status, select Enabled.
When you click Create, you will be taken back to the CloudFront tab, where you will see the distribution you just created (see Figure 3).
Figure 3. A configured distribution
On this page is a domain name, such as djdzxdmb99068.cloudfront.net. You must now go to your DNS server, configure the four CNAMEs, and have them point to the domain name of your distribution. Amazon Elastic Load Balancing works in a similar way, except you are creating four numbered names.
Place a test file in the Amazon S3 bucket you associated with the distribution. You should be able to see this document by browsing to your distribution's domain name, such as http://djdzxdmb99068.cloudfront.net/test.htm, to view a file called test.htm inside your bucket. If you get an access denied error, make sure you have public access enabled on the files (instructions vary depending on which tool you use to manage your Amazon S3 bucket). If the previous test worked, then you can try using the CNAMEs you created earlier, such as http://cdn1.smallpayroll.ca/test.htm.
You must copy the contents of your public directory to the Amazon S3 origin bucket. The simplest way to do so is to run:
s3sync.rb -r -p public/ smallpayroll-cdn: |
from the root directory of your Rails application. The
-r option means the copy is to be recursive.
The -p option makes all files publicly
readable.
If you want to automate this procedure, check Resources for a link to a gem that handles the task.
At a simple level, you can change all your image, JavaScript, and CSS links
to point to one of your CDN links instead of your Web server. If you used
the Rails URL helpers such as image_tag, you
can have Rails do the work for you. Add the following line to
config/environments/production.rb:
ActionController::Base.asset_host = "cdn%d.smallpayroll.ca" |
This code adds a single line to the production configuration—namely,
that static assets are to be served from the host defined as
asset_hosts. The %d
is, by default, expanded to the numbers 0 through 3, so you are telling
Rails to rotate between cdn0.smallpayroll.ca, cdn1.smallpayroll.ca,
cdn2.smallpayroll.ca, and cdn3.smallpayroll.ca. These are the same hosts
that you configured CloudFront to respond to. With four hosts, you can
expect browsers to download up to eight assets at a time, as browsers are
generally limited to two connections per host.
Now, your application will use the CDN where possible. Enjoy the improved speed!
Your application uses a Amazon CloudFront as a CDN, which will speed up page loads by making downloads faster and allowing for parallel downloading by the client. You have also updated your application to dynamically grow and shrink computing resources. Some of the launching and terminating is done on a schedule, and some is done in response to load. You have some scripts that you can extend to handle many more cases, as well.
One item lacking is management. At any given point, you probably don't know exactly how many computers are running. Deployments aren't automated yet, because the server list is always in flux. You really don't know how well the servers themselves are performing. Look for the final article in this series to address these problems.
Learn
-
AssetTagHelper: Check out the Ruby documentation for the
AssetTagHelpermodule. If you use these functions to generate links and images, you'll find that moving to a CDN is a breeze. -
Preparing a Rails Application for SVN shows how to manage any
Rails application through Subversion. It's trickier than you think because
of various log files.
-
Performance tuning considerations in your application server
environment (Sean Walberg, developerWorks, January 2009) discusses
ways to make your application faster, including using job servers.
- An Amazon EC2 instance has several pieces
of instance metadata that can help the instance learn about the
environment. Browse through this chapter of the Amazon EC2 documentation
along with the list of metadata categories, and get some ideas of what can be done.
- In the Cloud Computing area
on developerWorks, get the resources you need to develop and
deploy applications in the cloud and keep on top of recent cloud
developments.
-
Alfa
Jango's blog: Take CloudFront one step further and learn about how
to handle compressed assets and selectively serve assets from either your
application or the CDN.
-
In the developerWorks Linux zone,
find hundreds of how-to
articles
and tutorials, as well as downloads, discussion forums,
and a wealth of other resources for Linux developers and administrators.
-
Stay current with
developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
-
Attend a free developerWorks Live!
briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
-
Watch developerWorks on-demand demos
ranging from product installation and setup demos for beginners, to advanced functionality for experienced developers.
-
Follow developerWorks on Twitter, or subscribe
to a
feed of Linux tweets on developerWorks.
Get products and technologies
- Now that you've
got multiple AMIs inside Amazon S3, you might want to prune some old ones.
The Amazon S3 File Manager is a
Web-based file manager that rivals the features of many stand-alone
applications or browser plug-ins. If you delete an AMI, don't forget to
ec2-deregisterit. -
S3Sync is a helpful tool for copying
files to and from Amazon S3 as well as manipulating your buckets.
-
Capistranois a popular deployment package that acts in a similar manner to Rake. -
delayed_jobis a background job server that integrates well with Rails and ActiveRecord. This link is to thecollectiveideafork of the project, which seems to be the currently maintained stream. -
synch_s3_asset_hostis a gem that makes synchronizing your Amazon S3 origin bucket and your application's static files a breeze. -
Evaluate IBM products
in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the
SOA Sandbox
learning how to implement Service Oriented Architecture efficiently.
Discuss
-
Get involved in the My developerWorks community.
Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Sean Walberg has been working with Linux and UNIX systems since 1994 in academic, corporate, and Internet Service Provider environments. He has written extensively about systems administration over the past several years. You can contact him at sean@ertw.com.




