Putting the clouds to the test

Learn how different cloud platforms handle pre- and post-deployment testing

When you think about cloud computing and software testing, you want to be able to test from two perspectives — testing applications and services before they are released and testing after they are released. Pre-release cloud service testing is not trivial because cloud computing opens up the potential for highly distributed computing running on some opaque and heterogeneous infrastructures. Also, the structure of cloud computing itself has spawned a range of new services that can be used to facilitate testing; for example, the elasticity of the cloud improves load and performance testing by allowing developers to rapidly and cheaply scale their software. In this article, the author describes the interplay of cloud computing and software testing from these two perspectives.


John Rhoton (john.rhoton@gmail.com), Cloud Computing Strategist

John Rhoton photoJohn Rhoton is a technology strategist specializing in consulting to global enterprise customers, with a focus on public, private, and hybrid cloud computing. He speaks regularly at industry events on emerging technologies such as mobility, social networking, and virtualization and is the author of seven books, including Cloud Computing Explained (2009) and Cloud Computing Architected (2011).

16 April 2013

Also available in Chinese Russian

Through its abstraction of infrastructure, resource pooling, and elastic consumption model, cloud computing can have a profound impact on the design and development of applications. It is only natural to expect this disruption to have tangible implications for quality assurance.

Testing is an integral part of software engineering. As such, there is a close relationship between the development and deployment tools and the options available for testing. As demonstrated in this article, the development environment of each platform is virtually unique:

  • Google App Engine provides a software development kit (SDK),
  • Microsoft® offers a plug-in to Microsoft Visual Studio®, while
  • Force.com programmers rely heavily on Eclipse and
  • Amazon is largely agnostic to the tools and languages that run on its platforms.

Nonetheless, the software development life cycle (SDLC) is similar across vendors and services and usually involves four distinct phases:

  • Unit testing. Most projects begin with some amount of independent coding and prototyping, which engineers can initiate on their desktop.
  • Integration testing. To resolve the dependencies between modules, it is necessary to integrate the code at regular intervals.
  • Staging. As the software matures, it is useful to stage it in a production-like environment and ensure that it meets all its requirements.
  • Deployment. Finally, the application is deployed into production, where it remains until it is replaced by the next version.

We'll look at each of these four steps.

Unit testing

Programmers can take two different approaches to develop cloud-based applications:

  • They can generate the code locally and move it to the cloud when it has reached a minimum level of maturity.
  • They can use the cloud to host the development environment.

At present, most application developers take the first approach because it allows them to work independently of any connectivity and reduces the need for cloud-based resources. However, as cloud-based tools evolve and costs decline, you may very well see a trend toward complete development in the cloud.

In the interim, a development environment is typically installed on the programmer's local machine. Most unit testing is therefore a factor of the integrated development environment (IDE) in use — an element that is platform specific.

Google App Engine

The Google App Engine SDK includes all the functionality needed to run applications locally with limited performance and data volumes. To maintain a simplified development environment, Google supports Eclipse for Java™ developers. There is no equivalent for Python. However, Google does document how to install and configure the PyDev plug-in for the same results (Figure 1).

Figure 1. The PyDev plug-in for Eclipse
Image showing the PyDev plug-in for Eclipse

Windows Azure

To access the Windows® Azure™ platform, you typically use Visual Studio. Microsoft has made additional SDKs for Java and Ruby available to improve interoperability, so there is no strict limitation to the Microsoft .NET Framework, with varying levels of support for Java, Python, Ruby, PHP, OpenID, and Eclipse. Nonetheless, you will have much more information and community support available if you keep to a native Microsoft implementation, at least to get started.

What this means is that a Visual Studio developer must download and install both the Windows Azure SDK and the Windows Azure Tools for Microsoft Visual Studio.

After installing these components, a new set of solution templates appears in Visual Studio called Cloud Service (Figure 2).

Figure 2. Visual Studio Cloud Service
Image showing Microsoft's Visual Studio Cloud Service

These Cloud Services are similar to the Visual Studio web project types but are tailored for Windows Azure. The developer can test applications locally (on port 81) until they are ready and then publish the solution to Windows Azure. To support the testing phase, the SDK simulates the cloud offering with a development fabric and storage, called DevFabric and DevStorage. Although both of these environments are essential for any local debugging, it should be clear that they lack the full infrastructure for exhaustive testing.

In particular, it is impossible to perform load or performance testing of the application. Similarly, any interfaces to the local environment or machine may behave differently on the Windows Azure platform and will need to be retested in the cloud.

When the application has been tested locally, it can be uploaded to Windows Azure through Visual Studio and then managed through the Azure Services Developer Portal (Figure 3).

Figure 3. The Azure Services Developer Portal
Image showing the Azure Services Developer Portal

The portal offers facilities to stage the application before deploying it publicly. Users can also suspend instances when they are not needed and restart them at a later time.


The Force.com IDE is simply a plug-in to the Eclipse platform that connects directly to the Force.com services in the cloud and acts on the test environment (Figure 4).

Figure 4. The Force.com IDE
Image showing the Force.com IDE

It includes an embedded schema explorer for a live view of the application database schema. The browser can also view and edit any metadata components.

Because Apex code is automatically sent to the server when saved, the developer is always working with server-based data in staging mode. To deploy the application into production, it must first pass a minimum threshold of unit tests.

Even though it is convenient, there is no technical requirement to use the Force.com IDE. The full set of web services to deploy, compile, and test the application are exposed and documented, so it is possible to call them directly and indeed, even to develop another IDE that uses them.

IBM SmartCloud Enterprise

For those customers who want to leverage the benefits of a public cloud but also have enterprise-class requirements for security, customization, and integration with on-premise applications, IBM offers both Infrastructure as a Service (in the form of IBM SmartCloud™ Enterprise) and Platform as a Service (in the form of IBM SmartCloud Application Services). These two are closely connected because IBM SmartCloud Application Services automatically deploys virtual resources to IBM SmartCloud Enterprise. Together they deliver a secure, collaborative, cloud-based environment that supports the complete life cycle of accelerated application development, deployment, and delivery.

A core component of IBM SmartCloud Application Services is Workload Deployer, which provides access to IBM middleware virtual images and patterns to repeatedly create application environments that can be securely deployed and managed in a private cloud. Another strong point of IBM's solution is that it automatically integrates with and extends the IBM Rational® tools which are already in wide use for development and testing.

Integration testing

Integration testing involves assembling several modules and performing regression tests to ensure that they interoperate and deliver the required functionality. In contrast to unit testing, which developers conduct using white-box techniques, integration testing should be conducted by another testing team and generally employs black-box or gray-box techniques.

There is therefore a compelling case for running complex tests in the cloud. Especially for complicated applications, running all of the tests locally can become a tedious and time-consuming task. Deploying the application and testing it in the cloud can save significant time and local development resources. As the testing increases in scope, the differences between the SDK and the actual cloud computing environment that a PaaS vendor provides become more apparent. In particular, if an application has a complicated workflow, local testing can only cover a fraction of the actual workflow and data.

An additional benefit of cloud computing is the speed with which innovation can reach the user. In a highly integrated continuous deployment environment, a feature or a fix can go through all the steps in the SDLC in less than a day. Consequently, it is possible to deploy new production releases quickly and frequently, even multiple times per day.

Continuous integration (CI) implements a continuous process of applying quality control whereby small pieces of effort are applied frequently. It aims to improve the quality of software and to reduce the time taken to deliver it by replacing the traditional practice of applying quality control after completing all development.

A regular (nightly) build is not difficult to implement if it is automated. A single command should have the capability to build the system; for example, build tools such as make have existed for years. Other, more recent tools like Apache Ant, Apache Maven, MSBuild, or IBM Rational Build Forge are frequently used in CI environments. Automation of the build should include deployment into a production-like environment.

In many cases, the build script not only compiles binaries but also generates documentation, web pages, statistics, and distribution media. Ideally, the build is also at least partially self-testing.

Popular tools for CI, testing, and release make it easier for developers to integrate changes to the project and for users to obtain a fresh build. Such tools include CruiseControl and Hudson, as well as commercial tools such as IBM Rational Team Concert and Microsoft Team Foundation Server.


Staging is a more advanced phase of testing that typically runs in an environment that closely reflects the production one. Staging not only deals with testing all the functionality, but also load testing; this may be made accessible to customers to obtain their approval prior to final production release.

When the application grows more complex, it often makes sense to introduce multiple levels of staging and testing. Some mission-critical environments have as many as seven different environments along the path from development to production. Each represents a step closer to what the production environment is like.

Cloud computing has the potential to dramatically accelerate this trend. Because you can fire up instances on demand, little cost is associated with having several different variants to test multiple options in parallel. The cloud can also be useful if the need arises to roll back a live system to a previous version.

Staging in the cloud includes running the application code with a representative dataset. The actual technique for staging the application will depend on the platform used. In the case of a custom platform built on an infrastructure service, such as Amazon Elastic Compute Cloud (Amazon EC2), Rackspace, or GoGrid, the developer determines the whole staging operation. It could be as easy as using a duplicate instance of the production environment and running it with a different domain name.

In an environment like that of Google App Engine, this process would include running the application as a different version under the main production application ID. The workflow would process the data sets that reside in the production data store. Thus, it is necessary to identify and resolve any potential data store schema issues related to different representations of data between the old production and the new staging version of the application.

Windows Azure offers a separate staging environment for applications. The Azure Service Management Portal provides two slots — one for production and one for staging — that are identical except for the URL used to call them. You can therefore fully test the staging environment until you are comfortable that it is ready for production without affecting the production slot.

I'd like to mention load-testing tools at this point. A major advantage of cloud computing is the elasticity it provides to the testing infrastructure. This elasticity is particularly important for load and performance testing which can require significant resources, albeit typically for short periods of time only.

As you might expect, a variety of tools has emerged to cover this need. Some of them, such as SOASTA and LoadStorm, specifically target cloud testing. Others, such as HP LoadRunner and IBM Rational Performance Tester, have been around for some time but have also recently added cloud-specific capabilities.

For example, the Rational Performance Tester workbench facilitates load testing in the IBM SmartCloud to provision virtual test agents and generate virtual users on a pay-as-you-go basis. IBM Rational performance testing solutions enable performance tests to emulate large-population user transactions and thereby validate application scalability and diagnose application performance bottlenecks.


Production is the final phase of the SDLC. It must scale to meet the demands of the entire user base and should be largely error free because support can be costly. The execution platform typically includes a set of runtime environments (such as a Java Virtual Machine, Python, or PHP) and programming languages. It may also provide a set of library functions, data storage facilities, and integration mechanisms (such as a queueing system) for communication with other modules, internal and external.

The two main factors to consider during release are:

  • Whether the deployment is completed successfully.
  • That service is disrupted as little as possible during the transition.

The first point is extremely important. Assuming that the software was tested extensively during the staging phase, the main objective is to ensure that the deployment process itself maintains the integrity of the service. It is therefore vital to test the process of copying software and making it available with non-critical services before deploying the main user-visible services.

The second issue concerns the downtime of the application during the go-live period. In some cloud environments (like Google or Azure), it is possible to support deployment with zero downtime and no production disruptions provided the deployments are well planned. In other cases, deployments should be as small as possible to minimize any chance of interruption in production.

Some of the cloud SDKs offer nice tools to automate deployments. These tools have the benefit of diminishing the risk of an incomplete deployment and can also accelerate the process, thereby minimizing the downtime. For example, Google App Engine has a deployment tool in its SDK that you can use to migrate the application from the SDK to either a testing environment with a different application ID or version or to staging and production with different versions of the application. With Azure, it's as simple as requesting an in-place upgrade or swapping the virtual IP address of the staging and production slots using the service console. Similarly, for infrastructure services, you would typically just update the Domain Name System entry to point to the newly deployed service when it is ready.

Data migration

If the application's design ensures a clear separation of data and code, there should ordinarily be relatively little impact on the data during a deployment. However, two cases that may require at least some thought.

During the initial deployment, you may need to seed the data. Because there is no real risk before the service is live, you can attend to this case carefully and use any bulk migration processes that are necessary to transform and load the data for use.

A much more delicate situation is when you have to change the data structures. If you refactor the data or use alternate data sources, the interfaces may change and there is the potential for a significant number of problems and disruptions.

Needless to say, if you can avoid the problem by refraining from schema changes, it will make your life much easier. When you see no other options, you can still decouple the migration from the deployment by abstracting the data structures to an intermediate layer. If you use a virtual view of the data and allow it to proxy the actual requests, you can migrate the data structures independent of the deployment and potentially even over an extended period of time.

Release cycle

Part of the value proposition of cloud computing (and Software as a Service in particular) is the notion that the service can be updated frequently so new functionality reaches users regularly. This means software bugs are fixed quickly, security vulnerabilities are patched before they are exploited, and new features are added as soon as the developers can code and test them.

As I already alluded to, every deployment exposes the service to a risk of failure, as well as potential downtime, during the transition. The only way to manage this risk effectively within a regular or continuous deployment process is to comprehensively automate the deployment and test it carefully. Some of the tools used heavily for deployment in cloud-like environments include CFEngine, Puppet, and Chef.


No matter how careful you are in deploying new versions, there is always the chance that you won't discover a critical problem until it is already in production. Unless you have a rollback position, you could end up with production down for hours.

In practice, this means that you need to retain the most recent production releases in a ready state for quick redeployment. They might be instances at your cloud provider's site that can be easily switched with a namespace change. Or, they may simply be code that is retained in your repository. In the latter case, you must validate that you can package and deploy them with minimal delay in case of an emergency. You might also look at compartmentalizing the application. In addition to the general benefits that service-oriented architecture promises for loosely coupled applications, there are also advantages to a fine-grained deployment model. Such a model would not only reduce the rollback impact but minimize the effort required to deploy small fixes.


Cloud computing introduces additional complexity into the development process through the sheer number of components that need to interoperate to ensure success. Is it therefore vital to have a clear picture of cloud platforms and services to guarantee quality and reliability.

At the same time, the elasticity and automated provisioning of resources that cloud technologies offer radically improve the SDLC. The ability to launch massive load or penetration tests on demand allows engineering teams to easily and cheaply develop and test scalable applications. Similarly, mechanisms to switch seamlessly between versions and builds can accelerate new-feature deployment while reducing the risk of rollback.



Get products and technologies

  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.


  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Cloud computing on developerWorks

  • developerWorks Premium

    Exclusive tools to build your next great app. Learn more.

  • Cloud newsletter

    Crazy about Cloud? Sign up for our monthly newsletter and the latest cloud news.

  • Try SoftLayer Cloud

    Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.

Zone=Cloud computing
ArticleTitle=Putting the clouds to the test