Meeting the continuous challenge

Value creation and testing

DevOps, along with agile, is changing the way we think about software development, delivery, and deployment, but when DevOps is discussed, one critical piece of the technology puzzle is frequently overlooked: testing and the data on which it relies. This article attempts to shine a light on testing and data management in the burgeoning world of DevOps and cloud computing.


Grace Walker (, IT Consultant, Walker Automated Services

Grace Walker, a partner in Walker Automated Services in Chicago, Illinois, is an IT consultant with a diverse background and broad experience. She has worked in IT as a manager, administrator, programmer, instructor, business analyst, technical analyst, systems analyst, and Web developer in various environments, including telecommunications, education, financial services, and software.

10 December 2013

Also available in Chinese Russian

Recent history has demonstrated the importance and value of testing. When time lines are set inappropriately, and then begin to loom large, testing is one of the areas that tends to be reduced. As the recent health care problems in the United States demonstrate, testing is critical. When applications are delivered to the public, whether by government, corporation, or nonprofits, value must be delivered. When sites don't function properly or performance is inadequate, the testing regime—and particularly the lack thereof—is frequently the culprit.

DevOps, both the movement and the methodology, begins with the collaboration between developers and operations. It reflects the urgency underlying the omni-channel reality that drives today’s highly stochastic market—a mega global market that is the tool not only of the enterprise but also of the customer. This environment, buttressed by the Internet of Everything, is a challenging arena that requires constant monitoring to ensure an organization's market and financial position. The continuous integration and delivery cycle that is DevOps makes the creation of an appropriate set of test data challenging, complicated by the omni-channel environment for which adequate testing must be performed. In addition, moving to the cloud, another key component of the new agnostic information and communications technology model, requires testing in the cloud for user experience in addition to system performance and scalability.

Testing is an essential component of DevOps

DevOps includes testing. In fact, the role of quality assurance (QA) and testing is crucial in the DevOps-agile model, which is pushing development and testing together. Continuous integration and continuous delivery require continuous testing. The traditional tester role, which follows development and precedes operations, is becoming obsolete. It is common practice today for developers and testers to work together as part of a team. In this environment, their work is performed in near parallel.

The relationship of testing to development and operations is best represented in the diagram shown in Figure 1.

Figure 1. DevOps Venn diagram
Image showing a DevOps Venn diagram

Continuous testing eliminates bottlenecks by using virtualized dependent services, reducing the cost of provisioning and simplifying the test environment creation process. Implementing continuous testing, in addition to diminishing the costs associated with testing, helps development balance quality and speed. Continuous testing has tremendous value for operations also. It allows them to see how the created environment functions when supporting the application, giving the team time to make any necessary enhancements. Applications are exposed to production-like environments much earlier, in some instances from the beginning of the development process. This allows teams to test and validate applications sooner using a repeatable delivery process.

Of course, the work of testing doesn't stop at production. Testing must also be performed in the cloud—that is, TestOps. The drive to adopt DevOps is due in large part to the demands on the business to be more profitable as well as more responsive to market demands. In today’s socially aware and fast-paced market, speed, agility, and insight are the keys to success. Organizations seeking competitive advantage must develop and deliver applications and services rapidly or be left in the proverbial dustbin of history.

Automation is key

In the developerWorks article "Rebuild your IT operations to be application-centric" (see Resources for a link), the fictional company Acme Knowledge Services is described from the development and operations perspective. Now, let’s take a look at testing.

The first step for the Acme Knowledge Services testers was to join development and operations in using the Git repository for their test scripts and Atlassian JIRA for tracking. The JIRA plug-in, Zephyr for JIRA, provides the test management for JIRA, allowing the test team to become full partners on the DevOps team. Zephyr for JIRA also adds test capabilities for other processes involving planning, development, bug tracking, and reporting.

It was also important for the business to have access to the reports and metrics related to QA that IBM® Rational® Quality Manager supplied. So, Acme needed a tool that communicates between JIRA and Rational Quality Manager. This allows the testers to work with the DevOps team in their use of JIRA and still continue to support the businesses requirements by providing data to Rational Quality Manager. The tool they found was CM-Logic JazzConnect-JIRA, which allows organizations to sync Atlassian JIRA and Rational Quality Manager.

Test data still needs management

Test data management (TDM) is particularly significant for agile development. With each delivery, the code must be tested, because continuous integration and continuous delivery requires continuous testing. For testing to be valuable, data must be appropriately designed so it can support iterative test runs and at the same time impose a rigorous data governance regime.

DevOps relies on TDM to supply appropriate data to underpin project development. See Figure 2.

Figure 2. Test data management
Image showing test data management

TDM in this environment can become complex and extremely error prone, not to mention a major security or legal concern. Addressing the challenges of TDM is a precondition for a successful DevOps-agile-based project, because TDM is an essential function for organizations that plan to survive.

Acme made the decision to go with IBM InfoSphere® Optim™. InfoSphere Optim regulates the test data management process and ensures that all policies and standards are adhered to throughout the entire life cycle. It accelerates the delivery of refreshed test data and reduces risk by masking sensitive data.

TestOps, because it never stops

The main problem with the testing regime heretofore has been that testing stops after the application is in production. This model is no longer viable. Testing must begin in development, continue into production, and cease only when the application is sunset. Traditionally, the quality signal for testers has come from test results. In the era of big data, testers can and must begin to use the data signals from real users and real deployments as part of their quality strategy. As shown in Figure 3, when testing is performed in the cloud and based in the big data signal emanating from the organization's own system, you have the basis for TestOps.

Figure 3. TestOps diagram
Image showing a TestOps diagram

TestOps obtains its quality assessments from services actually running in production. It leverages the variety inherent in actual user interactions to discover bugs that could not be identified during preproduction testing. TestOps provides genuine and therefore extremely valuable metrics, because it is a natural and organic way to test software services and user interactions. To gain value from the big data emanating from production, testing must mimic the operations team and begin to focus on the live site as well as testing in preproduction. Production testing must be designed to not only gauge the performance of the system but also analyze the user experience. The user experience must be evaluated to determine the efficacy and usability of the software in the real world.

Exposure control and performance

It is often said that your business is not what you create; it's what your customers think about it. Marketing uses the term zero moment of truth, which refers to that critical moment when an individual fires up his or her computer, tablet, mobile phone, or other device and begins the process of learning about the given service or product that he or she is considering. This is a critical moment in the potential commercial transaction. Today, mobile customers are using phone-aided comparison shopping to make their decisions. Understanding this new, more savvy customer, who is well equipped to exploit the mobile market, is an economic imperative. When and how customers make their choices have changed and continue to evolve rapidly. This rapid change must be constantly monitored, just as the performance of the system itself must be monitored.

There are only three ways to determine what your customers are up to on your system. You can:

  • Make an educated guess
  • Listen to what they say
  • Watch what they do

Guessing is not very productive. An educated guess can bring positive results, but it is still a guess.

Asking is better and should be used, but you may not always ask the right questions or frame them in the right manner for a given user. It's also possible that the user may not want to communicate the complete truth for one reason or another.

Watching what and how your users actually perform, however, provides the truth of how they interact with your system. Because Acme Knowledge Services has moved to IBM SmartCloud®, it is able to use various methods of exposure control in production to evaluate its application. This is key in TestOps, because the cloud allows you to spin up environments quickly and repeatedly. In IBM SmartCloud, patterns of expertise can be created and used to perform various tests, such as canary deployments, A/B testing, and performance testing.

Canary deployments

The cloud adds a new dimension to testing. It is now possible to perform canary deployments—a critical DevOps best practice. Canary deployments allow you to test a few users based on various points, such as by browser type; by location; or by simply taking a percentage of the users, such as 10%, and exposing them to the version of the application you want to test.

Individuals using the Web are often involved in tests that an entity is conducting on its system. New versions of a site are frequently activated while the original code is still live and available to visitors. The comparison of system performance and user activity in the two versions gives you a solid basis for analysis.

Figure 4 provides an illustration of a basic canary deployment. In this example, one-third of the visitors to this site are directed to the N+1 version of the system, while the other two-thirds are directed to the current release of the system. If the N+1 site proves to be stable, accurate, and user friendly, it will eventually become the new release.

Figure 4. Canary deployment
Image showing a canary deployment

Canary deployment allows you to change the number of users being tested, the location from which the users must come, and even the type of browsers being used. You can find patterns that cause unexpected or unwanted results. Most importantly, you can easily turn off access to the N+1 version. No more fear of deploying only to discover that there are major issues. With canary deployment, if things don't go according to plan, simply remove access to the site under test.

A/B testing

In A/B testing, or split testing, two or more pages are subjected to test to evaluate the system's efficacy in general data performance and the conversion of legacy data. You should use A/B testing to evaluate new features in an atmosphere that your customer base is familiar and comfortable with. The minimum page evaluation is two pages, with no specific maximum.

A/B testing, as illustrated in Figure 5, helps you determine the best way to guide people to a specific page or set of information. A conversion rate of 57% is vastly superior to 34% and ultimately makes a huge financial difference.

Figure 5. A/B testing
Image showing A-B testing

Performance testing

Production performance testing ensures that the application is operating at optimum production load processing levels. It uncovers possible processing glitches on a scale that cannot be tested during preproduction. When new cars, trains, or planes are created, the designs are tested using simulators, but the vehicles still have to be "road" tested before they are ready for actual field operations. Not driving the car or putting the train on the tracks or having a test pilot take the plane through its paces would be unthinkable.

Use your big data

When Acme moved to the cloud, it chose IBM SmartCloud because it has embraced OpenStack. It also selected IBM InfoSphere BigInsights™ as the best option to manage its Apache Hadoop implementation. Hadoop is efficient in processing structured and—the critical element in this equation—unstructured data. Hadoop and InfoSphere BigInsights let you evaluate the exponentially multiplying amount of business-created data you generate on a daily basis.

InfoSphere BigInsights flattens the learning curve required to build big data applications by simplifying the process. It can also serve as a query-able archive, which allows Acme to collect, store, and dissect large quantities of actionable big data. InfoSphere BigInsights allows Acme to establish the mission-critical data and patterns, allowing the system it deployed to excel in the extraction of quick and inexpensive valued data.

Acme Knowledge Services can now analyze its big data, which allows the organization to make more informed and meaningful business and technology decisions.


The purpose of DevOps is to bring order and method to a world that can be both complex and chaotic by creating teams that include not only development and operations but also testing. Failure to implement proper testing procedures and methodologies is a sure suicide pill for any development project. Agility is intended to ensure the fast delivery of quality features and products, not the feeble illusion of the same. If an enterprise is not properly focused on vigorous testing processes, it runs the very real risk of having to either inform the customer that the mutually stipulated delivery will be delayed or worse still, trying to flimflam the client with some form of legerdemain intended to hide the true circumstances. In the end, it's all about user experience, because ultimately, your organization is what your customer thinks about it. Put on your best digital face, test early, and never stop.



Get products and technologies

  • Learn more about IBM SmartCloud.
  • Learn more about OpenStack technologies.
  • The Zephyr for JIRA tool description explains the tool's use and value. It also provides a trial download.
  • The Rational Quality Manager product description explains how this collaboration and automation tool helps you manage quality and testing. A trial version is also available for download.
  • Take a look at JazzConnect-JIRA, which makes Rational Quality Manager and JIRA work together to provide a seamless environment.
  • Learn how test automation with InfoSphere Optim Test Data Management enhances the testing process while ensuring good governance.
  • If you're interested in big data and the value it brings to your enterprise, check out InfoSphere BigInsights, which explains how this tool helps you take control of Hadoop.


  • Join the developerWorks community, a professional network and unified set of community tools for connecting, sharing, and collaborating.
  • The developerWorks DevOps community provides news, discussions, and training to help you and your organization with continuous delivery.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into DevOps on developerWorks

Zone=DevOps, Cloud computing
ArticleTitle=Meeting the continuous challenge