Best practices for SOA nonfunctional testing

In the course of developing a Service-Oriented Architecture (SOA) application, your organization will most likely have nonfunctional requirements (NFRs) that need significant implementation and testing. Shiv Asthana describes the best practices you should adhere to when testing nonfunctional requirements for applications built as part of an SOA environment.

Shiv Asthana (, IBM GBSC Test Management Team, IBM

Shiv Asthana's photoShiv Asthana is part of the IBM GBSC Test Management team based in Hyderabad, India, which tests SOA-based assets. Shiv has an MS in quality management from Birla Institute of Technology and Science, Pilani (BITS, Pilani) and has eight years of IT experience.

28 August 2008

Also available in Chinese


SOA is an IT architectural style that supports the transformation of a business into a set of linked services, or repeatable business tasks, that can be accessed when needed over a network. This might be a local network, the Internet, or a geographically and technologically diverse set of services from different locations that interact as though they were all installed on a local desktop. These services can coalesce to accomplish a specific business task, enabling businesses to quickly adapt to changing conditions and requirements. In other words, SOA works on the key principle of find, bind, and-execute, and is ideal for request/response type business needs.

Figure 1. The SOA implementation life cycle
The SOA implementation life cycle

When an SOA implementation is guided by strategic business goals and requirements that either resolve pain points or automate a certain manual task, business transformation is achieved. Some of the many benefits are:

  • Alignment of IT with the business.
  • Maximum reuse of IT assets (no rip and replace).
  • Automation of manual and repetitive tasks.
  • Compliance with industry standards and legal regulations.
  • Filling the white space between the IT silos of an organization.
  • Continually changing business processes that can easily be mapped as new services.

The promise of SOA, however, can be achieved only when the functional and nonfunctional requirements are built into the application and seamlessly integrated into the production environment. Therefore, perfect implementation and testing is imperative for the moment of truth.

The key challenge in NFR testing is posed when an SOA application is developed in a lab environment or as part of a new product portfolio of a software vendor where, in most cases, access to a production environment isn't available. Some of the other challenges include:

  • SOA applications have a lot of internal and invisible components.
  • Testing requires creation of stubs, which would substitute the legacy applications in a production environment.
  • Interoperability expectations are very high.
  • SOA applications run on a heavy middleware stack or a platform.
  • There's a high dependence of performance and load-bearing capability on the middleware stack and legacy applications (in production environment).
  • Because SOA is implemented to fill the enterprise-wide white-space gaps, the NFRs can be massive.
  • Early technology (SOA) adopters have limited access to test automation tools.
  • Depending on the business problem being solved, the architecture and components used for application deployment can change.
  • NFRs change more frequently than functional requirements.
  • Testing in a lab environment requires heavy investment in the test infrastructure.

Best practices

While there's no single best approach for testing software, the following list is based on my experience. Apart from addressing some of the challenges above, these recommendations might bring in other advantages as well:

  • Start by putting together a test team that's skilled enough to work on the target operating system and the middleware components you plan to use.
  • Analyze the requirements carefully, and look for self-contradicting requirements.
  • Understand the system context diagram to draw requirements for stubs.
  • Be sure to create a testability matrix in the beginning of the project. Identify the NFRs that can be tested, the ones that can't be tested, and where testing can be automated.
  • After creating the testability matrix, identify the requirements that can be tested with the help of any industry standard tool, such as IBM® Rational® Performance Tester, or see if a custom software program or script is required for testing automation.
  • Start planning early for the test Infrastructure. The requirement might be to test the application on various platforms, such as Microsoft® Windows®, Linux®, IBM AIX®, and so on. It's difficult to procure servers later.
  • Estimate the test data requirements and identify ways of creating an adequate volume of valid test data for volume and load testing.
  • At least test your SOA application for the nonfunctional aspects in Table 1.
Table 1. List of nonfunctional characteristics
Type of testTest definition
AccessibilityVerify the ability to access the functionality of the application.
Audit and controlVerify how easily it's possible to check the historic workflow and audit trail.
AvailabilityVerify that the application has a high uptime as stipulated in the service level agreement (SLA).
CompatibilityVerify if the application fits in a pre-existing (older) environment.
DocumentationVerify that the user guides provide the right instructions.
InstallationVerify if the application works on the defined middleware stack.
InteroperabilityVerify if application works after changing a vital component in the environment.
Load/VolumeVerify if the application processes the required number of transactions in a given time.
MaintainabilityVerify how easy it is to maintain the application after it goes into production.
PerformanceVerify if criteria like response time, throughput, concurrency and so on meet requirements.
ReliabilityVerify that the application works if stressed in production-like environment.
ScalabilityVerify if the application can meet the growing needs of a business.
SecurityVerify if the application has enough security to protect information theft.
ServiceabilityVerify if the application can be debugged without making any impact on the business.
UsabilityVerify that the application is usable from an end-user perspective.
Table 2. Comprehensive list of tools that can facilitate NFR testing of SOA applications
IBM Rational AppScanEnables SOA application security assessments in a centralized fashion and provides a fully integrated solution set. Rational AppScan is scalable with enterprise architecture that enables scanning for multiple applications simultaneously.
IBM Rational Functional TesterProvides testers with automated testing capabilities for functional and nonfunctional testing, regression testing, GUI testing, and data-driven testing. The test automation capability of the tool allows load and stress testing as well.
IBM Rational Performance TesterCaptures response time, page throughput and concurrency related metrics. This performance test-creation, execution, and analysis tool is for teams validating the scalability and reliability of complex SOA applications before deployment.
IBM Rational Performance Tester Extension for SOA QualityExtends performance and scalability testing to SOA applications. This is a load- and performance-testing and problem-analysis tool that, apart from the standard capabilities of Rational Performance Tester, helps find performance bottlenecks. It enables problem determination for SOA efficiency with broad platform-monitoring support for deployed Web services and collection and visualization of server resource data.

Note:IBM Rational RequisitePro, IBM Rational Test Manager, and IBM Rational ClearQuest® can be used for requirements management, test-case management, and defect management, respectively.

Identify the need for creating stubs and simulators for testing

As mentioned above, one of the challenges in SOA testing is creating stubs and simulators. Stubs and simulators substitute the legacy applications in a production environment and other external applications, which are required as back-end systems in production. (You won't have access to this.) Apart from the business domain knowledge, you can create stubs and simulators by analyzing the system context of the application being developed (see Figure 2).

Figure 2. Sample system context diagram of an SOA application
Sample system context diagram of an SOA application

In Figure 2, the legacy production systems (in blue) and the external system (in pink) require simulation as a stub to test the new application (needed for both functional and nonfunctional testing).

The test team can begin by understanding the functionality of these systems and then explore simple ways of simulating them for testing. In most cases, a simple database might help but should have all the tables and sample data to support a business transaction being tested.

Real-life implementation of these best practices

A group in IBM called the Global Business Solutions Center (GBSC) has developed and tested SOA applications called composite business services (CBSs), which run on the IBM SOA Foundation and a middleware stack, such as IBM WebSphere® Process Server, IBM WebSphere Application Server, IBM WebSphere Business Services Fabric, IBM WebSphere MQ, and IBM DB2® products. A CBS is a collection of related and integrated business services that provide a specific business solution and support a business process built on an SOA.

Typically a CBS is integrated with other applications, services, or both within a customer environment to deliver the desired business solution for the customer. The business solution consists of business services that are choreographed together. In other words, a CBS is not a stand-alone solution, but a solution can be created using one or more CBSs.

An end-to-end solution built using a CBS is targeted for industries like banking, healthcare, insurance, and retail, and for state governments. Given the nature and magnitude of these industries, you can see that the role of SOA is to enable the processing of high-volume and mission-critical business transactions and to support ongoing business change in the face of a heterogeneous environment.

This is why NFRs are so pivotal in transforming a client's business. The CBSs developed so far in GBSC have been tested for most of the requirements recommended above and, in some cases, have demonstrated outstanding performance benchmarks, such as the capability to process over 2 million transactions in a single batch submission, and reported excellent throughput and response times for various mission-critical business processes.

Testers used tools like Rational Performance Tester, IBM Tivoli® Composite Application Manager, and custom-built utility programs to create test data and automate the execution of certain test cases. I omitted regular test- and defect-management tools, like Rational Test Manager and Rational ClearQuest, from this list because they're used to support both functional and nonfunctional tests.

Figure 3 shows how the average response time has increased from 3 to 3.2 seconds when the concurrent user count changes from 1,000 to 1,150 for business transaction #1 in a sample application.

Figure 3. Performance testing: business transaction #1
Performance testing: business transaction #1

Figure 4 shows how the average response time has increased from 7.9 to 10.7 seconds when the concurrent user count changes from 1,000 to 1,150 for business transaction #2 in a sample application.

Figure 4. Performance testing: business transaction #2
Performance testing: business transaction #2

When you review metrics associated with NFRs, note that reference data is essential. Without that the numbers don't make sense. For instance, if you gather and publish performance-related benchmarks, at a minimum you should provide the following reference data:

  • Test environment details, such as server specifications (hardware and software specifications)
  • Client details or specifications of the desktop from where the test was executed
  • Details of the tool used for testing (for instance, Rational Performance Tester was used to generate the sample charts in Figure 1 and Figure 2).

What to watch out for

While testing NFRs is imperative for the successful adoption of SOA applications, remember that in the absence of specific quantified requirements (where possible) you can run into trouble. This is more likely when development and testing happens in a lab environment where nonfunctional criteria, such as concurrency and volume/load requirements, aren't quantified and the test team gets into benchmarking mode. At that point, the time and effort required to complete the project increases significantly.

However, this risk can be mitigated if a sampling plan is created and shared with all project stakeholders. Let's examine how to construct a sampling plan for a project where you need to determine the maximum number of transactions an SOA application is capable of processing.

First you tabulate all the permutations and combinations of transactions that can be potentially processed in a test environment. See the examples in Table 3.

Table 3. Identify different ways transactions can be processed
Number of transaction requests in a fileNumber of files in a single batch uploaded

Subsequently, when the application gets developed, process a small number of files and extrapolate from the numbers. Maybe upload 100 batches with 100 requests in a file and based on the time it takes to process this number you can estimate the time it would take to process other permutations and combinations of files shown in Table 3.

Depending on the time and resources available and your confidence level, you can decide where the benchmarking exercise can begin and conclude. Check the sampling plan (see Table 4) created from Table 3.

Table 4. Sampling plan, part 1: Target values for testing proposed are in bold from the example
Number of transaction requests in a fileNumber of files in a single batch uploaded

Given the fact that these target values can fail during the benchmarking mode, it helps to plan the test approach at this level itself. If, for instance, the test case with 100*40000 files fails, then how do you go about it? It's feasible and pragmatic only to proceed in a preplanned way; for example, if 100*40000 fails, then try 100*37500, then 100*35000, and so on. See more examples in Table 5.

Table 5. Sampling plan, part 2: Test approach if testing against the initial target value fails
Number of transaction requests in a fileNumber of files in a single batch uploaded

Limitations in achieving higher levels of nonfunctional capability

With SOA adoption, it's possible to uncover the full potential of your existing IT systems. But the limiting factors can be attributed to the various components used in the middleware stack, which is required to run SOA applications and the legacy applications that run in production. To exemplify this, consider using a rules engine, which can process only X number of transactions in a given time. If this rules engine is hooked to an SOA application, it becomes a constraint in achieving performance levels higher than X.


Hopefully, this article will help your organization in its journey towards SOA adoption. While the application of these best practices depends on your current SOA maturity level and opportunities for improvement, the list of challenges, nonfunctional characteristics, tools, and the real-life implementation example provided above might help you get a head start. And eventually, a successful implementation of NFRs will lead to a direct linkage between IT and business processes, which is the main goal.



Get products and technologies

  • Innovate your next development project with IBM trial software, available for download or on DVD.



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into SOA and web services on developerWorks

Zone=SOA and web services
ArticleTitle=Best practices for SOA nonfunctional testing