![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
- Read this article in our interactive digital edition format!
- Subscribe to
IBM Data Management magazine
Change is inevitable. Bringing new applications and new systems online is part of the job. But with change comes risk. Every new piece of code raises the question, Will it work the way it's supposed to? When it comes to business-critical systems, the answer must be yes.
The best way to know for sure is through testing. Done properly, testing reduces risks and lets you be a change champion without gambling. But what should a test measure, and how do you build effective tests?
To help answer these questions, we turned to experts: the Performance and Benchmark team at Infor, a major ERP provider based in Alpharetta, Georgia, that is known for its rigorous benchmark practices and skills. Hans Kamman, director of Development and Release Management, runs the Infor Performance and Benchmark team based in Barneveld, the Netherlands. Kamman and three of his performance engineers, Cees Padmos, Dick Westeneng, and Adrian Voortman, offered up advice on benchmarking and the basics of efficient, effective testing.
Creating an effective test is an exercise in balance. Your test must produce results that repeat when the application is installed and running in a live environment, but you don't have the time or budget to re-create your organization's entire production system. If you've decided that you need more specific information than a standard vendor benchmark can provide, you need to create just enough realism in three areas: data, users (or processes), and infrastructure. For more information on vendor benchmarks, see the sidebar "What goes into a vendor benchmark?"
For testing purposes, the goal is to create a data set that represents your organization's data but is tailored to the specific needs of testing. Copies of actual production data are a good place to start, but you'll almost certainly need to do some tweaking. Evaluate the quality of your test data by thinking in two dimensions: volume and distribution (skew).
The volume question may be the easiest: do you have enough data to create a realistic test load? From there, the decisions become more complicated. For example, when figuring out the correct distribution of your test data set, you obviously can't just make a million copies of the same record. But a perfectly even distribution isn't natural either—most data sets have areas of greater and lesser concentration.
You should also consider the impact of erroneous data on the system that you're testing. Decide whether you need to include some percentage of mistakes.
Finally, stay aware of privacy and security issues if you're sampling production data. You may need to set up auditable protocols that randomize, mask, or otherwise obscure your test data.
Of course, test data is inert—you need scripts to simulate the actions that your system will support, such as transactions for an OLTP database or queries on a data warehouse. Writing a good script requires you to understand what users do and how often and how fast they do it.
The foundation of a good script starts with a clear understanding of the business process that the application will support. For example, how many total users will there be, and how many will be logged in at the same time? How many active sessions can you expect, and what is the maximum number of simultaneous transactions that the system needs to support? Critical resources for good script design are articulate, involved folks from the business side who can work with you to develop effective specifications. "Your starting point should be to establish the user roles that will figure into the testing," says Padmos. "And don't forget that some business processes involve multiple user roles. For example, a purchase order may have to sit in one or more manager's queues for approval after the buyer sets it up."
Building effective test scripts requires imagination and awareness. For example, an automated script may need less than a second to fill out a query or data entry form that would take a human several minutes to complete. Therefore the test scenario needs to incorporate waiting time between steps in the test. You can expect notable human variations in this waiting time. "What happens, for example, if the keystroke speed of operators goes up by 10 percent? Does that actually ever happen? Does it happen only on the day after Thanksgiving or every day after the 10 a.m. caffeine break? And would it adversely affect system response time?" asks Kamman rhetorically. "While it's easier to write a script that tests every function sequentially, humans are almost never that methodical and predictable. To be realistic, the test scenarios should be able to simulate user behavior that can show almost random variations."
Once you have scripts that capture the necessary business processes and user behavior, set aside some time to review how the scripts access your data sets. Is there too much contention? Not enough? “For example, are purchase orders concentrated on a few vendors, or are they spread out evenly across several vendors?” asks Voortman. “Simply rotating through a hundred sample vendors in the same sequence isn’t realistic, but neither is issuing all purchase orders to a single vendor.” Of course, some contention does happen in real life: the warehouse that carries a popular product will have the highest transaction rate—a pressure situation that you’ll want to test.
The goal is to eliminate unrealistic conditions that will affect performance and skew results—and to identify unintended consequences. “One company that we worked with created a test script that attributed all orders to a single vendor,” says Kamman. “The idea was partially to be sure that the system could handle a specific volume, but they were also trying to reduce the amount of work that they needed to do on the test environment.” However, the strategy backfired slightly. “The high order volume caused the sample vendor to exceed its credit limit, and the system started to refuse orders,” says Westeneng.
How suitable is your test infrastructure?
Creating a test infrastructure that reflects your production systems can be a real challenge. Some companies can run a production-like environment on off-hours or weekends, but most do not have that luxury. If you’re struggling with this, you might want to ask a systems integration partner or software vendor for help. "Our consulting services, for example, may be able to create a close-enough version of our customer's production environment," says Kamman. “For specific requirements, the application provider may also have tools and scripts a customer can draw on for testing."
Recognize that running tests will likely be as much of a learning experience as building them. The initial run of a test may show unexpected results—usually substandard performance. That starts the detective work. Poor test results may have causes outside the application itself. Keep an open mind and be ready to pull in database, network, and hardware experts to help explain mysterious results. And while you aim for realism, understand that no test is foolproof. “You cannot model reality 100 percent,” cautions Kamman.
Testing an application carefully builds a deeper understanding of how it works and how your own IT environment affects applications and business processes. To bring new applications and versions online, it is essential to reduce risk and protect IT performance. The better the benchmarks, the more freedom you have to be a champion of change.
- Learn the optimal way to approach many issues involving DB2 at the Best practices for DB2 for Linux, UNIX, and Windows page.
- Learn more about DB2 on the DB2 for Linux, UNIX, and Windows page on developerWorks.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
