This item was orignally published in August, 2001 and refers to Rational Suite TestStudio
How many times have you surfed to a Web site to accomplish a task only to give up and go to a different site because the home page took too long to download? According to Juniper Communications, "46% of consumers will leave a preferred site if they experience technical or performance problems." In other words, if your site is slow, your users will find another, less frustrating way to do business!
With so many people relying on the Internet to perform a wide range of operations, application performance has become vital to the success of any Web solution. Consequently, many software companies have developed tools and methodologies to test and tune applications for performance. Many of these tools and methodologies have focused on optimizing system metrics rather than user experience. But let's face it users don't care what your throughput, bandwidth, or hits-per-second metrics prove or don't prove; they want a positive user experience.
So how do you truly predict and tune an application for optimized user experience? You must test the user experience firsthand. There are two ways to accomplish this. You can release your Web site straight into production, where data can be collected and the system can be tuned, with the great hope that the site won't crash or be painfully slow. Or you can simulate actual multiuser activity, tune the application, and repeat until the system is tuned before placing your site into production. Certainly the second alternative is the wiser choice, but how exactly do you simulate multiuser activity accurately? That's the question this series of articles attempts to answer.
The "User Experience, Not Metrics" series addresses topics related to determining true user experience and application performance tuning using IBM® Rational SuiteM® TestStudio® coupled with Noblestar's proven methodology of end-to-end performance engineering. This first article is intended to introduce you to the high-level concepts and terminology used throughout the series and to give you an overview of the articles that follow.
Understanding the terms and concepts that follow is critical to getting the most out of this series of articles.
Performance engineering is the process of identifying and eliminating application or system performance bottlenecks by exercising actual expected user patterns.
Workload distribution is a representation of the functions performed by a user community on a system, sometimes known as a user community model. For example, during the course of a day on a retail-based Web site, most users are shopping, some are searching for a specific product, some are finalizing purchases and checking out, while a single administrator may be updating product prices. A workload distribution is based on a percentage of users performing a specific function over a given period of time. Using the above example, a workload distribution could be: shopping -- 83%, searching -- 5%, checking out -- 10%, and administration -- 2%.
Performance goals, when viewed from a user-experience perspective, are expressed as a maximum allowable response time at a predetermined number of concurrent users. For example, "All static pages on the Web site will display in 8 seconds or less on a 56.6 kbs modem 95% of the time while 500 users are accessing the site according to the documented user community model."
Response time is measured from the end-user perspective. A good example of response time is a measurement of the time between when a user clicks the Login button and when the subsequent page is fully loaded.
Scalability is the ability of a Web site or application to handle an increased number of users while still meeting performance goals. A Web site or application is often said to scale gracefully if its performance degrades slowly when predicted user levels exceed expectations as opposed to becoming unusable or crashing.
Session duration is the total amount of time a single user is using the Web site during a single site visit (expressed in fractions of an hour). Session durations may vary by type of user.
The number of concurrent users is the total number of users who are actually accessing the site or who have active sessions at a specific instant in time.
Total hourly usage is the anticipated total amount of user activity on the site in an hour. This is calculated by multiplying the session duration by the number of concurrent users.
Baselines, in this context, are single-user, single-script tests that are recorded to provide a starting point for time comparisons.
A user delay is a wait time incorporated into a script so when that script is played back it plays at the same pace as an actual user.
Client-side processing time is a wait-time incorporated into a script that simulates the amount of time the user's computer spends processing requests, generating pages, or executing scripts passed to the client over the Web from the server.
This series of articles may use some common terms that have different meanings to different people. These terms are defined below for the context of this series.
A load test is a multiuser test that accurately simulates the expected user community, including user delays. These tests may be executed with differing user loads to find information such as the maximum number of users a site can support while still meeting the stated performance goals.
A stress test is any combination of scripts that are played back at a high user load excluding user delays. These types of tests are useful in determining system stability, and functionality under load, but are not valid for determining user experience.
Benchmarks are metrics gathered about system hardware and supporting software but not application code. For instance, Web server throughput and hits per second when accessing a large graphic can be determined through benchmark testing.
The "User experience, not metrics" series will publish monthly. All articles will include discussions of the practical applications of the methodology/technique being introduced, real- world examples and/or code samples, and "Now you try it" exercises. Each article will be identified as beginner, intermediate, or expert level. This level generally refers to the complexity of the code required to accomplish the technique being discussed. The concepts presented in each article will be applicable at all levels of expertise. The first 12 articles have already been identified and are described briefly in the following sections.
One of the keys to determining true user experience is to effectively model actual users and user communities. Most performance tuning approaches today don't account strongly enough for either the distribution of tasks across entire user communities or the high level of randomness among actual users. The three articles immediately following this one in the series discuss how to use TestStudio to accurately model both individual users and entire user communities from the application's perspective. Specific topics include:
- Part 2: How to model individual user delays
- Part 3: How to model individual user patterns
- Part 4: How to model groups of users
After modeling actual users, it's imperative to capture the actual system/application response time from the perspective of those users. Simply capturing those times isn't sufficient. The times are useless unless the patterns of those times can be interpreted. The next three articles discuss how to use Rational TestStudio to capture and interpret true experience times. Specific topics include:
- Part 5: What should I time and where do I put my timers?
- Part 6: What is an outlier and how do I account for one?
- Part 7: How to consolidate and interpret times
As much as I hate to admit it, stakeholders and decision makers need reports on results. I keep trying to convince my clients that all they need from me at the end of a performance engineering engagement is a Post-It note with either the words "Go Live" or "Don't" written on it, but they don't seem to think that provides enough value. If you have clients similar to mine, you'll be required to take the vast amount of data collected from a performance engineering effort (often several gigabytes) and consolidate it into a concise, yet meaningful report. The articles in this section will discuss what types of tests provide the most value to managers and decision makers, as well as how to use the data collected from Rational TestManager reports to create multiple run summaries. Specific topics include:
- Part 8: What tests add value to stakeholders?
- Part 9: How to summarize across multiple tests with accuracy
- Part 10: How to create a degradation curve
The final group of articles in this series will focus on specific advanced issues that have caused stress to the author. These articles take the format of case studies. Each case study outlines the specific need of the (unnamed) client in question, the author's thought process in developing a solution, an outline of the potential solutions, and a detailed description of the selected solution. Specific topics include:
- Part 11: How to handle secure session IDs
- Part 12: How to deal with conditional user path navigation (intelligent surfing)
- Part 13: How to work with unrecognized protocols
The lesson in this introduction to the "User experience, not metrics" series of articles is unmistakable: a user's point of view is a more reliable measure of Web site performance than today's customary metrics. This series of articles is designed to teach how multiuser activity can be simulated using IBM® Rational Suite® TestStudio™ and Noblestar's proven performance engineering methodology. The articles will share valuable information about how the methodology works and how the Rational toolset is utilized. The articles will even divulge useful tips on getting around those issues that have stumped the experts. I hope your interest has been piqued and that you'll read the whole series.
- "Managing Web Application Performance" by Juniper Communications (1998)
Currently, Scott Barber serves as the lead Systems Test Engineer for AuthenTec. AuthenTec is the leading semiconductor provider of fingerprint sensors for PCs, wireless devices, PDAs, embedded access control devices and automotive markets. He is also member of the Technical Advisory Board for Stanley-Reid Consulting, Inc.
With a background in consulting, training, network architecture, systems design, database design and administration, programming, and management, Scott has become a recognized thought leader in the field of performance testing and analysis. Before joining AuthenTec, he was a software testing consultant, a company commander in the United States Army and a government contractor in the transportation industry.
Scott is a co-founder of WOPR (the Workshop on Performance and Reliability), a semi-annual gathering of performance testing experts from around the world, a member of the Context-Driven School of Software Testing and a signatory of the Agile Manifesto. He is a discussion facilitator for the Performance and VU Testing forum on Rational DeveloperWorks and a moderator for the performance testing and Rational TestStudio related forums on QAForums.com. Scott speaks regularly at a variety of venues about relevant and timely testing topics. Scott's Web site complements this series and contains much of the rest of his public work. You can address questions/comments to him on either forum or contact him directly via e-mail.
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
