![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
From the IBM WebSphere Developer Technical Journal.
What makes a good system verification tester?
Having grown from a junior developer to a senior one, and eventually to software architect, I have worked on a wide variety of design and development projects in my eight-plus years in software development. In this time, I came to appreciate the importance and the challenges of testing, having even volunteered to test pieces I designed, in the spirit of "eating your own cooking." About a year ago, I became manager of a System Verification Test (SVT) group. What prompted me to write this column was a casual conversation with a tester in my department who, it turned out, was a former colleague of someone who had applied for a testing position in my department. When I asked the tester about the candidate, who was currently doing development work, her response was "I don't remember this particular person, but he must be a good tester since he is a developer."
I was a bit taken aback by this comment, but later on I realized that this general belief was fairly prevalent in both the development and testing communities at large. Certainly, there are excellent developers and excellent testers, but just because you are stellar at one does not necessarily mean you are also as good at the other. So here I am, now a test manager, here to explain the challenges involved and the unique qualities required to be a good SVT tester -- not only to attract top talent into the profession, but also to share the pride we have in our work, and perhaps gain a bit more respect for the testing discipline in general.
Be definition, the SVT group conducts system level testing. What is system level testing? That's a common question. My interpretation is that system level testing is performed to ensure that the intended customer value is delivered through the capabilities that the product offers.
So what is a "customer value?" The answer is in the reasons why customers buy and use a product in the first place. For example, I have used Microsoft® Word for many years. Recalling my personal experiences with the software, I remember the most annoying thing in early versions was its tendency to crash in the middle of editing, losing everything I had written since I last saved. Thankfully, it doesn't do that anymore. Even better, it periodically saves a document by itself, so even if I mistakenly exit without saving, I can still recover from the last auto-saved copy -- which to me was a fantastic feature. This experience as a customer made me realize that, while the product serves a general purpose, it is its individual features that really differentiate a product from its competitors, and it is these features that provide the values which make its customers loyal. Therefore, the most important trait of a good SVT tester is to have a clear understanding of the customer values of each feature.
While it may be easy to find the promised customer values of each feature from the introductory section of a design document or user manual, it is not trivial to incorporate them effectively as the core of the testing design. To achieve this, a tester needs to understand the adoption cycle of a technology, and where the technology currently lies on its adoption curve. He also needs to understand the competing technologies that serve similar customer values, and the collaborating technologies that are likely to be used together.
A common mistake I observe among newer SVT testers is that they tend to look at the technology (or the product feature) in isolation rather than holistically. As a result, the testing is more focused on the conformance to the design rather than on the true value that is delivered. For instance, when container-managed persistence was introduced as a part of the J2EE™ technology, the goal was to deliver a simpler and more portable programming model to access databases; the competing technologies were other persistence technologies, such as JDBC and persistent-able data objects. If the technology or feature is not easy to use, and cannot scale or perform at least as well as competing technologies, it is difficult for the feature to get adopted -- even if customers buy the product, they will not use that feature. Therefore, the SVT tester's job is to discover problems of this nature and report them before the product is out of the door.
Does it it work, or does it also work better?
A while back, a tester explained to me how the new secure conversation feature she was testing helps to make Web services perform better and more secure, with each invocation contacting a "trust server" for authentication. I wondered if the time used for this additional call to the trust server was small enough to avoid negating the time saved by this new feature -- and if this trust server introduced a single point of failure to the system.
As important as technical knowledge of the feature and its related features are for SVT testers, technical knowledge alone is often not sufficient. Using customer values as the core of feature testing also requires testers to go the extra mile and, in some cases, cross organizational boundaries. When I raised my concern about the impact of the trust server, the tester's reply was "I don't test performance, the performance team does that." Another time, I had a customer question regarding how the product handles transaction recovery under heavy load. The tester who performed the exact test scenario was unable to answer the question, because she "doesn't do load testing."
To some degree, this could be a reflection of how limitations have come to be defined within an organization. Of course, few organizational structures can address every situation perfectly. I believe that a good SVT tester is willing to go beyond just the testing of the feature itself and is vigilant to anything that could potentially prevent the feature from realizing its promised customer values, whether it's a competitive technology that outperforms the feature, collaborative features that are not integrated well, or the feature itself that falls short of the expectation or creates additional weaknesses in the system.
Just as features and products do not work in isolation, they do not stand still in time either. Products and technologies evolve at a faster pace than ever before. Good testers are lifetime learners and, more importantly, fast learners. The learning environment for SVT testers is very challenging. SVT testers must typically learn about features while the design is still in flux, the documentation is not yet available, and the code base is potentially unstable. When they begin to have a decent grasp of the feature, the release date is around the corner. As a result, many SVT testers learn features by experience, through trial and error. While this is often an effective learning style, it can be problematic for a tester to rely on this as the sole learning mechanism, since it can be difficult to tell if a feature's behavior is by design or caused by an error. This also implies an added danger if all information from developers is taken as being truly correct, which I tend to the "student mentality."
A much better way to learn is to start from the problem the feature intends to address, whether it be a business problem or a technical problem, then understand the key ideas behind how the feature solves the problem. The feature's capabilities represent the concrete form of those key ideas. In other words, good SVT testers will spend more time studying the problem than the solution. Testing is treated as the way to measure how good the feature is as a solution to the problem.
An SVT tester develops a good test scenario to effectively exercise the feature, armed with a solid understanding of the feature, its related features, and the problem it intends to address. Most of the time, the test scenario will not work as initially expected. It is the purpose of testing to flush out these problems. Good SVT testers analyze the role each component plays in the system, determines the problem, identifies the bottlenecks within the system, and collects evidence to verify their expectation. If the behavior of the system continues to oppose expectations, the testers can either redesign the test, or challenge the design of the feature. In short, good SVT testers are also good system analysts. To illustrate, one of our test applications was intended to stress J2EE capabilities, such as EJB components, Web services, and so on. However, the system had a bottleneck in the data access logic, and so while the overall CPU usage was high, the test scenario was considered stress testing, even though the stressed feature was not the intended target of the test.
Balancing risk and practicality
Another challenge all testers constantly face is practicality and prioritization. It is virtually impossible to test "everything." The entire universe of things that could be tested is typically too large to be covered within a given schedule, even more when multiple features are combined and the permutation grows exponentially. Add to this middleware products, like WebSphere Application Server, on top of which you can build applications. The number of ways applications can be designed is essentially unlimited. Therefore, the question becomes how to prioritize the test scenarios, and how much testing achieves an optimal balance between the cost and benefit. This still goes back to the customer values, technology adoption cycle, and the problems the features are intended to solve. Sure, the priority order could be off and important test scenarios could be missing, so there is always some level of uncertainty, but good SVT testers acknowledge the existence of risks, painstakingly identify the risks, and then manage them on an ongoing basis. They document the assumptions and omissions before the release of the product, and seek verifications and corrections post release through feedbacks from customers. The checks and balances and safeguards of a proven SVT methodology ultimately offsets such inherent uncertaincies.
I firmly believe that good SVT testers should be good programmers, and I believe this for several reasons:
- Unless you use the technology, you will not fully understand it.
- Without basic programming skills, testers cannot research, debug, and narrow down the root cause of a problem in a way that will help developers resolve the issue productively.
- Testers need sufficient programming skills to communicate with developers effectively and, perhaps more importantly, to earn the respect of the development community.
In my experience, testers without decent programming skill have low testing productivity and their opinions are not regarded as highly by developers, while testers with good programming skill get problem resolved more quickly and their feedback is respected more.
While it is of so surprise that programming skills would benefit testers of middleware products, I argue that technical skills are also very important for testers who work on end user products. All software products evolve over the time and the cost of regression testing increases accordingly. Without appropriate automated testing, test results cannot be reproduced reliably and regressed efficiently. As a product matures and the customer base increases, the cost of regression testing can become so high that it can eventually deter innovation and the pervasive improvements to the product if tests are not extensively automated. Without the support of proper programming skills, developing and maintaining automated test scenarios can be so costly that manual testing becomes more economical. Repetitive manual regression testing not only makes the testing job less interesting, but also becomes so expensive that management faces the tough choice of regression tests, new tests, or service cost.
Last, but not least, a good SVT tester needs a set of soft skills: communication, negotiation, and people skills. These are important for any job, and many people might not associate these skills with software testing, but they are very important skills for testers since a fair part of their job is to convince others that changes may need to be made. The developer's goal is to make things work while a tester's goal is to identify things that are not working. Rather than spark tension, good testers will clearly communicate a problem, diplomatically negotiate the commitment of changes, and effectively resolve any difference of opinion. Good SVT testers do not just do what they're told by developers; instead, they act as true peers to developers and work collaboratively to make the product better.
In summary, good SVT testers have a solid understanding of the customer values that each feature delivers, and makes these values the core of their testing -- even if it means going the extra mile to achieve it. Testers are not only quick learners, but they include the problem that the features intended to address in the start and the end of their learning. They manage risks and verify assumptions carefully, they are good system analysts and programmers, and they have good communication, negotiations, and people skills.
Although responsibilities can be divided, it's not impossible for one person to acquire all these skills. It could be a challenge, but who said testing was easy? As an SVT test manager, I make it the most important part of my job to acquire and develop these skills in myself and for the members of my team. There is much more to SVT testing than most people expect, and so I hope that this has helped explain the true scope of the testing task and the range of skills that are needed for the job to be done well. I hope most of all that anyone who may already possess these skills does not think he or she is too good to be a tester, and realize instead that it is a specialized and highly critical role in the development cycle.
Chinhua Q. Wang is a senior SVT manager with IBM working in WebSphere Application Server product development organization in Austin, Texas. She manages a group of SVT testers focusing on Web Service and Service Component Architecture testing. Prior her manager position, she was a senior software engineer and a leading developer in WebSphere development organization.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
