 |
|
The Software Quality Picture Dan Gouveia works for IBM Rational Software. Dan is a member of the sales team, ensuring Rational's software tools meet the demands of customers. His primary focus is on the Rational Software Quality solutions. Prior to being a sales engineer, Dan was a post-sales consultant. Before that, he was a QA Engineer in the start-up world. When Dan isn't working, he is out seeking his next big thrill mountain biking, scuba diving, shark diving, kayaking, skiing, snowshoeing, or camping.
|
|
|
The Summer Time Slow Down
The Summer Time Slow Down is upon us. Vacations are rampant. We find ourselves with some extra time between projects. Having it is a luxury. So ... what do we do with this extra time that is now sitting in our hands, oozing between our fingers? One suggestion would be to review a couple of projects and begin tightening the loose screws in the process.
Sometimes this can be a daunting task -- a leviathan, daring us to attack it. No problem. Simply de-construct things into smaller pieces. For each project that we review, look at the things we did. For instance, did we perform ...
1. Test Management?
2. Manual Test Script Creation?
3. Functional Test Automation?
4. Performance Testing?
Once we have divided things into smaller components, we can break out our magnifying glass and dig into each one. We can make this as formal or as informal as we want. However, we want to make it fun. For instance, we can set up a series of meetings that take over a meeting room during lunch. We can order up some food or make it a brown bag affair. Perhaps we even bring in some of everybody's favorite tunes. The idea is to create a series of collaborative sessions that our team wants to participate in. Once we create the desired environment, let the nit-picking being.
When we begin to dive into the smaller chunks we should treat it like we're newspaper reporters. We should ask the basic "who", "what", "when", "where", "why", and "how" questions. They will help us identify key pieces of information.
WHO - identifies the team members involved (both on the test team and within the extended teams, such as project management).
WHAT - identifies all of the things that influenced the project (test cases, requirements, schedules, tools, etc...).
WHERE - identifies where things took place (e.g. offshore? multiple sites?)
WHY - identifies the baseline of the objective (e.g. What was the mission? Was it to release software sans Sev0/Sev1 defects and have all Sev2 defects addressed via technotes?)
WHEN - identifies the overall schedules (iterations, phases, builds, etc...)
HOW - identifies how the project developed, tasks that occurred, risks that were mitigated, etc..
We can now start identifying critical pieces of each chunk. Let's try it with our Test Management example.
WHO - QA Analysts on the team, QA Manager on the team, Business Analysts from the Requirements Team, Project Manager for overall project
WHAT - Develop test cases, designs, schedules, and strategies
WHERE - Development is both onshore and offshore. Testing is being handled onshore.
WHY - Verify the quality of the application. Don't ship with Sev0 and Sev1 bugs.
WHEN - 6 month project. Multiple iterations, pushing out multiple builds per iteration. Next iteration doesn't occur if Sev0/Sev1 bugs exist.
HOW - Use Word to capture test plan, Excel to capture manual testing, and homegrown defect tracker
We have just collected facts about our test management. If we were to then identify problems that we encountered in our projects, we could apply them to our facts and begin to see correlations. For instance, some project issues may have been:
** Application didn't meet customer expectations
** Released software late
** Defect tracking wasn't managed well
OK!! So ... what if we look at some simple correlations:
Application didn't meet customer expectations -> Didn't trace test cases to requirements
Released software late -> Test schedule got pushed
Released software late -> Didn't use test automation tools.
We have just identified 3 loose screws. To tighten them, we need to look at the different answers that could apply. Could we invest in test automation tools for the next project? Do we develop a contingency plan for accelerated test schedules (e.g. test all critical and major requirements)?
Ultimately, we are looking at iteratively honing what we do. What can be better?
Categories
: [ Assurance | Automation | Metrics | Process | QA | Quality | Test ]
Jul 01 2008, 09:45:35 AM EDT
Permalink
|
|
Plug it in ... Turn it on ... Does it Smoke?
Got a smoke? A smoke test, that is!
Since my last entry, I have busy working with customers. When I engage with them, I look to find the types of test suites they run. I have been pretty surprised that the majority of these customers don't have a smoke test. Further, a good portion of those customers never heard of a smoke test. Hence, I figured this would be a good topic for the blog.
The smoke test is actually derived from our hardware brethren. The test was actually pretty simple for them. They would power the hardware device up and if it smoked ... it wasn't working! Pretty simple stuff!. In the software world, we're looking at applying a series of tests to see if the software is working. The tests are derived from the larger regression suite we run. In fact, we try to pull one or two test cases from each major area of functionality (or Use Case). Once aggregated, these tests help to assess the architectural stability of the current build. The idea behind this is that if one of these test cases fail, then the build isn't worth testing because a core/architectural piece of functionality has an issue.
So what's the value behind this? Well -- it is a way for us to ensure we spend our time wisely. For instance, if the print portion of our application has an issue, do we really want to run all of the tests for that build? What if the issue was as simple as the application can't print. The good majority of our tests would focusing on printing from the application (e.g. different page counts, different layouts, different printers, etc...). Do we really want to run those test cases, creating defects for each thing -- even though they are all related to the fact that the application's printing functionality is broken? Or would we rather look to see if the core problem is that the app just can't print! We would only need to log one defect. The smoke test can save us time. It can make us more efficient.
A few entries ago, I talked about what to automate. I mentioned that there are a lot of thoughts and opinions on this. My opinion is that, if you can, automating a smoke test would be a valuable tool. You could add it into the build process. When the application gets built, your smoke test gets executed. If this is done overnight, you can come in the next day to see what the results are.
We all know that smoking is bad for our health. However, smoke testing is great for our QA efforts.
Categories
: [ "automated | "automation" | "qa" | "smoke | "testing" | test" | testing" ]
May 16 2008, 08:34:04 AM EDT
Permalink
|
|
Another tool in the toolbox ... PurifyPlus
I think we can largely (and perhaps completely) agree that there are two major buckets we concern ourselves with when testing software:
1. Requirements - We want to make sure the application works per spec ... in other words we ensure that the application is doing what the customer asked for!
2. Code - We want to make sure that we kick ALL of the tires on the rig! It is never any fun when customers test our software for us!!
Rational has their finger on the pulse of requirements-based testing. We have the ability to tie requirements directly to test cases directly to defects (e.g. RequisitePro->ClearQuest Test Management/TestManager->CQ). Did you know that Rational also has its finger on the pulse of code-based testing? Ever hear of PurifyPlus? You typically find it on the developer's spice rack. When their cooking up their applications, they give a good dose of PurifyPlus. They primarily use it for the Purify feature. Purify allows them to look at their code for any memory issues (leaks, array errors, etc...). If the developer's don't mind you raiding their spice rack, check it out! I think you might find the PureCoverage portion quite useful.
As its name infers, PureCoverage allows you to get code coverage metrics. These are those lovely reports that tell you how much code your test suite is touching. This is the stuff dreams are made of. This let's you know if you're hitting all of the logic in your application (conditional, loops, etc...). It'll also give you an idea of whether you need more test cases/scripts or simply need to add more data to your tests (e.g. is your datapool broad enough to hit all of the code?).
I have been thinking about this lately. I believe the advent of server-side code pushed code-based tesing "out of sight ... out of mind" for a lot of QA teams. There is some great information on IBM developerWorks that discusses how to set up PurifyPlus to profile server-side code. For instance, here are two ... one for .NET and one for Java:
** .NET
** Java
Is it for everyone? Probably not! PurifyPlus focuses on certain technologies, not all technologies. However, you should consider making coverage-based testing part of your arsenal if you can.
Go ahead ... take a bite and chew on this for awhile!! Let me know what you think ...
Categories
: [ Automation | PureCoverage | PurifyPlus | QA | Test ]
Apr 17 2008, 04:54:55 PM EDT
Permalink
|
|
The Gray Scale of Test Automation
Wow ... I can't believe I let this much time lapse between entries!! I do apologize for the faithful readers, however few or many there are. I have lots of pots boiling away on the stove. Hopefully, they'll turn into the full-course menu that I am hoping for.
Gray is a color that we derive from a mixture of black and white. Mixing more black in creates a darker shade of gray. The addition of extra white lightens the shade of gray. Pretty simple thought ... I know! But it was a powerful enough thought for me to head down to the laboratory to stitch together the following idea:
THE GRAY SCALE OF TEST AUTOMATION
Let's break this down in the simplest of terms:
White = White Box Test Knowledge/Skills
Black = Black Box Test Knowledge/Skills
Think of this as a virtual management tool for building out a test automation team. If you were diligent, you would research the going rates for Testers, Test Automation Engineers, and Developers. You could put those in the scale and then figure out your budget and your needs. Testers would fall on the black side of the scale, Test Automation Engineers would fall in the middle of the scale, and developers would fall on the white side of the scale. Do you just need record/playback automation? You could hire people black side. Do you need automation framework/architectural design? Hire somewhere in the middle of the scale. How about people that not only develop tests, test automation, and frameworks ... but also can work on bug fixes. Hire on the black side of the scale. Ideally, you would want to look at picking people up all over the scale. Take a look at your needs and your budget. Ideally, you would pick up a couple of "dark gray" testers and filling the rest of the team with "black" testers. This would give you the ability to design the architecture that supports complex automation AND record-playback scripting. If you strike a deal with the development team, stealing some of their dollars, you could add the "white" tester. That's the one that changes the tire while the car is moving!!
Deep thoughts? Nah ... just a simple rambling of the mind. I am simply offering a thought on how to flesh out a test team. They don't all have to be hardcore automation engineers. However, you shouldn't just rely on your record-playback testing either.
C'mon ... just think about it would ya?!? If you don't like the idea ... let me know. I'm open to suggestions!
Categories
: [ QA | automation | code | coverage | test ]
Apr 01 2008, 05:50:30 PM EDT
Permalink
|
|
Do you believe?
It's a pretty simple question, really.
I'm in technical sales. This is a question that I ask myself all of the time. Do I believe in what I'm selling? To me ... that is the deciding factor for my success. If I don't believe in what I'm selling, my energy about the solution goes down. I become transparent. The company I am selling to doesn't buy into my presentation. If I do believe in what I'm selling, then I can generate the excitement that I seek from my customers. I can make an abstract concept tangible for them. I can stitch analogies into my presentation to help my audience draw upon their concrete knowledge about something and relate it back to what I'm talking about. This question shouldn't just lie in my domain. In fact, when I was a tester, it didn't.
I remember my first testing job. It was at a pre-IPO company back in the 90s. I worked with a bunch of ex-AOLers that missed the "big one". They were banking on the company -- that I just got hired by -- to be the next AOL. Seeing as how I am working at IBM today ... you can probably guess how successful that company was. Nonetheless, I learned a very valuable lesson during my first few weeks there. My manager sat me down and explained that it just wasn't about testing the software to make sure it works. It was about putting myself in the end-user's seat. Look at the software from their point of view. That was a pretty novel concept for me. It helped me to revisit my testing ideals. I moved forward, testing the company's software, not only looking for bugs but also what would make this software go from good to great. Did my enhancement requests always get heeded? No. We all know the realities of a software development project. There wasn't time. However, it led me to believe that I could affect the software. I was, after all, an end-user of sorts. This was something that I have carried with me to this day. I can make things great. If I look at things from an end-user perspective, I can help to develop the right solution for a customer. So what does this diatribe mean? You have to believe!
I assume that all of us, at some point or another, have taken a job for the sake of having a job. Sometimes the circumstances warrant it. Sometimes, we just want to escape the pounding of the humdrums at our current job. Regardless, when we go to test software, we need to believe in it. We need to believe that, as end users, this is going to be a great application. So don't just look at testing as covering your requirements. Don't just look at it as beating on the system. Look at it as though you were the end user purchasing the software. Think to yourself, what would make this great.
Until the next rambling ...
Categories
: [ Assurance | QA | Quality | Software | Testing ]
Mar 05 2008, 08:47:23 AM EST
Permalink
|
|
The Power of "Why?"
If necessity is the mother of invention, the question "why?" is the skeptical aunt. They are born and raised in the same family. We turn to them to nurture our thoughts and ideas. One to spawn the "aha" moments. One to keep us grounded.
What does this have to do with testing? Perhaps, I'll ask it another way. Why am I talking about this? Simply put, I believe that the power of "why?" allows us to hone our testing efforts as well as our personal development. As we develop our strategies for any given testing project, we should inject a healthy does of these "why?" questions. Further, we shouldn't just settle for the complacent answers. Don't live with "because that's how we always did it". You don't have to digest "because that's how it should be done".
A great example of this is SOA. A lot of companies are looking to bring in SOA-specific tools. Why? Often times it is due to the fact that the hype around SOA breeds a mindset that if a tool says "SOA" then we need it. Do they actually need it? Maybe! Testing tools for SOA are geared at headless web services. Basically, companies want to be able to test web services when there isn't an application built to consume them (e.g. a web or portal app). They bring these tools in to test the exposed methods of a service. Some companies will need to do this. Other companies will not. Why? Some companies create test apps for this. They are very rudimentary web pages that provide a UI for sending data to the service and receiving -- hence validating -- the information coming back. These companies will not need an SOA-specific tool.
Another example is the adoption of test automation. A lot of companies tend to think this is an absolute necessity at the time they think about it. Often times, it is not a wise choice. Why? Adopting testing tools should be a very strategic process. Part of the process is ensuring that you have the right project, at the right time, with the right team. Lobbing automation tools at a test team that is in the middle of a project may not be the wisest thing. Why? They will lose a great deal of time learning the tool and, more importantly, where to place the tool in regards to their test process.
Should you be paying any attention to this diatribe? Maybe! Why? It'll help to develop that niche piece of critical thinking that always keeps you on your toes. Why? Think of it as the little voice in your head that keeps you honing and re-honing the way you do things.
Time for me to split to the place where Tom Petty said it best -- "...into the great wide open ..."
Categories
: [ Assurance | Critical | Engineer | Planning | Quality | Test | Thinking ]
Feb 22 2008, 09:36:15 AM EST
Permalink
|
|
File Verifications
Good Morning --
I was going through some old code that I wrote a ways back for a customer engagements. It kind of reminded me of cleaning out and attic, basement, or garage. Why? Because I am amazed at how much stuff can actually amass in one given area. It is also similar in the sense that I will always find one thing that stirs up a pot of nostalgia. An old hat, a sweatshirt, a book, an audio tape, etc... It is that one thing that I will hold onto, wearing it, reading it, or playing it in the old, dusty tape player. The compulsive need for that item usually lasts about a week, until I realize why the object of my affection made its way to its land of exile in the first place. The tag in the hat or sweatshirt will stab me a few times, the book on how to tie a 1000 knots didn't get any easier, or crease that occurred in the audio tape 10 years ago never ironed itself out. That's where things differ with the code snippets that I keep. I can usually re-purpose code snippets, using them well past the week-long period of nostalgia.
The particular snippet that is cause for today's entry is one that deals with Checksum values. It was written in the context of a performance test. I was tasked with testing the performance of an FTP server. I was also tasked with ensuring that the files I retrieved maintained their integrity. The customer required that I verify the integrity of the files using their size and Checksum values. When I happened upon this, I thought to myself -- "Gee, this could be a pretty cool way to do file verifications in functional tests." It would help to eliminate the dependency on byte-by-byte file comparisons. It would also be simpler than writing a character-by-character verification.
First a brief description of what a Checksum is (a decent definition from http://info.ssl.com):
A checksum is a value that is used to check the integrity of data. Checksums are generated by a function that is dependent upon the data in question. For security purposes, checksums are generated by one-way hash functions. Once a checksum has been generated, it is either stored with or transmitted with the data in question. The integrity of the data can be checked by generating a new checksum. If the two checksums are identical, then the file has not changed. If the two checksums are different, then the data (or file) in question has been altered.
So basically this is a value that your developers will cook up. They'll create it, using an algorithm. Typically, it is stored in a "checksum file". This is a very small file that gets included with the data you want to verify. So how would you create a verification for this? Well ... let's boil it down into some simple steps:
1. Find out what checksum algorithm your developers are using
2. Obtain the checksum value (either from the developers or the checksum file)
3. Create your checksum verification
SAMPLE CHECKSUM VERIFICATION FOR RATIONAL FUNCTIONAL TESTER: (using the CRC32 algorithm in Apache's commons-net-1.4.1.jar)
===================================================================================================
public void verifyChecksum(String filename, long checksumDecimal, String checksumHex){
try{
File out = new File(filename);
long checksum = FileUtils.checksum(out, new CRC32()).getValue();
Long expectedDecimal = new Long(checksumDecimal);
Long actualDecimal = new Long(checksum);
String expectedHex = checksumHex;
String actualHex = Long.toHexString(checksum);
vpManual("Verify_DECIMAL_Checksum", expectedDecimal, actualDecimal).performTest();
vpManual("Verify_HEX_Checksum", expectedHex, actualHex).performTest();
} catch(IOException ioe){
System.out.println(ioe);
}
}
Anyhow ... just some thoughts to help you push the boundaries of your testing ...
It's time for me to make like Michael Jackson and "Beat It"
WOULD YOU LIKE TO SEE THE CHECKSUM METHOD FOR RFT .NET? LEAVE ME A COMMENT OR SEND ME A NOTE -- I'LL HOOK YOU UP
Categories
: [ Automated | Automation | Checkpoint | File | QA | Testing | Verification ]
Feb 14 2008, 09:22:43 AM EST
Permalink
|
|
What Rolls Downstairs And Out The Door And Over Your Neighbor's dog?
It's great for a snack ... It fits on your back ... It's LOG LOG LOG!
I figured that I'd give a little something for all of you "Ren and Stimpy" fans.
Anyhow ... I'm back! The NE Patriots have lost the big game. I am sidelined from mountain biking due to the weather. I don't have any money to go diving somewhere warm. I guess I'll have to return all focus to work. I would like to keep things interesting though. Let's go a little Tarantino on things. We'll start at the end of the automation cycle, look at Rational Functional Tester's XML logs, and wind ourselves back to the start.
IBM Rational came out with the ability to write XML-based logs within Functional Tester. A lot of people asked for this. They like to have control over publishing their logs, providing access to people that may not have the necessary software installed. I thought to myself, "wow -- this is cool!". Of course, I needed to dig in and see what could be done with these little beauties. Here's what happened ...
Pulling on the third book in my bookcase, I unearthed the secret door that led to my mad lab. I traversed the winding staircase until I came to the lab where so many experiments occurred, some good ... some bad! I pulled down the necessary implements for my latest foray into the unknown. I busted out some "Eye of Newt", "Toe of Frog", VB.NET, limited XSL knowledge, and Rational Functional Tester's XML logs. After the stench and smoke cleared, I was left with a neat little utility that allowed you to convert your Rational Functional Tester XML logs into pretty cool HTML files. I couldn't tell you the exact amounts of dark magic that went into this tool, however, I can tell you what I did with the XSL, VB.NET, and RFT logs.
The basis for the success of this utility is XSL. This is what is used to actually convert the XML into HTML. Basically, we apply an XSL stylesheet to the XML, telling it how is should look as HTML. The VB.NET piece is simply a means wrap this transformation practice into a user-friendly (read "WICKED SIMPLE") tool to use.
Want to take a look at it? Send me an email, I'll be happy to send things over to you. Please remember, since this was cooked up in my own secret laboratory, it is AS-IS software, not supported by IBM. That's why I also include the source code for you to play with.
LASTLY ... A SOLICITATION FROM YOURS TRULY. SEND ME SOME FUNKY PROBLEMS THAT WE CAN DISCUSS. WE CAN LOOK AT BUILDING ANY COOL UTILITIES, SCRIPTS, QUERIES, ETC... IF YOU CAN'T FIND THE SOLUTION ANYWHERE ELSE ... PERHAPS WE CAN HAMMER SOMETHING OUT!!
As the Pixies once asked - "Where is my mind?" -- I am off to go find it.
Categories
: [ Automation | Functional | Logs | QA | Rational | SQA | Test | Tester | Tools ]
Feb 05 2008, 09:35:04 AM EST
Permalink
|
|
Segmented or Round-Trip Automation
OK -- I am back from my hiatus. I am fully enabled and ready to help people with their software.
So ... my first couple of posts were kinda fluffy. The were based around just some welcome thoughts and crazy things like figuring out what your value is. Today, I would like to put some spin on the Rational tools. In particular, I would like to go ahead and talk a little about creating scripts with Rational Functional Tester and/or Rational Robot.
Chances are, when you take a test automation tool class, you will learn that there are two strategies for creating scripts. You can create segmented or round-trip scripts. Let's take a look at each strategy and comment on their usefulness and drawbacks:
Segmented scripts accomplish one piece of the whole. That is, it will pick up where another script left off, do something specific, and then leave off for another script to continue through the execution path (or terminate things if it was the last script). The key is that you will need a script to call each of the "segmented" pieces. It usually looks something like this:
==========================================================
public void testMain() {
callScript("Script1"); // start application & log in
callScript("Script2"); // walk to part "a" of your app
callScript("Script3"); // verify part "a" does something
.
.
.
callScript("ScriptN"); // Close app and clean up
}
==========================================================
Of course, you would have created Script1, Script2, Script3, ..., ScriptN. You simply call them in the specific sequence that you need them to run. Is this a good (e.g. useful) strategy? Sure it is. I would wager that most test automation shops follow this type of strategy. It allows you to create both helper/navigation scripts as well as the actual test scripts. Each script focuses on a specific task -- whether is setting up a database, starting the application, performing a verification, or some sort of other useful task. These scripts are typically small and easy manage. What's the downside? Their brittleness when included in a long regression test. If one of the first few tests fail, typically the rest aren't going to be of much use. They all depend on where the prior script leaves off. If one of the first few scripts fails and doesn't leave the application in the state for the next script to successfully run, you hit the domino effect -- every ensuing script is going to have issues because the application isn't in the state that it needs to be.
Round-trip scripts are best explained as scripts that do everything. This strategy is used so that each script become independent of the others. For example a round-trip script might look like the following:
==========================================================
public void testMain() {
startApp("some app");
navigational step 1; // walk to part "a" of your app
verification step; // verify part "a" does something
.
.
.
clean up step; // Close app and clean up
}
==========================================================
What one round-trip script accomplishes may take 3 or more segmented scripts. What's good about this strategy? Independence! You don't have to worry about your regression execution becoming completely useless if one of the first few segmented scripts fail (e.g. the domino effect mentioned above). Each round-trip script sets up what it needs to, does what it needs to, and cleans up what it needs to. What's the drawback? Waste. There is a lot of duplication that occurs (e.g. starting the application multiple times, doing the same navigational steps, performing the same clean up steps, etc...).
So which one is right for you? That is fully dependent on what your goals are. You will probably end up going with the segmented strategy. It provides for small, focused, easier-to-manage scripts. However ... let me leave you with this thought. Could you find use for round-trip scripts if you incorporated them into your Test and Defect management process? For example ... would it be useful to have round-trip scripts to handle the disparate list of defect fixes you need to verify for a new build? You wouldn't have to worry about them executing within a particular sequence.
Anyhow ... just topic to chew on. Until the next time ....
Categories
: [ automation | qa | software | test | testing ]
Jan 25 2008, 09:59:36 AM EST
Permalink
|
|
Stats and Statistics
OK - I'll admit right up front that I am not prepared. I spent my weekend focused on the New England Patriots. I hope you will provide me with a pass on my lack of readiness. Afterall -- I did have to figure out where I was going to watch the Patriots game, what I was going eat, what kind of beer I was going to drink, and if I would return to my house to watch the NFC Championship Game. I did take one golden nugget out of the games I watched though. Statistics!
It always amazes me the amount of statistics that are garnered for any given player, team, sport, etc... A lot of times there are some pretty far out statistics. The number of catches a receiver makes in the 2nd half of playoff football games in the month of January. The number of carries that a running back gets on a rainy day versus a snowy day versus a perfectly sunny day. The number of passes a quarterback makes in the first 3 minutes of football games that falls on an odd numbered day in the month of October. I'm exaggerating, of course! However, I challenge you to sit through a sporting event on television and pay attention to the statistics. They do get pretty funky. Soooo ... that will be today's topic.
Statistics just don't exist in sports. They are alive and kicking in the software development world. If you have any question, just ask a development manager how many lines of code there are for a particular application. Ask a developer how many components there are. Ask yourself, how many test cases you have. Don't forget the defects. There are trending statistics, density statistics, static statistics, etc... Hey -- it's all numbers, right? Our job, is to make sense of them.
Let me step outside of our statistics box. Let's look at software tool evaluations. I'm sure that you know what they are and have probably been involved with an evaluation or two. When we evaluate software tool, we're primarily looking for the product the best meets our needs. For instance, in the software test automation world, we want to find the tool that will best fit our testing needs. To find out what best meets our needs, we need to first establish what is of value to us. Is it ease of use? Is it a powerful scripting engine? Is it datapooling? Once we have our list of values established, it is a matter of assessing any given tool's features to see if they provide the value that you are looking for. What if we didn't set the list of values that we were looking for? Further, what if we attended a "feature dump" demo, where features were not correlated to the values that they provide. We would be left simply looking at what a tool does. I see the same truth about statistics. If we don't know what values we need from our statistics, they're just numbers.
So I ask you this ... do you know what your statistics mean to you? Can you make a "go/no go" decision based off of them? If you answer NO to either or both of these questions, you may want to peel the onion back and evaluate what your value list is. Are you looking for requirements coverage? Then your statistics will be tied to the number of test cases that validate requirements. Are you looking for code coverage? Then you will need to find out if your current test case suite covers the percentage of code you're looking to cover. How about defects? Does a density report necessarily tell you that one component is weaker than the others? Pretty much so! However, what about the flip side of that coin? Are you just testing that one component more than the others? Ahhh ... that starts speaking to test case metrics. It's all about the value, whether it is features of a tool or statistics in a development world.
Before I sign off, I would like to put this spotlight on test automation. I often hear customers say they are seeking 100% test automation. That's fantastic. It is all idealistic. I believe the majority of people would agree with me that you don't automate 100% of your test cases. Why? Sometimes you can't (do to technical issues like 3rd party controls). Sometimes you don't (the cost-benefit analysis would show it isn't worth spending the time and money automating certain test cases). Could you shoot for 40%? 60% 80%? Sure ... but does that statistic add value? In other words, would it be of more value to have a similar chunk of your testing automated than a disparate chunk? I would love to have an automated smoke test (e.g. a small percentage of a test suite). I am not a fan of disparate test automation, covering a decent percentage of my test suite. I like consistency. I like to know that a certain chunk of my testing is done versus a smattering here and there.
Anyhow ... as Led Zeppelin likes to say -- "Ramble On"
Categories
: [ automation | defects | management | qa | statistics | test | testing ]
Jan 21 2008, 09:01:35 AM EST
Permalink
|
|
First Order of Business
Welcome! Welcome! Welcome!
You have found the Software Entomology blog. The inaugural set of my musings anyhow.
Why is this blog named "Software Entomology"? Well ... Why do you call yourself a "Quality Assurance Engineer", or "QA Specialist", or "SQA Analyst"? Have you ever thought that you could be a "Software Guidance Counselor"? It is all semantics, afterall! No matter how you slice it, we look for defects, often called "bugs", in the applications that developers write ... Software Entomology, right?
When you come here, I don't wish for you to find stale, boring data points, surrounding testing (tools, processes, etc...). I want you to come here to be stretched a bit. What do I mean by that? I want you to come and read my blog when you want to think a zip code or two outside of your box. That means you'll find some outlandish ideas, inane thoughts, and topics that may challenge you. I want you to think of this as the laboratory where the MadTester dwells. The place where we'll concoct stereotypical smoking, bubbling, liquids in beakers, test tubes, and vials.
If you haven't figured it out from the two puzzle pieces above. I really want you to come here to relax and have some fun. Will you find code snippets here? Probably! Will you find some tips-n-tricks for Rational test tools here? Most likely! Will you chuckle at what we're talking about? I hope so.
So come on ... grab your lab coat and let's make some potions! Who knows -- maybe we'll turn ourselves in "Mr. Hydes" and beat some developers along the way.
Talk to you later!
Categories
: [ "test | automation | management" | qa | sqa | test ]
Jan 18 2008, 12:46:28 PM EST
Permalink
|
|
|
| S | M | T | W | T | F | S | | | | | 1 | 2 | 3 | 4 | | 5 | 6 | 7 | 8 | 9 | 10 | 11 | | 12 | 13 | 14 | 15 | 16 | 17 | 18 | | 19 | 20 | 21 | 22 | 23 | 24 | 25 | | 26 | 27 | 28 | 29 | 30 | 31 | | | | | | | | | | | Today |
|
