Managing test automation in continuous testing
monica914 060000WAY8 Visits (4848)
I’m still thinking about what the right content should be for continuous testing. My previous blog was about thinking about what test content matters. But there’s also the consideration of how much content and considering avoiding waste of building too much automation. I’ve had the experience several times of too much success (amazing, right?) in building automation. In those days, it was actually great to have a robust test automation suite that took 12-18 hours to run. We actually measured our success at testing by how much we could add to the suite. Since we were getting builds 1 or 2 times a week, it was no problem to run automation for almost a day. So it wasn’t actually “too much” success in that era.
However, continuous integration builds run at least one time a day, more often multiple times a day, and the testing cannot run that long. Yet in order to support continuous delivery, we absolutely need test automation. The testing problem is once again a series of delicate balance decisions, although these decisions are slightly different. For instance, if we need to keep the test automation to say under 30 minutes of runtime, is it worthwhile to invest in an automation framework? I’m going to still say “yes” because frameworks help ensure that the automation is structured and therefore more robust. It doesn’t help if the automation runs in under 30 minutes, finds a bunch of problems, and all those problems turn out to be automation defects, not application defects.
Another interesting new-ish question is how to keep the suite under 30 minutes and ensure it’s covering all the right things. As the team continues to build the software, they are continually adding test automation (or at least we hope so). It’s going to pretty quickly get to longer than 30 minutes of runtime. So how does the team prune? I’ve spent several days turning over ideas around this one and have decided that I would recommend a series of short test suites. The team establishes a time limit for the automated testing and adds new automation to a suite until it reaches that limit. Then start a new test suite. When running a build, the test suites are stepped through in order. I haven’t tried this, but I’m thinking it has some interesting promise for scaling. If you build every hour, maybe your test suites are 5 minutes long. But in a 24-hour period, you will be able to run 24 unique test suites of 5 minutes and this is a good amount of testing. If you build less frequently, say twice a day, maybe the team sets the test suite time limit to 45 minutes. It seems to me that a technique like this could really maximize the amount of test automation the team can consume and not have them wasting time looking at the suite to remove enough tests to keep the builds running quickly enough.
I’m sure there are some more questions that will occur to me soon as the topic of continuous testing is definitely percolating with me!