Pressing the ‘delete’ button on the test automation backlog
monica914 060000WAY8 Comments (4) Visits (11766)
There’s been a refrain I’m hearing lately: “The team has six weeks of manual testing at the end of every release cycle. In order to get to continuous delivery, we have to address all this technical debt and automate those tests– we can’t possibly find enough time or resource to do that. “
Are you sure?
Right now you’re thinking this is going to be another one of those exhortations to prioritize technical debt because it’s killing the team. Nope. I’m suggesting you forget about it and consider a new future.
Let’s say the team can really get to small batches (we will deal with this assumption later, just play along for a minute). Delivering small amounts of code continuously shouldn’t require massive amounts of regression tests. There’s not much change in each batch, so there’s, um, not that much you can break.
Ah, you say, but you never know who might’ve broken what & we have these complicated products and I just don’t understand.
Imagine for a minute that you embrace this idea. The team is delivering small batches. Every time a line of code is committed an automated test is committed with it. Any time a piece of old code is touched, a new automated test is committed with it. [Sidebar: These automated tests are not lightweight unit tests; they are acceptance tests that emulate user activity.] Every delivery kicks off the pipeline & the tests are run automatically. It’s easy to tell which code change broke something. Let’s call this idea “rolling forward.” Don’t create any new technical debt.
Because the team is delivering all the time, if a problem is found in production, the fix can also be delivered quickly. So far, so good. But what if the problem found in production is a security breach? Or data corruption? Uh, oh, not so good.
Now we do need to think about our test strategy and what test automation we need to have in order to succeed. This is going to be something far smaller than the bucket that takes 6 weeks manually. It’s going to be focused on the traditionally hardest areas to automate – the ones you can’t recover from. And there might be some fun ways to do this automation, like attempting to hack your app. Then again, it might be hard work that takes a big(ger) investment in automation. But consider the pay-off. These should be tests that find problems all the time, not stand-by functional tests that are not adding a lot of value. It may be that these tests are developed outside of the test frameworks for new code as those relate directly to application capability, which are often recoverable failures. Without question, though, you are automating tests that are important to the business rather than bringing forward old tests “just because.”
Now about that assumption – the team is delivering in small batches. You might have noticed some other assumptions (where is that developer committing that test to?). There are definitely some investments needed to be ready to roll forward. I have some thoughts about how to get your team there, hopefully avoiding some common pitfalls:
In addition, if you are lucky enough to be working on something where the application architecture is new or being revised, consider how you would automate your deployment and what is going to be the strategy for automating the tests when designing the architecture. For example, having an API layer that allows testing user flow through the application will save hundreds of hours of debugging false failures in test automation at the GUI layer.
If you’ve read this far, I hope you’re thinking about how you might apply these ideas to your project team or organization. Once you start thinking that not everything has to be done the old way, it can be freeing. Like the advice I got once from an Agile coach. I had about 500 emails when I got back from vacation. He said “Delete them all. If it’s important it will come up again.” So, I say delete those old regression tests and minimize your focus to the really bad-for-business problems (can’t actually wait for those to just come up again). You might find it liberating.
Once again - thanks to our "artist in residence" Darrel Rader for the illustration!