Continuous Testing and the Pipeline
monica914 060000WAY8 Visits (2989)
I seem to be talking and thinking about continuous testing, um, continuously lately. I had a little epiphany last week when I concluded that most of the pipeline is for testing. The pipeline starts with a continuous integration build where unit testing happens and a package, installable entity is produced. The pipeline ends when that packaged entity is deployed into production. Every step in between is for testing. Different kinds of testing, but they are all testing. So at a minimum, when there’s only one test stage, 30% of the pipeline is about testing. In practice, it is way more like 50-70% testing. Of course, looking again at Jez Humble & David Farley’s book “Continuous Delivery,” I realize the epiphany has been staring me in the face for some time now.
But it did cause me to reflect on each time I’ve hit the test automation wall. Anyone who has ever written a large body of test automation gets tired pretty soon of manually setting up the environment for running those automated tests. Sounds easy right? You can run the test automation tool from the command-line, can’t you? So starts the trek into achieving “unattended” or “headless” test execution. And the typical path to that includes what I call some hard-wiring. Like making sure the test automation tool is available by having a vm image that gets reverted to snapshot before each run. Maybe automating the test database by doing a restore from backup. Machines that are pre-configured and waiting around when no tests are running. Figuring out how to dance around authentication issues without creating a security hole (how do I pass the login password around???). Struggling to automate retrieving the right version of the test scripts. Stringing together a bunch of commandlines with runtime parameters and running those from a build tool with ANT, or a java process. First time I did it, I used PERL. Usually, to get this running with chewing gum and chicken wire took a lot of effort and a lot of time and sometimes a team of people.
Now try to scale that up to a pipeline. Where there are expectations that the test environment be a lot more like production. And where there are multiple test frameworks that run different tests for different reasons. It’s also no longer acceptable to have systems sitting idle waiting for a job that comes once a day. Okay, okay, that was never really acceptable, but it’s definitely harder to get away with now. And the pipeline is expected to be reliable – no more chewing gum! So these are the challenges I’m thinking about now. Application Release Automation tools like UrbanCode Deploy and access to Cloud infrastructure allow me to experiment with creating reusable automation to deploy the other part of the test environment… the parts needed to support automated test execution. That long list of stuff that my test automation relies on (why exactly did I need to use STAF?). My goal – to be able to run all my tests on multiple environments in parallel. No choosing between tests as I’d suggested in my earlier blog. Just run ‘em all. Why not? Now that seems to me, to be Continuous Testing. And since it’s most of the pipeline, worth investing the time to get it right.