The Right Tool for the Job - Building an Automation Testing Matrix
JoshGalde 270005VQUH Visits (2824)
For Desktop-based testing it’s a no-brainer: Use object-based scripting to maximize reuse across platforms/browsers. In today’s mobile world it really isn’t that simple. There are many different platforms, OS versions, form factors and carr
In order to tackle this problem, an Automation Engineer cannot simply look at it from a “one size fits all” perspective to create a set of objects and re-use them across all combinations of platforms. For example, there are fundamental differences in how an app behaves on iOS and Android, even with something as basic as a “back button” has its quirks.
Although these fundamental differences can be grouped together as a step or action, they are unique enough to not be able to simply share an object between the two OS’s.In some cases with mobile testing, you may be able to get to the object-level, however this usually requires that you instrument your app, or test on an emulator. While this fulfills a piece of your testing matrix, you will probably need to seek a couple tools to get this done across all platforms. In other cases, the content you are testing might be HTML-based and you can test by WebKit profiling. Again, part of your testing matrix is fulfilled, however you aren’t quite there.This may be enough to satisfy a short-term goal, but at some point you need to be testing on real mobile devices.
In order to truly automate on mobile, your mobile testing “Utility Belt” needs to be designed in such a way that allows for testing by object when possible, element when possible, and also be able to quickly fall back on text or image verification in order to satisfy all areas of your testing matrix and assure the highest quality of your mobile product. Having the flexibility to be able to choose how to get the testing done is paramount since as an Automation Engineer,you very rarely have a say in how a particular app or mobile web site is developed. The job requires you to sometimes understand functionality without necessarily being privy to the construction, and there is always a tight timeline to achieve results. The right tool for the job is a tool that takes all of this into consideration, and provides a platform to consolidate all of these different types of testing approaches.The first step is to determine the type of app you are testing. Is it fully native, fully web, or somewhere in between?
The second step is to find which pieces or steps of your test cases are reusable between each other, and can accept parameterization to fulfill the task. For instance, automating the selection of an item or link on your main screen of your app or landing page: Maximize reuse by engineering a parameter to accept different values, and reuse it across each test case. Although you may need to individually determine what type of verification you will use to achieve this on a per platform or device level, you will save time in the long run when you write additional test cases. The third step is to then group those pieces or steps together by device screens or pages. This way, as you write the test cases you have an organizational structure that is easy to identify by where you are within the app or site and where you need to navigate to next.Following these steps will provide a structure that can be grown to accommodate new features within an app or new sections within a mobile web site. As mobile devices become easier to automate against, this structure can easily adapt to emerging technologies that allow for greater reuse across platforms.
Note: This article was recently published in the April, 2013 issue of Automate Software Quality Magazine.