The economics of mobile application testing
The economics of mobile application testing
The year 2011 seems to be when a lot of companies are motivated to extend their business applications to support mobile devices. I get asked about how IBM Rational supports mobile application development at least once a day (usually more often than that!). As new smart phones and tablets are projected to become the "client of choice" for internet access ( ZDNet: As Microsoft's monopoly crumbles, its mobile future is crucial), business application support for them will only intensify in the future. In some ways, mobile application software development is similar to other kinds of software development. But there are some different challenges associated with developing mobile software too.
One of the areas where mobile application development poses a huge challenge is testing. Testing for mobile applications represents a step-jump in complexity and cost over more traditional applications. With the hundreds of different mobile devices in the market, there are a wider range of endpoint types on which the application must execute. And even if you limit the range of devices that you plan to support with the application, the list will still typically contain dozens of devices manufactured by different vendors and at different releases. Compare that list to the one that a typical desktop application needs to support.
And there are more variables thrown into the equation for mobile testing that aren't relevant for other kinds of software. The same model of device may function subtly different when connected to a different carrier network. And the quality of the network connection can have profound impact on the behavior of a mobile application. Even the movement of the mobile device itself may be an important factor in the behavior of the application (some applications specifically exploit device movement). How can a mobile application project address this quality control challenge?
You could rely on mobile device simulators and emulators (Mobile Emulators & Simulators: The Ultimate Guide) for your testing. Using this approach, a software program running on a desktop workstation takes the place of a real physical device. I think that the use of simulators and emulators for mobile application testing can be valuable for developer unit tests and other "early stage" testing of a mobile app. However, the fact remains that these programs are not the actual physical devices. Some of the device simulators and emulators are excellent, but some are not that good at replicating the real device. I don't know about you, but I would not put an important business application into production that was solely tested on a simulator / emulator without ever being tested on a real physical device.
You could plan on manual testing of the mobile app (Mob
There are mobile app testing solutions that rely on real physical devices tethered to a computer and running an agent program that the tests interact with and drive (Mac News Article about Eggplant). This approach has the advantage of using real physical devices for test with the added benefit of automation. You still have to buy all of the devices to be tested. And you are responsible for setting them up, tethering them to a computer and installing the test agent on them.
Another approach to address the mobile app testing challenge is to leverage what I call a "device cloud". The compute resources typically exposed by internet "cloud" services today are regular general purpose computers - virtual machines running the Linux or Windows operating system (NIST Definition of Cloud Computing). But a cloud could expose compute resources that are actually physical mobile devices. Instead of "renting" a Linux virtual machine for a few hours or days of testing, you could rent a specific model and release of a mobile device.
Does such a "device cloud" for mobile app testing exist? Yes, there is in fact just such a solution available today. IBM Rational Quality Manager (RQM) offers an adaptor for managing the execution of mobile app testcases on real mobile devices leveraging a device cloud controlled by the DeviceAnywhere system. Using this solution, you can write an automated script that executes the same test for your application on many different mobile devices. So you don't need a different script for each device. And the system can automatically capture screen shots of the mobile device if a testcase fails. The test failure data, including screen capture, is attached to a defect created by RQM which routes it to the application developer.
Now that I know about the "device cloud" automated testing solution, I can't imagine a more economical way to get the test coverage needed for mobile apps. I'll still use simulators for unit tests and to quickly check the app during the code-deploy-debug cycle of app development. But for fully automated functional testing and pre-production regression tests, I am really enthusiastic about the cost savings of the new device cloud test solution.
What do you think? Could a "device cloud" be in your future for mobile application testing? What other solutions for testing mobile apps have you seen?
About the author:
Leigh Williamson is an IBM Distinguished Engineer who has been working in the Austin, Texas lab since 1988, contributing to IBM's major software projects including OS/2, DB2, AIX, OpenDoc, Java, Component Broker, and WebSphere Application Server. His current role is as a member of the IBM Rational Software Chief Technology Officer team, influencing the strategic direction for products in the Rational brand and focusing on projects related to mobile device application development. Leigh holds a BS degree in Computer Science from NovaUniversity and a Masters degree in Computer Engineering from University of Texas at Austin. Leigh leads the discussions on the Rational Mobile Frontier blog, providing missives on the evolution of mobile software.