Several techniques for testing and validating mobile apps are in use today. The IBM point of view is that an effective mobile development project will employ all of the applicable techniques, because each technique has its strengths and weaknesses. The various available techniques are not mutually exclusive. The most effective mobile quality strategy balances the use of all forms of mobile test execution and rolls up the results from each into a comprehensive overall mobile app "quality metric."
IBM Mobile Quality Assurance is designed to fit well as a component of a comprehensive mobile app quality strategy. Such a strategy should include running a battery of automated mobile tests against each build of the mobile app to ensure that it meets minimum quality levels before employing IBM Mobile Quality Assurance services to distribute the build to human interactive testers, so that bugs and crashes can be effortlessly captured in rich context.
When a new build of the app passes the initial battery of automated tests, the developer can upload the app binary (.apk or .ipa file) to the IBM Mobile Quality Assurance service and identify the people who should be notified about the availability of this new build. This set of mobile app evaluators/testers receives an email notifying them about the new build. And when the tester clicks a link in the notification email, the new version of the app is automatically downloaded to their mobile device and installed, ready for immediate testing. The mobile app testers can be confident that they have the correct build of the app to be tested.
In-app bug reporting and feedback
As testers perform interactive manual testing of the app and encounter a defect of any kind, they can use IBM Mobile Quality Assurance in-app bug reporting capability to submit a defect right from inside the app being tested on their mobile device.
The tester simply shakes the mobile device and the app being tested goes into "bug reporting mode." This mode suspends normal behavior of the app and allows the user to capture one or more screen images from the mobile app. The tester can annotate the screen image with lines, circles, arrows—anything that fingers can draw.
After the screen image is captured, the tester is presented with a text box to be used to describe the defect. Once the description of the problem is entered, the tester taps on the Report button, and the defect information is sent over the network to the IBM Mobile Quality Assurance service. Along with the explicit information (screen images and text description) from the tester, rich technical details of the context of the mobile app and the device on which it was running are captured and sent as well.
Some of the rich context for each defect captured includes:
- Mobile device type
- Mobile operating system and release level
- Network in use, including carrier or wireless details
- Memory available and in use on the device
- Logging output up to the point when the defect is reported
- Battery level
This detailed technical information is invaluable for helping the mobile app developers troubleshoot the defect and understand the root cause of the problem.
In addition to the in-app bug reporting capability of IBM Mobile Quality Assurance services, every application crash is captured by the service logic. Each time the application crashes, the entire context of the application and the device on which it was running are captured at the moment of the crash. This critical, "must gather" data is sent over the network to the IBM Mobile Quality Assurance services, where it is analyzed and made available to the development team.
If you use IBM DevOps services for defect and work item tracking and management, you can configure IBM Mobile Quality Assurance services to automatically open work items for each crash or bug report that comes into the service.
You can leverage the crash data capture capabilities of IBM Mobile Quality Assurance services during the initial automated battery of tests on the app, during the manual interactive phase of testing, and even after the app has been released to the app store.
Additional analytics within the service recognize crashes that occur at the same spot in the mobile app and aggregate them so that you can see how many times a crash occurs at the same location in the app logic.
Quality is in the eye of the end user
The main focus for quality assurance of mobile apps is on the pre-production phases of the development lifecycle. However, quality assessment should not end when the app gets released into production. There is still very important data about the behavior of the mobile app "in the wild" that can be obtained and used to help the developers continuously improve the app.
It isn’t practical to expect that every conceivable defect in a mobile app can be caught and fixed before it is released into production. Even after they are deployed to the app store and installed on end users' devices, the best mobile apps continue to capture context information for every crash that occurs and deliver that information back to the app development team.
Besides outright crashes, it is valuable to solicit feedback from your end users about how they perceive the mobile app. Most popular mobile apps include some kind of "in-app feedback" mechanism so that users who would not take the time and effort to write a review in the app store can at least send the development team a short message about how they view the app. And it is especially important to capture the context of such feedback at the time it is submitted, so that the development team can know if special conditions are contributing to the impression conveyed in that user's feedback.
App store review comments offer a rich source of quality assessment also. If your mobile app has only a few dozen reviews in the app store, it is easy enough to read each one and gain insight into the sentiment of your users. But once the number of reviews reaches several dozen or more, you need an analysis tool to effectively and efficiently mine the key insights from that amount of data.
IBM Mobile Quality Assurance includes an app store review analysis capability that captures all of the review text and searches for a set of special "user sentiment" keywords in each review text. Analysis for app store reviews is organized into 10 distinct "attributes" about your mobile app, such as usability, stability, performance, and elegance.
You can drill down into each of the user sentiment attributes for your mobile app and see the analysis used to produce the score for that attribute. You can even see the specific list of reviews containing comments about that attribute.
This app store user sentiment is invaluable, especially when you correlate the sentiment expressed by the reviewers with hard technical evidence in the crash reports and in-app user feedback records.
Continuous delivery and quality assessment for mobile apps
“A best practice mobile app testing strategy strikes a balance between automated tests and interactive human-based testing.”
The ideal quality cycle begins by running a battery of automated tests against the output of the continuous integration build process for your mobile app. Once this initial battery of automated tests has verified that the latest build meets minimum quality criteria, the build can be distributed to a group of testers and internal evaluators who will perform the interactive testing.
After the mobile app has passed the automated and interactive tests, it is a candidate to be released to production use and distributed to real end users (presumably via a public app store or private enterprise app store). Even after the mobile app has been released into production, you can continue to obtain quality assessment data about the application.
IBM Mobile Quality Assurance services are designed to play a part in all three phases of the mobile quality regimen, making your interactive testing as efficient as possible and capturing critical must-gather technical data about crashes and bug reports filed by your testers. And, as the reviews for your app build up in the app store, use the sentiment capabilities of IBM Mobile Quality Assurance to gain insight about how your end users perceive your mobile app.
This comprehensive mobile quality strategy will give your team faster resolution of crashes and bugs in your app, and deeper insight into the conditions in which the app runs in the real world. You’ll be heading toward the coveted 5-star rating in no time!
- IBM Mobile Frontier blog: Follow Leigh and other experts in mobile app development.
- @leighawillia: Follow Leigh on Twitter.
- IBM MobileFirst: Learn about becoming a mobile enterprise.
Get products and technologies
- IBM Mobile Quality Assurance: Try it!