Mobile-first is the new watchword for both large and small organizations, which are adopting mobile delivery as the surest path to customer engagement. As mobile developers, architects, and strategists, we are challenged to come up with creative approaches that sometimes require big risks. We are doing everything that we can to anticipate and leverage the emergent trends of mobile. In the process, we are transforming the traditional application development lifecycle and the assumptions and priorities that drive it. None of this is comfortable, but it is certainly exciting. Fortunately, the mobile ecosystem is also expanding, with an array of new tools that developers can use to meet the demand for integrated, automated, and collaborative mobile solutions.
This article summarizes key points from our panel discussion for the August 2013 IBM developerWorks Mobile Frontiers webcast, "Mobile testing: Proven techniques to use when apps just have to work" (see Resources). As mobile specialists who work in IBM MobileFirst and partner organizations, we discussed technologies and approaches that help our clients master the complexity of mobile test environment; how teams can leverage techniques like test integration, test automation, and testing in the wild to help deliver high-quality mobile apps; and the growing value of the mobile-device cloud.
Mobile apps redefine quality
More than desktop and web applications, mobile apps are measured against a user-centric view of application quality. From this perspective, high quality no longer means just the absence of bugs. To evaluate the quality of a mobile app, you must first adopt a multi-dimensional lens. Both front-facing elements like usability, performance, and content, and back-end issues like functional quality and security need to be factored in.
It is crucial that you carefully assess application quality because an app that fails to meet users' expectations can be detrimental to your entire brand. Mobile app stores and social media give customers a large soapbox for bad reviews, and switching to a competitor's app is easy and affordable.
Developers must always balance the importance of user experience against multiple other quality factors, but the user’s experience really is king in mobile. For better or worse, we are weighing the trade-offs of application quality differently than we have in the past, which inevitably shifts both a developer's and a business manager's perception of what "high quality" means.
Requirements gathering still matters
Mobile-first is at a tipping point, with more enterprise customers who realize that the mobile app has become the primary way that customers interface with their organization. It’s no longer just a sideline or nice to have: in some cases the mobile app is driving the business and might serve as the face of the company.
As mobile apps overtake web apps, developers must rethink requirements gathering and requirements management for mobile development. The abundance of hardware and app distribution options makes mobile an exceptionally competitive field, so the edge of possibility is constantly being pushed. Mobile developers need to be unusually responsive to meet user demands for new functionality and features. That responsiveness is becoming the new norm.
In short, an effective requirements gathering process for mobile must involve the user base. You must know what prospective users want to entice them away from your competitors in the mobile app marketplace.
Mobile app testing is unique
Fragmentation creates new creative opportunities and choice for both developers and users. There is a downside to that choice during testing, however. Currently, on the market there are 12,000 unique Android devices alone. That level of fragmentation must be factored into the test matrix for a new mobile app.
Although they're smaller and more lightweight, mobile applications often are expected to have more features or more advanced functionality than their web equivalents. An airline application on the web requires check-in functionality, but its mobile equivalent needs to be able to display an e-ticket, track baggage, and more. Each of these features requires an extra test step.
While we're wrestling with the question of how to test richly featured mobile apps across multiple platforms, we're also facing lower production budgets for mobile apps than for web and desktop applications. The engineering and testing budget for a mobile app tends to be relatively slim.
The pace of mobile development is also intense. Deliveries often don’t follow the cadence that traditional developers are accustomed to. Mobile development requires a more agile approach, especially when it comes to testing and quality. You can't test a mobile app as exhaustively as you would a web or desktop app because the number of vectors to be covered is much greater.
Rethink your testing strategy
Mobile operations that use traditional techniques to determine test requirements will struggle. To be successful, adopters need to look at mobile as something different and rethink their approach completely. It's important to understand the trade-offs between time-to-market goals and the changing quality tolerances of your consumer base.
“A best practice mobile app testing strategy strikes a balance between automated tests and interactive human-based testing.”
Decide who is responsible for testing mobile apps in your organization. Is it the development team, a center of excellence, or some combination of the two? If you are using both internal and external testers, clearly define where the handoff occurs. Methodology should also be explicitly defined. Assessing available resources and the scope of testing helps guide these decisions.
The FURPS (functionality, usability, reliability, performance, supportability) model is a useful measure for testing mobile apps (see Resources). Extend it to include security, privacy, and content because all three play a large role in the user's perception of quality. Business managers should also be looking at the target market, the intention of the app, and the app's risk profile. Explicit and implicit quality feedback from developers, testers, and users should be continuously incorporated.
Balancing quality and time-to-market
Companies like Twitter, Facebook, and Google base their business model on getting new features to market. They give away free applications as a marketing strategy. Each of these companies learned to accept inconsequential bugs to launch a new feature while it is hot. A bug that does not impact 100,000 people might or might not get fixed. These changes have transformed the software development lifecycle over the past five to ten years.
Prioritizing time-to-market doesn't fit for every company, however. Large brand companies or operations that depend on trust cannot afford functional bugs that would lead consumers to question the safety of their data. For these companies, it is imperative to avoid security or even minor functionality flaws that might lead to poor user experience and damaged trust.
The challenge for financial providers is thus to:
- Engage customers with mobile apps
- Add new and differentiating features and capabilities
- Maintain exceptionally high quality and security in mobile applications
Keys to successful mobile testing
For effective mobile testing throughout the development process, plan to integrate and automate your testing and test in real-world scenarios and environments.
One thing that hasn't changed with mobile is the need to test early. Early testing can be a bigger challenge on a mobile app, however, because the app can depend on multi-tiered distributed systems, mainframes, ERP systems, and packaged application systems. All of these dependencies are typically developed and maintained by separate teams that are operating on different schedules with different budgets.
Integrating all of the platforms that support and drive a mobile app shouldn't be an obstacle to early testing. Service virtualization that uses solutions like IBM Rational Test Workbench and Rational Test Virtualization Server (see Resources) offers a viable workaround for some mobile integration and testing scenarios. Creating test stubs for the services that a mobile app will eventually depend on lets developers do integration testing while they move the app forward on schedule.
Developers can use Rational Test Workbench to automate the exercising of the UI from the front of the application and verify performance and functionality. IBM Rational Test Workbench is used for integration testing, virtualization of back-end services, and UI test automation. Virtualization and UI test automation are also used to test for unexpected environmental conditions.
Automation is or will become essential to most of our clients. Testing teams that are able to keep up with manual testing will eventually become overwhelmed. As organizations ramp up and see the central role that mobile plays, now and in the future, the pace of the development and release is going to accelerate. Automating testing will be mandatory.
For teams who develop an automated mobile testing strategy three questions are key:
- How will you create efficient test automation that runs on multiple devices?
- How will you create efficient automation that uses a minimal number of changes between versions?
- How will you create truly unattended test automation?
How a team responds to these questions determines the level of efficiency that can be derived from testing and the coverage reach that the project can hope to achieve.
Testing in the wild
In-the-wild testing refers to hiring professional testers to test your application in real-world scenarios and environments. The testing is done on real devices to ensure that applications work as they are intended to under various conditions. Testing in the wild isn't testing in the idealized conditions of a test lab. Instead, real users who represent your app's target market try out your application and determine whether it meets the standard of quality that you are aiming for.
In-the-wild testing complements testing in the lab. It is especially useful for testing your application's interface with localized services and with actual users in your target market. Ideally, tests in the wild integrate into application lifecycle management processes and align with in-house bug tracking systems.
Mobile app testing and quality management will remain a challenge for organizations that need to continuously deliver high-quality mobile apps. Three more factors on the horizon are likely to impact mobile testing:
- Incorporating test automation into the DevOps lifecycle as part of the release-and-delivery pipeline.
- Reducing fragmentation by improving standardization for security, development, and other factors. (This is familiar to firms that remember the early days of testing software for the web.)
- Transforming what is being done for mobile for emerging Internet-of-Things environments like cars, TVs, and refrigerators.
Mobile app testing and quality assurance are new IT industry frontiers and remain a constant challenge as organizations strive to deliver the highest-quality mobile apps possible; ones that engage and delight users while optimizing access to back-end systems and providing the security users expect.
- Webcast Replay: "Mobile Testing: Proven techniques to use when apps just have to work" (The Mobile Frontier, August 2013): Join the Mobile Global Rational User Community to see the replay or attend and view replays for more webcasts.
- "FAQ: What is IBM Mobile Quality Assurance?" (IBM developerWorks): Find out how IBM Mobile Quality Assurance helps developers instrument mobile apps to capture tester and live-user experiences.
- "How to test mobile applications with Rational Test Virtualization Server" (Manjula Sogi, IBM developerWorks, January 2014): Get a step-by-step introduction to end-to-end mobile application testing with Rational Test Virtualization Server, Rational Test Workbench, and Rational Quality Manager.
- "Add IBM Mobile Quality Assurance to your mobile quality regimen" (Leigh Williamson, IBM developerWorks, March 2014): Start today with IBM Mobile Quality Assurance, a cloud-based set of tools that assists developers with manual interaction testing for mobile apps.
- "Mobile testing with IBM Rational Test Workbench" (Pragati Maheshwari, IBM developerWorks, August 2013): A hands-on introduction to using IBM Rational Test Workbench to test an Android mobile app.
- "From manual to continuous automated deployment of mobile Worklight applications" (Joel Cayne, IBM developerWorks, February 2014): This tutorial shows you how to use IBM UrbanCode Deploy to define a DevOps deployment solution for IBM Worklight mobile artifacts.
- "Why testing in-the-wild is inevitable" (Jaime Saine, uTest Software Development Blog, March 2013): In-house testing still has its place, but market and environmental factors make testing in the wild an essential piece of the mobile application lifecycle.
- Visit the IBM developerWorks Mobile Frontier Blog and Mobile development zone for more articles like this one.
Get products and technologies
- IBM Rational Test Workbench: A comprehensive test automation solution for mobile applications, regression testing, integration technologies, and performance and scalability testing.
- Rational Test Virtualization Server: Enables the deployment of virtualized services, software, and applications for simplified, efficient testing.
- Get involved in the My developerWorks community: Connect with other developerWorks users while you explore the developer-driven blogs, forums, groups, and wikis.