Build mobile apps that work

Five tips from an expert panel

A panel of experts from IBM® MobileFirst and partner organizations uTest and Perfecto Mobile recently came together to discuss the current state-of-the-art in mobile application testing and quality assurance. This article distills what they had to say about how the mobile-first application lifecycle is redefining application quality and what developers and firms can do to manage the complexities of mobile development, including test integration and automation in cloud-backed mobile application environments.


Yan Tina Zhuo (, Product Manager, IBM

Photo of Yan (Tina) ZhuoYan (Tina) Zhuo is a product manager for mobile from IBM Rational, responsible for defining mobile solutions that enable customers to transform their businesses through mobile. Over the last 10 years with IBM, Tina has always enjoyed working with customers in different roles from presales to service, from development to product management. Tina holds a Master of Arts in Early Childhood Education from Boston College and a Master of Science in Information Systems from Northeastern University. She believes that good software engineers are as curious as young children.

Leigh Williamson (, IBM Distinguished Engineer, IBM

Photo of Leigh WilliamsonLeigh Williamson is an IBM Distinguished Engineer who has worked in the Austin, Texas lab since 1988, contributing to IBM’s major software projects that include OS/2, DB2, AIX, Java, WebSphere Application Server, and the IBM Rational portfolio of solutions. His current role is as a member of the Chief Technology Officer team, influencing the strategic direction for products that address the needs of software development teams. Leigh’s primary focus is on tools and best practices for mobile application development. His blog on mobile development topics is You can follow him on twitter @leighawilli. Leigh holds a Bachelor of Science degree in Computer Science from Nova University and a Masters degree in Computer Engineering from University of Texas at Austin.

Dennis Schultz (, Solution Architect, IBM

Photo of Dennis SchultzDennis Schultz is a Solution Architect, tester, and father of four. As a child, he had an insatiable curiosity of how things worked and a desire to take things apart. The all-too-common outcome of taking things apart was that he could not put them back together again. The desire to make amends for these transgressions eventually lead to his current position as a Solution Architect where he helps IBM clients to assemble products and tools into integrated solutions that resolve their challenges. Dennis holds a bachelor's degree in engineering. He has held various roles in IBM and currently resides in the Unleash the Labs organization.

10 April 2014

Also available in Russian Japanese

Mobile-first is the new watchword for both large and small organizations, which are adopting mobile delivery as the surest path to customer engagement. As mobile developers, architects, and strategists, we are challenged to come up with creative approaches that sometimes require big risks. We are doing everything that we can to anticipate and leverage the emergent trends of mobile. In the process, we are transforming the traditional application development lifecycle and the assumptions and priorities that drive it. None of this is comfortable, but it is certainly exciting. Fortunately, the mobile ecosystem is also expanding, with an array of new tools that developers can use to meet the demand for integrated, automated, and collaborative mobile solutions.

This article summarizes key points from our panel discussion for the August 2013 IBM developerWorks Mobile Frontiers webcast, "Mobile testing: Proven techniques to use when apps just have to work" (see Resources). As mobile specialists who work in IBM MobileFirst and partner organizations, we discussed technologies and approaches that help our clients master the complexity of mobile test environment; how teams can leverage techniques like test integration, test automation, and testing in the wild to help deliver high-quality mobile apps; and the growing value of the mobile-device cloud.

Mobile apps redefine quality

More than desktop and web applications, mobile apps are measured against a user-centric view of application quality. From this perspective, high quality no longer means just the absence of bugs. To evaluate the quality of a mobile app, you must first adopt a multi-dimensional lens. Both front-facing elements like usability, performance, and content, and back-end issues like functional quality and security need to be factored in.

IBM Mobile Quality Assurance

To support organizations that are adopting a mobile-first strategy, IBM has announced an open beta of IBM Mobile Quality Assurance (IBM MQA). IBM Mobile Quality Assurance acknowledges that user feedback and quality metrics are needed at every stage of development to improve user experience and mobile app quality. Engaging testers across the globe, it provides line-of-business professionals and development teams with insightful and streamlined quality feedback and metrics, enabling them to prioritize and act to support a dynamic mobile application strategy.

It is crucial that you carefully assess application quality because an app that fails to meet users' expectations can be detrimental to your entire brand. Mobile app stores and social media give customers a large soapbox for bad reviews, and switching to a competitor's app is easy and affordable.

Developers must always balance the importance of user experience against multiple other quality factors, but the user’s experience really is king in mobile. For better or worse, we are weighing the trade-offs of application quality differently than we have in the past, which inevitably shifts both a developer's and a business manager's perception of what "high quality" means.

Requirements gathering still matters

Mobile-first is at a tipping point, with more enterprise customers who realize that the mobile app has become the primary way that customers interface with their organization. It’s no longer just a sideline or nice to have: in some cases the mobile app is driving the business and might serve as the face of the company.

As mobile apps overtake web apps, developers must rethink requirements gathering and requirements management for mobile development. The abundance of hardware and app distribution options makes mobile an exceptionally competitive field, so the edge of possibility is constantly being pushed. Mobile developers need to be unusually responsive to meet user demands for new functionality and features. That responsiveness is becoming the new norm.

In short, an effective requirements gathering process for mobile must involve the user base. You must know what prospective users want to entice them away from your competitors in the mobile app marketplace.

Mobile app testing is unique

Fragmentation creates new creative opportunities and choice for both developers and users. There is a downside to that choice during testing, however. Currently, on the market there are 12,000 unique Android devices alone. That level of fragmentation must be factored into the test matrix for a new mobile app.

Although they're smaller and more lightweight, mobile applications often are expected to have more features or more advanced functionality than their web equivalents. An airline application on the web requires check-in functionality, but its mobile equivalent needs to be able to display an e-ticket, track baggage, and more. Each of these features requires an extra test step.

While we're wrestling with the question of how to test richly featured mobile apps across multiple platforms, we're also facing lower production budgets for mobile apps than for web and desktop applications. The engineering and testing budget for a mobile app tends to be relatively slim.

The pace of mobile development is also intense. Deliveries often don’t follow the cadence that traditional developers are accustomed to. Mobile development requires a more agile approach, especially when it comes to testing and quality. You can't test a mobile app as exhaustively as you would a web or desktop app because the number of vectors to be covered is much greater.

Rethink your testing strategy

Mobile operations that use traditional techniques to determine test requirements will struggle. To be successful, adopters need to look at mobile as something different and rethink their approach completely. It's important to understand the trade-offs between time-to-market goals and the changing quality tolerances of your consumer base.

A best practice mobile app testing strategy strikes a balance between automated tests and interactive human-based testing.


Decide who is responsible for testing mobile apps in your organization. Is it the development team, a center of excellence, or some combination of the two? If you are using both internal and external testers, clearly define where the handoff occurs. Methodology should also be explicitly defined. Assessing available resources and the scope of testing helps guide these decisions.

The FURPS (functionality, usability, reliability, performance, supportability) model is a useful measure for testing mobile apps (see Resources). Extend it to include security, privacy, and content because all three play a large role in the user's perception of quality. Business managers should also be looking at the target market, the intention of the app, and the app's risk profile. Explicit and implicit quality feedback from developers, testers, and users should be continuously incorporated.

Balancing quality and time-to-market

Companies like Twitter, Facebook, and Google base their business model on getting new features to market. They give away free applications as a marketing strategy. Each of these companies learned to accept inconsequential bugs to launch a new feature while it is hot. A bug that does not impact 100,000 people might or might not get fixed. These changes have transformed the software development lifecycle over the past five to ten years.

Prioritizing time-to-market doesn't fit for every company, however. Large brand companies or operations that depend on trust cannot afford functional bugs that would lead consumers to question the safety of their data. For these companies, it is imperative to avoid security or even minor functionality flaws that might lead to poor user experience and damaged trust.

The challenge for financial providers is thus to:

  • Engage customers with mobile apps
  • Add new and differentiating features and capabilities
  • Maintain exceptionally high quality and security in mobile applications

Keys to successful mobile testing

For effective mobile testing throughout the development process, plan to integrate and automate your testing and test in real-world scenarios and environments.

Test integration

One thing that hasn't changed with mobile is the need to test early. Early testing can be a bigger challenge on a mobile app, however, because the app can depend on multi-tiered distributed systems, mainframes, ERP systems, and packaged application systems. All of these dependencies are typically developed and maintained by separate teams that are operating on different schedules with different budgets.

Integrating all of the platforms that support and drive a mobile app shouldn't be an obstacle to early testing. Service virtualization that uses solutions like IBM Rational Test Workbench and Rational Test Virtualization Server (see Resources) offers a viable workaround for some mobile integration and testing scenarios. Creating test stubs for the services that a mobile app will eventually depend on lets developers do integration testing while they move the app forward on schedule.

Developers can use Rational Test Workbench to automate the exercising of the UI from the front of the application and verify performance and functionality. IBM Rational Test Workbench is used for integration testing, virtualization of back-end services, and UI test automation. Virtualization and UI test automation are also used to test for unexpected environmental conditions.

Test automation

Automation is or will become essential to most of our clients. Testing teams that are able to keep up with manual testing will eventually become overwhelmed. As organizations ramp up and see the central role that mobile plays, now and in the future, the pace of the development and release is going to accelerate. Automating testing will be mandatory.

For teams who develop an automated mobile testing strategy three questions are key:

  1. How will you create efficient test automation that runs on multiple devices?
  2. How will you create efficient automation that uses a minimal number of changes between versions?
  3. How will you create truly unattended test automation?

How a team responds to these questions determines the level of efficiency that can be derived from testing and the coverage reach that the project can hope to achieve.

Mobile-first test automation

Test automation is essential to a mobile-first strategy. Mobile teams work naturally with agile methodologies like test-driven design. So we expect to see more firms that automate testing up front to build a quality product from the start. As more agile teams enter mobile development, the lines between tester and developer are blurring. App quality is not just the responsibility of a designated tester anymore; it's the responsibility of the entire team.

Testing in the wild

In-the-wild testing refers to hiring professional testers to test your application in real-world scenarios and environments. The testing is done on real devices to ensure that applications work as they are intended to under various conditions. Testing in the wild isn't testing in the idealized conditions of a test lab. Instead, real users who represent your app's target market try out your application and determine whether it meets the standard of quality that you are aiming for.

In-the-wild testing complements testing in the lab. It is especially useful for testing your application's interface with localized services and with actual users in your target market. Ideally, tests in the wild integrate into application lifecycle management processes and align with in-house bug tracking systems.

In conclusion

Mobile app testing and quality management will remain a challenge for organizations that need to continuously deliver high-quality mobile apps. Three more factors on the horizon are likely to impact mobile testing:

  • Incorporating test automation into the DevOps lifecycle as part of the release-and-delivery pipeline.
  • Reducing fragmentation by improving standardization for security, development, and other factors. (This is familiar to firms that remember the early days of testing software for the web.)
  • Transforming what is being done for mobile for emerging Internet-of-Things environments like cars, TVs, and refrigerators.

Mobile app testing and quality assurance are new IT industry frontiers and remain a constant challenge as organizations strive to deliver the highest-quality mobile apps possible; ones that engage and delight users while optimizing access to back-end systems and providing the security users expect.



Get products and technologies

  • IBM Rational Test Workbench: A comprehensive test automation solution for mobile applications, regression testing, integration technologies, and performance and scalability testing.
  • Rational Test Virtualization Server: Enables the deployment of virtualized services, software, and applications for simplified, efficient testing.


  • Get involved in the My developerWorks community: Connect with other developerWorks users while you explore the developer-driven blogs, forums, groups, and wikis.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Mobile development on developerWorks

Zone=Mobile development, Rational
ArticleTitle=Build mobile apps that work