June 15, 2016 | Written by: Ville-Veikko Helppi
Categorized: How-tos | Mobile
Share this post:
Mobile app testing can be time-consuming, error-prone and expensive – if not done properly. Furthermore, mobile app testing on real devices is a critical task to ensure end-users of your mobile app have a positive user experience. It is absolutely necessity to make sure that the user experience is awesome, and that starts from the very first try.
Frankly, if mobile app isn’t thoroughly tested and be ready for a myriad of different device variants, the results can be disastrous. Users will abandon apps that do not work, crash or behave strange on their devices. Also, end-users don’t want to be treated like testing monkeys. When they find an issue with an app, you might never hear a word from them and they won’t come back – and uninstall of your app leads to lost revenue/usage.
Testdroid Cloud: Mobile app tests on physical devices
However, mobile app developers can prepare and make sure their app works bullet-proof on variety of today’s diverse device sets. Using real mobile devices with IBM Bluemix is nowadays easy, straightforward and provides the necessary tools to build apps that work well across variety of different users and their devices.
Mobile development is meant to be agile, and a lot of great tools and practices have been developed for that agility. However, breaking an ideal development flow with manual testing means that the automation flow is also broken – and it won’t be agile anymore. Mobile test automation has a significant value to offer and has shown tremendous growth in use among app, game and web developers, speeding up their doings and yielding robust and better results.
We’ll take a look at this blog how to get started with mobile application testing and how to use real mobile devices in IBM Bluemix integration with Testdroid Cloud service.
Getting Familiar with the Testdroid Cloud UI
First, log in to IBM Bluemix developer console to get all documents and authorized access to the Cloud service. Under Services check Mobile and you’ll see bunch of great services provided with IBM Bluemix. Click Testdroid Cloud and follow instructions to create an account, space and other necessary pieces.
Select a plan and create a space for your testing needs.
The next thing you see is the Cloud service dashboard:
Dashboard links back the landing page where summary of projects, test runs as well as update of recent changes in Testdroid Cloud will be shown.
- Projects – project management view where all user’s projects can be managed.
- Reports – reports view where users can create reports of projects and test runs.
- Device Groups – devices view where users can create their own device groups.
- Manual Testing – Testdroid Cloud’s Manual Testing feature wherein user can manually access any Testdroid Cloud device.
Step #1: Create a Project for Mobile App Testing
First, simply click Projects on the top menu. Give a name for your project, select a project type and click ‘+ Add’ icon:
Now, depending on which type of test automation and framework you are planning to use, Testdroid Cloud provides support for all the industry standard ones out-of-the-box. The most common frameworks used for mobile test automation are Appium, Calabash, Espresso, UI Automator, among the others.
Step #2: Upload Your Application and Tests
Now, you have successfully created a project and ready to start your first test run. Click a ‘+’ button on top right corner of the widget and the test run creation widget will guide you through all required steps for successful test run creation:
The test run launch wizard will start and the following steps will help you to start a test run:
- Upload Application – Click Choose File button to locate .APK or .IPA file from your local hard disk. For further configuration, click Next (either on bottom or top of the view).
- Upload Tests or Select ‘App Crawler’ – In this step, you can select what test type is in use. Some test automation files are packages as .APKs, .jar, .zip or some other formats. This depends on which test automation framework are you using.In addition, Testdroid Cloud provides a feature called App Crawler that automatically crawls through the app and tests its functionality by exercising UI components (e.g. button clicks, opening menus, changing views). App Crawler automatically handles the screenshot taking, log writing and generally it keeps the record of application’s status.When user selects ‘File’ instead of App Crawler the view asks user to locate test script files for the app.When proper test method has been selected, click Next (either on bottom or arrow on top).
- Select Devices You Want to Use. – When selecting devices for your test run, all cloud devices are in use and depending on which plan you selected in IBM Bluemix. This view shows all created device groups. User can also create a new device group by clicking the left-top icon with a ‘+’ button. Device groups are either for Android or iOS. By default, Free device group is selected.
NOTE! Device groups can include only Android or iOS devices.
- Advanced Configurations – The last view of test run creation wizard allows you to specify certain settings for your test runs. For example, user can specify test run name, tags, scheduling of the test run (if tests is desired to be executed simultaneously on all devices, one device at a time or on a first available device for test run).User can select the language for devices. There is great variety of different languages available and all devices will be configured with selected language for the test run.User can input login credentials for the application. If app requires user authentication at the beginning, login and password information can be included here.User can specify how test cases are used – e.g. from specific package or class and then definitely the name in given field.User can also specify hooks for finished tests by giving an URL. If this option is used, the POST call will be performed when test run is finished.
Start the test run by clicking Start button. You’ll now be redirected to Test Run view.
Step #3: Analyze Results from Device Runs
You’ll be redirected to results page where devices selected for a test run are shown, with their running status (Waiting, Running or Successful/Failed). Typical test run takes 2-5 minutes but it directed depends on what you have included in your tests. Some thorough test cases can take even 30 minutes to complete and in that kind of case application is thoroughly tested.
The test run view looks as follows:
You can click any of finished test runs to get all details of that specific test run. For example, test run data includes screenshots, full logs, performance statistics of CPU consumption and memory use, plus other highly useful data about the test run and how devices have behaved during the run.
First, the device run details include all test steps which are important to understand which test cases and aspects were tested during the run:
Second important and very useful way to see instantly if something went wrong with the test run are screenshots that are captured when something happens with the test run. The screenshots can be also compared against different device runs to instantly spot issues with scaling resolutions, changing landscape orientation and many other UI aspects that can work differently on devices, OS versions, OEM customized version etc.
Performance of the application under test can be seen from the following performance graph. This provides fine-grained details of CPU consumption and memory allocations/deallocations:
Logs are the DNA of application behavior and provide all necessary information for developers to fix whatever happens or goes wrong with the app. Logs can be also sorted out based on warning, errors or just regular debugging data thrown into log file:
Finally, all test assets produced during the test run are reserved in Logs section on the bottom of the device run view:
Bonus: Remote Manual Access to Devices
Testdroid Cloud Manual Testing feature provides users a way to take manual access to cloud devices. By clicking Manual Testing in the main menu, user will be directed to the Manual Testing setup view.
Users can upload their applications by clicking Upload new app on top of the widget and then select application for a manual session. It is also possible to simply start using a device by clicking on Start without and app.
Step #1: Select application for the session
The Manual Testing session gets started by clicking the application or by starting without. More applications can be uploaded and all are shown in this view:
Step #2: Select a device for the session
User can pick up the device and use filters on the left-hand side to quickly browse desired device. NOTE! If certain device is not available (used by some other user) it will be shown with grayed lock picture on top of it:
Step #3: All Done – You’re Using Real Device Remotely!
The Manual Testing session will be opened with the selected application installed on device:
During the session, users can use external buttons to do the following adjustments for the device, session and tests:
- Power – Switch the device off / end session.
- Volume – speaker volume can be turned up or down.
- Rotate – the device under session can be switched to portrait and landscape mode.
- Screenshot – Take a screenshot of device session. The screenshot will be shown in Taken screenshots widget next to the Manual Testing widget.
Users can also change the file (application), device or restart the session with current device using buttons on top of the widget.
Happy Testing Folks!