How do top mobile game developers test games?

How do top mobile game developers test games?

Designing, developing and testing mobile games is a unique experience in comparison to testing mobile apps, as well as web development and testing. But still the quality, robustness and device compatibility are critical factors for the success of a mobile game.

Awesome performance, slickness with smooth game-play, and user experience (gaming experience) are the top factors that either set a mobile game on the top row – or on bottom where nobody downloads and plays it.

How do those top game developers test their mobile games? And how do they overcome the device fragmentation, as well as ensure their games are performing well on each and every device out there? We’ll take a look at this topic in this blog.

3 critical criteria for popular mobile games

There are three things that are commonly accepted as the most important reasons why mobile games make it big time – or fail.

• User ratings: When looking at today’s top mobile games (at Google Play and Apple App Store), there is one important characteristic that is common for the top games: user ratings. Competition is fierce with mobile games today. Users have very short attention spans for your game, and the decision of whether it gets downloaded or not is typically done based on user ratings. Nobody wants to download bad performing, buggy or dubious games from app markets. In addition, users are very straightforward about their passionate opinions on games and comments/feedback end up in front of hundreds of millions of other potential users.

• Performance: Good and slick performance of mobile games provide better gaming experience and gamers are more likely promote games that work well. The real performance of mobile games can only be observed and measured on real devices. A majority of mobile games are graphics-intensive and basically stress two parts of the hardware: CPU and GPU. The optimization of any mobile game across different device variants, chipsets, and form factors requires testing on those real devices.

  

• Device compatibility: Too many mobile games are still limited to high-end devices even though those could run fine on mid-range and even low-end phones, if tested properly. Scaling the graphical content, limiting certain operations or simply allowing mobile game to fully utilize hardware are probably the easiest methods to get a game running on any device. However, it still requires the use of physical, real devices, for testing.

Lately, test automation has come into play and helped game developers to quickly and easily see how their games working across all possible device variants. This produces tons of great data of how the game can be optimized for end-user devices.

If you are looking for a way to get stared with real device testing, there are services available (such as Bitbar Public Cloud, previously known as Testdroid Cloud) that provides all critical tools, support for frameworks and plenty of material how to get started with mobile app and game test automation.

Shift of Paradigm: From Waterfall to Agile

Things that are done manually are hardly agile. Manual mobile game testing is not an exception, and it significantly slows down the release and delivery process. Non-agile, waterfall process looks pretty much like this:

In this way of thinking, testing is only taking place at the end in post-production – and that’s not efficient. How can testing start sooner and how can mobile developers use existing testing methodologies while the game is still under development? The answer is test automation and efficient use of it as part of the development process.

Mobile developers use pretty much the same methods as mobile app developers when building their games. They use continuous integration, agile methodologies, source code repositories, and DevOps tools. The use of these methods and tools make test automation a seamless part of the process and provide an easy way to start testing a game while it is still under development.

Testing is very important; nearly all of it can be included as part of the agile development process and every aspect of the game can be tested during the development – and during the production. For instance, many aspects of design, look-and-feel, and other important foundations for user experience can be tested even during pre-production.

Mobile test automation helps game developers to include and integrate testing as part of the agile development process. This will significantly improve the efficiency of getting implementation tested and produce results that enable early releases with less bugs.

Mobile test automation is a modern way to enable simultaneous, automatic testing of mobile games based on certain developer/user-given test scenarios.

The differences between app and game test automation

What are the major differences between testing a mobile application and a mobile game? There are a few important differences that developers, testers and QA folks must know when using test automation for mobile games.

User interface and its functionality

A user interface and its overall functionality will directly affect how successful your app or game will be. These two things, which encompass visual appeal and gameplay, are the most important things to get right – and you must ensure that device fragmentation doesn’t break any of these. Various things in the UI need to be tested:

• UI layouts and elements. Games especially are typically targeted at a high number of different screen resolutions and screen types. Regression testing should be done each and every time the UI’s layout changes to ensure that the game works.

• Menu structure and functions. Testing menu structures, functionality and behavior can be automated with instrumentation and the help of different test-automation frameworks.

• Screen orientation. Surprisingly, many apps and games out there get this one wrong. For example, if a screen’s orientation changes during an interaction, then what happens? What is supposed to happen? Does the app or game work well in both landscape and portrait modes?

• Screen resolution. A lot of screen resolutions exist, especially on Android, and auto-scaling will usually help developers. However, test your game across these resolutions to ensure that the graphics do not stretch.

Graphics performance

Good graphics performance of any mobile game is tightly related to good user experience. Stunning graphics, pictures, animations and all those beautiful effects will make the mobile game shine; but, if the performance lags, they are pretty much useless. In the gameplay, gamers want to see constant progress, feel that the game is fully implemented for their device, and that nothing slows down the game experience.

Performance needs to be consistent across all device variants among your users. Because of this, use test automation on as many real devices as possible. To determine how well your game responds to various levels of usage, including performance and battery usage, consider creating tests that last for hours. To determine whether your game runs effectively under a heavy load for a long time, run load (or stress) tests. These performance tests will measure, for example, how responsive your game is on real devices.

Usability and gaming experience

Testing usability, navigation flow and user experience simply cannot be done on a desktop with a mouse and keyboard. So, forget emulators and use only real devices. And to test how usable and entertaining your app is, consider these two important things:

• User interaction and responsiveness: Testing performance is critical because this will make or break the user experience. Performance lag, for example, is easy to expose with real devices.

• Background events: Interruptions, battery consumption and the effect of battery chargers on overall performance and usage all have a significant impact on the user experience — and entertainment value.

Multiplayer features

Nowadays, multi-user support is common in both apps and games. Testing multi-player capabilities is important and is naturally more challenging, requiring real users to measure performance. A typical case is a game communicating with the back-end server. In this case, connectivity is essential, to synchronize the back-end with devices that need to get information about the gameplay. You should test a ton of different scenarios, many of which could severely affect the game’s experience, resulting in negative feedback and the game being uninstalled by users.

Integration with social networks is another important factor. Being able to share something across an ecosystem, with friends or just with oneself, is essential in many apps. Test this thoroughly with real Android and iOS devices, with different OS versions and different device configurations, to assess functionality and ease of use.

Image recognition – Easiest way to automate mobile game testing

Many of the top mobile game development companies use image recognition to get their latest mobile games tested. All this is done in test automation context and industry-standard open source test automation frameworks are used.

Image recognition isn’t really a new thing. But in mobile game testing, it has recently gained popularity due to its easy setup, easy to use, and the way it can help separate developers from testers. Test scripts are driving the execution and testers can simply execute tests without programming skills.

With the help of Akaze and OpenCV, mobile developers can quickly build functions that compare the screen and graphics content in .png files. The idea is to give visual assets (.png files) as they are and test script will compare and perform action whenever those visual assets are shown on screen.

For example, with some of the test frameworks the timing and delays have been problematic. With this sort of implementation you don’t need to implement delays or any timing logic for scripts but instead the script can wait until certain visual assets are shown on screen.

What’s next

In my next blog article, I’ll take a deeper technical dive to show you how to get started with image recognition on Bitbar Public Cloud and how use real devices on cloud. Stay tuned!

References

• Best practices in mobile game testing (bitbar.com)

• Testdroid Cloud: Testing on Real Mobile Devices on Cloud (ibm.com)