Learning from user trends to improve testing coverage for mobile projects

Anybody who has ever worked on mobile apps understands this pain all too well. Because this market is so fragmented, ever-changing and at the same time so unbelievably interesting, it will be the focus of this article.

Coming up with a testing strategy is a complicated thing to do. Many different parties need to be satisfied, countless things have to be taken into consideration, there are many time constraints, and so on...The good part is that there are a few things that come for free and it’s only a matter of putting them to use.

Start looking at the information available

For either Android or iOS projects, the chances are that information is already available about what devices, operating systems, languages your users have. The data is segregated by different categories in Google Play Console (Android) or App Store Connect (iOS). In case these don’t satisfy your needs, there are other solutions available like Google Analytics, Firebase, Localytics or Appsee which can be used together with the default ones. 

Just remember, we’re talking about anonymized data which doesn’t include any PII (Personally identifiable information) and which users agreed to share with app developers. What we are interested in, are things like: 

• What version of the operating system do the majority of users have

• What types of devices are mostly used

• What languages are mostly used

• What’s the percentage of phone vs tablet users (this one can really be a game-changer, as supporting only one form factor will reduce risk and cut down on the time needed to test)

It’s only a matter of looking at the data and making some decisions based on that. Updating the testing coverage is a big and risky business, the whole team should be involved and made aware of this. 

While all this might sound like a silly or normal thing, there are many companies/projects which don’t really look at the data being collected. Such a waste...

PRO tip: Very closely connected to this, since you’re checking metrics and improving coverage, make sure to ALWAYS test on the lowest supported version (this is where a big part of the risk comes from). 

Constantly review your testing process

It’s very easy to fall into a trap and don’t improve the testing process. Doing what you’re doing is easy because you’ve become experienced and familiar with everything... But think if what you are currently doing still makes sense or if that time can be better spent on something else. 

Also, having information about how the product is being used is useless if it is not used. Consider having most users on Android 7.0 and the testing team using only Android 9.0 because those devices are faster and it makes their job easier. 

Let’s take another example: most of your customers are using the product in German. This means that there is a need to focus more resources on this language. Another reason why testing using German language is important is that German words tend to be very long. If you’re not handling this well, you might end up with strings which are cut, which are not displayed, or worse (think about words like Rechtsschutzversicherungsgesellschaften).

What would make sense for us as testers is: every 3 - 6 months take a look at the data and check if anything is different: 

• If the top 5 (or 10) most used devices are in the device rotation of your testing team 

• If the device pool needs to be refreshed with new devices

• If the most used operating system versions have changed

• If the ratio phone vs tablet has changed

• …

Don’t come unprepared to the battle

Testing can be considered a battle: the more prepared, the better the strategy one has, the higher the chances are of being successful. The battle is not between the testing team and the development team. It’s between the team, all the challenges and complexities that come with the software development life cycle. 

In more realistic terms, a big topic each year is the next operating system. Both Apple and Google tend to have one major release every 12 months or so, which means we have little time to prepare. Documentation is announced and finalized late, it changes without any notice and there’s of course the rest of the product that we need to think about and keep testing.

This is why, if no effort went into preparing for Android Q or iOS 13 (don’t forget iPad OS), this is a good moment to start thinking and planning. Once testing is done and your app works on the shiny new piece of software, make sure to keep this preparation in mind for future years to come.

Conclusion

This article’s purpose is to confirm that you’re already doing the right thing and if not, give you a chance to start a conversation and try to push things for going the right direction. Similarly to what the Agile manifesto says, it’s not about the tools, it’s about the people. People who analyze the data, take decisions and make the experience better.

Improving the coverage or designing your strategy when it comes to coverage is only the beginning. There are many other things that we can talk about. The same tools which have information about what OS versions and which devices users have, also have information about crashes, performance and much more. But this is a conversation for another time.

In case you’re having trouble with testing in your company/department, you can always reach out for advice, for exchanging ideas or for working with us.