Mobile App Upgrade Testing – How to plan

Mobile app upgrade testing is vital because if you are a smartphone user, you will know how often apps are updated. It’s come to a point where most users just flip on automatic updates and forget about them. Maybe they see the notification that informs them that 23 of their apps were updated overnight. Some users though may be interested in reading about the update so they can check the notes before approving the update.

If you’ve ever read app reviews, you may notice at some point in the reviews one or more users will mention that the app no longer functions after the latest update, or after they upgraded the device OS. You can guess how many stars those reviewers leave in the app store.

There are quite a few things that can occur with app updates. Based on the changes, some updates are obviously less likely to cause issues than others. Being informed about the changes going into the release can help identify the risks and areas to focus on. A version bump of an SDK (Software Development Kit) integrated with your app, for example, may behave differently across OS versions. It might work flawlessly on Android 9.0, but a change to the third-party SDK might affect Android 6.0. The same goes for iOS and SDK integrations. A surprising number of crashes occur after updates on both Android and iOS apps.

Alright, so we’ve established that update testing is a big deal and we should think carefully about how to approach it. Let’s go over some scenarios we should cover and some ways to cover them efficiently.

Upgrading Multiple App Versions

When we begin to consider which upgrade to test before a release, we should first look at some data. What app versions are currently being used in production and what percentage of the user base is using each?

The scope of update testing depends on how many versions need to be tested. If the last app update was several months prior, chances are any active users are all on the latest version by now. If the last update was only a few days ago and updates were released every week prior to that, then you may have a significant percentage of users on multiple versions. This will increase the scope a bit.

Assuming you have multiple versions to test, the logical place to start is with the version that has the highest usage percentage. Users who have auto-updates on would have the latest version, so this is likely the most used version. This is where you want to focus the most attention because issues with this upgrade would affect the biggest audience, and likely the most frequent users.

OS Versions

Now that you’ve figured out the priority of app versions to test with the latest build, you can now test that upgrade on multiple OS versions. Turning to analytics will once again give the data needed to decide where to focus testing. The OS version with the highest percentage of users, and then the app version with the highest percentage of users, will be the area to focus on. This would be the scenario to look at on multiple devices in different states. With the right set of test cases and diligent testing of upgrades for this set of users, you can be assured this user base won’t be affected.

With any OS update, it’s always best to start early. When a beta version is available, it’s good to do a sanity test on that OS version with each app update. Testing around that version should then ramp up as it gets closer to a public release.

Latest and Minimum

When testing app upgrades, another important test is the verification of upgrading with the minimum OS version and the most recently released OS version. The minimum OS version that is supported is at high risk for issues because some new app features may not be supported by the minimum supported version. The issue could manifest itself in a crash after the update.

On the other end of the spectrum is the latest OS version. Depending on the season, it could be very few users or a very high percentage of users. Even if the latest Android OS update was only just released and few users have it, it is another area where the risk is high. The OS version wouldn’t have been tested in public and there are still likely to be tweaks and bug fixes being worked out. Obviously, issues such as these need to be discovered and mitigated before the user base of that OS increases rapidly.

Important States and Scenarios

When testing app updates, there could be a number of states to look at depending on app functionality and desired behavior after an upgrade. Here are a few that would apply to a lot of apps:

• Authentication – If your app uses authentication in any way, it is imperative to ensure the user remains authenticated. Nobody enjoys launching an app and being required to authenticate after an update.
• Customization – Depending on the app and the customization options, this could be a big one. Apps that include favorites or UI customizations shouldn’t be lost.
• Notifications – Verify that notifications are still received, and that notification settings have not been affected following the upgrade.
• New or updated functionality – New screens, and especially those that use an updated third-party SDK version, should be viewed after an upgrade. Oftentimes, the app won’t crash if it is a clean install but will crash after an upgrade.

Conclusion

As with most planning for mobile app upgrade testing, the best place to start is by identifying the highest risk area. With the speed at which mobile app updates are released, you could find that upgrade testing is one of the most often completed test sets. With target versions changing frequently, it’s important to carefully consider the versions being tested and remember to stay up to date on the latest app usage data. Hopefully, these tips will help you to feel confident that you have created the best plan to test your app upgrades.