How to Test an App Icon

A quick and dirty way to A/B test before you ship.

If you work in mobile app product management, you no doubt use A/B tests to help drive all kinds of decisions around features, enhancements, bug fixes, and other changes to your app. But did you know you can even A/B test your app’s icon?

A great icon is important. It can be the difference between someone downloading your app over another. Once someone downloads your app, a great icon can help keep a user engaged by standing out on a home screen.

Whether you’re designing a new icon from scratch or optimizing an existing one, here’s one way to A/B test your icon that will get you fast results and only cost you a couple hundred bucks:

1. Design several variations of your icon. You’ll need at least two. For conclusive results, you probably shouldn’t test more than four or five at a time. If you need help designing your icon, check out this advice from Apple’s Mike Stern.
2. Create a simple mobile ad banner to promote your app. A mobile leaderboard or inline rectangle should do the trick. The emphasis of your banner should be on the icon.
3. Make a copy of this banner for each of the icon variations you want to test. For each copy of the banner, swap in the icon variation. Change nothing else within the banner. The only differences between these banners should be the icons.
4. For each banner, set up a mobile ad campaign on AdMob. If you haven’t used AdMob before, just sign up quick and head to the “Promote” section. Make sure to set up individual campaigns for each banner, not a single campaign with multiple creatives. If you want the test to be quick, you can probably go with a max CPC of around $0.50, and a max daily budget of $20 for each campaign. If you don’t care too much about getting results back quickly, feel free to lower the CPC and budget to spend a little bit less.
5. Watch the campaigns for a few days and see which banner gets the highest CTR. End the campaigns when the results stabilize. You should be able to get enough solid data in less than a week. Note that with short campaigns, AdMob sometimes runs over your daily budget (it gets balanced out over the course of a month), so watch the campaigns closely to monitor how much you’re spending.

And you’re done. Most likely, there will be a clear cut winner. Marketplace conditions are obviously very different than what users will experience when looking at your banner, so this is by no means a perfect method. However, it should give you a good sense of which icon is best.

Have you used mobile ad banners for other types of A/B or multivariate tests? If so, I’d love to hear about them on Twitter.

If you found this article helpful, please take a moment to click the “Recommend” button below. It really helps! Thanks.