App Store screenshot A/B testing: how to run a test that gives an answer
App Store screenshot A/B testing means showing part of your store visitors an alternative screenshot set and comparing how many of them download. On the App Store you do it with Product Page Optimization in App Store Connect; on Google Play with store listing experiments in Play Console. Both are built in and free to use.
This guide covers the limits of each tool, what to test first, how to build variants that produce a readable result, how long a test needs and how to read the outcome. The design rules for the screenshots themselves are in our App Store screenshot best practices.
How screenshot A/B testing works on each store
Both tools split traffic at random: a share of visitors sees a variant instead of your current page, and the store counts conversions for each group. In App Store Connect you set a traffic proportion that is split evenly among treatments, so 30% with three treatments gives each one 10%. Play Console splits its audience percentage equally across variants.
Apple shows treatments only on iOS 15 and iPadOS 15 or later, and product page optimization is not available for custom product pages or for App Store product pages on Apple Watch and iMessage. These are the limits from App Store Connect Help and Play Console Help (October 2026):
| Setting | App Store (Product Page Optimization) | Google Play (store listing experiments) |
|---|---|---|
| Variants per test | Up to 3 treatments against the original page | Up to 2 variants against the current listing |
| Tests at a time | One test | One default graphics experiment; up to 5 localized experiments, each with up to 5 languages |
| What you can test | App icon, screenshots, app previews | Icon, feature graphic, screenshots; descriptions in localized experiments |
| Maximum length | 90 days, or until you stop it | Stops automatically after 6 months |
| Before it starts | New metadata approved by App Review; alternate icons in the app binary | Published app, tested against the current version |
| When a result counts | Better or worse than the baseline with at least 90% confidence | Difference above your minimum detectable effect at your chosen confidence level |
What to test first
Each test should answer one question. Apple advises considering how many elements you change in a treatment so you can tell which one caused the result, and Play Console Help recommends testing one asset at a time. Start where most visitors look:
- The first screenshot. When there is no app preview, Apple shows the first one to three screenshots in search results, so frame 1 is seen by people who never open your page.
- The headline on frame 1: one clear benefit against a list of features.
- The message that leads. Apple names showcasing a certain character or value proposition as one of the things a test can show works better.
- The order: the same images with a different opener.
- Device presentation: plain UI, a flat device frame or an angled 3D device.
- Connected against separate frames, using the same screens in both. Our panoramic screenshots guide explains how to build the connected version.
- Seasonal or localized content. Apple lists seasonal content and culturally relevant content for a specific localization as changes that can lead to more downloads.
How to build screenshot variants that give a clear result
A treatment is a complete screenshot set. In App Store Connect you upload treatment screenshots per device size, and any device size you leave unedited uses the screenshots of your original product page. A test that changes only the iPhone set still shows your original iPad screenshots to iPad visitors.
Keep everything except the tested variable identical: the same app screens, order, background and export size. A treatment that changes the caption and the device angle at once can win, but you will not know which change did it.
Make the difference big enough to measure. Apple notes that a large lift needs less time to detect than a small one, and suggests testing a more dramatic change when a test is likely to be inconclusive. Treatment screenshots follow the same rules as your main set, so check them against the App Store and Google Play screenshot sizes before uploading.
Setting up the test in App Store Connect and Play Console
App Store: in App Store Connect, open the app, choose Product Page Optimization and create a test. Set the traffic proportion and the localizations (all are selected by default), then configure each treatment. The app must be live with the status Ready for Distribution or Pre-Order Ready for Distribution. New screenshots in a treatment must be approved by App Review before the test can start; tests without alternate icons can be submitted without a new app version, while alternate icons must be in the binary of the current version. Releasing a new version during a test can affect the results if it changes the tested assets.
Google Play: store listing experiments run on published apps, on the default store listing or a custom store listing. Choose a default graphics experiment (default language only; it excludes users who see localized graphics) or a localized experiment, pick the target metric, such as unique user install clicks, and set the audience, minimum detectable effect and confidence level. Play Console then estimates how long the experiment needs.
How long a screenshot test needs to run
There is no fixed minimum. App Store Connect shows results once at least five first-time downloads are attributed to the test and updates them daily, but Apple advises against ending a test early: wait until at least one treatment is declared better or worse than the baseline with at least 90% confidence. A test runs for up to 90 days, and App Store Connect can estimate the duration from your daily impressions and new downloads for the improvement you want to detect.
Traffic is the main constraint. Apple notes that more treatments and the localizations you select can make a test take longer, so a small app reaches a result sooner with one treatment and a high traffic proportion. On Google Play an experiment can report “More data needed” and stops automatically after six months.
Reading the results and applying the winner
App Store Connect labels each treatment Performing Better, Performing Worse, Collecting Data or Likely to be Inconclusive, next to its estimated conversion rate, improvement over the baseline and confidence. Apple’s analysis is Bayesian and combines test data with existing data about your product page. Applying a treatment copies its screenshots and app previews to your product page and ends the test; it cannot be undone, and an icon is not changed this way — you set the winning icon as the default in your next app version.
Play Console reports a winner, a draw when the difference is below your minimum detectable effect, or “More data needed”. You can keep the current listing if no variant wins.
A test that finds no winner is still a result: keep the original and test a bigger change next. Write down each hypothesis and outcome so the next test starts from what you learned.
Building test variants in Ecranify
In the full Ecranify editor a project holds several scenes, one per screenshot. Add Scene copies the current scene, so a variant starts as an exact duplicate: change only the caption, the device angle or the background, and every other setting stays the same. Batch export renders all scenes and downloads them as a ZIP, ready to upload as a treatment set.
Exports in the full editor are paid per render with credit packs; the free editor exports single screenshots at preset sizes. To try a layout before you build the variants, start in the free editor.
Frequently asked questions
Related guides & comparisons
Build every test variant from one scene
Duplicate a scene, change one thing, and export each variant at the exact App Store or Google Play size.