How Long to Run Screenshot A/B Tests

The Patience Challenge

One of the hardest aspects of A/B testing is waiting long enough for statistically valid results. The temptation to check early results and call a winner is strong - especially when one variant is outperforming. But ending tests prematurely is one of the most common and costly mistakes in A/B testing.

Statistical significance isn't just academic rigor - it's protection against false positives. If you end tests too early, you risk implementing changes that were actually just random variation, not genuine improvements. At scale, this can mean pursuing optimizations that actually hurt your conversion rate.

Understanding how to determine the right test duration saves you from both the cost of false positives and the opportunity cost of running tests longer than necessary.

Minimum Duration Guidelines

Your test needs to run long enough to capture representative user behavior. At minimum, this means running through at least one complete weekly cycle - ideally two.

User behavior varies significantly by day of week. Weekend users often differ from weekday users in intent, engagement, and conversion behavior. A test that runs only Tuesday through Thursday misses weekend patterns entirely and may produce skewed results.

Seven days is the absolute minimum for any test, but 14 days is generally recommended for reliable results. This ensures you capture at least two weekends and accounts for typical weekly fluctuations.

For apps with lower traffic, longer durations become necessary. You need enough conversion events to produce statistically meaningful comparisons. An app with 100 daily page views needs much longer than one with 10,000 to reach the same statistical confidence.

Calculating Required Sample Size

Statistical significance depends on both the size of the effect you're trying to detect and the number of observations in your test. Smaller effects require larger samples to detect reliably.

Before starting a test, determine the minimum effect size that would be meaningful for your business. If a 2% conversion improvement wouldn't justify implementation costs, you don't need a test sensitive enough to detect 2% changes.

Sample size calculators can help determine how many visitors you need. For a typical screenshot test aiming to detect a 10% relative improvement with 95% confidence, you might need 10,000-20,000 visitors per variant. Smaller apps may need to accept lower confidence thresholds or longer test durations.

Both Apple's Product Page Optimization and Google's Store Listing Experiments provide statistical significance indicators. Trust these rather than making your own calculations - they're designed for app store conversion patterns.

When to End Your Test

The ideal scenario: your test reaches statistical significance (typically 95% confidence) with a clear winner. Both app stores will indicate when this threshold is reached. At this point, implement the winner and move on to your next test.

Maximum test duration matters too. Apple limits Product Page Optimization tests to 90 days. Google's experiments have similar practical limits. If you haven't reached significance within these windows, you need to make a judgment call.

If your test is close to significance, consider whether extending it slightly might push it over the threshold. If after adequate duration your test shows no clear winner, that's itself a finding - the changes you tested probably don't matter much to users.

External factors can invalidate tests. If you run a promotion, get featured, or experience a sudden traffic spike partway through your test, consider whether this external factor might have skewed your results. Sometimes the cleanest approach is to restart the test under more stable conditions.

Don't peeking too frequently or emotionally react to early fluctuations. Set a calendar reminder to check results at appropriate intervals rather than obsessively monitoring daily numbers.