OpenAdLibraryOpenAdLibrary
Ad Creative & Funnels

How Long to Run Ad A/B Tests: Sample Size Math Made Practical

A/B test duration is arithmetic, not folklore: the required sample divided by expected daily observations. The sample-size math made usable, plus conversion delay, stop rules, and why no calendar floor or ceiling fits every test.

Editorial illustration: How Long to Run Ad A/B Tests: Sample Size Math Made Practical

Plan duration from the experimental unit, baseline rate, effect worth detecting, error tolerance, allocation and expected observations per day. Also allow for conversion delay and relevant cycles. There is no universal conversion-count or calendar threshold that makes every ad test conclusive.

Duration is an output, not a calendar choice#

Expected observations per variant affect the time needed to reach a planned sample. Conversion counts alone do not define the required sample: specify the baseline rate, effect size, design and uncertainty assumptions.

  • The baseline rate of the metric you are testing — CTR for creatives, conversion rate for landers and offers.
  • The smallest difference worth detecting. Detecting a 10% lift costs far more traffic than detecting a 50% lift, and most small lifts aren't worth the spend it takes to see them.
  • Daily volume per variant — set by your budget and bids.

Estimate the required observations under the chosen design and divide by expected observations per variant per day. Check whether the resulting window remains relevant; a four-week duration is not an automatic invalidity threshold.

The sample-size math, minus the statistics degree#

One rough planning approximation for two independent proportions uses equal allocation, a fixed horizon, a two-sided 5% significance level and 80% power:

n per variant ≈ 16 × p × (1 − p) ÷ d²

Here p is the assumed baseline event rate and d is an absolute rate difference. The result counts observations per variant, such as impressions or eligible visits, not conversions. The approximation needs checking for rare outcomes, unequal allocation, repeated users, clustering and other departures from the design.

Hypothetical calculations: p=0.005 and d=0.001 yield about 79,600 impressions per variant. p=0.02 and d=0.005 yield about 12,544 eligible visits per variant. If each such visit costs US$0.40, two arms cost about US$10,035 in media before other costs. These are planning scenarios, not industry benchmarks or promises of significance.

What you're testing Baseline Lift to detect Sample per variant
CTR (creative) 0.5% 20% relative ~80,000 impressions
CTR (creative) 0.5% 50% relative ~13,000 impressions
CVR (lander) 2% 25% relative ~12,500 clicks
CVR (lander) 2% 50% relative ~3,100 clicks
CVR (offer page) 5% 25% relative ~4,900 clicks

Under this approximation, halving d quadruples the planned sample. CTR and conversion answer different questions: a CTR improvement need not improve contribution or acquisition cost. Choose metrics and guardrails around the decision, rather than assuming impression-level testing is always cheaper or sufficient.

Allow for relevant cycles and conversion delay#

Choose a window that represents the relevant traffic and allows delayed outcomes to mature. Weekly patterns may matter, but seven days is neither a universal minimum nor proof that the design is representative.

The ceiling: long tests rot from the inside#

Longer windows can introduce changing conditions, but a test does not automatically cease to be controlled after three or four weeks. Assess randomization, concurrent exposure, drift, delayed outcomes and practical relevance; revise the design if needed.

Don't peek — or at least don't act on peeks#

Repeatedly checking a fixed-horizon test and stopping at the first favorable significance result can change its error rate. Prespecify the design and analysis, or use an appropriate sequential method. Operational stop conditions are distinct from a statistical success declaration.

Kill rules for when you can't afford significance#

Most native and affiliate buyers cannot fund 12,500 clicks per variant, and pretending otherwise produces fake rigor. The honest alternative is staged culling — cruder than significance testing, and standard practice among buyers who test dozens of creatives a month:

A financial stop rule can limit exposure but does not establish statistical significance. If screening variants before a confirmatory test, account for selection and multiple comparisons and use fresh, appropriately designed evidence for the final decision. A 2–3× CPA rule or a few thousand impressions is not universal validation.

Use competitor observations to form hypotheses#

Competitor captures can suggest creative hypotheses. A 30-day observation span does not establish that a competitor passed a profit test, and selecting a familiar angle does not reduce the sample needed for your chosen statistical design.

A worked example, end to end#

Say you run a $60-payout lead-gen offer with one campaign doing roughly 1,500 clicks a day across your test cells.

In a hypothetical equal-allocation test with p=0.03 and d=0.009, the approximation yields about 5,748 eligible visits per arm. At 1,500 total eligible visits per day, collecting both arms takes about 7.7 days before accounting for delay, cycles or other design requirements. An earlier screening stage does not guarantee a defensible winner after 10–12 days.

A planned duration combines sample requirements with expected eligible traffic and the timing of outcomes. It is not a guarantee that the experiment will produce a statistically or commercially decisive winner.

Frequently asked questions

How many conversions do I need per variant in an ad A/B test?
Required observations depend on the baseline rate, absolute effect, power, error tolerance and design. The displayed formula is a rough approximation for independent proportions, not a precise conversion-count rule.
Is 3 days long enough to run an ad A/B test?
It depends on the design, eligible traffic, relevant cycles and outcome delay. Three days can be too short for a particular question, but seven days is not a universal validity threshold.
Should I test ads on CTR or conversion rate?
Choose the metric for the decision and include downstream guardrails. CTR and conversion rate answer different questions; this article does not establish a universal 10–100× cost difference.
What happens if I stop an A/B test as soon as it hits significance?
You inflate your false-positive rate dramatically. Checking daily and stopping at the first significant reading means many 'winners' are noise that would have regressed with more data — the classic peeking problem. Pre-commit a sample size, use daily checks only for stop-loss conditions like broken tracking or runaway spend, and evaluate once when the sample is complete.
Why did my winning ad variant stop winning after the test?
Possible explanations include uncertainty, selection bias, changing traffic, delayed outcomes and actual changes in performance. The article has not established a usual cause; inspect the design and subsequent data.
The OpenAdLibrary Team
Written byThe OpenAdLibrary Team
Ad intelligence & native advertising research

We build OpenAdLibrary, the open ad-transparency platform. Every day our systems capture live native ads across Taboola, Outbrain, MGID, Revcontent, Teads, Yahoo and MSN, identify the real advertiser behind each one, and follow the click to its landing page. These guides distill what we see in that data so you can research the market faster.