How Long to Run Ad A/B Tests: Sample Size Math Made Practical
A/B test duration is arithmetic, not folklore: the required sample divided by expected daily observations. The sample-size math made usable, plus conversion delay, stop rules, and why no calendar floor or ceiling fits every test.

Plan duration from the experimental unit, baseline rate, effect worth detecting, error tolerance, allocation and expected observations per day. Also allow for conversion delay and relevant cycles. There is no universal conversion-count or calendar threshold that makes every ad test conclusive.
Duration is an output, not a calendar choice#
Expected observations per variant affect the time needed to reach a planned sample. Conversion counts alone do not define the required sample: specify the baseline rate, effect size, design and uncertainty assumptions.
- The baseline rate of the metric you are testing — CTR for creatives, conversion rate for landers and offers.
- The smallest difference worth detecting. Detecting a 10% lift costs far more traffic than detecting a 50% lift, and most small lifts aren't worth the spend it takes to see them.
- Daily volume per variant — set by your budget and bids.
Estimate the required observations under the chosen design and divide by expected observations per variant per day. Check whether the resulting window remains relevant; a four-week duration is not an automatic invalidity threshold.
The sample-size math, minus the statistics degree#
One rough planning approximation for two independent proportions uses equal allocation, a fixed horizon, a two-sided 5% significance level and 80% power:
n per variant ≈ 16 × p × (1 − p) ÷ d²
Here p is the assumed baseline event rate and d is an absolute rate difference. The result counts observations per variant, such as impressions or eligible visits, not conversions. The approximation needs checking for rare outcomes, unequal allocation, repeated users, clustering and other departures from the design.
Hypothetical calculations: p=0.005 and d=0.001 yield about 79,600 impressions per variant. p=0.02 and d=0.005 yield about 12,544 eligible visits per variant. If each such visit costs US$0.40, two arms cost about US$10,035 in media before other costs. These are planning scenarios, not industry benchmarks or promises of significance.
| What you're testing | Baseline | Lift to detect | Sample per variant |
|---|---|---|---|
| CTR (creative) | 0.5% | 20% relative | ~80,000 impressions |
| CTR (creative) | 0.5% | 50% relative | ~13,000 impressions |
| CVR (lander) | 2% | 25% relative | ~12,500 clicks |
| CVR (lander) | 2% | 50% relative | ~3,100 clicks |
| CVR (offer page) | 5% | 25% relative | ~4,900 clicks |
Under this approximation, halving d quadruples the planned sample. CTR and conversion answer different questions: a CTR improvement need not improve contribution or acquisition cost. Choose metrics and guardrails around the decision, rather than assuming impression-level testing is always cheaper or sufficient.
Allow for relevant cycles and conversion delay#
Choose a window that represents the relevant traffic and allows delayed outcomes to mature. Weekly patterns may matter, but seven days is neither a universal minimum nor proof that the design is representative.
The ceiling: long tests rot from the inside#
Longer windows can introduce changing conditions, but a test does not automatically cease to be controlled after three or four weeks. Assess randomization, concurrent exposure, drift, delayed outcomes and practical relevance; revise the design if needed.
Don't peek — or at least don't act on peeks#
Repeatedly checking a fixed-horizon test and stopping at the first favorable significance result can change its error rate. Prespecify the design and analysis, or use an appropriate sequential method. Operational stop conditions are distinct from a statistical success declaration.
Kill rules for when you can't afford significance#
Most native and affiliate buyers cannot fund 12,500 clicks per variant, and pretending otherwise produces fake rigor. The honest alternative is staged culling — cruder than significance testing, and standard practice among buyers who test dozens of creatives a month:
A financial stop rule can limit exposure but does not establish statistical significance. If screening variants before a confirmatory test, account for selection and multiple comparisons and use fresh, appropriately designed evidence for the final decision. A 2–3× CPA rule or a few thousand impressions is not universal validation.
Use competitor observations to form hypotheses#
Competitor captures can suggest creative hypotheses. A 30-day observation span does not establish that a competitor passed a profit test, and selecting a familiar angle does not reduce the sample needed for your chosen statistical design.
A worked example, end to end#
Say you run a $60-payout lead-gen offer with one campaign doing roughly 1,500 clicks a day across your test cells.
In a hypothetical equal-allocation test with p=0.03 and d=0.009, the approximation yields about 5,748 eligible visits per arm. At 1,500 total eligible visits per day, collecting both arms takes about 7.7 days before accounting for delay, cycles or other design requirements. An earlier screening stage does not guarantee a defensible winner after 10–12 days.
A planned duration combines sample requirements with expected eligible traffic and the timing of outcomes. It is not a guarantee that the experiment will produce a statistically or commercially decisive winner.







