Holdout Test (Advertising): Definition & Uses

A holdout test is an experiment design that withholds an ad or a marketing treatment from a portion of your audience, the holdout group, while the rest of the audience continues to receive it, then compares outcomes between the two groups to measure the treatment's real effect. It's the umbrella term for the broader family of causal-measurement experiments used in advertising, covering everything from a full regional pause to a narrow frequency-cap test.

The basic mechanics#

Split your audience into two groups that are as similar as possible on the traits that predict your outcome metric: past purchase behavior, region, device type, whatever matters for your business. One group, the holdout, sees no ads (or a public service message in their place); the other group sees the campaign as normal. After a set period, compare the conversion rate, revenue, or whatever metric you're testing between the two groups. The difference, adjusted for any baseline gap between the groups, is the treatment's incremental effect.

Why platforms build this in#

Most major ad platforms with a self-serve demand-side platform now offer some form of built-in conversion-lift or holdout testing, because advertisers kept asking the same question their own conversion tracking dashboard couldn't honestly answer: would these conversions have happened anyway? A dashboard showing a strong ROAS number is reporting associated conversions, not necessarily caused ones, and only a genuine holdout separates the two.

Where holdout tests are used across the funnel#

A full-channel holdout pauses an entire network or channel for a subset of users or regions, the design behind a regional lift test. A campaign-level holdout pauses one specific campaign while others continue, isolating that campaign's effect from the channel's baseline. A frequency holdout caps how many times a user sees an ad, then compares outcomes against an uncapped group, useful for measuring creative fatigue and diminishing returns.

Practical constraints#

Running a holdout test means deliberately not showing ads to some real, addressable customers for the test's duration, which has a real short-term cost. That's the tradeoff every incrementality-minded team accepts: give up some near-term reach and revenue in exchange for a trustworthy answer about whether the spend actually works. Sample size matters too. A holdout needs enough volume in both groups to detect a statistically meaningful difference; testing a low-traffic channel or a short campaign often just produces noise dressed up as a result.

Reading the result correctly#

A null result, no meaningful difference between exposed and holdout groups, is a real finding, not a failed test. It means the channel, at least for that audience and that period, wasn't adding incremental value beyond what would have happened anyway. Before scaling a channel based on strong reported numbers, check whether competitors in your vertical are also running heavy ad spend there, since a channel crowded with long-running creative, see ad longevity, is a reasonable proxy for one where holdout tests tend to confirm real lift. OpenAdLibrary's ad intelligence platform lets you check that activity by network and vertical before you commit test budget.