Incrementality Testing: Definition & How It Works

Incrementality testing is a controlled experiment that measures how much of your reported conversions an ad channel actually causes, rather than how many conversions the platform's pixel simply claims credit for. You hold back a group of users or geographies from seeing a specific channel's ads, then compare the outcome against a group that saw them. Whatever gap remains between the two groups is the channel's real, incremental contribution. Everything else is conversions that would have happened anyway.

Why platform-reported numbers overstate results#

Every ad network wants credit for a sale. If someone was already going to buy through organic search or a branded search click, and also happened to see a retargeting ad along the way, standard conversion tracking can still hand that credit to the ad inside its attribution window. Even softer credit, where a platform counts a conversion after someone merely saw an ad, adds to the problem; that mechanic is called view-through attribution. Incrementality testing strips this noise out by asking a blunter question: if this channel had spent zero dollars this month, would the conversion still have happened?

That's a different question from what a tracking pixel answers on its own, which is why teams running six-figure monthly native or social budgets eventually build an incrementality program alongside their platform dashboards, not instead of them.

How teams run it in practice#

Two designs cover most real-world cases. A geo-level test pauses a channel entirely in a set of matched regions and compares conversion rate against regions where spend continued. A user-level test uses the ad platform's built-in conversion-lift tooling, offered natively by several demand-side platforms, to withhold ads from a randomized slice of the audience.

Either way, the math is the same: incremental lift equals the test group's conversion rate minus the holdout group's conversion rate. A channel showing a strong self-reported ROAS in-platform might show close to zero incremental lift once a clean holdout removes the users who would have converted regardless.

What it's used for#

Media buyers run incrementality tests to decide budget allocation across networks, not just to audit one channel in isolation. A pattern we see across the index: brands running the same creative angle simultaneously across several networks often assume all of them are pulling weight, when a lift test shows one channel is mostly recapturing demand another already created.

Before running a formal test, most teams start by watching what competitors in their vertical are actually scaling, since a channel crowded with long ad longevity is a decent proxy for where incremental budget tends to work. OpenAdLibrary's ad intelligence tools let you check creative overlap and run duration across networks before you commit test ad spend to a holdout.

Common mistakes#

The holdout group needs to be large enough and random enough to detect a real signal, not just noise. A one-week test on a low-volume channel usually can't reach statistical significance. Teams also frequently run the test during a seasonal spike or a site-wide promotion, which contaminates both groups equally and makes the lift number meaningless. Run the test during a representative period, size the holdout with a proper power calculation, and don't stop early just because the interim number looks good, or bad.