Intro to Incrementality Analysis Series: Part I

Welcome to the first part of our Intro to Incrementality series, which uses real-world examples to explain how incrementality analysis and incremental lift can help brands and agencies isolate the effect of a particular media type or campaign variable in order to accurately measure its impact. Warning: this post involves (basic) math.

Isolating the Impact

As a marketer, the line between correlation and causation can get very blurry. In the modern ad landscape, brands run ads across many different media channels—from paid search to OTT/CTV, display to streaming audio. Customers are often exposed to several different media types—and several different ads within each media type—before making a purchase or converting. Therein lies the challenge: How do marketers know which specific ads, or which specific campaigns actually drove the conversion, when so many other campaigns and ads are running concurrently? How do they separate an individual channel from the rest, or from their own organic or native demand?

Enter incrementality. At its heart, this type of analysis compares a control (or unexposed group) against a test (or exposed group), measuring the difference between the two groups. The test group is an apt name, we’re introducing a new variable into the equation, whether that variable is a campaign, a publisher, a creative, or something else, and our hypothesis is that this variable will result in additional conversions. To measure that hypothesis, we need a baseline for what “additional” means, and that’s where the holdout group comes in—the holdout group establishes the floor above which we wish to measure.

Real-World Example

Let’s use a non-marketing example to understand how this might work. An NBA basketball team has two players who both make 40% of their foul shots. The team wants to improve that percentage, so it decides to hire a shooting coach. It designs a test to evaluate that coach, and assigns him to only one of the two players. Both players are told to do everything else they had been doing, exactly as they had been doing it. For instance, they’re told to spend the same amount of time in the gym, keep a similar diet, and maintain their same weight.

2020–21 Season:
Player A: 40% of free throws ┃ Player B: 40% of free throws

After a year, the player who worked with the shooting coach makes 80% of his free throws the following season, while the player not assigned the shooting coach makes 50% of his free throws. In this test, we have a new baseline (or control/holdout) of 50%, and a new variable introduced in the shooting coach, resulting in a new test result of 80%. Both players improved, but one player improved more, so how do we measure that incremental improvement and attribute it to the shooting coach?

2021–22 Season:
Player A (control): 50% of free throws ┃ Player B with Coach (test): 80% of free throws

To measure the impact of the shooting coach, we want to answer two questions:

1. After introducing the shooting coach, how much more likely is Player B to make a foul shot? We call this incremental lift. To answer this question, we use the formula:

(test: 80% – holdout: 50%) / holdout: 50%
The result is 60%

A lift of 60% equates to a multiple of 1.6. In other words, Player B is now 1.6 times as likely to make his foul shots. So, we can accurately say that Player B, after working with the shooting coach, was 60% more, or 1.6 times as likely, to make a free throw than he was without working with the shooting coach.

60% more likely to make a free throw

2. Of all the free throws Player B now makes (that new 80% figure), what percentage can we attribute to the shooting coach? The formula for incrementality is:

(test: 80% – holdout: 50%) / test: 80%
This result is 37.5%

Leveraging incrementality, we can determine the shooting coach is directly responsible for 37.5% of the shots made after he began working with the player,—that 37.5% of the shots would have otherwise been missed if Player B and the coach never worked together.

Back to the Subject

Let’s bring this concept back to marketing. Let’s replace the NBA players with a brand running search and social media campaigns. It observes two subsets of its consumers who are exposed to search and social media campaigns and finds that both groups, who resemble each other behaviorally and demographically, convert at .1%, or, put another way, of those consumers observed who were exposed to search and/or social media, one in every thousand buy its product.

The brand has a hypothesis: if the brand introduces CTV advertising into its media mix, the conversion rate will increase. In this example, OTT/CTV campaigns are standing in for our shooting coach.

So the test begins: the brand starts running OTT/CTV campaigns on one of the two subsets of consumers. For both groups, it keeps social media and search spending and campaigns static. At the end of the campaign, the brand analyzes the data and discovers that the group who was exposed to OTT/CTV ads in addition to search and social media ads converts at .2% (test) while the group exposed only to search and social media ads converts at .15% (holdout)

Using incremental lift analysis, we can determine that OTT/CTV was responsible for a 33.33% incremental lift in purchases

(test: .2% – holdout: .15%) / holdout: .15% = 33.33%

and has an incrementality percentage of 25%

(test: .2% – holdout: .15%) / test: .2% = 25%

In other words: Those exposed to the brand’s OTT/CTV campaigns are 33.33% more—or 1.33 times as—likely to purchase than those not exposed. It means OTT/CTV campaigns are directly responsible for 25% of all conversions recorded.

Incrementality aims to answer the question, would this person have converted otherwise if not exposed to a particular campaign or campaign variable? Just like the shooting coach, the OTT/CTV campaign provided a needed boost in conversion rate. By using incrementality analysis instead of simply looking at the new conversion rate, or the new number of conversions, the brand is able to isolate and measure that boost and understand more clearly the impact of every new variable it introduces into its marketing mix.

Coming up in our next post, we’ll discuss how brands and agencies can create and leverage holdout (or control) groups. Can’t wait? Check out our full insights report or speak to a member of our team today.