Search Incrementality Testing in Digital Advertising: Frameworks, Metrics, and Impact on Ad Revenue

In the fast-moving and ever-changing world of digital advertising, it’s easy to assume that each click, each impression or each conversion is simply the result of a perfectly placed ad at the right moment. Smart marketers are beginning to think about value for every dollar spent. Is that ad spend actually generating new customers, or would those sales have happened anyway?  Introducing search incrementality testing, the framework to helps marketers distinguish educated guesses from data-driven conclusions.

Understanding What Incrementality Really Means

Incrementality in digital advertising, at its most simplified level, relates to the additional value created by a marketing effort. It is the lift, the bump, the piece of revenue or engagement that can be attributed back to advertising. If there was no impact, a campaign could simply be a handy dandy cheerleader on the sidelines while customers independently make purchasing decisions.

Search incrementality testing is about uncovering true impact. In a universe where multi-touch attribution clouds your truth by handing out credits to every ad the customer sees, incrementality offers a clear picture of success. Instead of measuring every interaction, it asks a simple but powerful question, what changed because of the existence of this ad?

Testing the Waters: Frameworks for Search Incrementality

The frameworks for assessing incrementality are just as interesting as they are vital. Imagine an e-commerce brand wanting to discern whether their paid search ads on a platform like Google are resulting in new customers, or if they are simply cannibalizing the organic traffic they can get. The first step would be to build an experiment with a treatment and control group. The treatment group would be exposed to search ads while the control group would not.

One popular framework is geo testing. In this instance, the advertiser targets certain geo regions with the ads, while leaving other similar geo regions ad free. If the ads are running in New York, but nothing has run in Pennsylvania or Massachusetts, then any difference in sales could potentially be attributed to the ads. Care must be taken that the different regions are behaving similarly prior to the test, or this could skew the results.

A second method is time based testing. In this case, the advertiser runs search ads for a certain timeframe - two weeks for example - and then we compare all the data prior to and from the campaign. This approach is straightforward, but takes a careful process to eliminate other variables. For example, an advertiser may lose sales for reasons other than the advertising campaign - say both negative seasonality and the campaign.

When it is easier to target on an individual level, holdout testing is a standard tactic. Users are randomly assigned into groups in which one group sees ads and the other does not. This random assignment permits a comparison to be made, and it is particularly effective in online settings, where there are rich user-level data. This last statement is applicable to synthetic controls too.

Synthetic controls use a different variation of a control group. Instead of a real control group, the advertiser develops a prediction using historical data to determine what the performance of the treatment group would have been in the absence of advertising. Once again, this involves modeling and back-testing, but gives an answer to the case where a good control group in the real world is just too difficult to find.

Calculating Lift and Making Sense of the Numbers

Once the framework is established, and the campaign is over, you can now turn your attention to understanding the results. The most valuable metric is lift, which is the change in performance between treatment and control. In instances where the treatment group had a sales or conversion increase, the lift gives detail on that increase.
Lift is easy to calculate. Lift is simply the treatment group result less the control result. In this case, if the treatment group saw sales of 12,000 dollars and the actual, or predicted, baseline was 11,250 dollars, then the lift is 750 dollars.  The lift can then be applied towards measuring incrementality, which expresses causal effects as a percentage of total sales.

It is important to note that different frameworks provide information types. A geo test is likely to reveal geo differences, while user-level testing may expose differences in behaviors. The point, however, is the same: trying to detect advertising's hidden effect on outcomes.

Budget Decisions and the Revenue Ripple Effect

Incrementality testing has become such a big area of focus for one reason: it can actually help to change budgets permanently. Figuring out which ads delivered real revenue vs which ads only delivered growing spend can shift marketing budgets dramatically in a nights time.

Suppose a brand runs search ads for their product targeting specific keywords. The multi-touch attribution could indicate that these ads are a large contributing factor to sales, but an incrementality test can show that many of those sales would have occurred regardless of the ad spend, especially for loyal customers or users already intent on a purchase.

This distinction between attributed sales and incremental sales is extremely important. Incrementality provides brands with the ability to move budget from the performing ads and spend in a channel that drives actual, measured growth. It also helps ensure a brand does not build excessive investment in channels that are performing at a high level, but are really not new customer acquisition channels.

Moreover, the test results can reveal unexpected insights, like ads from a platform like TikTok or YouTube seemed ineffective in attribution reports, but when they tested for incrementality the ads exceeded expectations, and were bringing in new demand, not just repeat customers!

Challenges and the Path to Accurate Results

As a rule, doing incrementality tests is difficult because of a multitude of factors from outside your control that can create uncertain results such as competing timelines, holidays, weather events and even competitor activity, to name just a few. If you take the results from a test without consideration of these external factors, your test results may be misleading.

Ad fatigue is another factor to consider, especially in the case of a search ad, if the consumer sees the same search ad repeatedly, they will become blind to it over time. Depending on the creative approach to the ad (i.e. written content), a test may understate the performance of the campaign. Frequency is important to manage, just like rotating a creative to avoid fatigue.

Lastly, overlapping campaigns is an issue we can't ignore. When we run concurrent search and display campaigns, we may be unclear where the actual results are stemming from. Favoring sequential testing, with clear timelines, helps create focus and clarity around campaigns for results.

Regardless of the issues previously noted, having a tunnel vision approach, with a strongly constructed test plan, can help alleviate some, if not all, of the aforementioned issues. For every test, you are going to want to pay attention to the test duration, avoiding running a test in conjunction with any major promotional periods, and providing at least some segmentation in terms of audience behavior or region. The goal is to remove as much interference as possible to isolate the effect of the ads themselves.

Turning Insights into Action

The power of search incrementality testing lies in making more intelligent decisions. Rather than making assumptions or relying on overstated metrics, marketers can have confidence in what is working . If a campaign is driving high incrementality, a brand knows it is time to scale. If it is not, smart marketers will pivot, make adjustments and/or reallocate dollars.

Furthermore, the discovery found in one test can lead to more broadly informed marketing strategies. Brands can refine the keyword bidding, change the creative, and even update product positioning based on what the data shows.

In summary, search incrementality testing is not just another one of the marketing buzzwords bandied about. It is a fundamentally logical and experimental process that enables more efficient spending and more effective advertising. In a climate where every dollar spent in marketing counts, it is the north star that advocates for true value.

No Techcircle journalist was involved in the creation/production of this content.

Saurabh Mittal