Level Up your Performance Marketing Incrementality Rico Stuijt Data Expo 11-09-2024

Who is talking to you?

● Living in Alkmaar ● 10+ years XP in online marketing ● In my free time I enjoy spending time with family/friends, travel, sport and games ● With the company for 8+ years ● Favourite thing about working at JET is the chance to make impact ● Favourite takeaway is Chinese food ● Performance Measurement Team ● 11 people all located in Amsterdam ● Taking care of WEB/APP tracking, Attribution/Incrementality, Data structure, Analysis and reporting ● Main goal: Making sure that PMKT has maximum impact on total business. ● Main task: Setting up an incrementality measurement infrastructure so that PMKT teams can make strategy/budget decisions based on incrementality insights. About me Rico Stuijt - Principal Performance Measurement My Team Who am I? Me & JET My Role Job & Reward Framework

Global reach 19 countries We are a leading global online delivery marketplace Offering choice 731.000+ connected partners, from restaurant to retail Diversified customer base 81 million active customers A fantastic team 13K+ employees from over 100 different nationalities, operating from 20+ offices around the globe About the company JustEat Takeaway.com

Logistics Connected Partners Consumers Corporate Back Office Platform How it works

Why you need to care about incrementality

Attribution ≠ Incrementality Imagine a hypothetical case of two random users that are in-market to order food on JustEat: Not exposed to advertising from JustEat Orders a meal on JustEat Exposed to advertising from JustEat Orders a meal on JustEat Attribution Incrementality It would assign the order to the advertisement campaign since Henk placed an order after being exposed to the ad. It would not assign the order to the advertisement campaign, as the campaign had no incremental effect. There is no difference between the behaviour of the exposed (Henk) and the control (Anja) group. Anja Henk

Other advertisers’ ads Conversions Control Target 1.0 M 1.1M Random Assignment Attribution 1.1 M orders Incrementality 100K orders Spend = €300.000,- CPO = €0.28 CPiO = €3.00 cost per incremental order Why Incrementality?

So what does it mean to improve on incrementality Example SEA BRAND SEA NON BRAND SEA TOTAL Attribution CPO 2,- = 500 orders CPO 5,- = 200 orders 700 orders Incrementality CPiO 30,- = 33 orders CPiO 10,- = 100 orders 133 orders Budget 1000,- 1000,- 2000,- Example SEA BRAND SEA NON BRAND SEA TOTAL Attribution CPO 2,- = 250 orders CPO 5,- = 300 orders 550 orders Incrementality CPiO 30,- = 16 orders CPiO 10,- = 150 orders 166 orders Budget 500,- 1500,- 2000,- CPO 2,86 = CPiO 15,04 = CPO 3,64 = CPiO 12,05 = Optimise based on the incrementality insights +33 absolute orders on total business level +25% channel effectiveness

So what can it mean to improve on non calibrated attribution Example SEA BRAND SEA NON BRAND SEA TOTAL Attribution CPO 2,- = 500 orders CPO 5,- = 200 orders 700 orders Incrementality CPiO 30,- = 33 orders CPiO 10,- = 100 orders 133 orders Budget 1000,- 1000,- 2000,- Example SEA BRAND SEA NON BRAND SEA TOTAL Attribution CPO 2,- = 750 orders CPO 5,- = 100 orders 850 orders Incrementality CPiO 30,- = 50 orders CPiO 10,- = 50 orders 100 orders Budget 1500,- 500,- 2000,- CPO 2,86 = CPiO 15,04 = CPO 2,35 = CPiO 20,- = Optimise based on the attribution insights -33 absolute orders on total business level -25% channel effectiveness

So attribution can overvalue specific channels/efforts by a lot Attribution-only steering can harm your business Start incrementality measurement now! Recap why incrementality

Incrementality measurement methodologies and metrics

●Platform level test ●Before the auction (if any) users will be distributed in control and target groups Geo test Lift Test (Conversion/ Brand) Ghost bids Central control group Different methodologies Intent to treat ●Single or Multi-Platform test ●Holistic incrementality testing tool based on creating geo splits of test (exposed) and control geo’s (non-exposed). ●Platform level test ●Conversion Lift: Post auction cookie splitting showing our ad to X% of users and sending Y% the next ad in auction (control). ●Brand Lift: isolates two groups of users, presents a one-question survey or analyzes follow-up search activity PSA ●Platform level test ●Comparing exposed users of both groups ●Platform level test ●Similar as conversion lift but distribution will be made on entering the auction. ●Single or Multi-Platform test ●Intent to treat methodology but where control & target group are defined by advertiser and evenly distribution between groups is guaranteed.

●Platform level test ●Before the auction (if any) users will be distributed in control and target groups Geo test Lift Test (Conversion/ Brand) Ghost bids Central control group Different methodologies Intent to treat ●Single or Multi-Platform test ●Holistic incrementality testing tool based on creating geo splits of test (exposed) and control geo’s (non-exposed). ●Platform level test ●Conversion Lift: Post auction cookie splitting showing our ad to X% of users and sending Y% the next ad in auction (control). ●Brand Lift: isolates two groups of users, presents a one-question survey or analyzes follow-up search activity PSA ●Platform level test ●Comparing exposed users of both groups ●Platform level test ●Similar as conversion lift but distribution / analysis will be made on entering the auction. ●Single or Multi-Platform test ●Intent to treat methodology but where control & target group are defined by advertiser and evenly distribution between groups is guaranteed. To much noise / to expensive

●Platform level test ●Before the auction (if any) users will be distributed in control and target groups Geo test Lift Test (Conversion/ Brand) Ghost bids Central control group Different methodologies Intent to treat ●Single or Multi-Platform test ●Holistic incrementality testing tool based on creating geo splits of test (exposed) and control geo’s (non-exposed). ●Platform level test ●Conversion Lift: Post auction cookie splitting showing our ad to X% of users and sending Y% the next ad in auction (control). ●Brand Lift: isolates two groups of users, presents a one-question survey or analyzes follow-up search activity PSA ●Platform level test ●Comparing exposed users of both groups ●Platform level test ●Similar as conversion lift but distribution will be made on entering the auction. ●Single or Multi-Platform test ●Intent to treat methodology but where control & target group are defined by advertiser and evenly distribution between groups is guaranteed. To much noise / to expensive Still quite some noise / no uniform usages

Incremental orders CPiO Incremental NC CPiA Incremental … CPi… Incremental revenue / profit iROAS Cost Relative lift Significance GEO Lift GEO mismatch Conversion Lift Central Control Match rates Consent Consent (partly) Certain occasions APP share partly Certain occasions iOS extrapolation Non exposed target group = lower relative lift Important metrics you get back and the need for modeling Non exposed target group = lower relative lift

Relative lift can be misleading for PMKT Campaign A: 60% lift Campaign B: 30% lift & CPiO/CPiA of €30,- & CPiO/CPiA of €2,- Incremental orders CPiO Incremental NC CPiA Incremental … CPi… Incremental revenue / profit iROAS Cost Relative lift Significance CPiO Incremental Orders CPiA Incremental New Customers Significance 90% confidence Balance Metrics we care (less) about

Example: Campaign A: CPiA 4,- & CPiO 2,- Campaign B: CPiA 5,- & CPiO 1,- Budget is €1000,- for both Campaign A: 250 NC & 500 Orders Campaign B: 200 NC & 1000 Orders Understand the value of the New Customer NC = X orders in X years Might/Will differ per country/product/channel Example: 1 NC = avg 4 Orders in 1 year Campaign A: 1000 NC Orders & 500 Orders = 1500 Campaign B: 800 NC Orders & 1000 Orders = 1800 Example: 1 NC = avg 20 Orders in 1 year Campaign A: 5000 NC Orders & 500 Orders = 5500 Campaign B: 4000 NC Orders & 1000 Orders = 5000 Cost per Incremental Order Generation + NC Contribution Campaign A: €1000,- / 1500 = €0.67 Campaign B: €1000,- / 1800 = €0.56 Cost per Incremental Order Generation + NC Contribution Campaign A: €1000,- / 5500 = €0.18 Campaign B: €1000,- / 5000 = €0.20 Making the right decision - the CPiO & CPiA balance

Example: Campaign A: CPiA 4,- & CPiO 2,- Campaign B: CPiA 5,- & CPiO 1,- Budget is €1000,- for both Campaign A: 250 NC & 500 Orders Campaign B: 200 NC & 1000 Orders Understand the value of the New Customer NC = X orders in X years Might/Will differ per country/product/channel Example: 1 NC = avg 4 Orders in 1 year Campaign A: 1000 NC Orders & 500 Orders = 1500 Campaign B: 800 NC Orders & 1000 Orders = 1800 Example: 1 NC = avg 20 Orders in 1 year Campaign A: 5000 NC Orders & 500 Orders = 5500 Campaign B: 4000 NC Orders & 1000 Orders = 5000 Cost per Incremental Order Generation + NC Contribution Campaign A: €1000,- / 1500 = €0.67 Campaign B: €1000,- / 1800 = €0.56 Cost per Incremental Order Generation + NC Contribution Campaign A: €1000,- / 5500 = €0.18 Campaign B: €1000,- / 5000 = €0.2 Making the right decision - the CPiO & CPiA balance Think of your important lifetime components

Utilise the most appropriate methodology Understand to what level results need to be modeled Understand which metrics to use Unify result outcomes / understand lifetime components Change things to reach the optimal state Recap methodologies and metrics to use

Scalability A better way of working

Test incrementality ● power analysis ● setup and run ● results and interpretation ~ 2 months Adjust settings / budgets ~ 2 days Learning curve Get the platform algorithm up to speed ~ 3 weeks Incrementality testing is “time” consuming Test “one” thing Stability during a test is preferred Still two metrics CPO vs CPiO Conversion lift process Other notes

Accumulate incrementality results from different channels/campaigns during different seasonality and spend levels Generate multiple MTA models Make sure they are adjustable Select MTA model which correlates best with the incrementality results Use “safe” variable like CPO vs CPiO Calibrate MTA model “staged” with (new) incrementality results for the available channels Incrementality & MTA 1 2 3 4

Accumulate incrementality results from different channels/campaigns during different seasonality and spend levels Generate multiple MMM models Make sure they are adjustable Select MMM model which correlates best with the incrementality results Should roughly give similar results Calibrate MMM model further with (new) incrementality results in a staged manner Incrementality & MMM 1 2 3 4

Utilize the MMM model to help calibrating the MTA model for non incrementality channels MTA & MMM 1

Incrementality MTA MMM Triangle of optimisation Validation & calibration Validation & calibration “Leftover” calibration

1 metrics to look and steer on ● Calibrated MMM upper management ● Calibrated MTA lower management Will improve total business performance Ongoing optimisation Recap Scalability Use incrementality results for validating and calibrating your MTA & MMM models Less incrementality tests necessary

Thank you

Questions?