Slide 25
Slide 25 text
Powered by
Incrementality // Checklist for experiment design
3. An experiment should be designed with
clear methodology and objective in mind.
For example:
a. Conversion Lift based on geography (in
GAds UI) are optimal for calibration
b. Conversion Lift based on users (in GAds
UI) is the least comparable across MEM tools
c. GEO Experiments (open source code) use
1st-party data allowing for any comparison but
are resource-intensive
4. An experiment should have a comparable
scope.
In other terms, there should be parity between
the scope of the experiment and the scope of
corresponding attribution model or MMM.
1. An experiment should have a clear
hypothesis, based on evidence from:
a. Attribution or MMM results
b. Industry research
2. An experiment should have
comparables KPIs, so for instance it’s
important to know:
a. The amount of sales in attribution
depends on the attribution model, the
lookback window, etc
b. The amount of sales in MMM requires
2-3 years of historical sales
c. The amount of sales in incrementality
tests depends on the chosen methodology
(Conversion Lift, Geo Experiments, etc)