Slide 42
Slide 42 text
Example
5% of visitors in experimental population purchase
average purchase = $100, standard deviation = $40.
OEC: revenue
average spending = 0.05 $ $100 + 0.95 $ $0 = $5.00
Δ = sensitivity/std dev = 0.1 $ $+
$,-
= 0.0125
41
based on Kohavi et al., Controlled
experiments on the web: survey
and practical guide. 2008.
How many users do we need for experiment to detect 10% change in revenue?
/ =
16
Δ1
= 102,400 / =
16
(0.01 $ $5/$40)1
= 10.24M
Detect 10% change with 8 = 0.05, 9 = 0.20 Detect 1% change with 8 = 0.05, 9 = 0.20