論文読んだ「Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments 」

Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled ExperimentsMinyong Lee (Airbnb); Milan Shen (Airbnb) (KDD 2018)

Transcript

1. Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments Minyong Lee (Airbnb); Milan Shen (Airbnb) (KDD 2018)   @_stakaya 
   
2.  • !2AB)Music.1 -#43+( • "%'0AB)Music .15#43+( • &*  !2”$,43/”    2
3. • !2AB)Music.1 -#43+( • "%'0AB)Music .15#43+( • &*  !2”$,43/”     3
4. 
    • >,AB*)%5;4Bias <6
   %5?7 $  – Winner‘s Curse Bias=@ • 8+&(2 –#9'/0*:- .4  3Business Impact1  • AirbnbAB" ! 4
5.   •  3<AB$"&     
   )%* –@A/B$"&; #' → Gaussian 28 79B0.1 ;6!; / 2,NG –CA4  >5?: +- Wiki() =… 5
6. • n7"!#0  • i8("!#;*Metrics/& –5 /&),90  –… 62OK • <>631- –5'% 2 • +4&= : –31$ .2 +4&=  
     6
7. 
     • #& – b,# %( .+ – -*" !(  • ATotal True Effect – *" – A$)' $)' 7
8. 
     • AExpected Total True Effect • ATotal Estimated Effect – Total   – 
    
    8
9.   • 
    Expected Total True Effect 9
10. upward bias   10  
    i∈A   A  
       
11. upward bias   11 X_i > b_i ¥sigma_i I() 
       
12. upward bias   12 I(A) = 1– I(Not(A)) $  &
    X_i ≦ b_i ¥sigma_i'#I() ! 
    %"
13. upaward bias 
      13   
14. upaward bias   14  " 
    !  
    
15. upaward bias 
      15 
16. upaward bias 
      • # Bias • " •   –*$/(0%- &)  –!(  +'
     •  ., i=1, …,n  16
17. Selection bias with fixed p-values • p 
     Bias 
    •   
    Bias 17
18. 
    • Bias  –     –     18
19. 
    • Bias
    total true effect 19
20. "%  • Zhong and Prentice [25], Efron [7], and Xu, Craiu and Sun [23]
    A Bias  •   20    Gaussian '#&  !$  
21. 
    • Zhong and Prentice [25], Efron [7], and Xu, Craiu and Sun [23]
    A Bias 21
22. Bootstrap
     • Total true effect" 
     !  •   # 22
23. 
     • n=30 • • σ24$&5(shape=3, scale=1) .+ • ,3* 0  • … /6'(2%AB1,000) ! #"$-1 23
24. a
     24
    Figure 2
    
    
25. σ2
     25
    Figure 2
    
    
26. Code 26 library("ggplot2") theme_set(theme_grey(base_size=28)) #   # Zi|(−1.5 < Zi < 2) where Zi ∼ N(0.2,0.7^2) a <- qnorm(runif(10^5, pnorm(-1.5, mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7) ggplot(data.frame(value=z), aes(x = value, y = ..density..)) + geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) + xlim(c(-2.5, 2.5)) + theme_grey(base_size=28) #    
    Code   # σ^2 rform the inverse gamma distribution with shape parameter3 and scale param 1 sigma <- sqrt(1/rgamma(10^5, shape=3, scale=1)) ggplot(data.frame(value=sigma), aes(x = value, y = ..density..)) + geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) + xlim(c(0, 2)) + theme_grey(base_size=28)
27. v.s. 27
    Figure 3
    
    
28. v.s. 28
    Figure 4
    
    
29. v.s. 29
    Figure 5
    
    
30. Code 30 set.seed(71) size <- 30 a <- qnorm(runif(size, pnorm(-1.5, mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7) sigma <- sqrt(1/rgamma(size, shape=3, scale=1)) b <- qnorm(0.95, mean=0, sd=1) effect <- list() for(i in seq_len(10^3)){ x <- purrr::map_dbl(seq_len(size), ~ rnorm(1, mean=a[.x], sd=sigma[.x])) binary_win <- as.numeric(x/sigma > b) effect[[length(effect) + 1]] <- data.frame( # S_{A} sa=sum(x*binary_win), # T_{A} ta=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)), # T_{A, cond} tc=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)/(1 - pnorm((sigma * b - x)/sigma))*binary_win), # True effect te=sum(a*binary_win) ) } df <- dplyr::bind_rows(effect) # The total estimated effect v.s. The total true effect ggplot(df, aes(x=te, y=sa)) + geom_point() + geom_abline(slope=1, intercept=0) # The expected total true effect (conditional) v.s. The total true effect ggplot(df, aes(x=te, y=tc)) + geom_point() + geom_abline(slope=1, intercept=0) # The expected total true effect v.s. The total true effect ggplot(df, aes(x=te, y=ta)) + geom_point() + geom_abline(slope=1, intercept=0)
31.  
     • Market Dynamics team  31
    Figure 6 Holdout  
32. Experimentation Reporting Framework (ERF) At Airbnb • 100  Product Team  
     • 3,000 Metrics Monitoring • Winner‘s Curse Bias  $!0 32
    Figure 8
    # [17], [18]"
33. Aibnb
    • 53=LDαθG0 >  –'%( 7?5,$!*Dθ –8S7?=LDα • 9JRF2  '%(B /;  –MetricsNeutral *&" .)COA/B'% (TotalH@P'%(B/1 • Holdout#-+E< –MQ:53N4H@6AKI< 33
34. 
     • [7] Bradley Efron. 2011. TweedieâĂŹs formula and selection bias. J. Amer. Statist. Assoc. 106, 496 (Dec. 2011), 1602–1614. • [17] Will Moss. 2014. Experiment reporting framework. (May 2014). Retrieved February 16, 2017 from http://nerds.airbnb.com/experiment-reporting- framework • [18] Jan Overgoor. 2014. Experiments at Airbnb. (May 2014). Retrieved February 16, 2017 from http://nerds.airbnb.com/experiments-at-airbnb • [23] Lizhen Xu, Radu V Craiu, and Lei Sun. 2011. Bayesian methods to overcome the winner’s curse in genetic studies. The Annals of Applied Statistics (2011) • [25] Hua Zhong and Ross L Prentice. 2008. Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies. Biostatistics 9, 4 (Oct. 2008), 621–634. 34