論文読んだ「Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments 」

Transcript

1. Winner’s Curse: Bias Estimation for Total Effects of Features in

   Online Controlled Experiments Minyong Lee (Airbnb); Milan Shen (Airbnb) (KDD 2018)   @_stakaya 
   

2.  • !2AB)Music.1 -#43+( • "%'0AB)Music .15#43+( • &* 

   !2”$,43/”    2

3. • !2AB)Music.1 -#43+( • "%'0AB)Music .15#43+( • &*  !2”$,43/”

       3

4. 
    • >,AB*)%5;4Bias <6
   %5?7 $  – Winner‘s Curse Bias=@

   • 8+&(2 –#9'/0*:- .4  3Business Impact1  • AirbnbAB" ! 4

5.   •  3<AB$"&     
   )%*

   –@A/B$"&; #' → Gaussian 28 79B0.1 ;6!; / 2,NG –CA4  >5?: +- Wiki() =… 5

6. • n7"!#0  • i8("!#;*Metrics/& –5 /&),90  –… 62OK

   • <>631- –5'% 2 • +4&= : –31$ .2 +4&=  
     6

7. 
     • #& – b,# %( .+ – -*"

   !(  • ATotal True Effect – *" – A$)' $)' 7

8. 
     • AExpected Total True Effect • ATotal Estimated

   Effect – Total   – 
    
    8

9.   • 
    Expected Total True Effect 9

10. upward bias   10  
    i∈A  

    A  
       

11. upward bias   11 X_i > b_i ¥sigma_i I()

    
       

12. upward bias   12 I(A) = 1– I(Not(A)) $

     &
    X_i ≦ b_i ¥sigma_i'#I() ! 
    %"

13. upaward bias 
      13   

14. upaward bias   14  " 
    ! 

    
    

15. upaward bias 
      15 

16. upaward bias 
      • # Bias • " •

      –*$/(0%- &)  –!(  +'
     •  ., i=1, …,n  16

17. Selection bias with fixed p-values • p 
     Bias 
    

    •   
    Bias 17

18. 
    • Bias  –     –

        18

19. 
    • Bias
    total true effect 19

20. "%  • Zhong and Prentice [25], Efron [7], and

    Xu, Craiu and Sun [23]
    A Bias  •   20    Gaussian '#&  !$  

21. 
    • Zhong and Prentice [25], Efron [7], and Xu,

    Craiu and Sun [23]
    A Bias 21

22. Bootstrap
     • Total true effect" 
     !  •

      # 22

23. 
     • n=30 • • σ24$&5(shape=3, scale=1) .+ • ,3*

    0  • … /6'(2%AB1,000) ! #"$-1 23

24. a
     24
    Figure 2
    
    

25. σ2
     25
    Figure 2
    
    

26. Code 26 library("ggplot2") theme_set(theme_grey(base_size=28)) #   # Zi|(−1.5 <

    Zi < 2) where Zi ∼ N(0.2,0.7^2) a <- qnorm(runif(10^5, pnorm(-1.5, mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7) ggplot(data.frame(value=z), aes(x = value, y = ..density..)) + geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) + xlim(c(-2.5, 2.5)) + theme_grey(base_size=28) #    
    Code   # σ^2 rform the inverse gamma distribution with shape parameter3 and scale param 1 sigma <- sqrt(1/rgamma(10^5, shape=3, scale=1)) ggplot(data.frame(value=sigma), aes(x = value, y = ..density..)) + geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) + xlim(c(0, 2)) + theme_grey(base_size=28)

27. v.s. 27
    Figure 3
    
    

28. v.s. 28
    Figure 4
    
    

29. v.s. 29
    Figure 5
    
    

30. Code 30 set.seed(71) size <- 30 a <- qnorm(runif(size, pnorm(-1.5,

    mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7) sigma <- sqrt(1/rgamma(size, shape=3, scale=1)) b <- qnorm(0.95, mean=0, sd=1) effect <- list() for(i in seq_len(10^3)){ x <- purrr::map_dbl(seq_len(size), ~ rnorm(1, mean=a[.x], sd=sigma[.x])) binary_win <- as.numeric(x/sigma > b) effect[[length(effect) + 1]] <- data.frame( # S_{A} sa=sum(x*binary_win), # T_{A} ta=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)), # T_{A, cond} tc=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)/(1 - pnorm((sigma * b - x)/sigma))*binary_win), # True effect te=sum(a*binary_win) ) } df <- dplyr::bind_rows(effect) # The total estimated effect v.s. The total true effect ggplot(df, aes(x=te, y=sa)) + geom_point() + geom_abline(slope=1, intercept=0) # The expected total true effect (conditional) v.s. The total true effect ggplot(df, aes(x=te, y=tc)) + geom_point() + geom_abline(slope=1, intercept=0) # The expected total true effect v.s. The total true effect ggplot(df, aes(x=te, y=ta)) + geom_point() + geom_abline(slope=1, intercept=0)

31.  
     • Market Dynamics team  31
    Figure 6

    Holdout  

32. Experimentation Reporting Framework (ERF) At Airbnb • 100  Product

    Team  
     • 3,000 Metrics Monitoring • Winner‘s Curse Bias  $!0 32
    Figure 8
    # [17], [18]"

33. Aibnb
    • 53=LDαθG0 >  –'%( 7?5,$!*Dθ –8S7?=LDα • 9JRF2

     '%(B /;  –MetricsNeutral *&" .)COA/B'% (TotalH@P'%(B/1 • Holdout#-+E< –MQ:53N4H@6AKI< 33

34. 
     • [7] Bradley Efron. 2011. TweedieâĂŹs formula and selection

    bias. J. Amer. Statist. Assoc. 106, 496 (Dec. 2011), 1602–1614. • [17] Will Moss. 2014. Experiment reporting framework. (May 2014). Retrieved February 16, 2017 from http://nerds.airbnb.com/experiment-reporting- framework • [18] Jan Overgoor. 2014. Experiments at Airbnb. (May 2014). Retrieved February 16, 2017 from http://nerds.airbnb.com/experiments-at-airbnb • [23] Lizhen Xu, Radu V Craiu, and Lei Sun. 2011. Bayesian methods to overcome the winner’s curse in genetic studies. The Annals of Applied Statistics (2011) • [25] Hua Zhong and Ross L Prentice. 2008. Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies. Biostatistics 9, 4 (Oct. 2008), 621–634. 34