$30 off During Our Annual Pro Sale. View Details »

論文読んだ「Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments 」

論文読んだ「Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments 」

Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled ExperimentsMinyong Lee (Airbnb); Milan Shen (Airbnb) (KDD 2018)

Shinichi Takayanagi

September 26, 2018
Tweet

More Decks by Shinichi Takayanagi

Other Decks in Technology

Transcript

  1. Winner’s Curse: Bias Estimation for Total Effects
    of Features in Online Controlled Experiments
    Minyong Lee (Airbnb); Milan Shen (Airbnb)
    (KDD 2018)

    @_stakaya

    View Slide


  2. • !2AB)Music.1
    -#43+(
    • "%'0AB)Music
    .15#43+(
    • &*
    !2”$,43/”

    2

    View Slide

  3. • !2AB)Music.1
    -#43+(
    • "%'0AB)Music
    .15#43+(
    • &*
    !2”$,43/”


    3

    View Slide


  4. • >,AB*)%5;4Bias
    <6%5?7 $
    –Winner‘s Curse Bias=@
    • 8+&(2
    –#9'/0*:- .4
    3Business Impact1

    • AirbnbAB" !
    4

    View Slide


  5. • 3<AB$"&

    )%*
    –@A/B$"&; #' → Gaussian
    28 79B0.1 ;6!; /
    2,NG
    –CA4
    >5?:+-
    Wiki()=…
    5

    View Slide

  6. • n7"!#0
    • i8("!#;*Metrics/&
    –5 /&),90
    –… 62OK
    • <>631-
    –5'%2
    • +4&= :
    –31$ .2+4&=



    6

    View Slide


  7. • #&
    – b,# %(
    .+
    – -*" !(
    • ATotal True Effect
    – *"
    – A$)'$)'
    7

    View Slide


  8. • AExpected Total True Effect
    • ATotal Estimated Effect
    – Total


    8

    View Slide




  9. Expected Total True
    Effect
    9

    View Slide

  10. upward bias
    10

    i∈A


    A


    View Slide

  11. upward bias

    11
    X_i > b_i ¥sigma_i
    I()



    View Slide

  12. upward bias
    12
    I(A) = 1– I(Not(A)) $

    &X_i ≦ b_i ¥sigma_i'#I()!
    %"

    View Slide

  13. upaward bias

    13


    View Slide

  14. upaward bias
    14
    "
    !

    View Slide

  15. upaward bias

    15

    View Slide

  16. upaward bias
    • #
    Bias
    • "

    –*$/(0%-&)
    –!( +'
    • ., i=1, …,n
    16

    View Slide

  17. Selection bias with fixed p-values
    • p

    Bias


    Bias
    17

    View Slide


  18. • Bias



    18

    View Slide


  19. • Biastotal true effect
    19

    View Slide

  20. "%

    • Zhong and Prentice [25], Efron [7], and Xu, Craiu
    and Sun [23]A Bias


    20



    Gaussian
    '#&
    !$

    View Slide


  21. • Zhong and Prentice [25], Efron [7], and Xu, Craiu
    and Sun [23]A
    Bias
    21

    View Slide

  22. Bootstrap
    • Total true effect"

    !
    • #
    22

    View Slide


  23. • n=30

    • σ24$&5(shape=3, scale=1).+
    • ,3* 0
    • …
    /6'(2%AB1,000) !
    #"$-1
    23

    View Slide

  24. a
    24
    Figure 2

    View Slide

  25. σ2
    25
    Figure 2

    View Slide

  26. Code
    26
    library("ggplot2")
    theme_set(theme_grey(base_size=28))
    #
    # Zi|(−1.5 < Zi < 2) where Zi ∼ N(0.2,0.7^2)
    a <- qnorm(runif(10^5, pnorm(-1.5, mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7)
    ggplot(data.frame(value=z), aes(x = value, y = ..density..)) +
    geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) +
    xlim(c(-2.5, 2.5)) +
    theme_grey(base_size=28)
    # Code

    # σ^2 rform the inverse gamma distribution with shape parameter3 and scale param 1
    sigma <- sqrt(1/rgamma(10^5, shape=3, scale=1))
    ggplot(data.frame(value=sigma), aes(x = value, y = ..density..)) +
    geom_density(aes(alpha = 0.2), color="#4CAF50", fill="#4CAF50", show.legend=FALSE) +
    xlim(c(0, 2)) +
    theme_grey(base_size=28)

    View Slide

  27. v.s.
    27
    Figure 3

    View Slide

  28. v.s.
    28
    Figure 4

    View Slide

  29. v.s.
    29
    Figure 5

    View Slide

  30. Code
    30
    set.seed(71)
    size <- 30
    a <- qnorm(runif(size, pnorm(-1.5, mean=0.2, sd=0.7), pnorm(2, mean=0.2, sd=0.7)), mean=0.2, sd=0.7)
    sigma <- sqrt(1/rgamma(size, shape=3, scale=1))
    b <- qnorm(0.95, mean=0, sd=1)
    effect <- list()
    for(i in seq_len(10^3)){
    x <- purrr::map_dbl(seq_len(size), ~ rnorm(1, mean=a[.x], sd=sigma[.x]))
    binary_win <- as.numeric(x/sigma > b)
    effect[[length(effect) + 1]] <- data.frame(
    # S_{A}
    sa=sum(x*binary_win),
    # T_{A}
    ta=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)),
    # T_{A, cond}
    tc=sum(x*binary_win) - sum(sigma * dnorm((sigma * b - x)/sigma)/(1 - pnorm((sigma * b - x)/sigma))*binary_win),
    # True effect
    te=sum(a*binary_win)
    )
    }
    df <- dplyr::bind_rows(effect)
    # The total estimated effect v.s. The total true effect
    ggplot(df, aes(x=te, y=sa)) + geom_point() + geom_abline(slope=1, intercept=0)
    # The expected total true effect (conditional) v.s. The total true effect
    ggplot(df, aes(x=te, y=tc)) + geom_point() + geom_abline(slope=1, intercept=0)
    # The expected total true effect v.s. The total true effect
    ggplot(df, aes(x=te, y=ta)) + geom_point() + geom_abline(slope=1, intercept=0)

    View Slide



  31. • Market Dynamics team
    31
    Figure 6
    Holdout


    View Slide

  32. Experimentation Reporting Framework (ERF) At Airbnb
    • 100
    Product
    Team

    • 3,000
    Metrics
    Monitoring
    • Winner‘s Curse Bias
    $!0
    32
    Figure 8
    #
    [17], [18]"

    View Slide

  33. Aibnb
    • 53=LDαθG0>

    –'%(7?5,$!*Dθ
    –8S7?=LDα
    • 9JRF2 '%(B
    /;

    –MetricsNeutral *&" .)COA/B'%
    (TotalH@P'%(B/1
    • Holdout#-+E<
    –MQ:53N4H@6AKI<
    33

    View Slide


  34. • [7] Bradley Efron. 2011. TweedieâĂŹs formula and selection bias. J. Amer.
    Statist. Assoc. 106, 496 (Dec. 2011), 1602–1614.
    • [17] Will Moss. 2014. Experiment reporting framework. (May 2014). Retrieved
    February 16, 2017 from http://nerds.airbnb.com/experiment-reporting-
    framework
    • [18] Jan Overgoor. 2014. Experiments at Airbnb. (May 2014). Retrieved
    February 16, 2017 from http://nerds.airbnb.com/experiments-at-airbnb
    • [23] Lizhen Xu, Radu V Craiu, and Lei Sun. 2011. Bayesian methods to
    overcome the winner’s curse in genetic studies. The Annals of Applied
    Statistics (2011)
    • [25] Hua Zhong and Ross L Prentice. 2008. Bias-reduced estimators and
    confidence intervals for odds ratios in genome-wide association studies.
    Biostatistics 9, 4 (Oct. 2008), 621–634.
    34

    View Slide