A/B Testing at LINE NEWS

LINE株式会社 Jay Hori

LINE NEWSではデータに基づく意思決定のためにA/Bテストを実施しています。A/Bテストを効率的・効果的に実施していくための工夫、また、複雑なA/Bテストを含むリリースを安全に継続するための工夫を紹介します。キーワードはfeature flags、remote config、カナリアリリースです。

イベントURL: https://uit.connpass.com/event/291443/

Transcript

1. Jay Hori
   A/B Testing at LINE NEWS
   σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
   

   View Slide

2. Software Engineer
   Jay Hori
   LINEגࣜձࣾ
   ϑϩϯτΤϯυ։ൃηϯλʔ/UIT
   Hartwick College (New York)
   B.A. Cognitive Science
   ݴޠɾϑϨʔϜϫʔΫ
   TypeScript, JavaScript, React, Python
   

   View Slide

3. A/B Testing at LINE NEWS
   σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
   1. LINE NEWSʹ͍ͭͯ
   
   2. σϓϩΠͱϦϦʔεͷ෼཭
   
   3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
   
   4. Feature Con
   fi
   gurationͷ׆༻
   
   5. ·ͱΊ
   

   View Slide

4. A/B Testing at LINE NEWS
   σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
   1. LINE NEWSʹ͍ͭͯ
   2. σϓϩΠͱϦϦʔεͷ෼཭
   
   3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
   
   4. Feature Con
   fi
   gurationͷ׆༻
   
   5. ·ͱΊ
   

   View Slide

5. 7,700ສਓ
   MAU (2021೥8݄࣌఺)
   

   View Slide

6. • σʔλʹجͮ͘ҙࢥܾఆ
   
   • සൟͳA/Bςετ͕ඞཁ
   
   • A/Bςετӡ༻ͷੜ࢈ੑͱޮ཰ɺ
   ͦͯ҆͠શͳϦϦʔε͕ඞਢ
   LINE NEWS
   

   View Slide

7. σϓϩΠͱϦϦʔεͷ෼཭
   Decoupling deployment from releases
   σϓϩΠ
   ιʔείʔυΛ
   ϓϩμΫγϣϯ؀ڥ΁
   σϓϩΠ͢Δ
   ϦϦʔε
   ػೳΛϢʔβʔ΁
   ϦϦʔε͢Δ
   

   View Slide

8. A/B Testing at LINE NEWS
   Taking full control of experimental features
   1. LINE NEWSʹ͍ͭͯ
   
   2. σϓϩΠͱϦϦʔεͷ෼཭
   
   3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
   4. Feature Con
   fi
   gurationͷ׆༻
   
   5. ·ͱΊ
   

   View Slide

9. View Slide

10. Feature Flags
    ػೳΛΦϯΦϑ͢Δ
    O
    ff
    isEnabled: false
    On
    isEnabled: true
    

    View Slide

11. A/Bςετ
    Control Treatment
    ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
    

    View Slide

12. A/Bςετ
    ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
    Control Treatment 1 Treatment 2
    

    View Slide

13. A/Bςετ
    ΧϧʔηϧͱλΠϜϥΠϯΛൺֱ
    Carousel
    1%
    Timeline
    1%
    

    View Slide

14. Remote Config
    τϦʔτϝϯτάϧʔϓͷόϦΤʔγϣϯ΁ύϥϝʔλʔઃఆ
    Control
    
    isEnabled: false
    Treatment 1
    
    isEnabled: true
    
    numberOfArticlesToLoad: 30
    Treatment 2
    
    isEnabled: true
    
    numberOfArticlesToLoad: 10
    

    View Slide

15. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control
    
    isEnabled: false
    1%
    

    View Slide

16. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control
    
    isEnabled: false
    50%
    

    View Slide

17. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control
    
    isEnabled: false
    100%
    

    View Slide

18. A/B Testing at LINE NEWS
    Taking full control of experimental features
    1. LINE NEWSʹ͍ͭͯ
    
    2. σϓϩΠͱϦϦʔεͷ෼཭
    
    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
    
    4. Feature Con
    fi
    gurationͷ׆༻
    5. ·ͱΊ
    

    View Slide

19. Feature
    Configuration
    Feature Flags
    
    
    A/B Tests
    
    
    Remote Config
    
    
    Canary Releases
    =
    

    View Slide

20. ϦϞʔτ؅ཧ
    • JSONϑΥʔϚοτ
    
    • ϦϞʔτϦϙδτϦͰ؅ཧ
    
    • Con
    fi
    gurationαʔόʔ (Central Dogma) ʹอଘ͞ΕΔ
    Feature Configuration
    

    View Slide

21. JSONྫ
    • Feature
    fl
    ag
    
    • isEnabled
    
    • Remote con
    fi
    g
    
    • numberOfArticlesToLoadInitially
    
    • title
    Feature Configuration
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. A/B Testing at LINE NEWS
    Taking full control of experimental features
    1. LINE NEWSʹ͍ͭͯ
    
    2. σϓϩΠͱϦϦʔεͷ෼཭
    
    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
    
    4. Feature Con
    fi
    gurationͷ׆༻
    5. ·ͱΊ
    

    View Slide

26. ·ͱΊ
    1. େن໛ϓϩμΫτͰ͸A/Bςετͷੜ࢈ੑͱޮ཰͕ॏཁ
    
    2. σϓϩΠͱϦϦʔεΛ෼཭͢ΔػߏΛ࡞Δ
    
    3. Feature Con
    fi
    gurationͰޮ཰Խ
    
    4. ΧφϦΞϦϦʔεͰ҆શʹϦϦʔε͢Δ
    

    View Slide