A/B Testing at LINE NEWS

LINE株式会社 Jay Hori

LINE NEWSではデータに基づく意思決定のためにA/Bテストを実施しています。A/Bテストを効率的・効果的に実施していくための工夫、また、複雑なA/Bテストを含むリリースを安全に継続するための工夫を紹介します。キーワードはfeature flags、remote config、カナリアリリースです。

イベントURL: https://uit.connpass.com/event/291443/

Transcript

1. Jay Hori A/B Testing at LINE NEWS σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
2. Software Engineer Jay Hori LINEגࣜձࣾ ϑϩϯτΤϯυ։ൃηϯλʔ/UIT Hartwick College (New York) B.A. Cognitive Science ݴޠɾϑϨʔϜϫʔΫ TypeScript, JavaScript, React, Python
3. A/B Testing at LINE NEWS σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ 1. LINE NEWSʹ͍ͭͯ 2. σϓϩΠͱϦϦʔεͷ෼཭ 3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε 4. Feature Con fi gurationͷ׆༻ 5. ·ͱΊ
4. A/B Testing at LINE NEWS σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ 1. LINE NEWSʹ͍ͭͯ 2. σϓϩΠͱϦϦʔεͷ෼཭ 3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε 4. Feature Con fi gurationͷ׆༻ 5. ·ͱΊ
5. 7,700ສਓ MAU (2021೥8݄࣌఺)
6. • σʔλʹجͮ͘ҙࢥܾఆ • සൟͳA/Bςετ͕ඞཁ • A/Bςετӡ༻ͷੜ࢈ੑͱޮ཰ɺ ͦͯ҆͠શͳϦϦʔε͕ඞਢ LINE NEWS
7. σϓϩΠͱϦϦʔεͷ෼཭ Decoupling deployment from releases σϓϩΠ ιʔείʔυΛ ϓϩμΫγϣϯ؀ڥ΁ σϓϩΠ͢Δ ϦϦʔε ػೳΛϢʔβʔ΁ ϦϦʔε͢Δ
8. A/B Testing at LINE NEWS Taking full control of experimental features 1. LINE NEWSʹ͍ͭͯ 2. σϓϩΠͱϦϦʔεͷ෼཭ 3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε 4. Feature Con fi gurationͷ׆༻ 5. ·ͱΊ
10. Feature Flags ػೳΛΦϯΦϑ͢Δ O ff isEnabled: false On isEnabled: true
11. A/Bςετ Control Treatment ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
12. A/Bςετ ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ Control Treatment 1 Treatment 2
13. A/Bςετ ΧϧʔηϧͱλΠϜϥΠϯΛൺֱ Carousel 1% Timeline 1%
14. Remote Config τϦʔτϝϯτάϧʔϓͷόϦΤʔγϣϯ΁ύϥϝʔλʔઃఆ Control isEnabled: false Treatment 1 isEnabled: true numberOfArticlesToLoad: 30 Treatment 2 isEnabled: true numberOfArticlesToLoad: 10
15. ΧφϦΞϦϦʔε খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ Control isEnabled: false 1%
16. ΧφϦΞϦϦʔε খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ Control isEnabled: false 50%
17. ΧφϦΞϦϦʔε খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ Control isEnabled: false 100%
18. A/B Testing at LINE NEWS Taking full control of experimental features 1. LINE NEWSʹ͍ͭͯ 2. σϓϩΠͱϦϦʔεͷ෼཭ 3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε 4. Feature Con fi gurationͷ׆༻ 5. ·ͱΊ
19. Feature Configuration Feature Flags A/B Tests Remote Config Canary Releases =
20. ϦϞʔτ؅ཧ • JSONϑΥʔϚοτ • ϦϞʔτϦϙδτϦͰ؅ཧ • Con fi gurationαʔόʔ (Central Dogma) ʹอଘ͞ΕΔ Feature Configuration
21. JSONྫ • Feature fl ag • isEnabled • Remote con fi g • numberOfArticlesToLoadInitially • title Feature Configuration
25. A/B Testing at LINE NEWS Taking full control of experimental features 1. LINE NEWSʹ͍ͭͯ 2. σϓϩΠͱϦϦʔεͷ෼཭ 3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε 4. Feature Con fi gurationͷ׆༻ 5. ·ͱΊ
26. ·ͱΊ 1. େن໛ϓϩμΫτͰ͸A/Bςετͷੜ࢈ੑͱޮ཰͕ॏཁ 2. σϓϩΠͱϦϦʔεΛ෼཭͢ΔػߏΛ࡞Δ 3. Feature Con fi gurationͰޮ཰Խ 4. ΧφϦΞϦϦʔεͰ҆શʹϦϦʔε͢Δ