Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing at LINE NEWS

A/B Testing at LINE NEWS

LINE株式会社 Jay Hori

LINE NEWSではデータに基づく意思決定のためにA/Bテストを実施しています。A/Bテストを効率的・効果的に実施していくための工夫、また、複雑なA/Bテストを含むリリースを安全に継続するための工夫を紹介します。キーワードはfeature flags、remote config、カナリアリリースです。

イベントURL: https://uit.connpass.com/event/291443/

LINE Developers
PRO

September 11, 2023
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. Jay Hori
    A/B Testing at LINE NEWS
    σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ

    View Slide

  2. Software Engineer
    Jay Hori
    LINEגࣜձࣾ
    ϑϩϯτΤϯυ։ൃηϯλʔ/UIT
    Hartwick College (New York)
    B.A. Cognitive Science
    ݴޠɾϑϨʔϜϫʔΫ
    TypeScript, JavaScript, React, Python

    View Slide

  3. A/B Testing at LINE NEWS
    σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
    1. LINE NEWSʹ͍ͭͯ

    2. σϓϩΠͱϦϦʔεͷ෼཭

    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε

    4. Feature Con
    fi
    gurationͷ׆༻

    5. ·ͱΊ

    View Slide

  4. A/B Testing at LINE NEWS
    σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑Διϑτ΢ΣΞ։ൃઓུ
    1. LINE NEWSʹ͍ͭͯ
    2. σϓϩΠͱϦϦʔεͷ෼཭

    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε

    4. Feature Con
    fi
    gurationͷ׆༻

    5. ·ͱΊ

    View Slide

  5. 7,700ສਓ
    MAU (2021೥8݄࣌఺)

    View Slide

  6. • σʔλʹجͮ͘ҙࢥܾఆ

    • සൟͳA/Bςετ͕ඞཁ

    • A/Bςετӡ༻ͷੜ࢈ੑͱޮ཰ɺ
    ͦͯ҆͠શͳϦϦʔε͕ඞਢ
    LINE NEWS

    View Slide

  7. σϓϩΠͱϦϦʔεͷ෼཭
    Decoupling deployment from releases
    σϓϩΠ
    ιʔείʔυΛ
    ϓϩμΫγϣϯ؀ڥ΁
    σϓϩΠ͢Δ
    ϦϦʔε
    ػೳΛϢʔβʔ΁
    ϦϦʔε͢Δ

    View Slide

  8. A/B Testing at LINE NEWS
    Taking full control of experimental features
    1. LINE NEWSʹ͍ͭͯ

    2. σϓϩΠͱϦϦʔεͷ෼཭

    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
    4. Feature Con
    fi
    gurationͷ׆༻

    5. ·ͱΊ

    View Slide

  9. View Slide

  10. Feature Flags
    ػೳΛΦϯΦϑ͢Δ
    O
    ff
    isEnabled: false
    On
    isEnabled: true

    View Slide

  11. A/Bςετ
    Control Treatment
    ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ

    View Slide

  12. A/Bςετ
    ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
    Control Treatment 1 Treatment 2

    View Slide

  13. A/Bςετ
    ΧϧʔηϧͱλΠϜϥΠϯΛൺֱ
    Carousel
    1%
    Timeline
    1%

    View Slide

  14. Remote Config
    τϦʔτϝϯτάϧʔϓͷόϦΤʔγϣϯ΁ύϥϝʔλʔઃఆ
    Control

    isEnabled: false
    Treatment 1

    isEnabled: true

    numberOfArticlesToLoad: 30
    Treatment 2

    isEnabled: true

    numberOfArticlesToLoad: 10

    View Slide

  15. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control

    isEnabled: false
    1%

    View Slide

  16. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control

    isEnabled: false
    50%

    View Slide

  17. ΧφϦΞϦϦʔε
    খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞ΢τ͢Δ
    Control

    isEnabled: false
    100%

    View Slide

  18. A/B Testing at LINE NEWS
    Taking full control of experimental features
    1. LINE NEWSʹ͍ͭͯ

    2. σϓϩΠͱϦϦʔεͷ෼཭

    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε

    4. Feature Con
    fi
    gurationͷ׆༻
    5. ·ͱΊ

    View Slide

  19. Feature
    Configuration
    Feature Flags


    A/B Tests


    Remote Config


    Canary Releases
    =

    View Slide

  20. ϦϞʔτ؅ཧ
    • JSONϑΥʔϚοτ

    • ϦϞʔτϦϙδτϦͰ؅ཧ

    • Con
    fi
    gurationαʔόʔ (Central Dogma) ʹอଘ͞ΕΔ
    Feature Configuration

    View Slide

  21. JSONྫ
    • Feature
    fl
    ag

    • isEnabled

    • Remote con
    fi
    g

    • numberOfArticlesToLoadInitially

    • title
    Feature Configuration

    View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. A/B Testing at LINE NEWS
    Taking full control of experimental features
    1. LINE NEWSʹ͍ͭͯ

    2. σϓϩΠͱϦϦʔεͷ෼཭

    3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε

    4. Feature Con
    fi
    gurationͷ׆༻
    5. ·ͱΊ

    View Slide

  26. ·ͱΊ
    1. େن໛ϓϩμΫτͰ͸A/Bςετͷੜ࢈ੑͱޮ཰͕ॏཁ

    2. σϓϩΠͱϦϦʔεΛ෼཭͢ΔػߏΛ࡞Δ

    3. Feature Con
    fi
    gurationͰޮ཰Խ

    4. ΧφϦΞϦϦʔεͰ҆શʹϦϦʔε͢Δ

    View Slide