$30 off During Our Annual Pro Sale. View Details »

Mutual Collaboration between Data Analysis and Machine Learning Development at LINE: Continuous Improvement in Sticker Recommendation Engine

Mutual Collaboration between Data Analysis and Machine Learning Development at LINE: Continuous Improvement in Sticker Recommendation Engine

LINE DEVDAY 2021
PRO

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. None
  2. Self Introduction - Joined LINE Corporation as a Data Scientist

    in February 2019 - Working on data analysis for cross-service projects such as LINE POINT CLUB - Yu Otsuka
  3. Agenda - Data Science & Machine Learning department - A

    case of DS & ML collaboration - Continuous improvement in sticker recommendation engine
  4. - DS - Estimate business impact - Perform statistical analysis

    - ML - Provide tailor-made ML engine - Develop general ML platform DS & ML Department Data Science department Machine Learning department ML privacy team LPML department Data Science center
  5. Continuous improvement of ML System Plan ML design Check A/B

    test Action Find improvement Do ML integration .- .- %4 .- %4
  6. Continuous improvement in sticker recommendation engine A case of DS

    & ML collaboration
  7. Stickers For You - Personalized recommendation

  8. Stickers For You % purchase (within the top page) 40%

    # clicks per day 1.2M # packages 11M Purchase: As of September 2021 Click: As of August 2021 Package: As of April 2021
  9. Role of “Stickers For You” is changed Smart Channel -

    Many sticker touchpoints out of sticker shop - Smart Channel - Home Tab - LINE Official Account - and more … Home Tab
  10. Role of “Stickers For You” is changed 89% Paid users

    via both Stickers For You & others Paid users via Stickers For You only - Paid users via Stickers For You also use other touchpoints Period: FY2020
  11. Role of “Stickers For You” is changed As-Is - Offering

    stickers that all you need - Require simple recommendation To-Be - Offering stickers that enhances serendipity - Require highly diverse recommendation - Re-definition of Stickers For You
  12. As-Is Recommendation Model Naive Bayes-based collaborative filtering Business Requirement ›Simple

    recommendation Model Summary Pros ›Small number of parameters ›Very irrelevant items won’t be recommended Cons ›Poor exploration leads to no fresh items ›Poor generalization leads to cold start problem !(item' |f* , f, , . . . , f. ) ∝ !(item' )∏ 2 !(f2 |item' )
  13. New Recommendation Model Two-tower model based on Graph Convolution Network

    Business Requirement ›Enhance serendipity Model Summary Pros ›Highly diverse recommendation ›Better performance reported [1,2] Cons ›Large number of parameters ›ANN search sometimes picks unexpected items
  14. Metrics for Model Assessment Sales via Stickers For You Total

    sales Sales divided into major and minor package Contents Via Stickers For You Other than Stickers For You Major package Minor package
  15. Metrics for Model Assessment Sales via Stickers For You Total

    sales Sales divided into major and minor package Contents Via Stickers For You Other than Stickers For You Major package Minor package
  16. Total Sales Lifted, but … Major package Minor package Total

    Sales Lift (%) from As-Is model
  17. Total Sales Lifted, but … Major package Minor package Total

    Sales Impression Gap between sales and impression lifts Lift (%) from As-Is model
  18. What happened? - e.g. Users bought Osaka-dialect sticker → Model

    offers [another prefecture]-dialect sticker - Feedback to Machine Learning department - Too-much feature abstraction on sticker metadata
  19. Improved Recommendation Model Simplified two-tower model based on GCN Business

    Requirements ›Enhance serendipity ›Moderate feature abstraction Model Summary ›Simplification by using Light Graph Convolution layer [3]. ›Automated hyperparameter optimization. ›Better performance than the previous model in an offline test.
  20. Total Sales Lifted even more with enough serendipity Lift (%)

    from As-Is model Major package Minor package Total Sales Impression 1st model
  21. Total Sales Lifted even more with enough serendipity Lift (%)

    from As-Is model Major package Minor package Total Major package Minor package Total Sales Impression 1st model 2nd model
  22. Wrap-up

  23. Continuous improvement of ML System Plan ML design Check A/B

    test Action Find improvement Do ML integration .- .- %4 .- %4
  24. Thank you

  25. [1] Inductive Representation Learning on Large Graphs - https://arxiv.org/abs/1706.02216 [2]

    Controllable Multi-Interest Framework for Recommendation - https://arxiv.org/abs/2005.09347 [3] LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation - https://arxiv.org/abs/2002.02126 Reference