Mutual Collaboration between Data Analysis and Machine Learning Development at LINE: Continuous Improvement in Sticker Recommendation Engine

Yu Otsuka
LINE / LINE Platform Data Science Team / Data Scientist

https://linedevday.linecorp.com/2021/ja/sessions/79
https://linedevday.linecorp.com/2021/en/sessions/79
https://linedevday.linecorp.com/2021/ko/sessions/79

Transcript

2. Self Introduction - Joined LINE Corporation as a Data Scientist in February 2019 - Working on data analysis for cross-service projects such as LINE POINT CLUB - Yu Otsuka
3. Agenda - Data Science & Machine Learning department - A case of DS & ML collaboration - Continuous improvement in sticker recommendation engine
4. - DS - Estimate business impact - Perform statistical analysis - ML - Provide tailor-made ML engine - Develop general ML platform DS & ML Department Data Science department Machine Learning department ML privacy team LPML department Data Science center
5. Continuous improvement of ML System Plan ML design Check A/B test Action Find improvement Do ML integration .- .- %4 .- %4
6. Continuous improvement in sticker recommendation engine A case of DS & ML collaboration
7. Stickers For You - Personalized recommendation
8. Stickers For You % purchase (within the top page) 40% # clicks per day 1.2M # packages 11M Purchase: As of September 2021 Click: As of August 2021 Package: As of April 2021
9. Role of “Stickers For You” is changed Smart Channel - Many sticker touchpoints out of sticker shop - Smart Channel - Home Tab - LINE Official Account - and more … Home Tab
10. Role of “Stickers For You” is changed 89% Paid users via both Stickers For You & others Paid users via Stickers For You only - Paid users via Stickers For You also use other touchpoints Period: FY2020
11. Role of “Stickers For You” is changed As-Is - Offering stickers that all you need - Require simple recommendation To-Be - Offering stickers that enhances serendipity - Require highly diverse recommendation - Re-definition of Stickers For You
12. As-Is Recommendation Model Naive Bayes-based collaborative filtering Business Requirement ›Simple recommendation Model Summary Pros ›Small number of parameters ›Very irrelevant items won’t be recommended Cons ›Poor exploration leads to no fresh items ›Poor generalization leads to cold start problem !(item' |f* , f, , . . . , f. ) ∝ !(item' )∏ 2 !(f2 |item' )
13. New Recommendation Model Two-tower model based on Graph Convolution Network Business Requirement ›Enhance serendipity Model Summary Pros ›Highly diverse recommendation ›Better performance reported [1,2] Cons ›Large number of parameters ›ANN search sometimes picks unexpected items
14. Metrics for Model Assessment Sales via Stickers For You Total sales Sales divided into major and minor package Contents Via Stickers For You Other than Stickers For You Major package Minor package
15. Metrics for Model Assessment Sales via Stickers For You Total sales Sales divided into major and minor package Contents Via Stickers For You Other than Stickers For You Major package Minor package
16. Total Sales Lifted, but … Major package Minor package Total Sales Lift (%) from As-Is model
17. Total Sales Lifted, but … Major package Minor package Total Sales Impression Gap between sales and impression lifts Lift (%) from As-Is model
18. What happened? - e.g. Users bought Osaka-dialect sticker → Model offers [another prefecture]-dialect sticker - Feedback to Machine Learning department - Too-much feature abstraction on sticker metadata
19. Improved Recommendation Model Simplified two-tower model based on GCN Business Requirements ›Enhance serendipity ›Moderate feature abstraction Model Summary ›Simplification by using Light Graph Convolution layer [3]. ›Automated hyperparameter optimization. ›Better performance than the previous model in an offline test.
20. Total Sales Lifted even more with enough serendipity Lift (%) from As-Is model Major package Minor package Total Sales Impression 1st model
21. Total Sales Lifted even more with enough serendipity Lift (%) from As-Is model Major package Minor package Total Major package Minor package Total Sales Impression 1st model 2nd model
22. Wrap-up
23. Continuous improvement of ML System Plan ML design Check A/B test Action Find improvement Do ML integration .- .- %4 .- %4
24. Thank you
25. [1] Inductive Representation Learning on Large Graphs - https://arxiv.org/abs/1706.02216 [2] Controllable Multi-Interest Framework for Recommendation - https://arxiv.org/abs/2005.09347 [3] LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation - https://arxiv.org/abs/2002.02126 Reference