Libra suite – In-house toolset to improve LINE's ML product

Bo Zhang
LINE Machine Learning Planning Team Machine Learning Project Manager
https://linedevday.linecorp.com/2020/jp/sessions/1731
https://linedevday.linecorp.com/2020/en/sessions/1731

Transcript

2. Agenda › Introduction › Components › Case study › Future work
3. Introduction
4. Haku Chou Who am I? › Joined LINE from June 2019 › Working as a ML product/project manager › Data Labs Machine Learning Planning Team
5. Brief introduction to Libra suite What is it? Offline evaluation Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
6. Brief introduction to Libra suite What is it? Offline evaluation Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
7. Brief introduction to Libra suite What is it? Offline evaluation Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
8. Architecture How does it work Test group B Test group A ML Demonstration Monitoring
9. Architecture How does it work Test group B Test group A ML Demonstration Monitoring
10. Architecture How does it work Test group B Test group A ML Demonstration Monitoring
11. Architecture How does it work Test group B Test group A ML Demonstration Monitoring
12. Issues until Libra suite Why we developed Libra suite? Suffered from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
13. Issues until Libra suite Why we developed Libra suite? Suffered from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
14. Issues until Libra suite Why we developed Libra suite? Suffered from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
15. Issues until Libra suite Why we developed Libra suite? Suffered from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
16. Missions of Libra suite What do we expect from Libra suite? To conduct more AB test efficiently with less cost To offer standard demonstration tools for qualitative evaluation To accelerate the “model deploy -> analysis -> report -> release” cycle
17. Developers and users of Libra suite People involved in Libra suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
18. Developers and users of Libra suite People involved in Libra suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
19. Developers and users of Libra suite People involved in Libra suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
20. Components
21. Components of Libra suite 3 components
22. Libra › An in-house CMS to set A/B test up › Split users into multiple groups › Test spec. shared w/ other systems via JSON (for integration)
23. Libra Setup the AB test and demonstration
24. Libra Setup the AB test and demonstration
25. Libra As of 2020/10/18 services Support 26 A/B tests Conducted 462 users Used by 282
26. Libra How many AB tests conducted in 2020 0 5 10 15 20 25 30 35 40 Jan Feb Mar Apr May Jun Jul Aug Sep
27. Libra report › Aggregate AB test logs › Real-time and batch data integration › Manage metrics › Generate dashboard
28. Libra report AB test monitoring
29. Libra report AB test monitoring
30. Libra report As of 2020/10/18 services Support 8 A/B tests Recorded 78 min Minimum delay 10
31. Libra checker › Setup demonstration › Gather demonstration data › Generate demonstration web page
32. Libra checker Generate the demonstration dashboard
33. Libra checker Generate the demonstration dashboard
34. Libra checker As of 2020/09/30 services Support 5 groups Candidate groups max 11 items Items per group max 100
35. Case study
36. › From CF to end2end DNN › 5 AB tests › 3 regions: Japan, Thailand and Taiwan › Conducted seamlessly in 3 months AB test for sticker recommendation
37. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
38. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
39. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
40. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
41. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
42. 5 AB tests setup and ran by Libra The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
43. Bugs and oversights caught by Libra report The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
44. Bugs and oversights caught by Libra report The progress of sticker recommendation AB test Release Test #4 Further expanded target users in JP Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Deployment miss Delivery system setup miss Test #1 Collaborative Filtering to DNN
45. Bugs and oversights caught by Libra report The progress of sticker recommendation AB test Release Side effect Unexpected effect on AD’s impression Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Deployment miss Delivery system setup miss Test #1 Collaborative Filtering to DNN
46. AB test result For SMARTCH sticker recommendation daily Sticker sales + 900K(¥) daily Impression + 50M for Japan users Target users 250%
47. Future work
48. The upcoming features to Libra suite › Support more LINE services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
49. The upcoming features to Libra suite › Support more LINE services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
50. The upcoming features to Libra suite › Support more LINE services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
51. The upcoming features to Libra suite › Support more LINE services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
52. We are hiring! Join LINE’s machine learning team › https://linecorp.com/ja/career/position/1408 Server Side Engineer Machine Learning Engineer › https://linecorp.com/ja/career/position/589 MLOps Engineer › https://linecorp.com/ja/career/position/2255
53. Thank you