Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Libra suite – In-house toolset to improve LINE's ML product

Libra suite – In-house toolset to improve LINE's ML product

Bo Zhang
LINE Machine Learning Planning Team Machine Learning Project Manager
https://linedevday.linecorp.com/2020/jp/sessions/1731
https://linedevday.linecorp.com/2020/en/sessions/1731

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 26, 2020
Tweet

Transcript

  1. None
  2. Agenda › Introduction › Components › Case study › Future

    work
  3. Introduction

  4. Haku Chou Who am I? › Joined LINE from June

    2019 › Working as a ML product/project manager › Data Labs Machine Learning Planning Team
  5. Brief introduction to Libra suite What is it? Offline evaluation

    Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
  6. Brief introduction to Libra suite What is it? Offline evaluation

    Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
  7. Brief introduction to Libra suite What is it? Offline evaluation

    Online evaluation Release Model deploy Generate demonstration Setup A/B test Monitor and report
  8. Architecture How does it work Test group B Test group

    A ML Demonstration Monitoring
  9. Architecture How does it work Test group B Test group

    A ML Demonstration Monitoring
  10. Architecture How does it work Test group B Test group

    A ML Demonstration Monitoring
  11. Architecture How does it work Test group B Test group

    A ML Demonstration Monitoring
  12. Issues until Libra suite Why we developed Libra suite? Suffered

    from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
  13. Issues until Libra suite Why we developed Libra suite? Suffered

    from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
  14. Issues until Libra suite Why we developed Libra suite? Suffered

    from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
  15. Issues until Libra suite Why we developed Libra suite? Suffered

    from huge communication cost to integrate AB test with service side Lack of qualitative analytics and evaluation tools for ML development Too long stand-by between ML solution’s AB test and release
  16. Missions of Libra suite What do we expect from Libra

    suite? To conduct more AB test efficiently with less cost To offer standard demonstration tools for qualitative evaluation To accelerate the “model deploy -> analysis -> report -> release” cycle
  17. Developers and users of Libra suite People involved in Libra

    suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
  18. Developers and users of Libra suite People involved in Libra

    suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
  19. Developers and users of Libra suite People involved in Libra

    suite Organization LINE's Data Labs ML team: core dev. DS team: core users and testers. DV team: UI/UX design. Main contributors 5 developers PM: 1 Engineer: 3 Designer: 1 Main target users LINE’s service side: PM or engineer Data Labs: data scientists ML engineers PM
  20. Components

  21. Components of Libra suite 3 components

  22. Libra › An in-house CMS to set A/B test up

    › Split users into multiple groups › Test spec. shared w/ other systems via JSON (for integration)
  23. Libra Setup the AB test and demonstration

  24. Libra Setup the AB test and demonstration

  25. Libra As of 2020/10/18 services Support 26 A/B tests Conducted

    462 users Used by 282
  26. Libra How many AB tests conducted in 2020 0 5

    10 15 20 25 30 35 40 Jan Feb Mar Apr May Jun Jul Aug Sep
  27. Libra report › Aggregate AB test logs › Real-time and

    batch data integration › Manage metrics › Generate dashboard
  28. Libra report AB test monitoring

  29. Libra report AB test monitoring

  30. Libra report As of 2020/10/18 services Support 8 A/B tests

    Recorded 78 min Minimum delay 10
  31. Libra checker › Setup demonstration › Gather demonstration data ›

    Generate demonstration web page
  32. Libra checker Generate the demonstration dashboard

  33. Libra checker Generate the demonstration dashboard

  34. Libra checker As of 2020/09/30 services Support 5 groups Candidate

    groups max 11 items Items per group max 100
  35. Case study

  36. › From CF to end2end DNN › 5 AB tests

    › 3 regions: Japan, Thailand and Taiwan › Conducted seamlessly in 3 months AB test for sticker recommendation
  37. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  38. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  39. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  40. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  41. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  42. 5 AB tests setup and ran by Libra The progress

    of sticker recommendation AB test Release Test #4 Further expanded target users in JP Test #2 DNN to DNN end2end Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  43. Bugs and oversights caught by Libra report The progress of

    sticker recommendation AB test Release Test #4 Further expanded target users in JP Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Test #3 Expanded target users in JP Test #1 Collaborative Filtering to DNN
  44. Bugs and oversights caught by Libra report The progress of

    sticker recommendation AB test Release Test #4 Further expanded target users in JP Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Deployment miss Delivery system setup miss Test #1 Collaborative Filtering to DNN
  45. Bugs and oversights caught by Libra report The progress of

    sticker recommendation AB test Release Side effect Unexpected effect on AD’s impression Logic flaw Recommended items overlapping Test #5 Independent tests in TH&TW Deployment miss Delivery system setup miss Test #1 Collaborative Filtering to DNN
  46. AB test result For SMARTCH sticker recommendation daily Sticker sales

    + 900K(¥) daily Impression + 50M for Japan users Target users 250%
  47. Future work

  48. The upcoming features to Libra suite › Support more LINE

    services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
  49. The upcoming features to Libra suite › Support more LINE

    services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
  50. The upcoming features to Libra suite › Support more LINE

    services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
  51. The upcoming features to Libra suite › Support more LINE

    services › Allow users to commit customized query via workflow engine like Airflow Libra report Libra checker › Add support for historical comparison demonstration › Add support for cross-domain recommendation demonstration Libra › Add support for service side creative AB test › Add support for AB test based on item2item recommendation Future work
  52. We are hiring! Join LINE’s machine learning team › https://linecorp.com/ja/career/position/1408

    Server Side Engineer Machine Learning Engineer › https://linecorp.com/ja/career/position/589 MLOps Engineer › https://linecorp.com/ja/career/position/2255
  53. Thank you