Data Science drives improvement of LINE messenger

Data Science drives improvement of LINE messenger

Taro Takaguchi
LINE Data Science Team2 Senior Data Scientist / Manager
https://linedevday.linecorp.com/jp/2019/sessions/B1-3

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Data Science Drives Improvement of LINE Messenger >

    Taro Takaguchi > LINE Data Science Team2 Senior Data Scientist / Manager
  2. Agenda > Introduction: Data Science Team > Challenges in LINE

    App Improvement > Improvement of “Create Group” Feature > Data Science Tools
  3. LINE App Improvement Project Data Driven Diverse Team Users First

    In-house Development
  4. Introduction: Data Science Team

  5. Organization Data Science Team Engineering
 Infrastructure Data Platform Data Science

    And Engineering Center GROWTHY Platform Data Labs Machine Learning Team
  6. Coverage of Services Data Science Team 2 Data Science Team

    1 Data Science Team 3 Data Science Team Data Science Team 4 Stickers
  7. Company-Wide Collaboration Data Science Service Plan Data Science Is a

    Part of Interwoven Team Client Development Server Development UI / UX Design Legal Info Security … Machine Learning Data Platform
  8. Cycle of Projects User Research Development Test Feedback Plan

  9. User Research Learning Inside & Outside of Log Data Focused

    Interview Online Survey Dashboard Monitoring
  10. Plan & Development

  11. Test & Feedback Online A/B Tests in our Team (2018

    -) 20+ Metrics Monitored in a Test 100+ Users Targeted Globally in a Typical Test 10M+ > Online A/B testing in principle > In-depth data analysis of results
  12. Challenges in LINE App Improvement

  13. Many People Use LINE in Different Ways MAU (4 Main

    Regions) 164M Message Types 20+ MAU (JP) 82M
  14. No Single KPI

  15. Core Value of LINE App

  16. Closing the Distance Easy and Pleasant Communication With Close Friends

  17. Local Friend Network on LINE Me

  18. Diverse and Active Social Contexts Family School

  19. Group Feature

  20. Improvement of “Create Group” Feature

  21. Screen Order Was Not Intuitive? > 1. Set group icon

    and name > 2. Choose members to be invited User Research
  22. Swap the Screen Order > 1. Choose members to be

    invited > 2. Set group icon and name Plan
  23. No Difference > Lift in success rate of group creation

    Test JP TH TW ID No statistical significance 0
  24. Why Made No Difference?

  25. Dimensions of Analysis > Uniqueness of each region? > Time-dependency?

    > Success users? Failed Users? > … Feedback
  26. True Pain Point: Choosing Members Quickly JP TW TH ID

    Not proceed to “group icon and name” screen Feedback > Where failed users gave up?
  27. Potential of Further Improvement Fail Group Creation On the Last

    Day Never Use Group Creation Before Use Group Creation Before Succeed Group Creation Before Fail Group Creation Before 65% 7% 28% Feedback
  28. Show Recently Chatted Friends Section Plan > Scenario: 1-to-1 chat

    → Create group to invite friends
  29. Plan B > Put everything into 1 step to shorten

    the process Plan
  30. A/B Testing for 4 UI Patterns? Alphabetical + Recent Chat

    2 Step Control ʢAS-ISʣ Treatment 1 1 Step Treatment 3 Treatment 2 > Complete order: 6-pair comparisons > High false positive rate / Large sample size required Invitee list # Steps Test
  31. Solution: 2-Pair Comparison > Factorize effects into 2 comparisons >

    Moderate false positive rate with small sample size Alphabetical + Recent Chat 2 Step Control ʢAS-ISʣ Treatment 1 1 Step Treatment 2 Invitee list # Steps Test
  32. Feedback > Lift in success rate of group creation JP:

    T1 JP: T2 TH: T1 TH: T2 TW: T1 TW: T2 ID: T1 ID: T2 No statistical significance 0 No Difference
  33. JP TW TH ID Users Can Find Members Quickly >

    Reduction in typical time to complete group creation * statistically significant * * * 0 Control (AS-IS) vs. 2-step + Recent chat Feedback
  34. How About All-in-One Screen? > Users do not prefer all-in-one

    screen Feedback > Creation of 1-member groups ↑ ↓ > Success rate of ≥2-member groups
  35. How About All-in-One Screen? > Users do not prefer all-in-one

    screen > Creation of 1-member groups ↑ Feedback ↓ > Success rate of ≥2-member groups > Switch ‘Create group’ → ‘Create chat’ ↑
  36. How About All-in-One Screen? > Users do not prefer all-in-one

    screen > Creation of 1-member groups ↑ > Switch ‘Create group’ → ‘Create chat’ ↑ Feedback ↓ > Success rate of ≥2-member groups > Click Done button twice ↑
  37. Another Pain Point

  38. Where Is “Create Group” Button? User Research > (B) “Add

    friends” menu > (C) “Create chat” menu in Chats tab > (A) Home tab > (D) No idea ✓ ✓ ✓
  39. “Create Group” Option in Create Chat Menu Plan > Most

    of users know how to create a chat
  40. JP TH TW ID More Users Create Group Successfully Test

    > Lift in the number of users completing group creation * statistically significant * *
  41. All Good, Really? Feedback > Created groups just instead of

    chats?
  42. Good in Total Feedback > More users in JP created

    groups or chats Group Chat Group or Chat Lift 0
  43. All Good, Really? Feedback > Eat user traffic to Home

    tab and Add friends menu ?
  44. Good in Total Feedback > Add friends and official accounts

    > Click friends recommendation > View Home tab → → → > Click create chat button → > No negative effects on friending
  45. Next Phase… > Increase of “active” groups User Research Development

    Test Feedback Plan
  46. Data Science Tools

  47. Steps for A/B Test Data Analysis 1. Required sample size

    2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
  48. Steps for A/B Test Data Analysis 1. Required sample size

    2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
  49. > Presto / SparkSQL / SparkR / PySpark / Markdown

    > Company-wide accessible (under permission control)
  50. LIBRA > Split users into 1,024 randomized slots > No

    manual operation necessary Data Scientist Use Slots #654 & #987 Test Spec Distribution Server Slot #654: C Slot #987: T
  51. Steps for A/B Test Data Analysis 1. Required sample size

    2. Target user slots 3. Monitoring dashboard 4. Test results Report Data Scientist Planner Designer Developer
  52. LIBRA REPORT > Semi-automatic data summarization and visualization > Tables

    re-usable for test result analysis Data Scientist Processed Test Results Register Query
  53. R Shiny > Handy and flexible dashboard > Early detection

    of unexpected logs Also see Poster #P-B2 presented by Motoyuki Oki (DS team 2)
  54. Steps for A/B Test Data Analysis 1. Required sample size

    2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
  55. Conflr > R Markdown → Atlassian Confluence wiki > OSS

    developed by our team member (https://github.com/line/conflr)
  56. Wrap-Up

  57. LINE App Improvement Project Data Driven Diverse Team Users First

    In-house Development
  58. Thank You