Data Science drives improvement of LINE messenger

Taro Takaguchi
LINE Data Science Team2 Senior Data Scientist / Manager
https://linedevday.linecorp.com/jp/2019/sessions/B1-3

Transcript

1. 2019 DevDay Data Science Drives Improvement of LINE Messenger > Taro Takaguchi > LINE Data Science Team2 Senior Data Scientist / Manager
2. Agenda > Introduction: Data Science Team > Challenges in LINE App Improvement > Improvement of “Create Group” Feature > Data Science Tools
3. LINE App Improvement Project Data Driven Diverse Team Users First In-house Development
4. Introduction: Data Science Team
5. Organization Data Science Team Engineering
 Infrastructure Data Platform Data Science And Engineering Center GROWTHY Platform Data Labs Machine Learning Team
6. Coverage of Services Data Science Team 2 Data Science Team 1 Data Science Team 3 Data Science Team Data Science Team 4 Stickers
7. Company-Wide Collaboration Data Science Service Plan Data Science Is a Part of Interwoven Team Client Development Server Development UI / UX Design Legal Info Security … Machine Learning Data Platform
8. Cycle of Projects User Research Development Test Feedback Plan
9. User Research Learning Inside & Outside of Log Data Focused Interview Online Survey Dashboard Monitoring
10. Plan & Development
11. Test & Feedback Online A/B Tests in our Team (2018 -) 20+ Metrics Monitored in a Test 100+ Users Targeted Globally in a Typical Test 10M+ > Online A/B testing in principle > In-depth data analysis of results
12. Challenges in LINE App Improvement
13. Many People Use LINE in Different Ways MAU (4 Main Regions) 164M Message Types 20+ MAU (JP) 82M
14. No Single KPI
15. Core Value of LINE App
16. Closing the Distance Easy and Pleasant Communication With Close Friends
17. Local Friend Network on LINE Me
18. Diverse and Active Social Contexts Family School
19. Group Feature
20. Improvement of “Create Group” Feature
21. Screen Order Was Not Intuitive? > 1. Set group icon and name > 2. Choose members to be invited User Research
22. Swap the Screen Order > 1. Choose members to be invited > 2. Set group icon and name Plan
23. No Difference > Lift in success rate of group creation Test JP TH TW ID No statistical significance 0
24. Why Made No Difference?
25. Dimensions of Analysis > Uniqueness of each region? > Time-dependency? > Success users? Failed Users? > … Feedback
26. True Pain Point: Choosing Members Quickly JP TW TH ID Not proceed to “group icon and name” screen Feedback > Where failed users gave up?
27. Potential of Further Improvement Fail Group Creation On the Last Day Never Use Group Creation Before Use Group Creation Before Succeed Group Creation Before Fail Group Creation Before 65% 7% 28% Feedback
28. Show Recently Chatted Friends Section Plan > Scenario: 1-to-1 chat → Create group to invite friends
29. Plan B > Put everything into 1 step to shorten the process Plan
30. A/B Testing for 4 UI Patterns? Alphabetical + Recent Chat 2 Step Control ʢAS-ISʣ Treatment 1 1 Step Treatment 3 Treatment 2 > Complete order: 6-pair comparisons > High false positive rate / Large sample size required Invitee list # Steps Test
31. Solution: 2-Pair Comparison > Factorize effects into 2 comparisons > Moderate false positive rate with small sample size Alphabetical + Recent Chat 2 Step Control ʢAS-ISʣ Treatment 1 1 Step Treatment 2 Invitee list # Steps Test
32. Feedback > Lift in success rate of group creation JP: T1 JP: T2 TH: T1 TH: T2 TW: T1 TW: T2 ID: T1 ID: T2 No statistical significance 0 No Difference
33. JP TW TH ID Users Can Find Members Quickly > Reduction in typical time to complete group creation * statistically significant * * * 0 Control (AS-IS) vs. 2-step + Recent chat Feedback
34. How About All-in-One Screen? > Users do not prefer all-in-one screen Feedback > Creation of 1-member groups ↑ ↓ > Success rate of ≥2-member groups
35. How About All-in-One Screen? > Users do not prefer all-in-one screen > Creation of 1-member groups ↑ Feedback ↓ > Success rate of ≥2-member groups > Switch ‘Create group’ → ‘Create chat’ ↑
36. How About All-in-One Screen? > Users do not prefer all-in-one screen > Creation of 1-member groups ↑ > Switch ‘Create group’ → ‘Create chat’ ↑ Feedback ↓ > Success rate of ≥2-member groups > Click Done button twice ↑
37. Another Pain Point
38. Where Is “Create Group” Button? User Research > (B) “Add friends” menu > (C) “Create chat” menu in Chats tab > (A) Home tab > (D) No idea ✓ ✓ ✓
39. “Create Group” Option in Create Chat Menu Plan > Most of users know how to create a chat
40. JP TH TW ID More Users Create Group Successfully Test > Lift in the number of users completing group creation * statistically significant * *
41. All Good, Really? Feedback > Created groups just instead of chats?
42. Good in Total Feedback > More users in JP created groups or chats Group Chat Group or Chat Lift 0
43. All Good, Really? Feedback > Eat user traffic to Home tab and Add friends menu ?
44. Good in Total Feedback > Add friends and official accounts > Click friends recommendation > View Home tab → → → > Click create chat button → > No negative effects on friending
45. Next Phase… > Increase of “active” groups User Research Development Test Feedback Plan
46. Data Science Tools
47. Steps for A/B Test Data Analysis 1. Required sample size 2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
48. Steps for A/B Test Data Analysis 1. Required sample size 2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
49. > Presto / SparkSQL / SparkR / PySpark / Markdown > Company-wide accessible (under permission control)
50. LIBRA > Split users into 1,024 randomized slots > No manual operation necessary Data Scientist Use Slots #654 & #987 Test Spec Distribution Server Slot #654: C Slot #987: T
51. Steps for A/B Test Data Analysis 1. Required sample size 2. Target user slots 3. Monitoring dashboard 4. Test results Report Data Scientist Planner Designer Developer
52. LIBRA REPORT > Semi-automatic data summarization and visualization > Tables re-usable for test result analysis Data Scientist Processed Test Results Register Query
53. R Shiny > Handy and flexible dashboard > Early detection of unexpected logs Also see Poster #P-B2 presented by Motoyuki Oki (DS team 2)
54. Steps for A/B Test Data Analysis 1. Required sample size 2. Target user slots 3. Monitoring dashboard 4. Test results report Data Scientist Planner Designer Developer
55. Conflr > R Markdown → Atlassian Confluence wiki > OSS developed by our team member (https://github.com/line/conflr)
56. Wrap-Up
57. LINE App Improvement Project Data Driven Diverse Team Users First In-house Development
58. Thank You