Data Science drives improvement of LINE messenger

Taro Takaguchi
LINE Data Science Team2 Senior Data Scientist / Manager
https://linedevday.linecorp.com/jp/2019/sessions/B1-3

Transcript

1. 2019 DevDay
   Data Science Drives Improvement
   of LINE Messenger
   > Taro Takaguchi
   > LINE Data Science Team2 Senior Data Scientist /
   Manager
   

   View Slide

2. Agenda
   > Introduction: Data Science Team
   > Challenges in LINE App Improvement
   > Improvement of “Create Group” Feature
   > Data Science Tools
   

   View Slide

3. LINE App Improvement Project
   Data Driven
   Diverse
   Team
   Users First
   In-house
   Development
   

   View Slide

4. Introduction: Data Science Team
   

   View Slide

5. Organization
   Data Science
   Team
   Engineering

   Infrastructure
   Data
   Platform
   Data Science And
   Engineering Center
   GROWTHY
   Platform
   Data Labs
   Machine
   Learning Team
   

   View Slide

6. Coverage of Services
   Data Science
   Team 2
   Data Science
   Team 1
   Data Science
   Team 3
   Data Science Team
   Data Science
   Team 4
   Stickers
   

   View Slide

7. Company-Wide Collaboration
   Data
   Science
   Service
   Plan
   Data Science Is a Part of Interwoven Team
   Client
   Development
   Server
   Development
   UI / UX
   Design
   Legal
   Info
   Security
   …
   Machine
   Learning
   Data
   Platform
   

   View Slide

8. Cycle of Projects
   User Research Development
   Test
   Feedback
   Plan
   

   View Slide

9. User Research
   Learning Inside & Outside of Log Data
   Focused Interview Online Survey Dashboard Monitoring
   

   View Slide

10. Plan & Development
    

    View Slide

11. Test & Feedback
    Online A/B Tests in
    our Team (2018 -)
    20+
    Metrics Monitored
    in a Test
    100+
    Users Targeted
    Globally in a
    Typical Test
    10M+
    > Online A/B testing in principle
    > In-depth data analysis of results
    

    View Slide

12. Challenges in LINE App
    Improvement
    

    View Slide

13. Many People Use LINE in Different Ways
    MAU
    (4 Main Regions)
    164M
    Message Types
    20+
    MAU (JP)
    82M
    

    View Slide

14. No Single KPI
    

    View Slide

15. Core Value of LINE App
    

    View Slide

16. Closing the Distance
    Easy and Pleasant Communication With Close Friends
    

    View Slide

17. Local Friend Network on LINE
    Me
    

    View Slide

18. Diverse and Active Social Contexts
    Family
    School
    

    View Slide

19. Group Feature
    

    View Slide

20. Improvement of “Create Group”
    Feature
    

    View Slide

21. Screen Order Was Not Intuitive?
    > 1. Set group icon and name
    > 2. Choose members to be invited
    User Research
    

    View Slide

22. Swap the Screen Order
    > 1. Choose members to be invited
    > 2. Set group icon and name
    Plan
    

    View Slide

23. No Difference
    > Lift in success rate of group creation
    Test
    JP TH TW ID
    No statistical significance
    0
    

    View Slide

24. Why Made No Difference?
    

    View Slide

25. Dimensions of Analysis
    > Uniqueness of each region?
    > Time-dependency?
    > Success users? Failed Users?
    > …
    Feedback
    

    View Slide

26. True Pain Point: Choosing Members Quickly
    JP
    TW
    TH
    ID
    Not proceed to “group icon and name” screen
    Feedback
    > Where failed users gave up?
    

    View Slide

27. Potential of Further Improvement
    Fail Group Creation
    On the Last Day
    Never Use Group
    Creation Before
    Use Group
    Creation Before
    Succeed Group
    Creation Before
    Fail Group
    Creation Before
    65%
    7% 28%
    Feedback
    

    View Slide

28. Show Recently Chatted Friends Section
    Plan
    > Scenario: 1-to-1 chat → Create group to invite friends
    

    View Slide

29. Plan B
    > Put everything into 1 step to shorten the process
    Plan
    

    View Slide

30. A/B Testing for 4 UI Patterns?
    Alphabetical + Recent Chat
    2 Step
    Control
    ʢAS-ISʣ
    Treatment 1
    1 Step Treatment 3 Treatment 2
    > Complete order: 6-pair comparisons
    > High false positive rate / Large sample size required
    Invitee list
    # Steps
    Test
    

    View Slide

31. Solution: 2-Pair Comparison
    > Factorize effects into 2 comparisons
    > Moderate false positive rate with small sample size
    Alphabetical + Recent Chat
    2 Step
    Control
    ʢAS-ISʣ
    Treatment 1
    1 Step Treatment 2
    Invitee list
    # Steps
    Test
    

    View Slide

32. Feedback
    > Lift in success rate of group creation
    JP: T1 JP: T2 TH: T1 TH: T2 TW: T1 TW: T2 ID: T1 ID: T2
    No statistical significance
    0
    No Difference
    

    View Slide

33. JP TW TH ID
    Users Can Find Members Quickly
    > Reduction in typical time to complete group creation
    * statistically significant
    * * *
    0
    Control (AS-IS) vs. 2-step + Recent chat
    Feedback
    

    View Slide

34. How About All-in-One Screen?
    > Users do not prefer all-in-one screen
    Feedback
    > Creation of 1-member groups ↑
    ↓
    > Success rate of ≥2-member groups
    

    View Slide

35. How About All-in-One Screen?
    > Users do not prefer all-in-one screen
    > Creation of 1-member groups ↑
    Feedback
    ↓
    > Success rate of ≥2-member groups
    > Switch ‘Create group’ → ‘Create chat’ ↑
    

    View Slide

36. How About All-in-One Screen?
    > Users do not prefer all-in-one screen
    > Creation of 1-member groups ↑
    > Switch ‘Create group’ → ‘Create chat’ ↑
    Feedback
    ↓
    > Success rate of ≥2-member groups
    > Click Done button twice ↑
    

    View Slide

37. Another Pain Point
    

    View Slide

38. Where Is “Create Group” Button?
    User Research
    > (B) “Add friends” menu
    > (C) “Create chat” menu in Chats tab
    > (A) Home tab
    > (D) No idea
    ✓
    ✓
    ✓
    

    View Slide

39. “Create Group” Option in Create Chat Menu
    Plan
    > Most of users know how to create a chat
    

    View Slide

40. JP TH TW ID
    More Users Create Group Successfully
    Test
    > Lift in the number of users completing group creation
    * statistically significant
    *
    *
    

    View Slide

41. All Good, Really?
    Feedback
    > Created groups just instead of chats?
    

    View Slide

42. Good in Total
    Feedback
    > More users in JP created groups or chats
    Group Chat Group or Chat
    Lift
    0
    

    View Slide

43. All Good, Really?
    Feedback
    > Eat user traffic to Home tab and Add friends menu ?
    

    View Slide

44. Good in Total
    Feedback
    > Add friends and official accounts
    > Click friends recommendation
    > View Home tab
    →
    →
    →
    > Click create chat button →
    > No negative effects on friending
    

    View Slide

45. Next Phase…
    > Increase of “active” groups
    User Research Development
    Test
    Feedback
    Plan
    

    View Slide

46. Data Science Tools
    

    View Slide

47. Steps for A/B Test Data Analysis
    1. Required sample size
    2. Target user slots
    3. Monitoring dashboard
    4. Test results report
    Data Scientist Planner
    Designer
    Developer
    

    View Slide

48. Steps for A/B Test Data Analysis
    1. Required sample size
    2. Target user slots
    3. Monitoring dashboard
    4. Test results report
    Data Scientist Planner
    Designer
    Developer
    

    View Slide

49. > Presto / SparkSQL / SparkR / PySpark / Markdown
    > Company-wide accessible (under permission control)
    

    View Slide

50. LIBRA
    > Split users into 1,024 randomized slots
    > No manual operation necessary
    Data Scientist
    Use Slots
    #654 & #987
    Test Spec
    Distribution Server
    Slot #654:
    C
    Slot #987:
    T
    

    View Slide

51. Steps for A/B Test Data Analysis
    1. Required sample size
    2. Target user slots
    3. Monitoring dashboard
    4. Test results Report
    Data Scientist Planner
    Designer
    Developer
    

    View Slide

52. LIBRA REPORT
    > Semi-automatic data summarization and visualization
    > Tables re-usable for test result analysis
    Data Scientist
    Processed
    Test Results
    Register
    Query
    

    View Slide

53. R Shiny
    > Handy and flexible dashboard
    > Early detection of unexpected logs
    Also see Poster #P-B2 presented by Motoyuki Oki (DS team 2)
    

    View Slide

54. Steps for A/B Test Data Analysis
    1. Required sample size
    2. Target user slots
    3. Monitoring dashboard
    4. Test results report
    Data Scientist Planner
    Designer
    Developer
    

    View Slide

55. Conflr
    > R Markdown → Atlassian Confluence wiki
    > OSS developed by our team member (https://github.com/line/conflr)
    

    View Slide

56. Wrap-Up
    

    View Slide

57. LINE App Improvement Project
    Data Driven
    Diverse
    Team
    Users First
    In-house
    Development
    

    View Slide

58. Thank You
    

    View Slide